The Algorithmic Bridge

The Algorithmic Bridge

Share this post

The Algorithmic Bridge
The Algorithmic Bridge
PhDs Fail This 5th-Grade Riddle! Can You Solve It?
Copy link
Facebook
Email
Notes
More

PhDs Fail This 5th-Grade Riddle! Can You Solve It?

Sorry for the clickbait title, except it's not

Alberto Romero's avatar
Alberto Romero
Jun 26, 2024
∙ Paid
23

Share this post

The Algorithmic Bridge
The Algorithmic Bridge
PhDs Fail This 5th-Grade Riddle! Can You Solve It?
Copy link
Facebook
Email
Notes
More
5
2
Share
The Chess Players by Honoré Daumier, 1863-1867

A blog about AI that’s actually about people

I. The simple puzzle that eludes the best AIs

Studying the skill level of artificial intelligence systems is revealing. A stark contrast emerges when they perform at their best vs. their worst. Or when they succeed at the most difficult challenges but fail at the easiest.

The two best models in the world, Anthropic’s Claude Sonnet 3.5 and OpenAI’s GPT-4o surpass the 50% mark on the hardest reasoning benchmark, the GPQA (graduate-level “Google-Proof Q&A”). Here are a couple of examples from the GPQA paper:

Source

I’ve looked at a dozen questions from the benchmark (just for reference, I studied Aerospace Engineering and I’m a science enthusiast). My undergrad knowledge and expertise combined with a passion for learning would get me a ∼0%.

But while AI juggles quantum mechanics and organic chemistry, it struggles with this:

Sören Mindermann

I’ve done a few IQ tests—I’m familiar with this kind of puzzle—and I’m not a kid anymore so I’m not really proud to say this but I saw the solution right away—took me literally a few seconds. It’s so easy that people who know how IQ tests work may try at first to find a harder solution than the actual one.

I’d bet an average fifth-grader can solve this. Smart younger kids would, too.

How can Claude 3.5 and GPT-4o solve PhD-level problems but fail this puzzle?

This post is for paid subscribers

Already a paid subscriber? Sign in
© 2025 Alberto Romero
Privacy ∙ Terms ∙ Collection notice
Start writingGet the app
Substack is the home for great culture

Share

Copy link
Facebook
Email
Notes
More