Thank you, Keith. Sure, try here: https://arcprize.org/arc-agi/3 (note that this is the public set, which means the games are easier than those GPT-5.6 got a 7.8% on. It won't seem easy at first, but keep trying)
The goal of ARC-AGI is not to prove that a model has human-level intelligence, but to refute that it is the case. The philosophy of the test is Popper's falsification approach. Should GPT reach 100%, it will still be irrelevant. These benchmarks are saturated always in the same way: manually creating large datasets with games similar in spirit to those in ARC-AGI 3, so that the model will be prepared and optimized for the test. But intelligence consists precisely in facing completely novel situations: saturating the benchmark by accumulating experience defeats the purpose.
I wonder a lot about hill climbing and overfitting to benchmarks though. The ARC tests are supposed to measure fluid intelligence, yet qualitatively even cutting edge LLM fluid intelligence feels a lot worse in real world scenarios than, say, those ARC 2 results would suggest.
Fascinating, instructive and well worth the read. I am wondering if is it possible for humans to try ARC-AGI-3 type tests?
Thank you, Keith. Sure, try here: https://arcprize.org/arc-agi/3 (note that this is the public set, which means the games are easier than those GPT-5.6 got a 7.8% on. It won't seem easy at first, but keep trying)
The goal of ARC-AGI is not to prove that a model has human-level intelligence, but to refute that it is the case. The philosophy of the test is Popper's falsification approach. Should GPT reach 100%, it will still be irrelevant. These benchmarks are saturated always in the same way: manually creating large datasets with games similar in spirit to those in ARC-AGI 3, so that the model will be prepared and optimized for the test. But intelligence consists precisely in facing completely novel situations: saturating the benchmark by accumulating experience defeats the purpose.
Yeah
I wonder a lot about hill climbing and overfitting to benchmarks though. The ARC tests are supposed to measure fluid intelligence, yet qualitatively even cutting edge LLM fluid intelligence feels a lot worse in real world scenarios than, say, those ARC 2 results would suggest.
Certainly. ARC-AGI-2 is still a much easier challenge than life