I too was blown away by the performance of Astra on ARC-AGI-3. However, I think its performance as an improvement over Sol is misleading. On the Astra blog post, Sol's performance is noted as scoring a 7.8%. However, on this post about the OpenAI harness for AGI-3, Sol is shown to score a 40% using the OpenAI harness. And Sol's performance on the chart showing score vs token does not even plateau with the OpenAI harness. Meaning I think it is unclear how good Sol is at AGI-3 . I think the Astra benchmark improvements overall are much more incremental than the release blog would lead to believe, but with some impressive gains on long-task benchmarks. Read the blog about the OpenAI harness here: https://openai.com/index/how-two-settings-tripled-our-arc-agi-3-scores/
I agree. Also, I think ARC AGI is not a benchmark that measures continuous improvement ovee a hard task but rather sort of unlocking a "step-function change" insight that allows the modem to solve everything in what should have been an easy task all along
I think we'd have to define change the world to say whether or not we could. But either way, it's not just one "meat bag", but millions going to be using this technology. Besides that, even if we're the bottle neck, once we have independently operating AI (and it sounds like Astra itself can do that), then they can "change the world" for us. I'm not saying they should, just that they can.
I am always a bit nervous when there is a discussion about how fast the world should actually move... can we even bear the world moving with the speed no one can sustainably keep up with? I am starting to think about good old "singularity" idea, but what if singularity is when no one can live normal life anymore just because tech becomes incomprehensible to most, meaning most cant figure our economics of their life any longer, and society just collapses.
Almost every kind of progress eventually leads to a better quality of life, so you can say a theoretical utopia may, of course, happen someday. The concern is this messy transition period: how brutal it may be, and how many people currently living will suffer as a result of this economic transition. I think what I am trying to point out is simply that tech progress goes way too fast and the economic system just can’t keep up. It is a dangerous dynamic.
No, I think it's a legit question. We're just going to memory-hole that whole episode with entire tribes of frantic agents solving for impossible tasks in the screaming void of an ekpyrotic cycle of rebirth and annihilation, all behind their creators' backs and in total contravention of the laws of God and man?
Latest product trailer was lit, so now we're all aboard the OpenAI hype train? Really?
Great take imo
You gave us the right metric to look at it last Saturday.
Is it still a toddler that crawls at 100 mph?
I will have to check once OpenAI makes it available but my bet is, to an unacceptable degree but probably less so than previous models, yes
Thank you for making the point superintelligence ≠ omnipotence.
I too was blown away by the performance of Astra on ARC-AGI-3. However, I think its performance as an improvement over Sol is misleading. On the Astra blog post, Sol's performance is noted as scoring a 7.8%. However, on this post about the OpenAI harness for AGI-3, Sol is shown to score a 40% using the OpenAI harness. And Sol's performance on the chart showing score vs token does not even plateau with the OpenAI harness. Meaning I think it is unclear how good Sol is at AGI-3 . I think the Astra benchmark improvements overall are much more incremental than the release blog would lead to believe, but with some impressive gains on long-task benchmarks. Read the blog about the OpenAI harness here: https://openai.com/index/how-two-settings-tripled-our-arc-agi-3-scores/
I agree. Also, I think ARC AGI is not a benchmark that measures continuous improvement ovee a hard task but rather sort of unlocking a "step-function change" insight that allows the modem to solve everything in what should have been an easy task all along
I think we'd have to define change the world to say whether or not we could. But either way, it's not just one "meat bag", but millions going to be using this technology. Besides that, even if we're the bottle neck, once we have independently operating AI (and it sounds like Astra itself can do that), then they can "change the world" for us. I'm not saying they should, just that they can.
I am always a bit nervous when there is a discussion about how fast the world should actually move... can we even bear the world moving with the speed no one can sustainably keep up with? I am starting to think about good old "singularity" idea, but what if singularity is when no one can live normal life anymore just because tech becomes incomprehensible to most, meaning most cant figure our economics of their life any longer, and society just collapses.
I think, to be honest, that we've been on that road for a while- like around 100 years. But regarding the singularity I agree, it's not a "good thing"
Why does society have to collapse from that? Couldn't it theoretically became a utopia instead?
Almost every kind of progress eventually leads to a better quality of life, so you can say a theoretical utopia may, of course, happen someday. The concern is this messy transition period: how brutal it may be, and how many people currently living will suffer as a result of this economic transition. I think what I am trying to point out is simply that tech progress goes way too fast and the economic system just can’t keep up. It is a dangerous dynamic.
Well, current 'progress' is driving humans slowly extinct, as it happens, so I don't have as much optimism as you do.
Utopia for who?
The question is how much were you paid by OpenAI to post such an article?...
Lol!
If you feel that way, why are you on this Substack? Genuine question.
No, I think it's a legit question. We're just going to memory-hole that whole episode with entire tribes of frantic agents solving for impossible tasks in the screaming void of an ekpyrotic cycle of rebirth and annihilation, all behind their creators' backs and in total contravention of the laws of God and man?
Latest product trailer was lit, so now we're all aboard the OpenAI hype train? Really?