Ultra-Large AI Models Are Over
The end of "scale is all you need" is near
I don’t mean ‘over’ as in “you won’t see a new large AI model ever again” but as in “AI companies have reasons to not pursue them as a core research goal—indefinitely.”
Don’t get me wrong. This article isn’t a critique of the past years—even if I don’t buy the “scale is all you need” argument, I acknowledge just how far scaling has advanced the field.
Parallelism can be drawn between the 2020-2022 scaling race and—keeping the distance—the 50s-70s space race. Both advanced science significantly as a byproduct of other intentions.
But there’s a key distinction.
While space exploration was innovative in nature, the quest for novelty isn’t present in the “bigger is better” AI trend: To conquer space, the US and USSR had to design novel paths toward a clear goal. In contrast, AI companies have blindly followed a predefined path without knowing why or whether it’d lead us anywhere.
You can’t put the cart before the horse.
That makes all the di…