Ultra-Large AI Models Are Over

The end of "scale is all you need" is near

Ultra-large skyscrapers. Credit: Author via Midjourney

I don’t mean ‘over’ as in “you won’t see a new large AI model ever again” but as in “AI companies have reasons to not pursue them as a core research goal—indefinitely.”

Don’t get me wrong. This article isn’t a critique of the past years—even if I don’t buy the “scale is all you need” argument, I acknowledge just how far scaling has advanced the field.

Parallelism can be drawn between the 2020-2022 scaling race and—keeping the distance—the 50s-70s space race. Both advanced science significantly as a byproduct of other intentions.

But there’s a key distinction.

While space exploration was innovative in nature, the quest for novelty isn’t present in the “bigger is better” AI trend: To conquer space, the US and USSR had to design novel paths toward a clear goal. In contrast, AI companies have blindly followed a predefined path without knowing why or whether it’d lead us anywhere.

You can’t put the cart before the horse.

That makes all the di…