The Algorithmic Bridge

The Algorithmic Bridge

Share this post

The Algorithmic Bridge
The Algorithmic Bridge
Size Doesn't Matter
Copy link
Facebook
Email
Notes
More

Size Doesn't Matter

The era of large AI models is over

Alberto Romero's avatar
Alberto Romero
Apr 25, 2023
∙ Paid
28

Share this post

The Algorithmic Bridge
The Algorithmic Bridge
Size Doesn't Matter
Copy link
Facebook
Email
Notes
More
6
3
Share
Midjourney

Just kidding: we all know size matters. It's definitely true for AI models, especially for those trained on text data, i.e., language models (LMs). If there's one trend that has, above all others, dominated AI in the last five or six years, it is the steady increase in parameter count of the best models, which I've seen referred to as Moore’s law for large LMs. The GPT family is the clearest—albeit not the only—embodiment of this fact: GPT-2 was 1.5 billion parameters, GPT-3 was 175 billion, ~100x its predecessor, and rumors have it that GPT-4’s size, although officially undisclosed, has reached the 1 trillion mark. Not an exponential curve but definitely a growing one.

OpenAI was categorically following the godsend guidance of the scaling laws they themselves discovered in 2020 (that DeepMind later refined in 2022). The main takeaway is that size matters a lot. DeepMind revealed that other variables like the amount of training data, or its quality, also influence performance…

This post is for paid subscribers

Already a paid subscriber? Sign in
© 2025 Alberto Romero
Privacy ∙ Terms ∙ Collection notice
Start writingGet the app
Substack is the home for great culture

Share

Copy link
Facebook
Email
Notes
More