Clues Say Generative AI’s Future Will Be Revealed in 2024

To Zuckerberg and Altman: Alea Iacta Est

2024 is a special year for generative AI but for different reasons than the 2020-2023 period.

If I had to choose a word to define the previous years, I’d say growth.

In just four years, generative AI has been transformed from an abstract wish into a heterogeneous space that involves many types of companies working in synergy: model makers like OpenAI, data labelers like Scale AI, hardware vendors like Nvidia, device innovators like Rabbit, and narrow services built on top of language models like Jasper.

All of them pursue the same goal in one way or another: establishing generative AI as a respectable and mature industry.

Although some people defied the premise that generative AI was useful at all, almost no one denied it was a new thing on which we could place high expectations. We expected growth and we had growth. At the level of both research and products (just try to count the number of published worthwhile papers or founded AI startups).

So it’s growth that best defines the early period of the generative AI explosion, which goes from OpenAI’s GPT-3 in June 2020 (even GPT-2 in 2019 if we stretch it a little) to Google DeepMind’s Gemini in December 2023.

This one will also be a year of growth, of course, but it won’t be growth that’ll define it the best. We’re entering a new phase and a more appropriate word is required.

2024 will be the year of revelations.

Is GPT-4 just too good or the best we can get?

2023 began with a surprise. OpenAI’s GPT-4 was much better than anything else.

It ended with another — of opposite sentiment: The long-awaited Google DeepMind’s Gemini was not that much better than GPT-4 (for some people, it was even slightly inferior and only appeared to be better due to benchmark shenanigans).

The reason for both surprises was, interestingly, the same: GPT-4 is 2022 tech.

One year and a half after OpenAI finished training the model (summer 2022), no AI company in the world (including OpenAI) had managed to build and release a better system. GPT-4 created a gap so great between OpenAI and its competitors that it became effectively a moat.

Smaller AI startups and labs pursued OpenAI’s successes tapping into the power of open source, led by Meta and its Llama family, but none of them got anywhere close to GPT-4’s performance. Neither did Anthropic’s Claude 2 or Microsoft’s models or Amazon’s models. (The LLM arena leaderboard is the only reliable benchmark nowadays, you can check the podium: GPT-4, GPT-4 and, you guessed it, GPT-4).

Finally, everyone’s hope that promised to embody a new step forward (at least for those like me, looking from the sidelines) also failed.

Gemini Ultra missed the mark.

We could argue that Google DeepMind simply isn’t as good as OpenAI. That Gemini is not much better than a 1.5-year older AI model because the company that built it’s not as great. But that’s a hardly defendable argument when DeepMind built AlphaGo and AlphaFold, then Chinchilla and Gopher, and finally Gemini itself.

DeepMind’s history of amazing AI developments is, arguably, greater than OpenAI’s.

The other possibility, which I understand why AI people avoid thinking about, is that we’re very close to the ceiling of generative AI with the current state of the technology. Without breakthroughs (that might or might not happen), we won’t get to the next level soon enough. If that happens, expectations will suffer an unexpected blowback.

The Bitter Lesson applies only during times when ingenuous algorithmic breakthroughs aren’t needed. When they are, it breaks apart. If we’re entering one such period, neither scale, data, nor compute will get us to the other side.

So, it’s GPT-4 that good or it’s just the best we can get with what we have now?

Mark Zuckerberg’s strong bet on generative AI

2024 is an important year for another reason. Not only because we might hit what looks like an algorithmic ceiling (we don’t know for sure), but also because the stakes are at an all-time high.

If the ceiling is even there, it might not last long because a couple of very well-known AI companies promise to break it with everything they have.

Enter Meta CEO Mark Zuckerberg.

In a much-commented short video that Zuck shared on Threads, he stated explicitly (as far as I know, for the first time) that Meta’s goal is to build artificial general intelligence (AGI) and open-sourcing it. The same goal that OpenAI and DeepMind have been pursuing for years (except the part of making it open source, which has received substantial praise and surely has Altman and Hassabis shaking at home).

Zuck revealed that, by the end of 2024, Meta will have ~350,000 Nvidia H100s (in total, the equivalent of ~600,000 H100s if they include other GPUs). That is, as some have estimated, $20 billion worth of compute ready to train next-generation AI models.

He also said what we already knew: Meta is training Llama 3 and, as Yann LeCun hinted, the company will open-source it as they did with Llama 2 and LLaMA. (Note the definition of open source is looser than it once was; we know nothing about the data the models were trained on, for instance.)

In short: Meta is going all in.

Against OpenAI and Google but also toward AGI and the leadership in generative AI. Zuck’s message implies something else: he seems confident Meta can prove that the capability ceiling doesn't exist.

But if it does exist — if there’s a plateau in capabilities (which some experts believe is non-existent or at least that we’re not close yet, like 01.AI CEO Kai-Fu Lee said during a debate at the World Economic Forum) — they will hit it soon. And hard.

But if it does exist, what happens if they fail to break it with ~600k H100s?

How far can we get with what we’ve discovered?

Not many experts disagree with the notion of AGI, an admittedly confusing term.

Some prefer other names — LeCun thinks it’s a misnomer and uses human-level AI instead — but most believe we’ll get there however we call it. Zuck himself said he doesn’t really care about the exact definition of the concept behind the acronym, but that we need general intelligence for the incoming AI tools to really be next-gen.

I won’t discuss the name here either. What I care about is that even if people don’t want AI companies to build it, it is, at the very least, theoretically possible. No physical laws that we know of limit the existence of a similar or superior form of intelligence to humans (no physical law says it’s inevitable either, though).

The reason I bring this up is that the hypothetical ceiling I keep referring to is not a hard limit but a soft one in the form of what we can understand about what we’re doing (which is half a cognitive deficiency and half a technological insufficiency).

Perhaps we haven’t really figured out what we need to build the next generation of AI tools. Perhaps we can’t. I don’t know, but the possibility exists that our efforts fail in 2024. It’s not a crazy prediction. Whatever many GPUs Meta has gathered, if that possibility becomes real, they might not be enough.

What will happen if, after trying their hardest, these two top AI companies don’t manage to fulfill our expectations and break the capability ceiling that GPT-4 has seemingly, and unintentionally, set?

That’s why 2024 is a special year.

Will GPT-5 be good enough?

GPT-5 and Llama 3 are incoming anyway. Will they be good enough?

Sam Altman said recently in an interview at the World Economic Forum (here’s a translated version of the article) that “GPT-2 was very bad. GPT-3 was pretty bad. GPT-4 is bad. GPT-5 will be okay.” I interpret this as meaning GPT-5 will be significantly better than GPT-4, i.e. we will feel the difference using them, but perhaps not as good as he’d want.

The question is: Will this capability improvement reflect a similar path of progress as previous ones? Will GPT-5 be to GPT-4 what GPT-4 was to GPT-3? Will that satisfy us? For one, I already didn’t consider the GPT-3 to GPT-4 jump as impressive as the GPT-2 to GPT-3 was. Perhaps it’s easier to be surprised when you see a system go from absolutely dumb to mid than to go from mid to somewhat decent.

If GPT-5 is “only” a bit more decent, then it might not pass the bar of either our expectations or our perceptions. People (well, the obsessed ones) are expecting AGI anytime now. They’re also the most diehard advocates of AI. If AI companies lose them, they’re losing the bulk of their popular support.

2024: The year of the revelations

So Zuckerberg and Meta are going all in toward AGI with more value in the form of GPU compute than OpenAI has money in the bank. Fine. Altman and OpenAI will release GPT-5 sometime in 2024 to try to remain leaders and keep our enthusiasm intact. Also fine. (I don’t know what Google DeepMind plans to do after the Gemini disappointment, aggravated by the tricky demo, so I’ve left them out of this.)

They will try their hardest to provide evidence that we’re not done yet with the exponential (as Cohere CEO Aidan Gomez says, scaling hasn't yet been exhausted). That AI is growing as fast as ever or more. That breakthroughs will come our way and clear the mirage-like obstacles ahead. That the capability ceiling is non-existent. That AGI is around the corner.

Whether they succeed or fail, the near-term future of AI will be laid bare.

In going all in they’re making a final bet.

In 2024, the result of that bet will be revealed.