Even the best critical thinkers can fall victim to the contagious thrill and excitement caused by the semblance of a scientific paradigm shift. This effect is enhanced if recent history is plagued with unprecedented change. More so if the discipline we’re talking about is one that promises to redefine the world as we know it.
This is happening in AI and one of the clearest examples we can find of this vicious cycle of frenzy is Q*, the supposed AI breakthrough that OpenAI researchers reportedly discovered recently, presumably leading to the eventual boardroom coup that ended in CEO Sam Altman’s firing (only for him to be promptly reinstated afterward).
I write “supposed,” “reportedly,” and “presumably” because we literally only have a name, Q* (pronounced Q-star), and a vague description of its main skill: solving certain math problems. That’s all the information we got last week — enough to spark a wave of wild speculations about Q* and the future of AI. The truth is, we know nothing about Q*. So, why all the fuzz?
What’s the real reason we care about Q*?
Did Q* spook the OpenAI board or did we just need to believe it did?
To quickly put you in context, Reuters reported that a source “familiar with the matter” revealed that Q* was the “AI breakthrough” that caused the boardroom coup at OpenAI after “several staff researchers wrote a letter to the board of directors warning of a powerful artificial intelligence discovery that they said could threaten humanity.” Shortly afterward, The Information reported that Q*, the development resulting from a research breakthrough by Ilya Sutskever, had “raised concerns among some staff that the company didn’t have proper safeguards in place to commercialize such advanced AI models.”
Just to make a brief digression, I will say that writing “a source familiar with the matter,” although intended to protect anonymity, is a rather confusing way to emphasize that the information is trustworthy. This practice is becoming commonplace for tech magazines but it effectively covers up any trace for readers to follow, which only makes it all more mysterious and hardly verifiable. Indeed, The Verge’s Alex Heath and Platformer’s Casey Newton both reported that the board never received such a letter. The Atlantic’s Karen Hao wrote that “the researchers’ concerns did not precipitate the board’s actions.”
So, who do we trust now? If Q* didn’t prompt the coup, then what was the reason? Is Q* even worth all this debate built on thin air? Or, an alternative way to frame it: Why is the AI space so filled with misinformation?
Perhaps Q* wasn’t the reason the board fired Altman. Perhaps it’s not even a real breakthrough; as Hao says “It takes a long time for consensus to form about whether a particular algorithm or piece of research was in fact a breakthrough.” In any case, the news spread like wildfire through the Twittersphere, and people didn't hesitate to fuel the hype with rumors about Q* (I didn’t because I’ve made that mistake already). People immediately reacted to Q* as if it had been confirmed that it was, indeed, the motivation for the mutiny as an unprecedented achievement toward artificial general intelligence (AGI) — one worth worrying about.
I don’t blame the people who jumped to speculate (except the grifters, of course). If a news source you trust says an “AI breakthrough” spooked the board because OpenAI researchers were concerned, what else are you going to believe? We were hungry to close the explanatory gap that the board’s deafening silence left throughout the five-day crisis. Why did they risk an extremely successful AI startup’s survival? What did they know that we didn’t?
I argued it was likely a bunch of tiny discrepancies that compounded over time, starting from a hidden seed of potentially fracturing disagreement at the very conception of OpenAI. That was, of course, not sufficient: People don’t want to follow breadcrumbs each of which amounts to a minuscule portion of the puzzle.
What they wanted — the only thing that would satisfy their curiosity and skepticism — was the Big News. Q*, merely the shape of the shadow of a secretive project that hinted at something very important, nailed it.
Q* could be an interesting breakthrough but not one worth worrying about
Just to let you know what people think Q* could be (take all of this with a grain of salt, but at the same time, know that it makes sense), here’s a brief review of what people have been arguing. As I see it, the most succinctly reasonable hypothesis is Yann LeCun’s: Q* is “OpenAI attempts at planning.”
The explanation is straightforward enough. All leading AI labs, including OpenAI, Google DeepMind, FAIR, and possibly Anthropic, know that language models are insufficient to get AI to the next level. Auto-regressive models use their recent history of outputs to decide what to generate next, which is a poor alternative to planning. AI agents that can reliably act in the world need to have an internal world model that allows them to make predictions of potential actions before making them. That’s what planning is. (Some prominent experts believe SOTA language models have developed primitive forms of internal world models, but that’s a question for another article.)
The name Q* is likely a reference to both A*, a popular search algorithm (i.e., find the best path), and Q-learning, a popular reinforcement learning algorithm (i.e., choose the best action), which suggests OpenAI is trying to mix learning with search, which is a key ingredient for agents capable of planning and reasoning. As DeepMind co-founder Shane Legg shared recently:
I don’t think we’ll see systems that truly step beyond their training data until we have powerful search in the process.
That’s very likely what Google DeepMind’s Gemini is all about, too. As I wrote back then when the first information came out: “DeepMind’s Alpha family [reinforcement + search] and OpenAI’s GPT family [deep learning] each possess a secret sauce that has turbocharged progress toward human-level intelligence in recent years. This secret sauce mix is what I believe Google DeepMind plans to imbue into Gemini.”
It makes sense that Q* is a project focused on planning not just because planning (and reasoning, which is required for planning) is what AI needs next, but also because OpenAI has been giving hints for months (clear in retrospect) that they are working on this.
They hired Noam Brown in July this year. He co-created Libratus (poker) at CMU and Cicero (Diplomacy) at Meta AI. These breakthroughs in reasoning resulted from a very anthropomorphic insight from Brown: Allowing AI models to “ponder” before answering improved significantly their performance, just like humans do when we engage in our system 2 mode of thinking, as defined by Daniel Kahneman.
This is what Brown said about his work previous to OpenAI at the time he was hired by the company:
For years I’ve researched AI self-play and reasoning in games like Poker and Diplomacy. I’ll now investigate how to make these methods truly general. If successful, we may one day see LLMs that are 1,000x better than GPT-4.
There’s more. Just before his firing, Altman shared some cryptic words during the APEC summit that I originally assumed (before we knew about Q*) were related to the reason for the coup (the reason wasn’t perhaps the reality hidden behind the words, likely Q*’s existence as a threat to humanity, but Altman’s nonchalance towards it). He said:
On a personal note, like four times now in the history of OpenAI — the most recent time was just in the last couple of weeks — I’ve gotten to be in the room when we pushed the veil of ignorance back and the frontier of discovery forward.
A couple of weeks prior, during the first Developer Day, Altman concluded with a similarly enigmatic yet optimistic message:
What we launch today is going to look very quaint relative to what we're busy creating for you now.
All that said, there’s no reason to think Q* is spooky or a step toward increased AI existential risk (for AI doomers, any step forward is a step in that direction, so their opinion doesn’t really count here). Again, LeCun says it best: “People need to calm down.”
Material for grifters; food for enthusiasts
As Jim Fan argues, it doesn’t really matter what Q* is or what it does (although there have been some serious attempts at trying to elucidate those unasked questions). It is about planning most likely, and a combination of learning and search. The details are unimportant. We will surely know more trustworthy details about Q* in the short-term future.
That’s why this isn’t really an article about Q* (even though the above long-ish introduction is worth it in and of itself). This is, instead, a meta-article about people’s behavior in the face of hypothetical and underreported AI breakthroughs; why do they do what they do? How does that affect the industry? What does it mean for the future of the field as a whole if we don’t stop it now?
I said above that the buzz around Q* emerged out of a necessity to find answers: Why did the board really fire Altman? But although I think that’s a factor, I don’t think it’s the primary one. Let me clarify that I’m not against speculation per se — I’ve done it (wrongly, but also correctly) and find it to be an interesting exercise and often a worthwhile source of insight for readers as long as it’s framed according to the reliability of the sources and hypotheses.
But the debate on Q* wasn’t about that. It was a fantasy. Fan says that in his decade spent on AI, he’s “never seen an algorithm that so many people fantasize about. Just from a name, no paper, no stats, no product.”
More importantly, Q* was mostly not a fantasy aligned with finding the truth. Unbounded speculation that starts from the tiniest seed of truth (“just from a name”) is dangerous because it’s the perfect mix to attract grifters motivated by some vested interest who have not an ounce of devotion and respect for knowledge. We are all self-interested but these people’s self-interest happens to be orthogonal with finding the truth.
These grifters are everywhere. What makes them problematic is that AI is a gold mine and fantasizing about it is extremely profitable. There’s a constant influx of news, rumors, and speculation that may or may not reflect an underlying reality. But they don’t care. What matters to them is that, to a degree only comparable to the amount of material at their disposal, there are people waiting to feel excited, concerned, and fearful to not miss out, to get ahead, and to avoid being left behind.
You may think the reason for both the grifters and the enthusiasts to exist is that AI is making significant strides forward but I don’t buy it. AI is advancing but nowhere as fast — at least in aspects that are relevant to the world at large — as it seems to be.
The real reason all the unnecessary hype about Q* happened at all is that we’ve grown accustomed to getting so many AI revelations and announcements so fast that we can’t help but crave more. Those grifters only exist because there’s a demand to be satisfied.
Q* is a symptom of our Shiny Object Syndrome
There are no real culprits here (I despise grifters but they’re just playing the game). It’s just the workings of the market for attention and interest. We can however locate the original source that started it all: OpenAI’s impeccable marketing.
Didn’t you notice that just after GPT-3 was released in 2020 people were already asking for GPT-4? The same thing happened with ChatGPT in November last year. Unsurprisingly, when GPT-4 was announced in March people were already asking for GPT-5. GPT-4 was interesting until it was released. Shortly after, a considerable portion of interest shifted to GPT-5.
Altman himself noticed this weird phenomenon in an interview with Exponentially’s Azeem Azhar. “[GPT-4] felt like a big jump for a little while and now people are … ‘What have you done for me lately, where’s GPT-5?’ … people get used to anything; we establish new baselines very quickly.” Altman didn’t phrase it so much as a complaint but as a neutral observation of how the world works.
My reading is slightly less optimistic. Many people are enthusiastic about AI not for what it is as an end (e.g., useful tools) or what it could be ultimately (AGI) but to satisfy an urge, to get the next dopamine hit. Q* provided just that. Q* is the symptom of an illness we suffer from: The Shiny Object Syndrome.
We’re drawn to whatever’s new, because in our evolutionary history new info tended to matter. But now it doesn’t, because 99% of new info is clickbait, mass-produced and rushed-out to exploit our attraction to novelty. So stop chasing the new and seek info that’s stood the test of time.
The shiny-object metaphor is not confined to the realm of politics. Business strategy, technology and marketing consultants have all referred to ‘‘bright, shiny objects’’ (or ‘‘B.S.O.s’’) to describe the fickle tastes of modern life. Urban Dictionary identifies ‘‘S.O.S.’’ (‘‘shiny-object syndrome’’) as ‘‘a condition which causes an inability to focus on any particular person while online dating.’’
Shiny Object Syndrome is everywhere but it becomes more prevalent and accentuated if we get new things easily, in great amounts, and very fast. That’s exactly what’s going on in AI (especially since ChatGPT). It’s exactly the tragedy of OpenAI’s success and its impeccable marketing: Make it look easy and people will assume it is; give them a lot and they will ask for even more; give it to them really fast and they will yearn for the new thing even faster.
We get used to the current thing so fast that the only thing that truly satisfies our interest in AI is knowing that a new thing is coming our way. This sad phenomenon has no end. The reactions we saw about Q* last week are all about this. We don’t care anymore about ChatGPT or GPT-4 — they are great as technological advancements, as products, and as scientific reflections of what’s possible with AI, but they’re not eternal in their ability to satisfy our need for more. Q* isn’t either, so don’t be surprised if sometime soon we are talking about the next shiny object.
Great analysis. Most things are blown out of proportion these days. The ability to diagnose signal within all the noise is critical.