Hey, Alberto here! 👋 I publish long-form AI analysis covering culture, philosophy, and business. Paid subscribers get Monday how-to guides and Friday news commentary (on hold for vacation, returning soon). If you’d like to become a paid sub, here’s a button for that:
Dario Amodei, CEO of Anthropic, published an essay called “We Must Pace the Frontier” that could define the next years of the industry.
He proposes that the industry slow down capability improvements while it figures out a safe way forward. His argument is that AI is advancing too quickly—with the help of AI itself—and that the companies building it are not fully in control. He mentions the hack of Hugging Face as an example. Another could be the solution to Navier-Stokes.
His proposal has three stages:
Independent evaluators embedded inside AI companies to check what they do, something Anthropic is committing to.
Coordination between American and other democratic-world labs.
Eventually, some degree of international coordination (including China).
The timing is impeccable when you consider the following facts.
First, safety concerns have gone mainstream after Jacob Coxon’s tweet and resignation from Anthropic. Whether this was part of an orchestrated campaign or not is irrelevant.
Second, AI safety problems have become real with the many hacks that have taken place during the past months (the Hugging Face incident is merely the most prominent). Whether these hacks represent autonomous behavior from an uncontrollable technology or good old-fashioned industry-wide negligence is, again, irrelevant to the point.
Third, progress at the frontier is dominated by two American companies (I think it will be hard for the others to catch up to Anthropic and OpenAI). The Navier-Stokes solution makes this case best but you can also take a look at revenues or benchmark scores.
Those three facts suggest that Amodei’s worries 1) will be well received because others have paved the way, 2) are reasonable because the real world has felt the danger, and 3) need not compromise Western AI hegemony because China is too far behind.
His peers backed him up.
Soon after he published the essay, Sam Altman of OpenAI, Elon Musk of xAI, and Demis Hassabis of DeepMind expressed support for his proposal. Altman, who runs Anthropic’s main rival at the frontier, also announced that OpenAI would commit to allowing independent evaluators inside.
So that’s roughly where the industry stands right now. What about the governments?
President Donald Trump, who notably holds one key to this solution—the other key holder being Xi Jinping—said no to the slowdown. He said that “whoever wins AI wins,” and sprinkled some expletives on top. Altman masterfully responded indirectly with: “But you could win the Nobel Peace Prize, tho.” So the question is, which is larger, Trump’s ambition or Trump’s pride? Jinping, for his part, called Amodei’s call to a slowdown a “Cold War playbook.” The Global Times wrote that it was an attempt “to curb China’s Al development through technological barriers and regulatory monopolies, uphold Washington’s monopolistic hegemony in cutting-edge technology, and exclude China from the global Al governance system.”
With the fearful industry on one side and the fearless governments on the other, this is looking a lot like Oppenheimer vs Truman. Unfortunately for Amodei, technology is power but not more so than power itself.
So I guess it ain’t happening. But I’m going to take Amodei’s intentions seriously anyway. Let’s assume Trump’s ambition win over his pride and China falls even more far behind.
There are several readings of Amodei’s essay, spanning the entire spectrum from “this is a marketing ploy ahead of the IPO” to “pacing is nowhere near enough; we need to stop the technology entirely,” with every flavor of worry, enthusiasm, and cynicism in between. I’m going to take his words and worries at face value, for the simple reason that we don’t need to reject his message to find cracks in it.
To be clear, I think Amodei is mostly honest, as he usually is (a true believer). The fact that Anthropic is a company that needs to satisfy investors, turn a profit, and so on does not automatically nullify the possibility that he cares about other things too. He was calling for safety regulation long before he was in a position of power to impose either. Will he try to sway governments and the industry toward his preferred vision if he can? Of course. That does not make him dishonest.
And yet, even if we take him seriously, there are important flaws in his plan. He knows it.