Why the FLI Open Letter Won't Work

History sure does rhyme and AI is no exception

Midjourney

On March 22, the Future of Life Institute published an open letter where they “call on all AI labs to immediately pause for at least 6 months the training of AI systems more powerful than GPT-4.” I agree with the letter's intention but not completely with the premises (or the reasons that motivated it). In case you haven’t read it yet, I recommend you do it and then come back.

Before I continue, I want to share some contrarian takes (that I consider reasonable) with you: Emily M. Bender, Matthew Barnett, Arvind Narayanan, and Timnit Gebru. Here’s another—yet opposite from the above—by Scott Aaronson. And here’s Eliezer Yudkowsky’s more extreme take (“We need to shut it all down”). I also thoroughly recommend reading the level-headed response to criticisms (including those above) by Gary Marcus (who signed but isn’t in complete agreement). The debate is clearly more nuanced than it seems at first sight.

This article isn’t a defense of my stance. It is neither a review of the arguments in favor (e.g., the letter is great, it’s necessary, it has flaws but it’s important to sign it) nor against (e.g., the letter is plain stupid, it’s useless, or it’s completely mistaken in its motivation and proposal). It is, instead, a lateral view: my reasoning for why I believe it won’t work out—not now nor in the future. (Sam Altman’s reaction is a hint.)

We may not step twice on the same stone, but we’ll make sure to do it once

We love making predictions. We’re notably bad at it—not because we lack intelligence but because the task is truly a challenge of accuracy. But if there’s something we’re even worse at is preemptively acting according to the predictions we come up with (if they imply a sacrifice of some kind).

Despite being the smartest species on earth, we’re quite dumb when it comes to adequately assessing our ability to overcome future challenges—even those we see coming (hi, climate change).

Instead, we tend to learn after the fact. We may not step twice on the same stone (sometimes we do, though) but we surely step on it at least once. That’s our reality. Our arrogance pushes us forward with extreme conviction and it’s only the hard reality of the aftermath that stops us and forces us to revisit our priors, decisions, and mistakes.

Physicists knew nuclear weapons were dangerous. They knew it when they were making them, before making them, and after they were already made. Most opposed the Manhattan Project but, despite the pushback, the US still dropped two atomic bombs on Japan. Only after dozens of thousands of deaths and many more injured—who are still today, almost 80 years later, feeling the consequences of human arrogance—did we learn that maybe it’d have been better to not build them in the first place. Now our options are reduced to stopping the proliferation of this terrible invention and raising an armistice weakly sustained by a shared fear of complete destruction.

Three months after the bombings of Hiroshima and Nagasaki, J. Robert Oppenheimer, scientific director of the project, gave a farewell speech to his fellow scientists where he “justified pursuit of an atomic bomb as inevitable, stressing that scientists must expand man’s understanding and control of nature,” as summarized by the Atomic Heritage Foundation.

They saw it coming but did it anyway.

AI is going down the same path, arguably under a more prosaic motivation—money. It doesn’t matter how relentlessly experts warn about short-term (i.e., existing) and long-term risks or how many open letters—with many prominent scientists and researchers signing them—are crafted with the intention to stop the deployment of powerful AI models. We don’t seem to learn that way.

It’s not that OpenAI, Google, and Microsoft don’t see any risks going forward—they clearly accept them as real—but their actions reveal their implicit motto is “keep going until we fuck it up.” Keep going until they step over a stone that makes us all fall over. Then and only then they’ll rethink our path.

Risk ≠ harm: Stop the doom and gloom

Why stop when all those existential risks (either for “all of humanity” or only for those they deem expendable) are still imaginary? Science fiction shouldn’t drive progress—much less motivate halting it. That’s the reasoning going on in the minds of the people making these “ever-larger unpredictable black-box models with emergent capabilities,” as the open letter describes GPT-4-like language models.

But here’s the thing, and this goes mainly for those who argue we should accelerate instead: We don’t know which way powerful AI will go. The word “risk” implies just that. A risk isn’t yet a harm because it hasn’t happened, but implies an arbitrarily large possibility for it. Just imagine this argument by Daniel Jeffries applied to nuclear weapons for which a known risk did turn into real harm:

“Until we put technology out into the real world, we can't make it better. It's through its interaction with people and places and things that we figure it out. And it's real life feedback that makes it safer faster.”

Good on paper. Terrible in hindsight. Just like Oppenheimer’s farewell speech.

Let me be clear on one thing: doomsaying and fearmongering—not as the emotionally-charged terms they’re now (often used as rhetorical attacks) but as mere descriptors—are bad. Bad for people’s beliefs, bad for scientific progress, and bad as arguments. But I don’t think it’s wise to dismiss an opposite view as “fearmongering” when those who hold it have dedicated a significant part of their professional lives to the stuff they’re warning us about. Maybe they’re right that it's time “to pause and reflect.”

Risk denialism will triumph, though. Not because it’s a good argument—saying something bad hasn’t happened is as poor an argument as you can get to avoid taking measures to prevent harm from known risks. But because humans tend to be extremely hubristic before we step for the first time on a particular stone—more so those who hold the power that moves the world forward (or wherever).

After tripping over everyone knows what to do—and what we should have done to prevent it. But we never get the chance to apply hindsight wisdom in the first place.

History sure does rhyme

Sadly, the lessons we learn by failing once don’t seem to transfer to future stones of a different type. Maybe because the resulting temporary humility is easily overshadowed by renewed overconfidence. Nuclear weapons and AI systems don’t entail the exact same kind of risks so why would we fail this time? Maybe we’re just forgetful. Why, if not, would history repeat itself all the time?

We think we are the smartest, but we might actually be the dumbest in relation to our superior means of affecting the world—which include the set of actions that would lead us to extinguish ourselves—and all the lessons we should have learned from our past mistakes that we don't seem to have learned.

Phil Tanny, a loyal TAB reader, likes to ask a question that, given the evidence, has no satisfactory answer: Is there a reasonable upside that compensates for taking one more risk to our survival as species? The answer would be yes if we could prove to have a “reasonable upside,” but anything we might come up with is merely speculative. So the answer has to be no.

We’re going to do it anyway.

We might need to live through an analogous event to the nuclear bombing of Hiroshima and Nagasaki for the most stubborn to understand and internalize that AI-generated harm is, indeed, possible. (There are a lot of instances of ongoing harm by existing AI—typically underlined by AI ethics experts like Bender and Gebru—but I don’t count on the people I’m imaginarily arguing against here considering those at all.)

If the most pessimistic, like Eliezer Yudkowsky, Elon Musk, or the late physicist Stephen Hawking, are right, we may not have a second chance. I might not think this will happen, but my belief doesn’t matter—we’ll probably get to check it ourselves.