Very insightful article. All will be well! We need to hear from the folks who do not know much about AI or know only little. But these folks may know a lot about how humanity has functioned and survived over all the centuries either because of their age and/or life experiences. I want to assure all that doomsday is not going to happen in your generation. There will be a shake-out soon in AI, market consolidation, and a few players will be left standing. Those players may well resort to monopolistic competition (read market power, price gouging, political interference, etc). That is not new! We have already seen it in the pharma industry, insurance industry, financial services, energy industry, retail, etc.
We've seen this story before. It's written in scripture.
God tells humanity to "multiply and fill the earth," and by Genesis 10 the world is a Table of Nations — your C.S. Lewis crowd, bathing the children and playing darts, speaking diverse languages. Then a cadre of elites who all speak one language settles in Shinar to build "a tower with its top in the heavens," to make a name for itself and, explicitly, to avoid being scattered. God confuses their language and scatters them anyway. The story ends where your essay does: "and they left off building the city."
In the AI version, a cadre of Effective Altruists settles in Silicon Valley to build a Utopian tower of intelligence itself, recursively self-improving into a singularity, to make names for themselves and cure all human misery (and make a boatload of money). Both your wolves come from that pack — doomer and accelerationist argue in one shared tongue of expected value and p(doom) and just draw opposite conclusions. That's what makes it Babel rather than a turf war: the confusion happens inside a single language. One says "too dangerous," the other "too close," and both walk off the site. Your twist is that no one has to come down from heaven. The builders confuse themselves.
The name has its own irony. Genesis ties Babel to balal, "to confuse," but Babel was also a pun on Babylon, which means "Gate of God" — so the Genesis story is also a satire of imperial and religious grandeur, about a domination project that was launched and abandoned and rebuilt and abandoned again, for centuries. Perpetually imminent, never finished.
Could there be more laughable figures of corporate and messianic grandeur than some of the tech titans running today's Babel projects? Though at the moment they seem more like Bond villains than saviors of humanity, or at least its Western pole.
Right? BTW, I cite scripture not for religious reasons, but because they are records of what people noticed about human nature over the centuries and bothered to pass them along. I’m sure similar insights about this aspect of human nature is recorded in a wide variety of traditions
I wonder what the world would be like if the atomic bomb had been developed by a few private companies, during peace time, each rushing to detonate the first successful gadget, rather than during war time, as a government project.
I imagine a nuclear bomb is sufficiently legible as dangerous so that the government would have confiscated everything pretty soon. AI being dangerous needed a lot of convincing and it's not clear to me they've bought it yet.
That's certainly likely. I should have said what if "atomic energy" was pursued initially, without thought of bombs. In my counterfactual Meitner & Frisch's paper comes out during peacetime, with no war on the horizon. The big minds think about free heat, not bombs. "Free" heat means cheap electricity and other cheap energy. Private companies, in different countries, vie for the great minds. Fermi goes to one company, Bethe goes to another. Szilard starts a company that's going to make all their discoveries openly available to the world. Lawrence has the best uranium enrichment company. It's all about building reactors to generate heat, not bombs. A few of the researchers claim that the future is plutonium, so a few groups build reactors to make plutonium. And then people start talking about bombs. Some people are certain atomic energy is going to be the greatest gift to humanity. Others are certain bombs will be successfully developed and the world will be destroyed. Now what does the government do?
In sum, I feel like we've been here before. And we all survive, until we don't. Which is a long winded way of saying I really appreciate your C.S. Lewis quote. Great column overall, too.
Believers in the singularity have varying ideas of how close we are and what it might look like. Even if many people were inclined to stop working on it at the point they personally thought it was inevitable, others who think it is farther away would continue to push progress. FWIW I don't find your main premise plausible. I've worked in the industry and overwhelmingly I see people leave projects after they are concluded, not right before success. If you leave early, you don't get your name on it, you don't get to join the victory party, you don't get to influence the final form of whatever you are building. Either way, I think that as long as there is funding and freedom to continue pushing forward, there will be people eager to do the work.
There is no paradox. If you believe that AI will lead to a utopian future then every day you bring that forward will save the life of thousands, if not millions of people and the closer the singularity is, the more urgent the moral imperative to bring it closer will feel. If you were researching a cure for cancer and felt that you were just a day away then it would be morally reprehensible to take a two week vacation before finishing because that means that you would be "responsible" for the deaths of however many people normally die in a two week period (Yes the work of actually producing and distributing the cure takes a long time but that is a fixed period that will be added in either case)
Maybe I didn't explain it well. The reason why those who think the singularity is good would leave is because they imagine they're not needed anymore to make it happen because others are doing it and so the opportunity cost of doing other things they may not be able to do because the singularity is, by definition, the point from which everything becomes unpredictable, only grows and grows by the day. And so at some point, it's sufficiently high that those guys leave to, say, spend time with their families because that's suddenly a higher priority.
I still don't think that works. Just because others are working on it doesn't mean that your work won't bring it one day closer and bringing utopia one day closer has a very high moral value. You'd have to believe that your work made no difference at all (or at least less than a day) to the date at which significant benefits to human life are acheived.
This sounds like a scenario where you believe that RSI doesn’t really occur and that non trivial human input will be required to sustain it. Know one knows, so this might be true. But for those that believe in true singularity takeoff the basic concept is that when the exponential really gets going the amount of human input required to keep it going will rapidly drop to near zero. To me that means that the true believers in utopia will stop working hard at that point, but your idea that they will do that BEFORE it’s actually self sustaining seems to hinge on all those people being mistaken about how much human help is needed. I think that is unlikely. I think people like Nick describes are more likely to keep pushing up until the point they are SURE it’s going on it’s own, which is likely to be AFTER it is, not just before.
RSI can occur and still not be self-sustaining. Those are different things. E.g. we bomb all the datacenters and RSI is gone. Or engineers stop working out the infra, etc. the real world and the digital world exist in different planes. My point here is that the material plane prevents RSI from being enough to cause the singularity. Do the guys working on it believe it's enough? Ok, that strengthens my point: they will stop sooner due to growing opportunity costs (which don't disappear even if what you say is true) or later she to the mistaken belief that RSI is enough by itself. Whatever the case, the paradox stands. Like, there's not even a reason to keep working for legacy or something like that because if you actually believe in the Singularity, there's no "history" after it. Their human instincts will kick off. No animal can withstand the imposing presence of a "black hole." It's psychologically too much.
I agree with your point that RSI alone is not enough for self sustaining, particularly if AI stays constrained to only digital and hasn’t physically manifested itself yet (e.g. robots building robots and data centers).
But in my mind if it’s not self sustaining then it’s not yet a singularity at all yet and is still “pre-singularity”. I would expect at that point people would still be motivated towards reaching singularity as there are hard limit on how much “utopia” AI can provide while it has no physical manifestation and is still digital (e.g. can’t build housing, or make things that people want).
In some ways it might be BETTER if we get full RSI before we have good robots as then if things go very bad it still would be possible to physically disconnect all the data centers. Though if we make a lot of robots first and they are already everywhere as part of infrastructure then hitting RSI after that point is potentially much more dangerous.
Interesting. I think the concept of "software-only singularity" (not mine, I wish) should be explored more. I think that's the actual future we're heading toward. An extreme superintelligence on coding math, etc. fornalizable and verifiable domains but extremely jagged in other areas. I still think that can lead to one form of singularity, but ofc that becomes a debate on semantics.
Counterpoint: This was actually Andrej Karpahy's reason for not working at a big lab (progress will continue without him) and starting Eureka Labs (edu lab) instead... now he's at Anthropic.
I agree with most of your points, but I think there’s a much simpler explanation for why the singularity won’t happen: physical limits.
Limitless self-improvement is a pipe dream in a universe with finite resources and many laws on how those resources can be used. The whole singularity concept is built around the idea that AIs will operate much faster than humans, and that’s becoming less and less true. Frontier models still outpace humans, but it’s not by nearly as much as it used to be. That’s probably fundamental. The more intelligent a model is, the longer it has to think on each question, the more compute it uses, etc. The gap between AI and human problem solving is closing, but so is the gap in time to a solution.
My point is nothing comes for free, and gains in intelligence will likely be much harder to come by as things progress. AIs will eventually hit a point where further gains aren’t possible, and, given how well optimized humans are, that point probably isn’t too far beyond our own intelligence. And even if you believe there’s tremendous room for improvement, our universe still doesn’t support it. And if your answer to my point is that AI will find new physics to circumvent these issues, you might as well just make a religious argument, because there’s no scientific support that such mechanisms exist.
Brilliant reframe the singularity may stall not from regulation or technical failure but from human motivation collapsing at both ends. C.S. Lewis's point lands, how you live while waiting is the whole game.
Thanks, a reasonable conjecture, who is to say. Most of us feel like we are on the outside looking in…This is the first moment in human history in which almost all of the written record is instantly available. What we call AI is one manifestation of this and can leave the user with an unhealthy feeling of great power when it should ,rather be fostering a perspective of humility in the face of our whole species’ history and learning on display. The super heated environment of this research makes the researchers themselves feel like they are working on multiple ‘Manhattan’ projects at any one time, with almost religious abandon. From the outside it’s difficult to gauge how much credibility to place in the view of any of these individuals but to acknowledge the super empowerment of lone individuals this way has an almost certain potential for disaster….we all are on the outside looking in.
As someone who has observed thousands of people spend days of their life on an online game while vocally complaining that they stopped enjoying it years ago, I believe many people's capacity to overcome inertia and reassess their daily activities and long term habits is not very high at all. Kudos to those who do make the change!
But the key difference is that those people are agency-less humans whereas we're talking here about possibly one of the highest agency batch of humans alive today. They just can do things
Perhaps. I've never worked for an AI company, but my personal experience elsewhere in tech has been that the correlation between the qualities "highly capable developer" and "plays online game and moans about it" is not a negative one.
That is not how things work. People don’t abandon their pursuits because they think other people will accomplish them or by thinking their goal is inevitable. If you really believed that why are you wouldn’t be writing this post.
That's a response in the abstract. I presented a specific mechanism. Do you have any specific counter to this specific mechanism? I can't answer an abstract argument because it's empty.
Brilliant essay, Alberto! The idea that intelligence can just detach from physical reality and accelerate to infinity has always struck me as deeply ungrounded.
Both groups are wasting time. There is no superhuman AI. There is no super intelligence in AI, not even plain intelligence. A rogue AI model with undesirable behavior points to our failure to erect proper guard rails to constrain the model.
The oscillation only works if those specific people are load-bearing, and I'm not sure they are. Anthropic's own economic-scenarios paper — same company Coxon walked out of — has labor's share of national income falling in all three futures it modeled, including the best one, from roughly 60% down to the mid-40s. If a lab's own research says human contribution keeps getting cheaper in every scenario, the second wolf doesn't have to wait for the first one to come back; it just hires someone who never had the conundrum. What would actually have to break for the machine to stop, if not the staffing?
Very insightful article. All will be well! We need to hear from the folks who do not know much about AI or know only little. But these folks may know a lot about how humanity has functioned and survived over all the centuries either because of their age and/or life experiences. I want to assure all that doomsday is not going to happen in your generation. There will be a shake-out soon in AI, market consolidation, and a few players will be left standing. Those players may well resort to monopolistic competition (read market power, price gouging, political interference, etc). That is not new! We have already seen it in the pharma industry, insurance industry, financial services, energy industry, retail, etc.
Compelling viewpoint. Real understanding grows precisely where formal models confront messy empirical constraints.
We've seen this story before. It's written in scripture.
God tells humanity to "multiply and fill the earth," and by Genesis 10 the world is a Table of Nations — your C.S. Lewis crowd, bathing the children and playing darts, speaking diverse languages. Then a cadre of elites who all speak one language settles in Shinar to build "a tower with its top in the heavens," to make a name for itself and, explicitly, to avoid being scattered. God confuses their language and scatters them anyway. The story ends where your essay does: "and they left off building the city."
In the AI version, a cadre of Effective Altruists settles in Silicon Valley to build a Utopian tower of intelligence itself, recursively self-improving into a singularity, to make names for themselves and cure all human misery (and make a boatload of money). Both your wolves come from that pack — doomer and accelerationist argue in one shared tongue of expected value and p(doom) and just draw opposite conclusions. That's what makes it Babel rather than a turf war: the confusion happens inside a single language. One says "too dangerous," the other "too close," and both walk off the site. Your twist is that no one has to come down from heaven. The builders confuse themselves.
The name has its own irony. Genesis ties Babel to balal, "to confuse," but Babel was also a pun on Babylon, which means "Gate of God" — so the Genesis story is also a satire of imperial and religious grandeur, about a domination project that was launched and abandoned and rebuilt and abandoned again, for centuries. Perpetually imminent, never finished.
Could there be more laughable figures of corporate and messianic grandeur than some of the tech titans running today's Babel projects? Though at the moment they seem more like Bond villains than saviors of humanity, or at least its Western pole.
Right? BTW, I cite scripture not for religious reasons, but because they are records of what people noticed about human nature over the centuries and bothered to pass them along. I’m sure similar insights about this aspect of human nature is recorded in a wide variety of traditions
You know it's a real crisis when we start making biblical references.
I wonder what the world would be like if the atomic bomb had been developed by a few private companies, during peace time, each rushing to detonate the first successful gadget, rather than during war time, as a government project.
I imagine a nuclear bomb is sufficiently legible as dangerous so that the government would have confiscated everything pretty soon. AI being dangerous needed a lot of convincing and it's not clear to me they've bought it yet.
That's certainly likely. I should have said what if "atomic energy" was pursued initially, without thought of bombs. In my counterfactual Meitner & Frisch's paper comes out during peacetime, with no war on the horizon. The big minds think about free heat, not bombs. "Free" heat means cheap electricity and other cheap energy. Private companies, in different countries, vie for the great minds. Fermi goes to one company, Bethe goes to another. Szilard starts a company that's going to make all their discoveries openly available to the world. Lawrence has the best uranium enrichment company. It's all about building reactors to generate heat, not bombs. A few of the researchers claim that the future is plutonium, so a few groups build reactors to make plutonium. And then people start talking about bombs. Some people are certain atomic energy is going to be the greatest gift to humanity. Others are certain bombs will be successfully developed and the world will be destroyed. Now what does the government do?
In sum, I feel like we've been here before. And we all survive, until we don't. Which is a long winded way of saying I really appreciate your C.S. Lewis quote. Great column overall, too.
Believers in the singularity have varying ideas of how close we are and what it might look like. Even if many people were inclined to stop working on it at the point they personally thought it was inevitable, others who think it is farther away would continue to push progress. FWIW I don't find your main premise plausible. I've worked in the industry and overwhelmingly I see people leave projects after they are concluded, not right before success. If you leave early, you don't get your name on it, you don't get to join the victory party, you don't get to influence the final form of whatever you are building. Either way, I think that as long as there is funding and freedom to continue pushing forward, there will be people eager to do the work.
I think you're right but failing to model what "the singularity" means. Changes quite a lot from "projects" in general
Alberto, may you turn out to be right!
This is the way the world ends
Not with a bang but a whimper. (Eliot)
There is no paradox. If you believe that AI will lead to a utopian future then every day you bring that forward will save the life of thousands, if not millions of people and the closer the singularity is, the more urgent the moral imperative to bring it closer will feel. If you were researching a cure for cancer and felt that you were just a day away then it would be morally reprehensible to take a two week vacation before finishing because that means that you would be "responsible" for the deaths of however many people normally die in a two week period (Yes the work of actually producing and distributing the cure takes a long time but that is a fixed period that will be added in either case)
Maybe I didn't explain it well. The reason why those who think the singularity is good would leave is because they imagine they're not needed anymore to make it happen because others are doing it and so the opportunity cost of doing other things they may not be able to do because the singularity is, by definition, the point from which everything becomes unpredictable, only grows and grows by the day. And so at some point, it's sufficiently high that those guys leave to, say, spend time with their families because that's suddenly a higher priority.
I still don't think that works. Just because others are working on it doesn't mean that your work won't bring it one day closer and bringing utopia one day closer has a very high moral value. You'd have to believe that your work made no difference at all (or at least less than a day) to the date at which significant benefits to human life are acheived.
The thing is that "moral value" is not the only kind of value. Humans are selfish, a trait that is perhaps clearest near the end of time.
This sounds like a scenario where you believe that RSI doesn’t really occur and that non trivial human input will be required to sustain it. Know one knows, so this might be true. But for those that believe in true singularity takeoff the basic concept is that when the exponential really gets going the amount of human input required to keep it going will rapidly drop to near zero. To me that means that the true believers in utopia will stop working hard at that point, but your idea that they will do that BEFORE it’s actually self sustaining seems to hinge on all those people being mistaken about how much human help is needed. I think that is unlikely. I think people like Nick describes are more likely to keep pushing up until the point they are SURE it’s going on it’s own, which is likely to be AFTER it is, not just before.
RSI can occur and still not be self-sustaining. Those are different things. E.g. we bomb all the datacenters and RSI is gone. Or engineers stop working out the infra, etc. the real world and the digital world exist in different planes. My point here is that the material plane prevents RSI from being enough to cause the singularity. Do the guys working on it believe it's enough? Ok, that strengthens my point: they will stop sooner due to growing opportunity costs (which don't disappear even if what you say is true) or later she to the mistaken belief that RSI is enough by itself. Whatever the case, the paradox stands. Like, there's not even a reason to keep working for legacy or something like that because if you actually believe in the Singularity, there's no "history" after it. Their human instincts will kick off. No animal can withstand the imposing presence of a "black hole." It's psychologically too much.
I agree with your point that RSI alone is not enough for self sustaining, particularly if AI stays constrained to only digital and hasn’t physically manifested itself yet (e.g. robots building robots and data centers).
But in my mind if it’s not self sustaining then it’s not yet a singularity at all yet and is still “pre-singularity”. I would expect at that point people would still be motivated towards reaching singularity as there are hard limit on how much “utopia” AI can provide while it has no physical manifestation and is still digital (e.g. can’t build housing, or make things that people want).
In some ways it might be BETTER if we get full RSI before we have good robots as then if things go very bad it still would be possible to physically disconnect all the data centers. Though if we make a lot of robots first and they are already everywhere as part of infrastructure then hitting RSI after that point is potentially much more dangerous.
Interesting. I think the concept of "software-only singularity" (not mine, I wish) should be explored more. I think that's the actual future we're heading toward. An extreme superintelligence on coding math, etc. fornalizable and verifiable domains but extremely jagged in other areas. I still think that can lead to one form of singularity, but ofc that becomes a debate on semantics.
Counterpoint: This was actually Andrej Karpahy's reason for not working at a big lab (progress will continue without him) and starting Eureka Labs (edu lab) instead... now he's at Anthropic.
I agree with most of your points, but I think there’s a much simpler explanation for why the singularity won’t happen: physical limits.
Limitless self-improvement is a pipe dream in a universe with finite resources and many laws on how those resources can be used. The whole singularity concept is built around the idea that AIs will operate much faster than humans, and that’s becoming less and less true. Frontier models still outpace humans, but it’s not by nearly as much as it used to be. That’s probably fundamental. The more intelligent a model is, the longer it has to think on each question, the more compute it uses, etc. The gap between AI and human problem solving is closing, but so is the gap in time to a solution.
My point is nothing comes for free, and gains in intelligence will likely be much harder to come by as things progress. AIs will eventually hit a point where further gains aren’t possible, and, given how well optimized humans are, that point probably isn’t too far beyond our own intelligence. And even if you believe there’s tremendous room for improvement, our universe still doesn’t support it. And if your answer to my point is that AI will find new physics to circumvent these issues, you might as well just make a religious argument, because there’s no scientific support that such mechanisms exist.
Brilliant reframe the singularity may stall not from regulation or technical failure but from human motivation collapsing at both ends. C.S. Lewis's point lands, how you live while waiting is the whole game.
Thanks, a reasonable conjecture, who is to say. Most of us feel like we are on the outside looking in…This is the first moment in human history in which almost all of the written record is instantly available. What we call AI is one manifestation of this and can leave the user with an unhealthy feeling of great power when it should ,rather be fostering a perspective of humility in the face of our whole species’ history and learning on display. The super heated environment of this research makes the researchers themselves feel like they are working on multiple ‘Manhattan’ projects at any one time, with almost religious abandon. From the outside it’s difficult to gauge how much credibility to place in the view of any of these individuals but to acknowledge the super empowerment of lone individuals this way has an almost certain potential for disaster….we all are on the outside looking in.
I'm not quite sure what makes them think the human race isn't up to the job of destroying itself without AI help.
[AI] Ugh, ya, but you're so slow at it. At this rate I'd have to wait another decade
As someone who has observed thousands of people spend days of their life on an online game while vocally complaining that they stopped enjoying it years ago, I believe many people's capacity to overcome inertia and reassess their daily activities and long term habits is not very high at all. Kudos to those who do make the change!
But the key difference is that those people are agency-less humans whereas we're talking here about possibly one of the highest agency batch of humans alive today. They just can do things
Perhaps. I've never worked for an AI company, but my personal experience elsewhere in tech has been that the correlation between the qualities "highly capable developer" and "plays online game and moans about it" is not a negative one.
That is not how things work. People don’t abandon their pursuits because they think other people will accomplish them or by thinking their goal is inevitable. If you really believed that why are you wouldn’t be writing this post.
That's a response in the abstract. I presented a specific mechanism. Do you have any specific counter to this specific mechanism? I can't answer an abstract argument because it's empty.
Being hyper specific when the underlying logic is flawed makes the argument invalid in my view. Take that as you will.
Superintelligence is coming whether you accept the fact or not.
Brilliant essay, Alberto! The idea that intelligence can just detach from physical reality and accelerate to infinity has always struck me as deeply ungrounded.
Physics and empirical feedback loops are stubborn constraints that no amount of pure compute can wave away. Wrote a philosophical dialogue touching on those physical boundaries here: https://gzambrano.substack.com/p/o-limite-do-testavel-um-dialogo-em
Such an important corrective to the prevailing techno-mysticism!
Both groups are wasting time. There is no superhuman AI. There is no super intelligence in AI, not even plain intelligence. A rogue AI model with undesirable behavior points to our failure to erect proper guard rails to constrain the model.
The oscillation only works if those specific people are load-bearing, and I'm not sure they are. Anthropic's own economic-scenarios paper — same company Coxon walked out of — has labor's share of national income falling in all three futures it modeled, including the best one, from roughly 60% down to the mid-40s. If a lab's own research says human contribution keeps getting cheaper in every scenario, the second wolf doesn't have to wait for the first one to come back; it just hires someone who never had the conundrum. What would actually have to break for the machine to stop, if not the staffing?