Catastrophists versus accelerationists: Will AI destroy the world or save it?

Eliezer Yudkowsky, 46, and Nate Soares, 37, are convinced that if artificial intelligence (AI) systems continue to improve, they will eventually surpass human capabilities. And when that happens, humanity will go extinct. They argue this could occur in a matter of months or within a decade. The title of their latest book is blunt: If Anyone Builds It, Everyone Dies: Why Superhuman AI Would Kill Us All (Little, Brown & Co).

Yudkowsky and Soares are two of the leading figures among the doomers, or catastrophists. Recent advances in generative AI — the technology behind ChatGPT, Gemini, and Sora — have sparked a heated debate inside the industry about the technology’s potential. Distinct schools of thought have emerged. Doomers believe that once AI is sufficiently developed it will take the reins and decide to end civilization. For that reason, they recommend that states sign international treaties to curb AI’s advance, in the same way nuclear proliferation was limited during the Cold War.

In early 2023, an open letter signed by hundreds of AI researchers called for a six-month moratorium on research. “We signed it too, although we considered it far too short,” they write. So short, Yudkowsky wrote in an article published around that time in Time magazine, that each country’s allowed computing power should be limited and those who violate such limits should have their data centers “destroyed by air strike.”

At the other extreme are the boosters, or accelerationists, who take the opposite view: the development of superintelligence (the hypothetical intelligence that would surpass human intelligence) should be pursued because it will solve many of society’s problems. It will cure diseases, increase efficiency across processes, and help us work less. It will make us happier.

Doomers

There are prominent names associated with both currents. The doomers, precisely because they invoke the apocalypse, have greater traction in the U.S. media and internationally. Their movement carries the seal of respectability that comes from having Turing Award winners — considered the Nobel Prize of computing — among its defenders, such as Yoshua Bengio and Geoff Hinton, the latter also a Nobel laureate in physics. The fact that two of the fathers of machine learning, the technique that enabled AI’s major recent leap in capabilities, now oppose the technology they helped develop is used in Yudkowsky and Soares’ book as a weighty argument in favor of their position.

Another Turing Award winner and machine learning godfather, Yann LeCun, disagrees. He has mocked the doomers on social media. “We will design their desires,” he has said, for example. “The history of engineering is full of brilliant, eager optimists who dive headlong into new and fascinating problems that turn out to be infinitely harder than they expected,” Yudkowsky and Soares reply in the book.

This narrative is not confined to academia. Some entrepreneurs making AI possible have embraced similar rhetoric. Prominent among them is Sam Altman, CEO of OpenAI, who in May 2023 — months after the launch of ChatGPT — undertook a world tour with statespersons to showcase AI’s benefits and warn of its dangers.

Which of these two currents deserves more attention? It depends whom you ask. But sticking to the facts, both are equally detached from reality.

Does synthetic intelligence exist?

There is no scientific evidence that generative AI tools literally understand a given fragment of text. Yet, partly because of our own biases, people interpret a coherent response as evidence of intelligence. “Language models only manipulate form; they imitate how people use language in many different contexts,” linguist Emily Bender said in a recent interview with EL PAÍS.

“Machines don’t need to be intelligent in exactly the way humans are to be highly effective at predicting and steering the world,” Soares says by email. “AI developers are very good at improving machines every year. AI could be more effective than people because it can be faster than a human brain or operate with more complex cognitive algorithms,” adds the former Google and Microsoft employee.

That large language models can hold conversations with users, summarize texts, or solve mathematical problems may lead some to think they are intelligent or conscious. For now, however, they are programs that map patterns over enormous datasets. Why assume one of these programs could suddenly become conscious or pursue its own agenda? “No one knows whether AIs are conscious in the way humans are,” Soares replies. “They are huge, complicated systems that were not developed carefully like traditional software; they don’t follow instructions that were carefully programmed by humans. They are enormous trained entities that no one, not even their creators, understands,” he adds, referring to the opacity surrounding deep learning (the process by which a system takes training data and forms patterns on its own). As for the possibility that AI will develop its own goals, Soares argues that this is already happening. He cites Moltbook, the social network of AI agents, though he omits that someone placed those agents there (and likely assigned them roles).

Why would it want to kill us?

The human brain is the product of evolution and therefore carries built-in goals such as feeding, reproducing, and avoiding harm. Synthetic intelligence has not evolved, so it does not necessarily incorporate innate objectives. For some experts the question arises: can we ensure a potential intelligence will have goals that benefit us? That is the so-called alignment problem, a concept introduced by Soares and Yudkowsky in 2014.

Soares believes the solution is to limit the development of AI that is “becoming smarter and smarter in ways no one understands.” There is no need to eliminate large language models, self-driving cars, or AI that helps discover new drugs — only deep learning. “When our leaders finally understand how dangerous a superintelligence could be, they will surely end this suicidal race. AI poses far more risk than we are willing to accept in any other industry,” he says.

But whether such a superintelligence could actually come into existence remains to be seen. For now, it is speculative. The majority opinion is that it is “unlikely” or “very unlikely” we will see it, according to 76% of the 475 AI academics and professionals surveyed a year ago by the Association for the Advancement of Artificial Intelligence.

Two sides of the same coin

Some entrepreneurs within the AI sector itself, such as Altman or Alex Karp, Palantir’s CEO, also argue that AI could become immensely powerful — powerful enough to be a danger to society. The subtext is that because of that potential, investors should trust only the most capable companies (their own). And if the technology is so powerful, it would be foolish not to invest in it.

Emily Bender and Alex Hanna argue in their book The AI Con: How to Fight Big Tech’s Hype and Create the Future We Want, that AI catastrophists and accelerationists are two sides of the same coin. “Those who love AI say that superintelligence is inevitable and will solve all our problems. And those who hate it say it’s inevitable and will kill us all. Essentially the same thing, but with a different twist at the end,” Bender told EL PAÍS.

Sign up for our weekly newsletter to get more English-language news coverage from EL PAÍS USA Edition