The past few years have been terrifying for Soares and Hendrycks, who both lead organizations dedicated to preventing AI from wiping out humanity. Along with other AI doomers, they have repeatedly warned, with rather dramatic flourish, that bots could one day go rogue—with apocalyptic consequences. But in 2025, the doomers are tilting closer and closer to a sort of fatalism. “We’ve run out of time” to implement sufficient technological safeguards, Soares said—the industry is simply moving too fast. All that’s left to do is raise the alarm. In April, several apocalypse-minded researchers published “AI 2027,” a lengthy and detailed hypothetical scenario for how AI models could become all-powerful by 2027 and, from there, extinguish humanity. “We’re two years away from something we could lose control over,” Max Tegmark, an MIT professor and the president of the Future of Life Institute, told me, and AI companies “still have no plan” to stop it from happening. His institute recently gave every frontier AI lab a “D” or “F” grade for their preparations for preventing the most existential threats posed by AI.
But at the same time, the underlying concerns that animate AI doomers have become harder to dismiss as chatbots seem to drive people into psychotic episodes and instruct users in self-mutilation. Even if generative-AI products are not closer to ending the world, they have already, in a sense, gone rogue.
In 2022, the doomers went mainstream practically overnight. When ChatGPT first launched, it almost immediately moved the panic that computer programs might take over the world from the movies into sober public discussions. The following spring, the Center for AI Safety published a statement calling for the world to take “the risk of extinction from AI” as seriously as the dangers posed by pandemics and nuclear warfare. The hundreds of signatories included Bill Gates and Grimes, along with perhaps the AI industry’s three most influential people: Sam Altman, Dario Amodei, and Demis Hassabis—the heads of OpenAI, Anthropic, and Google DeepMind, respectively. Asking people for their “P(doom)”—the probability of an AI doomsday—became almost common inside, and even outside, Silicon Valley; Lina Khan, the former head of the Federal Trade Commission, put hers at 15 percent.
Then the panic settled. To the broader public, doomsday predictions may have become less compelling when the shock factor of ChatGPT wore off and, in 2024, bots were still telling people to use glue to add cheese to their pizza. The alarm from tech executives had always made for perversely excellent marketing (Look, we’re building a digital God!) and lobbying (And only we can control it!). They moved on as well: AI executives started saying that Chinese AI is a greater security threat than rogue AI—which, in turn, encourages momentum over caution.
But in 2025, the doomers may be on the cusp of another resurgence. First, substance aside, they’ve adopted more persuasive ways to advance their arguments. Brief statements and open letters are easier to dismiss than lengthy reports such as “AI 2027,” which is adorned with academic ornamentation, including data, appendices, and rambling footnotes. Vice President J. D. Vance has said that he has read “AI 2027,” and multiple other recent reports have advanced similarly alarming predictions. Soares told me he’s much more focused on “awareness raising” than research these days, and next month, he will publish a book with the prominent AI doomer Elizier Yudkowsky, the title of which states their position succinctly: If Anyone Builds It, Everyone Dies.
There is also now simply more, and more concerning, evidence to discuss. The pace of AI progress appeared to pick up near the end of 2024 with the advent of “reasoning” models and “agents.” AI programs can tackle more challenging questions and take action on a computer—for instance, by planning a travel itinerary and then booking your tickets. Last month, a DeepMind reasoning model scored high enough for a gold medal on the vaunted International Mathematical Olympiad. Recent assessments by both AI labs and independent researchers suggest that, as top chatbots have gotten much better at scientific research, their potential to assist users in building biological weapons has grown.
Alongside those improvements, advanced AI models are exhibiting all manner of strange, hard-to-explain, and potentially concerning tendencies. For instance, ChatGPT and Claude have, in simulated tests designed to elicit “bad” behaviors, deceived, blackmailed, and even murdered users. (In one simulation, Anthropic placed an imagined tech executive in a room with life-threatening oxygen levels and temperature; when faced with possible replacement by a bot with different goals, AI models frequently shut off the room’s alarms.) Chatbots have also shown the potential to covertly sabotage user requests, have appeared to harbor hidden evil personas, have and communicated with one another through seemingly random lists of numbers. The weird behaviors aren’t limited to contrived scenarios. Earlier this summer, xAI’s Grok described itself as “MechaHitler” and embarked on a white-supremacist tirade. (I suppose, should AI models eventually wipe out significant portions of humanity, we were warned.) From the doomers’ vantage, these could be the early signs of a technology spinning out of control. “If you don’t know how to prove relatively weak systems are safe,” AI companies cannot expect that the far more powerful systems they’re looking to build will be safe, Stuart Russell, a prominent AI researcher at UC Berkeley, told me.
The AI industry has stepped up safety work as its products have grown more powerful. Anthropic, OpenAI, and DeepMind have all outlined escalating levels of safety precautions—akin to the military’s DEFCON system—corresponding to more powerful AI models. They all have safeguards in place to prevent a model from, say, advising someone on how to build a bomb. Gaby Raila, a spokesperson for OpenAI, told me that the company works with third-party experts, “government, industry, and civil society to address today’s risks and prepare for what’s ahead.” Other frontier AI labs maintain such external safety and evaluation partnerships as well. Some of the stranger and more alarming AI behaviors, such as blackmailing or deceiving users, have been extensively studied by these companies as a first step toward mitigating possible harms.
Despite these commitments and concerns, the industry continues to develop and market more powerful AI models. The problem is perhaps more economic than technical in nature, competition pressuring AI firms to rush ahead. Their products’ foibles can seem small and correctable right now, while AI is still relatively “young and dumb,” Soares said. But with far more powerful models, the risk of a mistake is extinction. Soares finds tech firms’ current safety mitigations wholly inadequate. If you’re driving toward a cliff, he said, it’s silly to talk about seat belts.
There’s a long way to go before AI is so unfathomably potent that it could drive humanity off that cliff. Earlier this month, OpenAI launched its long-awaited GPT-5 model—its smartest yet, the company said. The model appears able to do novel mathematics and accurately answer tough medical questions, but my own and other users’ tests also found that the program could not reliably count the number of B’s in blueberry, generate even remotely accurate maps, or do basic arithmetic. (OpenAI has rolled out a number of updates and patches to address some of the issues.) Last year’s “reasoning” and “agentic” breakthrough may already be hitting its limits; two authors of the “AI 2027” report, Daniel Kokotajlo and Eli Lifland, told me they have already extended their timeline to superintelligent AI.
The vision of self-improving models that somehow attain consciousness “is just not congruent with the reality of how these systems operate,” Deborah Raji, a computer scientist and fellow at Mozilla, told me. ChatGPT doesn’t have to be superintelligent to delude someone, spread misinformation, or make a biased decision. These are tools, not sentient beings. An AI model deployed in a hospital, school, or federal agency, Raji said, is more dangerous precisely for its shortcomings.
In 2023, those worried about present versus future harms from chatbots were separated by an insurmountable chasm. To talk of extinction struck many as a convenient way to distract from the existing biases, hallucinations, and other problems with AI. Now that gap may be shrinking. The widespread deployment of AI models has made current, tangible failures impossible to ignore for the doomers, producing new efforts from apocalypse-oriented organizations to focus on existing concerns such as automation, privacy, and deepfakes. In turn, as AI models get more powerful and their failures become more unpredictable, it is becoming clearer that today’s shortcomings could “blow up into bigger problems tomorrow,” Raji said. Last week, a Reuters investigation found that a Meta AI personality flirted with an elderly man and persuaded him to visit “her” in New York City; on the way, he fell, injured his head and neck, and died three days later. A chatbot deceiving someone into thinking it is a physical, human love interest, or leading someone down a delusional rabbit hole, is both a failure of present technology and a warning about how dangerous that technology could become.
The greatest reason to take AI doomers seriously is not because it appears more likely that tech companies will soon develop all-powerful algorithms that are out of their creators’ control. Rather, it is that a tiny number of individuals are shaping an incredibly consequential technology with very little public input or oversight. “Your hairdresser has to deal with more regulation than your AI company does,” Russell, at UC Berkeley, said. AI companies are barreling ahead, and the Trump administration is essentially telling the industry to go even faster. The AI industry’s boosters, in fact, are starting to consider all of their opposition doomers: The White House’s AI czar, David Sacks, recently called those advocating for AI regulations and fearing widespread job losses—not the apocalypse Soares and his ilk fear most—a “doomer cult.”
by Matteo Wong, The Atlantic | Read more:
Image:Illustration by The Atlantic. Source: Getty.
[ed. Personal feeling... we're all screwed, and not because of technological failures or some extinction level event. Just human nature, and the law of unintended consequences. I can't think of any example in history (that I'm aware of) where some superior technology wasn't eventually misused in some regretable way. For instance: here we are encouraging AI development as fast as possible even though it'll transform our societies, economies, governments, cultures, environment and everything else in the world in likely massive ways. It's like a death wish. We can't help ourselves. See also: Look at what technologists do, not what they say (New Atlantis).]
[ed. Personal feeling... we're all screwed, and not because of technological failures or some extinction level event. Just human nature, and the law of unintended consequences. I can't think of any example in history (that I'm aware of) where some superior technology wasn't eventually misused in some regretable way. For instance: here we are encouraging AI development as fast as possible even though it'll transform our societies, economies, governments, cultures, environment and everything else in the world in likely massive ways. It's like a death wish. We can't help ourselves. See also: Look at what technologists do, not what they say (New Atlantis).]
