Even knowing they could kill everyone? Zoomer Doomer Paradiso: What Are AI Researchers Thinking?
For a group trying as hard as possible to make sure AI does not kill everyone (another shibboleth, this one significantly clearer), I expected them to be a lot more depressed and gloomy. This is far from the case. If you were to construct a world from first principles that provides for all the physiological, safety, belonging, esteem, and self-actualization needs of its inhabitants in a kind of Maslovian utopia, the solution would be roughly Berkeley shaped and centered around Constellation. If you believe you’re making progress on the most important problem in the world in a tight-knit and well-resourced community of intelligent coworkers who double as your closest friends, what more could you possibly want in life?
In some ways, Constellation seems less like an office and more like a strange idealized version of college. [...]
This is not to say Constellation is stress-free. Numerous Stellies speak of the long hours and burnout involved in trying to keep up with the exponential pace of AI progress. And in the same breath, they’ll express how much they love it and feed off the energy of late nights and the meaning it brings. If taking more time off means, however marginally, that models deployed at massive scale are more dangerous, it becomes difficult to justify relaxing much at all.
Plenty of organizations claim to be mission-driven and provide their employees with prettier environments and cushier amenities, but few claim stakes so high as preventing literal human extinction. Constellation is a paradise of purpose: the camaraderie, intellectual stimulation, long nights, and stress exist only because the feeling that what happens inside matters enormously.
From the outside, there is an obvious and unflattering comparison: the community looks millenarian and cult-like. They’re convinced that history is approaching a discontinuity after which ordinary human life will be either incomprehensibly better or permanently over and reorganize their careers and social lives around preparing for the shift. Indeed, coworkers become friends, friends become housemates and romantic partners, and the whole world becomes highly endogamous. This is not necessarily because anyone objects to dating outsiders; it’s more that the arguments permeate ordinary life so much that being with someone who is not “pilled” can become difficult. After all, if the most important follow-up question to “do you want to have kids?” is not “how many?” but “before or after ASI?”, you’re going to have issues dating the general populace. [...]
When the history of AGI is written, the organizations on those floors will be discussed alongside Anthropic and OpenAI in the way Oak Ridge, the Berkeley Rad Lab, and Los Alamos feature in the history of the Manhattan Project. Like Los Alamos, the population skews young, elite, and imbued with a sense that if they fail, everyone is thoroughly fucked. [...]
Each organization is shaped around what is called the “theory of change”, or ToC. These are always AI safety flavored but often sharply diverging on what they believe is the most important way to prevent disaster. MIRI is the most doomerish of the lot, with its co-founder Eliezer Yudkowsky and president Nate Soares publishing the book If Anyone Builds It, Everyone Dies last year. They believe short-term attempts to make AI more aligned are futile and campaign for building “off-switches” and enacting policies to block ASI. The AI Futures Project instead tries to make possible futures legible enough for governments and other decision-makers to prepare for them, while groups like METR and Redwood focus more narrowly on measuring dangerous capabilities and controlling advanced systems.
If AI safety is a cult, it does an awful job of disseminating and enforcing orthodoxy because anything is up for debate. There is enough heterogeneity among ToCs in AI safety that some believe others’ projects in the movement are truly net negative (“in expectation”, I feel obligated to add), either by advancing AI in a way that improves dangerous capabilities more than safety, generating bad optics, or simply wasting money. [...]
I grant that AI has superhuman computer-based abilities that can easily generate new viruses, hack critical infrastructure, and cause other problems worth taking very seriously. However, unlike the many software engineers and mathematicians I met whose strong suits involve building up beautiful and noiseless abstraction hierarchies with computers, I am deeply skeptical of similarly rapid AI advancements in physical technologies. I was amused by a phenomenon where those closest to math and computer science had the shortest timelines to ASI, and electrical and mechanical engineers had the longest. This is not because engineers in the physical world aren’t acquainted with the magic of frontier AI tools -- the best electrical engineers I know burn upwards of one billion tokens per day on automated experiments, simulation sweeps, and the like. They’re not as bullish because the physical world is more noisy, complicated, and slow than software.
The Radiant
Even though I did not find myself believing in their strongest visions of doom, it’s not too hard to understand the appeal if you psychologically inhabit their world for enough time. There’s a sense you’re right in the middle of world events, with near weekly articles from major outlets featuring METR and Redwood reports. There’s a world of pleasure, community, and meaning in what most believe to be the “probable pre-apocalypse”. There’s a shared map of the many possible roads to hell and a common goal of ensuring we never arrive. There’s a heaven at the last stop before hell. [...]
There is a sense in which the struggle against the legible, articulable Shoggoth may be more meaningful than what comes afterward. Meaning in rationalist circles is driven by expected impact relative to the counterfactual, so I began to wonder whether the world around Constellation is, in some sense, more of a paradise than the AI future they so desperately aim for. If they succeed and the AI is aligned, the handoff complete, and the hyperabundant future secured, what is there to work on that matters?
The AI safety community is deeply consequentialist and derives its status symbols, norms, and sense of purpose from expecting to enact change through work. Having high agency is universally lauded, and many of the pitches to join AI safety center on the belief that the next few years are the most pivotal time in history. If the appeal of your job and source of meaning is around shaping the future, then what is it like to come down from that world and live with only mundane consequences?
More abstractly, how can you be a consequentialist once you’re not meaningfully in control? Self-actualization through work is the capstone of Maslow’s pyramid, and without it, Constellation would be just like all the other elite college campuses from which its inhabitants came, perhaps with more rigorous epistemic standards.
Various Stellies I talked to didn’t seem too concerned with this outcome -- they’d have won. They plan to do math puzzles and write and stargaze and all the other typical hobbies of rats, but I find it hard to believe a culture which valorizes agency and impact so easily relinquishes its central virtues.
by Connor Blake, Boson Cutter | Read more:
Image: Constellation via
[ed. You can skip the first half (partying in Silicon Valley).]


















