But the rewards for disempowering ourselves are magnificent. The global economy booms, and Americans grow richer than ever before. We are given new cures and cheap energy and wondrous entertainments. Machines perform our labor and mood-enhancing drugs protect our psyches. We need not struggle — perhaps we need not even work — to live lives, long lives, of luxury and ease.
Would you press the button?
The A.I. conversation centers questions of catastrophe. I want to center the question of success: What if A.I. works not as its architects fear, but as its inventors and investors hope? Wander outside of Silicon Valley’s p(doom) — their shorthand for the probability that A.I. extinguishes the human race — and you might notice that their p(utopia) has problems, too. They imagine something like the world I sketched above, but more extreme, with A.I. playing the role assigned to China.
An amusing collision of these visions came when Jacob Coxon, the 27-year-old OpenAI and Anthropic whistle-blower, sat down with Jon Stewart on “The Daily Show.” Coxon was there to warn that A.I. might kill us all, but when Stewart offhandedly mocked the “future utopia” of the A.I. architects, Coxon stopped him. “Well, I am a believer in the future utopia,” Coxon said. But what he said next sounded grimmer: “So I heard you talking about jobs. I think there will be no jobs.”
Coxon went on to describe “the world that is being envisioned by the people building this technology” as “no jobs. Universal high income. The dividends from entirely automated economic production can be distributed among real humans.”
This vision, or some variant of it, is common enough at A.I. companies. I have heard it often. But most people, face to face with this idea, are incredulous at hearing this world — in which virtually all advanced cognitive work is done by A.I. — described as utopian. Stewart certainly was. “So we’ll all just be performing in a play?” he asked.
I tend toward skepticism of the jobless future. Human beings are good at nothing if not finding something to do. (Who would have imagined, in our grueling agrarian past, that podcasting could be a full-time job?) But I find it easy to imagine how human control and comprehension of much of society could be handed over to machines. A few recent developments — some reflecting the outer boundary of what advanced A.I. can do and the other reflecting its advancing utility in daily life — reveal the ways it is already happening.
A few weeks ago, OpenAI dumped more than 700 manuscripts proposing solutions to nearly 400 advanced mathematical problems all at once. If you’re not a mathematician, and you are not close to any mathematicians, this may have passed by unnoticed. Within mathematics, it was an earthquake.
I am not a mathematician, but my father is. The area of math he worked in had to do with the way electrons move through disordered crystals. Sitting at the center of his field were two problems that were more than just unsolved; they were, my father says, “totally out of reach — nobody had any idea how to solve them.” OpenAI’s systems solved them in an afternoon. “It is the end of mathematics as we know it,” he texted me. “It is the first time I am glad I have retired.”
The newly formed Association for Human Mathematics, which has about 1,000 members, quickly released a statement: “Mathematicians did not ask for this work to be done.” It’s an interesting sentence. Should mathematicians be able to decide which math problems do and do not get solved? How do we weigh the benefits and costs of a world where the frontier of mathematics is advancing rapidly, but human beings no longer understand it?
Imagine the OpenAI process running repeatedly, over a period of years: A.I. swarms solving problems no human being has solved, then using those solutions to pose new problems, which are solved by other A.I. swarms that have absorbed the work done by their predecessors, which allows for the formulation of yet more problems, and on and on. It is too much new math for humans to comprehend, but A.I.s can absorb it easily, and they do, driving forward advances in material sciences and energy production. Human beings do not understand all this new science, but they do benefit from it.
Is that world so different from the one we now inhabit? Few of us understand much about the borderlands of human knowledge. I cannot remotely comprehend the math my father did. How different is it if none of us can understand the math the A.I. does? And yet, a world where the farthest edge of knowledge and progress rests in algorithmic systems, rather than human minds, does seem different to me. Human control — which is, in my view, inseparable from human comprehension — is worth something, is it not?
As I was writing this, I read about a study in which theoretical physicists posed 50 complex, unpublished problems to advanced A.I. systems. These problems were difficult but solvable, taking a physicist who was expert in the particular area of the problem about a day to solve. The best A.I. solved 98 percent of them, at a cost of less than $3 per problem.
The physicists initially thought the A.I. had performed worse than it had, but the model insisted that it was the humans, not the A.I., that had erred, and it was right. “Frankly, I wasn’t expecting this day to come as soon as it has,” Suvrat Raju, one of the physicists behind the study, wrote. “After we found these results, I couldn’t sleep properly for a couple of nights.”
The grimmest A.I. stories often center on a single moment when superintelligence slips out of our control and begins plotting our demise. But the scenario I find more likely and more irresistible is called “gradual disempowerment.”
In this telling, there is no one moment when humanity loses control; there is instead a progression of small moments when we willingly give control away because it is the rational thing to do. Firms turn first more of their work and then more of their decision making over to advanced A.I.s because it gives them a competitive edge. Governments turn more battlefield decisions over to A.I. because it makes their forces swifter and more lethal. A.I. companies turn to recursive self-improvement — where the A.I. autonomously creates the next A.I. model — because it vaults them ahead of their competitors.
The key to this process is competitive pressure. Companies and governments and A.I. labs that resist are outcompeted by peers that give A.I.s more control and autonomy. Advances in A.I. systems — and, eventually, in A.I.-controlled drones and robotics — keep expanding the amount of work A.I.s can do better than human beings. Even assuming the A.I.s remain helpful and pliant, more and more of society’s basic functioning slips beyond the grasp of human beings.
by Ezra Klein, NY Times | Read more:
Image: Tamara Porras for The New York Times[ed. We're screwed. Not because there aren't well-intentioned and very capable people working on AI safety and future planning but because the incentives to keep going as fast and far as we can are too great. The next 2-3 years will likely be historic, as various crises converge (AI, politics, finance, climate). See also: The AI Doc: (Or How I Became an Apocaloptimist) on Netflix. Highly recommended.]
