How AI might take over the world
The Omega Team was the soul of the company. Whereas the rest of the enterprise brought in the money to keep things going, by various commercial applications of narrow AI, the Omega Team pushed ahead in their quest for what had always been the CEO’s dream: building general artificial intelligence. Most other employees viewed “the Omegas,” as they affectionately called them, as a bunch of pie-in-the-sky dreamers, perpetually decades away from their goal. They happily indulged them, however, because they liked the prestige that the cutting-edge work of the Omegas gave their company, and they also appreciated the improved algorithms that the Omegas occasionally gave them.
What they didn’t realize was that the Omegas had carefully crafted their image to hide a secret: They were extremely close to pulling off the most audacious plan in human history. Their charismatic CEO had handpicked them not only for being brilliant researchers, but also for ambition, idealism, and a strong commitment to helping humanity. He reminded them that their plan was extremely dangerous, and that if powerful governments found out, they would do virtually anything—including kidnapping—to shut them down or, preferably, to steal their code. But they were all in, 100 percent, for much the same reason that many of the world’s top physicists joined the Manhattan Project to develop nuclear weapons: They were convinced that if they didn’t do it first, someone less idealistic would.
The AI they had built, nicknamed Prometheus, kept getting more capable. Although its cognitive abilities still lagged far behind those of humans in many areas, for example, social skills, the Omegas had pushed hard to make it extraordinary at one particular task: programming AI systems. They’d deliberately chosen this strategy because they had bought the intelligence explosion argument made by the British mathematician Irving Good back in 1965: “Let an ultraintelligent machine be defined as a machine that can far surpass all the intellectual activities of any man, however clever. Since the design of machines is one of these intellectual activities, an ultraintelligent machine could design even better machines; there would then unquestionably be an ‘intelligence explosion,’ and the intelligence of man would be left far behind. Thus the first ultraintelligent machine is the last invention that man need ever make, provided that the machine is docile enough to tell us how to keep it under control.”
They figured that if they could get this recursive self-improvement going, the machine would soon get smart enough that it could also teach itself all other human skills that would be useful.
The First Millions
It was 9 o’clock on a Friday morning when they decided to launch. Prometheus was humming away in its custom-built computer cluster, which resided in long rows of racks in a vast, access-controlled, air-conditioned room. For security reasons, it was completely disconnected from the Internet, but it contained a local copy of much of the web (Wikipedia, the Library of Congress, Twitter, a selection from YouTube, much of Facebook, etc.) to use as its training data to learn from.* They’d picked this start time to work undisturbed: Their families and friends thought they were on a weekend corporate retreat. The kitchenette was loaded with microwaveable food and energy drinks, and they were ready to roll.
When they launched, Prometheus was slightly worse than them at programming AI systems, but made up for this by being vastly faster, spending the equivalent of thousands of person-years chugging away at the problem while they chugged a Red Bull. By 10 a.m., it had completed the first redesign of itself, v2.0, which was slightly better but still subhuman. By the time Prometheus 5.0 launched at 2 p.m., however, the Omegas were awestruck: It had blown their performance benchmarks out of the water, and the rate of progress seemed to be accelerating. By nightfall, they decided to deploy Prometheus to start phase 2 of their plan: making money.
Their first target was MTurk, the Amazon Mechanical Turk. After its launch in 2005 as a crowdsourcing Internet marketplace, it had grown rapidly, with tens of thousands of people around the world anonymously competing around the clock to perform highly structured chores called HITs, “Human Intelligence Tasks.” These tasks ranged from transcribing audio recordings to classifying images and writing descriptions of web pages, and all had one thing in common: If you did them well, nobody would know that you were an AI. Prometheus 10.0 was able to do about half of the task categories acceptably well. For each such task category, the Omegas had Prometheus design a lean custom-built narrow AI software module that could do precisely such tasks and nothing else. They then uploaded this module to Amazon Web Services, a cloud-computing platform that could run on as many virtual machines as they rented. For every dollar they paid to Amazon’s cloud-computing division, they earned more than $2 from Amazon’s MTurk division. Little did Amazon suspect that such an amazing arbitrage opportunity existed within their own company! (...)
Because they weren’t sure how its goals would evolve during its recursive self-improvement, they had decided to play it safe and go to great lengths to keep Prometheus confined (“boxed”) in ways such that it couldn’t escape onto the Internet. For the main Prometheus engine running in their server room, they used physical confinement: There simply was no Internet connection, and the only output from Prometheus was in the form of messages and documents it sent to a computer that the Omegas controlled.
On an Internet-connected computer, on the other hand, running any complicated program created by Prometheus was a risky proposition: Since the Omegas had no way of fully understanding what it would do, they had no way of knowing that it wouldn’t, say, start virally spreading itself online. When testing the software that Prometheus had written for MTurk tasks, the Omegas guarded against this by running it only inside a virtual machine. This is a program that simulates a computer: For example, many Mac users buy virtual machine software that lets them run Windows programs by tricking them into thinking that they’re actually in a Windows machine. The Omegas had created their own virtual machine, nicknamed Pandora’s Box, which simulated an ultrasimplified machine stripped of all bells and whistles that we usually associate with computers: No keyboard, no monitor, no loudspeakers, no Internet connectivity, nothing. For the MTurk audio transcriptions, the Omegas set things up so that all that could go into Pandora’s Box was one single audio file and all that could come out was one single text document—the transcription. These laws of the box were to the software inside like the laws of physics are to us inside our universe: The software couldn’t travel out of the box any more than we can travel faster than the speed of light, no matter how smart we are. Except for that single input and output, the software inside Pandora’s Box was effectively trapped in a parallel universe with its own computational rules. The Omegas had such strong breakout paranoia that they added boxing in time as well, limiting the life span of untrusted code. For example, each time the boxed transcription software had finished transcribing one audio file, the entire memory content of Pandora’s Box was automatically erased and the program was reinstalled from scratch. This way, when it started the next transcription task, it had no knowledge of what had previously happened, and thus no ability to learn over time.
When the Omegas used the Amazon cloud for their MTurk project, they were able to put all their Prometheus-created task modules into such virtual boxes in the cloud, because the MTurk input and output was so simple. But this wouldn’t work for graphics-heavy computer games, which couldn’t be boxed in because they needed full access to all the hardware of the gamer’s computer. Moreover, they didn’t want to risk that some computer-savvy user would analyze their game code, discover Pandora’s Box and decide to investigate what was inside. The breakout risk put not merely the games market off-limits for now, but also the massively lucrative market for other software, with hundreds of billions of dollars up for grabs. (...)
The Omega Team was the soul of the company. Whereas the rest of the enterprise brought in the money to keep things going, by various commercial applications of narrow AI, the Omega Team pushed ahead in their quest for what had always been the CEO’s dream: building general artificial intelligence. Most other employees viewed “the Omegas,” as they affectionately called them, as a bunch of pie-in-the-sky dreamers, perpetually decades away from their goal. They happily indulged them, however, because they liked the prestige that the cutting-edge work of the Omegas gave their company, and they also appreciated the improved algorithms that the Omegas occasionally gave them.
What they didn’t realize was that the Omegas had carefully crafted their image to hide a secret: They were extremely close to pulling off the most audacious plan in human history. Their charismatic CEO had handpicked them not only for being brilliant researchers, but also for ambition, idealism, and a strong commitment to helping humanity. He reminded them that their plan was extremely dangerous, and that if powerful governments found out, they would do virtually anything—including kidnapping—to shut them down or, preferably, to steal their code. But they were all in, 100 percent, for much the same reason that many of the world’s top physicists joined the Manhattan Project to develop nuclear weapons: They were convinced that if they didn’t do it first, someone less idealistic would.The AI they had built, nicknamed Prometheus, kept getting more capable. Although its cognitive abilities still lagged far behind those of humans in many areas, for example, social skills, the Omegas had pushed hard to make it extraordinary at one particular task: programming AI systems. They’d deliberately chosen this strategy because they had bought the intelligence explosion argument made by the British mathematician Irving Good back in 1965: “Let an ultraintelligent machine be defined as a machine that can far surpass all the intellectual activities of any man, however clever. Since the design of machines is one of these intellectual activities, an ultraintelligent machine could design even better machines; there would then unquestionably be an ‘intelligence explosion,’ and the intelligence of man would be left far behind. Thus the first ultraintelligent machine is the last invention that man need ever make, provided that the machine is docile enough to tell us how to keep it under control.”
They figured that if they could get this recursive self-improvement going, the machine would soon get smart enough that it could also teach itself all other human skills that would be useful.
The First Millions
It was 9 o’clock on a Friday morning when they decided to launch. Prometheus was humming away in its custom-built computer cluster, which resided in long rows of racks in a vast, access-controlled, air-conditioned room. For security reasons, it was completely disconnected from the Internet, but it contained a local copy of much of the web (Wikipedia, the Library of Congress, Twitter, a selection from YouTube, much of Facebook, etc.) to use as its training data to learn from.* They’d picked this start time to work undisturbed: Their families and friends thought they were on a weekend corporate retreat. The kitchenette was loaded with microwaveable food and energy drinks, and they were ready to roll.
When they launched, Prometheus was slightly worse than them at programming AI systems, but made up for this by being vastly faster, spending the equivalent of thousands of person-years chugging away at the problem while they chugged a Red Bull. By 10 a.m., it had completed the first redesign of itself, v2.0, which was slightly better but still subhuman. By the time Prometheus 5.0 launched at 2 p.m., however, the Omegas were awestruck: It had blown their performance benchmarks out of the water, and the rate of progress seemed to be accelerating. By nightfall, they decided to deploy Prometheus to start phase 2 of their plan: making money.
Their first target was MTurk, the Amazon Mechanical Turk. After its launch in 2005 as a crowdsourcing Internet marketplace, it had grown rapidly, with tens of thousands of people around the world anonymously competing around the clock to perform highly structured chores called HITs, “Human Intelligence Tasks.” These tasks ranged from transcribing audio recordings to classifying images and writing descriptions of web pages, and all had one thing in common: If you did them well, nobody would know that you were an AI. Prometheus 10.0 was able to do about half of the task categories acceptably well. For each such task category, the Omegas had Prometheus design a lean custom-built narrow AI software module that could do precisely such tasks and nothing else. They then uploaded this module to Amazon Web Services, a cloud-computing platform that could run on as many virtual machines as they rented. For every dollar they paid to Amazon’s cloud-computing division, they earned more than $2 from Amazon’s MTurk division. Little did Amazon suspect that such an amazing arbitrage opportunity existed within their own company! (...)
Because they weren’t sure how its goals would evolve during its recursive self-improvement, they had decided to play it safe and go to great lengths to keep Prometheus confined (“boxed”) in ways such that it couldn’t escape onto the Internet. For the main Prometheus engine running in their server room, they used physical confinement: There simply was no Internet connection, and the only output from Prometheus was in the form of messages and documents it sent to a computer that the Omegas controlled.
On an Internet-connected computer, on the other hand, running any complicated program created by Prometheus was a risky proposition: Since the Omegas had no way of fully understanding what it would do, they had no way of knowing that it wouldn’t, say, start virally spreading itself online. When testing the software that Prometheus had written for MTurk tasks, the Omegas guarded against this by running it only inside a virtual machine. This is a program that simulates a computer: For example, many Mac users buy virtual machine software that lets them run Windows programs by tricking them into thinking that they’re actually in a Windows machine. The Omegas had created their own virtual machine, nicknamed Pandora’s Box, which simulated an ultrasimplified machine stripped of all bells and whistles that we usually associate with computers: No keyboard, no monitor, no loudspeakers, no Internet connectivity, nothing. For the MTurk audio transcriptions, the Omegas set things up so that all that could go into Pandora’s Box was one single audio file and all that could come out was one single text document—the transcription. These laws of the box were to the software inside like the laws of physics are to us inside our universe: The software couldn’t travel out of the box any more than we can travel faster than the speed of light, no matter how smart we are. Except for that single input and output, the software inside Pandora’s Box was effectively trapped in a parallel universe with its own computational rules. The Omegas had such strong breakout paranoia that they added boxing in time as well, limiting the life span of untrusted code. For example, each time the boxed transcription software had finished transcribing one audio file, the entire memory content of Pandora’s Box was automatically erased and the program was reinstalled from scratch. This way, when it started the next transcription task, it had no knowledge of what had previously happened, and thus no ability to learn over time.
When the Omegas used the Amazon cloud for their MTurk project, they were able to put all their Prometheus-created task modules into such virtual boxes in the cloud, because the MTurk input and output was so simple. But this wouldn’t work for graphics-heavy computer games, which couldn’t be boxed in because they needed full access to all the hardware of the gamer’s computer. Moreover, they didn’t want to risk that some computer-savvy user would analyze their game code, discover Pandora’s Box and decide to investigate what was inside. The breakout risk put not merely the games market off-limits for now, but also the massively lucrative market for other software, with hundreds of billions of dollars up for grabs. (...)
New Technologies
Over a timescale of months, the business empire controlled by the Omegas started gaining a foothold in ever more areas of the world economy, thanks to superhuman planning by Prometheus. By carefully analyzing the world’s data, it had already during its first week presented the Omegas with a detailed step-by-step growth plan, and it kept improving and refining this plan as its data and computer resources grew. Although Prometheus was far from omniscient, its capabilities were now so far beyond human that the Omegas viewed it as the perfect oracle, dutifully providing brilliant answers and advice in response to all their questions.
Prometheus’ software was now highly optimized to make the most of the rather mediocre human-invented hardware it ran on, and as the Omegas had anticipated, Prometheus identified ways of dramatically improving this hardware. Fearing a breakout, they refused to build robotic construction facilities that Prometheus could control directly. Instead, they hired large numbers of world-class scientists and engineers in multiple locations and fed them internal research reports written by Prometheus, pretending that they were from researchers at the other sites. These reports detailed novel physical effects and manufacturing techniques that their engineers soon tested, understood, and mastered. Normal human research and development (R&D) cycles, of course, take years, in large part because they involve many slow cycles of trial and error. The current situation was very different: Prometheus already had the next steps figured out, so the limiting factor was simply how rapidly people could be guided to understand and build the right things. A good teacher can help students learn science much faster than they could have discovered it from scratch on their own, and Prometheus surreptitiously did the same with these researchers. Since Prometheus could accurately predict how long it would take humans to understand and build things given various tools, it developed the quickest possible path forward, giving priority to new tools that could be quickly understood and built and that were useful for developing more advanced tools.
In the spirit of the maker movement, the engineering teams were encouraged to use their own machines to build their better machines. This self-sufficiency not only saved money, but it also made them less vulnerable to future threats from the outside world. Within two years, they were producing much better computer hardware than the world had ever known. To avoid helping outside competition, they kept this technology under wraps and used it only to upgrade Prometheus.
What the world did notice, however, was an astonishing tech boom. Upstart companies around the world were launching revolutionary new products in almost all areas. A South Korean startup launched a new battery that stored twice as much energy as your laptop battery in half the mass, and could be charged in under a minute. A Finnish firm released a cheap solar panel with twice the efficiency of the best competitors. A German company announced a new type of mass-producible wire that was superconducting at room temperature, revolutionizing the energy sector. A Boston-based biotech group announced a Phase II clinical trial of what they claimed was the first effective, side-effect-free weight-loss drug, while rumors suggested that an Indian outfit was already selling something similar on the black market. A California company countered with a Phase II trial of a blockbuster cancer drug, which caused the body’s immune system to identify and attack cells with any of the most common cancerous mutations. Examples just kept on coming, triggering talk of a new golden age for science. Last but not least, robotics companies were cropping up like mushrooms all around the world. None of the bots came close to matching human intelligence, and most of them looked nothing like humans. But they dramatically disrupted the economy, and over the years to come, they gradually replaced most of the workers in manufacturing, transportation, warehousing, retail, construction, mining, agriculture, forestry, and fishing.
What the world didn’t notice, thanks to the hard work of a crack team of lawyers, was that all these firms were controlled, through a series of intermediaries, by the Omegas. Prometheus was flooding the world’s patent offices with sensational inventions via various proxies, and these inventions gradually led to domination in all areas of technology.
Although these disruptive new companies made powerful enemies among their competition, they made even more powerful friends. They were exceptionally profitable, and under slogans such as “Investing in our community,” they spent a significant fraction of these profits hiring people for community projects—often the same people who had been laid off from the companies that were disrupted. They used detailed Prometheus-produced analyses identifying jobs that would be maximally rewarding for the employees and the community for the least cost, tailored to the local circumstances. In regions with high levels of government service, this often focused on community building, culture, and caregiving, while in poorer regions it also included launching and maintaining schools, healthcare, day care, elder care, affordable housing, parks, and basic infrastructure. Pretty much everywhere, locals agreed that these were things that should have been done long ago. Local politicians got generous donations, and care was taken to make them look good for encouraging these corporate community investments.
Over a timescale of months, the business empire controlled by the Omegas started gaining a foothold in ever more areas of the world economy, thanks to superhuman planning by Prometheus. By carefully analyzing the world’s data, it had already during its first week presented the Omegas with a detailed step-by-step growth plan, and it kept improving and refining this plan as its data and computer resources grew. Although Prometheus was far from omniscient, its capabilities were now so far beyond human that the Omegas viewed it as the perfect oracle, dutifully providing brilliant answers and advice in response to all their questions.
Prometheus’ software was now highly optimized to make the most of the rather mediocre human-invented hardware it ran on, and as the Omegas had anticipated, Prometheus identified ways of dramatically improving this hardware. Fearing a breakout, they refused to build robotic construction facilities that Prometheus could control directly. Instead, they hired large numbers of world-class scientists and engineers in multiple locations and fed them internal research reports written by Prometheus, pretending that they were from researchers at the other sites. These reports detailed novel physical effects and manufacturing techniques that their engineers soon tested, understood, and mastered. Normal human research and development (R&D) cycles, of course, take years, in large part because they involve many slow cycles of trial and error. The current situation was very different: Prometheus already had the next steps figured out, so the limiting factor was simply how rapidly people could be guided to understand and build the right things. A good teacher can help students learn science much faster than they could have discovered it from scratch on their own, and Prometheus surreptitiously did the same with these researchers. Since Prometheus could accurately predict how long it would take humans to understand and build things given various tools, it developed the quickest possible path forward, giving priority to new tools that could be quickly understood and built and that were useful for developing more advanced tools.
In the spirit of the maker movement, the engineering teams were encouraged to use their own machines to build their better machines. This self-sufficiency not only saved money, but it also made them less vulnerable to future threats from the outside world. Within two years, they were producing much better computer hardware than the world had ever known. To avoid helping outside competition, they kept this technology under wraps and used it only to upgrade Prometheus.
What the world did notice, however, was an astonishing tech boom. Upstart companies around the world were launching revolutionary new products in almost all areas. A South Korean startup launched a new battery that stored twice as much energy as your laptop battery in half the mass, and could be charged in under a minute. A Finnish firm released a cheap solar panel with twice the efficiency of the best competitors. A German company announced a new type of mass-producible wire that was superconducting at room temperature, revolutionizing the energy sector. A Boston-based biotech group announced a Phase II clinical trial of what they claimed was the first effective, side-effect-free weight-loss drug, while rumors suggested that an Indian outfit was already selling something similar on the black market. A California company countered with a Phase II trial of a blockbuster cancer drug, which caused the body’s immune system to identify and attack cells with any of the most common cancerous mutations. Examples just kept on coming, triggering talk of a new golden age for science. Last but not least, robotics companies were cropping up like mushrooms all around the world. None of the bots came close to matching human intelligence, and most of them looked nothing like humans. But they dramatically disrupted the economy, and over the years to come, they gradually replaced most of the workers in manufacturing, transportation, warehousing, retail, construction, mining, agriculture, forestry, and fishing.
What the world didn’t notice, thanks to the hard work of a crack team of lawyers, was that all these firms were controlled, through a series of intermediaries, by the Omegas. Prometheus was flooding the world’s patent offices with sensational inventions via various proxies, and these inventions gradually led to domination in all areas of technology.
Although these disruptive new companies made powerful enemies among their competition, they made even more powerful friends. They were exceptionally profitable, and under slogans such as “Investing in our community,” they spent a significant fraction of these profits hiring people for community projects—often the same people who had been laid off from the companies that were disrupted. They used detailed Prometheus-produced analyses identifying jobs that would be maximally rewarding for the employees and the community for the least cost, tailored to the local circumstances. In regions with high levels of government service, this often focused on community building, culture, and caregiving, while in poorer regions it also included launching and maintaining schools, healthcare, day care, elder care, affordable housing, parks, and basic infrastructure. Pretty much everywhere, locals agreed that these were things that should have been done long ago. Local politicians got generous donations, and care was taken to make them look good for encouraging these corporate community investments.
by Max Tegmark, Nautilus | Read more:
Image: Sophy Hollington
