Part 2 of this post then covers Alex Turner’s resignation, and his story about how he tried and failed to prevent Google from signing up to allow the Department of War to use its models for essentially whatever the government wants, including autonomous weapons.
Demis Hassabis sold DeepMind to Google on condition that something like this would not happen. Yet here it is, happening. A cautionary tale. [...]
***
The Core Statement and RequestHe saying we are standing in the foothills of the singularity.
He is excellent about stating that this is big, really big, no bigger than that, it be big.
Demis Hassabis: I’ve spent my whole life working on AGI because I’ve always had a deep conviction that, if built and deployed responsibly, it would prove to be one of the most beneficial and transformative technologies ever invented. AGI cannot be compared to standard technological breakthroughs, not even ones as consequential as the internet or mobile - it is much more akin to the discovery of electricity or fire. If you stop to think about it, we’ve essentially found a way to make sand think. It’s miraculous.Things Left Unsaid
The magnitude of this technology’s impact will be unprecedented, perhaps 10x of the Industrial Revolution at 10x the speed. It will help us solve some of the biggest problems society faces from accelerating drug discovery to developing new clean energy sources to creating novel advanced materials. We could even reach a point where resources are no longer the limiting factor for human progress, leading to an amazing new era of abundance.
There is definitely a ‘don’t say the thing’ aspect of this, where he won’t name what the downside risks actually are. When Demis says ‘experts disagree’ he is rather avoidant about the way in which they disagree here.
Nate Soares (MIRI): I’m glad Demis acknowledges that this is a “pivotal moment in human history” during an “extremely intense” race. I’m disappointed that his proposed solution is a “standards body” to evaluate whether models are dangerous, with no plan for what to do once they are.Clearly this is strategic, but if you don’t already know, or are looking to not realize, it is very easy to come away thinking that Demis does mean the effect on jobs, even though when he says ‘safely’ he very much does not (primarily) mean that.
I’m glad he acknowledges that “experts disagree.” I’m annoyed that he glosses past how the disagreement is about whether there’s a ~5% or ≥50% chance of total catastrophe. We’ve gotta do better.
Aaron Scher: Glad to see AI CEOs speaking publicly about their views on AGI. I think Demis is wrong about his policy prescription: it’s far too little too late. When he says the experts disagree, he means that some think 5% this tech kills literally everybody, some at 40%, some at 90%.
The Proposal
Demis Hassabis: … On the horizon, we will need robust safeguards to maintain control of increasingly agentic, recursively self-improving systems - and tackle unknown issues that will only become clearer over time.He makes clear part of this is about giving us options, including for a slowdown.
… I’ve always believed in the power of human ingenuity and creativity to solve any problem. I’m confident that mitigating the technical risks related to AI is a challenge we can collectively address, but only if we give ourselves the time and space to get this next crucial step right. Currently, as a field and as a wider society, we aren’t doing that.
The strength of this approach is it would be technically focused, while at the same time supporting innovation and incentivising responsible behaviour. It is designed to keep up with the field’s acceleration and adapt to the biggest risks as they are identified, and could be ratcheted up if the seriousness of the situation demands, including coordinating a slowdown in development among the Frontier Labs if deemed necessary.Demis keeps it short, not offering many details. To the extent that he has laid out a proposal, it seems to be a good one. It is definitely an improvement on the margin.
Jack Clark (Anthropic): At this point, everyone at the frontier of AI agrees that third-parties should test out AI systems and use these to develop standards to feed into policy - excellent to see @demishassabis laying out a framework to do this!Thus I file this post and its ask, as high praise, under ‘the least you could do.’
Samuel Hammond: It is striking to see leadership at Google, Anthropic, OpenAI and Microsoft all fairly independently sounding warning alarms about an imminent technological acceleration.
A Good Start But Insufficient
I agree with Peter Wildeford that while better than nothing FINRA is not a great model here, with heightened risk of regulatory capture, and not a substitute for full government action. You need an SEC to your FINRA. That doesn’t mean don’t make the FINRA. It does mean you still need the SEC.
Would such a (at least partly) voluntary regime, only for models intended for release, and without a related binding intentional agreement, be sufficient to solve the problem? No, again it’s just way better than doing nothing, as Peter Wildeford and many others noted.
You do not need to believe, as Aaron Scher and Connor Leahy do below, that only a full halt would be sufficient here, to know we have a long way to go. Demis’s statements here, if you know what they actually mean, imply a level of danger and urgency that is not reflected in the proposal.
I agree with Peter Wildeford that while better than nothing FINRA is not a great model here, with heightened risk of regulatory capture, and not a substitute for full government action. You need an SEC to your FINRA. That doesn’t mean don’t make the FINRA. It does mean you still need the SEC.
Would such a (at least partly) voluntary regime, only for models intended for release, and without a related binding intentional agreement, be sufficient to solve the problem? No, again it’s just way better than doing nothing, as Peter Wildeford and many others noted.
You do not need to believe, as Aaron Scher and Connor Leahy do below, that only a full halt would be sufficient here, to know we have a long way to go. Demis’s statements here, if you know what they actually mean, imply a level of danger and urgency that is not reflected in the proposal.
Eli Tyre: > Initially, Frontier Labs would voluntarily share models with the Standards Body for review up to 30 days before release.
Is this proposal only intended to address risks from models that companies plan to release? If a company develops a frontier model and never releases it, only deploying it internally to develop even more powerful AI capabilities, are they thereby exempt from this oversight scheme?
Connor Leahy: While @demishassabis is right that we need urgent action to address risks as we approach AGI (and superintelligence, I’d add), the correct response to the threats is not a ‘self-regulatory organization’.
We need to prohibit superintelligence, not give industry regulatory power.
Aaron Scher: … The extinction threat, the “only a few short years”, the “10x the Industrial Revolution”—these aren’t indicators that point to “let’s evaluate models to understand their capabilities and have voluntary safety standards”. We need to back off, we need to halt the creation of ASI.
Point 2: I agree with the attached quote that we need more time. But I think Demis’s optimism is a vibe, not a trustworthy basis for predictions. Rob Miles says it best in this video, if an asteroid we’re headed earth’s way 200 years ago, we’d just die 🤷
Point 3: As others have pointed out, it’s not clear that this proposal would reduce risks from internal deployment (it seems to focus on public deployment and pre-deployment testing), but internal deployment is where much of the risk is.
Point 4: I don’t think the proposed body could actually enact, verify, and enforce a slowdown; there’s ambiguity about what’s voluntary. Again, I think we need a long-term international treaty and to actually back off, not just to slow down a little.
by Zvi Mowshowitz, DWV | Read more:
Image: uncredited
[ed. See also: The Voice of Google (New Yorker):]
Google’s founders, Larry Page and Sergey Brin, were bona-fide public figures by then, and self-made billionaires multiple times over, but in Charlie’s they were idols. They would often ascend the stage together, practically matching in sweat-wicking athletic clothes and Crocs. Larry had a dopey perma-smile, and seemed delighted by everything, especially Sergey. Sergey was the straight man, with a faint lilt, a product of his childhood in Russia, and an acrobatic build that made him look like he might launch into a handspring at any moment. Their charisma was unconventional, contextual; you had to be there. The audience of employees lapped up every word, giggled at every dad joke. During a Q. & A. portion of the proceedings, even adversarial questions were absorbed into the Google spirit—it all melted into laughs, love. Merriam-Webster had added “google” to the dictionary the year before. Fortune had crowned it the “Best Company to Work For” in America. Profits were, as the execs loved to boast, “up and to the right,” fuelled by an online-advertising machine that minted cash beyond Wall Street’s wildest dreams. But the company’s financial success felt almost incidental. What mattered, we told ourselves, was the mission—a conviction that technology could improve the world and that we were helping to build the future. The air in Charlie’s buzzed with collective belief.
That first meeting was the only one I’d ever attend as a pure spectator. By week two, I was working the event—cordoning off the Noogler section, handing out extra caps—and I soon began helping to draft bits of Larry and Sergey’s script. A portion of my time was spent supporting the P.R. team, and I started to pick up my first press requests, providing office tours to journalists eager to see the “Google experience” firsthand. I studied a “master workplace talking points” document, which was maintained with input from PeopleOps, which was Google-speak for human resources. This was the era of “bringing your whole self to work,” of shiny, smiling H.R. people doing press hits about the importance of valuing employees’ authentic personhood (always with a telling corollary: “Because that’s how people do their best work!”). I was required to attend a training on “conscious business” with a guy named Fred Kofman, an executive coach whom Sheryl Sandberg credited with shaping her “lean-in” ethos. The course was, theoretically, about living one’s courageous values, but its most salient lesson was that employees should take “unconditional accountability”—which, in practice, sounded a lot like never questioning the higher-ups. The message reiterated over and over was that there were two kinds of people in the world: victims and players. You wanted to be a player at all times.
Despite the lore, Google’s offices didn’t make a big first impression. The bulk of the campus had been quickly converted after its previous occupant went down in the fallout from the dot-com bust. The result was a complex of squat, one- or two-level buildings with metal and glass siding, surrounded by a moat of parking spaces, with Google signs plunked into the dirt out front. But there were plenty of amenities to point out—the massage rooms and nap pods, the dinosaur fossil, the wacky sensory-break touches like ball pits, swings, and yoga balls (even if no one actually seemed to use them). Foreign journalists seemed more skeptical than their American counterparts of perks such as lunch-break haircuts or on-site laundry rooms, which I’d heard described as letting Google be your “housewife.”
***
I started working at Google in the summer of 2007, straight out of college, as a “new-grad associate” in the communications department. My first week, I sat with more than a hundred other “Nooglers” (new Googlers) at the company’s weekly staff meeting, T.G.I.F., wearing matching company-issued propeller caps as a kind of ritual hazing. The venue was Charlie’s Cafe, a multilevel auditorium in the heart of the “Googleplex,” the company’s sprawling campus in Mountain View, California. The event felt less like a corporate meeting than like a weekly revival—part standup set, part science fair, part sermon, all of it fuelled by keg beer.Google’s founders, Larry Page and Sergey Brin, were bona-fide public figures by then, and self-made billionaires multiple times over, but in Charlie’s they were idols. They would often ascend the stage together, practically matching in sweat-wicking athletic clothes and Crocs. Larry had a dopey perma-smile, and seemed delighted by everything, especially Sergey. Sergey was the straight man, with a faint lilt, a product of his childhood in Russia, and an acrobatic build that made him look like he might launch into a handspring at any moment. Their charisma was unconventional, contextual; you had to be there. The audience of employees lapped up every word, giggled at every dad joke. During a Q. & A. portion of the proceedings, even adversarial questions were absorbed into the Google spirit—it all melted into laughs, love. Merriam-Webster had added “google” to the dictionary the year before. Fortune had crowned it the “Best Company to Work For” in America. Profits were, as the execs loved to boast, “up and to the right,” fuelled by an online-advertising machine that minted cash beyond Wall Street’s wildest dreams. But the company’s financial success felt almost incidental. What mattered, we told ourselves, was the mission—a conviction that technology could improve the world and that we were helping to build the future. The air in Charlie’s buzzed with collective belief.
That first meeting was the only one I’d ever attend as a pure spectator. By week two, I was working the event—cordoning off the Noogler section, handing out extra caps—and I soon began helping to draft bits of Larry and Sergey’s script. A portion of my time was spent supporting the P.R. team, and I started to pick up my first press requests, providing office tours to journalists eager to see the “Google experience” firsthand. I studied a “master workplace talking points” document, which was maintained with input from PeopleOps, which was Google-speak for human resources. This was the era of “bringing your whole self to work,” of shiny, smiling H.R. people doing press hits about the importance of valuing employees’ authentic personhood (always with a telling corollary: “Because that’s how people do their best work!”). I was required to attend a training on “conscious business” with a guy named Fred Kofman, an executive coach whom Sheryl Sandberg credited with shaping her “lean-in” ethos. The course was, theoretically, about living one’s courageous values, but its most salient lesson was that employees should take “unconditional accountability”—which, in practice, sounded a lot like never questioning the higher-ups. The message reiterated over and over was that there were two kinds of people in the world: victims and players. You wanted to be a player at all times.
Despite the lore, Google’s offices didn’t make a big first impression. The bulk of the campus had been quickly converted after its previous occupant went down in the fallout from the dot-com bust. The result was a complex of squat, one- or two-level buildings with metal and glass siding, surrounded by a moat of parking spaces, with Google signs plunked into the dirt out front. But there were plenty of amenities to point out—the massage rooms and nap pods, the dinosaur fossil, the wacky sensory-break touches like ball pits, swings, and yoga balls (even if no one actually seemed to use them). Foreign journalists seemed more skeptical than their American counterparts of perks such as lunch-break haircuts or on-site laundry rooms, which I’d heard described as letting Google be your “housewife.”
“Z is is all a big plot to control ze workers, no?” a French reporter said.
At that point, though, I was still learning to see Google through Google’s eyes. I learned to deflect these kinds of questions and pitied the askers, a little bit, for their cynicism.
