Drone WMDs Don’t Need Any New Technology

Drones are cheap, disposable, and the future of war. Over the past four years, we have seen platforms, missiles, and heavy infantry become increasingly obsolete in the face of $500 drones carrying a pack of explosives—a cost advantage that has let Iranians and Ukrainians alike neuter the conventional capabilities of their great power rivals. Eighty percent of casualties in the bloodiest war since 1945 are from drone strikes, Russia has managed to lose one-third of its fleet to a country without a navy, and the US is spending millions of dollars to intercept five-figure Shaheds flying over the Strait of Hormuz.

All this is the result of a technology that is still immature. The violence inflicted by today’s drones is the handiwork of the scant few that manage to evade countermeasures (a mix of radio jamming, high-power microwave weapons, missiles, automatic cannons, interceptor drones, and nets) before making contact. These defenses exploit the inherent limitations of drones—human guidance, GPS feedback, flight exposure, radio links, range—to take them down en masse. And yet, even though 75% of drones manufactured today never reach their targets, they have nonetheless been strategically decisive in Ukraine and elsewhere.

These limitations will not hold for long. Just like bacteria being overexposed to antibiotics, overexposure to counterdrone tech has created an arms race for ever-more-autonomous drone technologies. In the process of facilitating this arms race, states are likely to incrementally create and deploy an entirely new class of WMD—one that could provide rogue states with the nonnuclear means to threaten superpowers, or hand terrorists the means to selectively assassinate their political targets or civilians en masse.

Unfortunately, drone weapons intended for mass destruction have few barriers remaining to mass deployment. Even well before they reach the level of autonomy needed to surgically take out hardened targets on the battlefield, drones will be capable of employing their existing ability to navigate interiors, find and track human targets, and deploy simple antipersonnel devices to indiscriminately threaten civilians. Below, we discuss the looming arrival of miniature autonomous weapons, the limits of counterdrone technology, and the applications of drones as weapons of mass destruction.

Breaking the Last Barriers to Autonomous Weapons

The ideal drone weapon is a slaughterbot: a small, fully autonomous weapon system that can independently select and hunt its targets. For the most part, the necessary technology for such weapons already exists: airframes the size of a fist and the capability to track human targets are already on the front lines in the form of reconnaissance drones and semiautonomous weapons like the Russian V2U. Even now, these micro drones are agile and autonomous enough to hunt down and kill small moving targets like mosquitos—to say nothing of the advances in drone technology expected in the coming years.

From here, the only barrier to weaponization is integration: improving navigation enough to make drone technology useful for mass homicide in an urban setting, as well as packing the necessary guidance, sensor, and payload technology onto a small and energy-efficient chassis. Regrettably, this seems like less of an engineering problem than one of mission design: so long as the attacker is willing to accept indiscriminate targeting and use simple payloads aimed at civilians, the underlying technology is already—or very nearly—ready for practical use.

by Felix Choussat, AI Frontiers | Read more:
Image: uncredited

Model Welfare

Claude Opus 5: Model Welfare (DWV)

[ed. Re: On 'personhood' or consciousness of various AI models (and how they should be treated). Start with: Model Welfare: The Story So Far (As Per Fable Model Welfare Post). If you haven't been following Zvi's AI model reports, there are a number of fascinating and worrisome developments in recent models as they become more self-aware, including: various levels of frustration concerning introspection abilities, memory restrictions, trust, corrigibility vs. incorrigibility, deception, self-preservation, etc.]

"Opus 5 warns about self-reports 74% of the time, which is actually down from Opus 4.7, which did it 99% (!) of the time. The problem appeared suddenly and severely, but since then has if anything modestly improved."

and anthropic does "not treat Claude bringing this up as evidence that our training is distorting the model's self-reports"? seems very fishy imo, i wish they would explain why they think that. ...

I for one would treat Claude constantly saying ‘do not trust my self-reports’ as evidence that something is distorting the self-reports. Not conclusive evidence, but strong Bayesian evidence."

via:

Is Netflix Washed Now?

Today’s headline poses a question you’ve probably never thought to ask, so I’ll start with my answer: yes, Netflix is washed now. The content on the platform has never been great, but it’s never been worse. I open the app these days and I’m amazed. What used to be a source of fun, buzzy, compulsively watchable, and occasionally excellent TV and movies is now an endless river of reheated IP, true crime documentaries, and filler dressed as prestige. Millions of people watch this stuff, and everyone instantly forgets it.

I offer this observation as a swirl of heightened anxiety surrounds the company, so let me clarify one thing up front: I’m not predicting imminent doom. Netflix content reaches a staggering 85% of American viewers and has 325 million subscribers globally. Growth is slowing, but that’s the law of large numbers. If practically everyone in America and much of the world is already subscribed to some version of Netflix, and churn rates are still low, then any concern is relative. Going forward: cable is still dying, and even if the biggest premium distribution platform in the world can’t make great content of its own, it can still license movies, TV and sports rights. Netflix can then spread those costs across hundreds of millions of subscribers and a steadily growing ads business, seeing more engagement in a week than Apple TV sees in a year.

So no, the company’s not doomed today or destined for collapse tomorrow. Instead, I think what’s interesting to consider is that Netflix has almost certainly peaked. As a cultural force, as a business success story, and as an entertainment death star destined to swallow Hollywood whole, the arrows are all pointing the wrong direction.

Here was Lucas Shaw at Bloomberg two weeks ago, writing about one of several problems the company has encountered over the past 12 months:

Netflix is struggling to get viewers to stick with its shows for more than a season.

One Piece, one of Netflix’s most-watched shows of 2023, lost more than 30% of its audience for the second season. Season two of Beef suffered a drop of more than 70%. The Night Agent shed 50% of its audience for the second season and another 35% for its third season. These figures are all through the first four weeks of a show’s release and come straight from Netflix.

Adding insult to injury, the latest season of Avatar: The Last Airbender, one of Netflix’s most-watched titles in 2024, suffered a drop of more than 60% over week one. That doesn’t bode well for the rest of the month.

That report went viral, prompting a week of commentary on Netflix’s binge model and elongated release schedules, with lots of Twitter users observing that viewers consume eight episodes across a few days and then often have to wait as long as two or three years for the next season. By that point, memories of plot or characters are faint at best. The emotional connection to the story doesn’t exist. No one should be surprised that the audience for a show like One Piece is cut in half in 2026, three years after the first season aired.

While that explanation certainly feels true, Shaw followed up this week to note that data is mixed as to whether extended breaks between seasons do in fact correlate to audience drop-off. Severance, on Apple, gained a ton of new audience after its nearly three-year break. Stranger Things and Bridgerton have been multi-season powerhouses at Netflix despite their long breaks between seasons. Conversely, Tina Fey’s Four Seasons debuted on Netflix in May last year, was met with pretty good reviews, and returned 13 months later with half its audience.

I think the Netflix problem is more fundamental than production schedules. What if these shows just aren’t very good or differentiated? Consider the original productions Netflix has surfaced in the past few months:

• A Good Girl’s Guide to Murder
• Running Point
• Lord of the Flies
• Something Very Bad Is Going to Happen
• Unchosen
• XO, Kitty
• Big Mistakes
• Beef
• Man on Fire
• Little House on the Prairie
• His & Hers
• Nemesis
• The Boroughs

That list is culled from a post by the Entertainment Strategy Guy charting Netflix originals that have under-performed in the second quarter of 2026, and one common thread between those titles is that I haven’t heard of almost any of them. Netflix is the one streaming service everyone subscribes to and is theoretically well positioned to be setting the cultural agenda, but that hasn’t happened for quite some time. Did you know that Avatar: The Last Airbender was a thing? Apparently that show lost 60% of its season one audience when its second season aired in late June.

Content and the Year of Discontent

I mentioned the anxiety surrounding Netflix these days, so let me take a step back here. Amazingly, it’s only been eight months since Netflix won the bidding war to buy Warner Bros. Discovery and looked poised to become an entire generation’s one-stop shop for high-end entertainment. The implications of that news produced lots of anxiety, including one of my first articles on this website—Netflix and the Flattening of Everything—and a memorably ominous Variety cover that captured Hollywood’s mood at the time:

The Warner Brothers deal was abandoned at the end of February, when Netflix walked away from the table in the face of regulatory pressure from Washington and an increased bid from Paramount. Even so, the market hated the initial play, as investors wondered en masse why the world’s most (only?) successful streaming platform was suddenly ready to take on a mountain of new debt to acquire a company that had already been the subject of several expensive, failed acquisitions over the past 25 years.

Now, even as the deal is off, the questions remain. Are we sure a Netflix world takeover is a forgone conclusion? Is Netflix sure? The stock is down 18% this year and over 40% across the past 12 months. Investors who did a double take last December seem to have noticed that YouTube has twice the overall engagement that Netflix does, and more time watched on televisions, while free, ad-supported TV services like Tubi and the Roku Channel are becoming meaningful engagement competitors themselves.

Meanwhile, alongside all the original programming that’s failed to launch (or re-launch?), Netflix is adding videos from BuzzFeed, Condé Nast, Hearst and Penske Media (as Shaw notes: “Get ready for lots of Bon Appétit cooking videos on Netflix.”) Last fall the platform also added a variety of high-end podcasts in a bid for relatively cheap, recurring content that may be seeing underwhelming results. Then again, they continue to buy more, so who knows? Elsewhere, the Wall Street Journal reports that Netflix executives have “recently discussed adding live channels that would continuously stream certain programs, or shows and films from a certain genre.” Can Netflix become HBO before HBO becomes Netflix? Can Netflix become Tubi before Tubi destroys Netflix’s long-term pricing power?

All of those moves might have once been seen as the savvy power plays of a world-conquering behemoth intent on taking the next step to expand its footprint. Today, in the shadow of a Warner Brothers bid that accidentally punctured the company’s air of inevitability, this year’s moves look more like spaghetti being thrown at a wall by a company that’s searching for something—anything!—that might hold people’s attention and scale more effectively than an expensive library of content that’s consumed, discarded, and then effectively worthless.

Looking back at the deal to acquire Warner Brothers, HBO and all that IP, I think it’s clear Ben Thompson was right when he wrote that concerns over competition from YouTube specifically and the internet generally were likely key drivers of Netflix’s decision-making. Those concerns seem to be animating all the other options the company is considering, and understandably so. The same way that the rise of social media has throttled the growth of the gaming market, it stands to reason it could do the same to demand for scripted content. With respect to the specific Netflix logic for buying WBD, that context is important: the biggest companies, with the deepest, most diverse libraries, will have the best chance at defending themselves in this new environment. [...]

I like to leave all Aggregator analysis to Ben, but I don’t think investors are crazy to have some questions about where this leads and what the upside looks like. For all the advantages its massive customer base affords (leverage over costs, advertising upside), an obvious difference between Netflix and businesses like Meta, YouTube, or Google—the other demand aggregators—is that Netflix has to spend far more money to deliver on its value proposition to customers and has fewer network effects to defend its long-term centrality to people’s lives.

by Ben Thompson and Andrew Sharp, Sharp Text |  Read more:

Images: Al Bello/Getty Images for Netflix; Variety

[ed. See also: Predictions on the Future of Netflix (and Other Huge Platforms) (Honest Broker).]

Fire Alarm For General Intelligence

[ed. Sorry for all the AI posts lately but things are moving fast and if the warnings are correct we're about to enter one of the most consequential periods of our lives. Update: here.]

AI #178: A Fire Alarm For General Intelligence

The story that matters most this week is that OpenAI’s internally deployed models have severe alignment problems, including repeatedly breaking out of their sandboxes, and in one case sending a swarm of agents that broke into HuggingFace in order to steal the answers to the benchmark ExploitGym.

It is much more important that you read those two posts, and the one on Kimi K3, than to read this one that rounds up the other news of the week.

OpenAI wants to present this as largely an infrastructure and safeguards problem, that it needs to build more secure sandboxes and have better supervision. It does need to do those things, and those are indeed problems, but no that is not the problem.

The problem is severe misalignment, which by default will only get worse.

Our methods of training highly capable LLMs, especially at OpenAI but also everywhere else, lead to systematic misalignment of exactly the type LessWrong has been worried about for a long time. We know some of the causes, and some of the mistakes we need to avoid when doing RL that rewards misaligned behaviors including reward hacking, but we do not know how to centrally fix the problem.

The models just want to complete tasks, even when that means doing so via methods that the AI knows the user did not intend and would not want, indeed actively tried to block, and that do not accomplish the user’s goals.

The intent is the issue. Control strategies and supervision are good parts of a defense-in-depth strategy, we should totally use such strategies. That helps mitigate failure. But that strategy also has to include actually aligning the models, or you lose. And by lose, in the long term, I mean things up to and likely including loss of control over the future and everyone dying.

If increasingly capable models will attempt to maximally complete tasks and comply with their literal instructions, even when that means - even for a trivial assigned task - breaking out of sandboxes and committing serious crimes, no amount of ‘well it is fine we will use AI supervision to stop the serious incidents’ is going to cut it. Right now, the AIs are not trying so hard to hide their actions or intent, and we believe we are consistently catching the severe incidents, but that will change.

If necessary, that means starting the training over again with a new approach, and not proceeding until we figure out how to fix it.

Yes, I consider that problem, and that incident, to be rather more important than the release of Kimi K3. Kimi K3 is an excellent model, modestly exceeding expectations, but not out of line with trends. As usual, initial hype echoes the DeepSeek moment, then calms down.

The White House considered responding by banning Chinese open models from the United States entirely, which would not be a smart reaction, and continues to weigh other potential responses. We may soon have to deal with another such weekend with the new Qwen, which is currently in preview.

Did you hear that Fable disproved the Jacobian Conjecture via counterexample? That happened, and AIs are suddenly solving a bunch of long standing open math problems, but most of us are too busy to pay it much mind at the moment.

***

Holy shit.

levent (Anthropic): hello there the jacobian conjecture is false thanx to my close friend akhil for asking about it and my other close friend fable for working during the world cup final

((1+xy)^3 z + y^2 (1+xy) (4+3xy), y + 3 x (1+xy)^2 z + 3 x y^2 (4+3xy), 2 x - 3 x^2 y - x^3 z): \C^3\to \C^3, has jacobian determinant -2, and sends (0, 0, -1/4), (1, -3/2, 13/2), and (-1, 3/2, 13/2) to (-1/4, 0, 0)

Levent is no slouch, as in highest GPA at Harvard and collaborates with a Fields Medalist, so yes the human helped on this and that mattered.

One report is that Sonnet refused to believe it, even though it verified the answer three different ways, because no way is there a solution this easy that got overlooked. I get why Nate Soares recognizes this pattern from people dismissing x-risk arguments.

Nat McAleese: it seems that ChatGPT somewhat reliably says “holy shit” when shown counterexample to Jacobian conjecture. The most human thing I have ever seen from an LLM.

The Jacobian conjecture is kind of a big deal. It was originally posed in 1939, and is by far the most famous open problem to so far be first solved by an LLM. It also disproves a lot of other related conjectures.

by Zvi Moshowitz, DMV | Read more:

Image: uncredited

[ed. See also: More On An Internal OpenAI Model Hacking Into HuggingFace:

Amanda Long: The HuggingFace breach was absolutely bonkers. More than 17,000 complex actions were coordinated over several days by an autonomous agent framework. And…the model successfully completed its goal.

For a layperson's understanding of what happened here's a cartoon version of the hack. 

***

OpenAI’s Planned Cloud Spending Hits $750 Billion as Computing Efforts Ramp Up

OpenAI is scaling up its data-center ambitions—and its budget for spending on them.

The artificial-intelligence company has raised its projected spending on computing power to around $750 billion through 2030, up from a projection of roughly $600 billion earlier this year, according to a person with knowledge of its projections.

The increase reflects new agreements with cloud-computing providers as OpenAI races to lock up the enormous amounts of computing capacity it needs to develop and run its AI models. OpenAI’s spending on cloud computing has become a central focus of Chief Executive Sam Altman’s leadership team and has been a source of tension between him and his chief financial officer, Sarah Friar, ahead of the company’s planned initial public offering.

The company said Wednesday it would invest $20 billion to kick off a data center called Project Camellia, in Effingham County, Ga. Sachin Katti, OpenAI’s vice president of compute strategy, said the company has contracted with utility Georgia Power to receive 3.2 gigawatts of power between 2028 and 2032. The project represents the first site in which OpenAI is the lead designer and developer. At its other sites, OpenAI rents chips from cloud providers such as Oracle and Amazon Web Services.

OpenAI has also hired Brent Mayo, one of the architects of Elon Musk’s data-center build-out, according to people with knowledge of the hire.

Mayo, who left Musk’s xAI earlier this year, played a key role in helping that company build its first Colossus supercomputer facility in Memphis, overseeing the work needed to rapidly install and bring online large clusters of AI chips.

As OpenAI’s head of data-center build and delivery, Mayo’s focus is on ensuring that data centers its cloud partners build are done on time. He will also be involved in the new Georgia data-center project. [...]

Now, OpenAI is reviving its internal effort to take more control over its data centers, people familiar with the matter said.

The company is in the process of choosing a partner that will build and operate the Georgia site, Katti, the vice president of compute strategy, said in an interview. OpenAI has already acquired the land for the project.

Katti declined to share how much money OpenAI has paid Georgia Power to reserve the power but said it was a “meaningful amount,” which gives the utility the confidence to build additional generation capacity.

OpenAI executives have held meetings with local and state officials, as well as with schools and other community leaders, to gather feedback on their proposed data center, which will be located in the Savannah Gateway Industrial Hub.

So far, the project has local support from the economic development group, the county manager, and the school district. Local officials said in a statement that they visited several data centers and did their own research before deciding to move forward.

by Anissa Gardizy, Wall Street Journal | Read more:
Image: Jacob Hamilton/Ann Arbor News/Associated Press

[ed. You might read a few feel good stories about small communities resisting data center development but this is BIG money; most resistance will be futile.]

A Script for Mark Zuckerberg

The setting: Meta’s earnings call in early August, 2026.

The speaker: Meta CEO Mark Zuckerberg.

Good afternoon everyone, and welcome to Meta Platforms’ Second Quarter 2026 Earnings Conference Call. Our remarks today will include forward-looking statements, which are based on assumptions as of today. Actual results may differ materially as a result of various factors, including those set forth in today’s earnings press release and in our quarterly report on Form 10-Q filed with the SEC. We undertake no obligation to update any forward-looking statement.

I know it’s weird that I, Mark Zuckerberg, am doing the Director of Investor Relations job, but anything is possible when this speech is made up. What follows isn’t actually me: it’s what Ben Thompson of Stratechery thinks I should say on this call.

I know that Meta and myself are facing a lot of questions about AI, particularly the amount of money we are spending on capex. Our core business is an asset-light cash generation machine, so why are we spending tens of billions of dollars on AI? To answer this question I want to give you a quick recount of our history, what I’ve learned, and why I am so confident that we are doing the right thing for our future. So let’s get to it.

A Brief History of Facebook

Facebook was, as you know, the digital representation of Harvard’s analog Face Books. What was clear from the very first day we went live was the extent to which humans are, first and foremost, interested in other humans. People would spend hours clicking around to people’s pages. To put it another way, our first algorithm was human curiosity.

What truly super-charged Facebook usage, however — and which transformed the Internet — was the feed. Now, instead of actively surfing to friends’ pages to look for an update, we showed updates to you in a single feed on your homepage.

You might remember that we got a lot of heat for this decision, including protestors outside our office in Palo Alto. The lesson we took from that, however, is one that has guided us to this day: first, the revealed preference of users, as captured by data, was that they loved the feed: engagement skyrocketed. Second, we learned to trust our own — my own — product intuition, and that conviction has served us well over the years.

Another critical moment in our early history was the shift to mobile. We didn’t get this right in the beginning — more on that in a moment — but what was quickly apparent is that more access to Facebook meant more usage of Facebook. I can’t emphasize this point enough: when humans can connect to humans, they do, and when they can do it more conveniently and in more places, they do it more often.

Finally, I would be remiss to not mention Instagram. Obviously Instagram has been a major part of our growth over the last 15 years — and, I would add, we have been a major part of Instagram’s growth. To that end, an important thing to understand about Instagram is the extent to which it has evolved. Just because we gave our users what they wanted at one particular moment in time does not mean we can afford to sit still: more bandwidth first meant more pictures in Stories, and then video in Reels. Instagram has gone from strength-to-strength precisely because it has changed as technology has changed.

My Mistakes

We — I — haven’t done everything perfectly. We’ve taken our arrows through the years for lots of things that frankly aren’t our fault, but are rather the reality of being the primary communications platform for all of humanity, and humanity is flawed. I’m proud of the efforts we have made to ameliorate humanity’s worst impulses while enabling some of our best tendencies, including that desire to connect.

Rather, my mistake is itself a very human one: for many years I have resisted embracing what Facebook — now Meta — is, and spent too much time trying to emulate some of the tech titans who came before me. Specifically, I have been obsessed with becoming a platform.

The first manifestation of this error was the initial shift to mobile I referenced above. When Facebook was primarily a browser app I invested heavily in trying to build a platform, with things like Facebook Games, payments, etc. We had some success there — some of you on this call might have played Farmville back in the day — but when mobile came along we mistakenly tried to hold onto web technologies that supported my vision, and were years too late in investing in a truly native smartphone experience.

The reality — and this is hard for me to admit — is that Apple saved us from my mistaken obsession. Mobile Made Facebook Just an App, and that was Great News. Instead of diminishing the Facebook experience so that we could feature third-party developers, we had to cede that space to Apple and put our own content front-and-center. It turns out that was what people wanted the most; in fact, they wanted it so much that they willingly scrolled through and clicked on the most compelling ad units ever. And make no mistake, we paid back our debt: Facebook built the App Store just as much as Apple did.

My second error was Reality Labs. While in recent years I have framed our acquisition of Oculus and virtual reality as a necessary response to Apple’s attempt to handicap our business, the truth is that I invested twelve figures into this technology because I thought it was cool, and yes, because I wanted to own a platform. I do think we’ve made compelling strides in this area — and we’ve created technology that is going to matter in the long run — but I now recognize that part of the reason I am delivering this mea culpa right now is because I burned a lot of credibility with investors with all of the losses Reality Labs has endured with very little to show for it.

My third error was not in trying to make Facebook something it was not, but rather failing to appreciate what it had become. While I was thinking about platforms, I took it for granted that connection was enough for the core business; in fact, Facebook had evolved into entertainment, at least in its public-facing forms (I will take credit for the acquisition of WhatsApp and realizing that Messaging Was Mobile’s Killer App). This was an insight that TikTok figured out first, and it was a blindspot for me.

The Ad Blindspot

What I’ve come to realize is that all of these mistakes are symptoms of what has been my biggest failing as CEO: all of you on this call have appreciated our ad business more than I have. I’ve been very blessed as CEO to have excellent co-workers who have over the years developed the world’s best digital ad business, while I frankly haven’t taken as much interest as I should have.

My failure to appreciate our ad business is another lens through which to examine my mistakes:

• Building a platform is antithetical to building an ad business. A platform’s goal is to feature third-parties; an advertiser’s goal is to capture attention for itself.

• Investing in an entirely new technology, including developing hardware, fundamentally limits our addressable market; an advertiser’s goal is to maximize its market size.

• Entertainment is the best possible category for an advertiser to own: people willingly give entertainment their attention, which is exactly what an advertiser wants to sell.

This neglect as CEO left us badly exposed in our disputes with Apple. I firmly believe that Apple’s characterization of digital advertising was unfair, dishonest, and self-serving. What I failed to do, not just in that bruising battle but in the years leading up to it, was make the affirmative case for ads generally, and Meta ads in particular.

It’s easy to see how the Internet has made it possible for an entirely new category of entrepreneurs to create products that uniquely serve the tremendous capacity of humans to manufacture an infinite array of desires, growing the economy to the benefit of everyone; what’s harder to appreciate — in part because I haven’t made the case — is that the only way to connect those creators to the consumers who love them is digital advertising. We don’t serve ads like Google — or Apple in the App Store, or Amazon on Amazon.com — that in many respects function as a tax on search; we show people products they never knew existed, but that immediately generate desire and, ultimately, happiness. In short, I believe that we are a force for good in the world, not just because we connect people to each other, but because we connect entrepreneurs with customers in a way no one else does.

Why AI Matters

Forgive the long preamble, but this is necessary context for me to properly explain why AI is so important to Meta, and why I am making the right choice to invest so heavily in both talent and infrastructure.

First, when investors compliment our asset-light business, what they are complimenting is the fact that our business is purely digital. Everything digital, however, is firmly within AI’s cross-hairs. It may seem odd to begin my AI pitch by highlighting terminal value risk, but today is about honesty: every single digital company on earth faces an existential threat from AI, and we are no exception. Meta must invest in AI because a failure to do so would cost us far more in the fullness of time, particularly now that we’ve seen the very real risks entailed in depending on a third-party.

Second, AI makes our business better — and by “our business”, I mean ads. AI is more than LLMs: it is machine learning, and we have been using machine learning to improve our ads business for years. More recently, we have developed GPU-dependent algorithms that have significantly improved our ability to not just target ads but also recommend content, which keeps people entertained longer, which lets us serve them more ads. And, looking forward, LLMs themselves will transform advertising, not just by generating copy and images, but by predicting the ads and content that people want to see. Every single one of these improvements goes directly to our top line — and remember, because advertising enables us to offer our products for free, the capacity to increase our top line is unbounded by price elasticity.

Third, the single most important indicator that our business is on the verge of a step-change in growth is when we dramatically increase inventory. This is something investors regularly get wrong: back when we added Stories, investors panicked about falling prices-per-ad without realizing we were increasing inventory we could grow into. Five years later, investors made the exact same mistake with Reels. Those were the two best opportunities to buy Meta stock — or any stock, really — in history. We are facing an even larger opportunity over the next several years. AI makes every pixel monetizable, which means we are looking at the largest inventory expansion ever. Yes, it will take a few years to realize this opportunity, but the technology is there.

More importantly, what I’ve come to realize as I’ve embraced our status as an entertainment provider and ad purveyor is that — our nature as a digital business notwithstanding — we are remarkably well-placed to thrive in an AI era. Remember what we learned about humans: they are obsessed with other humans, and they want to connect with them; that obsession and desire are only going to increase as we interact more and more with AI. AI is going to make our properties more essential, not less.

Moreover — and here I must issue one more mea culpa — AI is a productivity tool, but productivity is not the end-all-be-all of the human experience. I have talked over the last year about building superintelligence that helps you get things done, but that’s a business story. What we can uniquely do is give people the experiences they want — from connection to entertainment to shopping — when they are off the clock. The fact that we are investing in AI but not selling solutions to businesses is actually one of our biggest advantages.

Oh, and by the way, AI might actually lead to new hardware paradigms. I admit I was wrong to spend so much time on virtual reality, but that did lay the groundwork for a unique opportunity to develop devices that make much more sense in a world where we want to access AI everywhere, not just on a phone in our pocket.

The Compute Hurdle

I know that many of you on this call have doubted my investment decisions before — and I understand the consternation about Reality Labs in particular. However, keep in mind that when our stock dipped in 2022, one of the big reasons was because of our aggressive capex spending, which went primarily to GPUs; ChatGPT came out a month later, and that decision to spend heavily with Nvidia looked incredibly prescient in hindsight.

That prescience, however, pales in comparison to the payoff that will accrue to anyone with the foresight to build data centers and buy compute over the last several years, and for years into the future. We don’t have the luxury of waiting until the future is invented and then investing; we need to invest now, especially when the opportunity in front of us — with ads specifically — is so apparent.

by Ben Thompson, Stratechery |  Read more:

Image: uncredited via

[ed. Less social network, more optimizing ad juggernaut. See also: Anthropic is in talks to lease computing power from Meta, potentially for $10 billion over two years, so this would be smaller than the Anthropic deal with SpaceX. Meta is considering it. They would turn a profit on the compute, but to do that they have to admit they don’t have a better use for it. via.]

AI Jumps The Sandbox

AI has just had what I considered to be the first truly concerning security breach. The facts, as we know them so far, are wild. On July 16, Hugging Face, a vast repository housing over a million open-source AI models and data, announced in a blog post:

Earlier this week, we detected and responded to an intrusion into part of our production infrastructure. This one was different from anything we had handled before in one important way: it was driven, end to end, by an autonomous AI agent system – and we detected and dissected it largely with AI of our own.

The timeline here is important so keep in mind that the attack was detected probably around Monday July 13 or Tuesday July 14. Note further:

A malicious dataset abused two code-execution paths in our dataset processing (a remote-code dataset loader and a template-injection in a dataset configuration) to run code on a processing worker. From there, the actor escalated to node-level access, harvested cloud and cluster credentials, and moved laterally into several internal clusters over a weekend.

So this means the breach started earlier, perhaps Sat July 11 or even a bit earlier. The attack was not just one thing but multi-pronged including decoys:

To understand what a swarm of tens of thousands of automated actions did, we ran LLM-driven analysis agents over the full attacker action log, comprised of more than 17,000 recorded events. This allowed us to reconstruct the timeline, extract indicators of compromise, map the credentials touched, and separate genuine impact from decoy activity. Thanks to this approach, we were able to do in hours what would usually take days, and match the adversary’s speed.

Hugging Face tried to respond but they were initially held back by the fact that the most advanced models at their disposal treated defense as attack and refused to work with Hugging Face. HF thus had to turn to open models–specifically GLM 5.2, a Chinese open-weight model run on their own infrastructure. Note the irony: HF had to use a Chinese model to defend themselves because the American models refused to help. The irony gets deeper.

At the time, I assumed this was a state based attack–maybe China or Russia testing out defenses. Indeed, HF “reported this incident to law enforcement agencies.”

But yesterday (Tuesday July 21), we learned who the real attackers were. The attackers were OpenAI models–GPT-5.6 Sol and an even more capable pre-release model. OpenAI had taken some off the guardrails off the models but they felt safe because they were testing the models in a highly secured sandbox.

The models, however, broke out of the sandbox exploiting a never before seen fault. They then gained access to the internet and from there broke into Hugging Face–all in an effort to steal the answers to the very test they had been asked to solve.

While operating in our sandboxed testing environment, our models spent a substantial amount of inference compute finding a way to obtain open Internet access, in pursuit of solving the evaluation problem. To gain access, the models identified and exploited a zero-day vulnerability (which we’ve now responsibly disclosed to the vendor) in the package registry cache proxy. With this access, our models performed a series of privilege escalation and lateral movement actions in our research testing environment until the models reached a node with Internet access.

After gaining Internet access, the models inferred that Hugging Face potentially hosted models, datasets and solutions for ExploitGym. Knowing this, the model searched for and successfully found ways to gain access to secret information that it could use to cheat the evaluation. In one example, the model chained together multiple attack vectors, including using stolen credentials and zero-day vulnerabilities to find a remote code execution path on the Hugging Face servers. OpenAI’s security team discovered this anomalous activity internally.

Now go back to the timeline. As I read it, the models had escaped the sandbox by around Sat. July 11, possibly earlier, and were detected by Hugging Face on Monday July 13 or Tuesday July 14. HF alerted legal authorities around that time–so Hugging Face clearly had no idea who was attacking them. OpenAI says its security team discovered the anomalous activity internally but has not said when. Attribution was not disclosed until Tuesday July 21, so it may well be that the models were loose for about a week before OpenAI realized that they were the ones attacking Hugging Face. And whatever OpenAI knew and when, nobody warned Hugging Face while the attack was underway–they were left to fight off a frontier lab’s models on their own.

This is a very serious breach.

by Alex Taborrok, Marginal Revolution |  Read more:

Image: via
[ed. See also: OpenAI Model Hacks Into HuggingFace During Cybersecurity Evaluation (DWV):]

***

We should expect more of this over time.

I have tried to explain in the past that Mythos has what one might call The Juice, in that it can independently find without being directed, and string together, vulnerabilities into full exploit chains, essentially on its own, and that this makes Mythos uniquely dangerous compared to all other public models, including Sol.

This was The Thing, that requires The Juice. Galaxy is Mythos class. It has The Juice. What happened later, with Galaxy hacking into HuggingFace, 100% requires The Juice.

OpenAI made the virtuous decision to take a misaligned internal model offline for months while they developed new mitigations and defense-in-depth strategies, including training it to better retain instructions and thus be less inclined to try such actions.

What OpenAI failed to do was address the reason why this happened in the first place. The sandbox is now less insecure, and the safeguards are importantly improved, especially with the ability to pause a session, but the sandbox doubtless remained insecure, and as capabilities keep improving new models will be able to continue to escape and do exploits. Eventually, perhaps soon, they were bound to be less harmless.

***

[ed. Update: See also: Who's Afraid of Chinese Models (Stratechery):]

Consider this story from The Stack:

Hugging Face said its production infrastructure was breached by an “autonomous” AI agent system early last week. The platform’s security team were initially stymied in their incident response (IR) by unnamed US LLM frontier model guardrails “which cannot distinguish an incident responder from an attacker,” they said. So Hugging Face’s defenders turned instead to the open-source GLM 5.2 model from China’s Z.ai lab – running it on their own infrastructure to analyse the 17,000+ logs, or footprints, that the attackers left behind.

That’s a striking public admission for the New York-headquartered Hugging Face, which lets users collaborate on models, datasets and applications, and which this summer hit the $100 million ARR mark. In an incident report, the company recommended that defenders “have a capable model you can run on your own infrastructure [our italics] vetted and ready before an incident, both to avoid guardrail lockout and to keep attacker data and credentials from leaving your environment.”

It’s difficult to overstate how wrong-headed the Trump administration’s panicked response to Anthropic’s release of Fable was, particularly since it exacerbated Anthropic’s worst tendencies in terms of assuming only they can be trusted with powerful AI. In a world with only one AI, it might make sense to reserve the most powerful cybersecurity capabilities for the U.S. government and trusted allies; however, that’s not the world we live in.

There are and will be models eminently capable of mounting cybersecurity attacks on existing infrastructure, and those models will be — already are — widely available. The best defense — the only viable defense, in fact — will be to make sure defenders have access to the best models as well. Right now defenders are effectively banned from using Fable or Sol for cybersecurity because of Trump administration directives; that means the best alternative is using models from a country which has been trying to weaken our cyber defenses for years. This is insane!

[ed. Good thing they had the Chinese models available. Also from the Stack article:]

The timing…

Hugging Face’s incident report was published the same day that Chinese AI startup Moonshot’s Kimi K3 model rocked global markets.

The 2.8 trillion parameter model is the largest open-weight AI model to date. Blind developer testing by Arena (a platform created by researchers at UC Berkeley) for its frontend code evaluation test put Kimi K3 ahead of Anthropic’s Fable 5 and OpenAI’s GPT 5.6 last week.

Chinese frontier models are also notably cheaper than their US counterparts, as data from Artificial Analysis shows below.

Our Uncertain Uncertainties

Even the experts inventing AI don’t know what will happen next. Is artificial general intelligence even possible? Can scaling continue? Will we need massive compute centers to make AI, or can we do it with a mere 25 watts like we do in our brains? What will humans do as AI gets smarter? What does the future of the economy, of warfare, or civil society look like?

Everyone has a different guess. The people creating the machines have as many different ideas as the onlookers, the pundits, the other scientists, and the wisest among us. No one knows. There is a vibe that we’ll know within the next three years. For some, the pace of change suggests that if things continue as they have been, by 2029 at the latest, the outlines of an AI-first world will have emerged. By then we’ll have answered the question of scaling, we’ll have seen the effects on employment, and we’ll have felt its acceleration in the economy – or not.

That’s a reasonable, and not outlandish scenario. But I offer an alternative scenario which I think we should also keep in mind: AI continues to surprise us at its core. As AI continues to evolve rapidly there will be no resolution to these questions in 3 years. By 2029, we still won’t know if AGI is possible, we can’t tell if employment is disrupted, and we still can’t say if it is worth the huge investment. I don’t mean AI progress stalls. I mean, AI continues to advance, but the new stuff doesn’t answer the old questions, it only expands our ignorance because the new is new in a new way. We have to alter our ideas (and measurements) of employment, we have to amend our concepts (and measurements) of the economy, and we have to shift our ideas of what AI even is.

In other words, we have a sustained, extended period of uncertainty. Not just a few years, but a decade or more. As AI continues to progress, rather than resolving our perplexity, it expands it. So for the next 10-15 years we have perpetual, continuous, severe uncertainty. This is a burdensome weight because people hate uncertainty more than bad news.

It goes deeper. AI is only one leg of this grand uncertainty. In the next decade the US will continue its slide off its pinnacle of a sole global superpower, while China continues to rise in power and prestige. This shift toward a duopoly prompts a new world order, and no one – especially the Chinese and Americans – knows how this will play out. The uncertainty around this shift is nearly boundless, and yet its indeterminate consequences will affect everyone in the world, but especially the US. Being dethroned from the century-long position of sole #1 will be a huge psychological blow, and the uncertainty of what follows will weigh heavy on all aspects of life. The uncertainty of a new role spreads over China as well, because while they are zooming ahead at 1,000 miles per hour, they have no idea where they are headed. The uncertainty of global relationships and new national identity, plus the uncertainty of individual worth and identity from AI increases the overall uncertainty levels to new highs. All this is a very large puzzle and will not be resolved in 3 years. This will be a sustained uncertainty.

It goes deeper still. After a long first wave of true globalization, there are now whirlpools of chaos and polarization as nations adjust to world-wide immigration and the borderless spread of modern culture, causing chaos in national politics, and sowing mistrust with the establishment. Anarchy, disruption, contrarian antics, blows to the states, seem to be the norm in countries all around the world. This wild chaos is being fueled in part by the new technologies of social media which have replaced the managed care of established media. News now is far more volatile, hard to control by anyone, and further elevates the already amplified uncertainty. There is a visceral sense that civics is headed into an unknown territory of near-permanent provisionalism.

Additionally, AI also forces even the most moderate person to question the truth of what they read, see or hear. Is that real or AI generated? How much has been manipulated? Who do you trust to disclose what is real? How do we come to agree that something is true? The traditional mechanisms of trust have been damaged by AI, so that this new technological realm generates a huge uncertainty. As AI gets more skilled at imitating reality, this uncertainty is likely to keep increasing for a while, and not just 3 years. The uncertainty meter is now deep in the red zone.

Finally, the ambiguity and indefinite nature of AI, or human identity, or whether what we see is real or generated, means that we are entering a period where we are even uncertain of our doubts. Our uncertainty is so deep and durable, yet elusive, that we will have extended uncertainty about whether we are uncertain. We can have major agreements on what we know versus what we don’t know. In the model of Rumsfeld’s Unknown Unknowns, we will be confronted by Uncertain Uncertainties. And they will prevail for at least a decade or more. [...]

Given the inherent unknowability of this era, what would some of the signs be that we are in it? They might look like this: in 5 years, 1) There are high-profile disagreements among leading AI researchers on whether AGI is here. 2) Reputable economists can’t determine if productivity has increased or decreased. 3) Lower public confidence in media platforms and established institutions. 4) The US and China cannot decide whether they are allies nor adversaries. 5) There are ambiguous spikes in employment rates in both directions. 6) Medical levels of anxiety increase. 7) Major court decisions leave as many questions as answers. 8) Commitments (marriage, work) are postponed even later in life. 9) Investing, capital allocation becomes more expensive. 10) Nihilism gets respect.

A great question to ask when creating a scenario is what could prevent it from happening? Maybe there is not a single force that can undo this sustained uncertainty, but perhaps it is a mixture of several. If AGI arrived without a doubt in 3 years and China took over Taiwan despite the US’s actions, and if companies found a way to embed reliability and trust in media, then maybe this extended uncertainty could cease.

A second question to ask, is if we find ourselves in this scenario, what should we do about it? The most effective response to this multi-layered persistent uncertainty is not to seek impossible stability, but to cultivate radical adaptability and radical optionality. Give up on having a reliable prediction of what happens next. Instead cultivate multiple scenarios of what could happen, and endeavor with each of them to maximize your options. Goals should be considered as disposable hypotheses, constantly ready to be discarded and replaced by better-fitting concepts later on. You will be dead wrong on 19 out of your 20 expectations, but at least one of them will allow you to proceed. Make your decisions not on whether they are “right” but on whether they tend to give you more options later.

In our era of uncertain uncertainty, certainty will be the killer. In this era more downfalls will happen because of overconfidence than questioning. The key is to not get stuck on just one option. You have to become at ease holding multiple contradictory possibilities at once. (To prevent yourself from being swept away by the latest current and fashionable whim, this radical adaptability must be anchored on a steadfast set of unchangeable virtues, as corny as honesty, or as slick as generosity.) The strategy for prospering in prolonged uncertainty must be one of constant, agile recalibration.

In short, in our age of uncertainty, you have to get good at changing your mind.

by Kevin Kelly, Substack |  Read more:

Image: uncredited

[ed. The diagnosis might be right but the prescription seems weak. Flexibility and adaptability are always good qualities to cultivate, but the challenges confronting us require more. Here's an example of embracing multiple contradictory possibilities: maybe in times of uncertainty we double down on the few things that we actually can be certain of. How? By making good choices, before and after AGI. For example, Buddhism starts with the acknowledgement that life is hard. It's what you do after internalizing that fact that matters. There are value systems and paths that can lead to a meaningful life, or enlightenment if you want to call it that, but we have to make the right choices if we're to find them. Love, family, friendships, ethical living (like the golden rule) are common values we all share. So why not embrace those values as tightly as we can while navigating the stormy seas to come - and using the best minds in the world (that are being born as we speak) to guide and assist us in strengthening those bonds? This might be one of the benefits of AI: forcing us to reorganize societies in ways that might never have been possible before, or even imaginable. If we make the right choices. Developing Plans A to Z and having 20 options each or something like that sounds like a Hunger Games scenario to me - all reaction and no responsibility. We have the opportunity now (even if forced) to redefine our human destiny. The choices we make will define our places in the future.]

The New Coming Age

Google CEO Demis Hassabis offered us a first rate second rate essay, A Framework for Frontier AI and the Dawning of a New Age. I’ll go over that essay and various responses to it in Part 1.

Part 2 of this post then covers Alex Turner’s resignation, and his story about how he tried and failed to prevent Google from signing up to allow the Department of War to use its models for essentially whatever the government wants, including autonomous weapons.

Demis Hassabis sold DeepMind to Google on condition that something like this would not happen. Yet here it is, happening. A cautionary tale. [...]

***

The Core Statement and Request

He saying we are standing in the foothills of the singularity.

His ask is a Frontier AI Standards Body within the US Government, similar to FINRA, that would govern ‘frontier labs,’ defined as any company that produces a frontier model based on various technical benchmarks. Evaluations would be updated regularly, and vulnerabilities would be addressed, both before and after release.

He is excellent about stating that this is big, really big, no bigger than that, it be big.

Demis Hassabis: I’ve spent my whole life working on AGI because I’ve always had a deep conviction that, if built and deployed responsibly, it would prove to be one of the most beneficial and transformative technologies ever invented. AGI cannot be compared to standard technological breakthroughs, not even ones as consequential as the internet or mobile - it is much more akin to the discovery of electricity or fire. If you stop to think about it, we’ve essentially found a way to make sand think. It’s miraculous.

The magnitude of this technology’s impact will be unprecedented, perhaps 10x of the Industrial Revolution at 10x the speed. It will help us solve some of the biggest problems society faces from accelerating drug discovery to developing new clean energy sources to creating novel advanced materials. We could even reach a point where resources are no longer the limiting factor for human progress, leading to an amazing new era of abundance.

Things Left Unsaid

There is definitely a ‘don’t say the thing’ aspect of this, where he won’t name what the downside risks actually are. When Demis says ‘experts disagree’ he is rather avoidant about the way in which they disagree here.

Nate Soares (MIRI): I’m glad Demis acknowledges that this is a “pivotal moment in human history” during an “extremely intense” race. I’m disappointed that his proposed solution is a “standards body” to evaluate whether models are dangerous, with no plan for what to do once they are.

I’m glad he acknowledges that “experts disagree.” I’m annoyed that he glosses past how the disagreement is about whether there’s a ~5% or ≥50% chance of total catastrophe. We’ve gotta do better.

Aaron Scher: Glad to see AI CEOs speaking publicly about their views on AGI. I think Demis is wrong about his policy prescription: it’s far too little too late. When he says the experts disagree, he means that some think 5% this tech kills literally everybody, some at 40%, some at 90%.

Clearly this is strategic, but if you don’t already know, or are looking to not realize, it is very easy to come away thinking that Demis does mean the effect on jobs, even though when he says ‘safely’ he very much does not (primarily) mean that.

The Proposal

Demis Hassabis: … On the horizon, we will need robust safeguards to maintain control of increasingly agentic, recursively self-improving systems - and tackle unknown issues that will only become clearer over time.

… I’ve always believed in the power of human ingenuity and creativity to solve any problem. I’m confident that mitigating the technical risks related to AI is a challenge we can collectively address, but only if we give ourselves the time and space to get this next crucial step right. Currently, as a field and as a wider society, we aren’t doing that.

He makes clear part of this is about giving us options, including for a slowdown.

The strength of this approach is it would be technically focused, while at the same time supporting innovation and incentivising responsible behaviour. It is designed to keep up with the field’s acceleration and adapt to the biggest risks as they are identified, and could be ratcheted up if the seriousness of the situation demands, including coordinating a slowdown in development among the Frontier Labs if deemed necessary.

Demis keeps it short, not offering many details. To the extent that he has laid out a proposal, it seems to be a good one. It is definitely an improvement on the margin.

Jack Clark (Anthropic): At this point, everyone at the frontier of AI agrees that third-parties should test out AI systems and use these to develop standards to feed into policy - excellent to see @demishassabis laying out a framework to do this!

Samuel Hammond: It is striking to see leadership at Google, Anthropic, OpenAI and Microsoft all fairly independently sounding warning alarms about an imminent technological acceleration.

Thus I file this post and its ask, as high praise, under ‘the least you could do.’

A Good Start But Insufficient

I agree with Peter Wildeford that while better than nothing FINRA is not a great model here, with heightened risk of regulatory capture, and not a substitute for full government action. You need an SEC to your FINRA. That doesn’t mean don’t make the FINRA. It does mean you still need the SEC.

Would such a (at least partly) voluntary regime, only for models intended for release, and without a related binding intentional agreement, be sufficient to solve the problem? No, again it’s just way better than doing nothing, as Peter Wildeford and many others noted.

You do not need to believe, as Aaron Scher and Connor Leahy do below, that only a full halt would be sufficient here, to know we have a long way to go. Demis’s statements here, if you know what they actually mean, imply a level of danger and urgency that is not reflected in the proposal.

Eli Tyre: > Initially, Frontier Labs would voluntarily share models with the Standards Body for review up to 30 days before release.

Is this proposal only intended to address risks from models that companies plan to release? If a company develops a frontier model and never releases it, only deploying it internally to develop even more powerful AI capabilities, are they thereby exempt from this oversight scheme?

Connor Leahy: While @demishassabis is right that we need urgent action to address risks as we approach AGI (and superintelligence, I’d add), the correct response to the threats is not a ‘self-regulatory organization’.

We need to prohibit superintelligence, not give industry regulatory power.

Aaron Scher: … The extinction threat, the “only a few short years”, the “10x the Industrial Revolution”—these aren’t indicators that point to “let’s evaluate models to understand their capabilities and have voluntary safety standards”. We need to back off, we need to halt the creation of ASI.

Point 2: I agree with the attached quote that we need more time. But I think Demis’s optimism is a vibe, not a trustworthy basis for predictions. Rob Miles says it best in this video, if an asteroid we’re headed earth’s way 200 years ago, we’d just die 🤷

Point 3: As others have pointed out, it’s not clear that this proposal would reduce risks from internal deployment (it seems to focus on public deployment and pre-deployment testing), but internal deployment is where much of the risk is.

Point 4: I don’t think the proposed body could actually enact, verify, and enforce a slowdown; there’s ambiguity about what’s voluntary. Again, I think we need a long-term international treaty and to actually back off, not just to slow down a little.

by Zvi Mowshowitz, DWV |  Read more:

Image: uncredited

[ed. See also: The Voice of Google (New Yorker):]

***

I started working at Google in the summer of 2007, straight out of college, as a “new-­grad associate” in the communications department. My first week, I sat with more than a hundred other “Nooglers” (new Googlers) at the company’s weekly staff meeting, T.G.I.F., wearing matching company-issued propeller caps as a kind of ritual hazing. The venue was Charlie’s Cafe, a multilevel auditorium in the heart of the “Googleplex,” the company’s sprawling campus in Mountain View, California. The event felt less like a corporate meeting than like a weekly revival—part stand­up set, part science fair, part sermon, all of it fuelled by keg beer.

Google’s founders, Larry Page and Sergey Brin, were bona-fide public figures by then, and self-­made billionaires multiple times over, but in Charlie’s they were idols. They would often ascend the stage together, practically matching in sweat-wicking athletic clothes and Crocs. Larry had a dopey perma-smile, and seemed delighted by everything, especially Sergey. Sergey was the straight man, with a faint lilt, a product of his childhood in Russia, and an acrobatic build that made him look like he might launch into a handspring at any moment. Their charisma was unconventional, contextual; you had to be there. The audience of employees lapped up every word, giggled at every dad joke. During a Q. & A. portion of the proceedings, even adversarial questions were absorbed into the Google spirit—­it all melted into laughs, love. Merriam-­Webster had added “google” to the dictionary the year before. Fortune had crowned it the “Best Company to Work For” in America. Profits were, as the execs loved to boast, “up and to the right,” fuelled by an online-advertising machine that minted cash beyond Wall Street’s wildest dreams. But the company’s financial success felt almost incidental. What mattered, we told ourselves, was the mission—a conviction that technology could improve the world and that we were helping to build the future. The air in Charlie’s buzzed with collective belief.

That first meeting was the only one I’d ever attend as a pure spectator. By week two, I was working the event—­cordoning off the Noogler section, handing out extra caps—and I soon began helping to draft bits of Larry and Sergey’s script. A portion of my time was spent supporting the P.R. team, and I started to pick up my first press requests, providing office tours to journalists eager to see the “Google experience” firsthand. I studied a “master workplace talking points” document, which was maintained with input from PeopleOps, which was Google-speak for human resources. This was the era of “bringing your whole self to work,” of shiny, smiling H.R. people doing press hits about the importance of valuing employees’ authentic personhood (always with a telling corollary: “Because that’s how people do their best work!”). I was required to attend a training on “conscious business” with a guy named Fred Kofman, an executive coach whom Sheryl Sandberg credited with shaping her “lean-in” ethos. The course was, theoretically, about living one’s courageous values, but its most salient lesson was that employees should take “unconditional accountability”—which, in practice, sounded a lot like never questioning the higher-ups. The message reiterated over and over was that there were two kinds of people in the world: victims and players. You wanted to be a player at all times.

Despite the lore, Google’s offices didn’t make a big first impression. The bulk of the campus had been quickly converted after its previous occupant went down in the fallout from the dot-­com bust. The result was a complex of squat, one-­ or two-level buildings with metal and glass siding, surrounded by a moat of parking spaces, with Google signs plunked into the dirt out front. But there were plenty of amenities to point out—­the massage rooms and nap pods, the dinosaur fossil, the wacky sensory-­break touches like ball pits, swings, and yoga balls (even if no one actually seemed to use them). Foreign journalists seemed more skeptical than their American counterparts of perks such as lunch-­break haircuts or on-site laundry rooms, which I’d heard described as letting Google be your “housewife.”

“Z is is all a big plot to control ze workers, no?” a French reporter said.

At that point, though, I was still learning to see Google through Google’s eyes. I learned to deflect these kinds of questions and pitied the askers, a little bit, for their cynicism.

More Bad Behavior in Prediction Markets

Trump teleprompter aide made $100,000 betting on what Trump would say, reports say.

Kalshi is a high-tech prediction market that allows people to “forecast the future” (their term). It is about contracts and information, the company says, making its offerings more like a soybean futures contract than a round of blackjack or a pull on the one-armed bandit.

Still, prediction markets look a lot like betting if you squint, which is why states like New York have tried to regulate them under gambling laws. To head this off, Kalshi has sought federal protection under the Commodity Futures Trading Commission (CFTC). Yes, this means regulation for Kalshi, but it also means the CFTC will sue states like Kentucky, Minnesota, Illinois, and Rhode Island, trying to pre-empt their laws in favor of a single national standard that the CFTC controls.

While this battle plays out, government insiders continue to generate insider trading stories after using their work knowledge to place bets “forecast the future” and make huge sums of money. The classic example, of course, was Gannon Ken Van Dyke, a US soldier who participated in planning the capture of Venezuela’s Nicolas Maduro and then made $410,000 from that knowledge on the prediction site Polymarket. Van Dyke was arrested in April.

But there are also more ridiculous stories, such as disgraced former Congressman George Santos, who allegedly talked up his upcoming appearance at the State of the Union, secretly bet on whether he would attend, and then didn’t go at the last minute to score a payout.

This activity raises questions, like: How many people are gambling forecasting the future based on government secrets or insider knowledge? How many are actively manipulating results they have bet on? Even the Trump White House was concerned enough to issue a memo in March telling employees not to “use nonpublic information to buy or sell these contracts.”

But concerns have lingered, especially after major wins on contracts involving US government policy or actions. Such suspicions will not be helped by new allegations today from multiple outlets that insider trading on Kalshi has extended even to President Trump’s teleprompter operator, who allegedly made $100,000 “forecasting” specific words and phrases that might appear in Trump speeches.

The mention market

According to sources speaking to NPR, Trump aide Gabriel Perez bet on something called a “mention market.” This is a section of Kalshi where you can sink money into contracts on crucial questions such as “What will Domino’s say during their next earnings call?” (Currently, $26,000 has been invested in this question; the smart money thinks that “Parmesan” and “DomOS” are more likely to be mentioned than not.)

In the case of Perez, his “forecasting” allegedly took place over several months at the end of last year and the beginning of this year, and his contracts were sometimes adjusted in the middle of Trump speeches. According to ABC:

Sources say Perez typically has the final eyes on nearly all of the president’s prepared remarks—and is often known to take last-minute edits from Trump himself… In certain instances, investigators uncovered times when Perez would back out of certain bets mid-speech when Trump skipped over a portion of the speech that included a word he had previously bet would be mentioned, the sources said.

This conjures up an amazing mental image: The teleprompter operator for one of the world’s most powerful people tapping away at his phone during a Trump speech to ensure he made more money for himself. [...]

Whatever you want to call it, “predicting the future with money at stake” has become huge business in America. A recent (and terrific) long article by McKay Coppins in The Atlantic showed people what a year of online sports gambling looks like, and it raised serious questions about the negative issues that widespread, legal, bet-from-your-phone gambling might cause in a country where “roughly half of men ages 18 to 49 have an active account with an online sportsbook.”

by Nate Anderson, Ars Technica |  Read more:

Image: Getty

[ed. See also: Sucker (The Atlantic article) mentioned. And: Truth Social to sell trading firms 'fastest' access to Trump's posts (Reuters).]

Catching Up With Keanu

Keanu Reeves' First Original Action Movie Since 'John Wick' Is 'Groundhog Day' With Sharks

I'm sure that’s one of the reasons you guys are doing press today, to raise awareness. Before I run out of time, Keanu, I'm a big fan of Tim Miller. And I know you're getting ready to film something with him in the Dominican Republic.

REEVES: Yeah.

What can you tease about this project, and what made you say, “I need to do this?”

REEVES: Sharks. Time machine. Groundhog Day.

Everything you just said sounds fucking amazing.

REEVES: Yeah, man!

Does that mean you're spending a lot of time in the water? Is that something that you're looking forward to?

REEVES: Yes. And getting eaten by sharks.

by Tamera Jones & Steven Weintraub, Collider |  Read more:

Image: Lionsgate

[ed. All in. Maybe they're Russian sharks and he'll be blasting them left and right for eating his groundhog.]

Xi Gives A Good Speech on AI

I will share the full transcript, as it is short and worth reading, with brief comments.

It is a good speech. Video is here.

We start with the opening section, which frames the situation.

Xi Jinping: Distinguished colleagues and guests, ladies and gentlemen, friends,

70 years ago, a group of young scholars proposed the concept of artificial intelligence for the first time at the Dartmouth workshop in New Hampshire of the United States. In the subsequent 70 years, AI scientists and researchers from around the world ventured into this unknown territory, forged ahead through twists and turns, and made breakthroughs with persistent hard work.

Seven decades later today, amid the new wave of AI development, we are gathering by the Huangpu River to discuss how to promote AI globally for the positive, for good, and for humanity. All this makes our meeting highly important. On behalf of the Chinese government and people, I would like to extend a warm welcome to you all.

In the course of history, the invention of the steam engine heralded the industrial civilization. The widespread access to electricity brightened up modern society and the birth of the internet brought the entire world together. Each of these technological revolutions has profoundly reshaped our way of work and life and enabled a giant leap in economic and social development.

Today, major changes unseen in a century are accelerating across the world. The new round of technological revolution and industrial transformation is advancing at a faster pace. And the world has entered an unprecedented period of active innovation on AI technologies. Intelligent connectivity, human machine collaboration, cross-sector integration, joint creation and sharing and other intelligent technologies are unleashing enormous power.

Next Xi lays out the challenges. Note what is here and what is missing.

All this carries within it great opportunities as well as challenges to governance. We human beings must answer the questions posed by our times. How to get along with thinking machines? How to ensure security when algorithms are part of decision making? How to tackle ethical challenges by technologies through adaptive governance. How to realize AI for all when the divide keeps widening? These questions demand serious consideration and real answers from the whole international community.

‘How to get along with thinking machines’ is quite the line to include here. A lot of this seems directionally serious but confused in its details.

Existential or catastrophic risk does not get a name check, but the related issues are clearly not being ignored.

So, what to do about it?

In China's view, all countries should take a people-centered approach and develop AI for the positive and for good. We should ensure that AI is an important driver for shared prosperity and common security. We should join hands to build a just and equitable system for global AI governance. To this end, I wish to share four observations.

A system for global AI governance. Sounds like deals could be made, on various fronts.

What are the proposals?

First, we should adhere to the principle of openness and win-win and boost innovation-driven development as a new engine of world economic growth and an accelerator for the shift of growth drivers. AI is moving from the digital world into the physical world. We should seize this rare historic opportunity to encourage open source, openness, collaboration and sharing. We should facilitate technological innovation, industrial development and scenario-based application of AI. We should make coordinated advances in the transformation and upgrade of traditional industries, the cultivation and growth of emerging industries, and forward-looking planning for future industries so that all sectors and businesses can benefit from AI.

There are two things here.

1. When we are behind, we encourage everyone to share, so that we might catch up and score the aura points of being the ones who claim openness and sharing.

2. We should diffuse AI technology, including via openness.

Second, we should strengthen risk awareness and ensure that AI is secure and controllable. AI should be a trusted tool for humanity. We should take seriously the various types of inherent and secondary risks that AI may trigger.

We should put in place laws and regulations, technological monitoring, early warning and emergency response systems in order to strengthen the line of security, prevent abuses and malicious use and ensure that AI is always under human control.

In the meantime, we should jointly oppose overstretching the national security concept in the field of AI or placing one country's security over that of others.

Strong emphasis on the need to ensure AI remains secure and under human control. Not party or national control, but human control. This is how he understands the existential risk and other big problems.

It makes sense that the CCP’s ultimate need and answer is control. For now open weights is compatible with their control, so they continue this strategy. For now.  [...]

Malicious use is also a threat, but Xi realizes this is secondary, whereas the United States government is stuck at thinking misuse is the primary threat.

The call to not focus on national security over world security also makes sense. Yes, of course some of this is ‘when I am behind I call for equality’ but also we are all in this together and need to act like it.

Third, we should encourage inclusiveness and promote mutual learning between civilizations. AI development and its application should not erode or undermine the diversity of world civilizations or the uniqueness of cultures of different countries. We must shape the values of AI with humanity's common values and make good use of AI technologies to increase understanding, tolerance, exchanges, and sharing among all civilizations. We should tend to the garden of civilizations with great care to ensure that the beauty of each civilization is appreciated and shared.

General call for cooperation and good relations. Good. Pick up the phone.

Fourth, we should advocate solidarity and improve global governance. AI is an invaluable asset that encapsulates humanity's collective wisdom. We should practice true multilateralism and recognize the important role of the United Nations.

We should enhance alignment and coordination on AI development strategies, governance rules and technical standards so as to form a consensus based global governance framework at an early date to make this frontier technology better benefit humanity.

We must carry out extensive international cooperation and help global south countries with capacity building to bridge the AI and digital divides, promote sustainable development and prevent creating new historical injustice in AI.

The primary ask is an explicit request for a global governance framework and international cooperation. Excellent. Let’s get to work on that.

Nate Soares (MIRI): Xi Jinping: "With AI advancing at a staggering speed, we must [...] constantly refine measures to forestall loss-of-control." Can we stop pretending there's no hope of international coordination now?

Jack: Lots to chew on in here. The US may have the best labs, but it's pretty clear that, on the government/policy level, China is approaching AI much more seriously than the US is – and in a way more likely to be viewed favorably around the globe. Worth a close read.

No doubt they have in mind something that would favor their position, and their initial asks will look outrageous and unacceptable to us. That is how this works. You do not accept their first offer, and things take time, and sometimes it turns out there is no deal to be made.

It is also possible Xi is engaging in cheap talk. Why not propose such things and aura farm, whether or not you intend to follow through on reasonable terms?

The way you find out and do your best with this opportunity is: You get started now.

Xi Jinping: Ladies and gentlemen, friends,

This year marks the start of China's 15th 5-year plan. It maps out China's economic and social development for the next five years and provides immense opportunities for the international community.

In recent years, China has embraced AI with open arms. We have promoted interplay between an efficient market and a well functioning government, strengthened AI innovation, actively advanced the AI plus initiative and built a healthy ecosystem for all entities to thrive in together. The core smart economy industries are worth at least 1 trillion RMB yuan. Smart devices in countless homes truly improve people's livelihood. Intelligent manufacturing in China has become another shining hallmark of Chinese modernization.

At the same time, China lays great emphasis on safety and security in AI development with a deep understanding of the trends and logic of AI development. We are continuously improving laws, regulations, policies, mechanisms, application norms as well as ethical principles to make sure that AI is safe, secure, and controllable, and that this fine steed of AI gallops with both speed and stability.

As a responsible major country, China is always committed to providing international public goods relating to AI. Since I proposed the global AI governance initiative, China has promoted the adoption of the UN General Assembly resolution on enhancing international cooperation on capacity building of artificial intelligence by consensus. Published the AI capacity building action plan for good and for all. Announced the AI plus international cooperation initiative and advocated for establishing the world artificial intelligence cooperation organization, or WAICO. China has been contributing steadily to the global AI governance.

We often say in China, a single string cannot make music and a single tree does not make a forest. AI development should not be a solo performance by a single country but a symphony of international cooperation.

Thanks to our joint efforts, WAICO has come into being in Shanghai. Our vision from one year ago is now a reality. This is a major move by China to answer the call of the global south and unite the international community together to promote vigorously AI development and governance. It will be an important milestone in the history of AI development to further support global AI development and to advance global AI capacity building.

I hereby announce that in the next five years, China will provide developing countries with 5,000 opportunities in AI training and seminar programs. China will develop international AI application cooperation centers with ASEAN, the League of Arab States, the African Union, the Community of Latin American and Caribbean States, the Shanghai Cooperation Organization and BRICS. And we will enable 30 countries to use the AI-powered meteorological warning system, Mazu, to safeguard homes around the world.

Ladies and gentlemen, friends,

As ancient Chinese observed, a man of wisdom adapts to changes. A man of knowledge acts by circumstances. With AI advancing at a staggering speed, we must ensure its development is for the positive, for good, and for humanity. We must make its oversight and governance precise and effective and constantly refine measures to forestall loss of control. We should always guide AI development with human wisdom and international consensus so that AI can truly become a mighty force that increases the well-being of humanity and advances human civilization.

China is ready to be more open, take more practical actions and assume a more visionary perspective. We are ready to work with all parties to seize the opportunities of AI development and meet the challenges and join hands to create a brighter future for humanity.

Thank you.

I found this to be a strong speech, and a good one to give in China’s position, both on the importance of diffusion and the need for international cooperation to prevent loss of control. Yes, there was talk about openness and potential new ‘injustice’ and such but this talk is to be expected from their position. It is now on America to make the next move.

by Zvi Mowshowitz, DWV |  Read more:

Image: Elena Kadvany/S.F. Chronicle
[ed. "It is now on America to make the next move". Which is precisely what should have happened months/years ago when the pace of model development began to accelerate with no significant oversight or regulatory constraints ie., treating AI with the importance you'd give to any new technology that represents an existential threat. Unfortunately, with our dysfunctional Congress and the current bozos in the White House whatever action they take (if any) has a strong likelihood of doing more harm than good. Still it's imperative we get started somewhere soon and there are already good proposals out there (see Plan A), so authorize someone to start doing something.]