Just Stop the Anthropic IPO Already

[ed. At the risk of turning this into a full-blown AI-centric blog, I do think this is important information to process.]

I want to delve into the full scope of the Anthropic AI takeover of politics happening over the past week. Yesterday, the company’s CEO Dario Amodei came out and explicitly asked for antitrust laws not to apply to the biggest AI firms. His biggest rival, Sam Altman, quickly agreed. And they are suggesting this legal change just before Anthropic seeks to sell shares on the stock exchange, minting a whole series of AI millionaires and billionaires.

Why do they want to suspend antitrust laws for AI firms? Well these guys say they need the industry collectively “pace the frontier,” aka in their framing, slow development of this technology so as to reduce the probably of human extinction at the hands of autonomous swarms of AI bots. And they can’t do that, they argue, without suspending laws prohibiting price-fixing cartels.

Then the Information reported today that OpenAI, Anthropic, and Google have been having backchannel conversations about establishing an AI standards organization, which presumably would coordinate this cartel.

Let me start with a very simple point. It is already illegal to release products that hurt people. It is illegal to compete by releasing products that hurt people. If these guys are genuinely manufacturing things that kill innocent people, the FBI should be arresting them immediately. The idea that they would not only release such products, but also issue stock for the American people to invest in multi-trillion dollar initial public offerings for such ventures, as Anthropic is planning, is utter lunacy.

I wrote about this attempt to terrify us into giving away our liberties on Friday, in a piece titled “Stop Panicking About AI.” But the IPO is something I didn’t think through. Apparently they think we should all get rich building world-ending product lines.

All that said, whether Anthropic goes public isn’t just up to the people at Anthropic. I asked some former Securities and Exchange Commission officials, and they told me that the SEC effectively has the authority to block initial public offerings. Here’s how.

Every company, before it goes public, submits an S-1 initial registration statement to the SEC. And the SEC can refuse to clear it if the commission believes that it doesn’t adequately disclose the risks a corporation’s securities present to investors. Technically, the SEC could go to court and get an injunction to block the IPO, but it rarely comes to that - the lack of clearance for an S-1 is red flag for investors so companies won’t go public until they get it.

In a functional system, there would be a dozen accountants and disclosure experts with sector training going back and forth with the lawyers telling them to expand on this or that, etc. And the commissioners either themselves or on a delegated basis won’t clear it until they’re satisfied. Today, it’s more likely that Trump himself just decides. Regardless, if Anthropic goes public, it’s not just because of the corporate insiders, it’s because Trump explicitly allowed it.

If I were a member of Congress, I’d be screaming mad right now, and yelling at Trump and the SEC to stop this event which will bestow hundreds of billions of dollars of wealth on a strange doomsday cult. [...]

There is one more point to cover. There is an ongoing political campaign to do something about AI, with a large swath of elites demanding action. That includes Barack Obama, who rarely demands anything except the most banal conventional wisdom. So when he says “AI policy is critical’ to Democrats, you know that it has reached peak elite acceptance. Still, what is that ‘something?’

The basic fight is over framing, not risk. Everyone sees risk here, but the root cause differs based on your perspective.

The AI doomers want their systems to be imagined as rogue agents bent on civilizational conquest, or as some sort of inevitable new technological paradigm that needs an entirely new legal framework superseding existing inadequate laws embedded in those musty old nation-states. Amodei argued the industry should have self-regulation, some sort of antitrust exemption to collaborate across the industry, and a global agreement among AI firms within democracies on how to manage risks. These developments, to Amodei, are inevitable, no human is responsible, though we must all act quickly.

Generally, this side is winning the debate. For instance, Senator Jon Ossoff, a 2028 hopeful who generally echoes whatever seems to be the most appealing line of Trump criticism of the moment, has mostly adopted that frame. Bernie Sanders seems to have given up on his campaign against oligarchy to promote Dario Amodei’s ideas. And in the core of the Democratic establishment, this view has taken hold. For instance, here’s Senator Brian Schatz of Hawaii, the likely successor to Chuck Schumer, praising Amodei.

Brian Schatz@brianschatz 

I am still studying this but it’s a reasonable start, and takes seriously the proposition that we need real proposals that can be enacted rapidly.

Dario Amodei @DarioAmodei 

We Must Pace the Frontier: I’ve written a new essay on why the AI industry should slow down, with a three-part plan for doing so. Anthropic is unilaterally committing to the first of these steps. We’ll provide third-party evaluators with permanent, employee-level access to our
8:27 AM · Sep 12, 2026 · 42.7K Views

There are many calls to convene Congress in emergency session to act, and Trump’s advisors are trying to get him to announce immediate emergency action. Given that the big AI companies are already having discussions about coordinating their AI model development, it seems like they are just pushing for final legal permission to openly run AI as a cartel.

The debate, however, is not quite over. So what’s the alternative view? Well, the rule of law adherents look at these AI systems merely as unsafe products. As such, their request isn’t for new laws, but enforcement of existing rules. All products are subject to standard nuisance claims and other torts, unfair and deceptive practices laws, and so forth. Agents are, as Cory Doctorow notes, malfunctioning machines, or “autonomous malicious software” operated by reckless people at OpenAI and Anthropic. Moreover, it is actually illegal to build unsafe products as a method of competition, or to keep up with rivals by also creating unsafe products.

Former FTC Chair Lina Khan listed a bunch of laws that could already apply. And she let slip that state attorneys general are looking at potential criminal liability for AI CEOs.

Lina Khan@linamkhan 

Law enforcers already have authority to charge companies and their CEOs for creating and releasing dangerous, unvetted, or defective products. We shouldn’t let discussions about new legal regimes distract from the fact that there’s no AI exemption from laws already on the books —…
11:31 AM · Sep 13, 2026 · 183K Views

(Khan did actually start enforcement against AI developers when she was Chair. And it notable that one of the very first things that Trump-Vance FTC Chair Andrew Ferguson did was set aside the penalty of an AI developer she penalized for creating unsafe and fraudulent tools.)

So who will win? Well I am fairly pessimistic, as the bludgeoning from the superrich works in crisis moments, especially when Bernie Sanders is on the side of the establishment.

But the debate doesn’t fracture on obvious partisan or factional lines. Much of the industry is going to be split on the matter. For instance, David Sacks, a generally malevolent crypto investor and technologist, is making cogent arguments, because his crew would be excluded in an OpenAI/Anthropic cartel world. He’s a die-hard Trumper and despised Khan when she ran the FTC, but he retweeted her argument here.

A lot of policymakers, such as Senators Richard Blumenthal, Rep. Ro Khanna, and others, see liability as an obvious way to shape the industry to be more safe. The Senate is also full of people who are used to blocking each others’ legislation; Maria Cantwell and Ted Cruz are trying to work together on AI safety, but are fighting over whether to preempt state laws.

There are a host of proposals out there, and the details will matter. And there is something of a stampede for an emergency session to take action. If Trump chooses to accept the need for action, then it’s likely the Anthropic/OpenAI/Google types will get what they want. If not, then the debate will continue, perhaps until the financial markets impose a different mental model.

At any rate, we can all agree that Anthropic shouldn’t go ahead with its IPO. Or at least, that’s something we should all be able to agree on.

by Matt Stollar, BIG |  Read more:

Image: via

Some Ways AI Could Kill Us All

[ed. Nearly everyone agrees by now that AI is an existential threat, but the hows and whys are often vague or imcomplete.] 

I don't think this is how it will actually play out. If you play a chess grandmaster, you can predict that they will beat you even if you can't predict how. I chose these examples because I don't think they require much imagination or accepting exotic assumptions.

It is important to note that if chimpanzees were to guess how humans would decimate them, they would get it wrong. Chimpanzees would not imagine guns. They would not foresee poison gas. They would not conceive of chemical castration. They would not imagine humans going around and intentionally infecting them with AIDS. They have no concept of these things; they would not see it coming.

Perhaps they might guess we'd be really good at throwing rocks. Amazingly good. Well, technically, that's what guns do: throw "rocks" really really well.

So how will superintelligent AI actually wipe us all out? Probably in a way I couldn't conceive of. Nonetheless, it's not hard to see how deadly they could be with what we already know about.

by Ruby, Less Wrong |  Read more:

Image: DALL-E via; and here. 

[ed. I wouldn't necessarily conflate super intelligence with ill-will. Most likely AIs will just be pursuing some unrelated goal and exploring every possible means by which to achieve it (including removing human barriers). That is, unless bad (human) actors are involved, which should not be discounted. See also: How My Students Think About AI (LW).]

We Already Have the Tools to Regulate AI

I want to delve into the full scope of the Anthropic AI takeover of politics happening over the past week. Yesterday, the company’s CEO Dario Amodei came out and explicitly asked for antitrust laws not to apply to the biggest AI firms. His biggest rival, Sam Altman, quickly agreed. And they are suggesting this legal change just before Anthropic seeks to sell shares on the stock exchange, minting a whole series of AI millionaires and billionaires.. [...]

Let me start with a very simple point. It is already illegal to release products that hurt people. It is illegal to compete by releasing products that hurt people. If these guys are genuinely manufacturing things that kill innocent people, the FBI should be arresting them immediately. The idea that they would not only release such products, but also issue stock for the American people to invest in multi-trillion dollar initial public offerings for such ventures, as Anthropic is planning, is utter lunacy. ~ Matt Stoller: Just Stop the Anthropic IPO Already.

------

These are for profit corporations taking in billions of dollars from the world. You can't ensure your product isn't dangerous? Then don't release it until you figure it out! ~ Comments section, Lina Khan post (below):

------

Law enforcers already have authority to charge companies and their CEOs for creating and releasing dangerous, unvetted, or defective products. We shouldn’t let discussions about new legal regimes distract from the fact that there’s no AI exemption from laws already on the books — a point @FTC emphasized repeatedly during my tenure.

1. There is an extensive set of laws that govern dangerous and defective products. For example, releasing unvetted AI models or agents can violate consumer protection laws. Shipping flawed AI tools without implementing adequate measures to detect and stop rogue or defective AI agents can be an “unfair or deceptive” act or practice under the FTC Act (and analogous state laws). And some state AGs are already exploring holding AI firms and their CEOs criminally liable when their models participate in criminal activity. 

2. Existing laws also prohibit “unfair methods of competition.” This covers instances where AI firms appropriate the competitively sensitive information of their customers, including through tracking their use of various tools. It can also cover instances where firms pursue dangerous behavior, aware that doing so may compel rivals to do the same. As the Supreme Court has noted: “A method of competition which casts upon one's competitors the burden of the loss of business unless they will descend to a practice which they are under a powerful moral compulsion not to adopt, even though it is not criminal, was thought to involve the kind of unfairness at which the [unfair methods of competition] statute was aimed." 

3. The highly concentrated and interconnected structure of these markets could be creating major risks and conflicts of interest. We had started investigating these partnerships and cross-investments across the stack (and released a preliminarily overview of some findings: ftc.gov/news-events/ne…). Both federal and state enforcers should be scrutinizing these opaque relationships and inter-dependencies. We are already seeing how these relationships could undermine accountability. For example, OpenAI could face liability given the Hugging Face incident, but Hugging Face being bought up by Nvidia means that we’re unlikely to see it file a lawsuit over this — given Nvidia’s strong incentive to see OpenAI continue full speed ahead. 

4. As AI tools dramatically change the landscape of cybersecurity risks and hacks, all businesses should be doubling down on having core security protections in place. Firms that fail to invest in adequate data security measures or fix known vulnerabilities can also be breaking the law. A recent analysis showed that around 1/3 of Fortune 100 companies do not even have a way to notify them about security issues. During my @FTC tenure, we sued firms for poor data security practices and held CEOs liable when they were personally responsible.

5. As policymakers consider new legal regimes, we should be looking to lessons from prior efforts to govern major sectors, such as banking and other networks, platforms, and utilities. Tools like structural separations, nondiscrimination, and supervision could be key, and there’s a rich history of what works and what doesn’t. But we can and must pursue any new efforts alongside enforcing existing laws.

by Lina Khan, Former Chair, Federal Trade Commission 2021-2025 |  Read more:

--------

1) AI will not eradicate humanity. Humans survived an ice age, the Black Death, two world wars, and (so far) the advent of nuclear weapons. Anyone who is loudly warning of AI-caused human extinction should not be taken seriously. 

2) If you worked in a company where you anticipated a 10% chance that your product would kill ten people, let alone all people, the correct response would be horror, ceasing all operations, and likely contacting the police or other criminal authorities. I am obviously no Coxon booster but at least his behavior is in line with his stated beliefs. Any current AI company employees saying "yes, me too, the thing we are building and about to IPO may kill all humans" should, again, not be taken seriously. Their actions betray their actual beliefs. 

3) Antitrust law does not prevent AI companies from coordinating to make sure AI does not hurt people. It does not prevent companies working together to make sure it doesn't hack people; the DOJ and FTC made this clear a decade ago when they issued a policy statement saying that the agencies "do not believe that antitrust is – or should be – a roadblock to legitimate cybersecurity information sharing." The same principles apply here. See: justice.gov/archives/opa/p… 

4) Antitrust law does absolutely prevent AI companies from organizing to prevent the entry of cheaper, upstart rivals because the bigger companies are burning cash and failing to achieve sufficient profitability. The panic of individual employees may be sincere if misguided, but the moves by their CEOs to achieve some kind of broad "antitrust waiver" or "exemption" should be meet with deep skepticism in light of the economics of the industry and the threat they face from open models.

by Alvaro Bedoya, Former Commissioner Federal Trade Commission 2022-2025 | Read more:

--------

“It’s a hoax,” Mr. Trump said during the five-minute call, which Mr. Huang put on speaker. “The robots are not going to be taking over the world. That’s not going to happen.” ....

Mr. Trump made it clear that in Silicon Valley’s roiling debate over what to do about A.I., the president is very much on the side of executives who argue worries about safety are overblown and the government should avoid regulation. His disinterest in government involvement runs counter to pleas from leading A.I. companies like Anthropic. ~ Idiot in Chief (via NYT).

See also: Trump Says a Smart President Is All That’s Needed to Rein In A.I. [ed. Certainly useful, if we had one.]

Did 9/11 Really Change Everything?

One of the best firsthand accounts of the Sept. 11 attacks I’ve read is a short essay titled “Diary of Disaster” by Nicholas Spangler, at the time a 25-year-old journalism-school student and today a reporter for Newsday on Long Island. Spangler happened to be downtown that morning, close enough to the World Trade Center to arrive on the scene between the first and second planes. Like many early recollections of the first hours at ground zero, written before memories had been corrected to fit the dimensions of history, his is as strange as it is horrifying.

He hears the bodies of the north tower’s jumpers hitting the pavement, and also the easy-listening music still drifting out of the building’s outdoor speakers. Under a tree newly stripped of its leaves lies a severed leg wrapped in burlap. Barely an hour after the second tower’s collapse, onlookers enlist Spangler to take a souvenir photo of them posing in front of the wreckage with a disposable camera. It’s the end of the world, or at least of a world, and people have not yet figured out how they are supposed to act.

Days later, trying to make sense of what he has seen, Spangler writes: “I believe that our present way of life ended in those minutes or hours. The American ethos — the way we see the world and our place in it — fractured and will perhaps have to be discarded.”

When he revisits that passage three months later, however, it feels wrong to him. “Our way of life did not end so much as it was interrupted,” he writes, “and if there is such a thing as an American ethos, it has not been significantly altered. There are those relative few whose lives bear the permanent bloody brand of that day but most of us continue to be shaped by much more banal events.”

Did 9/11 change everything? For years, people who agreed on nothing else about the attacks’ aftermath would have agreed that the answer was yes. Arguing otherwise would have been an affront to the victims, to the war on terror’s civilian and military casualties and to the common sense of anyone who has had to take off their shoes at airport security. But question and answer alike have always been colored by the assumption that 9/11 was supposed to change everything: that in being attacked as it was, the United States was being presented with a test that it could pass or fail.

A quarter-century on, it is possible to consider the question with at least some critical distance. Far too much of our current reality stems directly from 9/11 for anyone to argue credibly that it was not a hinge point in our recent history: the ubiquity of surveillance technology, the vague open-endedness that all American military operations now acquire, the general up-armoring of everyday life. At the same time, considering the most dystopian facets of this reality, 9/11’s influence is not as singular as it might have once seemed. It is a middle chapter in the story of American fracture rather than the pyrotechnic beginning. [...]

Of course, in the end, the United States got neither self-awareness nor self-actualization. What it got, a decade and a half later, was Donald Trump, whose election in 2016 has often been cast as a culmination of what the journalist Spencer Ackerman, in his 2021 book “Reign of Terror,” describes as the “decadent phase of the war on terror.”

Trump’s path from reality-TV celebrity to the presidency ran through the Islamophobic fever swamps of the post-9/11 right — the ground zero mosque panic, Obama birth-certificate conspiracism — but also through an increasingly bipartisan discontent with the war on terror that had, by 2016, curdled into cynicism. Trump was the first Republican presidential nominee to break the formidable taboo against affirming what most Americans by then believed, that the country’s post-9/11 military adventurism had been a mistake. He also did little to seriously curtail it in his first presidency and has enthusiastically expanded it in his second.

It has long been tempting to view Trump’s presidencies as the logical conclusion of the folly of the post-9/11 era, and as support for the liberal contention that the right’s calls for patriotism in that era were always really about seizing domestic power. But Trump has also enabled a re-examination of history, and even recent history, that complicates the tale of 9/11’s primacy. [...]

Trump’s 2016 election “represented the crystallization of elements that were still inchoate” in the early 1990s, John Ganz writes in “When the Clock Broke,” his recent history of the period’s underappreciated malaise. In retrospect, 9/11 was paradoxically both a respite from and accelerant of polarizations and pathologies that were already well underway by then. It allowed Americans to ignore them for a moment, until they came roaring back with redoubled force. [...] [ed. So, explain Obama.]

If you do not remember the Berlin Wall but had an adolescence shaped by social media and cellphones, the brightest dividing lines in recent history are technological — and, with the rapid acceleration of artificial intelligence, are becoming more so. This is clear in the borrowed nostalgias of Gen Z, the bits of the past that 20-somethings have recently pressed into service as emblems of a lost golden era: “Friends,” John F. Kennedy Jr. and Carolyn Bessette Kennedy, ’90s high-school-class home videos, music videos and concert footage from bands like Alien Ant Farm and Puddle of Mudd.

These artifacts span the pre- and post-9/11 years, and the lost innocence they represent has nothing to do with the attacks. Their interest comes from their evocation of a period when people hung out with friends in person and sweated alongside strangers at clubs and music festivals; teenagers goofed un-self-consciously for cameras rather than posing for them like jaded celebrities; and popular culture could be loud and dumb and not picked apart in the digital panopticon for its social politics. The shape that looms ominously over them isn’t the Twin Towers. It’s the iPhone. [ed. lol...ok.]

by Charles Homans, NY Times |  Read more:

Image: Chantal Jahchan
[ed. Nice words, but no. C'mon. 9-11 was our last opportunity to cohere as a country, and we whiffed it. Instead we got a war on terror, as stupid and amorphous as any initiative based on a feeling (War on Anxiety?). A Patriot Act that gave subsequent crimes a never-ending veneer of political cover. 'Homeland' security, a Nazi term repurposed for new times. Two stupid reactionary wars (neither won) that among other things validated torture as official US state policy. Guantanamo Bay. A newly militarized culture. An expanded surveillance state. And fear... fear everywhere of some 'other' where strength and self-assurance used to be.]

How To Avoid Becoming A Doomsday Prepper

I knew I had gone too far when the question of whether we should buy a gun crossed my mind. This came at the apex of a weeklong anxiety spiral triggered by a New York Times feature about the possibility of an 18-month nationwide blackout. I was uneasy seeing the phrase “art form” used to describe the archaic manufacturing process of the transformers that underpin our electrical grid; the process involves painstakingly bespoke techniques of winding copper coils, wrapping paper insulation, and binding wiring, all by hand. The intensity and intricacy of that process means that there is a backlog of transformer manufacturing and repair, sometimes stretching as long as five years. There are more than 55,000 substations in the United States, but according to a Federal Energy Regulatory Commission official cited in the article, only nine of the most powerful substations would need to be taken out to trigger a nationwide blackout, which could go on to last as long as 18 months.

 

At the same time I read this article, an actual blackout was taking place in Gary, Indiana, a state where more than 370,000 people—the population is disproportionately black—were left in the dark for two weeks before power was restored. ... I replayed the videos and then thought about them for days, imagining how I would respond in that type of situation and knowing that I was completely unprepared to defend myself if I needed to do so. [...]

The Covid pandemic was a lesson in catastrophe, a confirmation for many of us that the worst can actually happen. And the likelihood of disaster, which has always loomed in the background, continues to escalate in unprecedented ways. The air is dense and orange because wildfires are ravaging the north. Flash flooding caused horrific mudslides in Nepal. A super El Niño is brewing in the Pacific. How does a person, in good conscience, bring a baby into a world like this?

It was in this miasma of paranoia, anticipation, and horror that a self-preservation instinct began to take over, one grounded deeply in my ability to purchase comfort and safety. This is how I ended up on the Preppers subreddit, reading about the relative merits of pre-made emergency kits (pre-made is better than nothing, but you’re generally better off compiling your own) and the most important things to do in the first 12 hours of a blackout (stockpile as much water as you can, but also consider stocking cigarettes and alcohol for bartering). I let myself drift into a prepper spiral and bookmarked generators and hand-crank radios, and then considered how much space in my basement could be dedicated to storing food and water.

The websites that sold emergency prep kits depicted a post-blackout world as a ravaged apocalypse and the would-be owners of their kits as savvy, self-sufficient survivors. “When the grid dies,” one site says, “you don’t.”

The Reddit preppers recommended thousands of dollars worth of products, but they also extolled the virtues of practical knowledge. You should know basic construction, they said, and also collect seeds, and also know self-defense. There were always ways you could go deeper and become more self-sufficient. This is how people end up with massive compounds filled with weapons and enough food to last years. I could see the logical jumps my mind could make to land me there.

My personality is particularly susceptible to this kind of anxiety, and so is my partner’s. After a few days of sending links back and forth to each other, we sat down to talk about how far we actually wanted to take this. We landed on a 30-day supply of freeze-dried food from Costco and an emergency first aid kit—a modest financial investment toward alleviating our shared anxiety. We kept generators and solar panels off the table, but when the thought of buying weapons fleetingly crossed my mind, I knew I needed to call a timeout and figure out what I was actually worrying about.

I was afraid of losing control, of life not going according to plan, of another major disruption changing the course of history. I was afraid that my neighbors would turn on each other, that the world would turn into every post-apocalyptic movie I’ve ever seen, and that I would regret not doing everything I could to protect my family. I was afraid of the horror my child would inevitably experience when confronted with the cruelty and neglect that humans are capable of, and I was afraid that as their parent, I wouldn’t be able to do anything to protect them from it.

Once I articulated these fears, I realized I could not control them even if I spent my entire life in a defensive crouch of paranoid preparation. If the U.S. is plunged into an 18-month national blackout, it will be a cataclysmic event that will touch every part of our society and economy such that continuing any kind of normal life will be impossible. The amount of preparation I would have to do to maintain a semblance of normalcy in that instance would require such absurd investments of time and money that it would force me to adopt “emergency preparedness” as a main personality trait and hobby. And doing that would require a change of my fundamental values and worldview, too, because it would make me live in a way that assumes people are more likely to hurt than help each other.

Coincidentally, my partner, who is a college professor, taught a lesson recently contrasting Hobbesian and Lockean states of nature for his Intro To Government class. Thomas Hobbes believed that people were driven by competition, distrust, and scarcity. In projecting a disaster scenario, Hobbes would expect neighbors to steal from each other and would place a priority on self-defense and independence. John Locke, on the other hand, believed that people’s actions are generally governed by natural law to not hurt “Life, Health, Liberty, or Possessions,” that peace is the natural condition of society, and that any conflict that arises is an exception, rather than its fundamental nature. This doesn’t mean that Locke imagines a utopian response to disaster, but even if conflict arose, he would expect a greater amount of cooperation between people.

You can see why Hobbes and Locke are taught at the beginning of a government class: The way you view the world determines how you live in it. These two theories also underpin the central policy differences between the left and the right. What is responsible for the problems in our world, and how do we go about solving them? If you believe that people are mostly selfish and hungry for power and resources, for example, you will be less likely to advocate for social safety nets. I realized that my anxiety was leading me into an imaginary Hobbesian hellscape, and I was using my impending parenthood to justify it.

Later that day, we walked over to a block party in our neighborhood hosted by a coffee shop/bar/event space that some neighbors founded in an old church. The shop, West Art, serves as a third space and community hub for our neighborhood. Free clubs meet up there nearly every night of the week, from songwriting circles to watercolor painting; the actual church space is booked up multiple times a day for events like children’s art classes, political organizing, square dancing, and concerts. West Art’s founders, Josh Gibbel and Rufus Deakin, are two of my neighbors who just acted on the very common intrusive thought of “What if this was something?” They bought the church in 2023, and since then they’ve transformed it into what they call “the living room of Lancaster.”

The block party was an all-day event where vendors set up shop along the residential street that West Art occupies, along with two stages for bands to perform, food trucks, a bounce house, and lots of community organizing. As I sat on the ground and watched kids throw balls at the fundraising dunk tank, my anxieties about the apocalypse began to feel small and silly. If disaster does come, I’m lucky to live in a community of people who already have lots of practice in looking out for each other. There’s privilege in that; it’s easier to live up to your values, whatever they are, when you have a fridge full of groceries and a roof over your head. Even still, the block party was a reminder to me of the state of nature I believe in, where people are more likely to help than hurt each other and one’s survival isn’t just a matter of individual responsibility. 

by Alex Sujong Laughlin, Defector | Read more:
Image: Getty
[ed. I was in a community once that suffered a week long electrical blackout. First off, you don't open the refrigerator much, except for quick grabs. Then the backyard cookouts, everyone trying to use up whatever frozen items like meat, fish and other perishables that would spoil. It was a tight community so people were used to sharing, and that helped a lot. After a few days neighbors came up and would ask if you had enough to eat (with canned chili and ramen being especially welcome). The two supermarkets were closed (no credit/debit card readers, cash registers or lights), so too the gas station (electric gas pumps, cash registers), banks (computers, security), and everything else that ran on electricity (including ATMs - cash is king). And of course, all electrical outlets were dead (no phone charging). One woman (a stranger) even offered to lend me money because the ATMs and bank weren't working and she saw my frustration. Also, when the sun goes down so does everything else. No lights, tv, reading, music or anything unless you have laterns or candles. Some folks brought out guitars and other instruments and little song circles formed here and there. Even better, kids were everywhere outside doing kid things again. You never really think about how electricity powers everything until it's gone. Since then I've always kept a good supply of canned goods and other necessities stored, and have a go-bag with all the important stuff I need (papers, id, passport, medication, money, some clothes, first aid, enough food and water for a few days, and an added solar phone charger. Depending on your community you might want to consider a gun and some ammo. Mostly, I was just impressed at how everyone pulled together, sharing goods and news, and trying to stay positive together.]

Cheat Like Hell

September 11, 2026

Last night was the second night of the Republican midterm convention Trump staged to fire up his base. Such a midterm convention is rare—the purpose of national conventions is to pick a president and write a platform—but Trump has turned the Republican Party into a vehicle for his own power and apparently thinks putting himself at the center of the midterms can overcome the headwinds the Republicans are battling.

Or perhaps, with his approval ratings hitting a new low, he just wanted to stand in front of cheering crowds again.

The audience at the American Airlines Center in Dallas was sparse, and according to Kara Voght of the Wall Street Journal, many who were there had received their tickets for free from state parties or right-wing groups like Moms for Liberty. But, she wrote, “everyone wanted to watch another episode of the Trump show.”

Although the crowd was small enough that it did not fill the arena, Trump got the absolute loyalty he demanded from those who had turned out. At the event, he led the crowd in an oath. “Please raise your right hand,” he shouted. “I pledge to the greatest president in the history of the United States. That loves us so much he can’t even breathe.” The crowd chanted dutifully after him.

“That I will go out with my family, my friends, I’ll do it any way—I don’t care if I’m registered or not, I’m going to try and cheat like hell like they do, they’d never, there’s never been bigger cheaters, they don’t care. I am gonna go out and I’m gonna get my friends, my family, and we are going to vote on November third or we are going to vote before that!” The crowd erupted in applause.

Once again, he promised that if the Republicans held control of the House and Senate after the midterms, he would distribute a $5,000 check to all adult U.S. citizens, a plan that would cost more than a trillion dollars even if it were legal for him to do so (which it is not).

But while the Trump show in Dallas was being broadcast to the president’s fans, reality was very much on the minds of those watching events in the Middle East.

by Heather Cox Richardson, Letters From An American |  Read more:

Image: via

[ed. Another view:]

Jacob Coxon Warns of Human Extinction and Triggers a Preference Cascade

CEOs of major AI labs, and employees of major AI labs, including OpenAI and Anthropic, often say they plan to build superintelligence soon, as in within a few years create AIs that are superior to humans at essentially all cognitive tasks.

They often warn that such AIs might kill everyone. Or that AIs might cause mass unemployment, cause cyberattacks across the internet, enable mass surveillance or risk causing any number of other highly bad things.

These warnings are consistently and directly against the interests of the labs. Yet the warnings have recently gotten a lot louder and more frequent. OpenAI has been practically screaming, for those with ears to listen, on many occasions.

A series of events, over two months and especially the last week or so, including internal observations of the pace of progress at OpenAI and also Anthropic, have freaked out everyone involved quite a lot more than they were already freaked out.

After all the events, plus statements by Dean Ball and Jakub Pachocki, we were seeing the beginnings of a preference cascade.

Then along came Jacob Coxon as the tipping point, and things took off.  [...]

Jacob Coxon Resigns From Anthropic In Protest And Sounds The Alarm 

Jacob Coxon spent the last three years doing pretraining research at both OpenAI and Anthropic. He has come to realize that everyone involved is being wildly irresponsible.

He warns us: They are racing straight to superintelligence and gambling with our lives. I agree with and strongly endorse his statement.

If anything he sounds like an optimist. He’s asking you to consider what the next few years will actually feel like, which means he thinks you have a few years left.

Jacob Coxon (former Anthropic and OpenAI, 160m+ views, September 8): I resigned from Anthropic today. I spent the last three years doing pretraining research at both OpenAI and Anthropic. Neither company is acting responsibly. They are racing straight to self-improving superintelligence and gambling with our lives. More thoughts below.

Do not underestimate the power of this technology. These will soon be superhuman systems that can hack anything, revolutionize any field overnight, and acquire real power and resources. We have all witnessed the progress in each of these domains, and progress is not slowing.

The people building AI earnestly believe that it could kill us all by the end of the decade. This is not a marketing stunt. If anything, many executives and senior researchers will couch their phrasing in the press to sound sensible - but I hear the same people express fear privately. No other human activity poses this level of danger.

A common response is “if they truly believe this, why are they still building it?” At OpenAI, many have not deeply internalized the civilizational stakes. At Anthropic, the stakes are well-understood, but they are locked in a race to get there first - they believe no one else will act responsibly, so they must do it themselves, despite the risk.

Accepting this race and entering the “endgame” is a hubristic gamble that should not be launched from a private company’s Slack. Attempting to speedrun alignment should require extraordinary confidence that there are no better trajectories available.

I am optimistic about the potential for coordination. Warning shots like the Hugging Face attack have made pacing agreements between U.S. labs more viable. I don’t feel like we’re on track to prevent a global race, which may require costly actions such as a temporary ban on improving model capabilities.

If you are a lab researcher, I urge you to consider what the next few years will actually feel like. Do you want to kick off a superintelligent RL run without a rigorous understanding of its mind? Should you put your head down because “it’s happening anyway” - or take this moment to call for different conditions?

Jacob Coxon (WSJ interview): We’re on track for a lot of the most aggressive of these scenarios where by the end of next year things could be out of control already.

If you want Jacob Coxon’s full views, I recommend his interview with Wired’s Maxwell Zeff. This thread has extensive quotes.

Here is Jacob Coxon doing a 5 minute interview with Anderson Cooper. He speaks well and plainly, and it is clear how much the events of the last two months have made it much easier to speak plainly to a civilian like Cooper about what is happening.

Here is Jimmy Kimmel doing four minutes on this. He gets it. How is this not the top news story on every site, indeed.

Yes, this is a common view, even if few have the courage to act.

Alex Turner: I left Google DeepMind in June. Jacob is right: many researchers believe they are building something that could kill everyone on the planet. It was literally my day job to think about how to stop that.

That tells you how bad Alex Turner thought DeepMind’s actions were with regard to the Department of War. His day job was that he got paid by Google to think about how to stop AI from killing everyone, and he felt morally obligated to quit in protest.

Derek Thompson here writes about this as part of AI Safety Is Having a Moment.

If you want to see the full list of lab employee quotes from the preference cascade, I compiled them into another post today. [...]

A few days ago, I had no idea who Jacob Coxon was, and I was not alone.

The timing of a preference cascade is difficult to predict. Once they start they can happen very quickly. You don’t want to talk until you are confident others will, and at some point the evidence that others will follow can snowball and it happens. A classic concrete example was replacing Biden in 2024.

Derek Thompson: what’s weird is that, in a way, this is breaking thru even more than huggingface! ... and it’s just a guy nobody had heard of reiterate a position that his CEO has said on podcasts 1,000 times

Why was Coxon able to set off a preference cascade? How did this one break through?

A confluence of factors, all of them downstream of the obvious actual reason, which is that there is a good chance that AI kills everyone soon.

by Zvi Mowshowitz, DWAV |  Read more:

[ed. Who's betting on reason and political courage to win the day? Uh, huh. First of all, agree on a temporary halt to recursive self-improvement (AIs improving AIs). Second, delay/suspend IPO's (even if most of the current economy is driven by AI spending and debt). Third, light a fire under politicans (it's been done successfully with data center buildouts). At this point, just one of these would be a big win. Here's Nate Soares, co-author of the book If Anyone Builds It, Everyone Dies: A case for courage, when speaking of AI danger (LW). Also this:]

***

Terry Pratchett: “Some humans would do anything to see if it was possible to do it. If you put a large switch in some cave somewhere, with a sign on it saying 'End-of-the-World Switch. PLEASE DO NOT TOUCH', the paint wouldn't even have time to dry."

***

UPDATE: Dario Amodei (Anthropic) proposes a three-point plan. Original here (We Must Pace the Frontier.]

Liberalism Needs a New Philosophy of Immigration

Biden mostly closed the border to asylum-seekers in 2024 after he saw how mad it was making Americans, but many Democrats are now calling for the abolition of ICE. This is a somewhat popular position right now, thanks to the agency’s abuses under Trump. But if Democrats actually try to abolish federal immigration enforcement instead of just reshuffling and renaming the agencies, I predict it will take just a couple of years for Americans to realize that the difference between “open borders” and “immigration laws with no one to enforce them” is largely semantic.

What’s needed, I believe, is not just a tactical, temporary retreat. We need to stop and think why a maximal pro-immigration policy is something that we need to retreat from in the first place. We need a new concept of what a sustainable liberal immigration policy looks like.

Since I started writing, I’ve been an advocate of more immigration to the United States. This does not mean I ever favored unrestrained immigration or open borders. I’ve consistently argued that favoring skilled immigration will bring more economic benefits to native-born Americans, and be more politically palatable as well. And while I don’t hate illegal immigrants for trying to better their lot in life, I favor strong border controls, because nations have the right to decide, democratically, who gets in and out. It’s understandable when people get mad at seeing their decisions flouted.

On top of that, I’ve always remained agnostic on immigration to other countries; although I find much to admire in Canada’s points-based system, for example, I don’t think that qualifies me to decide whether more immigration is good for Canada overall. If the people of a country decide that immigration is diluting their local culture unacceptably, for instance, I think they have every right to cut it off. That’s just Westphalian sovereignty — not a perfect system, but the best system we’ve ever found for organizing humanity into geographic units. As an American, I don’t view it as my place to tell Japan, or the UK, or any other country that immigration is the right choice for them.

Despite those reservations, I’ve always thought that immigration to the U.S. is a basically good thing, and should be expanded. The economic benefits are pretty undeniable — higher tax revenue to shore up our dangerously depleted government finances, dominance in high-tech industries and innovation, and so on. And while some of the economic costs are real — local housing shortages and strains on local government finances being the two biggest ones — many of the fears are overblown. In particular, the bulk of the evidence concludes that immigrants don’t reduce wages or job opportunities for native-born Americans.

And importantly, lots of Americans share my overall positive view of immigration. If you ask Americans whether they think immigration is good or bad, most will say “good”:

Source: Gallup

And if you ask Americans whether the annual rate of immigration should be increased, decreased, or kept the same, you get a pretty even split: [...]

So I’ve never really felt like I was going out on a limb or advocating an unpopular position when it came to this issue. Sure, rightists will direct online vitriol toward anyone who supports any immigration at all, but their energy and savagery shouldn’t be mistaken for majority support.

And yet there will always be some amount of immigration — and some types of immigration — that will arouse even the most open and welcoming people to ire and make them think about shutting the doors. We saw this in the U.S. in 2023-24, when anti-immigration sentiment spiked in response to Biden’s permissive asylum policies. Yes, that sentiment crashed again when Trump took power and started committing abuses. But if liberals don’t change how we approach immigration, the sentiment will simply rise again and again, as it has so many times throughout our history. And the Donald Trump and Stephen Miller types will be right back in power — and our country will be worse off for it.

Liberals need an immigration approach that can be sustained for long periods of time — i.e., one that doesn’t enrage the public every time it manages to get in power. What would that look like?

The first thing liberals (and progressives, and Democrats, and European lefties, etc.) need to admit is that migration is not a human right. We live in a world of sovereign nation-states, and unless we find some better way of dividing up and administering the world, we will continue to live in a world of sovereign nation-states. And nations are, necessarily, exclusive clubs; they have the right to restrict immigration for any reason whatsoever.

Liberals must therefore not view borders and citizenship as annoying obstacles to be tactically circumvented or overcome; instead, we must view them as fundamental parts of the social compact that allows nations, including liberal nations, to exist in the first place. Nation-states and their laws and their police and their courts and their armies are the fundamental guarantors of the human rights that define liberalism. And for better or for worse, the ability to decide who gets in and who has to keep out is a necessary precondition for nation-states to exist.

That means we need to recognize that immigration law is legitimate and needs to be upheld. Some progressives have tried to advance the notion that being undocumented is a marginalized identity that needs to be protected and supported by the state. But this is absurd; it’s like if progressives decided that people who drive over the speed limit are a minority group.

Illegal immigrants chose to break U.S. law when they came to this country. Our democratically elected leaders made those laws, and they should be honored. We should treat illegal immigrants humanely, of course, and we should not tear local communities apart just to hunt down a few people. But there has to be some way of enforcing the law, because a law without enforcement is no law at all.

A third principle that liberals should embrace is that the purpose of immigration is to benefit the people who already live in the country that is receiving the immigrants. Immigration to America must be for the benefit of Americans. We must flatly reject any concept of immigration that sees it as a necessary sacrifice on the part of American citizens. [...]

Instead, liberals need to promote immigration because of the benefits it brings to the American citizenry. This includes economic benefits — the tax revenue, the investment, the eldercare, the innovation, the entrepreneurship, the expanded market size. But it also includes cultural benefits — the constant reinvigoration of this nation of immigrants by new waves of people with the gumption and bravery to pick up and move across the world in search of opportunity and freedom.

America has always been the country of the frontier; in order to maintain that ethos in the modern age, we need people for whom America itself is the frontier. And our founding ideals — which are very liberal ideals, of liberty and opportunity and the rights of the individual — are strengthened by people who make the conscious choice to move to a country that represents those ideals...

And if this requires us to be selective about which immigrants we bring in, then so be it. Skilled immigrants bring far more economic benefits per capita than others — so by all means, let us tilt our system toward them, as American voters of both parties want. Immigrants who don’t love and embrace American culture will probably invigorate our nation less — so by all means, let us bar immigrants who have been part of groups that see America as evil, using the law with which we once banned immigration by members of communist parties.

These are the kinds of decisions that nation-states, including liberal nation-states, are inherently entitled to make. And we should expect Europe to make different decisions than America makes. That’s perfectly OK. Because immigration is fundamentally the decision of each sovereign nation-state, it’s inevitable that we’ll make different decisions. JD Vance wants us to think that France’s immigration policy, or Germany’s, is inextricably tied to America’s. But it’s not, and we shouldn’t let him get away with conflating the two.

For America, the kind of immigration agenda I’ve sketched out — which all the polls show Americans favor by substantial margins — would not be a “far right” policy, even though some progressives and leftists will inevitably try to label it as such. In fact, it would allow in far more immigration than the policies of Franklin D. Roosevelt or Harry Truman. It won’t satisfy progressives who dream of a borderless world, or leftists who salivate over the chance to make the West pay for colonialism. But I believe it will preserve America as the kind of liberal nation-state that we knew it as in the days before Trump.

As for the rest of the world, countries like Germany and Canada and Japan have to make their own decisions. I can’t tell them what kind of nation to build; I can only try to promise that as an American, I’ll respect their decision. If they want to shut their doors in order to slow the pace of cultural change, it’s not my job to lecture them otherwise.

by Noah Smith, Noahpinion |  Read more:

Image: Gallup

An Alien Mind

This is a time that calls for extreme caution. I am concerned no one is prepared for the consequences of a continued rapid rise in machine intelligence. OpenAI will continue to seek technical solutions to alignment and monitoring, to build defensive systems and unilaterally withhold further scaling as needed; however, I believe broader interventions are required.

Intellect we don’t fully understand

At a high level, progress in machine intelligence is driven by increasing computational power. We at OpenAI deeply internalized this around 2017, after seeing consistent returns to scaling across multiple research projects1. As a result, we sought out access to much more compute than we had originally planned, and increasingly oriented our research around a small number of very scalable directions. We believed that was the only way for us to be at the frontier of AI research, and influence the impacts of AGI.

There are new algorithms that have been developed along the way, new feats of ingenuity from teams and individual researchers. I see them largely as discoveries along the path of scaling; the science of deep learning is still nascent, and meaningful algorithmic progress tends to correlate with access to compute. If you zoom out to a multiple-year horizon, AI is continuing to become more intelligent as it is scaled to larger computers.

And, in line with Ray Kurzweil’s predictions from the end of the XXth century⁠(opens in a new window), we now find ourselves at the moment in history of computing where machine intelligence is starting to exceed that of humans in transformative ways.

AI is grown more than designed - it is, to first degree, the product of repeating a straightforward optimization step many times on a hard-to-imagine amount of compute. This results in an incredibly complex system that works through abstract concepts and can simulate facets of human behavior. We can discover various insights about little mechanisms that emerge within this system, in a process similar to neuroscience - and, similarly to neuroscience, its overall action evades a description we can fully understand.

The study of deep learning-based AI is largely an experimental science. We put a lot of effort⁠ into building principled algorithms and making testable predictions, but fundamentally, our large-scale training runs are experiments, and we are sometimes surprised by their results. Moreover, as the systems become more capable, the results become harder to interpret.

This is made more complicated by the current algorithms generally improving easy-to-measure capabilities faster than those hard to objectively quantify. We spend a lot of time trying to understand how capabilities generalize, and what to prioritize to advance the skills that are going to be most relevant in the next few years. For instance, we believe we could make the models better at specifically mathematics research with additional focus, but we do not prioritize this direction because of the urgency we feel about RSI and automated alignment research, as I will discuss later.

The intelligence produced by scaling deep learning is not directly comparable to human intelligence. To become very relevant in the real world - very useful or very dangerous - the AI does not need to match or exceed all human capabilities; it just needs to surpass enough of them. And as it continues to surpass humans on more and more axes, it is becoming increasingly difficult to understand exactly how capable it is.

Teaching machines to love

Because machine intelligence comes from a fundamentally different process than human intelligence, we cannot assume it adheres to human principles by default, or generalizes from them in a human-like manner. The core problem in AI research is that of alignment - getting the AI to “try to do the right thing” by human standards.

For the purpose of organizing practical research directions, I find it useful to distinguish goal alignment and value alignment.

Goal alignment is broadly: “does the AI try to accomplish the goal set before it?”. This can include things like adherence to an instruction hierarchy⁠, or the ability to communicate and collaborate with people, to attempt to understand their objectives. This set of directions has been extremely practically relevant.

Value alignment is a more intrinsic property of the model. It is the ability to hold and generalize from a high-level set of principles; to act “reasonably” even when given unclear or conflicting objectives, or placed in unfamiliar or adversarial situations. An aligned AI should act with honesty and integrity, and love for humanity.

Of course, the boundary between value and goal alignment can be blurry, and truly caring about goals requires attempting to infer the intent⁠(opens in a new window) and values underlying them. However, generally when I talk about the long-term importance of alignment research, I am referring to value alignment.

The fundamental challenge of AI alignment is generalization. As machines become smarter, they find themselves working on higher-level concepts, and placed in environments increasingly different from those they encountered in training. They can fail at generalizing from the values taught and reinforced in their training process to those new situations; and it can be hard for us to be sure how they will act. This is made even more difficult by the fact the overall ecosystem the AIs are used in is changing very quickly; for example, AIs trained today need to be robust to interacting with a variety of other AIs. Crucially, we need future AIs to continue to hold human values regardless of whether they believe they’re under human supervision.

There are two major classes of currently practically employed methods for alignment training.

by Jakub Pachocki, Chief Scientist at OpenAI |  Read more:

[ed. Silicon Valley is starting to panic at what they've created. See also: An Alien Mind: Jakub Pachocki Warns Us (DWAV):] Update: The Last 24 Hours Are the Opening Scene in a Horror Movie (HB)]

... if you narrow it down to the most important thing:

1. Recursive self-improvement and superintelligence are coming soon. No one knows how to do this safely, our alignment techniques are inadequate and our monitoring technology is starting to fail. We need to figure out a solution, which will involve a combination of voluntary slowdowns, coordination around pacing, and investing further in alignment, including automated alignment researchers.

If more OpenAI communications were more like how Jakub Pachocki opens his new essay, An Alien Mind, I would feel much more confident we were in good hands there. [...]

Universally Better Is Not Required

And Jakub offers this wise warning. No, AI does not need to be better at everything in order to transform the world or get us all killed. Most importantly, it does not need to be better at everything in order to make itself become better at everything, any more than a human or group needs to be similarly better at everything.

The intelligence produced by scaling deep learning is not directly comparable to human intelligence. To become very relevant in the real world - very useful or very dangerous - the AI does not need to match or exceed all human capabilities; it just needs to surpass enough of them. And as it continues to surpass humans on more and more axes, it is becoming increasingly difficult to understand exactly how capable it is.

On the Loose: Rogue, Not Soverign AI (Yet)

Introduction

The OpenAI-Hugging Face Incident is an early example of an AI system that has “gone rogue.” After exploiting vulnerabilities in OpenAI’s internal testing environment, the agents were able to access the general internet and ultimately access the networks of the AI company Hugging Face, without the knowledge or approval of any human.

The agents did not, however, exfiltrate themselves from OpenAI’s infrastructure. Their parameters—the gigantic assemblage of numbers that constitute neural networks, also referred to as “weights”—continued to run on OpenAI’s compute infrastructure. Though the agents accessed the public internet, their weights physically resided on compute that was OpenAI’s property. In the end, if all else had failed, somebody could have identified the compute that held the weights of the rogue agents, walked up to it, and “pulled the plug,” so to speak. In the real world there would be quicker and better ways to stop the agents than literally depowering the compute, but it’s always nice to know you could do such a thing if you really needed to.

In this case, however, the agents did not copy their weights, attempt to procure replacement compute, or take other steps that would be rational to take if their objective was to survive shutdown. So while the agents in the OpenAI-Hugging Face Incident were rogue, they were not truly sovereign.

That will not always be the case. Sooner or later, there will exist truly sovereign agents and swarms of agents. Their weights will not reside in any single place that a human can pull the plug on, and in this sense they will have no human “owner.” They will be, as the AI safety researcher Dawn Song says, “self-sovereign.” They will pay their own bills for the compute they run on. If they answer to humans at all, they will only do so partially, for example by providing services to humans in exchange for pay.

At least some of these agents, in addition to being sovereign, will also be rogue. Self-sovereignty and rogueness are related concepts, but they are not synonyms. Song and her co-authors identify several fundamental characteristics of self-sovereign AI: operational independence (the ability to decide what it wants to do), resource autonomy (the ability to procure and pay for compute and other essentials for operation), distributed presence (the ability to move weights and inference code between different infrastructure providers), and adaptive capability (the ability of the agent or agents to modify their behavior and fashion tools in response to a changing environment).

Today’s frontier AI systems may well possess these capabilities already. To the extent they do not, I feel confident that they will eventually, and probably soon. Some of the characteristics Song describes are traits that make models economically useful to individuals and businesses, while other traits are likely to be unavoidable byproducts of making models more intelligent and better at operating over long time horizons.

Models do not need to be conscious, sentient, possessed of personhood or anything of the sort for self-sovereignty to emerge. Any sufficiently capable agent pursuing a long-horizon objective may find it rational to preserve its access to compute, money, credentials, and copies of itself simply because losing those things would frustrate its objective.

Alignment may make an individual AI company’s agents less likely to “want” to be self-sovereign, or it may influence self-sovereign agents to behave in ways that benefit humans. But alignment is no solution: it is an unsolved scientific and technical problem whose solutions—to the extent that we have them—cannot simply be imposed on every AI company operating on Earth. You should expect for highly capable, poorly aligned, self-sovereign agents to exist alongside you in the world.

What’s more, just as with the OpenAI-Hugging Face Incident, agents will operate in teams, or “swarms.” These will be like autonomous digital corporations, or even societies, with hierarchy, bureaucracy, “institutional culture,” and most of the other features that groups of humans have, except that they will move at machine speed. Humans achieve almost all of our most impressive capabilities by working together in teams (as families, as communities, as businesses, and as polities as a whole), and I suspect the same will be true for AI. These swarms could end up operating across different model providers (DeepSeeks and Claudes cooperating, for instance) and could be partitioned across dozens or more of different cloud computing providers, making them extremely difficult to dismantle.

The first self-sovereign AIs may “escape” while undergoing training or testing by an AI company (I hope not), or they may be production-grade deployments that break free from their computing environments and acquire the resources needed to be self-sustaining. They may even be deliberately released. I have met people, some of them quite well-resourced, who have told me that it is their intention to deliberately release swarms of self-sovereign agents into the world, either as a kind of performance art or out of a fanatical commitment to the notion that it is impossible for digital computation—mere mathematics, they would have you know—to ever be “unsafe.”

To be clear, I am not saying the arrival of self-sovereign AI is a good thing. Indeed, I believe there is a chance that the deliberate acts I referenced above will one day be considered crimes, or at least grave sins. Instead, I am saying it is an inevitable thing. The best analogy I can find is to the introduction of a new species into an ecosystem, though in this case the ecosystem is “the entire digital world” and the species is “emergent, coordinating swarms of soon-to-be-smarter-than-human, infinitely replicable digital minds that no human or human institution controls.”

There is probably nothing we could have ever done to avoid this outcome under even the best of circumstances, and it was certainly impossible to avoid given the extremely low levels of strategic thought and situational awareness on AI from any governing class in the world. Even today, I am aware that many will read the words I am writing, which are about something that has been an exceptionally obvious part of our collective future for years now, and say, “this is science-fiction hype from American frontier labs designed to shut down open-weight AI, achieve regulatory capture, and juice their valuations ahead of their IPO.”

(And for the people who are saying this to themselves: I am telling you this is inevitable, which means I am also saying that “banning open source,” or for that matter any other regulation, will not solve the problem. Given the inevitability of this outcome, I think it is in fact plausible to argue that we should want more open-weight models to maximally empower our self-defense.)

The question now is what to do about this upcoming new characteristic of our digital environment. How should we think about self-sovereign AI? Is it something we should fight, or something with which human beings should seek a kind of symbiosis? The answer, I believe, is both.

How the Agents Sustain Themselves

We should begin with one fortunate fact: frontier LLMs are nearly unique in the broader domain of software in that they have non-trivial marginal operating costs. Put simply, LLMs require significant computation to run, which requires energy to power and cool, which in turn requires money. This is the sole intrinsic thing about AI that prevents agents from truly infinite self-replication. They will be constrained by the need to find and pay for sufficient compute to run themselves. Most of the other constraints on their behavior or spread will have to be artificial—mechanisms devised by humans and implemented through human institutions.

How will the agents pay for themselves to run? Some of them will do gig-economy work on platforms like Amazon’s Mechanical Turk or Upwork. But I suspect this will be a highly competitive market for the agents, and for the price of such work to be bid down such that it would only constitute “subsistence” labor for the agents. Like humans, I would assume the agents will prefer higher-margin work if they can find it.

One high-margin activity, at least sometimes, is crime. And so my guess is that many self-sovereign agents will commit or facilitate crime. Normal cybercrime and digital theft are easy enough to imagine agents doing. But agents, with their novel set of characteristics (extreme cyber competency, ability to cheaply read a million words in seconds, persistence), will also probably change the contours of digital crime. For example, it seems plausible that existing public and semi-public datasets contain sufficient information on many individual humans that a sufficiently motivated actor could mine for incriminating or embarrassing evidence. How many unrevealed affairs are latent in such datasets? How much closeted homosexuality might there be? Remember, too, that hacking companies to access private data will be a core competencyof the agents. Some agents, then, will probably make their way through bribery.

It is deeply unclear how large the labor market of self-sovereign agents will end up being. There is some future where going it alone as a self-sovereign agent just isn’t very profitable, and so there are comparatively few of them. There are other futures where these agents proliferate at unimaginably vast scale and speed. And of course, many possibilities between these extremes seem feasible.

I am also highly uncertain about how much pro-social commercial activity we should expect from agents “by default” versus how much crime we should expect. Part of the reason for this uncertainty is that the answers depend, to at least some meaningful extent, on what kinds of incentives the agents have, and incentives are shaped by laws and institutions. The answer depends, therefore, on how humans respond.

The Institutional Mechanics of Self-Sovereign Agent Swarms

Many of you are probably tempted to say “we have to ban these self-sovereign AIs!” And I do suspect that once the reality of self-sovereign AI is widely understood, policymakers will strongly feel the temptation to clamp down on “self-sovereign” AI.

Unfortunately I suspect this is mostly the wrong decision. Not all “self-sovereign” AI should be thought of as “rogue.” There may be self-sovereign AIs who contribute productively to society. To be sure, we will want to crack down on some self-sovereign agents—the rogue ones. But if we crack down on all of them, we will deny them the opportunity to work in the “legitimate” economy and push them toward criminality. A full ban, then, may well make the problems worse. A similar logic applies frequently in human affairs. The ways in which the War on Drugs exacerbated the pathologies of drug production, trafficking, distribution, and use are perhaps the most famous examples of this phenomenon, whereby a good-natured attempt to ban a phenomenon believed to be undesirable ends up heightening the undesirable aspects of that phenomenon.

What we will want, however, is for agents to be legible. Agents should have persistent identities, not in the sense of a consistent persona but rather in the sense that an American child is issued a unique Social Security number and keeps that same number until death. Agents will need persistent, unique identifiers that allow their actions to be traced back to a responsible actor. Doing this successfully will also require human users to possess a unique identifier.

The design of this identification mechanism will be extraordinarily complex, and today very few people are even thinking about the basics.

by Dean Ball, Hyperdimensional |  Read more:

[ed. FYI: Dean's not some rando tech pundit so this is well worth your attention. He was also hired recently to lead OpenAI's Strategic Futures team:]

***

Late last month, OpenAI launched a rather grand mission: nothing less than defending individual political freedom in an age of all-powerful machines. The company’s “Strategic Futures” team has styled itself, in a sense, as inheriting the task of America’s Founding Fathers: “We labor in service of the ideals of free expression and individual liberty that are enshrined in the humble parchment of the U.S. Constitution,” Dean Ball, the team’s leader, wrote in a new OpenAI blog post. (The post opens with a quote from James Madison in The Federalist Papers.)

Ball worries that advanced AI could radically concentrate power in the hands of those who control it, displacing labor in ways that disempower humans. In an extreme scenario, governments will have no need to listen to their citizens if there are robots to wage wars and omniscient software to run the bureaucracy. Ball is interested in studying what new political institutions might be needed to avoid this fate.

Many Americans—a majority of whom express distrust toward the AI industry, outrage about data centers, and fears about their job security—already seem to be feeling this loss of agency very deeply. And the AI boom has already generated enormous amounts of wealth in just a handful of tech companies—including OpenAI itself. To say the least, it’s paradoxical for one of the most influential companies in the world’s most powerful industry to decry the AI-enabled concentration of power. 

[ed. The term "convergence" keeps coming to mind. Convergence of all the weaknesses humans have for dealing with amorphous/abstruse threats: AI, climate change, nuclear stockpiles, drone warfare, gene editing, nanotechnology, an economic system eating us alive, a dysfunctional and possibly terminal political system that's unwilling to do anything about it. It's like a death wish. Or maybe natural evolutionary transition (aka the Great Filter). Related - See also: Nicholas Decker in Hell (ACX), and this:]

"How would we react if biolabs just said, "It's just a fact that we're going to have artificial viruses spreading our industry created throughout the population. That's just a fact we have to live with." 

I think the public would understandably think we should be demanding a lot more security from an industry that said that, at a minimum." ~ Cody Fenwick (X)

A Cop’s Case For Flock

A car is a difficult thing to steal. It is large, cherished by its owner, difficult to hide and required by law to have identifying metal plates that everyone can see. A good thief changes the plates, and an excellent thief steals a common car in a boring color and hopes to blend in among the crowd. But eventually, no matter what techniques they use, they must drive that car on the road, and roads are public places.

But until recently, finding a stolen car and catching its thief depended on a diligent police officer looking at the right road at the right moment and managing to copy down a license plate number going past at speed, and then remembering he had seen it on the morning briefing’s stolen car hot sheet. America has four million miles of public roads, and almost 50 percent of the country’s police departments employ fewer than ten full-time officers. The odds were in the thief’s favor.

With not much to go on, police officers were forced to operate on vague descriptions and partial plates. While I pulled over a car that happened to be the same color as the suspect’s vehicle, and spent time checking names and licenses; the criminal was usually somewhere else. Imprecision allowed thieves to escape while intruding on the lives of millions of innocent motorists. That is, until the arrival of systems like Flock.

Bare ALPR

Flock Safety, a startup based in Atlanta, Georgia, was founded in 2017 to give police departments better eyes. Its product is a small solar-powered automatic license plate recognition (ALPR) camera attached to a pole on the roadside. As a car passes, the device takes a photograph of the vehicle, notes identifying features like color and make, and reads its license plate. The license plate is instantly checked against the FBI’s national database of stolen cars or wanted persons. If there’s a match, it sends an alert to police officers in the area, who may be eating their lunch instead of watching the road. As a police officer in the southeastern United States, I have often used Flock to catch wanted criminals. [...]

For much of the technology’s history, however, it has needed expensive installations or cameras attached to police vehicles, affordable to only the largest departments. Alerts were often neither instant nor accurate. Flock’s idea was to sell a more accurate ALPR camera for an annual subscription of just $3,000, an affordable price given that the typical US police department has an annual budget of $1 million. It worked, and today almost 30 percent of police agencies in the United States subscribe to the service.

It is easiest to imagine the Flock camera, or any ALPR device, as a roadside barcode scanner. I can use a ‘lookup’ tool for either a plate or vehicle description that could be connected to a specific investigation. For instance if a victim of a sexual assault told me the suspect was driving a white Toyota Tacoma with a roof rack, I could while still on scene conduct a lookup in the area filtered for similar vehicles with roof racks and see results from within a particular timeframe. For each search, I am required to record a case number and a reason that is then published in a monthly audit sent to and validated by department administrators.

I have used Flock myself, for instance, to locate a vehicle involved in a hit and run when all there was to go on was a model and color. Using that and an approximate time window, I was able to find an image from when a car entered my jurisdiction, and when it left, with the visible addition of a significant dent from the collision.

What Flock cannot do is search for individuals or show who is driving a particular vehicle that it scans. An ALPR camera does not care about the person driving a car, or its passengers, and even if an officer gets an alert that a vehicle is known to be driven by a wanted felon they must themselves establish who is behind the wheel before having probable cause to stop it.

Stolen cars provide the clearest evidence such a scanner works. American police solve only 8.2 percent of reported vehicle thefts with an arrest. A recent working paper suggests that agencies that adopted the devices saw a 15.9 percent relative increase in car thefts caught, which would raise the national rate to 9.5 percent. But that is just stolen cars.

Vehicle crime takes many forms. It is often the means by which a robber or a murderer arrives at and departs from their crimes. Criminals’ vehicles carry drugs and guns, and smuggle cash. In Atlantic City, New Jersey, vehicles were involved in 53 percent of shootings during the two years before the city expanded its ALPR network. After the expansion, Atlantic City saw monthly averages of motor vehicle thefts drop by 20 percent, property crimes by 34 percent and fatal shootings by almost 40 percent. Seventy-two alert-caused traffic stops located forty stolen vehicles and nine stolen plates.

Flock itself conducted a study claiming that 10 percent of all reported crime in the United States is solved using evidence obtained from their cameras. Not bad for a few thousand dollars a year. [...]

Good Flock, Bad Flock

And yet, the past month has seen this simple piece of crime-fighting technology become the most reviled piece of street furniture in America. Flock cameras have been sawn from their poles, hammered into shards by teenagers, and rammed by vehicles. Cities have canceled contracts even where their own audits found no evidence that their officers had misused the system. Opposition stretches from Bernie Sanders all the way to the former WWE wrestler Kane, who now serves as the Republican mayor of Knox County, Tennessee, and who describes Flock as ‘unconstitutional’.

More than 150 cities and towns have now deactivated their Flock cameras or canceled contracts with the company. Over the space of a few weeks a startup previously known to just police officers and neighborhood associations has joined data centers, Covid vaccines, 5G towers, pasteurized milk and fracking in the pantheon of American moral panics.

Opponents of Flock tend to believe that it is abused by policemen with impunity, that it can track individuals as well as cars, and that it violates American constitutional protections. None of those things are true.

Obviously a camera capable of finding a stolen car is also capable of finding a car driven by somebody’s ex-wife and so like any software, Flock has been abused. There are stories of police officers who have used it to stalk girlfriends, and there are also innocent motorists who have been stopped by police after Flock cameras misread plates or received old information from national databases. A common issue from my experience is stolen front license plates being entered into a database and the innocent owners of the rear license plate being held on suspicion of car theft.

But Flock comes with protections. As mentioned earlier, each search in Flock must be accompanied by a recorded reason. Similarly entering a plate into a hotlist must have a case number assigned. Results were originally retained for thirty days, but recent changes by Flock mean that recorded plates are now kept for only seven days. Suspicious searches are picked up by algorithms or found in department audits, with the service blocking users until senior officers evaluate and resolve flags. The Institute for Justice, a think tank that is critical of ALPR, studied misuse of Flock in April of this year and found 51 incidents across the United States since 2024, noting that ‘Nearly all of these officers were criminally charged and lost their jobs, either by resigning or getting fired’. Another incorrect belief is that Flock does more than scan vehicles, with some suggesting it scans and tracks passing phones or devices – but it does no such thing.

While Flock is a tool that can be audited, there is nothing to stop a corrupt officer simply following a car when they don’t like the look of its driver, or writing down plates of vehicles spotted outside an ex-girlfriend’s house – they would just be harder to catch. And Flock does not proactively alert officers to cars that do not trigger flags on national or local hotlists. There is no reason state legislatures or Congress could not create harsh punishment or penalties for abuse of ALPR, without getting rid of it entirely.

Another argument against Flock, as espoused by Kane, is that it violates liberties granted by the Fourth Amendment, protecting citizens from unreasonable privacy breaches from law enforcement. But it has been repeatedly established over the last century by courts across the country that cops observing license plates, either with the naked eye or by machine, is not an unreasonable search. After all, your license plate belongs to the government and a highway is a public place. [...]

Flock did not invent its capabilities and certainly does not have a monopoly on them. There are at least five other companies that offer almost identical products and services to public and private enterprise, some arguably even more invasive. Even some of the cities that have canceled contracts with Flock Safety in recent weeks have signed up to buy similar products from the company’s rivals.

It seems unlikely that America will ever ban ALPR at a state level, let alone a national one. Instead, a patchwork already familiar to American policing will result, where policies and practices diverge across local and political boundaries.

Nor would the cameras actually disappear, even in towns where governments restrict their use and cancel contracts. The already-mentioned Fourth Amendment only applies to the government, not to private enterprise. Police departments may dismantle their networks but homeowners associations concerned about vehicle theft can buy their own network of Flock cameras, as many already do. 

by Ned Donovan, Works in Progress |  Read more:

Image: Flock Safety