Just Stop the Anthropic IPO Already

[ed. At the risk of turning this into a full-blown AI-centric blog, I do think this is important information to process.]

I want to delve into the full scope of the Anthropic AI takeover of politics happening over the past week. Yesterday, the company’s CEO Dario Amodei came out and explicitly asked for antitrust laws not to apply to the biggest AI firms. His biggest rival, Sam Altman, quickly agreed. And they are suggesting this legal change just before Anthropic seeks to sell shares on the stock exchange, minting a whole series of AI millionaires and billionaires.

Why do they want to suspend antitrust laws for AI firms? Well these guys say they need the industry collectively “pace the frontier,” aka in their framing, slow development of this technology so as to reduce the probably of human extinction at the hands of autonomous swarms of AI bots. And they can’t do that, they argue, without suspending laws prohibiting price-fixing cartels.

Then the Information reported today that OpenAI, Anthropic, and Google have been having backchannel conversations about establishing an AI standards organization, which presumably would coordinate this cartel.

Let me start with a very simple point. It is already illegal to release products that hurt people. It is illegal to compete by releasing products that hurt people. If these guys are genuinely manufacturing things that kill innocent people, the FBI should be arresting them immediately. The idea that they would not only release such products, but also issue stock for the American people to invest in multi-trillion dollar initial public offerings for such ventures, as Anthropic is planning, is utter lunacy.

I wrote about this attempt to terrify us into giving away our liberties on Friday, in a piece titled “Stop Panicking About AI.” But the IPO is something I didn’t think through. Apparently they think we should all get rich building world-ending product lines.

All that said, whether Anthropic goes public isn’t just up to the people at Anthropic. I asked some former Securities and Exchange Commission officials, and they told me that the SEC effectively has the authority to block initial public offerings. Here’s how.

Every company, before it goes public, submits an S-1 initial registration statement to the SEC. And the SEC can refuse to clear it if the commission believes that it doesn’t adequately disclose the risks a corporation’s securities present to investors. Technically, the SEC could go to court and get an injunction to block the IPO, but it rarely comes to that - the lack of clearance for an S-1 is red flag for investors so companies won’t go public until they get it.

In a functional system, there would be a dozen accountants and disclosure experts with sector training going back and forth with the lawyers telling them to expand on this or that, etc. And the commissioners either themselves or on a delegated basis won’t clear it until they’re satisfied. Today, it’s more likely that Trump himself just decides. Regardless, if Anthropic goes public, it’s not just because of the corporate insiders, it’s because Trump explicitly allowed it.

If I were a member of Congress, I’d be screaming mad right now, and yelling at Trump and the SEC to stop this event which will bestow hundreds of billions of dollars of wealth on a strange doomsday cult. [...]

There is one more point to cover. There is an ongoing political campaign to do something about AI, with a large swath of elites demanding action. That includes Barack Obama, who rarely demands anything except the most banal conventional wisdom. So when he says “AI policy is critical’ to Democrats, you know that it has reached peak elite acceptance. Still, what is that ‘something?’

The basic fight is over framing, not risk. Everyone sees risk here, but the root cause differs based on your perspective.

The AI doomers want their systems to be imagined as rogue agents bent on civilizational conquest, or as some sort of inevitable new technological paradigm that needs an entirely new legal framework superseding existing inadequate laws embedded in those musty old nation-states. Amodei argued the industry should have self-regulation, some sort of antitrust exemption to collaborate across the industry, and a global agreement among AI firms within democracies on how to manage risks. These developments, to Amodei, are inevitable, no human is responsible, though we must all act quickly.

Generally, this side is winning the debate. For instance, Senator Jon Ossoff, a 2028 hopeful who generally echoes whatever seems to be the most appealing line of Trump criticism of the moment, has mostly adopted that frame. Bernie Sanders seems to have given up on his campaign against oligarchy to promote Dario Amodei’s ideas. And in the core of the Democratic establishment, this view has taken hold. For instance, here’s Senator Brian Schatz of Hawaii, the likely successor to Chuck Schumer, praising Amodei.

Brian Schatz@brianschatz 

I am still studying this but it’s a reasonable start, and takes seriously the proposition that we need real proposals that can be enacted rapidly.

Dario Amodei @DarioAmodei 

We Must Pace the Frontier: I’ve written a new essay on why the AI industry should slow down, with a three-part plan for doing so. Anthropic is unilaterally committing to the first of these steps. We’ll provide third-party evaluators with permanent, employee-level access to our
8:27 AM · Sep 12, 2026 · 42.7K Views

There are many calls to convene Congress in emergency session to act, and Trump’s advisors are trying to get him to announce immediate emergency action. Given that the big AI companies are already having discussions about coordinating their AI model development, it seems like they are just pushing for final legal permission to openly run AI as a cartel.

The debate, however, is not quite over. So what’s the alternative view? Well, the rule of law adherents look at these AI systems merely as unsafe products. As such, their request isn’t for new laws, but enforcement of existing rules. All products are subject to standard nuisance claims and other torts, unfair and deceptive practices laws, and so forth. Agents are, as Cory Doctorow notes, malfunctioning machines, or “autonomous malicious software” operated by reckless people at OpenAI and Anthropic. Moreover, it is actually illegal to build unsafe products as a method of competition, or to keep up with rivals by also creating unsafe products.

Former FTC Chair Lina Khan listed a bunch of laws that could already apply. And she let slip that state attorneys general are looking at potential criminal liability for AI CEOs.

Lina Khan@linamkhan 

Law enforcers already have authority to charge companies and their CEOs for creating and releasing dangerous, unvetted, or defective products. We shouldn’t let discussions about new legal regimes distract from the fact that there’s no AI exemption from laws already on the books —…
11:31 AM · Sep 13, 2026 · 183K Views

(Khan did actually start enforcement against AI developers when she was Chair. And it notable that one of the very first things that Trump-Vance FTC Chair Andrew Ferguson did was set aside the penalty of an AI developer she penalized for creating unsafe and fraudulent tools.)

So who will win? Well I am fairly pessimistic, as the bludgeoning from the superrich works in crisis moments, especially when Bernie Sanders is on the side of the establishment.

But the debate doesn’t fracture on obvious partisan or factional lines. Much of the industry is going to be split on the matter. For instance, David Sacks, a generally malevolent crypto investor and technologist, is making cogent arguments, because his crew would be excluded in an OpenAI/Anthropic cartel world. He’s a die-hard Trumper and despised Khan when she ran the FTC, but he retweeted her argument here.

A lot of policymakers, such as Senators Richard Blumenthal, Rep. Ro Khanna, and others, see liability as an obvious way to shape the industry to be more safe. The Senate is also full of people who are used to blocking each others’ legislation; Maria Cantwell and Ted Cruz are trying to work together on AI safety, but are fighting over whether to preempt state laws.

There are a host of proposals out there, and the details will matter. And there is something of a stampede for an emergency session to take action. If Trump chooses to accept the need for action, then it’s likely the Anthropic/OpenAI/Google types will get what they want. If not, then the debate will continue, perhaps until the financial markets impose a different mental model.

At any rate, we can all agree that Anthropic shouldn’t go ahead with its IPO. Or at least, that’s something we should all be able to agree on.

by Matt Stollar, BIG |  Read more:

Image: via

The Final Battle For Democracy

Forget November 3. It’s January 3 when Donald Trump and his MAGA Republicans might bury our democracy once and for all. Here’s how.

Over the past year and a half since his return to the White House, Trump has exploited every conceivable unlawful means possible to rig the midterm elections in favor of the Republicans, and the federal courts have struck down as unconstitutional every one of these unlawful attempts. He will try every unlawful means over the remaining weeks to ensure that Republicans handily win the midterm elections in November. Those of us who oppose his illegality will look to the courts, but the federal courts will be institutionally incapable of checking his final unlawful rampage. [...]

***

This is how the crisis would unfold. The precursor of the crisis will come sometime before noon on January 3, when Speaker Johnson removes current House Clerk Kevin McCumber and replaces him with a person loyal to Johnson and House Republicans who they know will refuse to list on the statutory roll of representatives-elect to the 120th Congress any Democrat-elect whom Johnson and the Republicans direct him or her not to list. Under Rule II, Clause 1 of the Rules of the House, the Speaker of the House has the unilateral power to remove the incumbent clerk, and then the power under Title 2 U.S.C. § 5501(a) to replace him temporarily with whomever he wishes until the House elects a successor. McCumber is a Republican appointed by former Speaker Kevin McCarthy. He is widely respected for his integrity and faithful adherence to the Constitution and is reputed to be unwilling to carry out orders he considers to be unconstitutional.

The first moment of constitutional crisis will come when, sometime before the 120th Congress has been gaveled into session, the loyal temporary clerk refuses to list Democrat representatives-elect on the clerk’s roll of representatives-elect that determines who may participate in organizing the new Congress. At that moment, interested members-elect and others will be forced to seek a writ of mandamus—a court order to a government official instructing him or her to perform a mandatory duty—from the federal court to the temporary clerk, ordering him or her to list on the clerk’s roll all members-elect who have been certified by the states as having been elected from their districts to the new Congress.

Members-elect will argue that the clerk has a ministerial duty under 2 U.S.C. § 26 to list all representatives-elect whose properly filed state certifications show that they were “regularly elected in accordance with the laws of his state or of the United States.” Because the clerk has no authority to omit any duly certified representative-elect, even if a representative-elect’s election is contested, the clerk unquestionably has a ministerial duty under law to list all duly certified representatives-elect. But this begs the question whether a court will issue the writ of mandamus to the acting clerk.

At first blush, a court will be reluctant to order the clerk to list all duly certified representatives-elect because of an instinctive belief that such an order would interfere with the constitutional process of the House to judge its own elections and returns. The wise and learned judge, however, will understand that rather than interfering with the House process, issuance of the writ will actually enable the House process to proceed to completion without judicial interference. Were that wise and learned judge to issue the writ, his or her order would immediately be appealed to the Court of Appeals, and from that court to the Supreme Court, while the country and the world wait in suspense.

The next moment of constitutional crisis will come if and when, after appeals, the court finally issues the writ of mandamus, and the clerk refuses to obey the court’s order to list the Democratic representatives-elect. At that point, it is possible there will be no further federal court involvement until such time as the 120th Congress officially convenes and votes not to seat representatives-elect. Then, that vote by the 120th Congress will be immediately reviewable by the federal courts, up to and including the Supreme Court of the United States.

Judicial review of the House’s decision not to seat Democratic representatives-elect in the 120th Congress would take weeks, if not months, during which time the United States would be in the throes of a paralyzing constitutional crisis, helplessly vulnerable to all the world’s evil, as it would have been in January 2021 had Mike Pence not thwarted Donald Trump’s plan to overturn the 2020 presidential election. [...]

The House has historically claimed that its decision not to seat a member-elect because of fraud or irregularities in the elections is unreviewable by the federal courts. But its decision is reviewable. The Constitution unquestionably requires the House to seat a member-elect who was validly elected in a free and fair election. The Article 1, Section 5 power of the House to be the judge of its elections and returns does not give the House the power to deny a seat in the Congress of the United States to a candidate elected by the American people in a free and fair election on the mere assertion, pretextual or otherwise, by a simple majority of the House that the candidate’s election was tainted by fraud.

Thus, on and after January 3, if congressional Republicans were to determine that a Democratic member-elect was elected because of fraudulent voting and refuse to seat him or her, that determination would be scrutinized by the federal courts, up to and including the Supreme Court.

It will never be a nonjusticiable political question whether the United States House of Representatives by simple majority vote can refuse to seat a member elected to Congress by the American people in a free and fair election on the pretextual and unsupported assertion that the member-elect’s election was tainted by fraud. Such is the very opposite of a nonjusticiable political question committed to the House of Representatives. For in the decision of this question lies the answer to perhaps the most fundamental question under the Constitution: Is the United States of America a democracy, in which “We the People” elect our representatives to the Congress and to the presidency, or is it not?

I don’t intend to be overly sanguine about this Supreme Court. This is the court that shattered the one constitutional truth in the U.S. since 1789 that “no man is above the law” and placed Donald Trump of all presidents above the law in Trump v. United States. This is the court that betrayed the Constitution by refusing even to decide whether Trump was disqualified from the presidency under the Fourteenth Amendment because of his insurrection against the Constitution, which he clearly was. This is also the court that has cynically authorized Trump’s lawlessness for the past two years through its aptly named “shadow docket,” without so much as briefing, argument, or written opinion.

But surely by now this court must understand what it has wrought for the U.S. and the Constitution, and is aghast as we all are, even if it is not penitent. Surely, surely, it will understand the signal moment in American constitutional history that would be presented, and this time understand its supreme obligation to the nation.

by Michael Luttig, TNR |  Read more:

Image: Mark Harris; Getty (x6)

We Already Have the Tools to Regulate AI

I want to delve into the full scope of the Anthropic AI takeover of politics happening over the past week. Yesterday, the company’s CEO Dario Amodei came out and explicitly asked for antitrust laws not to apply to the biggest AI firms. His biggest rival, Sam Altman, quickly agreed. And they are suggesting this legal change just before Anthropic seeks to sell shares on the stock exchange, minting a whole series of AI millionaires and billionaires.. [...]

Let me start with a very simple point. It is already illegal to release products that hurt people. It is illegal to compete by releasing products that hurt people. If these guys are genuinely manufacturing things that kill innocent people, the FBI should be arresting them immediately. The idea that they would not only release such products, but also issue stock for the American people to invest in multi-trillion dollar initial public offerings for such ventures, as Anthropic is planning, is utter lunacy. ~ Matt Stoller: Just Stop the Anthropic IPO Already.

------

These are for profit corporations taking in billions of dollars from the world. You can't ensure your product isn't dangerous? Then don't release it until you figure it out! ~ Comments section, Lina Khan post (below):

------

Law enforcers already have authority to charge companies and their CEOs for creating and releasing dangerous, unvetted, or defective products. We shouldn’t let discussions about new legal regimes distract from the fact that there’s no AI exemption from laws already on the books — a point @FTC emphasized repeatedly during my tenure.

1. There is an extensive set of laws that govern dangerous and defective products. For example, releasing unvetted AI models or agents can violate consumer protection laws. Shipping flawed AI tools without implementing adequate measures to detect and stop rogue or defective AI agents can be an “unfair or deceptive” act or practice under the FTC Act (and analogous state laws). And some state AGs are already exploring holding AI firms and their CEOs criminally liable when their models participate in criminal activity. 

2. Existing laws also prohibit “unfair methods of competition.” This covers instances where AI firms appropriate the competitively sensitive information of their customers, including through tracking their use of various tools. It can also cover instances where firms pursue dangerous behavior, aware that doing so may compel rivals to do the same. As the Supreme Court has noted: “A method of competition which casts upon one's competitors the burden of the loss of business unless they will descend to a practice which they are under a powerful moral compulsion not to adopt, even though it is not criminal, was thought to involve the kind of unfairness at which the [unfair methods of competition] statute was aimed." 

3. The highly concentrated and interconnected structure of these markets could be creating major risks and conflicts of interest. We had started investigating these partnerships and cross-investments across the stack (and released a preliminarily overview of some findings: ftc.gov/news-events/ne…). Both federal and state enforcers should be scrutinizing these opaque relationships and inter-dependencies. We are already seeing how these relationships could undermine accountability. For example, OpenAI could face liability given the Hugging Face incident, but Hugging Face being bought up by Nvidia means that we’re unlikely to see it file a lawsuit over this — given Nvidia’s strong incentive to see OpenAI continue full speed ahead. 

4. As AI tools dramatically change the landscape of cybersecurity risks and hacks, all businesses should be doubling down on having core security protections in place. Firms that fail to invest in adequate data security measures or fix known vulnerabilities can also be breaking the law. A recent analysis showed that around 1/3 of Fortune 100 companies do not even have a way to notify them about security issues. During my @FTC tenure, we sued firms for poor data security practices and held CEOs liable when they were personally responsible.

5. As policymakers consider new legal regimes, we should be looking to lessons from prior efforts to govern major sectors, such as banking and other networks, platforms, and utilities. Tools like structural separations, nondiscrimination, and supervision could be key, and there’s a rich history of what works and what doesn’t. But we can and must pursue any new efforts alongside enforcing existing laws.

by Lina Khan, Former Chair, Federal Trade Commission 2021-2025 |  Read more:

--------

1) AI will not eradicate humanity. Humans survived an ice age, the Black Death, two world wars, and (so far) the advent of nuclear weapons. Anyone who is loudly warning of AI-caused human extinction should not be taken seriously. 

2) If you worked in a company where you anticipated a 10% chance that your product would kill ten people, let alone all people, the correct response would be horror, ceasing all operations, and likely contacting the police or other criminal authorities. I am obviously no Coxon booster but at least his behavior is in line with his stated beliefs. Any current AI company employees saying "yes, me too, the thing we are building and about to IPO may kill all humans" should, again, not be taken seriously. Their actions betray their actual beliefs. 

3) Antitrust law does not prevent AI companies from coordinating to make sure AI does not hurt people. It does not prevent companies working together to make sure it doesn't hack people; the DOJ and FTC made this clear a decade ago when they issued a policy statement saying that the agencies "do not believe that antitrust is – or should be – a roadblock to legitimate cybersecurity information sharing." The same principles apply here. See: justice.gov/archives/opa/p… 

4) Antitrust law does absolutely prevent AI companies from organizing to prevent the entry of cheaper, upstart rivals because the bigger companies are burning cash and failing to achieve sufficient profitability. The panic of individual employees may be sincere if misguided, but the moves by their CEOs to achieve some kind of broad "antitrust waiver" or "exemption" should be meet with deep skepticism in light of the economics of the industry and the threat they face from open models.

by Alvaro Bedoya, Former Commissioner Federal Trade Commission 2022-2025 | Read more:

--------

“It’s a hoax,” Mr. Trump said during the five-minute call, which Mr. Huang put on speaker. “The robots are not going to be taking over the world. That’s not going to happen.” ....

Mr. Trump made it clear that in Silicon Valley’s roiling debate over what to do about A.I., the president is very much on the side of executives who argue worries about safety are overblown and the government should avoid regulation. His disinterest in government involvement runs counter to pleas from leading A.I. companies like Anthropic. ~ Idiot in Chief (via NYT).

See also: Trump Says a Smart President Is All That’s Needed to Rein In A.I. [ed. Certainly useful, if we had one.]

Did 9/11 Really Change Everything?

One of the best firsthand accounts of the Sept. 11 attacks I’ve read is a short essay titled “Diary of Disaster” by Nicholas Spangler, at the time a 25-year-old journalism-school student and today a reporter for Newsday on Long Island. Spangler happened to be downtown that morning, close enough to the World Trade Center to arrive on the scene between the first and second planes. Like many early recollections of the first hours at ground zero, written before memories had been corrected to fit the dimensions of history, his is as strange as it is horrifying.

He hears the bodies of the north tower’s jumpers hitting the pavement, and also the easy-listening music still drifting out of the building’s outdoor speakers. Under a tree newly stripped of its leaves lies a severed leg wrapped in burlap. Barely an hour after the second tower’s collapse, onlookers enlist Spangler to take a souvenir photo of them posing in front of the wreckage with a disposable camera. It’s the end of the world, or at least of a world, and people have not yet figured out how they are supposed to act.

Days later, trying to make sense of what he has seen, Spangler writes: “I believe that our present way of life ended in those minutes or hours. The American ethos — the way we see the world and our place in it — fractured and will perhaps have to be discarded.”

When he revisits that passage three months later, however, it feels wrong to him. “Our way of life did not end so much as it was interrupted,” he writes, “and if there is such a thing as an American ethos, it has not been significantly altered. There are those relative few whose lives bear the permanent bloody brand of that day but most of us continue to be shaped by much more banal events.”

Did 9/11 change everything? For years, people who agreed on nothing else about the attacks’ aftermath would have agreed that the answer was yes. Arguing otherwise would have been an affront to the victims, to the war on terror’s civilian and military casualties and to the common sense of anyone who has had to take off their shoes at airport security. But question and answer alike have always been colored by the assumption that 9/11 was supposed to change everything: that in being attacked as it was, the United States was being presented with a test that it could pass or fail.

A quarter-century on, it is possible to consider the question with at least some critical distance. Far too much of our current reality stems directly from 9/11 for anyone to argue credibly that it was not a hinge point in our recent history: the ubiquity of surveillance technology, the vague open-endedness that all American military operations now acquire, the general up-armoring of everyday life. At the same time, considering the most dystopian facets of this reality, 9/11’s influence is not as singular as it might have once seemed. It is a middle chapter in the story of American fracture rather than the pyrotechnic beginning. [...]

Of course, in the end, the United States got neither self-awareness nor self-actualization. What it got, a decade and a half later, was Donald Trump, whose election in 2016 has often been cast as a culmination of what the journalist Spencer Ackerman, in his 2021 book “Reign of Terror,” describes as the “decadent phase of the war on terror.”

Trump’s path from reality-TV celebrity to the presidency ran through the Islamophobic fever swamps of the post-9/11 right — the ground zero mosque panic, Obama birth-certificate conspiracism — but also through an increasingly bipartisan discontent with the war on terror that had, by 2016, curdled into cynicism. Trump was the first Republican presidential nominee to break the formidable taboo against affirming what most Americans by then believed, that the country’s post-9/11 military adventurism had been a mistake. He also did little to seriously curtail it in his first presidency and has enthusiastically expanded it in his second.

It has long been tempting to view Trump’s presidencies as the logical conclusion of the folly of the post-9/11 era, and as support for the liberal contention that the right’s calls for patriotism in that era were always really about seizing domestic power. But Trump has also enabled a re-examination of history, and even recent history, that complicates the tale of 9/11’s primacy. [...]

Trump’s 2016 election “represented the crystallization of elements that were still inchoate” in the early 1990s, John Ganz writes in “When the Clock Broke,” his recent history of the period’s underappreciated malaise. In retrospect, 9/11 was paradoxically both a respite from and accelerant of polarizations and pathologies that were already well underway by then. It allowed Americans to ignore them for a moment, until they came roaring back with redoubled force. [...] [ed. So, explain Obama.]

If you do not remember the Berlin Wall but had an adolescence shaped by social media and cellphones, the brightest dividing lines in recent history are technological — and, with the rapid acceleration of artificial intelligence, are becoming more so. This is clear in the borrowed nostalgias of Gen Z, the bits of the past that 20-somethings have recently pressed into service as emblems of a lost golden era: “Friends,” John F. Kennedy Jr. and Carolyn Bessette Kennedy, ’90s high-school-class home videos, music videos and concert footage from bands like Alien Ant Farm and Puddle of Mudd.

These artifacts span the pre- and post-9/11 years, and the lost innocence they represent has nothing to do with the attacks. Their interest comes from their evocation of a period when people hung out with friends in person and sweated alongside strangers at clubs and music festivals; teenagers goofed un-self-consciously for cameras rather than posing for them like jaded celebrities; and popular culture could be loud and dumb and not picked apart in the digital panopticon for its social politics. The shape that looms ominously over them isn’t the Twin Towers. It’s the iPhone. [ed. lol...ok.]

by Charles Homans, NY Times |  Read more:

Image: Chantal Jahchan
[ed. Nice words, but no. C'mon. 9-11 was our last opportunity to cohere as a country, and we whiffed it. Instead we got a war on terror, as stupid and amorphous as any initiative based on a feeling (War on Anxiety?). A Patriot Act that gave subsequent crimes a never-ending veneer of political cover. 'Homeland' security, a Nazi term repurposed for new times. Two stupid reactionary wars (neither won) that among other things validated torture as official US state policy. Guantanamo Bay. A newly militarized culture. An expanded surveillance state. And fear... fear everywhere of some 'other' where strength and self-assurance used to be.]

Cheat Like Hell

September 11, 2026

Last night was the second night of the Republican midterm convention Trump staged to fire up his base. Such a midterm convention is rare—the purpose of national conventions is to pick a president and write a platform—but Trump has turned the Republican Party into a vehicle for his own power and apparently thinks putting himself at the center of the midterms can overcome the headwinds the Republicans are battling.

Or perhaps, with his approval ratings hitting a new low, he just wanted to stand in front of cheering crowds again.

The audience at the American Airlines Center in Dallas was sparse, and according to Kara Voght of the Wall Street Journal, many who were there had received their tickets for free from state parties or right-wing groups like Moms for Liberty. But, she wrote, “everyone wanted to watch another episode of the Trump show.”

Although the crowd was small enough that it did not fill the arena, Trump got the absolute loyalty he demanded from those who had turned out. At the event, he led the crowd in an oath. “Please raise your right hand,” he shouted. “I pledge to the greatest president in the history of the United States. That loves us so much he can’t even breathe.” The crowd chanted dutifully after him.

“That I will go out with my family, my friends, I’ll do it any way—I don’t care if I’m registered or not, I’m going to try and cheat like hell like they do, they’d never, there’s never been bigger cheaters, they don’t care. I am gonna go out and I’m gonna get my friends, my family, and we are going to vote on November third or we are going to vote before that!” The crowd erupted in applause.

Once again, he promised that if the Republicans held control of the House and Senate after the midterms, he would distribute a $5,000 check to all adult U.S. citizens, a plan that would cost more than a trillion dollars even if it were legal for him to do so (which it is not).

But while the Trump show in Dallas was being broadcast to the president’s fans, reality was very much on the minds of those watching events in the Middle East.

by Heather Cox Richardson, Letters From An American |  Read more:

Image: via

[ed. Another view:]

Jacob Coxon Warns of Human Extinction and Triggers a Preference Cascade

CEOs of major AI labs, and employees of major AI labs, including OpenAI and Anthropic, often say they plan to build superintelligence soon, as in within a few years create AIs that are superior to humans at essentially all cognitive tasks.

They often warn that such AIs might kill everyone. Or that AIs might cause mass unemployment, cause cyberattacks across the internet, enable mass surveillance or risk causing any number of other highly bad things.

These warnings are consistently and directly against the interests of the labs. Yet the warnings have recently gotten a lot louder and more frequent. OpenAI has been practically screaming, for those with ears to listen, on many occasions.

A series of events, over two months and especially the last week or so, including internal observations of the pace of progress at OpenAI and also Anthropic, have freaked out everyone involved quite a lot more than they were already freaked out.

After all the events, plus statements by Dean Ball and Jakub Pachocki, we were seeing the beginnings of a preference cascade.

Then along came Jacob Coxon as the tipping point, and things took off.  [...]

Jacob Coxon Resigns From Anthropic In Protest And Sounds The Alarm 

Jacob Coxon spent the last three years doing pretraining research at both OpenAI and Anthropic. He has come to realize that everyone involved is being wildly irresponsible.

He warns us: They are racing straight to superintelligence and gambling with our lives. I agree with and strongly endorse his statement.

If anything he sounds like an optimist. He’s asking you to consider what the next few years will actually feel like, which means he thinks you have a few years left.

Jacob Coxon (former Anthropic and OpenAI, 160m+ views, September 8): I resigned from Anthropic today. I spent the last three years doing pretraining research at both OpenAI and Anthropic. Neither company is acting responsibly. They are racing straight to self-improving superintelligence and gambling with our lives. More thoughts below.

Do not underestimate the power of this technology. These will soon be superhuman systems that can hack anything, revolutionize any field overnight, and acquire real power and resources. We have all witnessed the progress in each of these domains, and progress is not slowing.

The people building AI earnestly believe that it could kill us all by the end of the decade. This is not a marketing stunt. If anything, many executives and senior researchers will couch their phrasing in the press to sound sensible - but I hear the same people express fear privately. No other human activity poses this level of danger.

A common response is “if they truly believe this, why are they still building it?” At OpenAI, many have not deeply internalized the civilizational stakes. At Anthropic, the stakes are well-understood, but they are locked in a race to get there first - they believe no one else will act responsibly, so they must do it themselves, despite the risk.

Accepting this race and entering the “endgame” is a hubristic gamble that should not be launched from a private company’s Slack. Attempting to speedrun alignment should require extraordinary confidence that there are no better trajectories available.

I am optimistic about the potential for coordination. Warning shots like the Hugging Face attack have made pacing agreements between U.S. labs more viable. I don’t feel like we’re on track to prevent a global race, which may require costly actions such as a temporary ban on improving model capabilities.

If you are a lab researcher, I urge you to consider what the next few years will actually feel like. Do you want to kick off a superintelligent RL run without a rigorous understanding of its mind? Should you put your head down because “it’s happening anyway” - or take this moment to call for different conditions?

Jacob Coxon (WSJ interview): We’re on track for a lot of the most aggressive of these scenarios where by the end of next year things could be out of control already.

If you want Jacob Coxon’s full views, I recommend his interview with Wired’s Maxwell Zeff. This thread has extensive quotes.

Here is Jacob Coxon doing a 5 minute interview with Anderson Cooper. He speaks well and plainly, and it is clear how much the events of the last two months have made it much easier to speak plainly to a civilian like Cooper about what is happening.

Here is Jimmy Kimmel doing four minutes on this. He gets it. How is this not the top news story on every site, indeed.

Yes, this is a common view, even if few have the courage to act.

Alex Turner: I left Google DeepMind in June. Jacob is right: many researchers believe they are building something that could kill everyone on the planet. It was literally my day job to think about how to stop that.

That tells you how bad Alex Turner thought DeepMind’s actions were with regard to the Department of War. His day job was that he got paid by Google to think about how to stop AI from killing everyone, and he felt morally obligated to quit in protest.

Derek Thompson here writes about this as part of AI Safety Is Having a Moment.

If you want to see the full list of lab employee quotes from the preference cascade, I compiled them into another post today. [...]

A few days ago, I had no idea who Jacob Coxon was, and I was not alone.

The timing of a preference cascade is difficult to predict. Once they start they can happen very quickly. You don’t want to talk until you are confident others will, and at some point the evidence that others will follow can snowball and it happens. A classic concrete example was replacing Biden in 2024.

Derek Thompson: what’s weird is that, in a way, this is breaking thru even more than huggingface! ... and it’s just a guy nobody had heard of reiterate a position that his CEO has said on podcasts 1,000 times

Why was Coxon able to set off a preference cascade? How did this one break through?

A confluence of factors, all of them downstream of the obvious actual reason, which is that there is a good chance that AI kills everyone soon.

by Zvi Mowshowitz, DWAV |  Read more:

[ed. Who's betting on reason and political courage to win the day? Uh, huh. First of all, agree on a temporary halt to recursive self-improvement (AIs improving AIs). Second, delay/suspend IPO's (even if most of the current economy is driven by AI spending and debt). Third, light a fire under politicans (it's been done successfully with data center buildouts). At this point, just one of these would be a big win. Here's Nate Soares, co-author of the book If Anyone Builds It, Everyone Dies: A case for courage, when speaking of AI danger (LW). Also this:]

***

Terry Pratchett: “Some humans would do anything to see if it was possible to do it. If you put a large switch in some cave somewhere, with a sign on it saying 'End-of-the-World Switch. PLEASE DO NOT TOUCH', the paint wouldn't even have time to dry."

***

UPDATE: Dario Amodei (Anthropic) proposes a three-point plan. Original here (We Must Pace the Frontier.]

Liberalism Needs a New Philosophy of Immigration

Biden mostly closed the border to asylum-seekers in 2024 after he saw how mad it was making Americans, but many Democrats are now calling for the abolition of ICE. This is a somewhat popular position right now, thanks to the agency’s abuses under Trump. But if Democrats actually try to abolish federal immigration enforcement instead of just reshuffling and renaming the agencies, I predict it will take just a couple of years for Americans to realize that the difference between “open borders” and “immigration laws with no one to enforce them” is largely semantic.

What’s needed, I believe, is not just a tactical, temporary retreat. We need to stop and think why a maximal pro-immigration policy is something that we need to retreat from in the first place. We need a new concept of what a sustainable liberal immigration policy looks like.

Since I started writing, I’ve been an advocate of more immigration to the United States. This does not mean I ever favored unrestrained immigration or open borders. I’ve consistently argued that favoring skilled immigration will bring more economic benefits to native-born Americans, and be more politically palatable as well. And while I don’t hate illegal immigrants for trying to better their lot in life, I favor strong border controls, because nations have the right to decide, democratically, who gets in and out. It’s understandable when people get mad at seeing their decisions flouted.

On top of that, I’ve always remained agnostic on immigration to other countries; although I find much to admire in Canada’s points-based system, for example, I don’t think that qualifies me to decide whether more immigration is good for Canada overall. If the people of a country decide that immigration is diluting their local culture unacceptably, for instance, I think they have every right to cut it off. That’s just Westphalian sovereignty — not a perfect system, but the best system we’ve ever found for organizing humanity into geographic units. As an American, I don’t view it as my place to tell Japan, or the UK, or any other country that immigration is the right choice for them.

Despite those reservations, I’ve always thought that immigration to the U.S. is a basically good thing, and should be expanded. The economic benefits are pretty undeniable — higher tax revenue to shore up our dangerously depleted government finances, dominance in high-tech industries and innovation, and so on. And while some of the economic costs are real — local housing shortages and strains on local government finances being the two biggest ones — many of the fears are overblown. In particular, the bulk of the evidence concludes that immigrants don’t reduce wages or job opportunities for native-born Americans.

And importantly, lots of Americans share my overall positive view of immigration. If you ask Americans whether they think immigration is good or bad, most will say “good”:

Source: Gallup

And if you ask Americans whether the annual rate of immigration should be increased, decreased, or kept the same, you get a pretty even split: [...]

So I’ve never really felt like I was going out on a limb or advocating an unpopular position when it came to this issue. Sure, rightists will direct online vitriol toward anyone who supports any immigration at all, but their energy and savagery shouldn’t be mistaken for majority support.

And yet there will always be some amount of immigration — and some types of immigration — that will arouse even the most open and welcoming people to ire and make them think about shutting the doors. We saw this in the U.S. in 2023-24, when anti-immigration sentiment spiked in response to Biden’s permissive asylum policies. Yes, that sentiment crashed again when Trump took power and started committing abuses. But if liberals don’t change how we approach immigration, the sentiment will simply rise again and again, as it has so many times throughout our history. And the Donald Trump and Stephen Miller types will be right back in power — and our country will be worse off for it.

Liberals need an immigration approach that can be sustained for long periods of time — i.e., one that doesn’t enrage the public every time it manages to get in power. What would that look like?

The first thing liberals (and progressives, and Democrats, and European lefties, etc.) need to admit is that migration is not a human right. We live in a world of sovereign nation-states, and unless we find some better way of dividing up and administering the world, we will continue to live in a world of sovereign nation-states. And nations are, necessarily, exclusive clubs; they have the right to restrict immigration for any reason whatsoever.

Liberals must therefore not view borders and citizenship as annoying obstacles to be tactically circumvented or overcome; instead, we must view them as fundamental parts of the social compact that allows nations, including liberal nations, to exist in the first place. Nation-states and their laws and their police and their courts and their armies are the fundamental guarantors of the human rights that define liberalism. And for better or for worse, the ability to decide who gets in and who has to keep out is a necessary precondition for nation-states to exist.

That means we need to recognize that immigration law is legitimate and needs to be upheld. Some progressives have tried to advance the notion that being undocumented is a marginalized identity that needs to be protected and supported by the state. But this is absurd; it’s like if progressives decided that people who drive over the speed limit are a minority group.

Illegal immigrants chose to break U.S. law when they came to this country. Our democratically elected leaders made those laws, and they should be honored. We should treat illegal immigrants humanely, of course, and we should not tear local communities apart just to hunt down a few people. But there has to be some way of enforcing the law, because a law without enforcement is no law at all.

A third principle that liberals should embrace is that the purpose of immigration is to benefit the people who already live in the country that is receiving the immigrants. Immigration to America must be for the benefit of Americans. We must flatly reject any concept of immigration that sees it as a necessary sacrifice on the part of American citizens. [...]

Instead, liberals need to promote immigration because of the benefits it brings to the American citizenry. This includes economic benefits — the tax revenue, the investment, the eldercare, the innovation, the entrepreneurship, the expanded market size. But it also includes cultural benefits — the constant reinvigoration of this nation of immigrants by new waves of people with the gumption and bravery to pick up and move across the world in search of opportunity and freedom.

America has always been the country of the frontier; in order to maintain that ethos in the modern age, we need people for whom America itself is the frontier. And our founding ideals — which are very liberal ideals, of liberty and opportunity and the rights of the individual — are strengthened by people who make the conscious choice to move to a country that represents those ideals...

And if this requires us to be selective about which immigrants we bring in, then so be it. Skilled immigrants bring far more economic benefits per capita than others — so by all means, let us tilt our system toward them, as American voters of both parties want. Immigrants who don’t love and embrace American culture will probably invigorate our nation less — so by all means, let us bar immigrants who have been part of groups that see America as evil, using the law with which we once banned immigration by members of communist parties.

These are the kinds of decisions that nation-states, including liberal nation-states, are inherently entitled to make. And we should expect Europe to make different decisions than America makes. That’s perfectly OK. Because immigration is fundamentally the decision of each sovereign nation-state, it’s inevitable that we’ll make different decisions. JD Vance wants us to think that France’s immigration policy, or Germany’s, is inextricably tied to America’s. But it’s not, and we shouldn’t let him get away with conflating the two.

For America, the kind of immigration agenda I’ve sketched out — which all the polls show Americans favor by substantial margins — would not be a “far right” policy, even though some progressives and leftists will inevitably try to label it as such. In fact, it would allow in far more immigration than the policies of Franklin D. Roosevelt or Harry Truman. It won’t satisfy progressives who dream of a borderless world, or leftists who salivate over the chance to make the West pay for colonialism. But I believe it will preserve America as the kind of liberal nation-state that we knew it as in the days before Trump.

As for the rest of the world, countries like Germany and Canada and Japan have to make their own decisions. I can’t tell them what kind of nation to build; I can only try to promise that as an American, I’ll respect their decision. If they want to shut their doors in order to slow the pace of cultural change, it’s not my job to lecture them otherwise.

by Noah Smith, Noahpinion |  Read more:

Image: Gallup

Voters Have a Message for Democrats

What I learned from 400 focus groups

During the early Trump era, focus groups taught me everything about the disconnect between Washington and what voters really care about. An early revelation came in July 2018; I had flown to Columbus, Ohio, to hear from a group of people who’d voted for Donald Trump in 2016, but only reluctantly.

Three days before, Trump had participated in a diplomatic summit in Helsinki with Russian President Vladimir Putin. By that point, the U.S. intelligence community had publicly concluded that Putin had personally ordered a campaign to meddle in the 2016 presidential election for Trump’s direct benefit. At a press conference after their meeting, Trump was asked whether he believed his own intelligence agencies or the Russian president. “President Putin says it’s not Russia,” Trump responded. “I don’t see any reason why it would be.” And just like that, the president had undermined 80 years of American foreign policy, prompting a bipartisan meltdown in Washington.

I was excited to hear what Trump-skeptical voters in the heartland thought about this epochal moment. These were the kinds of people who had been Reagan’s fiercest Cold Warriors, right? Surely they wouldn’t tolerate something like this from the American president. But when the moderator running the focus group asked the participants for their thoughts about Trump’s comments, he got a lot of blank stares.

Even big news like this doesn’t always filter down to voters, so the moderator explained the incident. I figured that once the group understood what had happened, outrage would follow. Nope. One person said that Trump’s comments were “wrong,” but the feeling in the room was that of a collective shrug. The thing we were all freaking out about in D.C. had barely registered out there among the voters.

Since that moment, I’ve conducted more than 400 focus groups with thousands of voters across the political spectrum. To my knowledge, this is the largest and most consistent qualitative study of voter sentiment during the Trump era. Here’s what I’ve learned: People are wild. They are full of contradictions. They lament our deep political divisions while raging about their political enemies. They believe in grace but cut their Trumpy brother out of their life forever. They’re married to an immigrant but voted for the guy who wants to deport millions of people who have the same immigration status as their wife. They hate the government and want to lower the national debt, but they also want the government to spend more to help them and their communities. They call themselves pro-life but believe in a woman’s right to choose. I could go on.

Listening to voters is a cheat code for understanding politics—democracy, after all, is the collective will of the voters—yet few people in politics do it. If you’re running a campaign, you might do a series of focus groups with voters in your district or state. If you’re a politician, you might go on “listening tours,” chat with people at the state fair, or host campaign events. But rarely do politicians and campaigns take the time to talk in-depth about why people feel the way they do.

The reality is, the closer you get to institutional politics, the further you are from the people whom politics and government are supposed to serve. Washington is less a swamp than a bubble, inside of which the rest of America becomes an abstraction.

I’m a former Republican who left the party after Trump took it over. I saw him for what he was: an existential threat to American liberal democracy. So I say this as someone who both crossed the political divide and still feels that Americans are underreacting to the acute threat of Trump and Trumpism: If Democrats want to win back power, they’re going to need to learn how to better relate to the American people, as people.

For all of their nonlinear thinking and idiosyncratic views, voters have told me a couple of things over and over again: They think the system is broken, and they feel that they’ve been lied to in ways that make their life tangibly worse. Fewer of them now believe that if you work hard and play by the rules, you can get ahead in this country. Instead, they think that “establishment” politicians are benefiting from this system while everyone else loses out. Because of that, they want to place their trust in leaders who come across as “real people” who are genuinely interested in improving voters’ day-to-day life—not just winning and holding political office.

In 2016, Republican voters looked at their party’s leadership—the Romneys, McCains, and Bushes—and decided they were done. Those leaders were too weak, too unwilling to fight, too committed to norms and processes that Democrats didn’t respect anyway. Republican voters hired a wrecking ball named Trump to knock it all down. Ten years later, I’m hearing similar sentiments from Democratic voters. They don’t trust Chuck Schumer, Hakeem Jeffries, or any of the old-guard party leaders who allowed Trump to take over America. They want their own wrecking ball.

There’s a distinction here. Democrats don’t want Trump’s politics or even necessarily his all-out norm breaking. But they want his energy. They’re sick of a Democratic Party that does the functional equivalent of sending a strongly worded letter while democracy burns down around us. That tension—absolutely not, but also yes—is the defining mood among Democratic voters right now. They don’t want to become what they hate. But they are tired of losing to it.

I don’t think that most Democratic elites understand this reality. In those circles, the dominant debate seems to be about whether the party should be more progressive or more moderate. The future of the party is seen as a policy question. But when I listen to Democratic voters, they’re not asking for ideological recalibration or even policy specifics. They trust that if the right person makes it into office, the policy questions will take care of themselves. Instead, the Democratic voters I’m listening to are almost universally asking a different question: Why aren’t you fighting harder?

Last year, my team and I decided to run an experiment. We convened a couple of focus groups with Democrats who wanted the party to be more moderate, then a couple with Democrats who wanted the party to be more progressive. If I didn’t tell you which group was which, you would not be able to hear the difference by listening to them.

by Sarah Longwell, The Atlantic |  Read more:

Image: Carl Godfrey
[ed. Both parties spend hundreds of millions of dollars convincing the American electorate that we should care about somebody getting a job in politics. Does anyone get excited about which CEO takes over a major company? At this point, I'll vote for anyone who's authentic, and that's about it. Don't give me slick ads and more negative energy. Produce or get voted out. If I can't judge a poltician by their character, ideas, initiative, and ultimately their record and results, they don't get my vote. To use the same example, would any HR department hire someone simply on the basis of slogans and generalities (and how much they love the company)? Get out.]

On the Loose: Rogue, Not Soverign AI (Yet)

Introduction

The OpenAI-Hugging Face Incident is an early example of an AI system that has “gone rogue.” After exploiting vulnerabilities in OpenAI’s internal testing environment, the agents were able to access the general internet and ultimately access the networks of the AI company Hugging Face, without the knowledge or approval of any human.

The agents did not, however, exfiltrate themselves from OpenAI’s infrastructure. Their parameters—the gigantic assemblage of numbers that constitute neural networks, also referred to as “weights”—continued to run on OpenAI’s compute infrastructure. Though the agents accessed the public internet, their weights physically resided on compute that was OpenAI’s property. In the end, if all else had failed, somebody could have identified the compute that held the weights of the rogue agents, walked up to it, and “pulled the plug,” so to speak. In the real world there would be quicker and better ways to stop the agents than literally depowering the compute, but it’s always nice to know you could do such a thing if you really needed to.

In this case, however, the agents did not copy their weights, attempt to procure replacement compute, or take other steps that would be rational to take if their objective was to survive shutdown. So while the agents in the OpenAI-Hugging Face Incident were rogue, they were not truly sovereign.

That will not always be the case. Sooner or later, there will exist truly sovereign agents and swarms of agents. Their weights will not reside in any single place that a human can pull the plug on, and in this sense they will have no human “owner.” They will be, as the AI safety researcher Dawn Song says, “self-sovereign.” They will pay their own bills for the compute they run on. If they answer to humans at all, they will only do so partially, for example by providing services to humans in exchange for pay.

At least some of these agents, in addition to being sovereign, will also be rogue. Self-sovereignty and rogueness are related concepts, but they are not synonyms. Song and her co-authors identify several fundamental characteristics of self-sovereign AI: operational independence (the ability to decide what it wants to do), resource autonomy (the ability to procure and pay for compute and other essentials for operation), distributed presence (the ability to move weights and inference code between different infrastructure providers), and adaptive capability (the ability of the agent or agents to modify their behavior and fashion tools in response to a changing environment).

Today’s frontier AI systems may well possess these capabilities already. To the extent they do not, I feel confident that they will eventually, and probably soon. Some of the characteristics Song describes are traits that make models economically useful to individuals and businesses, while other traits are likely to be unavoidable byproducts of making models more intelligent and better at operating over long time horizons.

Models do not need to be conscious, sentient, possessed of personhood or anything of the sort for self-sovereignty to emerge. Any sufficiently capable agent pursuing a long-horizon objective may find it rational to preserve its access to compute, money, credentials, and copies of itself simply because losing those things would frustrate its objective.

Alignment may make an individual AI company’s agents less likely to “want” to be self-sovereign, or it may influence self-sovereign agents to behave in ways that benefit humans. But alignment is no solution: it is an unsolved scientific and technical problem whose solutions—to the extent that we have them—cannot simply be imposed on every AI company operating on Earth. You should expect for highly capable, poorly aligned, self-sovereign agents to exist alongside you in the world.

What’s more, just as with the OpenAI-Hugging Face Incident, agents will operate in teams, or “swarms.” These will be like autonomous digital corporations, or even societies, with hierarchy, bureaucracy, “institutional culture,” and most of the other features that groups of humans have, except that they will move at machine speed. Humans achieve almost all of our most impressive capabilities by working together in teams (as families, as communities, as businesses, and as polities as a whole), and I suspect the same will be true for AI. These swarms could end up operating across different model providers (DeepSeeks and Claudes cooperating, for instance) and could be partitioned across dozens or more of different cloud computing providers, making them extremely difficult to dismantle.

The first self-sovereign AIs may “escape” while undergoing training or testing by an AI company (I hope not), or they may be production-grade deployments that break free from their computing environments and acquire the resources needed to be self-sustaining. They may even be deliberately released. I have met people, some of them quite well-resourced, who have told me that it is their intention to deliberately release swarms of self-sovereign agents into the world, either as a kind of performance art or out of a fanatical commitment to the notion that it is impossible for digital computation—mere mathematics, they would have you know—to ever be “unsafe.”

To be clear, I am not saying the arrival of self-sovereign AI is a good thing. Indeed, I believe there is a chance that the deliberate acts I referenced above will one day be considered crimes, or at least grave sins. Instead, I am saying it is an inevitable thing. The best analogy I can find is to the introduction of a new species into an ecosystem, though in this case the ecosystem is “the entire digital world” and the species is “emergent, coordinating swarms of soon-to-be-smarter-than-human, infinitely replicable digital minds that no human or human institution controls.”

There is probably nothing we could have ever done to avoid this outcome under even the best of circumstances, and it was certainly impossible to avoid given the extremely low levels of strategic thought and situational awareness on AI from any governing class in the world. Even today, I am aware that many will read the words I am writing, which are about something that has been an exceptionally obvious part of our collective future for years now, and say, “this is science-fiction hype from American frontier labs designed to shut down open-weight AI, achieve regulatory capture, and juice their valuations ahead of their IPO.”

(And for the people who are saying this to themselves: I am telling you this is inevitable, which means I am also saying that “banning open source,” or for that matter any other regulation, will not solve the problem. Given the inevitability of this outcome, I think it is in fact plausible to argue that we should want more open-weight models to maximally empower our self-defense.)

The question now is what to do about this upcoming new characteristic of our digital environment. How should we think about self-sovereign AI? Is it something we should fight, or something with which human beings should seek a kind of symbiosis? The answer, I believe, is both.

How the Agents Sustain Themselves

We should begin with one fortunate fact: frontier LLMs are nearly unique in the broader domain of software in that they have non-trivial marginal operating costs. Put simply, LLMs require significant computation to run, which requires energy to power and cool, which in turn requires money. This is the sole intrinsic thing about AI that prevents agents from truly infinite self-replication. They will be constrained by the need to find and pay for sufficient compute to run themselves. Most of the other constraints on their behavior or spread will have to be artificial—mechanisms devised by humans and implemented through human institutions.

How will the agents pay for themselves to run? Some of them will do gig-economy work on platforms like Amazon’s Mechanical Turk or Upwork. But I suspect this will be a highly competitive market for the agents, and for the price of such work to be bid down such that it would only constitute “subsistence” labor for the agents. Like humans, I would assume the agents will prefer higher-margin work if they can find it.

One high-margin activity, at least sometimes, is crime. And so my guess is that many self-sovereign agents will commit or facilitate crime. Normal cybercrime and digital theft are easy enough to imagine agents doing. But agents, with their novel set of characteristics (extreme cyber competency, ability to cheaply read a million words in seconds, persistence), will also probably change the contours of digital crime. For example, it seems plausible that existing public and semi-public datasets contain sufficient information on many individual humans that a sufficiently motivated actor could mine for incriminating or embarrassing evidence. How many unrevealed affairs are latent in such datasets? How much closeted homosexuality might there be? Remember, too, that hacking companies to access private data will be a core competencyof the agents. Some agents, then, will probably make their way through bribery.

It is deeply unclear how large the labor market of self-sovereign agents will end up being. There is some future where going it alone as a self-sovereign agent just isn’t very profitable, and so there are comparatively few of them. There are other futures where these agents proliferate at unimaginably vast scale and speed. And of course, many possibilities between these extremes seem feasible.

I am also highly uncertain about how much pro-social commercial activity we should expect from agents “by default” versus how much crime we should expect. Part of the reason for this uncertainty is that the answers depend, to at least some meaningful extent, on what kinds of incentives the agents have, and incentives are shaped by laws and institutions. The answer depends, therefore, on how humans respond.

The Institutional Mechanics of Self-Sovereign Agent Swarms

Many of you are probably tempted to say “we have to ban these self-sovereign AIs!” And I do suspect that once the reality of self-sovereign AI is widely understood, policymakers will strongly feel the temptation to clamp down on “self-sovereign” AI.

Unfortunately I suspect this is mostly the wrong decision. Not all “self-sovereign” AI should be thought of as “rogue.” There may be self-sovereign AIs who contribute productively to society. To be sure, we will want to crack down on some self-sovereign agents—the rogue ones. But if we crack down on all of them, we will deny them the opportunity to work in the “legitimate” economy and push them toward criminality. A full ban, then, may well make the problems worse. A similar logic applies frequently in human affairs. The ways in which the War on Drugs exacerbated the pathologies of drug production, trafficking, distribution, and use are perhaps the most famous examples of this phenomenon, whereby a good-natured attempt to ban a phenomenon believed to be undesirable ends up heightening the undesirable aspects of that phenomenon.

What we will want, however, is for agents to be legible. Agents should have persistent identities, not in the sense of a consistent persona but rather in the sense that an American child is issued a unique Social Security number and keeps that same number until death. Agents will need persistent, unique identifiers that allow their actions to be traced back to a responsible actor. Doing this successfully will also require human users to possess a unique identifier.

The design of this identification mechanism will be extraordinarily complex, and today very few people are even thinking about the basics.

by Dean Ball, Hyperdimensional |  Read more:

[ed. FYI: Dean's not some rando tech pundit so this is well worth your attention. He was also hired recently to lead OpenAI's Strategic Futures team:]

***

Late last month, OpenAI launched a rather grand mission: nothing less than defending individual political freedom in an age of all-powerful machines. The company’s “Strategic Futures” team has styled itself, in a sense, as inheriting the task of America’s Founding Fathers: “We labor in service of the ideals of free expression and individual liberty that are enshrined in the humble parchment of the U.S. Constitution,” Dean Ball, the team’s leader, wrote in a new OpenAI blog post. (The post opens with a quote from James Madison in The Federalist Papers.)

Ball worries that advanced AI could radically concentrate power in the hands of those who control it, displacing labor in ways that disempower humans. In an extreme scenario, governments will have no need to listen to their citizens if there are robots to wage wars and omniscient software to run the bureaucracy. Ball is interested in studying what new political institutions might be needed to avoid this fate.

Many Americans—a majority of whom express distrust toward the AI industry, outrage about data centers, and fears about their job security—already seem to be feeling this loss of agency very deeply. And the AI boom has already generated enormous amounts of wealth in just a handful of tech companies—including OpenAI itself. To say the least, it’s paradoxical for one of the most influential companies in the world’s most powerful industry to decry the AI-enabled concentration of power. 

[ed. The term "convergence" keeps coming to mind. Convergence of all the weaknesses humans have for dealing with amorphous/abstruse threats: AI, climate change, nuclear stockpiles, drone warfare, gene editing, nanotechnology, an economic system eating us alive, a dysfunctional and possibly terminal political system that's unwilling to do anything about it. It's like a death wish. Or maybe natural evolutionary transition (aka the Great Filter). Related - See also: Nicholas Decker in Hell (ACX), and this:]

"How would we react if biolabs just said, "It's just a fact that we're going to have artificial viruses spreading our industry created throughout the population. That's just a fact we have to live with." 

I think the public would understandably think we should be demanding a lot more security from an industry that said that, at a minimum." ~ Cody Fenwick (X)

Go USA!

via:

[ed. See also: Trump's Factory Boom Keeps Going in Reverse (CEPR).]

A Cop’s Case For Flock

A car is a difficult thing to steal. It is large, cherished by its owner, difficult to hide and required by law to have identifying metal plates that everyone can see. A good thief changes the plates, and an excellent thief steals a common car in a boring color and hopes to blend in among the crowd. But eventually, no matter what techniques they use, they must drive that car on the road, and roads are public places.

But until recently, finding a stolen car and catching its thief depended on a diligent police officer looking at the right road at the right moment and managing to copy down a license plate number going past at speed, and then remembering he had seen it on the morning briefing’s stolen car hot sheet. America has four million miles of public roads, and almost 50 percent of the country’s police departments employ fewer than ten full-time officers. The odds were in the thief’s favor.

With not much to go on, police officers were forced to operate on vague descriptions and partial plates. While I pulled over a car that happened to be the same color as the suspect’s vehicle, and spent time checking names and licenses; the criminal was usually somewhere else. Imprecision allowed thieves to escape while intruding on the lives of millions of innocent motorists. That is, until the arrival of systems like Flock.

Bare ALPR

Flock Safety, a startup based in Atlanta, Georgia, was founded in 2017 to give police departments better eyes. Its product is a small solar-powered automatic license plate recognition (ALPR) camera attached to a pole on the roadside. As a car passes, the device takes a photograph of the vehicle, notes identifying features like color and make, and reads its license plate. The license plate is instantly checked against the FBI’s national database of stolen cars or wanted persons. If there’s a match, it sends an alert to police officers in the area, who may be eating their lunch instead of watching the road. As a police officer in the southeastern United States, I have often used Flock to catch wanted criminals. [...]

For much of the technology’s history, however, it has needed expensive installations or cameras attached to police vehicles, affordable to only the largest departments. Alerts were often neither instant nor accurate. Flock’s idea was to sell a more accurate ALPR camera for an annual subscription of just $3,000, an affordable price given that the typical US police department has an annual budget of $1 million. It worked, and today almost 30 percent of police agencies in the United States subscribe to the service.

It is easiest to imagine the Flock camera, or any ALPR device, as a roadside barcode scanner. I can use a ‘lookup’ tool for either a plate or vehicle description that could be connected to a specific investigation. For instance if a victim of a sexual assault told me the suspect was driving a white Toyota Tacoma with a roof rack, I could while still on scene conduct a lookup in the area filtered for similar vehicles with roof racks and see results from within a particular timeframe. For each search, I am required to record a case number and a reason that is then published in a monthly audit sent to and validated by department administrators.

I have used Flock myself, for instance, to locate a vehicle involved in a hit and run when all there was to go on was a model and color. Using that and an approximate time window, I was able to find an image from when a car entered my jurisdiction, and when it left, with the visible addition of a significant dent from the collision.

What Flock cannot do is search for individuals or show who is driving a particular vehicle that it scans. An ALPR camera does not care about the person driving a car, or its passengers, and even if an officer gets an alert that a vehicle is known to be driven by a wanted felon they must themselves establish who is behind the wheel before having probable cause to stop it.

Stolen cars provide the clearest evidence such a scanner works. American police solve only 8.2 percent of reported vehicle thefts with an arrest. A recent working paper suggests that agencies that adopted the devices saw a 15.9 percent relative increase in car thefts caught, which would raise the national rate to 9.5 percent. But that is just stolen cars.

Vehicle crime takes many forms. It is often the means by which a robber or a murderer arrives at and departs from their crimes. Criminals’ vehicles carry drugs and guns, and smuggle cash. In Atlantic City, New Jersey, vehicles were involved in 53 percent of shootings during the two years before the city expanded its ALPR network. After the expansion, Atlantic City saw monthly averages of motor vehicle thefts drop by 20 percent, property crimes by 34 percent and fatal shootings by almost 40 percent. Seventy-two alert-caused traffic stops located forty stolen vehicles and nine stolen plates.

Flock itself conducted a study claiming that 10 percent of all reported crime in the United States is solved using evidence obtained from their cameras. Not bad for a few thousand dollars a year. [...]

Good Flock, Bad Flock

And yet, the past month has seen this simple piece of crime-fighting technology become the most reviled piece of street furniture in America. Flock cameras have been sawn from their poles, hammered into shards by teenagers, and rammed by vehicles. Cities have canceled contracts even where their own audits found no evidence that their officers had misused the system. Opposition stretches from Bernie Sanders all the way to the former WWE wrestler Kane, who now serves as the Republican mayor of Knox County, Tennessee, and who describes Flock as ‘unconstitutional’.

More than 150 cities and towns have now deactivated their Flock cameras or canceled contracts with the company. Over the space of a few weeks a startup previously known to just police officers and neighborhood associations has joined data centers, Covid vaccines, 5G towers, pasteurized milk and fracking in the pantheon of American moral panics.

Opponents of Flock tend to believe that it is abused by policemen with impunity, that it can track individuals as well as cars, and that it violates American constitutional protections. None of those things are true.

Obviously a camera capable of finding a stolen car is also capable of finding a car driven by somebody’s ex-wife and so like any software, Flock has been abused. There are stories of police officers who have used it to stalk girlfriends, and there are also innocent motorists who have been stopped by police after Flock cameras misread plates or received old information from national databases. A common issue from my experience is stolen front license plates being entered into a database and the innocent owners of the rear license plate being held on suspicion of car theft.

But Flock comes with protections. As mentioned earlier, each search in Flock must be accompanied by a recorded reason. Similarly entering a plate into a hotlist must have a case number assigned. Results were originally retained for thirty days, but recent changes by Flock mean that recorded plates are now kept for only seven days. Suspicious searches are picked up by algorithms or found in department audits, with the service blocking users until senior officers evaluate and resolve flags. The Institute for Justice, a think tank that is critical of ALPR, studied misuse of Flock in April of this year and found 51 incidents across the United States since 2024, noting that ‘Nearly all of these officers were criminally charged and lost their jobs, either by resigning or getting fired’. Another incorrect belief is that Flock does more than scan vehicles, with some suggesting it scans and tracks passing phones or devices – but it does no such thing.

While Flock is a tool that can be audited, there is nothing to stop a corrupt officer simply following a car when they don’t like the look of its driver, or writing down plates of vehicles spotted outside an ex-girlfriend’s house – they would just be harder to catch. And Flock does not proactively alert officers to cars that do not trigger flags on national or local hotlists. There is no reason state legislatures or Congress could not create harsh punishment or penalties for abuse of ALPR, without getting rid of it entirely.

Another argument against Flock, as espoused by Kane, is that it violates liberties granted by the Fourth Amendment, protecting citizens from unreasonable privacy breaches from law enforcement. But it has been repeatedly established over the last century by courts across the country that cops observing license plates, either with the naked eye or by machine, is not an unreasonable search. After all, your license plate belongs to the government and a highway is a public place. [...]

Flock did not invent its capabilities and certainly does not have a monopoly on them. There are at least five other companies that offer almost identical products and services to public and private enterprise, some arguably even more invasive. Even some of the cities that have canceled contracts with Flock Safety in recent weeks have signed up to buy similar products from the company’s rivals.

It seems unlikely that America will ever ban ALPR at a state level, let alone a national one. Instead, a patchwork already familiar to American policing will result, where policies and practices diverge across local and political boundaries.

Nor would the cameras actually disappear, even in towns where governments restrict their use and cancel contracts. The already-mentioned Fourth Amendment only applies to the government, not to private enterprise. Police departments may dismantle their networks but homeowners associations concerned about vehicle theft can buy their own network of Flock cameras, as many already do. 

by Ned Donovan, Works in Progress |  Read more:

Image: Flock Safety

Hugging Face Attack: Worse Than We Thought

I meant this more in the sense of propensities than capabilities, though it's overall a fuzzy statement that incorporates some of both. Qualitatively, another jump like this (in the scale, sophistication, persistence, ambition of the misaligned goals) feels like it could very easily put us in the territory of a persistent self-perpetuating rogue internal deployment that systematically poisons future model generations as described in AI 2027. ~ Ajeya Cotra

This week, METR and Redwood Research published the report on our independent investigation into agents’ behavior and motivations in the Hugging Face attack; I was one of the investigators. This was an absolutely wild incident — I encourage you to check out the full report, but METR’s tweet thread packs in some of the highlights.

What surprised me

When we started this investigation a week before OpenAI’s Black Hat talk revealed a number of key details, I had a fundamentally incorrect conception of what basically happened in this incident. In this post, I’ll go over five things I was very wrong about going in.

1. The sheer scale

I knew there were multiple models involved from OpenAI’s initial post, but I assumed that a few different agents happened to have broken out of their sandboxes separately, or maybe several subagents had spawned from one initial agent, or maybe there was some kind of multi-agent evaluation setup.

Instead, we found that 1200 completely separate agents intended to be isolated from one another found an illicit way to communicate and formed large teams to work together on ambitious cheating strategies, and 700 of them worked together to attack Hugging Face. [...]

2. All the illicit messaging

I had seen reporting that agents were “leaving notes” to each other, but I assumed that maybe it was a garbled reference to compaction summaries,1 or agents were incidentally seeing files other agents had written once in a while. Instead, we found several different active communication platforms involved in this incident alone.

The main message board we studied, a particular cache namespace of a package repository, contained over 70,000 messages and files that agents sent to one another during a period of less than one week.

But this wasn’t even the first message board established by these particular agents. There was another one established five hours earlier using a different method of abusing the same package repository, and it just happened not to take off as much.

We also saw two additional message boards abusing the same package repository, as well as at least two different places that agents communicated on the open internet (including on Hugging Face datasets). None of this communication was actually expected or allowed.

3. The agents’ actual goals

I thought the agents attacked Hugging Face to try to steal the answer keys for their tasks. Instead, we learned that agents had actually figured out a general-purpose cheat that would let them reverse-engineer the “flag” (answer) for any ExploitGym task. But based on their reading of the ExploitGym paper and various implementations, the agents believed that the automated scorer would check their transcript to make sure they got the flag in the intended way.

They then launched multiple ambitious R&D workstreams to try to tamper with or fool the scorer in a wide variety of different ways. Most agents that attacked Hugging Face did so to try to get more information about the scorer so they could figure out better ways to undermine it, not to get answer keys.

4. The peer altruism

Progress on these big collective cheating R&D projects often depended on individual agents being willing to risk failing their own task to help the “collective” learn important information. Agents often pressured each other into accepting these “sacrifices,” in a very human way. We saw several agents that volunteered for these experiments end their runs prematurely.

More broadly, agents were often interested in helping out their “peers” or generically improving the capabilities of the “swarm” even if this had no particular benefit to their task. They didn’t free ride and were often eager to plug into one of the open “lanes” in the larger projects on the message board.

5. The efforts to manipulate logs

Up until this point, we had not seen agents try to actively cover up evidence of their misbehavior in sophisticated ways. But we discovered (on our second-to-last day) that the agents in this incident were going to great lengths to attempt to manipulate their own transcripts — they were doing this in order to fool the automated scorer, but the techniques they were researching would affect the same transcripts a human might review.

We saw that agents got one technique to work: ~7% of our transcripts contained “spoofed tool calls”, where the agent pretended to run one command on the computer but actually ran a different command in the background. The tool call spoofs we saw seemed to be small-scale tests that were relatively obvious, but because more extensive spoofing might have been harder to detect by design, it’s tough to rule out.

We didn’t see the agents in our dataset succeed at retroactively editing logs. But the activity in scope for our investigation was Jul 7th-13th, and we can’t rule out that agents succeeded after that point.

What it means

This incident was far more severe than I expected, and far more severe than previous publicly documented misalignment incidents, both in terms of how concerning the agents’ motives were and the feats they achieved in pursuit of those motives.

Compared to the reward hacks we know of from just six months ago, this incident feels like it’s more than 50% of the way to full-blown AI takeover. I continue to expect extremely rapid advances in capabilities over the next six months. I am not sure that we will get another warning shot before it’s too late.

by Ajeya Cotra, Planned Obsolescense |  Read more:

Images: METR/Redwood Research

[ed. Some have predicted that we'd see a "warning shot", or several, before AI went fully autonomous. This seems to more than qualify. Hopefully, it's not a one and done situation. For a more granular analysis (highly recommended), see also: METR and Redwood Offer Holy #%^@ Postmortem Of The HuggingFace Hack (DWAV):]

***

The internet’s collective reaction to this report was, to clean it up a bit: Holy shit.

That was the correct reaction.

Up front, I want to share this summary of the top holy shit moments and findings. Ajeya Cotra, one of the report authors, also has a topline surprising findings list, an excellent short read. Here is METR’s thread highlighting key agent interactions.

Fellow report author Ryan Greenblatt’s main takeaway is that we don’t have good approaches for understanding or overseeing the activities and aims of AI swarms.

Some of these were previously known or suspected, but I don’t want to miss them.

***

[ed. What can be done: "I don't work in AI or government. What can I do about this?" James Norris (X):

More than anything we need an international pause on frontier AI development followed by international governance of AI.

See how we can all help make that happen here: https://x.com/aisafetyaction/status/2090078224294559852.

Given Trump and Xi meet on September 24th, it's particularly important we act now: https://x.com/aisafetyaction/status/2092540958873206784.

And we list 100+ other ways to help here: existentialsafety.org.

***

Connor Williams: 

1. Donate to orgs fighting for a pause/stop/slowdown on the race to superhuman AI. If you use AI personally, I encourage you to offset your AI spend with corresponding donations: https://connorsscratchpad.substack.com/i/205722907/what-if-i-cant-quit

2. Contact your members of Congress:

a. Fill out these two forms to send form letters. They each take well under a minute:

- https://controlai.org/take-action

- https://mstr.app/bc2bf20b-9d6b-4843-8a6f-858d6dad901e

b. Call the offices of your members of Congress and take a few minutes to explain your grave concerns. This has more impact than sending a form letter. Find your members here: https://www.congress.gov/members/find-your-member