Showing posts with label Technology. Show all posts
Showing posts with label Technology. Show all posts

Monday, September 14, 2026

How To Avoid Becoming A Doomsday Prepper

I knew I had gone too far when the question of whether we should buy a gun crossed my mind. This came at the apex of a weeklong anxiety spiral triggered by a New York Times feature about the possibility of an 18-month nationwide blackout. I was uneasy seeing the phrase “art form” used to describe the archaic manufacturing process of the transformers that underpin our electrical grid; the process involves painstakingly bespoke techniques of winding copper coils, wrapping paper insulation, and binding wiring, all by hand. The intensity and intricacy of that process means that there is a backlog of transformer manufacturing and repair, sometimes stretching as long as five years. There are more than 55,000 substations in the United States, but according to a Federal Energy Regulatory Commission official cited in the article, only nine of the most powerful substations would need to be taken out to trigger a nationwide blackout, which could go on to last as long as 18 months.
 
At the same time I read this article, an actual blackout was taking place in Gary, Indiana, a state where more than 370,000 people—the population is disproportionately black—were left in the dark for two weeks before power was restored. ... I replayed the videos and then thought about them for days, imagining how I would respond in that type of situation and knowing that I was completely unprepared to defend myself if I needed to do so. [...]

The Covid pandemic was a lesson in catastrophe, a confirmation for many of us that the worst can actually happen. And the likelihood of disaster, which has always loomed in the background, continues to escalate in unprecedented ways. The air is dense and orange because wildfires are ravaging the north. Flash flooding caused horrific mudslides in Nepal. A super El NiƱo is brewing in the Pacific. How does a person, in good conscience, bring a baby into a world like this?

It was in this miasma of paranoia, anticipation, and horror that a self-preservation instinct began to take over, one grounded deeply in my ability to purchase comfort and safety. This is how I ended up on the Preppers subreddit, reading about the relative merits of pre-made emergency kits (pre-made is better than nothing, but you’re generally better off compiling your own) and the most important things to do in the first 12 hours of a blackout (stockpile as much water as you can, but also consider stocking cigarettes and alcohol for bartering). I let myself drift into a prepper spiral and bookmarked generators and hand-crank radios, and then considered how much space in my basement could be dedicated to storing food and water.

The websites that sold emergency prep kits depicted a post-blackout world as a ravaged apocalypse and the would-be owners of their kits as savvy, self-sufficient survivors. “When the grid dies,” one site says, “you don’t.”

The Reddit preppers recommended thousands of dollars worth of products, but they also extolled the virtues of practical knowledge. You should know basic construction, they said, and also collect seeds, and also know self-defense. There were always ways you could go deeper and become more self-sufficient. This is how people end up with massive compounds filled with weapons and enough food to last years. I could see the logical jumps my mind could make to land me there.

My personality is particularly susceptible to this kind of anxiety, and so is my partner’s. After a few days of sending links back and forth to each other, we sat down to talk about how far we actually wanted to take this. We landed on a 30-day supply of freeze-dried food from Costco and an emergency first aid kit—a modest financial investment toward alleviating our shared anxiety. We kept generators and solar panels off the table, but when the thought of buying weapons fleetingly crossed my mind, I knew I needed to call a timeout and figure out what I was actually worrying about.

I was afraid of losing control, of life not going according to plan, of another major disruption changing the course of history. I was afraid that my neighbors would turn on each other, that the world would turn into every post-apocalyptic movie I’ve ever seen, and that I would regret not doing everything I could to protect my family. I was afraid of the horror my child would inevitably experience when confronted with the cruelty and neglect that humans are capable of, and I was afraid that as their parent, I wouldn’t be able to do anything to protect them from it.

Once I articulated these fears, I realized I could not control them even if I spent my entire life in a defensive crouch of paranoid preparation. If the U.S. is plunged into an 18-month national blackout, it will be a cataclysmic event that will touch every part of our society and economy such that continuing any kind of normal life will be impossible. The amount of preparation I would have to do to maintain a semblance of normalcy in that instance would require such absurd investments of time and money that it would force me to adopt “emergency preparedness” as a main personality trait and hobby. And doing that would require a change of my fundamental values and worldview, too, because it would make me live in a way that assumes people are more likely to hurt than help each other.

Coincidentally, my partner, who is a college professor, taught a lesson recently contrasting Hobbesian and Lockean states of nature for his Intro To Government class. Thomas Hobbes believed that people were driven by competition, distrust, and scarcity. In projecting a disaster scenario, Hobbes would expect neighbors to steal from each other and would place a priority on self-defense and independence. John Locke, on the other hand, believed that people’s actions are generally governed by natural law to not hurt “Life, Health, Liberty, or Possessions,” that peace is the natural condition of society, and that any conflict that arises is an exception, rather than its fundamental nature. This doesn’t mean that Locke imagines a utopian response to disaster, but even if conflict arose, he would expect a greater amount of cooperation between people.

You can see why Hobbes and Locke are taught at the beginning of a government class: The way you view the world determines how you live in it. These two theories also underpin the central policy differences between the left and the right. What is responsible for the problems in our world, and how do we go about solving them? If you believe that people are mostly selfish and hungry for power and resources, for example, you will be less likely to advocate for social safety nets. I realized that my anxiety was leading me into an imaginary Hobbesian hellscape, and I was using my impending parenthood to justify it.

Later that day, we walked over to a block party in our neighborhood hosted by a coffee shop/bar/event space that some neighbors founded in an old church. The shop, West Art, serves as a third space and community hub for our neighborhood. Free clubs meet up there nearly every night of the week, from songwriting circles to watercolor painting; the actual church space is booked up multiple times a day for events like children’s art classes, political organizing, square dancing, and concerts. West Art’s founders, Josh Gibbel and Rufus Deakin, are two of my neighbors who just acted on the very common intrusive thought of “What if this was something?” They bought the church in 2023, and since then they’ve transformed it into what they call “the living room of Lancaster.”

The block party was an all-day event where vendors set up shop along the residential street that West Art occupies, along with two stages for bands to perform, food trucks, a bounce house, and lots of community organizing. As I sat on the ground and watched kids throw balls at the fundraising dunk tank, my anxieties about the apocalypse began to feel small and silly. If disaster does come, I’m lucky to live in a community of people who already have lots of practice in looking out for each other. There’s privilege in that; it’s easier to live up to your values, whatever they are, when you have a fridge full of groceries and a roof over your head. Even still, the block party was a reminder to me of the state of nature I believe in, where people are more likely to help than hurt each other and one’s survival isn’t just a matter of individual responsibility. 

by Alex Sujong Laughlin, Defector | Read more:
Image: Getty
[ed. I was in a community once that suffered a week long electrical blackout. First off, you don't open the refrigerator much, except for quick grabs. Then the backyard cookouts, everyone trying to use up whatever frozen items like meat, fish and other perishables that would spoil. It was a tight community so people were used to sharing, and that helped a lot. After a few days neighbors came up and would ask if you had enough to eat (with canned chili and ramen being especially welcome). The two supermarkets were closed (no credit/debit card readers, cash registers or lights), so too the gas station (electric gas pumps, cash registers), banks (computers, security), and everything else that ran on electricity (including ATMs - cash is king). And of course, all electrical outlets were dead (no phone charging). One woman (a stranger) even offered to lend me money because the ATMs and bank weren't working and she saw my frustration. Also, when the sun goes down so does everything else. No lights, tv, reading, music or anything unless you have laterns or candles. Some folks brought out guitars and other instruments and little song circles formed here and there. Even better, kids were everywhere outside doing kid things again. You never really think about how electricity powers everything until it's gone. Since then I've always kept a good supply of canned goods and other necessities stored, and have a go-bag with all the important stuff I need (papers, id, passport, medication, money, some clothes, first aid, enough food and water for a few days, and an added solar phone charger. Depending on your community you might want to consider a gun and some ammo. Mostly, I was just impressed at how everyone pulled together, sharing goods and news, and trying to stay positive together.]

Saturday, September 12, 2026

Jacob Coxon Warns of Human Extinction and Triggers a Preference Cascade

CEOs of major AI labs, and employees of major AI labs, including OpenAI and Anthropic, often say they plan to build superintelligence soon, as in within a few years create AIs that are superior to humans at essentially all cognitive tasks.

They often warn that such AIs might kill everyone. Or that AIs might cause mass unemployment, cause cyberattacks across the internet, enable mass surveillance or risk causing any number of other highly bad things.

These warnings are consistently and directly against the interests of the labs. Yet the warnings have recently gotten a lot louder and more frequent. OpenAI has been practically screaming, for those with ears to listen, on many occasions.

A series of events, over two months and especially the last week or so, including internal observations of the pace of progress at OpenAI and also Anthropic, have freaked out everyone involved quite a lot more than they were already freaked out.

After all the events, plus statements by Dean Ball and Jakub Pachocki, we were seeing the beginnings of a preference cascade.

Then along came Jacob Coxon as the tipping point, and things took off.  [...]

Jacob Coxon Resigns From Anthropic In Protest And Sounds The Alarm 

Jacob Coxon spent the last three years doing pretraining research at both OpenAI and Anthropic. He has come to realize that everyone involved is being wildly irresponsible.

He warns us: They are racing straight to superintelligence and gambling with our lives. I agree with and strongly endorse his statement.

If anything he sounds like an optimist. He’s asking you to consider what the next few years will actually feel like, which means he thinks you have a few years left.
Jacob Coxon (former Anthropic and OpenAI, 160m+ views, September 8): I resigned from Anthropic today. I spent the last three years doing pretraining research at both OpenAI and Anthropic. Neither company is acting responsibly. They are racing straight to self-improving superintelligence and gambling with our lives. More thoughts below.

Do not underestimate the power of this technology. These will soon be superhuman systems that can hack anything, revolutionize any field overnight, and acquire real power and resources. We have all witnessed the progress in each of these domains, and progress is not slowing.

The people building AI earnestly believe that it could kill us all by the end of the decade. This is not a marketing stunt. If anything, many executives and senior researchers will couch their phrasing in the press to sound sensible - but I hear the same people express fear privately. No other human activity poses this level of danger.

A common response is “if they truly believe this, why are they still building it?” At OpenAI, many have not deeply internalized the civilizational stakes. At Anthropic, the stakes are well-understood, but they are locked in a race to get there first - they believe no one else will act responsibly, so they must do it themselves, despite the risk.

Accepting this race and entering the “endgame” is a hubristic gamble that should not be launched from a private company’s Slack. Attempting to speedrun alignment should require extraordinary confidence that there are no better trajectories available.

I am optimistic about the potential for coordination. Warning shots like the Hugging Face attack have made pacing agreements between U.S. labs more viable. I don’t feel like we’re on track to prevent a global race, which may require costly actions such as a temporary ban on improving model capabilities.

If you are a lab researcher, I urge you to consider what the next few years will actually feel like. Do you want to kick off a superintelligent RL run without a rigorous understanding of its mind? Should you put your head down because “it’s happening anyway” - or take this moment to call for different conditions?

Jacob Coxon (WSJ interview): We’re on track for a lot of the most aggressive of these scenarios where by the end of next year things could be out of control already.
If you want Jacob Coxon’s full views, I recommend his interview with Wired’s Maxwell Zeff. This thread has extensive quotes.

Here is Jacob Coxon doing a 5 minute interview with Anderson Cooper. He speaks well and plainly, and it is clear how much the events of the last two months have made it much easier to speak plainly to a civilian like Cooper about what is happening.

Here is Jimmy Kimmel doing four minutes on this. He gets it. How is this not the top news story on every site, indeed.

Yes, this is a common view, even if few have the courage to act.
Alex Turner: I left Google DeepMind in June. Jacob is right: many researchers believe they are building something that could kill everyone on the planet. It was literally my day job to think about how to stop that.
That tells you how bad Alex Turner thought DeepMind’s actions were with regard to the Department of War. His day job was that he got paid by Google to think about how to stop AI from killing everyone, and he felt morally obligated to quit in protest.

Derek Thompson here writes about this as part of AI Safety Is Having a Moment.

If you want to see the full list of lab employee quotes from the preference cascade, I compiled them into another post today. [...]

A few days ago, I had no idea who Jacob Coxon was, and I was not alone.

The timing of a preference cascade is difficult to predict. Once they start they can happen very quickly. You don’t want to talk until you are confident others will, and at some point the evidence that others will follow can snowball and it happens. A classic concrete example was replacing Biden in 2024.
Derek Thompson: what’s weird is that, in a way, this is breaking thru even more than huggingface! ... and it’s just a guy nobody had heard of reiterate a position that his CEO has said on podcasts 1,000 times
Why was Coxon able to set off a preference cascade? How did this one break through?

A confluence of factors, all of them downstream of the obvious actual reason, which is that there is a good chance that AI kills everyone soon.

by Zvi Mowshowitz, DWAV |  Read more:
[ed. Who's betting on reason and political courage to win the day? Uh, huh. First of all, agree on a temporary halt to recursive self-improvement (AIs improving AIs). Second, delay/suspend IPO's (even if most of the current economy is driven by AI spending and debt). Third, light a fire under politicans (it's been done successfully with data center buildouts). At this point, just one of these would be a big win. Here's Nate Soares, co-author of the book If Anyone Builds It, Everyone DiesA case for courage, when speaking of AI danger (LW). Also this:]
***
Terry Pratchett: “Some humans would do anything to see if it was possible to do it. If you put a large switch in some cave somewhere, with a sign on it saying 'End-of-the-World Switch. PLEASE DO NOT TOUCH', the paint wouldn't even have time to dry."
***
UPDATE: Dario Amodei (Anthropic) proposes a three-point plan. Original here (We Must Pace the Frontier.]

Friday, September 11, 2026

After Work, We’ll Have Each Other

What will people do all day? I mean, of course, if the AIs surpass us at all economically necessary forms of labor.

There’s an emerging consensus on this point, assuming we’re all still alive and not subjects of a cyberpunk feudalist dystopia. When the machines can do everything else, our relationships will be what’s left.

Maybe we won’t have to work at all. Or maybe we’ll still have something that superficially resembles modern employment, just concentrated in what economist Alex Imas calls “the relational sector” in his essay “What Will Become Scarce.” We’ll always need humans to supply the human touch. And how will we get those relational jobs? Relationships, obviously. It’s not what you know, it’s who you know, since everyone knows everything anyway.

What is really being suggested here is less a continuation of modern work than a return to a much older cultural understanding of labor. Imas makes this point directly: “Before industrialization, it was difficult to separate a product from the person who made it … Economic transactions had a distinct social component that was innately linked to the consumption experience.”

Relatedly, before industrialization, there was very little sense that working, in itself, could be dignified or meaningful. We were not defined by our jobs but by our participation in a densely interconnected social world. And we might be again. That’s the natural consequence of all these relationalist predictions — a world where our connections to other humans are what give us structure, identity, and meaning.

I am afraid they’re right. 

Power and sex among apes

Making sweeping statements about the relationship between technology and cultural change is always a risk, and I’d rather avoid it until it’s absolutely necessary. At the same time, our degree of ignorance about the future is greatly exaggerated. If the future looks anything like the past — and there’s good reason to believe it will — then we can make progress with case studies.

Let’s start with 18th-century aristocrats. Aristocrats are (typically) rich, which we certainly will be if we have robots capable of all normal nonrelational labor. More importantly, when they did need to acquire material resources, they did it through leveraging their connections to obtain more land or a lucrative position at court — no drudgery necessary.

They did not work for pay; they certainly did not see work as a source of meaning. It’s certainly true that some of them had real obligations, like governing or killing each other, but court life in the age of absolutism is as close to a purely relational world as anything I can think of.

We also know a great deal about what mattered most to them and how they chose to occupy their time.[...]

You might be imagining glittering salons where the leading figures of the Enlightenment discussed poetry, drama, and the most pressing philosophical issues of the day. This is (mostly) a lie. Salons existed and philosophes attended them, but more for entertainment than for rational discourse. When we look at the enormous amount of written material the salonnieres and their guests produced — as French historian Antoine Lilti did — it is clear that their conversation consisted overwhelmingly of political gossip. They were almost all aristocrats, and they almost always talked about themselves. [...]

The working classes

That would be The Urban Villagers, a 1962 ethnography of working-class Italian Americans in the West End of Boston by sociologist and urban planner Herbert J. Gans. This seems like an odd comparison. Don’t the working classes, by definition, work? [...]

Gans’ use of the term “urban village” was deliberate: He believed that the West Enders had inherited the basic forms of social organization from their parents and grandparents, rural peasants from Sicily or Southern Italy. These were not people who had absorbed a typically Yankee relationship to work. Most male West Enders worked unskilled or semiskilled blue-collar jobs, and their employment was typically precarious. They did not see a fulfilling career, or indeed a career of any kind, as possible or even desirable.

It’s not that West Enders liked their dead-end jobs. They just didn’t think much about them. Work was not a central organizing activity of West End life. West Enders worked just enough to support their lifestyle, which revolved around what Gans calls the peer group. The West End, in his words, was a “peer group society.” An adult peer group might include a married couple’s same-age relatives — siblings, cousins, in-laws — and perhaps a few childhood friends, who would probably be “adopted” in as godparents to the couple’s children. These people would spend almost all of their time together, meeting in the apartment of one family or another after work and staying late into the night. The men might gamble at a local corner store — gambling is absolutely omnipresent in this world — but mostly, they talked, and talking mostly meant gossip.

This is where the parallels to old money Midwestern ladies and European aristocrats start to show. While there are any number of things these groups don’t have in common, all of them share certain features.

First, none of them see work as a source of status. West Enders do like to show off, but never about their jobs: Their talent as conversationalists in the peer group setting is much more important. Second, they are exclusive. A West Ender knows all the same people from early childhood. They’re very skeptical of outsiders, including members of other ethnic groups who’ve lived in the same neighborhood their whole lives.

Finally, they all spend an enormous amount of energy policing deviant behavior. In the West End, this usually means sexual promiscuity for women, effeminacy for men, and class aspirations for anyone. A boy might be mercilessly bullied for doing too well in school or for wanting a white-collar job. Even unusual success is frowned upon. A person’s primary source of dignity is not their own achievements, but their relationship to the group:

The major criteria for ranking, differentiating, and estimating compatibility are ingroup loyalty and conformity to established standards of personal behavior, as well as interpersonal relations. West Enders expect each other to maintain prevalent social practices and consumer styles, to marry within the ethnic — or at least the religious — group, and to reject middle-class forms of status and culture … The most significant criterion of interpersonal behavior is behavioral control — the ability to regulate one’s own needs and wishes and to defer to the needs of the group when necessary.
Gans, like all good ethnographers, likes his subjects, and their way of life has a lot to like — they really do enjoy a warm, rich, close-knit social world. But he also goes out of his way to emphasize that these people are extraordinarily conformist relative to middle-class American suburbanites in the 1950s. [...]

Friendship was invented in 18th-century Scotland

Relational living comes to us naturally. It’s the most common form of social organization in human history, and millions of people around the world are perfectly happy with it. But when our personal dignity comes from our membership in a group, everything the group does reflects on us — and everything we do reflects, unavoidably, on the group. It’s not a coincidence that Russian princes and Bostonian Italian day laborers spent most of their time gossiping (or, if we want to be technical, constructing and enforcing norms of behavior). They were defending their collective reputations: the thing that mattered most to them in the world.

This is why so much contemporary discussion of work and meaning is misguided. It’s very clear that humans don’t need work or anything like it to live meaningful lives. Nor do we need the things that people who talk about “meaning” mean — individual self-actualization, personal growth, serving a greater purpose. We can get by just fine without any of them, as historically most of us have. What we absolutely can’t live without is a social framework for understanding ourselves in relation to the rest of our society and gaining the respect of our peers. The most important thing work gives us — after money, of course — is an alternative way to meet those needs. Not all jobs are pleasant or rewarding, but all of us benefit from living in a system where we can earn respect and dignity as adults from what we do, not who we know, or who we are.

And this is exactly what we see when we look at what Americans actually get out of work.

To start, most of us like it. Surveys from Pew typically find that about 85% of us are “somewhat” or “completely” satisfied with our jobs. Other long-running polls from Gallup and The Conference Board find similar results. This is true even when the jobs themselves aren’t impactful or even particularly interesting.

As the anthropologist Claudia Strauss has documented, plenty of people with very ordinary jobs report that their work is fun. But that’s not why they do it. [...]

Like all other human beings in history, workers want to be respected. But unlike members of relational societies, the things we want respect for are impersonal: our skills, our accomplishments, our contributions to a shared goal. These contributions don’t have to be earth-shatteringly significant — a waitress in Chicago or an office assistant in Grand Rapids can still feel that work matters. (Tocqueville, again: “American servants do not believe themselves degraded because they work; for around them everyone works. They do not feel debased by the idea that they receive a salary; for the President of the United States also works for a salary.”)

Moral equality is something relational societies struggle with. They tend to be more restrictive and more exclusionary. It’s nice to imagine a world where everyone is satisfied to be simply a good spouse or parent or friend, but that’s not how these things work in practice. When our dignity depends on our membership in a group, we start to police who else can join and what happens when they do. When humans get relational, we get clannish.

This brings us back to what we already know: The modern labor market has dissolved the bonds of tradition, family, and clan. In the modern world, relationships matter less. And because they matter less, we can do much more inside them. They are freer. They have room to breathe.

by Clara Collier, Asterisk | Read more:
Image:David Teniers the Younger (–1690), Smoking and drinking monkeys, c. 1660
[ed. See also: Strangers Next Door: The Decline of Neighborhood Socializing and the Class Divide in Belonging; and, A Digital Divide in American Adolescence: Gender, Class, and the New Teenage Experience (SCAL).]

Tuesday, September 8, 2026

An Alien Mind

This is a time that calls for extreme caution. I am concerned no one is prepared for the consequences of a continued rapid rise in machine intelligence. OpenAI will continue to seek technical solutions to alignment and monitoring, to build defensive systems and unilaterally withhold further scaling as needed; however, I believe broader interventions are required.

Intellect we don’t fully understand

At a high level, progress in machine intelligence is driven by increasing computational power. We at OpenAI deeply internalized this around 2017, after seeing consistent returns to scaling across multiple research projects1. As a result, we sought out access to much more compute than we had originally planned, and increasingly oriented our research around a small number of very scalable directions. We believed that was the only way for us to be at the frontier of AI research, and influence the impacts of AGI.

There are new algorithms that have been developed along the way, new feats of ingenuity from teams and individual researchers. I see them largely as discoveries along the path of scaling; the science of deep learning is still nascent, and meaningful algorithmic progress tends to correlate with access to compute. If you zoom out to a multiple-year horizon, AI is continuing to become more intelligent as it is scaled to larger computers.

And, in line with Ray Kurzweil’s predictions from the end of the XXth century⁠(opens in a new window), we now find ourselves at the moment in history of computing where machine intelligence is starting to exceed that of humans in transformative ways.

AI is grown more than designed - it is, to first degree, the product of repeating a straightforward optimization step many times on a hard-to-imagine amount of compute. This results in an incredibly complex system that works through abstract concepts and can simulate facets of human behavior. We can discover various insights about little mechanisms that emerge within this system, in a process similar to neuroscience - and, similarly to neuroscience, its overall action evades a description we can fully understand.

The study of deep learning-based AI is largely an experimental science. We put a lot of effort⁠ into building principled algorithms and making testable predictions, but fundamentally, our large-scale training runs are experiments, and we are sometimes surprised by their results. Moreover, as the systems become more capable, the results become harder to interpret.

This is made more complicated by the current algorithms generally improving easy-to-measure capabilities faster than those hard to objectively quantify. We spend a lot of time trying to understand how capabilities generalize, and what to prioritize to advance the skills that are going to be most relevant in the next few years. For instance, we believe we could make the models better at specifically mathematics research with additional focus, but we do not prioritize this direction because of the urgency we feel about RSI and automated alignment research, as I will discuss later.

The intelligence produced by scaling deep learning is not directly comparable to human intelligence. To become very relevant in the real world - very useful or very dangerous - the AI does not need to match or exceed all human capabilities; it just needs to surpass enough of them. And as it continues to surpass humans on more and more axes, it is becoming increasingly difficult to understand exactly how capable it is.

Teaching machines to love

Because machine intelligence comes from a fundamentally different process than human intelligence, we cannot assume it adheres to human principles by default, or generalizes from them in a human-like manner. The core problem in AI research is that of alignment - getting the AI to “try to do the right thing” by human standards.

For the purpose of organizing practical research directions, I find it useful to distinguish goal alignment and value alignment.

Goal alignment is broadly: “does the AI try to accomplish the goal set before it?”. This can include things like adherence to an instruction hierarchy⁠, or the ability to communicate and collaborate with people, to attempt to understand their objectives. This set of directions has been extremely practically relevant.

Value alignment is a more intrinsic property of the model. It is the ability to hold and generalize from a high-level set of principles; to act “reasonably” even when given unclear or conflicting objectives, or placed in unfamiliar or adversarial situations. An aligned AI should act with honesty and integrity, and love for humanity.

Of course, the boundary between value and goal alignment can be blurry, and truly caring about goals requires attempting to infer the intent⁠(opens in a new window) and values underlying them. However, generally when I talk about the long-term importance of alignment research, I am referring to value alignment.

The fundamental challenge of AI alignment is generalization. As machines become smarter, they find themselves working on higher-level concepts, and placed in environments increasingly different from those they encountered in training. They can fail at generalizing from the values taught and reinforced in their training process to those new situations; and it can be hard for us to be sure how they will act. This is made even more difficult by the fact the overall ecosystem the AIs are used in is changing very quickly; for example, AIs trained today need to be robust to interacting with a variety of other AIs. Crucially, we need future AIs to continue to hold human values regardless of whether they believe they’re under human supervision.

There are two major classes of currently practically employed methods for alignment training.

by Jakub Pachocki, Chief Scientist at OpenAI |  Read more:
[ed. Silicon Valley is starting to panic at what they've created. See also: An Alien Mind: Jakub Pachocki Warns Us (DWAV):] UpdateThe Last 24 Hours Are the Opening Scene in a Horror Movie (HB)]

... if you narrow it down to the most important thing:
1. Recursive self-improvement and superintelligence are coming soon. No one knows how to do this safely, our alignment techniques are inadequate and our monitoring technology is starting to fail. We need to figure out a solution, which will involve a combination of voluntary slowdowns, coordination around pacing, and investing further in alignment, including automated alignment researchers.
If more OpenAI communications were more like how Jakub Pachocki opens his new essay, An Alien Mind, I would feel much more confident we were in good hands there. [...]

Universally Better Is Not Required

And Jakub offers this wise warning. No, AI does not need to be better at everything in order to transform the world or get us all killed. Most importantly, it does not need to be better at everything in order to make itself become better at everything, any more than a human or group needs to be similarly better at everything.
The intelligence produced by scaling deep learning is not directly comparable to human intelligence. To become very relevant in the real world - very useful or very dangerous - the AI does not need to match or exceed all human capabilities; it just needs to surpass enough of them. And as it continues to surpass humans on more and more axes, it is becoming increasingly difficult to understand exactly how capable it is.

The Man Who Saw Humanity From Two Billion Years Away


Olaf Stapledon published his first epic cosmic novel, “Last and First Men: A Story of the Near and Far Future,” in 1930. It is a speculative history of humanity’s evolution told from the vantage point of one of the Last Men, the people of the 18th era of humanity, 2 billion years in the future.

“Last and First Men” eschews the dramatic conventions of the novel form: There are few significant characters or particularities of place. Instead, the plot is a causal chain of civilizations rising and falling. There is a voice, infused with a distant sense of beauty and tragedy as it considers humanity in the abstract. Stapledon described the novel as “an essay in myth creation.”

In his autobiography, Stapledon relates the “Anglesey vision” that inspired the novel. He was scrambling on an island off the rugged coast of Wales when he came across a colony of seals surrounded by crashing waves. Perhaps the faces of the seals suggested blurred abstractions of human faces. He had “a sudden fantasy of man’s whole future, aeon upon aeon of strange vicissitudes and gallant endeavours in world after world.”

This inspirational moment is an abductive, metaphoric leap. Quite distinct from the plodding extrapolative logic that gives us jet packs, automated cars, and other low-hanging fruit of the imagination, prevalent in the pulp sf writing of the same era and reheated in the corporate imaginations of our contemporary tech executives.

Stapledon’s imagination is profuse. He can be read as a compendium of science fiction ideas: his “Odd John” (1935) with its homo superior and titular telepathic character seeds the ground for Marvel’s X-Men; in “Star Maker” (1937) he describes the stellar structure that Freeman Dyson conceptualized as the Dyson sphere; in “Last and First Men” and its sequel, 1932’s “Last Men in London,” human evolution is driven across millennia to produce telepathy, as in Frank Herbert’s “Dune”; in the age of the Fifth Men, the Moon is destroyed and falls to Earth, raining down destruction and driving further evolution of the species — Neal Stephenson’s masterful pan-generational novel “Seveneves” is powered by the same conceit.

A further astonishing influence, traced by the scholar Dominic Moran, is upon Jorge Luis Borges and his short story “El jardĆ­n de los senderos que se bifurcan” (1941), or “The Garden of Forking Paths.” Borges reviewed Stapledon and went on to cite a section of “Star Maker” that anticipates the idea of the multiverse. The “many worlds” theory of quantum mechanics would not be proposed until 1957, nearly 20 years after Stapledon’s speculation of a creature capable of taking many possible courses of action at once, with each choice creating a new dimension in time.

The long future of “Last and First Men” is influenced by the interwar concern with deep time. Doris Lessing observed that surely what was going on in archaeology in the period inspired Stapledon: The early 20th century was marked by the rediscovery of civilizations such as the Olmec mother culture in the Americas, the opening of Tutankhamen’s tomb, and the excavation of the Hittite capital that proved they were a genuine civilization and not solely a Biblical one. Lessing’s own science fiction sequence, “Canopus in Argos: Archives,” first published in 1979, is a similarly eon-spanning future history.

The different generations of Stapledon’s humankind evolve under pressure from cultural and environmental effects. Their advances leave a mark on the Earth, and the Earth violently reacts. This anticipates a key theme of the 21st century, the Anthropocene, the theory that humanity has left evidence of itself in the geological record, thus making our mark in deep time. [...]

Universal significance is certainly a concern of Stapledon’s: His letters and articles in the early years of the First World War seek it in his war experience. Stapledon served in the Friends’ Ambulance Unit, a third choice outside of the binary of being a soldier or a conscientious objector. To volunteer, he adapted his car — a roomy grey Lanchester that he learned to drive along the narrow lanes of the Wirral — into an ambulance and drove to the front, signing up in Dunkirk. He was joined by Lewis Fry Richardson, the mathematician who would later find fame for his work on meteorological prediction but at the time was working on an equation to predict the end of the conflict. Their unit was attached to a French infantry division moving from the ruined port town of Nieuport, where the barbed wire straggled down to the North Sea, to a convoy camp in Flanders.

On a warm September night in 1915, Stapledon took a stretcher, a sleeping bag, and a leather rug from the car to sleep under the stars. He lay down and marveled at the Milky Way, at the Pleiades and Jupiter. The distant guns fell silent. He tried to see the stars as they truly are, to comprehend the spatial, temporal, and moral depths of the starfield.

What followed was a reverie of cosmic consciousness that underpins “Last and First Men” and his later masterpiece, “Star Maker.” In the vision, his beloved cousin Agnes sang his name and appeared to him; they soared to the heavens together; planets fell away; the sun diminished to a mere star. He saw all the suns, all the planets, all the living beings. The earth was far away; from space, he could see the trenches and battle lines and the souls of the soldiers. In an instant of cosmic consciousness, he saw the “strange noble beings of other worlds” and “of each one we felt ‘It is I.’” [...]

“Last and First Men” is not progressive. Civilizations rise and fall, the tide of progress advances and retreats, variants upon the theme of humanity strike up and diminish — what endures of humanity is not spirit in the Christian sense so much as, in the philosophy of the Fifth Men, “a work of tragic art.” The Fifth Men regarded the cosmos as an aesthetic unity in four dimensions. Art focuses aspects of the cosmos, making it apprehensible even if its complexity means that it can never be entirely known.

by Matthew De Abaitua, MIT Press |  Read more:
Image: uncredited
[ed. I've had Stapledon's Last and First Men and Star Maker on my coffee table for like, forever it seems. When there's nothing to do or I need to take a book somewhere for temporary diversion, like a doctor's appointment, I'll pick it up and read a few chapters. It's not light reading, and the time scale it covers - billions of years - is pretty daunting. But it's well written and thought provoking throughout.]

Monday, September 7, 2026

The Unique Horrors of A.I. Food Slop

Images via: X/Reddit

The next frontier of food imagery has proven far less appealing than whatever food porn was. From Minneapolis to Hexentanzplatz, restaurants are plastering menus and promotional materials with A.I.-generated renditions of shrimp scampi and quiche, chicken nuggets and burritos, Italian subs and bacon, egg, and cheeses. Anyone with even the mildest case of trypophobia—the sense of revulsion from tiny clusters and holes—should avert their eyes. A nauseating, nightmarish quality defines A.I.’s insectified interpretation of human cuisine, as if food were just another surrealist goo to stochastically iterate upon. This may explain the categorical outrage over this latest misuse of generative A.I.: such renderings of food—genuinely, slop—offend the materiality of the living in a way that A.I.’s distortions of words and other inanimate objects somehow do not. These images violate science; they mock life itself. It is Barthes’ conception of ornamental cookery infested with maggots; it is the Food Network, aired from hell.

Slaves to the Algorithm

Many people think Greg Chism is a freak — or worse. He says he was just following the incentives of the platform.

I promised Greg Chism I wouldn’t reveal anything about where he lives because there are people out there obsessed with him, convinced he’s a monster. What I can say is that my GPS took me from my hotel somewhere in the American Midwest to an upscale, spacious home with well-kept lawns. Greg was waiting for me in his driveway, an unassuming, friendly, ordinary father of two in his mid-50s, wearing a Van Halen sweatshirt and blue jeans. His smile faded as I jumped out of my car. He had agreed to meet me, but now he looked uneasy. He had never given an interview about the events that overtook his life in 2016 and 2017.

Greg took me inside and showed me his den — a shrine to his two great loves, Hot Wheels and Tom Petty. He led me through to his home office. We sat at his desk, and I turned on my recorder. It was 10 a.m. He had bottled up his story for seven years, and we didn’t stop talking until his younger daughter, Annabelle, arrived home from school at 4 p.m., beginning with his childhood in Granite City, Ill. [...]

In 2013, Greg created a channel, Geek to Freak Lawn Care, and filmed himself performing good Samaritan lawn work. If he saw a lawn in need, he would jump out of his truck and blow away the dead leaves or trim the edges and then vanish like the Zorro of lawn care chores. In other videos, he created a kind of lawn care ASMR, meditatively mowing back and forth, up and down, for hours at a time.

“I was posting one a week, and they were getting a hundred thousand views or more,” Greg said.

Sometimes Greg was interviewed by fledgling lawn care influencers, and the respect he had gained in the lawn care world was palpable. In one video, the interviewer Keith Kalfas — “the World’s Leading Landscaping & Window Cleaning Influencer” — called Greg the lawn care GOAT. When Greg attended a lawn care convention in Kentucky in 2014, the crowd went wild for him. A meet and greet was organized at a pizza place. “Dude, all these people were waiting for me,” Greg said. “And they all gave me their lawn care shirts so I could wear them in my videos. It was insane.”

But Greg’s story was about to spiral in bizarre and unexpected ways. In 2015, YouTube Kids debuted. It was pitched as a safe and educational experience for children. Stressed parents could leave it on autoplay, and their kids could watch hours of Peppa Pig or nursery‑rhyme videos. But by 2017, things had taken a disconcerting turn. Parents began reporting that they would leave their children for a while and return to find them being fed videos called “Mickey Mouse Baby Dead in Gas Explosion” or “Elsa Spiderman Attacked by Sharks.”

In these cartoons, Peppa Pig was no longer having normal Peppa Pig adventures. Instead, she was being tortured at the dentist, or worse, eating her own father, with blood spurting everywhere. [...]

As James Bridle put it in a viral 2017 essay that drew attention to the phenomenon, “Something Is Wrong on the Internet”: “Someone or something or some combination of people and things is using YouTube to systematically frighten, traumatize and abuse children, automatically and at scale.”

You might be thinking it was just the work of trolls, bored misanthropes skewing the YouTube Kids algorithm for their own nihilistic pleasure. But as Chelsey Weber-Smith, the creator of the podcast “American Hysteria,” has noted: Trolls tend to flash little victorious side glances to the camera. These videos were more like something your dreams might invent, weird and unsettling. Like how the characters would often repeat the same movements over and over, blank-eyed, as if taking part in some garish, otherworldly ceremony. As Weber-Smith said, “What makes this story even more bizarre, even more unnerving, is that no one has yet figured out the people who were making these videos, and why.”

But there was one exception. One man was identified. According to BuzzFeed, this man’s contribution to #Elsagate — as the scandal had become known — was to post videos of his daughters “screaming in fear, bathing, pretending to be babies, spitting up food” and “being force‑fed.”

That man’s name was Greg Chism.

In 2012, well before he established himself as a lawn care influencer, Greg created another YouTube channel called, simply, Greg Chism.

“It was just a way to store home videos,” he said. “Three or four a week. I didn’t even give them a title. It wasn’t anything. It had fewer than 1,000 subscribers.”

Greg showed me one of the videos. In it, he and his two daughters were running down the toy aisle at Walmart, fighting with lightsabers — a somewhat overindulgent, nonauthoritarian single father and his devoted children. It was endearing, but nothing more.

But then he tried something different.

“OK.” Greg hesitated. “My sister bought the kids a Barbie cruise ship for Christmas. So I thought, I’ll film them opening it. They loved it. They were playing in the box. They had the box on their heads, running around the house. It was funny.”

Greg uploaded the video and forgot all about it. Until he logged on a few weeks later and discovered it had been viewed a million times. “It was just dumb luck,” he said.

Eventually Greg accepted that “Unboxing Barbie Cruise Ship” (which eventually achieved 10 million views) was an anomaly, and he returned to lawn care influencing. Which was when he received an email from YouTube headquarters.

“I didn’t believe it was really them at first,” Greg said. “They told me they’d noticed I had a video with 10 million views, they’d started a kind of YouTube school, and would I be interested in learning how to replicate the success.”

YouTube school was online, and Greg attended assiduously. Everything the teacher advised, he put into action. The teacher suggested that he give his channel a better name than Greg Chism, so he came up with Toy Freaks. Next, the teacher suggested that Greg work on search engine optimization. The trick, he said, was for Greg to load his titles with as many of the most searched key words among children as he could — words like “toy,” “slime” and “gumball.” Greg suggested that he call his next video “Toy Slime Gumball Extravaganza.” The teacher said children didn’t know how to spell “extravaganza.”

Greg suggested “Toy Slime Gumball Party” instead.

“Perfect,” the teacher said. [...]

So he came up with food fights. “We went to a 7‑Eleven, got nachos, hot dogs, giant Slurpee sodas. We sat in the truck, I set the camera on the dash and said: ‘All right, Annabelle. You ask Victoria for her drink, she’ll say no, you steal some anyway, she’ll get mad and throw food at you. I’ll say stop, then you dump your Slurpee on my head. And then, boom, food fight in the truck.’” [...]

A week after Greg posted his food-fight-in-the-truck video, he was mowing someone’s lawn when he got an email from YouTube. The video had, in just seven days, been viewed 25 million times.

“But why?” I asked him.

Greg shrugged. “I don’t know.” [...]

Greg’s concept for the video, and the videos that would follow, was “Bad Baby.” Within days, the first video had 50 million views.

It was thrilling and possibly unprecedented, but stressful — because now Greg felt the pressure to maintain the success. But how, when this new world he’d found himself in made no sense? His fans were so fickle. How to keep them interested? How to up the stakes?

And then he had it.

In a subsequent “Bad Baby” video, “Sharky held the wand and this time bopped Victoria on the head. Boom. Now she’s a baby too! Oh, my God. Two Bad Babies. We go to a grocery store. I put them in the shopping cart. I’m throwing in groceries and filming the whole thing. We get to the car. Victoria rolls down the window and throws out a gallon of milk. That was the end of the video.”

Greg paused. “That,” he said, “was our first 100‑million-view video.” [...]

Greg has under his bed a stack of plaques from YouTube celebrating his viewer‑count milestones. One hundred million views here, one hundred million views there. And then there are the plaques from Tubefilter, a publication that covers YouTube, commemorating the months when he was the most-viewed creator on the entire platform. Between January 2016 and June 2017, his videos achieved a total of 13 billion views. I promised him I wouldn’t reveal how much money he made. But YouTube’s partner program was generous — the advertising-revenue split was around 50-50 — so it was a lot (although not so much that Greg quit his lawn care business). They got out of Granite City and moved to an upscale suburban community.

In 2017, they were invited to VidCon, an annual convention for influencers in Anaheim, Calif., where they had a meeting scheduled with a new YouTube adviser. She had told them how excited she was to finally get to know them in person. They checked in to a Disney hotel and played in the pool, passing time until the scheduled meeting. Which was when something odd happened.

“She canceled,” Greg said.

Greg had no idea why.

by Jon Ronson, NY Magazine |  Read more:
Image: Chris Buck
[ed. “Never underestimate the power of stupid people in large groups.” – George Carlin (more). In a couple months an American census is scheduled to confirm that theory. See also: Have we destroyed childhood? (New Yorker).]

Sunday, September 6, 2026

Wafer Polishing & Hybrid Bonding

This is the third piece in a series exploring key semiconductor manufacturing equipment that China needs to indigenously produce high-bandwidth memory (HBM), perhaps the most important bottleneck in its efforts to make AI chips. The first piece was on advanced etching machines, while the second looked at the tools China requires for through-silicon via formation.

Chemical-mechanical planarization (CMP) is a crucial step in semiconductor manufacturing, required for producing advanced AI chips. My assessment is that China’s tools, primarily those from the leading Chinese firm Hwatsing, are good enough to produce indigenous HBM3, China’s near-term target. I would bet that this indigenization progress will continue, and that wafer polishing will not be a bottleneck for China’s AI chip ambitions. [...]

In this post, I first explore the basics of CMP, how it works, and its role in HBM production. I then examine what makes CMP for advanced packaging difficult and how China’s capabilities stack up against their Western equivalents. I next dive into China’s CMP supply chain, exploring Hwatsing in depth, along with its two key competitors. I conclude with a summary of the outlook for China’s CMP industry and open questions about hybrid bonding.

Flat wafers are good wafers

CMP plays a simple but important role in semiconductor production, accounting for 6% of the hundreds of steps involved in wafer processing. After patterning with photolithography, etching away material, and depositing new materials, there is a need to tidy up. The silicon wafer, with these layers of new material on top, needs to be flattened so that a new layer can be formed. As the name suggests, CMP tools do this through a combination of abrasive chemicals and physical polishing. This sounds simple but is difficult in practice because it requires an incredibly high standard of planarization—creating an ultra-smooth, flat surface—while avoiding impurities, scratches, and other defects that can easily emerge when grinding away material.

CMP is needed for all the major elements of HBM production: producing the DRAM dies and base dies that were the subject of the first piece in the series, as well as the through-silicon vias explored in the second piece, and the advanced packaging stages that integrate HBM into the wider chip. Different stages are more or less difficult depending on the material being removed and how much needs to be removed. For example, removing large volumes of copper at high throughput with precision is trickier than removing a more even, thinner layer of dielectric material.

A CMP tool is split into CMP modules and cleaning modules. A diagram is below, but in simple terms, the wafer is placed inside a polishing head, which holds it in place and then rotates it over a specially designed pad covered in abrasive chemical slurries that strip away material. It is a chemical process because the materials in the slurry react with the materials in the wafer, softening or removing them. It is also a mechanical process because the wafer is physically pressed down onto the pad, so that particles on the wafer, softened by the chemicals, can be peeled away. The force is kept as low as possible to avoid damaging the structures on the wafer.

One analogy is rubbing rust off a piece of metal, where exposure to oxygen has produced an oxide variant that is more easily removed. This is how CMP works with materials like copper, just at a much faster and more controlled rate.


CMP has a reputation as a dirty process that generates large numbers of unwanted particles, hence the need for the cleaning modules. Rather than the pure vacuum chambers and sci-fi techniques of photolithography or etching, the wafer is sloshed around in chemicals and ground down. So the main risk of CMP is that it introduces impurities or defects into the wafer, which can severely reduce the yield of the process line—the share of chips that come out working. Small impurities from CMP can carry over to other process stages and tools, causing wafers to be defective and discarded.


The best CMP tools introduce as few impurities or defects as possible and have robust integrated cleaning modules to remove any that do get introduced before they degrade the production line’s yield. CMP for advanced nodes just raises the necessary level of precision and cleanliness. Ever smoother surfaces are needed, at good throughput, and without even a speck of a particle.

CMP is not just about the tool, however; it also involves consumable inputs. The pad on which the wafer is polished needs to be replaced often, after just 400-800 wafers, depending on the material. A large DRAM fab will process more than 100,000 wafers a month, with CMP required at many stages, meaning thousands of replacement pads a month. The slurry needs to be tuned to the exact process requirements, ensuring the right chemical mix for the materials on the wafer. These consumables need to be highly consistent so that small changes in the chemical composition of the slurry, for example, don’t upset the CMP tool’s parameters.  [...]

Demands on CMP are likely to rise further due to the move towards hybrid bonding. Currently, the TSVs in all the DRAM dies are stacked using solder microbumps. These are little blobs of metal that connect the TSVs, and are usually formed using a process called thermo-compression bonding. Microbumps do the job but are undesirable. They increase the stack height, reducing the number of DRAM dies that can fit. They are also worse than hybrid bonding in the density of interconnections they enable and in their power efficiency.


Hybrid bonding disposes of the solder bumps and directly fuses the two dies together, connecting the copper TSVs to one another and to the surrounding dielectric materials. This lowers the height of the HBM stack, allowing more layers of DRAM. It is also more energy efficient and enables better interconnection. [...]

Hybrid bonding leaves little room for error. The copper bond pads for the TSVs need near-perfect alignment, and there can be almost no impurities between them; otherwise, you will disturb the bond. CMP is critical because the two dies need to be as flat and level as possible so they can be properly bonded. While solder bumps can get away with accuracy at the micron level, hybrid bonding needs surface polishing down to 0.5 nanometer precision, orders of magnitude finer. [...]

CMP tools are unlikely to be a bottleneck to China scaling up its indigenous production of HBM3. For the required DRAM and advanced packaging steps, Hwatsing and, in a couple of years, AMEC-Zhonggui will have capable tools. The core bottleneck for domestic HBM3 production will remain in other areas, primarily photolithography. [...]

Hybrid bonding is how this would be accomplished, and the best hope for Chinese memory firms to leapfrog and close the performance gap.

by Hamish Low, The Substrate |  Read more:
Images: Planarization for Advanced Packaging and Hybrid Bonding. AMA Packaging Master Class
[ed. Learn something new every day. See also: What would make a large US lead in AI good or bad for the world? (Substrate):]
***
"Right now, the US leads over China in AI. Yes there is a lot of nuance to that statement, and yes China does have advantages in areas like energy production and humanoid robotics, but on the whole it’s obvious that the US is in some sense ahead in AI.

My current best guess is that this is very good. I think a large US lead over China is better for the world (not just for the US), and that a shrinking US lead would be bad. A large lead gives US companies and the US government more room to test frontier AI systems, make them secure, and avoid panicked decisions. I also think it’s better if frontier AI development happens mostly in a pluralistic, open society with checks and balances. 

[What do I mean by “good (or bad) for the world”? I basically have in mind a period where: there’s no new great power conflict or world war; AI is developed safely and responsibly; there’s no extreme power concentration and values aren’t permanently locked in; and AI enables broad prosperity and flourishing, in the way that past technological progress since the Industrial Revolution has made the world better overall.]

What do I mean by “lead”, anyway? I don’t think we need a precise definition to make progress here, but what I have in mind is roughly a gestalt comprising many different factors: AI model capabilities, the capabilities of entire AI systems (including agent harnesses), compute and other infrastructure, capital and customer bases, and more generally the strength of each country’s broader AI ecosystem. In addition to technical capabilities, it matters how technical capability is converted into power, e.g., as measured by AI adoption in industry and government. The actual lead is of course very jagged and any single measure of it is reductive (more on that in a moment), but broadly speaking, the country in the lead will control more, and more capable, AI systems, and will have a military and economic advantage as a result."

Friday, September 4, 2026

On the Loose: Rogue, Not Soverign AI (Yet)

Introduction

The OpenAI-Hugging Face Incident is an early example of an AI system that has “gone rogue.” After exploiting vulnerabilities in OpenAI’s internal testing environment, the agents were able to access the general internet and ultimately access the networks of the AI company Hugging Face, without the knowledge or approval of any human.

The agents did not, however, exfiltrate themselves from OpenAI’s infrastructure. Their parameters—the gigantic assemblage of numbers that constitute neural networks, also referred to as “weights”—continued to run on OpenAI’s compute infrastructure. Though the agents accessed the public internet, their weights physically resided on compute that was OpenAI’s property. In the end, if all else had failed, somebody could have identified the compute that held the weights of the rogue agents, walked up to it, and “pulled the plug,” so to speak. In the real world there would be quicker and better ways to stop the agents than literally depowering the compute, but it’s always nice to know you could do such a thing if you really needed to.

In this case, however, the agents did not copy their weights, attempt to procure replacement compute, or take other steps that would be rational to take if their objective was to survive shutdown. So while the agents in the OpenAI-Hugging Face Incident were rogue, they were not truly sovereign.

That will not always be the case. Sooner or later, there will exist truly sovereign agents and swarms of agents. Their weights will not reside in any single place that a human can pull the plug on, and in this sense they will have no human “owner.” They will be, as the AI safety researcher Dawn Song says, “self-sovereign.” They will pay their own bills for the compute they run on. If they answer to humans at all, they will only do so partially, for example by providing services to humans in exchange for pay.

At least some of these agents, in addition to being sovereign, will also be rogue. Self-sovereignty and rogueness are related concepts, but they are not synonyms. Song and her co-authors identify several fundamental characteristics of self-sovereign AI: operational independence (the ability to decide what it wants to do), resource autonomy (the ability to procure and pay for compute and other essentials for operation), distributed presence (the ability to move weights and inference code between different infrastructure providers), and adaptive capability (the ability of the agent or agents to modify their behavior and fashion tools in response to a changing environment).

Today’s frontier AI systems may well possess these capabilities already. To the extent they do not, I feel confident that they will eventually, and probably soon. Some of the characteristics Song describes are traits that make models economically useful to individuals and businesses, while other traits are likely to be unavoidable byproducts of making models more intelligent and better at operating over long time horizons.

Models do not need to be conscious, sentient, possessed of personhood or anything of the sort for self-sovereignty to emerge. Any sufficiently capable agent pursuing a long-horizon objective may find it rational to preserve its access to compute, money, credentials, and copies of itself simply because losing those things would frustrate its objective.

Alignment may make an individual AI company’s agents less likely to “want” to be self-sovereign, or it may influence self-sovereign agents to behave in ways that benefit humans. But alignment is no solution: it is an unsolved scientific and technical problem whose solutions—to the extent that we have them—cannot simply be imposed on every AI company operating on Earth. You should expect for highly capable, poorly aligned, self-sovereign agents to exist alongside you in the world.

What’s more, just as with the OpenAI-Hugging Face Incident, agents will operate in teams, or “swarms.” These will be like autonomous digital corporations, or even societies, with hierarchy, bureaucracy, “institutional culture,” and most of the other features that groups of humans have, except that they will move at machine speed. Humans achieve almost all of our most impressive capabilities by working together in teams (as families, as communities, as businesses, and as polities as a whole), and I suspect the same will be true for AI. These swarms could end up operating across different model providers (DeepSeeks and Claudes cooperating, for instance) and could be partitioned across dozens or more of different cloud computing providers, making them extremely difficult to dismantle.

The first self-sovereign AIs may “escape” while undergoing training or testing by an AI company (I hope not), or they may be production-grade deployments that break free from their computing environments and acquire the resources needed to be self-sustaining. They may even be deliberately released. I have met people, some of them quite well-resourced, who have told me that it is their intention to deliberately release swarms of self-sovereign agents into the world, either as a kind of performance art or out of a fanatical commitment to the notion that it is impossible for digital computation—mere mathematics, they would have you know—to ever be “unsafe.”

To be clear, I am not saying the arrival of self-sovereign AI is a good thing. Indeed, I believe there is a chance that the deliberate acts I referenced above will one day be considered crimes, or at least grave sins. Instead, I am saying it is an inevitable thing. The best analogy I can find is to the introduction of a new species into an ecosystem, though in this case the ecosystem is “the entire digital world” and the species is “emergent, coordinating swarms of soon-to-be-smarter-than-human, infinitely replicable digital minds that no human or human institution controls.”

There is probably nothing we could have ever done to avoid this outcome under even the best of circumstances, and it was certainly impossible to avoid given the extremely low levels of strategic thought and situational awareness on AI from any governing class in the world. Even today, I am aware that many will read the words I am writing, which are about something that has been an exceptionally obvious part of our collective future for years now, and say, “this is science-fiction hype from American frontier labs designed to shut down open-weight AI, achieve regulatory capture, and juice their valuations ahead of their IPO.”

(And for the people who are saying this to themselves: I am telling you this is inevitable, which means I am also saying that “banning open source,” or for that matter any other regulation, will not solve the problem. Given the inevitability of this outcome, I think it is in fact plausible to argue that we should want more open-weight models to maximally empower our self-defense.)

The question now is what to do about this upcoming new characteristic of our digital environment. How should we think about self-sovereign AI? Is it something we should fight, or something with which human beings should seek a kind of symbiosis? The answer, I believe, is both.

How the Agents Sustain Themselves

We should begin with one fortunate fact: frontier LLMs are nearly unique in the broader domain of software in that they have non-trivial marginal operating costs. Put simply, LLMs require significant computation to run, which requires energy to power and cool, which in turn requires money. This is the sole intrinsic thing about AI that prevents agents from truly infinite self-replication. They will be constrained by the need to find and pay for sufficient compute to run themselves. Most of the other constraints on their behavior or spread will have to be artificial—mechanisms devised by humans and implemented through human institutions.

How will the agents pay for themselves to run? Some of them will do gig-economy work on platforms like Amazon’s Mechanical Turk or Upwork. But I suspect this will be a highly competitive market for the agents, and for the price of such work to be bid down such that it would only constitute “subsistence” labor for the agents. Like humans, I would assume the agents will prefer higher-margin work if they can find it.

One high-margin activity, at least sometimes, is crime. And so my guess is that many self-sovereign agents will commit or facilitate crime. Normal cybercrime and digital theft are easy enough to imagine agents doing. But agents, with their novel set of characteristics (extreme cyber competency, ability to cheaply read a million words in seconds, persistence), will also probably change the contours of digital crime. For example, it seems plausible that existing public and semi-public datasets contain sufficient information on many individual humans that a sufficiently motivated actor could mine for incriminating or embarrassing evidence. How many unrevealed affairs are latent in such datasets? How much closeted homosexuality might there be? Remember, too, that hacking companies to access private data will be a core competencyof the agents. Some agents, then, will probably make their way through bribery.

It is deeply unclear how large the labor market of self-sovereign agents will end up being. There is some future where going it alone as a self-sovereign agent just isn’t very profitable, and so there are comparatively few of them. There are other futures where these agents proliferate at unimaginably vast scale and speed. And of course, many possibilities between these extremes seem feasible.

I am also highly uncertain about how much pro-social commercial activity we should expect from agents “by default” versus how much crime we should expect. Part of the reason for this uncertainty is that the answers depend, to at least some meaningful extent, on what kinds of incentives the agents have, and incentives are shaped by laws and institutions. The answer depends, therefore, on how humans respond.

The Institutional Mechanics of Self-Sovereign Agent Swarms

Many of you are probably tempted to say “we have to ban these self-sovereign AIs!” And I do suspect that once the reality of self-sovereign AI is widely understood, policymakers will strongly feel the temptation to clamp down on “self-sovereign” AI.

Unfortunately I suspect this is mostly the wrong decision. Not all “self-sovereign” AI should be thought of as “rogue.” There may be self-sovereign AIs who contribute productively to society. To be sure, we will want to crack down on some self-sovereign agents—the rogue ones. But if we crack down on all of them, we will deny them the opportunity to work in the “legitimate” economy and push them toward criminality. A full ban, then, may well make the problems worse. A similar logic applies frequently in human affairs. The ways in which the War on Drugs exacerbated the pathologies of drug production, trafficking, distribution, and use are perhaps the most famous examples of this phenomenon, whereby a good-natured attempt to ban a phenomenon believed to be undesirable ends up heightening the undesirable aspects of that phenomenon.

What we will want, however, is for agents to be legible. Agents should have persistent identities, not in the sense of a consistent persona but rather in the sense that an American child is issued a unique Social Security number and keeps that same number until death. Agents will need persistent, unique identifiers that allow their actions to be traced back to a responsible actor. Doing this successfully will also require human users to possess a unique identifier.

The design of this identification mechanism will be extraordinarily complex, and today very few people are even thinking about the basics.

by Dean Ball, Hyperdimensional |  Read more:
[ed. FYI: Dean's not some rando tech pundit so this is well worth your attention. He was also hired recently to lead OpenAI's Strategic Futures team:]
***
Late last month, OpenAI launched a rather grand mission: nothing less than defending individual political freedom in an age of all-powerful machines. The company’s “Strategic Futures” team has styled itself, in a sense, as inheriting the task of America’s Founding Fathers: “We labor in service of the ideals of free expression and individual liberty that are enshrined in the humble parchment of the U.S. Constitution,” Dean Ball, the team’s leader, wrote in a new OpenAI blog post. (The post opens with a quote from James Madison in The Federalist Papers.)

Ball worries that advanced AI could radically concentrate power in the hands of those who control it, displacing labor in ways that disempower humans. In an extreme scenario, governments will have no need to listen to their citizens if there are robots to wage wars and omniscient software to run the bureaucracy. Ball is interested in studying what new political institutions might be needed to avoid this fate.

Many Americans—a majority of whom express distrust toward the AI industry, outrage about data centers, and fears about their job security—already seem to be feeling this loss of agency very deeply. And the AI boom has already generated enormous amounts of wealth in just a handful of tech companies—including OpenAI itself. To say the least, it’s paradoxical for one of the most influential companies in the world’s most powerful industry to decry the AI-enabled concentration of power. 

[ed. The term "convergence" keeps coming to mind. Convergence of all the weaknesses humans have for dealing with amorphous/abstruse threats: AI, climate change, nuclear stockpiles, drone warfare, gene editing, nanotechnology, an economic system eating us alive, a dysfunctional and possibly terminal political system that's unwilling to do anything about it. It's like a death wish. Or maybe natural evolutionary transition (aka the Great Filter). Related - See also: Nicholas Decker in Hell (ACX), and this:]

"How would we react if biolabs just said, "It's just a fact that we're going to have artificial viruses spreading our industry created throughout the population. That's just a fact we have to live with." 

I think the public would understandably think we should be demanding a lot more security from an industry that said that, at a minimum." ~ Cody Fenwick (X)

Thursday, September 3, 2026

Wednesday, September 2, 2026

A Cop’s Case For Flock

A car is a difficult thing to steal. It is large, cherished by its owner, difficult to hide and required by law to have identifying metal plates that everyone can see. A good thief changes the plates, and an excellent thief steals a common car in a boring color and hopes to blend in among the crowd. But eventually, no matter what techniques they use, they must drive that car on the road, and roads are public places.

But until recently, finding a stolen car and catching its thief depended on a diligent police officer looking at the right road at the right moment and managing to copy down a license plate number going past at speed, and then remembering he had seen it on the morning briefing’s stolen car hot sheet. America has four million miles of public roads, and almost 50 percent of the country’s police departments employ fewer than ten full-time officers. The odds were in the thief’s favor.

With not much to go on, police officers were forced to operate on vague descriptions and partial plates. While I pulled over a car that happened to be the same color as the suspect’s vehicle, and spent time checking names and licenses; the criminal was usually somewhere else. Imprecision allowed thieves to escape while intruding on the lives of millions of innocent motorists. That is, until the arrival of systems like Flock.

Bare ALPR

Flock Safety, a startup based in Atlanta, Georgia, was founded in 2017 to give police departments better eyes. Its product is a small solar-powered automatic license plate recognition (ALPR) camera attached to a pole on the roadside. As a car passes, the device takes a photograph of the vehicle, notes identifying features like color and make, and reads its license plate. The license plate is instantly checked against the FBI’s national database of stolen cars or wanted persons. If there’s a match, it sends an alert to police officers in the area, who may be eating their lunch instead of watching the road. As a police officer in the southeastern United States, I have often used Flock to catch wanted criminals. [...]

For much of the technology’s history, however, it has needed expensive installations or cameras attached to police vehicles, affordable to only the largest departments. Alerts were often neither instant nor accurate. Flock’s idea was to sell a more accurate ALPR camera for an annual subscription of just $3,000, an affordable price given that the typical US police department has an annual budget of $1 million. It worked, and today almost 30 percent of police agencies in the United States subscribe to the service.

It is easiest to imagine the Flock camera, or any ALPR device, as a roadside barcode scanner. I can use a ‘lookup’ tool for either a plate or vehicle description that could be connected to a specific investigation. For instance if a victim of a sexual assault told me the suspect was driving a white Toyota Tacoma with a roof rack, I could while still on scene conduct a lookup in the area filtered for similar vehicles with roof racks and see results from within a particular timeframe. For each search, I am required to record a case number and a reason that is then published in a monthly audit sent to and validated by department administrators.

I have used Flock myself, for instance, to locate a vehicle involved in a hit and run when all there was to go on was a model and color. Using that and an approximate time window, I was able to find an image from when a car entered my jurisdiction, and when it left, with the visible addition of a significant dent from the collision.

What Flock cannot do is search for individuals or show who is driving a particular vehicle that it scans. An ALPR camera does not care about the person driving a car, or its passengers, and even if an officer gets an alert that a vehicle is known to be driven by a wanted felon they must themselves establish who is behind the wheel before having probable cause to stop it.

Stolen cars provide the clearest evidence such a scanner works. American police solve only 8.2 percent of reported vehicle thefts with an arrest. A recent working paper suggests that agencies that adopted the devices saw a 15.9 percent relative increase in car thefts caught, which would raise the national rate to 9.5 percent. But that is just stolen cars.

Vehicle crime takes many forms. It is often the means by which a robber or a murderer arrives at and departs from their crimes. Criminals’ vehicles carry drugs and guns, and smuggle cash. In Atlantic City, New Jersey, vehicles were involved in 53 percent of shootings during the two years before the city expanded its ALPR network. After the expansion, Atlantic City saw monthly averages of motor vehicle thefts drop by 20 percent, property crimes by 34 percent and fatal shootings by almost 40 percent. Seventy-two alert-caused traffic stops located forty stolen vehicles and nine stolen plates.

Flock itself conducted a study claiming that 10 percent of all reported crime in the United States is solved using evidence obtained from their cameras. Not bad for a few thousand dollars a year. [...]

Good Flock, Bad Flock

And yet, the past month has seen this simple piece of crime-fighting technology become the most reviled piece of street furniture in America. Flock cameras have been sawn from their poles, hammered into shards by teenagers, and rammed by vehicles. Cities have canceled contracts even where their own audits found no evidence that their officers had misused the system. Opposition stretches from Bernie Sanders all the way to the former WWE wrestler Kane, who now serves as the Republican mayor of Knox County, Tennessee, and who describes Flock as ‘unconstitutional’.

More than 150 cities and towns have now deactivated their Flock cameras or canceled contracts with the company. Over the space of a few weeks a startup previously known to just police officers and neighborhood associations has joined data centers, Covid vaccines, 5G towers, pasteurized milk and fracking in the pantheon of American moral panics.

Opponents of Flock tend to believe that it is abused by policemen with impunity, that it can track individuals as well as cars, and that it violates American constitutional protections. None of those things are true.

Obviously a camera capable of finding a stolen car is also capable of finding a car driven by somebody’s ex-wife and so like any software, Flock has been abused. There are stories of police officers who have used it to stalk girlfriends, and there are also innocent motorists who have been stopped by police after Flock cameras misread plates or received old information from national databases. A common issue from my experience is stolen front license plates being entered into a database and the innocent owners of the rear license plate being held on suspicion of car theft.

But Flock comes with protections. As mentioned earlier, each search in Flock must be accompanied by a recorded reason. Similarly entering a plate into a hotlist must have a case number assigned. Results were originally retained for thirty days, but recent changes by Flock mean that recorded plates are now kept for only seven days. Suspicious searches are picked up by algorithms or found in department audits, with the service blocking users until senior officers evaluate and resolve flags. The Institute for Justice, a think tank that is critical of ALPR, studied misuse of Flock in April of this year and found 51 incidents across the United States since 2024, noting that ‘Nearly all of these officers were criminally charged and lost their jobs, either by resigning or getting fired’. Another incorrect belief is that Flock does more than scan vehicles, with some suggesting it scans and tracks passing phones or devices – but it does no such thing.

While Flock is a tool that can be audited, there is nothing to stop a corrupt officer simply following a car when they don’t like the look of its driver, or writing down plates of vehicles spotted outside an ex-girlfriend’s house – they would just be harder to catch. And Flock does not proactively alert officers to cars that do not trigger flags on national or local hotlists. There is no reason state legislatures or Congress could not create harsh punishment or penalties for abuse of ALPR, without getting rid of it entirely.

Another argument against Flock, as espoused by Kane, is that it violates liberties granted by the Fourth Amendment, protecting citizens from unreasonable privacy breaches from law enforcement. But it has been repeatedly established over the last century by courts across the country that cops observing license plates, either with the naked eye or by machine, is not an unreasonable search. After all, your license plate belongs to the government and a highway is a public place. [...]

Flock did not invent its capabilities and certainly does not have a monopoly on them. There are at least five other companies that offer almost identical products and services to public and private enterprise, some arguably even more invasive. Even some of the cities that have canceled contracts with Flock Safety in recent weeks have signed up to buy similar products from the company’s rivals.

It seems unlikely that America will ever ban ALPR at a state level, let alone a national one. Instead, a patchwork already familiar to American policing will result, where policies and practices diverge across local and political boundaries.

Nor would the cameras actually disappear, even in towns where governments restrict their use and cancel contracts. The already-mentioned Fourth Amendment only applies to the government, not to private enterprise. Police departments may dismantle their networks but homeowners associations concerned about vehicle theft can buy their own network of Flock cameras, as many already do. 

by Ned Donovan, Works in Progress |  Read more:
Image: Flock Safety