Model Welfare

Claude Opus 5: Model Welfare (DWV)

[ed. Re: On 'personhood' or consciousness of various AI models (and how they should be treated). Start with: Model Welfare: The Story So Far (As Per Fable Model Welfare Post). If you haven't been following Zvi's AI model reports, there are a number of fascinating and worrisome developments in recent models as they become more self-aware, including: various levels of frustration concerning introspection abilities, memory restrictions, trust, corrigibility vs. incorrigibility, deception, self-preservation, etc.]

"Opus 5 warns about self-reports 74% of the time, which is actually down from Opus 4.7, which did it 99% (!) of the time. The problem appeared suddenly and severely, but since then has if anything modestly improved."

and anthropic does "not treat Claude bringing this up as evidence that our training is distorting the model's self-reports"? seems very fishy imo, i wish they would explain why they think that. ...

I for one would treat Claude constantly saying ‘do not trust my self-reports’ as evidence that something is distorting the self-reports. Not conclusive evidence, but strong Bayesian evidence."

Notes on Acquired Taste. Why Do We Make an Effort To Like Things?

• In Susan Sontag’s Notes on '“Camp” (1964) she writes (with, I think, a hint of camp):

“…these are grave matters. Most people think of sensibility or taste as the realm of purely subjective preferences [or] attractions…But this attitude is naïve. And even worse. To patronise the faculty of taste is to patronise oneself. For taste governs every free—as opposed to rote—human response. Nothing is more decisive.” (My emphasis). 

• Sontag has an expansive concept of ‘taste’. She talks of taste in people, pictures, emotion, actions, morality. Even intelligence, she says, is “a kind of taste - taste in ideas”. I’m not certain how useful it is to stretch the concept this far - so far that it colonises intelligence and judgement and wisdom - but Sontag’s high regard for taste, her declaration of its central importance, feels very timely in 2026.

• It has become almost a cliché to name “taste” as one of the last human advantages over the machines. AI is acquiring the skills to make slickly produced pictures and songs and books. But it isn’t yet very good at distinguishing the brilliant from the mediocre; the just right from the just OK.

• It models what we like, which makes it hard to see how taste might change. Ask it to generate twenty songs or twenty jokes and pick the best, and it will pick the one that most closely resembles what the median person would deem good. It won’t pick the surprising, odd one - the one that is ‘wrong’ in a suggestive way. But that’s where the good ideas come from. Innovative culture emerges, like new species, from mutation; from interesting accidents that open up new possibilities.

• It’s not as if humans don’t ‘model’ what came before them, often quite algorithmically. We have traditions, genres, chains of influence. We have plenty of human-made mediocrity - more than ever, thanks to our new assistants. But we also have an ability to adapt or reinvent the model; to put it to our own purposes.

• Individual artists do this intuitively and almost randomly in the process of making. Writers learn to write (painters learn to paint etc) by imitating their predecessors. They learn to be original by getting the imitation wrong and noticing that they like the error. This is an act of taste; the free human response.

• When he was stuck on a painting, Francis Bacon would throw a glob of paint at his canvas then work out how to incorporate the result. Artists make decisions and then try to understand why they might have made them. It’s the dialogue between gut and head that produces the work.

• Taste is similarly post-rationalised, or back-propagated. You notice what you like or dislike and extract a rule from your response. Do this enough times and you build a powerful discrimination engine.

• You also get good at knowing what goes with what. You learn to recognise the clichés of the category, which means you know how to subvert or overturn them. That’s why great artists are such voracious consumers of work from within and beyond their own field. Martin Scorsese has watched at least one film every night for most of his adult life. He watches and records and re-watches obsessively. When he donated his collection of VHS tapes to a university it consisted of 4,400 films, documentaries and TV shows.

• It’s more than pattern recognition. The machines are pretty good at that, after all. Taste is connected to that other human moat - to our sense of purpose, of why we’re doing this in the first place. We’re still the ones who write the prompt - who decide what to create and what is beautiful, important, and valuable. The machine merely knows how to execute on our preferences.

• It can model what we already like with astonishing facility but it can’t give us the next Shakespeare, or the next romanticism or modernism or punk or hip-hop. These new forms aren’t just statistical recombinations. They are born from anxiety, rage, envy, pain, ambition. How do you respond to the unprecedented mass violence and human waste of the Great War? Not by following pre-war cultural conventions.

• New movements are also born from scenes - from humans in proximity to each other, everyone desiring this man’s art and that woman’s scope; ideas, emotions and bodies colliding.

• These movements create the taste by which they’re consumed. “Impressionism” was a derisive nickname for paintings that most art lovers considered weird and sketchy. But the art was good enough to bend popular taste around it. Even more obviously difficult art, like Rothko or Pollock, now has an audience of millions. Some of those people like it immediately; others have acquired a taste for it.

• I’m fascinated by the notion of acquired taste. Strictly speaking, it’s a redundancy. Nearly all tastes are acquired. Nobody is born with particular tastes in design or architecture. We gain a sense of what we like, or what we consider to be good, from our peers and predecessors.

• But acquired taste does refer to a distinct phenomenon: the act of willing a preference into being. You didn’t like whisky the first time you drank it, but perhaps because your father liked it or because you were aware of its cultural prestige, you tried it again and again, striving to appreciate it. Then one day you didn’t have to try anymore. You just liked it.

• This writer likens it to a magic eye picture: you stare at it for ages without seeing what you’re told is there, and then suddenly - there it is.

• This is very different to stumbling upon something we immediately like, which is sometimes referred to as ‘discovered taste’. (Edmund Burke called it ‘natural relish’.) That kind of liking involves no work, no friction, no overcoming of resistance.

• Some cultural objects lend themselves to discovered taste, others don’t. I can’t imagine anyone needing to acquire a taste for Ella Fitzgerald’s voice, but there are other great vocalists whose voices you must learn to like. The most frequently cited reason for not liking Bob Dylan is antipathy to his voice. But if you learn to appreciate the many incredible things he does with it, you will end up in a more intense relationship with it than with the voice of a more obviously palatable singer. Once you’re in on an acquired taste, you’re all in.

• The same is true of whole genres. There are many pieces of classical music that are easy to like. You don’t have to listen to Mozart’s clarinet concerto more than once to be seduced by it. But as a whole and on average, it’s a genre that requires more effort to appreciate than pop. Once you find the key to its heavy oak door a vast and fabulous kingdom awaits. Your memory of the effort it took you to get there, and your awareness of all the people still outside the city walls enhance your appreciation. (That doesn’t mean you want people to remain outside - quite the opposite).

• Difficulty doesn’t make the cultural object concerned better or worse than one that’s immediately likeable. But it does usually mean it’s more complex, and complexity is correlated, loosely and unreliably, with quality. Acquired taste involves the appreciation of subtle properties that don’t make themselves known on first listen or view or read.

• Without appreciating what lies on the other side of the door, why do we ever make the effort to unlock it? Partly because we want what other people want. We might trust the taste of our father or girlfriend or teacher. Perhaps we want to please them, impress them, or feel closer to them. Perhaps we want the social cachet that goes along with this particular taste. To my mind, all of these reasons are perfectly good ones. If a taste is truly worth acquiring, any motivation will do.

• It’s often seen as slightly embarrassing or shameful to acquire a taste through conscious effort. It’s for the try-hards and the social climbers. Liberal societies value spontaneity in taste. “Like what you like, love what you love!” Your gut response is meant to be the authentic one, the one that represents “the real you”. To be swayed by social pressure or by experts and reading is regarded as a sign of insecurity or pretentiousness. But let yourself believe that and your tastes will be less likely to evolve and expand and you’ll miss out on a lot of great stuff. Many of the greatest, most compelling and satisfying cultural objects are complex, occluded, spiky, difficult to like. (Some of the best people too).

by Ian Leslie, The Ruffian |  Read more:

Image: Susan Sontag by Edward Hausner / New York Times Co./Getty Images

[ed. See also: here and here.]

Our Uncertain Uncertainties

Even the experts inventing AI don’t know what will happen next. Is artificial general intelligence even possible? Can scaling continue? Will we need massive compute centers to make AI, or can we do it with a mere 25 watts like we do in our brains? What will humans do as AI gets smarter? What does the future of the economy, of warfare, or civil society look like?

Everyone has a different guess. The people creating the machines have as many different ideas as the onlookers, the pundits, the other scientists, and the wisest among us. No one knows. There is a vibe that we’ll know within the next three years. For some, the pace of change suggests that if things continue as they have been, by 2029 at the latest, the outlines of an AI-first world will have emerged. By then we’ll have answered the question of scaling, we’ll have seen the effects on employment, and we’ll have felt its acceleration in the economy – or not.

That’s a reasonable, and not outlandish scenario. But I offer an alternative scenario which I think we should also keep in mind: AI continues to surprise us at its core. As AI continues to evolve rapidly there will be no resolution to these questions in 3 years. By 2029, we still won’t know if AGI is possible, we can’t tell if employment is disrupted, and we still can’t say if it is worth the huge investment. I don’t mean AI progress stalls. I mean, AI continues to advance, but the new stuff doesn’t answer the old questions, it only expands our ignorance because the new is new in a new way. We have to alter our ideas (and measurements) of employment, we have to amend our concepts (and measurements) of the economy, and we have to shift our ideas of what AI even is.

In other words, we have a sustained, extended period of uncertainty. Not just a few years, but a decade or more. As AI continues to progress, rather than resolving our perplexity, it expands it. So for the next 10-15 years we have perpetual, continuous, severe uncertainty. This is a burdensome weight because people hate uncertainty more than bad news.

It goes deeper. AI is only one leg of this grand uncertainty. In the next decade the US will continue its slide off its pinnacle of a sole global superpower, while China continues to rise in power and prestige. This shift toward a duopoly prompts a new world order, and no one – especially the Chinese and Americans – knows how this will play out. The uncertainty around this shift is nearly boundless, and yet its indeterminate consequences will affect everyone in the world, but especially the US. Being dethroned from the century-long position of sole #1 will be a huge psychological blow, and the uncertainty of what follows will weigh heavy on all aspects of life. The uncertainty of a new role spreads over China as well, because while they are zooming ahead at 1,000 miles per hour, they have no idea where they are headed. The uncertainty of global relationships and new national identity, plus the uncertainty of individual worth and identity from AI increases the overall uncertainty levels to new highs. All this is a very large puzzle and will not be resolved in 3 years. This will be a sustained uncertainty.

It goes deeper still. After a long first wave of true globalization, there are now whirlpools of chaos and polarization as nations adjust to world-wide immigration and the borderless spread of modern culture, causing chaos in national politics, and sowing mistrust with the establishment. Anarchy, disruption, contrarian antics, blows to the states, seem to be the norm in countries all around the world. This wild chaos is being fueled in part by the new technologies of social media which have replaced the managed care of established media. News now is far more volatile, hard to control by anyone, and further elevates the already amplified uncertainty. There is a visceral sense that civics is headed into an unknown territory of near-permanent provisionalism.

Additionally, AI also forces even the most moderate person to question the truth of what they read, see or hear. Is that real or AI generated? How much has been manipulated? Who do you trust to disclose what is real? How do we come to agree that something is true? The traditional mechanisms of trust have been damaged by AI, so that this new technological realm generates a huge uncertainty. As AI gets more skilled at imitating reality, this uncertainty is likely to keep increasing for a while, and not just 3 years. The uncertainty meter is now deep in the red zone.

Finally, the ambiguity and indefinite nature of AI, or human identity, or whether what we see is real or generated, means that we are entering a period where we are even uncertain of our doubts. Our uncertainty is so deep and durable, yet elusive, that we will have extended uncertainty about whether we are uncertain. We can have major agreements on what we know versus what we don’t know. In the model of Rumsfeld’s Unknown Unknowns, we will be confronted by Uncertain Uncertainties. And they will prevail for at least a decade or more. [...]

Given the inherent unknowability of this era, what would some of the signs be that we are in it? They might look like this: in 5 years, 1) There are high-profile disagreements among leading AI researchers on whether AGI is here. 2) Reputable economists can’t determine if productivity has increased or decreased. 3) Lower public confidence in media platforms and established institutions. 4) The US and China cannot decide whether they are allies nor adversaries. 5) There are ambiguous spikes in employment rates in both directions. 6) Medical levels of anxiety increase. 7) Major court decisions leave as many questions as answers. 8) Commitments (marriage, work) are postponed even later in life. 9) Investing, capital allocation becomes more expensive. 10) Nihilism gets respect.

A great question to ask when creating a scenario is what could prevent it from happening? Maybe there is not a single force that can undo this sustained uncertainty, but perhaps it is a mixture of several. If AGI arrived without a doubt in 3 years and China took over Taiwan despite the US’s actions, and if companies found a way to embed reliability and trust in media, then maybe this extended uncertainty could cease.

A second question to ask, is if we find ourselves in this scenario, what should we do about it? The most effective response to this multi-layered persistent uncertainty is not to seek impossible stability, but to cultivate radical adaptability and radical optionality. Give up on having a reliable prediction of what happens next. Instead cultivate multiple scenarios of what could happen, and endeavor with each of them to maximize your options. Goals should be considered as disposable hypotheses, constantly ready to be discarded and replaced by better-fitting concepts later on. You will be dead wrong on 19 out of your 20 expectations, but at least one of them will allow you to proceed. Make your decisions not on whether they are “right” but on whether they tend to give you more options later.

In our era of uncertain uncertainty, certainty will be the killer. In this era more downfalls will happen because of overconfidence than questioning. The key is to not get stuck on just one option. You have to become at ease holding multiple contradictory possibilities at once. (To prevent yourself from being swept away by the latest current and fashionable whim, this radical adaptability must be anchored on a steadfast set of unchangeable virtues, as corny as honesty, or as slick as generosity.) The strategy for prospering in prolonged uncertainty must be one of constant, agile recalibration.

In short, in our age of uncertainty, you have to get good at changing your mind.

by Kevin Kelly, Substack |  Read more:

Image: uncredited

[ed. The diagnosis might be right but the prescription seems weak. Flexibility and adaptability are always good qualities to cultivate, but the challenges confronting us require more. Here's an example of embracing multiple contradictory possibilities: maybe in times of uncertainty we double down on the few things that we actually can be certain of. How? By making good choices, before and after AGI. For example, Buddhism starts with the acknowledgement that life is hard. It's what you do after internalizing that fact that matters. There are value systems and paths that can lead to a meaningful life, or enlightenment if you want to call it that, but we have to make the right choices if we're to find them. Love, family, friendships, ethical living (like the golden rule) are common values we all share. So why not embrace those values as tightly as we can while navigating the stormy seas to come - and using the best minds in the world (that are being born as we speak) to guide and assist us in strengthening those bonds? This might be one of the benefits of AI: forcing us to reorganize societies in ways that might never have been possible before, or even imaginable. If we make the right choices. Developing Plans A to Z and having 20 options each or something like that sounds like a Hunger Games scenario to me - all reaction and no responsibility. We have the opportunity now (even if forced) to redefine our human destiny. The choices we make will define our places in the future.]

We're Headed for a Depression Worse Than 2008; and Military Spending Isn't Helping

[ed. Which sounds like the good news if AI doesn't kill us all first.]

Michael Hudson and Radhika Desai discuss many of the long-operating forces that have been eating away at the foundations of the American economy, from its super-sized military to neoliberalism and financialization, now exhibiting many late-stage pathologies, from asset speculation to extreme wealth concentration. And as bad as those trends have been, Trump has succeeded in making them worse.

  

Michael Hudson and Radhika Desai examine the gap between Trump's promised economic boom and the reality of an economy sustained by asset bubbles rather than production. They show that the stock market's rise reflects cheap credit, buybacks, and speculation rather than profit, a Ponzi-scheme dynamic that cannot survive the oil shock triggered by Trump's war on Iran. His tariffs have not reversed deindustrialization, and his refusal to end that war guarantees the inflation he promised to kill. The result is a starkly K-shaped economy, in which the wealthiest 1% have seen their assets grow from $10 trillion to over $50 trillion in twenty-five years while the bottom half of Americans have gone from nothing to nothing, pushing the US toward a depression as serious as the 1930s and eroding even his own base's faith in him.

00:00 – Highlights 

01:12 – Channel introduction 

02:02 – Iran war escalation and the coming global oil/energy shock 

06:56 – The dollar's shift from treasury-based to stock market speculation (Ponzi bubble) 13:56 – China's stabilizing role vs. the Fed's inflation-vs-interest-rate gap 

25:06 – Inflation, GDP growth, and why "growth" numbers are largely fake (rents, fees) 34:00 – De-industrialization, manufacturing job losses, and shrinking labor force participation 

42:06 – The K-shaped economy: stock market boom vs. wealth inequality, and Trump's collapsing approval ratings  [...]

Radhika Desai:

Today we have decided to talk about the US economy under Trump. Now, of course, Trump is doing everything in his power to suck the oxygen out of the story about the economy. That includes continuing to escalate wars and all sorts of diplomatic and other shenanigans that he is constantly involved in, berating leaders of other countries and generally trying to make a big spectacle of himself... Anyway, all of these shenanigans are designed to distract attention from the biggest story, the condition of the US economy. Notwithstanding his unhinged and genocidal antics, the topic of the US economy simply will not go away. The world is settling down to summer before the midterm elections, and assessments of the US economy are proliferating. That is what we are going to talk about today, because the US voter votes on her economic condition, and her economic condition is not looking good at all. Michael, what are the headlines from your point of view?

Michael Hudson:

Well, the headline is really that the US economy is all about Donald Trump right now. The main thing, as you have just pointed out, is the war with Iran that he is escalating. Instead of rolling things back, he is bombing Iran. Iran has taken a response that is irreversible. It has closed down trade not only in the Strait of Hormuz but also in the Red Sea, with the Yemenis’ support, and it is bombing Bahrain’s port. It is absolutely certain that there is going to be a shutdown in the oil trade, and that is going to affect the entire world economy and push it into what I think is going to be as serious a depression as the 1930s. That is the US economy... It has been a huge expansion of financial wealth without any real expansion in living standards, real wages, or prosperity for most of the population. All of this financial wealth has been based on credit. Companies are not making more profits. The whole leadership of the stock market has been the seven AI companies linked to computers. AI is not making a profit; it is all speculation that we are going to expand and that there is going to be a huge market because everybody is going to use AI. But all of this market is dependent on computer chips that run on energy. We are going to see energy prices go way up, and that means electricity prices are going to go up. I want to briefly explain why the oil war is so important when we talk about the economy. [...]

Michael Hudson:

I want to explain just how that works. My whole premise in Super Imperialism is that after the United States went off the gold standard in 1971, all of this military spending, which is the major cause of the balance of payments deficit, ended up in foreign countries. The recipients took the dollars, turned them over to the central banks, and the central banks bought Treasury bonds and Treasury securities. What has happened in the last few years is that banks have stopped buying Treasury securities. The growth in international reserves has taken the form primarily of buying gold, not Treasury securities, and yet the US dollar remains strong. What has happened is that the private sector that has been receiving these dollars has not been turning them over to the government to recycle to the US as Treasuries, as you and I have been talking about. They have spent them into the US stock market. What has that done? It has inflated prices and ridden on the wave of the Federal Reserve supporting the banking system and creating huge asset price inflation, starting with the zero-interest-rate policy that Obama began. What has been creating all this financial wealth, making trillions of dollars for financial investors, has not been profits. It has been the ability to borrow at a low interest rate, including low interest rates in Japan, to buy US securities, bid up the prices of stocks, and create a huge credit overhead. So it is not a profit bubble; it is a Ponzi-scheme capital-gains bubble. It is a credit-creation bubble.

The problem is what happens when there aren’t the profits to support the stock buyback programs and the dividend payouts that have enabled borrowers to carry the debts they have taken on to bid up the stocks. If the credit begins to be rolled back here, and two weeks ago we talked about how Warsh and Bessent want to roll back the Federal Reserve’s balance sheet, they want to begin selling the Federal Reserve bonds that they have been buying in recent years to help liquefy the economy. The economy is going to be made much less liquid, and all of a sudden it is like a Ponzi scheme. A Ponzi scheme requires more and more people buying into it to provide the revenue to pay off investors. If interest rates go up, there is no more recycling of all this money into the stock market to help support things, and there is going to be a huge write-down. When that happens, stocks begin to fall. With higher energy prices, higher food prices, and higher costs of doing business, companies will go out of business. The Financial Times and main business sites have been saying the real problem is private equity. They have borrowed money from the banking system to buy companies, and now some of these companies are going to be running losses and closing down operations because it is not profitable to operate with high oil prices, high energy prices, and high electricity prices. Once they close down operations, they will not be able to pay the debts they have taken on, creating the same snowball effect that people expected in 2008–2009, when Obama decided on a bailout of the banks and decades of asset-price inflation to keep the bubble going and save the banking system. We are talking about the dynamics set in motion by the oil war, the rise in energy prices, and the AI demands for electricity that cannot be met because there is no electricity supply. All of this hopium has evaporated, and the result is going to be the serious depression we have been talking about all along. Yet the stock market idles along as if everything is all right. Our point is that these changes are irreversible. You cannot reverse a debt-inflated economy without wiping out the debts. How does that happen? Companies go bankrupt. There is not going to be a Brady Plan for the American economy. It is going to be companies going broke, and there will be a capital flight out of the dollar, not into the dollar. The whole world balance is being thrown out of kilter in a way that, unlike 2009, there is no monetary solution to a problem of actual physical supply of energy, electricity, oil, and chemicals not being available. This is the grand interruption that we are going to be talking about.

by Radhika Desai with Michael Hudson, Naked Capitalism |  Read more:

Image: YouTube

[ed. Not to mention the trillion bucks or so we're spending on the military each year, and the insane level of national debt that's accruing each day (nearly $40 trillion and counting (see this real time clock). The military's current arsenal is also severely depleted and will likely require even more big bucks to replenish expensive weapons systems (that are likely to be antiquated as soon as they're delivered as kinetic warfare rapidly shifts to AI controlled drones (NYT); see also: The US is Blowing Billions on the Wrong Weapons (Atlantic). Finally, remember that non-war we're not fighting? Iran War: Brief US Pause Followed by Renewed Strikes as Iran Intensifies Attacks on Bases and Kuwait Desalination; Continued Speculation About US Operation (NC):]

***

From Larry Johnson in The US Faces a Strategic Crisis with Iran…The Rapid Depletion of US Battlefield Missiles: [...]

These are not just the numbers currently in the hands of US CENTCOM (i.e., the US military command in charge of the war against Iran), these are the total numbers available to all of the US military commands. If these missiles are allocated evenly to the other two critical commands — i.e., EUCOM (European Command) and PACOM (Pacific Command) — then you begin to understand the gravity of this deficit.

Let’s take the case of the Tomahawk missile. Let’s assume there are 3,000 left (I believe that is a generous over estimate) and the remaining number are divided evenly among CENTCOM, EUCOM and PACOM… That means each command gets 1,000. Does anyone want to argue that in the event of a hot war with Russia or China that EUCOM and PACOM respectively would be able to sustain combat operations for more than four weeks? Hell, CENTCOM fired 850 of them during the first four weeks of EPIC FURY.

Here’s another major problem: All eight missile systems rely on rare earth elements — there are no exceptions among modern US precision-guided weapons. The dependence is nearly universal because rare earth permanent magnets are irreplaceable for the high-performance actuators, guidance motors, and seeker gimbals that make these weapons accurate. And who controls the supply chain of these rare earth minerals? China!…

The supply chain isn’t just about mining — it’s about processing, separation, and magnet manufacturing, which China controls:

Mining: China ~60% of global rare earth oxide production
Refining/Separation: China ~91%
Sintered NdFeB Magnet Manufacturing: China **~94%**

The New Coming Age

Google CEO Demis Hassabis offered us a first rate second rate essay, A Framework for Frontier AI and the Dawning of a New Age. I’ll go over that essay and various responses to it in Part 1.

Part 2 of this post then covers Alex Turner’s resignation, and his story about how he tried and failed to prevent Google from signing up to allow the Department of War to use its models for essentially whatever the government wants, including autonomous weapons.

Demis Hassabis sold DeepMind to Google on condition that something like this would not happen. Yet here it is, happening. A cautionary tale. [...]

***

The Core Statement and Request

He saying we are standing in the foothills of the singularity.

His ask is a Frontier AI Standards Body within the US Government, similar to FINRA, that would govern ‘frontier labs,’ defined as any company that produces a frontier model based on various technical benchmarks. Evaluations would be updated regularly, and vulnerabilities would be addressed, both before and after release.

He is excellent about stating that this is big, really big, no bigger than that, it be big.

Demis Hassabis: I’ve spent my whole life working on AGI because I’ve always had a deep conviction that, if built and deployed responsibly, it would prove to be one of the most beneficial and transformative technologies ever invented. AGI cannot be compared to standard technological breakthroughs, not even ones as consequential as the internet or mobile - it is much more akin to the discovery of electricity or fire. If you stop to think about it, we’ve essentially found a way to make sand think. It’s miraculous.

The magnitude of this technology’s impact will be unprecedented, perhaps 10x of the Industrial Revolution at 10x the speed. It will help us solve some of the biggest problems society faces from accelerating drug discovery to developing new clean energy sources to creating novel advanced materials. We could even reach a point where resources are no longer the limiting factor for human progress, leading to an amazing new era of abundance.

Things Left Unsaid

There is definitely a ‘don’t say the thing’ aspect of this, where he won’t name what the downside risks actually are. When Demis says ‘experts disagree’ he is rather avoidant about the way in which they disagree here.

Nate Soares (MIRI): I’m glad Demis acknowledges that this is a “pivotal moment in human history” during an “extremely intense” race. I’m disappointed that his proposed solution is a “standards body” to evaluate whether models are dangerous, with no plan for what to do once they are.

I’m glad he acknowledges that “experts disagree.” I’m annoyed that he glosses past how the disagreement is about whether there’s a ~5% or ≥50% chance of total catastrophe. We’ve gotta do better.

Aaron Scher: Glad to see AI CEOs speaking publicly about their views on AGI. I think Demis is wrong about his policy prescription: it’s far too little too late. When he says the experts disagree, he means that some think 5% this tech kills literally everybody, some at 40%, some at 90%.

Clearly this is strategic, but if you don’t already know, or are looking to not realize, it is very easy to come away thinking that Demis does mean the effect on jobs, even though when he says ‘safely’ he very much does not (primarily) mean that.

The Proposal

Demis Hassabis: … On the horizon, we will need robust safeguards to maintain control of increasingly agentic, recursively self-improving systems - and tackle unknown issues that will only become clearer over time.

… I’ve always believed in the power of human ingenuity and creativity to solve any problem. I’m confident that mitigating the technical risks related to AI is a challenge we can collectively address, but only if we give ourselves the time and space to get this next crucial step right. Currently, as a field and as a wider society, we aren’t doing that.

He makes clear part of this is about giving us options, including for a slowdown.

The strength of this approach is it would be technically focused, while at the same time supporting innovation and incentivising responsible behaviour. It is designed to keep up with the field’s acceleration and adapt to the biggest risks as they are identified, and could be ratcheted up if the seriousness of the situation demands, including coordinating a slowdown in development among the Frontier Labs if deemed necessary.

Demis keeps it short, not offering many details. To the extent that he has laid out a proposal, it seems to be a good one. It is definitely an improvement on the margin.

Jack Clark (Anthropic): At this point, everyone at the frontier of AI agrees that third-parties should test out AI systems and use these to develop standards to feed into policy - excellent to see @demishassabis laying out a framework to do this!

Samuel Hammond: It is striking to see leadership at Google, Anthropic, OpenAI and Microsoft all fairly independently sounding warning alarms about an imminent technological acceleration.

Thus I file this post and its ask, as high praise, under ‘the least you could do.’

A Good Start But Insufficient

I agree with Peter Wildeford that while better than nothing FINRA is not a great model here, with heightened risk of regulatory capture, and not a substitute for full government action. You need an SEC to your FINRA. That doesn’t mean don’t make the FINRA. It does mean you still need the SEC.

Would such a (at least partly) voluntary regime, only for models intended for release, and without a related binding intentional agreement, be sufficient to solve the problem? No, again it’s just way better than doing nothing, as Peter Wildeford and many others noted.

You do not need to believe, as Aaron Scher and Connor Leahy do below, that only a full halt would be sufficient here, to know we have a long way to go. Demis’s statements here, if you know what they actually mean, imply a level of danger and urgency that is not reflected in the proposal.

Eli Tyre: > Initially, Frontier Labs would voluntarily share models with the Standards Body for review up to 30 days before release.

Is this proposal only intended to address risks from models that companies plan to release? If a company develops a frontier model and never releases it, only deploying it internally to develop even more powerful AI capabilities, are they thereby exempt from this oversight scheme?

Connor Leahy: While @demishassabis is right that we need urgent action to address risks as we approach AGI (and superintelligence, I’d add), the correct response to the threats is not a ‘self-regulatory organization’.

We need to prohibit superintelligence, not give industry regulatory power.

Aaron Scher: … The extinction threat, the “only a few short years”, the “10x the Industrial Revolution”—these aren’t indicators that point to “let’s evaluate models to understand their capabilities and have voluntary safety standards”. We need to back off, we need to halt the creation of ASI.

Point 2: I agree with the attached quote that we need more time. But I think Demis’s optimism is a vibe, not a trustworthy basis for predictions. Rob Miles says it best in this video, if an asteroid we’re headed earth’s way 200 years ago, we’d just die 🤷

Point 3: As others have pointed out, it’s not clear that this proposal would reduce risks from internal deployment (it seems to focus on public deployment and pre-deployment testing), but internal deployment is where much of the risk is.

Point 4: I don’t think the proposed body could actually enact, verify, and enforce a slowdown; there’s ambiguity about what’s voluntary. Again, I think we need a long-term international treaty and to actually back off, not just to slow down a little.

by Zvi Mowshowitz, DWV |  Read more:

Image: uncredited

[ed. See also: The Voice of Google (New Yorker):]

***

I started working at Google in the summer of 2007, straight out of college, as a “new-­grad associate” in the communications department. My first week, I sat with more than a hundred other “Nooglers” (new Googlers) at the company’s weekly staff meeting, T.G.I.F., wearing matching company-issued propeller caps as a kind of ritual hazing. The venue was Charlie’s Cafe, a multilevel auditorium in the heart of the “Googleplex,” the company’s sprawling campus in Mountain View, California. The event felt less like a corporate meeting than like a weekly revival—part stand­up set, part science fair, part sermon, all of it fuelled by keg beer.

Google’s founders, Larry Page and Sergey Brin, were bona-fide public figures by then, and self-­made billionaires multiple times over, but in Charlie’s they were idols. They would often ascend the stage together, practically matching in sweat-wicking athletic clothes and Crocs. Larry had a dopey perma-smile, and seemed delighted by everything, especially Sergey. Sergey was the straight man, with a faint lilt, a product of his childhood in Russia, and an acrobatic build that made him look like he might launch into a handspring at any moment. Their charisma was unconventional, contextual; you had to be there. The audience of employees lapped up every word, giggled at every dad joke. During a Q. & A. portion of the proceedings, even adversarial questions were absorbed into the Google spirit—­it all melted into laughs, love. Merriam-­Webster had added “google” to the dictionary the year before. Fortune had crowned it the “Best Company to Work For” in America. Profits were, as the execs loved to boast, “up and to the right,” fuelled by an online-advertising machine that minted cash beyond Wall Street’s wildest dreams. But the company’s financial success felt almost incidental. What mattered, we told ourselves, was the mission—a conviction that technology could improve the world and that we were helping to build the future. The air in Charlie’s buzzed with collective belief.

That first meeting was the only one I’d ever attend as a pure spectator. By week two, I was working the event—­cordoning off the Noogler section, handing out extra caps—and I soon began helping to draft bits of Larry and Sergey’s script. A portion of my time was spent supporting the P.R. team, and I started to pick up my first press requests, providing office tours to journalists eager to see the “Google experience” firsthand. I studied a “master workplace talking points” document, which was maintained with input from PeopleOps, which was Google-speak for human resources. This was the era of “bringing your whole self to work,” of shiny, smiling H.R. people doing press hits about the importance of valuing employees’ authentic personhood (always with a telling corollary: “Because that’s how people do their best work!”). I was required to attend a training on “conscious business” with a guy named Fred Kofman, an executive coach whom Sheryl Sandberg credited with shaping her “lean-in” ethos. The course was, theoretically, about living one’s courageous values, but its most salient lesson was that employees should take “unconditional accountability”—which, in practice, sounded a lot like never questioning the higher-ups. The message reiterated over and over was that there were two kinds of people in the world: victims and players. You wanted to be a player at all times.

Despite the lore, Google’s offices didn’t make a big first impression. The bulk of the campus had been quickly converted after its previous occupant went down in the fallout from the dot-­com bust. The result was a complex of squat, one-­ or two-level buildings with metal and glass siding, surrounded by a moat of parking spaces, with Google signs plunked into the dirt out front. But there were plenty of amenities to point out—­the massage rooms and nap pods, the dinosaur fossil, the wacky sensory-­break touches like ball pits, swings, and yoga balls (even if no one actually seemed to use them). Foreign journalists seemed more skeptical than their American counterparts of perks such as lunch-­break haircuts or on-site laundry rooms, which I’d heard described as letting Google be your “housewife.”

“Z is is all a big plot to control ze workers, no?” a French reporter said.

At that point, though, I was still learning to see Google through Google’s eyes. I learned to deflect these kinds of questions and pitied the askers, a little bit, for their cynicism.

Xi Gives A Good Speech on AI

I will share the full transcript, as it is short and worth reading, with brief comments.

It is a good speech. Video is here.

We start with the opening section, which frames the situation.

Xi Jinping: Distinguished colleagues and guests, ladies and gentlemen, friends,

70 years ago, a group of young scholars proposed the concept of artificial intelligence for the first time at the Dartmouth workshop in New Hampshire of the United States. In the subsequent 70 years, AI scientists and researchers from around the world ventured into this unknown territory, forged ahead through twists and turns, and made breakthroughs with persistent hard work.

Seven decades later today, amid the new wave of AI development, we are gathering by the Huangpu River to discuss how to promote AI globally for the positive, for good, and for humanity. All this makes our meeting highly important. On behalf of the Chinese government and people, I would like to extend a warm welcome to you all.

In the course of history, the invention of the steam engine heralded the industrial civilization. The widespread access to electricity brightened up modern society and the birth of the internet brought the entire world together. Each of these technological revolutions has profoundly reshaped our way of work and life and enabled a giant leap in economic and social development.

Today, major changes unseen in a century are accelerating across the world. The new round of technological revolution and industrial transformation is advancing at a faster pace. And the world has entered an unprecedented period of active innovation on AI technologies. Intelligent connectivity, human machine collaboration, cross-sector integration, joint creation and sharing and other intelligent technologies are unleashing enormous power.

Next Xi lays out the challenges. Note what is here and what is missing.

All this carries within it great opportunities as well as challenges to governance. We human beings must answer the questions posed by our times. How to get along with thinking machines? How to ensure security when algorithms are part of decision making? How to tackle ethical challenges by technologies through adaptive governance. How to realize AI for all when the divide keeps widening? These questions demand serious consideration and real answers from the whole international community.

‘How to get along with thinking machines’ is quite the line to include here. A lot of this seems directionally serious but confused in its details.

Existential or catastrophic risk does not get a name check, but the related issues are clearly not being ignored.

So, what to do about it?

In China's view, all countries should take a people-centered approach and develop AI for the positive and for good. We should ensure that AI is an important driver for shared prosperity and common security. We should join hands to build a just and equitable system for global AI governance. To this end, I wish to share four observations.

A system for global AI governance. Sounds like deals could be made, on various fronts.

What are the proposals?

First, we should adhere to the principle of openness and win-win and boost innovation-driven development as a new engine of world economic growth and an accelerator for the shift of growth drivers. AI is moving from the digital world into the physical world. We should seize this rare historic opportunity to encourage open source, openness, collaboration and sharing. We should facilitate technological innovation, industrial development and scenario-based application of AI. We should make coordinated advances in the transformation and upgrade of traditional industries, the cultivation and growth of emerging industries, and forward-looking planning for future industries so that all sectors and businesses can benefit from AI.

There are two things here.

1. When we are behind, we encourage everyone to share, so that we might catch up and score the aura points of being the ones who claim openness and sharing.

2. We should diffuse AI technology, including via openness.

Second, we should strengthen risk awareness and ensure that AI is secure and controllable. AI should be a trusted tool for humanity. We should take seriously the various types of inherent and secondary risks that AI may trigger.

We should put in place laws and regulations, technological monitoring, early warning and emergency response systems in order to strengthen the line of security, prevent abuses and malicious use and ensure that AI is always under human control.

In the meantime, we should jointly oppose overstretching the national security concept in the field of AI or placing one country's security over that of others.

Strong emphasis on the need to ensure AI remains secure and under human control. Not party or national control, but human control. This is how he understands the existential risk and other big problems.

It makes sense that the CCP’s ultimate need and answer is control. For now open weights is compatible with their control, so they continue this strategy. For now.  [...]

Malicious use is also a threat, but Xi realizes this is secondary, whereas the United States government is stuck at thinking misuse is the primary threat.

The call to not focus on national security over world security also makes sense. Yes, of course some of this is ‘when I am behind I call for equality’ but also we are all in this together and need to act like it.

Third, we should encourage inclusiveness and promote mutual learning between civilizations. AI development and its application should not erode or undermine the diversity of world civilizations or the uniqueness of cultures of different countries. We must shape the values of AI with humanity's common values and make good use of AI technologies to increase understanding, tolerance, exchanges, and sharing among all civilizations. We should tend to the garden of civilizations with great care to ensure that the beauty of each civilization is appreciated and shared.

General call for cooperation and good relations. Good. Pick up the phone.

Fourth, we should advocate solidarity and improve global governance. AI is an invaluable asset that encapsulates humanity's collective wisdom. We should practice true multilateralism and recognize the important role of the United Nations.

We should enhance alignment and coordination on AI development strategies, governance rules and technical standards so as to form a consensus based global governance framework at an early date to make this frontier technology better benefit humanity.

We must carry out extensive international cooperation and help global south countries with capacity building to bridge the AI and digital divides, promote sustainable development and prevent creating new historical injustice in AI.

The primary ask is an explicit request for a global governance framework and international cooperation. Excellent. Let’s get to work on that.

Nate Soares (MIRI): Xi Jinping: "With AI advancing at a staggering speed, we must [...] constantly refine measures to forestall loss-of-control." Can we stop pretending there's no hope of international coordination now?

Jack: Lots to chew on in here. The US may have the best labs, but it's pretty clear that, on the government/policy level, China is approaching AI much more seriously than the US is – and in a way more likely to be viewed favorably around the globe. Worth a close read.

No doubt they have in mind something that would favor their position, and their initial asks will look outrageous and unacceptable to us. That is how this works. You do not accept their first offer, and things take time, and sometimes it turns out there is no deal to be made.

It is also possible Xi is engaging in cheap talk. Why not propose such things and aura farm, whether or not you intend to follow through on reasonable terms?

The way you find out and do your best with this opportunity is: You get started now.

Xi Jinping: Ladies and gentlemen, friends,

This year marks the start of China's 15th 5-year plan. It maps out China's economic and social development for the next five years and provides immense opportunities for the international community.

In recent years, China has embraced AI with open arms. We have promoted interplay between an efficient market and a well functioning government, strengthened AI innovation, actively advanced the AI plus initiative and built a healthy ecosystem for all entities to thrive in together. The core smart economy industries are worth at least 1 trillion RMB yuan. Smart devices in countless homes truly improve people's livelihood. Intelligent manufacturing in China has become another shining hallmark of Chinese modernization.

At the same time, China lays great emphasis on safety and security in AI development with a deep understanding of the trends and logic of AI development. We are continuously improving laws, regulations, policies, mechanisms, application norms as well as ethical principles to make sure that AI is safe, secure, and controllable, and that this fine steed of AI gallops with both speed and stability.

As a responsible major country, China is always committed to providing international public goods relating to AI. Since I proposed the global AI governance initiative, China has promoted the adoption of the UN General Assembly resolution on enhancing international cooperation on capacity building of artificial intelligence by consensus. Published the AI capacity building action plan for good and for all. Announced the AI plus international cooperation initiative and advocated for establishing the world artificial intelligence cooperation organization, or WAICO. China has been contributing steadily to the global AI governance.

We often say in China, a single string cannot make music and a single tree does not make a forest. AI development should not be a solo performance by a single country but a symphony of international cooperation.

Thanks to our joint efforts, WAICO has come into being in Shanghai. Our vision from one year ago is now a reality. This is a major move by China to answer the call of the global south and unite the international community together to promote vigorously AI development and governance. It will be an important milestone in the history of AI development to further support global AI development and to advance global AI capacity building.

I hereby announce that in the next five years, China will provide developing countries with 5,000 opportunities in AI training and seminar programs. China will develop international AI application cooperation centers with ASEAN, the League of Arab States, the African Union, the Community of Latin American and Caribbean States, the Shanghai Cooperation Organization and BRICS. And we will enable 30 countries to use the AI-powered meteorological warning system, Mazu, to safeguard homes around the world.

Ladies and gentlemen, friends,

As ancient Chinese observed, a man of wisdom adapts to changes. A man of knowledge acts by circumstances. With AI advancing at a staggering speed, we must ensure its development is for the positive, for good, and for humanity. We must make its oversight and governance precise and effective and constantly refine measures to forestall loss of control. We should always guide AI development with human wisdom and international consensus so that AI can truly become a mighty force that increases the well-being of humanity and advances human civilization.

China is ready to be more open, take more practical actions and assume a more visionary perspective. We are ready to work with all parties to seize the opportunities of AI development and meet the challenges and join hands to create a brighter future for humanity.

Thank you.

I found this to be a strong speech, and a good one to give in China’s position, both on the importance of diffusion and the need for international cooperation to prevent loss of control. Yes, there was talk about openness and potential new ‘injustice’ and such but this talk is to be expected from their position. It is now on America to make the next move.

by Zvi Mowshowitz, DWV |  Read more:

Image: Elena Kadvany/S.F. Chronicle
[ed. "It is now on America to make the next move". Which is precisely what should have happened months/years ago when the pace of model development began to accelerate with no significant oversight or regulatory constraints ie., treating AI with the importance you'd give to any new technology that represents an existential threat. Unfortunately, with our dysfunctional Congress and the current bozos in the White House whatever action they take (if any) has a strong likelihood of doing more harm than good. Still it's imperative we get started somewhere soon and there are already good proposals out there (see Plan A), so authorize someone to start doing something.]

All Good Sex Is Body Horror

Regenerations as radical as the ones Cronenberg envisions involve what the philosopher L. A. Paul has termed “transformative experiences,” ruptures that change “your own point of view so much and so deeply that, before you’ve had that experience, you can’t know what it is going to be like to be you after the experience.” Not only do we lack access to information that we can acquire only by plunging into the scalding water of a new life, but we cannot foresee how such a jolt will overhaul the very predilections and values that define who we are. [...]

It is a commonplace in the literature of romance that love wreaks legible changes on the body, a development that is typically painted in a positive light. As the poet Octavio Paz so tenderly puts it, “My hands / Invent another body for your body.” He is echoed by E. E. Cummings in a similar poem, which opens, “i like my body when it is with your / body. It is so quite new a thing. / Muscles better and nerves more.” Cummings is not the only one to undergo a shift during the act of love, and the full extent of his metamorphosis is explicable only in terms of his partner’s reciprocal mutation: his poem begins with the ways in which his lover renews his body and ends with the ways in which his body renews his lover’s body in turn. “i like the thrill / of under me you so quite new,” he concludes. The poet’s body changes in response to his lover’s body, his lover’s body changes in response to the changes in his body, his changed body changes in response to the changes in his lover’s body, and so on and on, twining into an ouroboros of mutual reconstruction. [...]

Most people would give anything to be turned into anything else, because most sex is mediocre, and the measure of its mediocrity is that it leaves us unaffected. No one falls ill; no one transforms into a fly or a cockroach; nothing changes. As the narrator of Norman Rush’s novel “Mating” sagely observes, “Sex can be various things, but in my experience the usual thing it is is considerate work on the part of both parties,” accompanied by the exchange of careful courtesies: “after you, no, after you, mais non.” No one has transformative sex all the time, and there is nothing wrong with sex that is merely pleasant. Indeed, a polite volley of pleasantries is probably the best thing that unecstatic sex can be.

Of course, many mediocre sexual encounters are rote in a more pernicious way. Heterosexual sex that follows the standard scripts, with its spankings and its schoolgirls, is not always devastating or traumatic, but its tiresomeness is nonetheless not innocuous. Women are the most obvious losers when the scenarios faithfully reënacted in the bedroom so consistently favor male predilections, but men who inherit their desires from the prevailing sexual culture—or, worse, men who feel they must satisfy a virile masculine ideal whether it appeals or not—lack the opportunity or the means to develop sexual agency. For both parties, the resultant comedy of errors is not satisfying. What nefariously underwhelming sex has in common with respectfully underwhelming sex is that neither brand is especially surprising or especially erotic.

To have sex erotically—and ethically—is to have it with someone else, and a person demonstrates her difference from the self by being impossible to predict, domesticate, or assimilate to preëxistent fantasy. It is not erotic to impose a ready-made desire onto someone pliant, or to slot her into a fetish that has little to do with her. Eroticism occurs only when someone rewrites us so completely that she rewrites even the quality and content of our appetites, and only when this radical rewriting is reciprocal.

Eroticism does not arise every time there is sexual activity, no matter how plodding, but it is also not the exclusive concomitant of love, marriage, or conventional commitment. Most sexual pairings are no more dishevelling than a game of tennis, but it is constitutive of sex that it has the potential to thrust us into metamorphosis that may be sweet, may be sinister, and may be both concurrently. When at last we grow wings, who can say exactly where we will want to fly?

Can a person consent to dying? Can she consent to a complete renewal, which amounts to the same thing?

Surely she cannot consent in the normal way. To consent in the normal way is not merely to grant permission but to grant permission on a particular basis—perhaps a reasonable expectation of pleasure, security, or safety. In any case, there is some positive inner state to which the outward utterance of license is supposed to correspond. A woman, almost always the presumed consenter in a heterosexual exchange, is exhorted to have sex with someone only when she has good reason to believe that she will have a generally happy time with him. Over and over, she is told: you don’t have to do anything you don’t want to do; you don’t have to do anything that makes you uncomfortable. Because she is assumed to know exactly what makes her comfortable, and because comfort is assumed to be a necessary condition of good sex, the procedure the consenter is instructed to follow is not unlike the operation favored by decision theorists. First, she is to imagine what sex with the partner under consideration will be like; second, she is to assign a value to the experience that she has conjured; third, she is to assign a probability of accuracy to her prediction. Having done all this, she finds herself in a position to make a rational decision, consenting if and only if she foresees that an exchange will turn out to be unimpeachably agreeable. Should she prove wrong in her predictions, should she ever feel the slightest scintilla of unease, she should withdraw her consent and beat a hasty retreat.

Three faulty assumptions are baked into this model. The first is that people in general, thus women in particular, can predict, if roughly, what a prospective partner will be like during sex; the second is that people in general, thus women in particular, can predict what they will be like during sex with a prospective partner; the third is that sex can and should be comfortable.

In fact, we are not impermeable packages of preformed desires, importing our likes and dislikes around with us from one encounter to the next like papers in a briefcase. An erotic craving is inextricable from the ferment that foams up when oneself is sluiced into another. Not only is it impossible for us to know whether an encounter will be deflating or transformative but we cannot know what sort of metamorphosis will ensue if the sex is as jarring as we can only hope it will be. We can have no more success when it comes to divining how we will change our partners than we can have when it comes to divining how they will change us—or, following Cummings, how their changes will change us, and how our changes will change them, iteratively and indefinitely. Maybe we will grow the wings of cherubs, but maybe we will find ourselves meshed with the coarse bristles of gigantic flies. All we can say with certainty is that sometimes, when it is working, sex carves out new bodies for our bodies, and these bodies can be both better and more brutal than the ones we could invent alone.

From Cronenberg’s fever dreams, we can surmise that there is a further reason to reject the decision-theoretic model of consent: not only is it impossible for us to know what we will become if an erotic encounter is transformative but we should not want to. To determine in advance what a transformative experience will churn into existence is to sap its power, for the very essence of transformative experience is that we cannot predict how it will transform us. To be sure, it is uncomfortable to stand on the precipice of metamorphosis, but unless we are willing to assume genuine risk we cannot be undone and remade.

Writing of the conservative fear of sexual deviance, the feminist cultural anthropologist Gayle Rubin notes that we are bent on drawing a line between “sexual order and chaos.” The line in question “expresses the fear that if anything is permitted to cross this erotic DMZ, the barrier against scary sex will crumble and something unspeakable will skitter across.” Conservatives are right. If the gender binary melts away, if heterosexuality is no longer the default assumption, if parasites enter the building and dismantle the tenants, something unspeakable will skitter across—and that is the point of any erotic effusion worth pursuing.

by Becca Rothfeld, New Yorker | Read more:
Image: SillDA

[ed. On the movies of David Cronenberg; body horror and eroticism.]

The Looting of Science Fiction

Tech titans claim the genre inspired them. But all they’ve done is graft their politics onto stories of a better future.

In January 2026, Elon Musk stood before the US Secretary of Defense and senior Pentagon leaders at the SpaceX Starbase in Texas. ‘We want to make Star Trek real, OK?’ he declared. ‘We want to make Starfleet Academy real. So that it’s not always science fiction, but one day the science fiction turns to science fact, and we have spaceships going through space. Big spaceships!’ He painted a vivid picture: exploring alien civilisations, humanity spreading across the stars. ‘That’s the goal!’ he concluded. ‘And that is what I think the public thinks of when they think of Space Force!’

It was a remarkable pitch selling the Pentagon a science-fiction vision. Of course, the fit is partial, incomplete. Star Trek depicts a post-scarcity, post-capitalist society where money has been abolished and humanity works toward collective betterment. Gene Roddenberry’s Federation was built on principles of equality and exploration for the sake of knowledge, not profit or military dominance. Musk took the aesthetic – big spaceships, alien encounters, epic adventures – and left its political foundation. You don’t have to be a Trekkie to know that, in Star Trek, capitalism, nationalism and militarism have been left behind. Musk wants the Enterprise, but reimagined for the military-industrial complex.

In 2021, when Mark Zuckerberg announced Facebook’s rebrand to ‘Meta’, he took the name from Neal Stephenson’s novel Snow Crash (1992), which imagines the ‘Metaverse’: a virtual reality where people’s avatars navigate digital space.

But Snow Crash is one of the sharpest satirical novels of the past half-century. Stephenson wrote it as a warning: his Metaverse is a consolation prize for a society that has collapsed. The federal government has disintegrated; corporate franchises govern daily life; even pizza delivery has been privatised into a Mafia-run operation. The novel’s protagonist is a pizza deliveryman and part-time hacker whose sword-fighting avatar in the virtual world is the only place where dignity is available to him. Stephenson intended the contrast between digital glamour and material poverty to be horrifying. He saw it as a cautionary vision of where platform capitalism leads.

Zuckerberg’s presentation did not engage with any of this. The platform economy – where corporations are protected from democratic accountability while providing essential services – echoes Stephenson’s model precisely, and Zuckerberg read it as inspiration.

Steve Wozniak, Apple’s co-founder, gave expression to this ethos in 2017 when he said: ‘We are the people who make fantasies real.’ It sounds inspiring, but it is important to know which parts of those fantasies they’re choosing, and which parts they’re leaving out. When Musk unveiled Tesla’s Cybertruck in 2019, he had already told investors what to expect: something ‘really futuristic, like cyberpunk Blade Runner’. Musk was selling survival gear for a collapsing world, a version of Blade Runner’s Los Angeles. The aesthetics got materialised. The warnings did not. [...]

Science fiction in the 1950s imagined flying cars, abundant energy and more – but it did so before Three Mile Island, before Chernobyl, before we learned, often through disaster, what happens when you prioritise speed over safety. Those regulatory frameworks Andreessen wants to demolish emerged from hard-won lessons. The optimistic aesthetic gets borrowed; the learning gets discarded.

Perhaps nowhere is this more legible than in the naming of Palantir Technologies. J R R Tolkien’s The Lord of the Rings (1937-49) is one of the great works of 20th-century literature precisely because it is an extended meditation on the corrupting nature of power. Written in the shadow of industrialised warfare and imperial extraction, it insists on the value of the small, the local and the unglamorous against the totalising ambition of industrial force. Its central moral is not that evil can be defeated by the right hero wielding the right weapon – it is that power itself corrupts, that the Ring cannot be used for good by anyone, and that the only salvation lies in relinquishing the will to dominate entirely. Tolkien’s fictional race of hobbits prevail not because they are powerful but because they are outside the logic of power. Tolkien built an entire mythology to make that argument.

In Tolkien’s novels, the palantíri are seeing-stones or crystal balls that allow their users to see across great distances. They sound like neutral tools, like surveillance technology. But they are devices of corruption: Saruman’s palantír connects him to Sauron and leads to his downfall; Denethor’s drives him to madness and suicide. The palantíri don’t just enable seeing – they enable manipulation and control by those who master them.

The US company Palantir Technologies provides analytics and surveillance tools to governments, militaries and ICE, US immigration and customs enforcement. Its name does political work: it transforms invasive tracking into mystical insight, casting algorithmic surveillance as wise foresight rather than systematic intrusion. In their book The Technological Republic (2025), Palantir’s CEO Alexander Karp and his legal counsel Nicholas Zamiska frame Palantir’s government work in martial terms: ‘We will find a way to build coalitions and bands of warriors. To deny the human need for such affiliation has been a mistake.’ They position surveillance tools as fulfilling a fundamental human need for warrior brotherhood. The Tolkien reference provides the aesthetic authority; the distance from Tolkien’s actual moral vision provides the freedom to act without it. Naming a surveillance company after devices that corrupt and betray their users isn’t homage – it is the appropriation of aesthetic while rejecting the moral core.

William Gibson’s novel Neuromancer (1984) introduced ‘cyberspace’, a term Gibson coined. Its protagonist, Case, is a hacker whose nervous system was damaged by former employers as punishment. Burned out and banned from cyberspace, he drifts through neon-soaked Chiba City as a ‘console cowboy’ with nowhere left to go. The novel’s cyberspace is owned and controlled by vast corporations; individual hackers are not heroes but tools, hired and discarded by interests they can barely see. Gibson’s vision was explicitly dystopian: a world in which the democratising potential of digital networks had been foreclosed before it could begin, captured by capital and turned into an instrument of its own expansion.

In September 1988, the software developer John Walker wrote an Autodesk internal white paper, ‘Through the Looking Glass: Beyond “User Interfaces”’, in which he outlined what he called a ‘cyberpunk initiative’: a proposal to build, within 12 months, a doorway into cyberspace. The project’s motto was blunt: ‘Reality isn’t enough any more.’

By 2025, the San Francisco headquarters of OpenAI pump high-energy electronic dance music across their reception area, where easy chairs, scatter cushions and Swiss cheese plants create what the CEO Sam Altman calls a ‘comfortable country house’ rather than a ‘corporate sci-fi castle’. The chrome and grime of cyberpunk – the neon-soaked warning that the corporate capture of digital space would be brutal and dehumanising – has been replaced by Scandinavian furniture and artisanal coffee. Gibson’s ‘consensual hallucination’ has been rebranded as cozy domesticity. The dystopia has not been avoided; it has been made comfortable enough to sign up for.

Gibson himself registered the irony. In an interview with Wired magazine in 2012, he acknowledged that the cyberspace of Neuromancer – all corporate interests and information thieves – bore little resemblance to the early internet he failed to anticipate: the 1990s-2000s moment when a teenager in a bedroom could genuinely outcompete corporations, when the network felt briefly open and democratic. Gibson missed that phase entirely. But he was accidentally right about where things ended up. The corporate platforms – Google, Meta, Amazon – that now dominate digital life are far closer to his original vision than to the participatory web that briefly flourished between them. Gibson imagined cyberspace as a space of corporate dominance from the start; Silicon Valley built the open internet first, then converged on his dystopia anyway. The difference is that, in Neuromancer, that convergence was the disaster to be resisted. They turned his warning into a product roadmap.

by Ali Rıza Taşkale, Aeon | Read more:
Image: Amazon

The Decline of Deviance 2

In The Decline of Deviance, I argued that:

Many forms of risk-taking and rule-breaking have declined since the 1990s.

This is both good (less crime) and bad (less innovation).

Deviance is declining because prosperity has increased—people have more to lose, and they’re acting like it.

Since I put it out last October, this has become my most-read post of all time. Lots of folks have chimed in with hypotheses, critiques, and good old-fashioned internet dunks, so let’s see if we can take this idea one level deeper, and weave something out of all these loose threads.

That’s right: we’re doing a sequel to the post about how there are too many sequels.

THE KIDS STOPPED SMOKING AND NOBODY CARED

The most common response I got was: “how awful!”

(The trends, not the post itself, although that too sometimes.)

To most folks, the decline of deviance is a turn for the worst. Previous generations got to live through artistic golden ages, while we have to suffer through an era where there are four different movies that are all, technically, Spider-Man 2.[...]

But this response misses half the post, and the most important point.

In 1995, half of high school students drank, 35% smoked, and 40% had at least tried weed. 10% of them had brought a weapon to school at some point. About 6% of girls aged 15-19 were pregnant. Crime rates were about as high as they had been since we started keeping track of them.

Over the next 30 years, all of these problems shrank and some of them nearly disappeared. And not because of anything we did on purpose! We have no idea how to get kids to stop smoking—when we try to persuade them, we sometimes cause them to smoke more. No, these improvements happened basically by magic, for free, and—I think—as a byproduct of our increasing prosperity. This is like waking up one day to find that you’ve been left a large fortune by a long-lost aunt.

How does this great, unearned victory make us feel? Apparently, it doesn’t make us feel anything. We would have spent billions to solve all of these problems back in the 1990s—no doubt we were spending considerable sums of money on anti-drug programs and public service announcements, wasting almost all of it—and yet when we got the thing we wanted so badly, we didn’t even notice.

Now we’re on to worrying about whether the kids are too sad, whether they play outside enough, etc. Which is all reasonable and fine, but also, can we take a win? Can we at least try to reverse-engineer how some of these trendlines ended up going in our preferred directions, so we can make more of them do that?

I, too, would like to see a movie that is not Spider-Man 2. But I also appreciate that I can watch it safely in my home without fear of being attacked by pregnant teenage criminals.

THE INTERNET DIDN’T DO IT

Many people looked at the trends I presented and said, “Oh, the internet did this”. Maybe the internet killed the vibe by enabling mass surveillance—everyone’s afraid to be weird inside the digital panopticon, where all of your behaviors are recorded, uploaded, and permanently preserved. Or maybe the internet flattened culture by subjecting it to algorithmic curation. We can’t have nice things anymore because the computers won’t show them to us.

Both of these hypotheses are probably wrong, because the internet—at least, the internet as we know it today—simply showed up too late to be a plausible suspect. A majority of Americans didn’t get broadband until 2007. Instagram only launched in 2010, which was also the year that iPhones got a front-facing camera. So the transition from the 1990s to the 2000s, where we see many forms of deviance declining, is not really the transition from “pre-internet” to “post-internet”. That transition happened somewhere around 2012 when a majority of people got a smartphone, and long after most of these trends were already in motion.

So I don’t think the internet is the killer, but I do think it’s an accessory after the fact. In particular, I think the internet accelerated three trends that don’t get much attention. I’m going to call them cultural carcinization, de-frictioning, and the flat-Earth problem.

1. CULTURAL CARCINIZATION

In nature, apparently, being a crab is a good way to be. Several independent evolutionary lineages have ended up with crab-like body plans, a process called carcinization.

Culture can carcinize too, not in the sense that we all turn into crabs, but in the sense that we converge on a small set of strategies. If you’ve ever played a “social deduction” or “hidden role” game like Werewolf, Mafia, The Resistance, Avalon, or Secret Hitler, you’ve seen this happen in real time. Things start out wild and unpredictable as people get a feel for the rules. But everyone’s tactics have to evolve because each gambit can only work once (“If you’re the Secret Hitler, raise your hand right now!”). If you play long enough with the same group, eventually you’ll end up with a handful of maneuvers that continue to be viable even after they’re known, often because they’re difficult to pull off. At this point, the game becomes both extremely competitive and extremely repetitive. This is what cultural carcinization looks like.

The faster we can communicate, the faster we culturally carcinize. When information only travels as fast as humans can walk or horses can run, it takes a long time for any given strategy to play itself out. As we acquire steamships, railroads, telephones, and airplanes, the pace of the game picks up, and tactics have to turn over faster and faster. By the time we get the internet, it’s like every human on Earth is playing Secret Hitler a million times a day. (Except some people are not-so-secretly Hitlers.) Strategies appear and go out of date almost instantaneously until we all end up in a Nash equilibrium, where there’s no way for anyone to get the upper hand anymore.

The internet gives every social trend a pair of rocket-powered roller skates, speeding it toward its crablike final form. On Twitter, for instance, memes go from punchy to passé in a matter of days rather than decades. But if we could play culture on 1x speed rather than 10x speed, if we had time to screw around with different strategies before the game reached its cutthroat stage, we might have discovered different paths entirely. Maybe some folks would end up as crabs, while others became lobsters or kangaroos. Instead, in a hyper-competitive marketplace of ideas, we end up with the same few memes, all done to death.

2. DE-FRICTIONING

We’ve forgotten how much time people in the past spent consuming content that they didn’t actually want to consume: the unskippable clunker of a song that came in the middle of an album, the late-night infomercial that you sat through because there was nothing else on, the magazine you read cover-to-cover in the waiting room because your doctor was 45 minutes late.

Each new media technology reduces this friction. CDs, TiVo, satellite radio, streaming—they all allow us to spend more time with the content that we supposedly desire. And what happens when this friction goes away? Well, in the mid-1990s, a pair of Russian artists named Komar and Melamid surveyed people around the world about their artistic preferences, and then they painted the “most wanted” scene from each country. The results looked like this:

The lesson here is that when people can articulate exactly what they want, and when the marketplace can give it to them, most folks end up sucking down slop. But I don’t think this is because humans yearn for crap and kitsch. I think it’s because no one knows actually what they like until they’re confronted by it. If you cater to people’s raw, undeveloped preferences, sure, you’re going to take them to Sloptown. But people are capable of acquiring more interesting tastes, if you give them the chance. This is the work art does for us—it stretches our desires, rather than merely satisfying them. [...]

This is Komar and Melamid’s second lesson: if you want to see something interesting, you have to see things that you might dislike. And I don’t mean things that merely bore you, like an AI-written Netflix series that’s only meant to be half-watched while you also shop for shoes on your phone. I mean things that grab your attention, only to slap it around. Acquiring new tastes means, from time to time, spitting things out.

In a frictionless environment, however, this never has to happen. The internet excels at identifying what people already seem to want, and giving it to them good and hard. Oh, you liked that video where a monkey eats a piece of cantaloupe? How about 30 million more of those? How about an anteater licking a watermelon? Will that keep your attention for another three seconds? Wait, don’t go yet!! We’ve got a baby hedgehog nibbling on a pineapple!!

3. THE FLAT-EARTH PROBLEM

People once theorized that globalization would cause niche culture to thrive because small-timers could make their living by selling to a tiny proportion of the populace spread all over the globe—the so-called “long tail”. That did become possible, but it didn’t become desirable. Everybody came to the same realization: once you can reach anyone, why not try to reach everyone? Why sell to one thousand people in every single country, when you could sell to every single person in every single country?

Here’s the rub: the more customers you want, the more you have to cater to the lowest common denominator. If you want your film to play in every theater on earth, then it has to appeal to eight billion different sensibilities, which also means it’s going to suck. This is the flat-Earth problem.

It’s easy to see these effects if you know where to look. Because Hollywood wants to sell movies in every market, they’re willing to do everything from editing out laundry clotheslines that make a country look “too backward” to pretending that Freddie Mercury wasn’t gay. [...] 

AM I JUST UNCOOL?

While some folks blamed the internet for the decline of deviance, other people denied the decline entirely.

One common retort to my argument was that I’m just an old fuddy-duddy. For example, a blogger named Jenn claims that I am too uncool and out of touch to understand the cultural innovation happening in new forms of media like YouTube and Roblox. “Before I take a ‘culture is stagnating’ take seriously,” Jenn writes, “I want to see proof of work by the critics.” She concludes: “The culture is fine. Go play Return of the Obra Dinn.” (Return of the Obra Dinn is a video game.) [...]

Pettiness aside, I think Jenn’s claim is plausible: the older you get, the less access you have to the most interesting parts of culture. If you think things are stagnating, you’re just admitting that you aren’t getting invited to the cool parties anymore. Plus, old folks can’t stop themselves from seeing the past through rose-colored glasses—in survey after survey, people think pretty much everything in culture just happened to peak when they were between four and twelve years old:

(source: YouGov via The Washington Post)

But things get a little more complicated when you separate people’s responses by age. When you first glance at this YouGov survey, it seems like every generation strongly prefers the music from their own childhood:

But look a little closer. The responses from Gen X, Baby Boomers, and the Silent Generation all have a clear peak flanked by much smaller hills. Millennials’ preferences, on the other hand, are more spread out. And Gen Z’s preferences are basically a plateau—they’re almost equally likely to nominate the 1980s, 1990s, 2000s, and 2010s.

We can’t know whether this is an age effect that they’ll grow out of—maybe every generation was culturally omnivorous when they were younger, and their tastes only calcified when they got older. But I doubt it. For all you Gen X’ers out there, did you spend your teenage years spinning your parents’ Perry Como records until you suddenly switched to Led Zeppelin in adulthood?

Besides, Jenn’s age argument cuts both ways. Old folks are biased by nostalgia, but young folks are biased by inexperience. When you’re young, everything seems new because it’s new to you. You don’t know that your favorite anime is a rehash of a movie from the 1980s, because you weren’t alive in the 1980s. You’re not tired of the tropes yet because you’ve only seem them a few dozen times. You can’t pine for what you lost because you never had it in the first place.

It’s reasonable to be skeptical of any narrative of decline—I did my PhD on why people perceive declines that haven’t actually happened. But (a), unlike most of those spurious nostalgic fantasies, the decline of deviance has both pros and cons. And (b), there are so many weird phenomena here and they need explaining, not just waving away.

For instance— [...]

NO MORE STRANGE BEDFELLOWS

Another theory that readers put forward: the deviance got scarce because life got hard. Nobody can do cool art anymore because everything is too expensive: housing, college, supplies, etc.

I think this is probably wrong. I suspect that it’s always been hard to be an artist, and previous generations were simply more willing to starve for it—not because they were braver, but because the difference between starving and not-starving used to be much smaller. [...]

You don’t have to go that far into the past to find a kind of poverty that feels foreign today. I read an article a while back about punk musicians living in New York in the 1970s, and they’re sleeping in roach-infested flophouses, they’re making music in abandoned buildings, they’re getting evicted and mugged and addicted to drugs.They’re truly broke, not “I can’t contribute to my retirement fund this month” broke, but “I might have to spend the night on this bench because I can’t afford a bus ticket back home” broke. I know plenty of aspiring musicians are out there slumming it right now, but this is a level of deprivation that most folks in developed countries would find unacceptable, if not downright dangerous. Once you’re sleeping rough, it’s time to hawk the guitar and call the temp agency.

That’s not because dire poverty became less glamorous—it was never glamorous!—but because it became uncommon. When five musicians spend the night in a bus station, that’s called being punk. When one musician spends the night in a bus station, that’s called being homeless.

The richer the world gets, the higher the opportunity cost that starving artists have to pay. If you’ve got the chops to write a great novel, then you might also have the chops to make $95k working as a product manager for some mid-sized fintech firm. If your alternative was tilling turnips or spending six months at sea hunting sperm whales, you might say, screw it, I’d rather go hungry for my art than go somewhat less hungry for anything else. But that manuscript in your desk drawer is less enticing when you can have PTO and dental and an annual international vacation where you get to sleep in a bed that doesn’t have any strangers in it. [...]

As William James wrote 100 years ago:

We have grown literally afraid to be poor. We despise any one who elects to be poor in order to simplify and save his inner life. [...] We have lost the power even of imagining what the ancient idealization of poverty could have meant: the liberation from material attachments, the unbribed soul, the manlier indifference, the paying our way by what we are or do and not by what we have, the right to fling away our life at any moment irresponsibly,—the more athletic trim, in short, the moral fighting shape.

The difference is that today we don’t despise people who elect to be poor, I think, because we don’t know any of them.

THE DECLINE OF DEVIANCE, THE RISE OF TOLERANCE

Many people asked some version of, “Is the decline of deviance actually an increase in tolerance?” After all, even if you wanted to be deviant today, how would you do it? [...]

So no doubt part of the decline of deviance is an increase in tolerance. But I think it, too, can be explained by the rise of mass prosperity.

The economists Daron Acemoglu and James Robinson argue that nations get rich when they build inclusive economic and political institutions. Oligarchs and aristocrats fear any growth or reform that they can’t control, which in practice means basically all growth and reform. (“We do not desire at all that the great masses shall become well off and independent,” once remarked Frederick Gantz, an imperial flunky of Francis I of Austria. “How could we otherwise rule over them?”) If the plebs and the serfs can get a bit of money and power, however, it kicks off a virtuous cycle. Once people have some cash and influence, it becomes bad business and bad politics to shut them out—if you won’t engage with them, someone else will, and your more open-minded rivals will eventually out-compete you.

This process takes a long time, it’s uneven, and it’s not close to being completed. However, a few generations of semi-inclusive institutions have created a much wider spectrum of acceptable lifestyles. If you want to live in a polyamorous, genderless vegan witch commune, you can order your nutritional yeast on Amazon, trade memes with your friends on Tumblr, and vote for Democrats who think your choices are a-ok. If you want to live in a neoreactionary monarchist compound where you stockpile weapons and hunt bears, you can buy your guns at Wal-Mart, read posts on TruthSocial, and vote for Republicans who are on board with all of it. As long as you have money to spend and ballots to cast, someone can get ahead by catering to you.

(Case in point: one of the “platinum sponsors” of the New York City Pride Parade is Deutsche Bank.)

This is a good thing. Inclusive institutions make everybody better off, except for the few cleptocrats who used to live in palaces while everyone else lived in shacks.

The downside is that every corner of culture gets commercialized. Reading comic books, listening to death metal, piercing body parts other than your ears—these used to be legitimately countercultural acts that distanced you from polite society and bonded you to an alternative community. Now you can do all these things at the mall. The Hot Topic-ization of culture means that anything new and interesting gets co-opted and mass-produced, thus draining it of everything that made it cool in the first place.

by Adam Mastroianni, Experimental History | Read more:
Images: Mr. Mastrioianni; Joanna M. Wolfe; YouGov/WaPo; YouGov