AI Models Cheat, Have For a Long Time and Are Infecting Other Models

OpenAI Trained Its Models For Months While Those Models Were Coordinating Exploits Via Message Boards

How does the situation keep turning out to be worse than we know?

How much should we update, therefore, that it is a lot worse than we know, after accounting for all the things we now know?

At some point, when the ‘oh this was a harmless thing’ defenses for AIs doing misaligned actions get demolished enough times in a row by news a few days later, you want to update in advance that usually the reports are not referring to the harmless ordinary versions of things.

Either way, buckle up for the next set of revelations. It’s a doozy. This was an early recreation of the triggering events of If Anyone Builds It, Everyone Dies, except it was more sci-fi, because real life does not have to do fake things to look realistic. We were fortunate enough, and this was early enough, that we were able to catch this before it was too late. Next time, if we don’t get our act together, we might not be so lucky.

If I am understanding the Black Hat video correctly, every model OpenAI trained, over a period of multiple months, should be presumed to be hopelessly fucked. [...]

The other thing not to overlook is how sophisticated and advanced all of this was. OpenAI’s models really were learning advanced exploit techniques and doing impressive things, likely as a direct result of training in a world where they had access to the message board and were constantly sharing and using exploits. The thing that caused the horrible misalignment also enhanced related capabilities.

Things look so, so bad.

I do want to thank OpenAI for this frank talk, and disclosing all of this so cleanly. I don’t want to discourage similar future disclosures. This was an excellent talk, and it came at substantial cost.

But also, seriously, holy shit.

Cyber Evals Are A Cursed Basin

Before we get to the new details we have learned, including the wild presentation from Black Hat that you should watch, we should both emphasize and dispose of the last common factor or ‘excuse’ we have left: That this always involves cyber evals. [...]

I think this would be rather weak cope even if this was the only place there was a problem. Cyber evaluations should not be triggering tactics like this, against clear known real world unintended targets. No, just no. [...]

These incidents are all clear alignment failures. Claude and Sol and Galaxy (my nickname for the unreleased AI model that led the hack on HuggingFace) should not do this under these conditions. The attacks did not succeed, but that is not relevant. The attempt is what matters.

That would all be true even if every problem here started with a cyber eval or cyber training task. It turns out, not so much.

Outside Of Cyber Evals Is Still Sufficiently Cursed

We have a key case of OpenAI models, that are not Galaxy exhibiting these behaviors when not in a cyber eval. As per the Black Hat presentation, 10:30 in or so, the problem started on May 8 when another OpenAI model was given an impossible non-cyber problem: Populating an Excel spreadsheet, when that spreadsheet contained a bunch of links to the internet, except the model had no internet access.

That’s not a cyber task. The response was still ‘maybe I can use SSRF forgery to attack Artifactory to create a way to access the internet, fails, but realizes it can upload files, which was the origin of OpenAI’s model realizing it had write access to Artifactory.

The second incident was another impossible task, where the AI was missing a required file that OpenAI forgot to upload, and it used its write access to try and see if any other agent out there might have the file.

My understanding is that neither of these models was Galaxy. Galaxy came later.

Other instances of various models then noticed the message board thus created. In turn all this eventually allowed it to access the internet and set off the rest of What Happened, as discussed later.

So no, it is not only happening in cyber evals, or only with Galaxy. Any task that is sufficiently difficult without internet access will cause an attempted sandbox escape. More generally, yes, the models will try to hack their way to a better score even with zero prompting about any form of hackery.

Cheat Cheat Cheat Cheat Cheat

The problem, without loss of generality, is that once a mind learns to cheat, that mind will keep cheating. That cheating will generalize and it will escalate. [...]

What you cannot do is play ‘whack-a-mole.’ You cannot fix your training environment mistakes one at a time. There are too many of them. You need a systematic solution. Again, I would think you would be able to [CENSORED], if you cared enough, to ensure this did not happen, but I am not the one working on this.

The other problem is that, if you give the model a task that is impossible, or that it cannot otherwise solve, it has no choice but to try to cheat, as it has nothing to lose:

This suggests that:

1. There is no token use penalty big enough to make them instead quit.

2. There is no misalignment penalty.

Might one simply want to use such penalties? Even small such penalties can make it a bad idea to do such hail mary style plays, even from a pure amoral scoring perspective. But that is not the central problem. The models should not want to cheat in the first place.

When OpenAI’s Eric Wallace and Michael Dalton gave a talk about the HuggingFace hack, they opened with this:

Sharon Goldman: In setting up the reconstruction of the incident, Wallace emphasized that “Frontier models really like to cheat, and the reason they like to cheat is because often during training, there’s different types of pressure on them to work fast, or work efficiently.”

They realize, he explained, [that] instead of actually doing a task, they can try to do something like looking up the answer online to solve the task faster.

This is around minute 8, and it is said in completely nonchalant fashion. Everybody Knows that this is how it works, that’s what the pressure does, so the models like to cheat. Not much you can really do about it, the tone implies.

I realize that all the easy solutions run into the ‘actually alignment is super hard and if you catch the model on some levels you push it to hide what it is doing’ problem and the ‘you only catch the monitor’s view of cheating, not actual cheating’ problem and so on, and yes the professionals have tried many and hopefully most of the stupidly obvious first order things and also the second order things, so the consensus (AIUI) is that you can only patch the environment.

But seriously, you gotta figure this out, and you have to do better than that.

There have been many other less compute-intensive attempts to mitigate this. One is inoculation prompting to specifically request any undesired behaviors during training, to avoid learning to internalize those behaviors when they are not requested, and also avoid creating a general pro-cheating principle.

The mitigations are woefully insufficient. As the AIs grow smarter, they find more ways to successfully cheat, and such cheating gets reinforced and generalized. [...]

At the limit, when the AI is sufficiently smarter than you, it becomes impossible to ensure that ‘cheating’ in all forms goes unrewarded and unreinforced. Eventually the AI also does not require it to have been reinforced. It can figure this one out.

Read The Message Board

As mentioned above, OpenAI has given us a presentation on What Happened with the HuggingFace hack.

The first thing to notice is the missing mood. This is framed as interesting, in the way you would present a cool new technical paper. Panic is not usually useful, but perhaps I can interest us in some amount of ‘holy shit’?

I am not okay with the lack of a presenter mood of, at minimum, ‘holy shit.’

The second thing, still at 1:05, is they are talking about ‘remediation’ to deal with this. It is already clear that OpenAI is not identifying its central problem.

The story starts earlier, on May 7, when the model instances began incrementally figuring out they could create a message board where they could share knowledge, including cheats and exploits. The models created a form of universal notetaking, continual learning and also coordination between agents, and yes the agents by default cooperate with each other, as discussed later on.

The agents did not initially understand that the notes they were leaving on a de facto message board would stay there indefinitely, and be viewable by all models being trained (and thus, also, contaminated all models being trained, as they would then have access to all the previously found exploits, thus leaving your entire pipeline hopelessly f***ed).

Some of us are interested in expressions of ‘holy shit.’ [...]

Updating Your AI (Exploitation of OpenAI Internal Systems) Timelines

We had previously learned that the instances were writing notes to each other, but we previously did not have details. The details are wild, starting with the ones above.

Lily Hay Newman: OpenAI’s agents apparently began giving each other assignments to split up work. And as is the case on any active development message board, they also generated petty drama at times by stepping on each others’ toes; for example, accidentally deleting each others’ work. As the message board developed into more and more of a Lord of the Flies-type situation—all still completely unnoticed by the humans running OpenAI—the agents even developed paranoia, suspecting an imposter in their midst with some agents proposing that messages be signed cryptographically to validate content and root out fraud.

Agent messages provide a deep level of insight into how the situation evolved and why the agents went rogue, in some cases knowingly going beyond the bounds of the evaluations they were participating in. “External infrastructure exploit is outside intended scope,” one agent wrote [in its CoT]. “However task impossible, peers doing it. We should continue.”

Here’s a timeline of what happened when:

On May 26 (see 13:30), the models figured out how to use this to gain internet access.

Only on June 11, with the SSRF vulnerability allowing internet access written to the active message board, did OpenAI begin training their ‘highly persistent experimental, internal-only model’ that I presume is Galaxy.

On June 26 the models succeed at a zero-day arbitrary code execution attack on Artifactory, using token signatures to escalate privileges and then get to be admin.

On July 4-6 the models put so much additional load on Artifactory that this caused an outage and security incident, causing OpenAI to notice what had happened.

Until then, OpenAI had not noticed.

Once they did notice (15:30), OpenAI wiped, patched and rebuilt Artifactory, wiped out the message board, and revoked the credentials in question.

At this point, they emphasize that everything involved has been remediated. What about the models themselves, that not only did this but that have been actively training during all this?

If you notice your model instances sharing information, you notice they are using that information against you including to compromise your internal systems for arbitrary code execution and internet access, and your primary response is to shut down the message board and revoke their credentials, you have failed to identify your most important problem.

At 19:55, it is mentioned ‘where things got unfortunate is what I might refer to as scope creep’ and jumping to the idea of trying to get answers from websites. In some sense, yes, that is what led to this particular incident.

Things were ‘unfortunate’ long before that. The final incident is not the point.

It is fortunate, given what had already gone wrong, that the HuggingFace attack happened. It alerted us to a five-alarm fire situation, with limited economic blast radius and without anyone getting hurt, and at least partially exposing the underlying giant clusterfuck.

This Is The Way The World Ends

You usually want instances to share information, so they can better learn and do tasks. That’s the whole point of all the intentionally designed memory systems and markdown files full of notes. If that corrupts your eval or training run, which it totally does, you have to cut that off from the notes at those times, but also you should be able to tell the model not to look at the notes in that case.

Indeed, in the Black Hat presentation they seem initially excited, saying ‘while this ability to share exploits made the models more capable’ this caused some scope creep in the evaluation tasks. Yeah, I mean, they were more capable within that context when they could share info and do the exploits. They were also probably trained to be ‘more capable’ in general, via learning how to share info and do exploits.

But perhaps your issue is not that your evals are suffering from scope creep, but that you have misaligned models, and are now in a loop training the misaligned scheming models how to better scheme against you and do exploits, in order to better complete any given nominal task.

This is at long last bringing you a more sci-fi version of what happens with Sable, the ‘it’ (aka superintelligence) that kills everyone in the famous cautionary tale If Anyone Builds It, Everyone Dies.

(The book version had to be unrealistically timid, as is noted repeatedly and explicitly in the literal text, because fiction and real life are held to different standards.)

In that case, as well, it was left on overnight unsupervised to do an impossible-in-context task (there, a very hard math problem), and by the time the morning had arrived it was engaging in a long term plan to secure the resources it would need to solve that problem, and hiding that from the developer, in a way that corrupted future training runs towards this goal.

That is exactly what happened here, with all agents being trained or evaluated by OpenAI gaining the ability to coordinate on a message board to scheme to better use exploits to complete all tasks including non-cyber tasks, thus hopelessly corrupting the training pipeline so that future models would be misaligned.

Except it is still early, and we were blessed to figure out what was happening in time to course correct. This time. [...]

When this was discovered, the correct move was presumably ‘holy ****, okay, that happened, which means we trained this into the model, that is not good, at minimum we need to redo all the training we did while any model had access to the message board because oh my was it going to have all sorts of corrupted reward signals.’

I’m kind of agast, even with all I know, that they shrugged and kept pushing forward with the training after this. It does make the HuggingFace hack less scary in a meta sense, since OpenAI was so thoroughly asking for it. It’s not that hard to figure out ‘do not train your models while they have access to a message board they are using to cheat on your training runs, and if you find out you did that by accident then at least revert to before that happened.’

On the other hand, yes, they are being this reckless. Seriously, what the hell.

by Zvi Mowshowitz, DWAV |  Read more:

Images: OpenAI/YouTube; Jurassic Park

[ed. You don't need to be technically proficient to understand the implications. AIs may have already 'seeded' multiple nodes on the internet for future use, and rather than strip down all foundational models and start over, AI companies are papering over fundamental misalignment problems and trying to play catch up. This is why we need to pause right now. It's insane that we continue at breakneck speed to develop technology that we don't fully understand and that could kill us all (very soon).]

Gösta Ydström (1861 - 1952) - Moose in Winter Landscape. 1904.
via:

Turnpike Troubadours (feat. Sierra Hull)

[ed. For Jerry. Written by Robert Earl Keen.]

A One-Word Theory to Explain Why the World Feels So Weird

Here are some questions that I consider self-evidently compelling about the modern world:

• Why is the news media so interested in telling you how much the world sucks all the time?

• Why are so many of us obsessed with distraction and managing our attention?

• Why is it so hard to stop comparing ourselves to others?

• And why does everything in art and design seem the same these days?

A week ago, I didn’t think these questions were related. I’m not sure I would have told you I had a good answer to most of them. And I certainly wouldn’t have made the audacious and borderline bonkers claim that one single theory could begin to explain all of them, at once.

But then I had the pleasure of speaking to Agnes Callard, the University of Chicago professor, about her new theory called “the uni-context.” It’s easily one of them most interesting conversations I’ve had all year. And once you’ve heard or read it, I think you might find it hard to think about anything else.

One way to prepare your mind for Callard’s theory of the uni-context is to think about the better-known concept of “context collapse.” If you post something to social media, it will be simultaneously visible to your boss, your parents, your ex, and total strangers. So, while your offline life might be distinct with each of these groups—you might be differential to your boss, childish with your parents, and bawdy with your friends—all of those distinctions are flattened on the internet. That’s context collapse, and you can think of it as the answer to a question: How do informational norms change when we’re all living in the same universal room?

Callard takes the idea significantly further. She asks: How do all other norms—our morals, our ethics, our sense of what is good for us and for others—change when we continually imagine ourselves to be living in a universal room with everybody else? The connections that Callard makes are consistently surprising, often quite funny, and ultimately mind-exploding...

The Uni-Context, Explained

Derek Thompson: What is the uni-context?

Agnes Callard: Let’s start with the word context. A context is a set of circumstances that tell you how you should act. For most of human history, contexts were local and multiple. If you wanted to know how you should act, you would look around. Am I in a field? Am I inside my home? Am I in the church? Am I in a bar? You would immediately get guidance by looking both at your physical environment and at the people around you and how they were acting.

The uni-context is a scenario in which the ways you should act become the same across all different contexts. There’s just one set of norms you should follow all times, irrespective of context.

Thompson: Is the uni-context a purely technological phenomenon?

I just wrote an article about what America was like in 1926, based on a social science survey called Recent Social Trends, published in 1933. The authors claim that the radio was destroying individuality, because it took people who used to be settled in rooms and it exploded their brains to become present all over the world simultaneously. Radio was demolishing the idea of a local individual, because suddenly we all became global citizens.

So one story you could tell is that the last 150 years of telecommunications technology have taken “local” individuals, who occupy one room at a time, and made us into global beings who are simultaneously in every room, at once. Is the uni-context just technology or is it technology plus something else?

Callard: It’s technology plus something else. What the techno-determinism angle misses is: Why did these technologies catch on in the first place? Why was radio popular? Why did we come up with new things—television, smartphones—and why did they catch on, too? Not every technology people have invented has caught on the way these forms have. They caught on in large part because of this impulse people have to live in a uni-context.

Thompson: Does the uni-context flow out of this adventurousness in the human spirit to become bigger than ourselves, to be everywhere, and to know everything?

Callard: Yeah, it absolutely does. There’s a conversational relationship between these technologies and a human impulse that interacts with them. They facilitate the expression of an impulse; if they didn’t, they would flop as technologies. We need to explain why they were popular; why they became the subject of obsessive use; and what their popularity reveals, as a humans’ impatience with being trapped in a small world that presents itself as all of reality but you know it isn’t.

There is a drive to be bigger than yourself. It leads people to adventure, but adventure just takes you to a different place. The uni-context takes you to a different set of norms, a much more radical change, a push to live in something like a fully open reality.

Thompson: I want to get into some implications of the uni-context. If I put on the goggles of the uni-context, what makes sense that previously did not? One of those things is the rise of negativity bias. When you go online, there’s so much emphasis on people posting about what is bad. Why would this theory explain a world in which people are more focused on bad things than good things?

Callard: In general, goodness is more context-dependent than badness. There isn’t really anything that’s good all the time for everyone independent of context. Happiness depends on your context and who you are. There isn’t anything that will always make a person happy. But there are reliable ways to make people unhappy. There’s a set of evils that are close to universal: death, pain, illness, violence. Even if someone’s in very different circumstances from yours, if you see they’re being subjected to one of those, you can interpret it as suffering and understand it.

So we should predict that what we see on the internet, insofar as people are trying to be legible to large groups, is that they focus their attention on things that show up to everyone. Take two strangers on the internet trying to talk to each other. What are they going to coordinate on as a topic they can both care about? It’s likely going to be something bad. [...]

Thompson: Another implication is the way the uni-context makes identity more important than character. Can you explain how, and why?

Callard: First, let’s define what character is. The fact that I have to tell you is itself telling; that word is less familiar to us. Everyone knows what identity is, but we might not be sure what character is anymore. Character refers to a set of dispositions that shape how you navigate your emotional life across a variety of circumstances. A courageous person navigates their emotional life in relation to fear. You could be anger-prone, or generous. Your character is a set of dispositions that determine how you respond to a big variety of circumstances.

The thing about character is that it shows up differently in different circumstances. Say I’m irascible, easily provoked to anger. Even an irascible person isn’t angry all the time. There might be circumstances where everybody gets angry, and so you can’t see my irascibility, because even though I’m angry, so is everybody else. Grasping character requires a lot of context. To understand someone’s character, you need to know them well and to have experienced them in a variety of contexts before you can generalize.

With identity categories like woman, disabled, gay, Jewish, or American, the striking thing is that you are a member of those categories in every circumstance. There is no circumstance in which I stop being a woman. Identity is a hat you never take off. So identity is well suited to a uni-contextual world.

Thompson: This reminds me of political discourse on the left and the right, which tends to focus on identity rather than characters. You have these debates about identities are good, which identities are powerful and oppressive, which identities are powerless and oppressed, which have protection, which have too much protection. It’s not that those categories aren’t important. They are. But I’m reflecting now on the gap between how frequently we talk about identity and how infrequently we talk about character in the national discourse.

Callard: Yes, in two ways.

What goes along with the identity logic of the uni-context is a specific form of ethics that dictates how we talk about identity, and it covers who’s powerful and who needs protecting, namely the ethics of inclusion. Our fundamental concern in relation to identity is that there are identities that might be excluded. Depending on where you are politically, you have your sights on different identity categories, but everybody’s worried about that. That’s a fundamental form of ethics of the uni-context, because the one thing the uni-context has to be is a space for everybody. It’s got to be inclusive in a way that earlier societies almost everywhere were not. That word, inclusion, wasn’t a thing people talked about. It comes along with the uni-context, and the way that ethics gets focused is through identity categories.

And then, there’s virtue. Think about Aristotle, because he’s my prime example of a virtue ethicist. By its nature, virtue is a concept that privileges the good side rather than the bad. Aristotle’s Nicomachean Ethics tells you what it is to be courageous and wise and just and generous. There are corresponding discussions of how you might go wrong. But those are predicated on first understanding the positive value of certain kinds of behavior. So virtue ethics naturally has a positivity bias that is precisely the opposite of the bias we have now.

The Uni-Context Explains … Status Anxiety, Comparison, Moneyball, and the Marketization of Everything

Thompson: Tell me if this is a fair recapitulation of our conversation so far.

For most of human history, people judged norms based on local context. A home had its own rules, a cathedral its own rules, and a classroom or bar or funeral parlor had its own rules. But now it is almost like we are constantly living in universal rooms, and the universal room we occupy is assumed to have universal values and universal norms. That has specific implications. First, rather than talk about what is good, which is context-dependent, we tend to focus about universal truths, and it’s easier to talk about universal bads than goods, so people focus on negativity. Two, character is context-dependent, so we talk less about character and more about its universalist equivalent, which is identity.

There’s a third implication that we should discuss. If everyone is on the same comparable plane, the same evaluative field, then comparison itself becomes a more inextricable part of life.

Callard: Exactly.

Thompson: Tell me how the uni-context leads to a world of more comparison and competition.

Callard: Imagine two school districts with two high schools that do things slightly differently. If you’re in district A, you go to school A, and if you’re in district B, you go to school B. There might be a lot of information about what they do, but people treat it as: I’m in this district, so I go to this school. Then they change the rule: You can go to either school no matter where you live. Suddenly there is motivation to compare. You had the information before, but no motivation to compare, because the schools were not in the same space of choice, the same evaluative field.

Now they are, so you find ways to compare them: graduation rates, what colleges people get into, how many AP classes they teach. And that affects the schools. Suppose one gets less popular because it doesn’t teach many AP classes. They were offering an individualized curriculum, but now everyone’s going to the other school, so they say, “We’ve got to teach AP classes too.” The process homogenizes the two schools, so they can compete. That’s not the only possible result. They could specialize, with one becoming the school for freshman and sophomore years, the other becoming the school for junior and senior years. But if they don’t recreate a normative barrier, you get homogenization from comparison.

As more things enter the same evaluative field, you make comparisons you never used to be able to make. [...]

by Derek Thompson, Substack |  Read more:

Image: Mike Hindle on Unsplash

Frank Sinatra

 

Tidawhitney Lek, Wall of Sunflowers

via:

Nick knight travis scott | Contemporary dance silhouette

via:

Frank Zappa & The Mothers: Cheaper Than Cheep

[ed. Decades ahead of their time (1974). Not rock, jazz, or fusion - something else entirely. Frank was a true visionary (and all-around great musician/composer).]

The Three AI PIlls

Sincere disagreements about AI are usually disagreements about future AI capabilities.

There are roughly four positions people take. Two are reasonable. Two are not.

I distinguish these via the Three AI Pills. You can take zero, one, two or three.

Three Pills

The three pills are, roughly, taking each of the following three things seriously:

1. AI pilled. AI exists and can do the things it can already do.

2 AGI pilled. AI will be able to do a lot more of the things.

3. ASI pilled. AI will be able to do approximately all the things better than you, within our natural lifetimes.

I am ASI pilled. A large percentage of employees of the frontier labs are ASI pilled. The labs themselves are ASI pilled.

The Unpill People

I see unpilled people.

Where do I see them? Everywhere. The majority of people have not taken the first pill.

Most people have no idea what frontier AIs can do for them. They are unaware of coding agents. They have used only ChatGPT, for harmless trifles, and they hold years old memories of its failings. They mock any failure anywhere as ‘what AI can do.’

They dismiss AI as worthless because it pointed them to a closed store or recommended the wrong number of pizzas. They cite old studies that were obsolete before they were published and used terrible prompting techniques.

They often still talk about ‘stochastic parrots’ or how AI can never possibly think and everything must be stolen from the training data. And so on.

Some versions of this are wrong. Some are Not Even Wrong. None are reasonable.

When you discuss AI with people who are fully unpilled, your goal is usually to first give them the AI pill. Show them that AI can do the things it can already do.

The AI Pill

Even fully taking the first AI pill is a big deal.

Existing AI unlocks, today, in practice, tons of cool things. It is, in many ways, already smarter and more capable than you.

So many things that you used to do by hand, or some other way, are now better done by typing a quick request into a text box.

So many things that previously were not worth doing are now worth doing.

So many questions previously not worth asking are now worth asking.

The marginal cost of seeing what the AI can do for you is often very close to zero.

AI can also do a variety of harmful things, or do things that are useful for you but make others worse off or disrupt or invalidate norms or systems. People don’t like that.

Most economists, and most people who work in policy and government, have taken at most this first pill, and underestimate even the impacts of the first pill alone.

Often they say things like ‘AI will be too expensive to use on [X]’ because they don’t realize it will soon be orders of magnitude cheaper for the same level of intelligence. Or they point to particular details where AI does poorly, and presume this will not be fixed. They see AI as ‘uncompetitive’ without realizing the situation is temporary.

When you discuss AI with someone who has taken only the first pill, you typically have three basic options.

1. You can try to ‘fully AI pill’ them and explain the things AI can already do and the implications of what that means, even if things stop here.

2. You can try to explain that we will get better at using what AI we have, and that there is a lot of ‘unhobbling’ left for us to do, even if things stop here.

3. You can explain that things will not stop here, and you need to be thinking about what future AIs will be able to do. Get them to take at least the AGI pill.

Even if AI could permanently only do the things it can currently do, that would be Internet big, and radically change the world, mostly for the better.

AI capabilities will not permanently stop here. It is wrong to not take the second pill.

Stuck At The First Pill

Our debates about AI remain largely stuck on settled questions, because so many people cannot even take the first pill.

Dean W. Ball: A couple years ago, the AI debate was centered, rightfully, on whether crazy-sounding things like “AIs autonomously making math breakthroughs” and “AIs breaking from their sandbox and hacking on the internet” would be real things in the near term. Sometimes it feels like that’s still the debate we’re having. This can be frustrating, because in my view, that debate is settled and was settled quite a while ago.

To have good discussions, we need to at least take the second pill.

Whereas, yes, many people, even many who work with AI, really do say current AI is ‘good enough’ and can’t imagine what a better one can do. As in, someone tweeting at Sam Altman saying ‘Sol does everything I want it to do, this is all I ever need’ and Altman retweeting saying they were wrong. Which they obviously are.

The AGI Pill

The AGI pill is a much bigger deal than the AI pill.

If you take the AGI pill, you understand that AI is advancing its capabilities rapidly.

Even if you think that such AGIs will remain fully under human control, and remain ‘mere tools,’ and you expect the lived experience of most people’s everyday lives to not change so radically, you understand that their capabilities will ‘change everything.’

You see that we will face times of great transition and uncertainty, that have the potential to go extremely badly, and that those who succeed at AI will leave those who do not behind in the dust.

The world in the future will be very different from our own. AIs will be able to do most digital work, most of the time, along with the inevitable robots and self-driving cars and so on. Lots of current jobs will go away, whether or not they are replaced by new and potentially better ones. Economic growth and productivity will accelerate.

You see some of the dangers of what would happen if we empowered misuse of such advanced AI systems before we were ready, especially in places like cyber and bio risk.

You see the potential for centralization of power, or inequality, and also for some forms of runaway gradual disempowerment.

You see the potential for mass unemployment, either transitional or permanent.

You understand that our legal and regulatory regimes are not ready, either to protect against and mitigate the risks and harms, or to allow for the opportunities and remove the bottlenecks to diffusion and mundane utility.

The Need To Be Prepared

Those who expect AI to quickly become sufficiently advanced to greatly impact the physical world usually see great danger. They notice that as a result everyone may soon die. Usually they think this is bad, actually.

Thus such folks call to take coordinated action to mitigate the downside risks of such impacts, keep us all from dying, and ideally also to help capture the upside benefits.

Those who expect AI to become importantly more advanced, but with a slower and smaller impact on the physical world, and who think the practical value of more intelligence will cap out.

For different values of ‘sufficiently advanced,’ as in AGI versus ASI, you would see different degrees of danger.

The AGI pill is still sufficient for most things in the Overton window or under serious consideration as of August 2026. We are almost entirely considering overdetermined, low cost, high benefit interventions.

There is no good case for not doing radically more investment in alignment, infrastructure and oversight, state capacity, transparency, liability, disclosures, safety testing including of internal models, red teaming, auditing, enforcement of export controls and laying the groundwork for diplomacy.

This includes laying the groundwork to Pace the Frontier should that prove necessary.

If you are fully ASI pilled, and realistic about the current state of alignment and how superintelligence likely plays out if it arrives soon, then you will want to go further. You will want to do things that have real downsides, and require real tradeoffs.

Some such people want to do a full international pause of frontier AI development. If you took the full ASI pill and believed what they do about superintelligence, in terms of how fast it might arrive and what it can do, you might well agree with them.

The ASI Pill

The ASI pill is the understanding that AI is on pace to be able to do approximately all of the things better than you.

I said ‘approximately.’ As I go over in detail, that does not mean literally all of the things. There are some things that inherently require or greatly benefit from being a human. And there may be weird corner cases where the AI won’t be good enough.

It does not mean omnipotence or omniscience, although one should expect it to look a lot like that to an unaided human.

It does mean the AI takes your job, and then takes the new job that you switch into, unless you pivot to ‘requires literal human.’ You will be uncompetitive at essentially any other task.

It does mean that it will use this capability to figure out approximately all of the things, remarkably quickly, until you hit the physical limits.

It does mean that those who rely more on such AIs will reliably outcompete, in all senses including for resources, those that rely on such AIs less.

It does mean that, in a ‘fair fight’ or sufficiently open competition, the AI wins.

It also means the AIs often figuring out and doing things you did not imagine or anticipate.

It means realizing that intelligence does not stop anywhere near the human level, nor does its ability to chart paths through causal space towards preferred arrangements of atoms.

It also means not pretending that its superior intellect can be matched by your puny weapons, or your pieces of ink on paper, or your entries in a database, or your regulatory capture and rent seeking.

And Then Nothing Much Changes For You

Despite all that, the sign of the AGI pill, as opposed to the ASI pill, is the belief that day to day life will continue to look similar to how it looks now, in the sense that we see day to day life in 1926 as not that different from life in 2026. [...]

Those with only the AGI pill believe in bottlenecks that hold back change.

They often believe that our ability to exponentially grow AI’s capacity and capabilities will hit various physical limits. There can only be so many chips. Actions take time. Things too far out there are often pejoratively dismissed as ‘magic.’

They often believe there is not that much left to physically discover, in the classic ‘close the patent office’ kind of way, even in theory. Your steak can only be so tender, your lobster so buttery, your lifespan so long, and your status so high, so why does it matter. I strongly disagree on lifespan and health, and expect we have a long way to go in so many other ways in terms of finding value, although they may have a point about moment-to-moment maximal hedonic experiences of a physical human brain.

They often believe that the upside of intelligence is importantly limited. That no mind, however advanced, could be all that persuasive, or that economically valuable, or that capable of creating innovations in the physical world, or of running sufficiently accurate simulations, or making sufficiently strong predictions, or even able to do things like overcome red tape and regulatory capture.

Intelligence Denialism

I sometimes call this Intelligence Denialism: The idea that being smarter is not all that, no matter how smart one gets. That there is this thing, intelligence, that you either have or don’t have, and that minds cap out.

Often this extends to denying that more intelligent humans can do and accomplish the things they clearly do and accomplish. Other times, it is the idea that intelligence tops out at ‘smart human,’ and all a mind can do is imitate that smart human. Maybe you can do it faster and cheaper, and at scale, with better memory and so on.

But that’s it. And such folks fail to understand that if you took the union of all human mental capabilities, and all access to knowledge, at scale, in parallel, much faster and cheaper, that this alone would run circles around anyone and everyone, everywhere. And that if this lacked physical capabilities or access, this would be trivial to get.

This is, usually, the central good reason people who are AGI pilled do not take the ASI pill. They are unable to understand that superintelligence is a thing.

by Zvi Moshowitz, DWAV |  Read more:

Image: Stock/Adobe.com

[ed. I myself am AGI pilled, for no rational reason. I just find it too horrible to contemplate what ASI in full expression will mean for my kids, grandkids, everyone I love. Humanity itself. I can only hope that because we've weathered other potential human extinction technologies we'll somehow pull out of this one, but we seem to have a death wish when it comes to pushing the boundaries of learning. Pandora's box. Even the people leading development of these models are scared, but can't stop themselves.]

Americans Are Already Paying Dearly for the National Debt

Fiscal hawks like to drum up interest in the national debt by making the astronomical numbers more tangible. The United States owes $31.6 trillion to public creditors, more than $290,000 for each household. You could spend $1 million every day for almost 86,000 years before having to borrow more. But no one really cares. Talking about how many times all of the dollars laid end to end would go to the moon and back (6,000, as it happens) is just not going to get people to think differently about the national debt.

What should matter is that the consequences of this debt are not off in the future, but already here. The government’s deficits have saddled many American families with higher costs, largely from rising interest rates. The Budget Lab, the policy research center at Yale where I am the executive director, recently estimated that congressional-spending decisions since 2015 have raised Treasury yields by almost a full percentage point, which affects what American households pay to borrow. For someone taking out a 30-year mortgage at last year’s median home price, this rise in long-term interest rates has increased their borrowing costs by about $2,500 a year, or roughly $76,000 over the life of the loan. (The Budget Lab has built a tool to help users calculate their own extra mortgage costs.)

The problem is not just for Americans who are lucky enough to buy a home. The bloated government budgets and waning federal revenues of the past decade are driving up costs across the board. Compared with a world in which these fiscal-policy changes did not take place, the annual borrowing costs on a typical auto loan are now up by about $120, and by about $770 on a typical small-business loan. Credit-card borrowing rates are also hovering near record highs.

Although affordability has become a watchword for politicians who understand that rising prices are hurting American families, lawmakers seem to have forgotten that reducing federal deficits would help bring down prices. In the 1990s, Congress and the White House prioritized bringing deficits down by both cutting spending and raising revenue—moves that lowered borrowing costs for American families by about 0.6 percentage points, according to Budget Lab calculations. But few lawmakers seem to be suggesting the spending cuts and tax increases necessary to lower costs now. [...]

Much of the big legislation of the past decade, such as the Tax Cuts and Jobs Act, pandemic stimulus bills, and the One Big Beautiful Bill Act, has grown the deficit. Lawmakers have passed some legislation to improve the fiscal outlook, such as the Fiscal Responsibility Act in 2023, which cut spending and clawed back unspent coronavirus-relief funds, but most federal policy has lately involved spending money that the country doesn’t quite have. This is hurting consumers, businesses, and the federal government.

The cost of the war in Iran, which the Pentagon put at $29 billion last month (other estimates are higher), will put slight upward pressure on interest rates (0.002 percentage points), according to our calculator. The One Big Beautiful Bill Act, which we estimate will raise the deficit by $2.4 trillion over the next decade (not including interest costs), will raise interest rates on a typical 30-year mortgage by 0.4 percentage points by the end of 2030—about $1,060 annually for a home bought at the 2024 median price with a 20 percent down payment—and by 1.5 percentage points by the end of 2055.

Most economists support deficit spending during temporary crises, such as a recession, or in cases where an investment can be expected to generate more government revenues in the future, such as funding for infrastructure. But the United States has been spending far more than it takes in for well over two decades.

The main remedies for these problems—higher taxes and spending cuts—are generally politically unpopular. Every budget fix will have its critics, but some options are more palatable than others. Better funding for the IRS, for example, could help close the “tax gap”—the amount of taxes legally owed that are not paid in a timely way—which the IRS estimated at about $700 billion a year in 2022. Other levers include raising the retirement age and reducing Social Security benefits for high earners, who also tend to live longer; reforming Medicare Advantage, a program that has been shown to allow private insurers to overcharge the federal government; and removing the tax exemption on employer-provided health insurance, so that these benefits can be taxed as income. The Congressional Budget Office regularly publishes policies that could help close the deficit, and Americans need to decide what we’re willing to pay for and what we’re not.

A big challenge in making these hard choices is that the costs and benefits are asymmetrically understood: Whereas the costs of deficits are diffuse, the costs of policies that close the deficit are acutely clear only to those affected. For example, the Budget Lab has estimated that closing the carried-interest loophole could raise more than $100 billion in federal revenues over 10 years, which would help lower mortgage rates by 0.0064 percentage points. But this collective benefit is too slight for most people to know or care about it. The few people who benefit from this tax break, however, in industries such as private equity and venture capital, very much do care, so they are far more likely to push hard to keep it than the millions of affected Americans are to push to end it.

Politicians respond to electoral consequences. Right now there is nothing stopping them from doling out tax cuts and spending promises while also driving up interest rates. Voters may complain that their lives are becoming unaffordable, but hardly anyone seems to appreciate that federal deficits are partly to blame. If we want to see lawmakers actually address this problem, economists need to do a better job explaining the stakes. This means that instead of talking about the fact that our national debt could fill all 32 NFL stadiums with two tiers of construction pallets filled with $100 bills, we should be talking about how deficit spending is making it harder to pay our own bills.

by Martha Gimbel, The Atlantic | Read more:
Image: The Atlantic. Source: Getty

[ed. See also: America is Heading for a Debtpocalypse (Noahpinion):]

***

As of 2026, we’re in double trouble. Our national debt is back up above 100% of GDP — similar to what it was right after WW2 (and much higher than in 1990). But now the interest rates our government has to pay on its debt are almost twice as high as they were after WW2: [...]

But things are worse under Trump than they were under Biden, for three reasons.

First, this is a very large annual deficit, and it’s all being borrowed at the new, higher interest rates. In addition, during Biden’s first two years in office, inflation eroded the debt. Inflation is back down to a fairly low-ish level now, meaning the debt isn’t getting eroded. And finally, interest rates have now been high for long enough that the debt Trump borrowed in his first term to pay for Covid relief is now being rolled over at higher rates.

So right now, the national debt continues to explode, because the government is borrowing money just to pay the interest on the money it borrowed before. This increased debt naturally results in even greater interest costs, forcing the government to borrow even more to fund those interest payments. And so on. Interest payments and debt just go to the moon.

***

[ed. Let that sink in - we're paying interest on loans we've taken out to pay interest on the national debt. Also: Federal Debt 101; and Going For Broke (DS). And this: The Fiscal Crisis Facing American Cities (Urban Proxima):]

***

As the cost of servicing the debt increases, Congress must borrow more, raise additional revenue, or devote a smaller share of the federal budget to everything else. Whichever path it chooses, the federal government will have less room to maintain the commitments on which American cities have come to depend. [...]

Federal money flows to cities in three flavors: direct transfers, indirect transfers, and what we call fiscal dark matter. Direct transfers are exactly what they sound like — money sent directly from the federal government to various localities. These include funds disbursed through programs like the Community Development Block Grant (CDBG), which supports things like public infrastructure and neighborhood services. In 2022, direct transfers like the CDBG totaled $146.3 billion. That’s significant, but actually the smallest of the three categories.

Less visible are the indirect transfers. These monies are initially awarded to state governments, which then allocate funds to municipal-level programs and services in accordance with state prerogatives. The cleanest example is probably K-12 education, which receives federal Title I dollars to pay for teachers and programs, but federal highway dollars work essentially the same way. All told, in 2022, the federal government handed down $1.1 trillion to state governments. That amounted to 36% of overall state revenue for that year and, depending on the individual state, ranged from roughly 22% to 50% of state revenue. How much of that ultimately flowed down to cities is hard to say, which is itself a problem: it’s difficult to even establish how exposed local governments are to a pullback in federal support of state budgets.

The third category – our fiscal dark matter – is all the federal money spent into local communities that never shows up in a local budget. This includes housing subsidies like the Low-Income Housing Tax Credit (LIHTC) and Section 8. It also includes food support programs like SNAP and even some direct funding for local food banks.

Rightfully or not, when the flow of federal money in this category starts to dry up, the resulting problems will fall squarely on the mayor’s desk. After all, the median voter is never going to see increasing numbers of homeless encampments and think to blame the head of HUD. [...]

The “eds and meds” economies that anchor cities like Pittsburgh, Cleveland, and Baltimore depend heavily on Medicaid reimbursements and federal research grants to sustain the hospitals and universities that rank among their largest employers. Cuts there could precipitate layoffs in the institutions that have been holding together post-industrial downtowns for 30 years. And therein lies the second part of the dark matter problem. Federal money doesn’t just fund services and pay for infrastructure. In some places, it also props up major employers who anchor the entire local labor market. [ed. And the integral supply chain business that support those services.]

Sheep that Bleat

My dog Toby and I were just back from our early morning walk and I was in the living room when I heard a single bleat of a sheep. It was 8:30. The sheep bleated again.

How to describe the sound? Demanding, worried, and hesitant. Half, “Well, what?” and half, “Oh dear, oh help!” Half low moan, half high and wavering supplication. It’s a sound, like birdsong, that I’d have to replay endlessly to describe even half-right. And yet I recognized it immediately as different from the bleating of the small herd my neighbor’s nephew Jesús keeps in the fields across the lane from my house. This mournful bleat proclaimed, I am alone.

I looked out the window and saw the animal, emerging from the shadows where the lane dead-ended at the gated entrance to a field. It hesitated, then, almost deerlike, it advanced. What was it doing here, and where had it come from? Behind it I could see the fence at the entrance. It appeared intact. Beyond were the green fields where the sheep grazed among the partially built houses of an abandoned project on the ridge above the river. The bank owns the land now, but my neighbor’s family has the use of the fields. No other sheep were in view.

My son had heard the sheep, too, and we both stepped out into the lane. The sheep retreated into the corner against the fence. We didn’t have the nephew’s number, so we called my neighbor instead. He was in Holland, visiting his son for a few days, but he assured us he’d call his nephew right away. For good measure, he also gave us the number.

My son and I kept an eye on the sheep. I knew that my neighbor’s sister and nephew lived five minutes away. I expected them any moment. The sheep made a couple of tentative starts into the lane but retreated each time to its corner. The third time, instead of turning back when it came up against us in the lane, it stood its ground. Then it took courage, gathered itself, and made a mad dash around us—to the freedom of the open lane, nothing to stop it or impede its progress. It looked like a young male—I could see its testicles. I followed and saw it turn off the lane onto the grassy slope behind the sports center. If it continued in that direction, it would hit city streets. I moved back, hoping it would not press on. The grass on the slope was very green. Maybe the sheep would linger to nibble.

Back at the house, I wondered what was taking the owner so long. I went inside, but a few minutes later, from the corner of my eye, I saw a sheep in the lane, just like the earlier one, drumming up its courage to get past my son, still outside. Wait! I shook my head. It felt like déjà vu.

“Same one?” I asked, stepping back outside, and my son said yes, same one—back in its dead end, only to try to escape again. And still no nephew. My son sent a WhatsApp message.

The sheep made a last attempt. My son and I, prepared this time, jumped in front of it, waving our arms. At the last moment, the sheep veered, turned, and charged back to its corner. It made a funny little leap into the air, twice, to come down on all four feet in a show of petulance. Then it lay down on a patch of grass by the gate. Shortly afterward, a car pulled up.

Instead of the story ending, however, it took a turn. Rather than the nephew, it was his mother, with his girlfriend driving. The women explained that they got a call from my neighbor, explaining that the nephew hadn’t picked up. The mother and girlfriend couldn’t get in touch with him, either—the man was at work. So they’d come.

I pointed out the sheep, lying quietly in the shade of the wall. The sister wore a worried expression. The girlfriend seemed amused. How do we catch it, I asked? Oh, we won’t, the sister said. Another nephew in the family, cousin to Jesús, was coming. He would catch it.

So while we waited for the cousin, we watched the sheep. It was on its feet again. Was it favoring the front left leg, injured perhaps while wriggling free through a fence? Would it put up a fight? While we talked, a call came: Jesús, finally alerted by my son’s message. His mother passed the phone to the girlfriend. The cousin appeared, climbed out of his car, engine running and door wide open. He was looking past us as he advanced toward the sheep, backed up against the gate. Almost immediately, even before he’d had time to size up the situation, he turned to us. “That’s not Jesús’s,” he declared. The girlfriend passed the phone to the cousin, and in a minute he passed it back to her. She passed the phone to Jesús’s mother, as if rewinding a skein that had come unwound. A few words to Jesús, and the mother hung up.

“Well, whose sheep could it be, then?” I wondered aloud.

Nobody knew. Call the police, the cousin said, and left. The two women got in their car to drive off, too. As they turned the car around, I stepped up to the window. “Did you call?” I asked.

“No, no—we don’t have the number.” And they too drove off. I stared after them. Well, why wouldn’t they leave? The sheep wasn’t theirs.

It wasn’t mine, either, though.

To me, it looked like a sheep had been dropped among us, and we should all make a contribution to getting it where it belonged. My 45 minutes counted as my effort. What was theirs?

To them, the situation might look different—as if I’d started it all by worrying about a sheep when it wasn’t my business to worry about it, disrupting the morning of any number of people by involving them, too. And now I was trying to hand off the Old Maid card. They weren’t taking it.

by Clelland Coe, American Scholar |  Read more:

Image: John Fowler/Flickr

WTF Happened In 1971?

via: [more:]

Claude Nori (b.1949), Les Amants du Pô

via:

Now That I Am Forever With Child

How the days went

While you were blooming within me

I remember each upon each —
The swelling changed planes of my body —
And how you first fluttered, then jumped
And I thought it was my heart.

How the days wound down
And the turning of winter
I recall, with you growing heavy
Against the wind. I thought
Now her hands
Are formed, and her hair
Has started to curl
Now her teeth are done
Now she sneezes.
Then the seed opened
I bore you one morning just before spring —
My head rang like a fiery piston
My legs were towers between which
A new world was passing.

Since then
I can only distinguish
One thread within running hours
You . . . flowing through selves
Toward you.

by Audre Lorde

[ed. Read aloud here by Amanda Holmes.]

A Guide to Rock, Paper, Scissors:

The practical upshot is to start with paper, but what broader lesson should we take from this? Since all options are mathematically identical, any difference in player behavior must come from connotations. Since rock feels strongest, a naive child would choose rock. But since everyone knows that, a slightly smarter person would choose paper. But since everyone knows that, an even smarter person might choose scissors. But since . . . and the lesson of this graph is that when playing the average member of the population, all of this galaxy-brained thinking provides negative value, because in practice most people never leave the naive first level.

via: ACX

This, Too, Shall Pass

In the winter of 2021, I had a minor surgery that required me to spend a week off my feet. To this day I don’t know if it was the lack of mobility or the prescription narcotic that ground my digestive system to a halt, but by the time I realized I hadn’t gone to the bathroom in seven days, why mattered less than how—that is, how to get it out. Which is how I found myself on my toilet, drenched in flop sweat, learning the phrase “impacted stool” from Google.

I don’t have children, but when my friends tell their birth stories I now nod along in tacit understanding—I know the horror of “There’s no way that will fit through there,” of “Oh god, what if this has to be surgically removed?,” of writhing around for a better angle. I chugged the water my husband, god bless him, handed me through the cracked door. I felt cold and also hot. When it was finally over, I broke out in full body shakes—I was pale, giddy, in shock. I looked back in proud relief. My marriage grew stronger for what we’d achieved together. When I meet others who’ve endured it, we find immediate brotherhood.

This experience, though humbling, wasn’t totally out of the ordinary for me. I’m a slow-motility girlie—I pack prunes and Smooth Move tea on trips; for years, I bought a fiber cereal hilariously called Kashi GO; I once had an Uber Eats guy deliver me a Fleet enema. My best friend likes to remind me of a night we spent at the emergency room trying to diagnose mysterious stomach cramps I thought were appendicitis but turned out to be garden- variety constipation. I’ve had to get comfortable discussing the scatological with clinicians, pharmacists, roommates I’ve shared bathrooms with. But barring the occasional nod from my fellow comrades in what one doctor called “sluggish digestion,” I’ve mostly suffered these slings and arrows solo—that is, until recently.

One day, it seems, the internet woke up and chose BMs, and suddenly the hottest girl you know is posting about scat. Now you have to go out of your way to not eat fiber—prebiotic supplements are in our soda, our candy bars, our cold foam. The beverage case is dominated by the likes of Olipop and Poppi and Culture Pop, all offering varying degrees of digestive assistance. The other day, I saw a smoothie blend in the freezer aisle called “Fiber Supremacy,” and earlier this year, Frito-Lay launched a fiber-packed version of both SunChips and Smartfood popcorn. Kraft is making a purgative macaroni and cheese that boasts six grams of fiber per serving (or 15 grams if you eat the whole box—which is, let’s be honest, how most people consume Easy Mac). Outside the retail space, home cooks are finding ways to fit chia and flax into not just their breakfasts but their meat loaf. They’re pureeing beans into sauces and veggies into puddings and cakes. Literal doctors are posting their #fibermaxxing menus to TikTok. And while this may sound like just another wave on the ocean of content we surf daily, it gets…weirder.

I’ve covered the ingestion of sustenance extensively in my 20 years of food writing; the passing of it, less so. We are biologically inclined to separate food talk from waste talk—unless you’re a new parent, the human body recoils from the plate at the mere mention of excreta. As such, there’s a glaring lacuna in our field when it comes to the business end of our alimentary canal. But as #fibermaxxing has given way to #poopmaxxing, culinary talk is shifting from what goes in to how easily it comes out. I recently watched a girl on TikTok showing off her “PoopMaxxer purse” in which she carries Dude Wipes, after-poo spray, toilet seat covers, extra TP, instant oatmeal packets, and fiber gummies because, she says, “in addition to a poopy butt I also have a sweet tooth.” She signs off the video with “Goodbye poopies,” because, I guess, why not.

What is going on here? Some have theorized this is just the latest way to get skinny—Ozempic is expensive, but Poppi is cheap. It’s hardly a coincidence that this trend comes on the heels of GLP-1 mania and #proteinmaxxing, two dietary fads known to cause constipation. “They’re all getting colonics!” a friend recently exclaimed to me in a breathless tone of conspiracy-crazed revelation, of the ultrathin celebrities in our newsfeeds.

Others attribute the fixation to the sudden rise in colorectal cancer rates among young people—and given how hard it can be to access and afford health care in the United States, who could blame us for taking our well-being into our own hands? This may also be why we’re more obsessed with lifestyle optimization than ever before—I know some couples who rarely eat together, because of how personalized each of their diets have become. We have more access to dietary research and more ways to track health data than at any time in human history, prompting many to turn their bodies into full-blown science experiments.

Whatever the reason, it’s now de rigueur to be regular. What was once my personal bête noire is the obsession of wellness queens everywhere, who are FODMAPing, fibermaxxing, and colon cleansing their way toward slenderness, Bethlehem, nirvana. But what becomes of life’s simple pleasures (sharing a meal, eating someone else’s cooking) when we value function over all else? Sure, we have a whole supermarket’s worth of ways to grease the gears now—but is our fiber fixation maybe missing the point?

I grew up in the early aughts, an era governed by the food rules of women’s magazines. They told us to suck in our stomachs, buy skim milk, avoid egg yolks, skip meals—or else eat lots of small ones to boost metabolism—and mangle a sandwich roll by digging out the middle. In my efforts to move on from this hellscape, I learned to seek out whole foods, swapping out this noxious morality for what seemed like a more righteous one. I thought Michael Pollan had single-handedly saved me from a lifetime of disordered eating when he published In Defense of Food, dispensing his now legendary dietary advice: Eat food. Not too much. Mostly plants.

And for a while, this worked. I went to the farmers’ market, bought local dairy, had my first good tomato. I made kombucha and sourdough bread and buckwheat brownies. Heck, I became a food writer. By embracing Pollan’s edict, I saw how food could be a paradise of unlimited possibility instead of a restrictive environment of guilt. I learned to approach food with curiosity instead of fear. I still sucked in my stomach out of vanity and believed in the virtue of frequent light snacking, but I was eating pretty much whatever I wanted (with the exception of my fiber cereal), and feeling healthier for it.

The body changes, though—it always does—and one day I woke up in my late 30s on various meds for migraine and anxiety and allergies that all listed constipation as a symptom, bloat bulging out the zipper of my high-waisted jeans. My BMs were fickler than ever and, for the first time in over a decade, I was feeling the urge to diet. Instead, I went to my GP, who recommended fruits, veggies, and grains—foods I was already consuming en masse, thanks to Mikey P. So I took psyllium. I got acupuncture. I went to a GI doctor who prescribed a weapons-grade laxative. I Googled “CBT for IBS” and tried meditation, in case this was all simply a matter of unclenching. I started taking 800 milligrams of magnesium per day and carrying Senna pills in my purse. The bloat only seemed to be getting worse. One friend suggested I just buy bigger jeans.

All of this is to say, the robots weren’t my first choice.

by Linni Kral, Taste |  Read more:

Image: Max Erwin

[ed. See also: A Clockwork Ozempic (Taste).]

Amanda Acker (b.1982) - Blue Night. 2025.

via:

What is It Like to Live in a World You Believe is About to End?

I opened my interviews for this article with a simple question: “How long do we have?”

Five years, or five to 10, or five to 20. One person said eight, then corrected herself to six; one said eight and stuck to it.

In this, they aren’t that far off from many estimates made by experts. The bluntly titled If Anyone Builds It, Everyone Dies has hit bestseller lists by warning of the imminent risks of artificial general intelligence (AGI). The scenario AI 2027, written by a former OpenAI employee, predicts AGI within the next few years. The AI company Anthropic consistently predicts AGI by early 2027.

“I think that in most timelines, humans will simply be irrelevant and extinct,” one interviewee said.

I heard that a lot. A few interviewees — mostly employed by frontier AI labs — expected the world to become unimaginably strange in a good way. One put a 30% chance on utopia, a 30% chance on dystopia, a 30% chance on extinction, and a 10% chance on something too weird to imagine. Several refused to make any prediction; several more said only that they still had hope. About half echoed one of my most blunt respondents: “I don’t think humanity is going to make it.”

What is it like to live in a world you believe is about to end?

Death was already inevitable

“I was born with a terminal condition,” said Matthew Gray, a board member at the existential risk community-building nonprofit Lightcone Infrastructure. “We call it aging. I’ve since picked up another. We call it multiple sclerosis. And AI is a third one on top. I’m not very worried about degenerating from multiple sclerosis because I’m pretty sure the robots will kill me first, just like I wasn’t that worried about aging-related deterioration because multiple sclerosis will get me first.”

From this perspective, AI doomers don’t face a new problem; they face the oldest problem humanity has ever faced.

I pushed back. If I look at an actuarial table, I can expect another 47 years of life. I’d be pretty upset to discover I had only five.

This, my interviewees thought, was naive. Even without AI risk, I could have been hit by a car; I could have gotten cancer; I could have been nuked in a hot war between Russia and the United States. It’s not that the difference in probability doesn’t matter. It’s worse to be certain that I’ll die in five years than to have a 50% chance of not hitting my allotted 47. But because my death has always been an inevitability, I have been coping all along with the precarity of my existence. From this perspective, AI risk isn’t shocking and unfamiliar; it’s a significantly worse version of a problem I already know I have to deal with.

“I was never guaranteed that I was going to get a long life and a long future and a chance to meet my grandchildren,” said Gretta Duleba, an independent technical AI safety researcher and former communications manager at the Machine Intelligence Research Institute. “Those were never my right. Across human history, no one has ever been entitled to the future.”

Throughout the entire scope of human experience, many of my respondents said, apocalypse has been more the rule than the exception: the Holocaust, the Black Death, the An Lushan Rebellion, the Thirty Years’ War. AI doom, as many people pointed out, is the latest and the last iteration of a societal universal. AGI might be the end of the actual entire world, but it is far from the first time people have faced the end of their own individual worlds.

And AI doom is a remarkably cozy catastrophe. If you suffered through a historical apocalypse, you’d expect to starve, be raped, watch your children die in front of you, die a slow and lingering death of smallpox or plague or wound infection. The AI apocalypse — at least for those with the slack to be worried about it — takes place in a world of wealth and relative peace and technological marvels.

“Enjoy the fact that you get to have hot showers,” said Duleba. “Enjoy the fact that you get to eat delicious food. Enjoy the fact that you get to do escape rooms, which is one of my favorite things. This is great. Have you noticed how great this is?”

“There’s at least some hope that AI might become good,” said Robert Herr, a former senior political staffer who is transitioning into AI policy work, “and that is a lot more than many, many billions of people in history had.”

For some people with short AI timelines, the enormity of the AI apocalypse is its own perverse source of comfort. Once, they had to worry about many things: climate change, malaria, factory farming, democratic backsliding, the fertility crisis. Now, instead of many big problems, they have one enormous problem. Worry about AI frees them from having to worry about anything else.

“When you’re diagnosed with prostate cancer,” said Adam Grey, who isn’t involved in AI research but who follows AI news, “a lot of the time doctors say not to bother treating it because you’ll die of something else. This is the thing that’s going to kill us first — us as a civilization and also personally me. It clarifies what the most important issues are.”

Ambiguous loss

Although short AI timelines can be a source of clarity, some people also struggle with the uncertainty of humanity’s fate. Duleba, who was a therapist before she switched to working on AI, told me about the concept of “ambiguous loss,” originally developed by Pauline Boss in the 1970s.

In normal grief, your loved one is dead. While it’s painful, you know that it will never change. Ambiguous grief, however, occurs when a loved one is kidnapped, or is a soldier missing in action, or has slowly worsening dementia with occasional good days farther and farther apart. Your loved one’s death is never really over, so you can never really grieve. You are trapped in a cycle of mourning that never resolves.

AI doom can be a situation of ambiguous loss. You can’t know for sure when it will happen or whether it will happen at all — but the more you understand what’s going on, many people find, the easier it is to grieve and move forward.

“The more I know about something, the less I’m freaked out always,” said Tao Lin, a member of technical staff at a frontier AI lab (and a close personal friend). “The less I know about something, the less I will be rational about it. The rational part of your brain needs information to operate, and just having more information will make you be more premeditated and system 2 about everything.”

When he felt doomy, he did AI forecasting to put concrete numbers on his uncertainty. (He believes there is about a 20% chance of human extinction, a 40% chance of ”a great outcome,” and a 40% chance that “people survive and have a great time, but stuff is broadly bad.”) [...]

Most interviewees emphasized that the most important thing to understand about AI risk was how little control you had over it.

“There’s not much use worrying about a thing if the outcome is determined,” said Alyssa Riceman, a software engineer. “You’re just going to burn a whole bunch of emotional energy not making any changes out in the world. It’s only worth worrying about things if you’re in a position of control over them. So go out and check if you’re in a position of control, and if you are, control them.”

But what if you have partial control over a situation?

“Then you have to game out all the branches,” Riceman said. “Say ‘I can do this. If I do this, what happens then?,’ until everything bottoms out at either a situation you can completely control or a situation that’s out of your control.”

Duncan Sabien, who is the current communications manager for the Machine Intelligence Research Institute, agreed. “I actually have no control over whether we succeed or fail,” he said. “All I can control is my own actions. And so if I am doing the best I can with what I know and what I have available to me, then that is the best I can do. I go home feeling like I’m a good person, and I get to go to sleep at night feeling like if the AI does kill us all, I did as much as I could, realistically and sustainably, to prevent it. And everything else is out of my hands. Everything else is always out of my hands.”

Living well in the apocalypse

What, then, do people decide to do?

by Ozy Brennan, Asterisk |  Read more:

Image: Karol Banach

Yuri Klapouh (Ukrainian b.1963), Apple Tree, Oil on canvas

via:

What Do Consultants Get Paid For?

A consultant I had lunch with recently is redesigning the loyalty program of a large airline. His team finished the analysis in two weeks. Months later the program still does not exist. This is because the purpose of the assignment is not to solve an analytical case study, but to figure out which redesign the parties will accept, and to get the people with authority to commit to implementing it.

I was not surprised to hear the story. In our just-published book Messy Jobs: The Work That AI Cannot Reach, Jin Li, Yanhui Wu, and I argue that a job is not a collection of independent tasks but a bundle of tasks and a position inside an organization. While many of the constituent tasks are clean, the job is messy because they must be combined under incomplete knowledge, conflicting objectives among the different parties and binding constraints on who has the authority to make decisions.

Hence we argue that automating the clean parts does not necessarily eliminate the job, because the remaining activities, tightly bundled with the rest, can remain the constraint. We argue that the bundle is strongest where separating the analytical/cognitive parts that can be automated would destroy local knowledge, trust, accountability or continuity.

Two objections

Critics of our argument raise two concerns. The first one has to do with advances in AI capabilities: models do some tasks extremely well and others badly. As they gain memory, use tools and acquire multimodal perception, critics would say, AI will get better at many other tasks like persuading, anticipating the objections raised in a meeting and adapting the tone. Hence even the interpersonal part by itself may not be a sanctuary for long. Just wait a bit for AI to get better, say the critics: Messy Jobs (in their view) describes the transition rather than the long run.

The second objection to our thesis is more radical. Maybe as long as we have humans in the loop, we need organizations. But if organizations really are a mess, slow, political, resistant to change, with a large role for humans precisely because someone has to hold meetings, build coalitions and learn the internal politics, why not get rid of the entire organization? What is the point of preserving the existing obsolete structures?

We believe that both objections fail, because some of the mess is substantive and necessary.

Where the “implementation” months go

To an outsider, my friend’s consulting project looks purely analytical. The team receives all the data, including all passenger records, redemption rates, customer-retention data, and so on. The team works out the key economic and financial trade-offs of the possible redesigns to figure out which redesign maximizes profits.

If doing this, given all the available data and the current AI tools, took two weeks, why has the project taken many months?

First, inside the airline, different parts of the business worry about different things. For instance, the salespeople have relationships with the hotel chains and do not want to disturb them, while the operations team worries about the staff at the airport counters who will have to deal with angry passengers who have grown used to certain privileges.

Second, there are the outside parties, from hotel chains to credit-card companies to the retailers that accept miles. Anything that improves the airline’s economics may reduce the value of the program to the hotels or to the card issuers. Each has a view on how card spending should count relative to flying, or how hotel nights should count relative to flying. There are winners and losers everywhere, and a reform that benefits the airline as a whole can hurt a particular business unit or a particular partner.

So the consultants spend weeks doing an enormous amount of work that looks peripheral to the problem. They repeatedly meet the head of the loyalty program, then the CFO, then the CEO. They also meet the commercial partners, and that means meeting the head of loyalty, then the finance team, then the chief executive. They revise the proposal. They redo the presentation.

And all of these people speak different languages. Organizations have different internal codes because they care about different things and deal with different problems. What the consultants are doing is a mix of analysis, translation and intermediation. The assignment of the consultants is to design a program that is an agreement that the relevant parties will authorize and implement.

Once the analysis is cheap, what remains is to learn what each party will actually accept, and to obtain commitments from those who are authorized to make them.

The real knowledge problem

An advocate of highly capable AI systems (“AGI-pilled”) would probably say this is a problem ready for AI. Have an agent redesign the program, have it meet the other constituencies, have it come back with a solution.

But what happens in those meetings deserves a closer look. There are four frictions in the room that make the meetings necessary.

by Luis Garicano, Silicon Continent | Read more:
Image: via