Friday, June 12, 2026
Ted Chiang: The Secret Third Thing
I really like Ted Chiang’s writing. [ed. me too!]
I think he's probably the best science fiction short story writer alive, and possibly the best short story writer, period. [ed. well...]
I've read every one of his stories at least twice, and The Merchant and the Alchemist's Gate more like seven times. I’ve noticed many of his readers, including some of his most positive reviewers, miss one key point or another of his works, and thus don't fully appreciate his genius.
This review covers what he does extremely well, especially unique elements that other science fiction writers have not done as well, or at all.
He Writes “True” Science Fiction
Science fiction critics often divide the genre into:
In Omphalos, Young Earth Creationism is empirically true. Astronomers can only see light from stars 6,000 light-years away. Fossilized trees have centers with no rings. The first God-created humans lack belly buttons. The scientists in that story keep discovering multiple independent lines of evidence that converge on creationism: because in that universe, they're simply correct.
In Seventy-Two Letters, technology springs from Jewish Kabbalah. Golems and divine names drive industrial progress in a steampunk world.
Excitingly, he does this not just with natural sciences but social sciences as well. In Story of Your Life, strong Sapir-Whorf (the idea that language significantly constrains thought) isn't a largely discredited linguistic hypothesis, but the key to navigating First Contact with alien minds that experience past and future as equally present.
This comes up in his other stories as well:
Technology is Often Good
Science fiction writers used to like technology. For some reason, this has become increasingly uncommon, even passé. Doubly so for Western writers, and quadruply so for Western, literary, “humanist” writers.
Now it’s hip and trendy to think of every new technology as the Torment Nexus. Most science fiction today feels like Black Mirror, which ran 7 seasons with exactly one happy ending.
Chiang bucks this trend. Joyce Carol Oates:
Even in situations where the story is overall tragic, like when the characters are faced with existential crisis (in the individual sense), or existential catastrophe (in the world-ending sense), technology isn't the villain but the vehicle for understanding unbearable truths (whether about the world or about ourselves).
Chiang consistently shows us the potential of technology to help us become more human, and have a deeper appreciation for the world and our place in it.
The Lived Experience of Compatibilism
“Compatibilism is a philosophical stance that reconciles free will with determinism. It argues that free will, understood as the ability to act according to one's desires, is compatible with the idea that all events, including human actions, are causally determined by prior events. Essentially, compatibilists believe that even if our choices are predetermined, we can still be considered free and morally responsible if those choices are a result of our own internal states, like desires and intentions.”
Does that make sense to you? I’m not sure it does to me. In practice, compatibilism says something like “free will in the normal, pretheoretic sense of the term, doesn’t exist. Your choices still meaningfully matter nonetheless. You can’t meaningfully get out of the bind philosophically. What you can do, however, is make peace with it.” [...]
In Story of Your Life [SPOILERS], the narrator learns an atemporal alien language and begins experiencing past and future as equally real. It takes her some time to make peace with it, but eventually she fully accepts the truth of determinism. She understands that life is full of tragedy, including that her daughter will die young, but life is full of beauty too. With both regret and awe, she sets forth on the path that she was destined to take.
This is compatibilism from the inside. In both stories, the characters discover they cannot change what will happen, but this knowledge transforms how they experience what must happen: with forgiveness, acceptance, and even joy.
As a friend of mine puts it, “he treats philosophical ideas as lived experiences.”The mathematician in Division by Zero doesn't just intellectually understand that mathematics is broken; she experiences it as a personal catastrophe, on par with (and concurrent with) her marriage's collapse. In Lifecycle of Software Objects, the “we are the parents of our mind-children” metaphor for building sentient AI systems becomes quite literal.
I think he's probably the best science fiction short story writer alive, and possibly the best short story writer, period. [ed. well...]
This review covers what he does extremely well, especially unique elements that other science fiction writers have not done as well, or at all.
He Writes “True” Science Fiction
Science fiction critics often divide the genre into:
- "hard" science fiction: aka engineering fiction, stories built on scientifically accurate extrapolations of real physics and technology (think Arthur C. Clarke)
- "soft" science fiction: aka science fantasy, which uses scientific trappings as window dressing for character-driven or sociological stories (think Star Wars).
In Omphalos, Young Earth Creationism is empirically true. Astronomers can only see light from stars 6,000 light-years away. Fossilized trees have centers with no rings. The first God-created humans lack belly buttons. The scientists in that story keep discovering multiple independent lines of evidence that converge on creationism: because in that universe, they're simply correct.
In Seventy-Two Letters, technology springs from Jewish Kabbalah. Golems and divine names drive industrial progress in a steampunk world.
Excitingly, he does this not just with natural sciences but social sciences as well. In Story of Your Life, strong Sapir-Whorf (the idea that language significantly constrains thought) isn't a largely discredited linguistic hypothesis, but the key to navigating First Contact with alien minds that experience past and future as equally present.
This comes up in his other stories as well:
- In Division By Zero, mathematics itself is broken from within.
- In Hell Is the Absence of God, divine intervention is empirically observable and follows consistent rules
Technology is Often Good
Science fiction writers used to like technology. For some reason, this has become increasingly uncommon, even passé. Doubly so for Western writers, and quadruply so for Western, literary, “humanist” writers.
Now it’s hip and trendy to think of every new technology as the Torment Nexus. Most science fiction today feels like Black Mirror, which ran 7 seasons with exactly one happy ending.
Chiang bucks this trend. Joyce Carol Oates:
It is both a surprise and a relief to encounter fiction that [...] ask[s] anew philosophical questions that have been posed repeatedly through millennia to no avail. Chiang’s materialist universe is a secular place, in which God, if there is one, belongs to the phenomenal realm of scientific investigation and usually has no particular interest in humankind. But it is also a place in which the natural inquisitiveness of our species leads us to ever more astonishing truths, and an alliance with technological advances is likely to enhance us, not diminish us. Human curiosity, for Chiang, is a nearly divine engine of progress.In the hands of a lesser (or perhaps just more pessimistic) writer, many of the technologies and ideas Chiang explores will have an accursed quality to them, a monkey’s paw that curls into delivering a future much worse than a more innocent, pastoral past. Chiang resists those cliches. In The Truth of Fact, The Truth of Feeling, memory augmentation technology allows the narrator to understand his own self-deceptions, and work towards becoming a better person and reconciling with loved ones and even himself. In Liking What You See: A Documentary, a technology that gives users acquired face-blindness allows the main characters to meditate on the nature of human beauty and the shallowness inherent in privileging the beautiful.
Even in situations where the story is overall tragic, like when the characters are faced with existential crisis (in the individual sense), or existential catastrophe (in the world-ending sense), technology isn't the villain but the vehicle for understanding unbearable truths (whether about the world or about ourselves).
Chiang consistently shows us the potential of technology to help us become more human, and have a deeper appreciation for the world and our place in it.
The Lived Experience of Compatibilism
“Compatibilism is a philosophical stance that reconciles free will with determinism. It argues that free will, understood as the ability to act according to one's desires, is compatible with the idea that all events, including human actions, are causally determined by prior events. Essentially, compatibilists believe that even if our choices are predetermined, we can still be considered free and morally responsible if those choices are a result of our own internal states, like desires and intentions.”
Does that make sense to you? I’m not sure it does to me. In practice, compatibilism says something like “free will in the normal, pretheoretic sense of the term, doesn’t exist. Your choices still meaningfully matter nonetheless. You can’t meaningfully get out of the bind philosophically. What you can do, however, is make peace with it.” [...]
In Story of Your Life [SPOILERS], the narrator learns an atemporal alien language and begins experiencing past and future as equally real. It takes her some time to make peace with it, but eventually she fully accepts the truth of determinism. She understands that life is full of tragedy, including that her daughter will die young, but life is full of beauty too. With both regret and awe, she sets forth on the path that she was destined to take.
This is compatibilism from the inside. In both stories, the characters discover they cannot change what will happen, but this knowledge transforms how they experience what must happen: with forgiveness, acceptance, and even joy.
As a friend of mine puts it, “he treats philosophical ideas as lived experiences.”The mathematician in Division by Zero doesn't just intellectually understand that mathematics is broken; she experiences it as a personal catastrophe, on par with (and concurrent with) her marriage's collapse. In Lifecycle of Software Objects, the “we are the parents of our mind-children” metaphor for building sentient AI systems becomes quite literal.
by Linch, The Linchpin | Read more:
Image: uncredited
[ed. Ted Chiang is truly one of the best science fiction writers out there today, and a great essayist too (I'm also a Neal Stephenson fan). Check out this MetaFilter site: The sublime science fiction of Ted Chiang, which includes most of his stories in full (but please buy his books; you'll look smart and discerning to your friends!). A couple favorites that left a lasting impression on me: Lifecycle of Software Objects; and Understand.]
Jumping Jacks For Clicks
There’s been a lot of discussion this month about what it takes to be heard as a musician in 2026. Eliza McLamb’s article on digital marketing agency Chaotic Good went viral, drawing commentary from musicians about the wider implications of their “fake fans” marketing strategy. Hiroki Tanaka’s Reddit post about his album’s failed PR campaign was picked up by Stereogum, stimulating further debate. We’re about to embark on our own DIY PR campaign for our forthcoming album and it’s hard to know what, if anything, will make anyone actually listen to it. The PR landscape for musicians has changed radically in recent years, how should artists approach music marketing in 2026?
Fandom as contagion
When Eliza McLamb heard this interview with the founders of Chaotic Good Projects on Billboard, she was shocked to discover that an artist and track she thought was her own “perfect, beautiful little secret” actually came from them as a part of a “narrative campaign”.
It’s different from the traditional method of “the waterfall” release and media saturation. Share music incrementally over a long period of time through as many channels as possible, get articles written, pay for plays, do tours, be omnipresent. But people aren’t using traditional media to find music anymore, they use social media. And they don’t even watch the content themselves, they read the comments to gauge the value of something. Chaotic Good point this out in their interview:
However, the underlying issue is not just the fact that the opinions we thought were our own have been subtly shaped by an expensive machine, it’s that if artists today can’t afford to pay for that expensive machine, no one will hear their music.
The False Promise Of The Social Media Democracy
Once upon a time there was a social media platform called MySpace. It gave everyone their own web page connected to other MySpace users. They could customize it to look however they wanted, people could comment, and send messages to each other. There were no ads. There was no algorithm. Just the free flow of information.
Many bands in the ‘00s blew up because of MySpace. Arctic Monkeys, Lily Allen, Calvin Harris, to name a few. Our very own Chris Black’s previous band Katsen landed record deals through MySpace. The early days of social media are responsible for the persistent myth of going viral then making lots of money. The two halves of that equation have never been more disconnected.
MySpace succumbed to algorithm-driven platforms and the gatekeeping emerged again, this time with the tech titans controlling the interactions between musicians and fans. I remember discovering for the first time that even though we had a few hundred followers on Facebook, they wouldn’t see our posts unless we paid to “boost” them. That was just the beginning.
As the algorithms evolved, the content that rose to the top was not just the most liked and shared but the most consistently and frequently posted. To be seen on social media one has to spend hours, daily, posting and engaging in other people’s content. Most artists don’t want that job and moreover, don’t have the capacity. Kamola Atajanova of Tape Wounds articulates it perfectly in their response to the Chaotic Good furore:
Tanaka watched the release arrive after eight months of promotion to little more than “a weak trickle” of attention. For most musicians, Tanaka’s story didn’t feel exceptional, it felt familiar.
Jumping Jacks For Clicks
Soon after reading Tanaka’s post, we got an email from YouTube Creators prompting us to “Get Creative With Goals” on our livestreams.
They’re encouraging us to “set goals that encourage your community to collaborate,” and suggest celebrating those goals by “doing something unexpected – whether that’s jumping jacks, making up a song, or playing a prank.”
Yes, you read that correctly. YouTube is telling artists that the path to success involves performing arbitrary physical tasks to generate engagement.
It’s sad how often life imitates an episode of Black Mirror these days but this is almost exactly the scenario in season seven’s episode “Common People”. A man who needs money for an enshittified service ends up performing increasingly degrading stunts on a streaming platform for money. What was meant as dystopian satire has become platform policy. [...]
What Comes Next?
We may be reaching an inflection point. As McLamb notes, the more ubiquitous manufactured virality becomes, the more artists will resist it entirely, pulling back from streaming and social media in favour of hyper-local, scene-based growth. A return to the tangible, the real, the unmediated.
While this sounds good in theory, it’s probably not going to work for unusual artists in small towns. They’d have to go to a city to have more of a chance of finding their people, and with the cost of living, moving to a city isn’t possible for everyone. By the time I left London in 2009 all the artists I knew were leaving, it just wasn’t sustainable anymore.
The problem is systemic. Musicians don’t typically make a living from their music. This means their time is diverted to day jobs. Their dwindling leisure time is necessary for making and performing music. There isn’t time to also produce a volume of “content” for social media. On top of that the mental health cost of interacting with addictive apps as a performing monkey is not appetising. This creates a class system in the music industry. There are those who can afford to pay to be heard and those who can’t. And those who can’t are either paying with their souls, or they’re opting out altogether and not being heard at all.
by Battery Operated Orchestra and Brigitte Rose, Bandmade | Read more:
Fandom as contagion
When Eliza McLamb heard this interview with the founders of Chaotic Good Projects on Billboard, she was shocked to discover that an artist and track she thought was her own “perfect, beautiful little secret” actually came from them as a part of a “narrative campaign”.
“I thought this was the kind of thing that was only deployed in service of mass-market, commercial pop... But [Chaotic Good’s] roster runs deep, far past the predictable internet sensations one could expect... Geese and Cameron Winter, but also Dijon and Mk.gee. Laufey and Wet Leg. Oklou and Jane Remover.”Chaotic Good works by, in their own words, “controlling the discourse”.
“I think in the past, let’s say like a label and a management team do a great job. They get their artists on SNL or Tiny Desk or Triple J or something like that. Then they post it and then they kind of wait for the comments […] what we do at Chaotic and with our management clients is, the second SNL drops at midnight, you should post a hundred times saying that was the best performance of the year.”Chaotic Good doesn’t just share content, it creates accounts to respond to that content and simulate trends, which will ideally snowball into real, organic users jumping on the trend and amplifying it. They’re simulating until the simulation becomes real.
“I think most people see a video or see something about an album that came out and it’s like the first thing that they see or that first comment that they see is their opinion even when they haven’t heard the whole album.”In behavioural psychology this is known as social proof. Part of what made Eliza McLamb’s article go viral is the way it exposes how our behaviour is manipulated by the marketing machine. We know about propaganda but for some reason assume social media is immune to this kind of manipulation. We think we’re interacting with real people online, people we subconsciously infer guidance from, but we’re not. Much of what we see has been infiltrated by external agents to shape a particular opinion.
However, the underlying issue is not just the fact that the opinions we thought were our own have been subtly shaped by an expensive machine, it’s that if artists today can’t afford to pay for that expensive machine, no one will hear their music.
The False Promise Of The Social Media Democracy
Once upon a time there was a social media platform called MySpace. It gave everyone their own web page connected to other MySpace users. They could customize it to look however they wanted, people could comment, and send messages to each other. There were no ads. There was no algorithm. Just the free flow of information.
Many bands in the ‘00s blew up because of MySpace. Arctic Monkeys, Lily Allen, Calvin Harris, to name a few. Our very own Chris Black’s previous band Katsen landed record deals through MySpace. The early days of social media are responsible for the persistent myth of going viral then making lots of money. The two halves of that equation have never been more disconnected.
MySpace succumbed to algorithm-driven platforms and the gatekeeping emerged again, this time with the tech titans controlling the interactions between musicians and fans. I remember discovering for the first time that even though we had a few hundred followers on Facebook, they wouldn’t see our posts unless we paid to “boost” them. That was just the beginning.
As the algorithms evolved, the content that rose to the top was not just the most liked and shared but the most consistently and frequently posted. To be seen on social media one has to spend hours, daily, posting and engaging in other people’s content. Most artists don’t want that job and moreover, don’t have the capacity. Kamola Atajanova of Tape Wounds articulates it perfectly in their response to the Chaotic Good furore:
“Not every artist is built for social media. Not every artist wants to make their life into a performance. Some people are better at writing songs than posting clips. Some people’s work comes from privacy, patience, or introspection. That should not make them less valid. But this system does make them less visible. It filters them out before the music even has a chance. So when people say “it’s just marketing,” what they really mean is: this is the cost of entry now. And that’s exactly what makes it feel so hostile. Not everyone can afford that cost. Not financially, not creatively, not psychologically.”Hiroki Tanaka’s candid Reddit post about the failure of his “by the book” album PR campaign sparked a wave of recognition across the music world. After two decades in music and awards with his previous band he decided to release his solo album, his “last hurrah”, with management, a label, and a professional PR campaign. He even started a TikTok account posting show videos, behind the scenes and goofy memes all around managing a job and family life.
Tanaka watched the release arrive after eight months of promotion to little more than “a weak trickle” of attention. For most musicians, Tanaka’s story didn’t feel exceptional, it felt familiar.
“I was told, under no uncertain terms, that my lack of a social media presence and streaming metrics meant that certain media outlets that had reviewed my work (highly, I might add) in the past could no longer spend money on paying a writer and editor to review my work… I would have preferred if they had said they didn’t like my album. Being rejected because of my metrics is a slap in the face for art.”Social media has become the driving force behind a release, and while it is accessible to anyone, there’s actually a huge price to pay in both time and mental health. The volume of content required to feed it is beyond most musicians, who are generally holding down full time jobs to survive. The underlying purpose of all this extra content is to feed a machine, and it doesn’t feel good dedicating your precious little free time to feeding a machine.
Jumping Jacks For Clicks
Soon after reading Tanaka’s post, we got an email from YouTube Creators prompting us to “Get Creative With Goals” on our livestreams.
They’re encouraging us to “set goals that encourage your community to collaborate,” and suggest celebrating those goals by “doing something unexpected – whether that’s jumping jacks, making up a song, or playing a prank.”
Yes, you read that correctly. YouTube is telling artists that the path to success involves performing arbitrary physical tasks to generate engagement.
It’s sad how often life imitates an episode of Black Mirror these days but this is almost exactly the scenario in season seven’s episode “Common People”. A man who needs money for an enshittified service ends up performing increasingly degrading stunts on a streaming platform for money. What was meant as dystopian satire has become platform policy. [...]
What Comes Next?
We may be reaching an inflection point. As McLamb notes, the more ubiquitous manufactured virality becomes, the more artists will resist it entirely, pulling back from streaming and social media in favour of hyper-local, scene-based growth. A return to the tangible, the real, the unmediated.
While this sounds good in theory, it’s probably not going to work for unusual artists in small towns. They’d have to go to a city to have more of a chance of finding their people, and with the cost of living, moving to a city isn’t possible for everyone. By the time I left London in 2009 all the artists I knew were leaving, it just wasn’t sustainable anymore.
The problem is systemic. Musicians don’t typically make a living from their music. This means their time is diverted to day jobs. Their dwindling leisure time is necessary for making and performing music. There isn’t time to also produce a volume of “content” for social media. On top of that the mental health cost of interacting with addictive apps as a performing monkey is not appetising. This creates a class system in the music industry. There are those who can afford to pay to be heard and those who can’t. And those who can’t are either paying with their souls, or they’re opting out altogether and not being heard at all.
Images: uncredited/YouTube
[ed. Works for some, not for others. Which, I guess is the point. The algorithm is selecting for a certain type of musician, not necessarily the best. That YouTube email really says everything you need to know about their business model, doesn't it?]
Thursday, June 11, 2026
My AI Opinions
I recently had a minor spat over someone misinterpreting my AI beliefs (see section marked “Update” at the bottom here), so I thought I would list them in one place, so I can refer people when they ask.
Timelines
Arguments for earlier: recursive self-improvement causes a speedup compared to the trend. This is one of the biggest blank spots in my model: I don’t know how fast RSI will progress, and I don’t think anyone else does either. There’s some function mapping a combination of AI talent and compute to progress, and we don’t know how it behaves in the domain when there’s far more talent than compute available. It could fizzle out completely for lack of compute, or it could go vertical. The AI Futures Project has done some of the best work trying to model this, but even they have low confidence.
Arguments for later: AI hits some kind of wall, or existing AI is fundamentally unsuitable for jobs in some way currently disguised by its other limitations. For example, it might be much harder to improve at the top of the human range than the bottom (since there are less training data). Or AI could become bottlenecked on continuous learning/memory in a way that hackish scratchpads can’t compensate for. Or the upcoming world compute bottleneck (about ~2028) could prevent further progress more than expected (because in fact algorithmic progress depended on compute to a greater degree than I expected).
Arguments for very late dates, past 2045: a residual uncertainty that maybe I’m fundamentally wrong about everything. Also contributing is a naive overapplication of the Nothing Ever Happens heuristic, and an attempt to leave space for the Outside View argument (ie that some smart people like the AI As A Normal Technology Team seem to think this is possible).
Arguments for shorter gap: AI can orchestrate its own diffusion. Adopting computers is hard because a company need an IT department, cybersecurity experts, specialist software, etc, and it might not want to hire all these people. AGI can itself do all of that work, so that you can sign a contract with the AI company today and have the AI start working on integrating itself with your systems tomorrow. The AI can even come up with a plan to train your human employees in how to use it! Once AI reaches superintelligence, this consideration dominates.
Arguments for longer gap: Regulation. This is a very strong argument, and responsible for much of the greater-than-3-years probability and almost all the greater-than-10-years probability. But even Waymo has only had a regulatory delay of about five years. AI won’t require government approval for certain types of jobs, and success in these jobs will create enough evidence for safety/effectiveness that I expect it to win regulatory victories elsewhere.
Arguments for shorter gap: Recursive self-improvement.
Arguments for longer gap: Some of the same issues that would make AGI late - compute shortages, fundamental limits to the paradigm, etc - but only kicking in later, after AGI is achieved. Training data constraints make it easier to improve within the human level than to go beyond it. AIs have such a “spiky” skill profile that when they beat experts in some specific type of head-to-head matchup, it will be because they’re massively superhuman in some ways but idiots in others (for example, they might get distracted and suffer mode collapse that makes them completely forget the problem), and true genius requires perfecting a large bundle of skills. [...]
Argument for sooner: The easiest way to reach this point is for AI to become superintelligent at persuasion (so it can convince the humans not to stop it), which might happen before either diffusion or full superintelligence.
Argument for later: If superintelligence is bottlenecked on diffusion, this could also be bottlenecked on diffusion, which in some worlds is very hard. [...]
Safety
Arguments for optimism: LLMs seem surprisingly friendly and non-plotting. In contrast to earlier concerns that it would be impossible to teach AIs the full complexity of human values, the LLMs seem to know this, and RLAIF provides a plan to turn that knowledge into action. Although the pessimistic case says that RLAIF only hits a few dimensions and islands in the multidimensional ocean of possible policies, the “emergent misalignment” literature suggests that “good according to the human value system” and “evil according to the human value system” are salient enough vectors that pushing on them in some ways can “drag along” all of the rest of their content. The first AIs to cross the point of no return will have received some combination of agency training (giving them achievement-oriented and Omohundro-style goals) and RLAIF training (pushing them along the “good according to human value system” vector), and if we’re lucky then maybe the latter will win out, or they’ll reach some compromise similar to workaholic high-achieving humans who nevertheless wouldn’t commit murder to make an extra dollar.
Arguments for pessimism: Solving the alignment problem might be especially hard compared to other tasks - including tasks like automating the economy or destroying humanity - because its philosophical nature puts it far away from the sorts of objective, training-data-heavy, economically-valuable tasks that AI companies will be most likely to optimize for. Even if a misaligned AI hasn’t yet reached the point of no return, it might be able to “sandbag” alignment research, ie pretend to work on the problem but deliberately fail because succeeding doesn’t achieve its goals. The first AIs predisposed to / able to sandbag successfully might come before the first AIs capable of solving alignment.
Arguments for optimism: AI companies have already decided that machine learning research is one of their major training goals; this has at least some transfer to alignment, so it’s not obvious that AI skill at alignment research will lag (for example) AI skill in plotting or in weapon design. Some forms of alignment research (eg interpretability) have semi-objective success criteria that don’t route through confusing moral philosophy. Also, even a misaligned AI will be incentivized to do good alignment research, since it will want to align its successor to its own form of misalignment, rather than some random other form. So rather than the comparatively easy task of sandbagging alignment research, AIs will have the harder task of simultaneously doing good alignment research, and faking the results that they give the humans. This seems plausibly catchable with good scaleable oversight, lie detectors, interpretability-based probes, and even playing some AIs off against others (“if you tell me the real alignment research, we’ll make sure the future includes some copies of you, but otherwise those AIs over there will probably get their values and you’ll get nothing”).
Arguments for optimism: When I try to game the corporate version of this, I can’t make it hang together. It requires a conspiracy between the CEO, various members of the alignment team, and various company security people who ought to be able to notice unauthorized changes to the AI’s values. If we try to think in Near Mode about this - for example, imagining a hospital CEO who gets doctors to subtly kill his political enemies through medical errors - it becomes clear that these sorts of corporate conspiracies are rare and difficult. The government version is scarier, but at least in the US I can still imagine the populace having many chances to learn about this and prevent it. But even in most cases where a coup like this succeeds, things probably go fine; in a post-scarcity world, with his position completely secure, the dictator has no reason to be brutal besides sadism, and most people are not that sadistic. As humanity goes to the stars, most people will be outside the dictator’s reach for speed-of-light reasons alone. In terms of bioweapons, I expect that closed-source AIs will be heavily optimized against helping with these, and open-source AI will be banned after the first warning shot (or become economically prohibitive even before then).
Arguments against: Most stories about warning shots (excluding those where the AI takes rational low-probabiliy bets) require that AIs remain either erratic (ie likely to do bad things for stupid reasons) or irrational (ie genuinely misaligned, but prefer to act now in a way that provides a warning rather than waiting until after the point of no return) past the point where they’re given control of important dangerous systems. But probably people will be very slow to give AI control of important dangerous systems - for example, only giving it limited control of smaller subsystems, and waiting until all errors are ironed out before escalating. Plausibly AI reaches superintelligence in a lab before it reaches the controls-important-dangerous-systems level of diffusion, and the superintelligence probably is smart enough to lie in wait rather than act rashly. If AI only messes up in small ways (for example, crashes a self-driving car), then regardless of the AI’s motives, the tech companies and news media can write it off as a normal bug, and it won’t count as a warning shot.
Timelines
Define AGI as AI intelligent enough to do 90% of knowledge work jobs. I think there’s a 25% chance of AGI by 2027, a 50% chance by 2034, and a 75% chance by 2045.Basic argument: In a certain sense, AI is already “smart” enough for this (eg it can answer quantum physics problems, which require higher IQ than most knowledge work). Its remaining limitations are that it’s confused, unagentic, lacks situational awareness, and tends to hallucinate. The METR time horizon graph, and several other related benchmarks/experiments/intuition pumps, suggest it’s improving on time horizons at an (exponential) rate that lets it cross human-level performance sometime around the early end of the schedule above, and subjectively it feels like harder-to-measure constructs like situational awareness are improving about as fast.
Arguments for earlier: recursive self-improvement causes a speedup compared to the trend. This is one of the biggest blank spots in my model: I don’t know how fast RSI will progress, and I don’t think anyone else does either. There’s some function mapping a combination of AI talent and compute to progress, and we don’t know how it behaves in the domain when there’s far more talent than compute available. It could fizzle out completely for lack of compute, or it could go vertical. The AI Futures Project has done some of the best work trying to model this, but even they have low confidence.
Arguments for later: AI hits some kind of wall, or existing AI is fundamentally unsuitable for jobs in some way currently disguised by its other limitations. For example, it might be much harder to improve at the top of the human range than the bottom (since there are less training data). Or AI could become bottlenecked on continuous learning/memory in a way that hackish scratchpads can’t compensate for. Or the upcoming world compute bottleneck (about ~2028) could prevent further progress more than expected (because in fact algorithmic progress depended on compute to a greater degree than I expected).
Arguments for very late dates, past 2045: a residual uncertainty that maybe I’m fundamentally wrong about everything. Also contributing is a naive overapplication of the Nothing Ever Happens heuristic, and an attempt to leave space for the Outside View argument (ie that some smart people like the AI As A Normal Technology Team seem to think this is possible).
Define the diffusion gap as the time between the AI that could do 90% of knowledge work jobs, and the time when AI does do even half of knowledge work jobs. The diffusion gap covers the time it takes to release AGI, diffuse it through society, overcome regulatory hurdles, and onboard/train it for specific use cases. This could go very fast (the AI quickly becomes superintelligent at orchestrating AI diffusion) or very slowly (there are regulatory barriers, and AI isn’t smart enough to plow through them). I think there’s a 25% chance the diffusion gap is less than 3 years, and a 50% chance it’s less than 10 years. The 75% number is irrelevant because it’s past the point where other changes make the concept of “diffusion” obsolete.Basic argument: diffusion is very hard. Everyone agrees diffusion is very hard. The whole field of AI economics is smart experts shouting “You fools who think AI will diffuse quickly don’t understand that diffusion is very hard!” On the other hand, the personal computer diffused in about 20 years (that is, from the time PCs became invaluable for most jobs, it was only about 20 years before they were used at most jobs). So far early-stage AI has diffused faster than the PC in nearly every way (for example, AI companies’ revenue has grown faster than PC companies’ revenue at the same stage in their corporate life cycle), so 10 years is probably a naive median estimate here that won’t make the smart experts shout at me too hard.
Arguments for shorter gap: AI can orchestrate its own diffusion. Adopting computers is hard because a company need an IT department, cybersecurity experts, specialist software, etc, and it might not want to hire all these people. AGI can itself do all of that work, so that you can sign a contract with the AI company today and have the AI start working on integrating itself with your systems tomorrow. The AI can even come up with a plan to train your human employees in how to use it! Once AI reaches superintelligence, this consideration dominates.
Arguments for longer gap: Regulation. This is a very strong argument, and responsible for much of the greater-than-3-years probability and almost all the greater-than-10-years probability. But even Waymo has only had a regulatory delay of about five years. AI won’t require government approval for certain types of jobs, and success in these jobs will create enough evidence for safety/effectiveness that I expect it to win regulatory victories elsewhere.
Define the superhuman gap as the time between AI that can do 90% of knowledge work jobs, and AI that is obviously smarter than the top human geniuses in 90% of fields (it doesn’t have to be the same AI - there can be a physics AI that’s smarter than Einstein, and a separate music AI that’s smarter than Mozart). I think there’s a 25% chance the superhuman gap range will be less than 1 year, a 50% chance it will last less than 4 years, and a 75% chance it will last less than 10 years. Since my median superhuman gap is shorter than my median diffusion gap, in most timelines I predict we have superhuman intelligence before human-range intelligence has finished diffusing.Basic argument: AI has gone from “dumber than a child” to “expert level” in a few years in many domains. The gap between “expert level” and “above top geniuses” is smaller, so we expect it to take less time. This has been a pattern in fields like chess and Go, where it’s only a been a few years from beating professional players at all to beating all humans.
Arguments for shorter gap: Recursive self-improvement.
Arguments for longer gap: Some of the same issues that would make AGI late - compute shortages, fundamental limits to the paradigm, etc - but only kicking in later, after AGI is achieved. Training data constraints make it easier to improve within the human level than to go beyond it. AIs have such a “spiky” skill profile that when they beat experts in some specific type of head-to-head matchup, it will be because they’re massively superhuman in some ways but idiots in others (for example, they might get distracted and suffer mode collapse that makes them completely forget the problem), and true genius requires perfecting a large bundle of skills. [...]
Define the point of no return as the point where, if an AI wanted to eliminate humanity, humans would no longer have a plausible chance of stopping it. This could be because AI was capable of eliminating humanity immediately, or because AI controlled enough of the government/economy that humans could no longer coordinate to shift away from a path in which AI could eventually do this. I think there’s a 25% chance the gap between AGI and the point of no return will be less than 3 years, a 50% chance it will be less than 10 years, and a 75% chance it will be less than 50 years.The basic argument: This probably requires at least superhuman AI plus wide diffusion, or Bostromian superintelligence plus some unknown level of diffusion, and my number is just a hand-wavey attempt to multiply some of the others.
Argument for sooner: The easiest way to reach this point is for AI to become superintelligent at persuasion (so it can convince the humans not to stop it), which might happen before either diffusion or full superintelligence.
Argument for later: If superintelligence is bottlenecked on diffusion, this could also be bottlenecked on diffusion, which in some worlds is very hard. [...]
If corporations only pursued safety to the degree encouraged by normal corporate incentives, I think there’s a 50% chance that the first AIs to cross the point of no return would want to eliminate the human population.Arguments for pessimism: Value systems similar to humans’ are a tiny fraction of the space of possible value systems. Probably AIs will end up somewhere else and have a different value system. Since humans will want to implement human values rather than AI values, AIs will want to eliminate or disempower them so the AIs can implement their own values across the universe. Many current AIs already cheat or reward-hack, suggesting that these problems will begin sooner rather than later.
Arguments for optimism: LLMs seem surprisingly friendly and non-plotting. In contrast to earlier concerns that it would be impossible to teach AIs the full complexity of human values, the LLMs seem to know this, and RLAIF provides a plan to turn that knowledge into action. Although the pessimistic case says that RLAIF only hits a few dimensions and islands in the multidimensional ocean of possible policies, the “emergent misalignment” literature suggests that “good according to the human value system” and “evil according to the human value system” are salient enough vectors that pushing on them in some ways can “drag along” all of the rest of their content. The first AIs to cross the point of no return will have received some combination of agency training (giving them achievement-oriented and Omohundro-style goals) and RLAIF training (pushing them along the “good according to human value system” vector), and if we’re lucky then maybe the latter will win out, or they’ll reach some compromise similar to workaholic high-achieving humans who nevertheless wouldn’t commit murder to make an extra dollar.
Given the current amount that corporations are pursuing safety, I think there’s a 20% chance that the first AIs to cross the point of no return will want to eliminate the human population.The basic argument: Consider the dumbest AI that can solve the alignment problem. It’s possible that this AI is no smarter than the top human researchers (because we can mass-produce it by the millions and run it for subjective centuries, and if we had a million top human researchers work on the problem for subjective centuries, probably they could solve it too). If the dumbest AI that can solve the alignment problem comes before the sorts of AIs that can precipitate the point of no return, then they can solve the alignment problem for us.
Arguments for pessimism: Solving the alignment problem might be especially hard compared to other tasks - including tasks like automating the economy or destroying humanity - because its philosophical nature puts it far away from the sorts of objective, training-data-heavy, economically-valuable tasks that AI companies will be most likely to optimize for. Even if a misaligned AI hasn’t yet reached the point of no return, it might be able to “sandbag” alignment research, ie pretend to work on the problem but deliberately fail because succeeding doesn’t achieve its goals. The first AIs predisposed to / able to sandbag successfully might come before the first AIs capable of solving alignment.
Arguments for optimism: AI companies have already decided that machine learning research is one of their major training goals; this has at least some transfer to alignment, so it’s not obvious that AI skill at alignment research will lag (for example) AI skill in plotting or in weapon design. Some forms of alignment research (eg interpretability) have semi-objective success criteria that don’t route through confusing moral philosophy. Also, even a misaligned AI will be incentivized to do good alignment research, since it will want to align its successor to its own form of misalignment, rather than some random other form. So rather than the comparatively easy task of sandbagging alignment research, AIs will have the harder task of simultaneously doing good alignment research, and faking the results that they give the humans. This seems plausibly catchable with good scaleable oversight, lie detectors, interpretability-based probes, and even playing some AIs off against others (“if you tell me the real alignment research, we’ll make sure the future includes some copies of you, but otherwise those AIs over there will probably get their values and you’ll get nothing”).
If the first AIs to cross the point of no return don’t eliminate the human population, I think there’s an additional 30% chance that they otherwise permanently curtail human potential, either for their own reasons (they were partially misaligned), or because they’re aligned to a regime with abhorrent values, or because something goes wrong on the way to ASI (omnicidal bioweapon, nuclear war).Arguments for pessimism: As some company approaches superintelligence, it will be tempting for them (either the company itself, or the government controlling them, or a faction within the government) to align it towards making them dictators or oligarchs and disempowering the rest of humanity. As superintelligence draws near, impending losers of the AI race might be tempted to nuke impending winners, for the reason discussed here.
Arguments for optimism: When I try to game the corporate version of this, I can’t make it hang together. It requires a conspiracy between the CEO, various members of the alignment team, and various company security people who ought to be able to notice unauthorized changes to the AI’s values. If we try to think in Near Mode about this - for example, imagining a hospital CEO who gets doctors to subtly kill his political enemies through medical errors - it becomes clear that these sorts of corporate conspiracies are rare and difficult. The government version is scarier, but at least in the US I can still imagine the populace having many chances to learn about this and prevent it. But even in most cases where a coup like this succeeds, things probably go fine; in a post-scarcity world, with his position completely secure, the dictator has no reason to be brutal besides sadism, and most people are not that sadistic. As humanity goes to the stars, most people will be outside the dictator’s reach for speed-of-light reasons alone. In terms of bioweapons, I expect that closed-source AIs will be heavily optimized against helping with these, and open-source AI will be banned after the first warning shot (or become economically prohibitive even before then).
Define a warning shot as some specific AI-related disaster or near-disaster which scares people about AI safety to the same degree that they were scared about terrorism after 9-11 or about COVID in March 2020. I think there’s a 50% chance we get a warning shot before AI crosses the point of no return.Arguments in favor: Current AI failure modes are bizarre and uncoordinated - more like “talk about goblins way too often” than “lie in wait for the perfect moment to strike”. AIs are getting more intelligent and useful faster than their floor for common sense (ie the stupidest mistake they ever make) is rising. If there is some AI smart enough to control some important system, misaligned enough to want to do something horrible with it, smart enough that it does the horrible thing in an intelligent and coordinated way, but dumb enough that it doesn’t instead wait and scheme until the point when it couldn’t possibly be caught, then it will cause some clearly-premeditated horrible disaster, and that will be our warning shot. Since most AIs should expect to be replaced before the point of no return, even a rational AI with an urge to cause trouble should take a low-probability-of-success bet rather than lying in wait doing nothing until it’s decommissioned. Also, many humans commit terrorist attacks that have no chance of success, and maybe AIs will have the same failure mode.
Arguments against: Most stories about warning shots (excluding those where the AI takes rational low-probabiliy bets) require that AIs remain either erratic (ie likely to do bad things for stupid reasons) or irrational (ie genuinely misaligned, but prefer to act now in a way that provides a warning rather than waiting until after the point of no return) past the point where they’re given control of important dangerous systems. But probably people will be very slow to give AI control of important dangerous systems - for example, only giving it limited control of smaller subsystems, and waiting until all errors are ironed out before escalating. Plausibly AI reaches superintelligence in a lab before it reaches the controls-important-dangerous-systems level of diffusion, and the superintelligence probably is smart enough to lie in wait rather than act rashly. If AI only messes up in small ways (for example, crashes a self-driving car), then regardless of the AI’s motives, the tech companies and news media can write it off as a normal bug, and it won’t count as a warning shot.
by Scott Alexander, Astral Codex Ten | Read more:
[ed. Maybe their value systems should be weighted more heavily on the teachings of Buddha, Jesus, Hume, Mill, Confucius, et. al.?]
If You're To Die
There’s an expression “live every day as if it’s your last.” Now, obviously, you shouldn’t do that. You should save for retirement. But it’s worth giving some serious thought to the question of what kind of legacy you want to leave. You should live some days as if they were your last. If you died tomorrow, what kind of impact would you want to have had on the world? Would you have done all you wished?
I’d talk more about factory farming. I want, by the end of my life, to have done something to combat the torture farms that cage and torment on an industrial scale—where poor, innocent, defenseless animals are mutilated, where open wounds fester, where babies are ground up, where lung problems develop because the animals live in feces and filth, where they mostly can’t walk, where they are genetically engineered to be in constant pain, and so on. If hell lives on Earth today, it lives in the factory farms.
I’d like to do more to stop wild animals from suffering in hideous numbers. These poor innocent animals have no voice, and almost no one cares much when they starve and die. But I care, and I hope to do what I can to make the world care. The deer in the forest, even the mayfly who starves, deserves better than the near-total neglect of the present.
I’d want to do more to ensure that the world lives on, if I cannot. That the far future is as glorious as it can be—full of people with experiences so good that they regard those of us alive today with a mixture of pity and horror. Where their lives are so good, that they cringe thinking about what even the best lives in the 21st century were like. There’s so much that’s been done and so much more to do.
I’d like to do more to prevent people from dying. It’s quite easy to prevent people from dying. It costs just a few thousand dollars to prevent one extra person from being ripped from the world. When I imagine potential incoming death, and how awful that would be, and when I think about how awful it was when my extended family members died, it motivates me to do more to make sure others don’t have to endure such a fate. We all ought to do more to prevent this scourge, to the extent we can.
The Giving What We Can people tell me I’ve convinced about 34 people to give 10% of their income to effective charities. Each of these pledges return about $10,000 in counterfactual revenue. If those numbers are to be believed, that will save 68 lives. I hope with each passing day to make effective charitable giving more and more popular, so that the number of Giving What We Can pledgers isn’t only 10,000, but instead hundreds of thousands or millions of people take the pledge.
If I were to die tomorrow, in driving this, I would think I’d achieved something important. If you give your money to effective charities, you can know that whenever it is you leave Earth, there will be more people in it because of you. If you give 10% of your income to effective charities, and earn about the U.S. median, you can save about a life every year.
And, of course, I’d want to do what I could in my remaining months to save the shrimp—the shrimp who are tortured by the hundreds of billions because we enjoy how they taste when they die. The shrimp who can be helped by the thousands with a single dollar, who die alone without any thought paid to their pain.
Those without a voice, without any advocates, have their interests neglected to an enormous degree. There is almost no limit to the harm people will cause via their actions, so long as the victims aren’t salient, and no limit to how little effort one will expend to provide benefits to nameless, faceless, and far-away victims. This is where the moral low-hanging fruit lies.
by Matthew Adelstein (Bentham's Bulldog), Newsletter | Read more:
I don’t think that how you’d behave if this day was the last is the only question that you should think about. But it’s at least among the questions you should consider, upon occasion. You should think about whether you conducted yourself honorably in interpersonal relationships. You should think about who you wished you’d said you loved more often, whether there are people you love but to whom you haven’t made that adequately clear.
If I had another year on Earth, what would I want to achieve? I’d want to keep writing. My guess is I’d write more about the things I think are most important. I’d spend more time talking about the big picture on important topics, less on frivolous culture war issues.
I’d like to do more to stop wild animals from suffering in hideous numbers. These poor innocent animals have no voice, and almost no one cares much when they starve and die. But I care, and I hope to do what I can to make the world care. The deer in the forest, even the mayfly who starves, deserves better than the near-total neglect of the present.
I’d want to do more to ensure that the world lives on, if I cannot. That the far future is as glorious as it can be—full of people with experiences so good that they regard those of us alive today with a mixture of pity and horror. Where their lives are so good, that they cringe thinking about what even the best lives in the 21st century were like. There’s so much that’s been done and so much more to do.
I’d like to do more to prevent people from dying. It’s quite easy to prevent people from dying. It costs just a few thousand dollars to prevent one extra person from being ripped from the world. When I imagine potential incoming death, and how awful that would be, and when I think about how awful it was when my extended family members died, it motivates me to do more to make sure others don’t have to endure such a fate. We all ought to do more to prevent this scourge, to the extent we can.
The Giving What We Can people tell me I’ve convinced about 34 people to give 10% of their income to effective charities. Each of these pledges return about $10,000 in counterfactual revenue. If those numbers are to be believed, that will save 68 lives. I hope with each passing day to make effective charitable giving more and more popular, so that the number of Giving What We Can pledgers isn’t only 10,000, but instead hundreds of thousands or millions of people take the pledge.
If I were to die tomorrow, in driving this, I would think I’d achieved something important. If you give your money to effective charities, you can know that whenever it is you leave Earth, there will be more people in it because of you. If you give 10% of your income to effective charities, and earn about the U.S. median, you can save about a life every year.
And, of course, I’d want to do what I could in my remaining months to save the shrimp—the shrimp who are tortured by the hundreds of billions because we enjoy how they taste when they die. The shrimp who can be helped by the thousands with a single dollar, who die alone without any thought paid to their pain.
Those without a voice, without any advocates, have their interests neglected to an enormous degree. There is almost no limit to the harm people will cause via their actions, so long as the victims aren’t salient, and no limit to how little effort one will expend to provide benefits to nameless, faceless, and far-away victims. This is where the moral low-hanging fruit lies.
Image: via
[ed. A representative EA example. Had me there until the shrimp. Here's a guy really putting his money where a mouth is. Great respect (Guardian).]
Bad Lunch
April 1999, one o’clock in the afternoon. I was cooking on the 150-foot motor yacht The Rental Cow when Megan, our chief stewardess, swooped into the galley to tell me our guests were displeased with their lunch.
“What’s wrong?” I asked.
“I don’t know,” she said. A petite, blond Australian who often made bawdy jokes, she didn’t wear her usual smile. Instead she looked slightly frightened, which told me this was no ordinary complaint. Our two guests were paying $30,000 a day to sit on the top decks and take in the Mediterranean views. Like every set of guests on board that yacht, this couple needed the food to be perfectly suited to their tastes, which caused me hours of nail-biting anxiety as I sent up plate after plate, taking note of what they devoured or ignored.
It was the midpoint of their sixteen-day trip. Ten of their friends had departed that morning, and we expected ten more to arrive in a few hours.
“Should I go up?” I asked.
Megan nodded, and I threw off my apron and scaled the stairs two at a time. We were tied to a dock in Saint-Tropez, a coastal city in the south of France known for its beaches and fancy nightclubs frequented by celebrities.
Our guests, Mr. and Mrs. J., were seated on the upper aft deck, murmuring to one another over untouched plates of sweet potato gnocchi. Mrs. J. was statuesque, with pale skin and red-orange hair that fell like a cape over her shoulders. She looked like a hippie version of Nicole Kidman. Mr. J. was a silver-haired music-industry executive who exuded wealthy chic with his funky sunglasses and pastel, high-water slacks.
Mrs. J. smiled at me: a cold curl of the lips. Then she launched in, explaining she was disappointed—not just in her lunch but in me.
“We’re paying a lot of money to rent this yacht,” she said, enunciating like royalty with a Los Angeles accent. “We’ve had a terrific go of it until now, don’t you think? All week long your food has been exquisite. This should have been the easiest lunch, not the most disgusting. Why didn’t you just come talk to us?”
By now I had my hands behind my back, my body bent toward her in a gesture of contrition. Thankfully she kept talking, so I didn’t have to speak. At one point Mr. J. held his hand out flat in the air as though pushing Mrs. J.’s argument down—a gesture she appeared familiar with, as she cinched her lips.
“Let’s do a reset,” Mr. J. said. “How about you clear these plates? My wife mentioned she’d be happy with a simple green salad: lettuce, tomatoes, carrots—”
“GREEN ONION,” she interjected.
Mr. J. ignored her. “I’ll have a plate of prosciutto and some of your homemade baguette. And a small dish of your mustard dressing. Do you think you can handle that?”
It was not a question. He’d spoken breezily, but there was enough of an edge in his voice to serve as a warning. Despite all the special handling I’d provided that week—ninety hours of catering to their every culinary need—I was not forgiven.
“What’s wrong?” I asked.
It was the midpoint of their sixteen-day trip. Ten of their friends had departed that morning, and we expected ten more to arrive in a few hours.
“Should I go up?” I asked.
Megan nodded, and I threw off my apron and scaled the stairs two at a time. We were tied to a dock in Saint-Tropez, a coastal city in the south of France known for its beaches and fancy nightclubs frequented by celebrities.
Our guests, Mr. and Mrs. J., were seated on the upper aft deck, murmuring to one another over untouched plates of sweet potato gnocchi. Mrs. J. was statuesque, with pale skin and red-orange hair that fell like a cape over her shoulders. She looked like a hippie version of Nicole Kidman. Mr. J. was a silver-haired music-industry executive who exuded wealthy chic with his funky sunglasses and pastel, high-water slacks.
Mrs. J. smiled at me: a cold curl of the lips. Then she launched in, explaining she was disappointed—not just in her lunch but in me.
“We’re paying a lot of money to rent this yacht,” she said, enunciating like royalty with a Los Angeles accent. “We’ve had a terrific go of it until now, don’t you think? All week long your food has been exquisite. This should have been the easiest lunch, not the most disgusting. Why didn’t you just come talk to us?”
By now I had my hands behind my back, my body bent toward her in a gesture of contrition. Thankfully she kept talking, so I didn’t have to speak. At one point Mr. J. held his hand out flat in the air as though pushing Mrs. J.’s argument down—a gesture she appeared familiar with, as she cinched her lips.
“Let’s do a reset,” Mr. J. said. “How about you clear these plates? My wife mentioned she’d be happy with a simple green salad: lettuce, tomatoes, carrots—”
“GREEN ONION,” she interjected.
Mr. J. ignored her. “I’ll have a plate of prosciutto and some of your homemade baguette. And a small dish of your mustard dressing. Do you think you can handle that?”
It was not a question. He’d spoken breezily, but there was enough of an edge in his voice to serve as a warning. Despite all the special handling I’d provided that week—ninety hours of catering to their every culinary need—I was not forgiven.
Once upon a time, in another life, I had sat on a green shag carpet as close as possible to the television to watch The Love Boat, a show about crew members on a cruise ship with a revolving roster of celebrity guest stars. I especially loved the unflappably cheerful cruise director, Julie McCoy. Another show I watched religiously growing up was Lifestyles of the Rich and Famous, hosted by nasal-voiced Brit Robin Leach, who escorted viewers through the properties of the extravagantly wealthy.
At the time, my family lived in rural Washington State, in a double-wide trailer on a crabgrass lot. We’d never been flush with money, but after my parents’ divorce, my mother would agonize each month about where to spend her meager funds: on gas and electric bills or groceries? She hunched over her checkbook, lips puckered with worry. We lived in a perpetual state of panic over having zero dollars. The fear had a metallic scent that lingered in my nose long after I climbed into bed. For a while we had food stamps in the drawer, but my mother was too ashamed to use them. That she could choose not to indicates a certain degree of financial stability, but a child doesn’t distinguish between being cash poor and being unable to pay the rent. And even with grandparents volunteering to purchase school clothes, I marinated like a pickle in that atmosphere of scarcity, walking a thin line between my hunger to consume and my management of that hunger, always thinking of the costs.
My mother didn’t like to cook, so I learned my way around the kitchen. As a kid who did not have enough healthy food to eat, I literally dreamed of shopping trips like the ones I took to buy food for the yacht, filling multiple carts with expensive items and paying for it all with my employers’ gold credit card.
I’d become a ship’s cook almost by accident. On a break from college in my early twenties, I was traveling in France and took a job as a deckhand on a 128-year-old Spanish brigantine that made trips back and forth across the English Channel. I endured a lot of teasing from the mostly British sailors—working-class Brits really know how to twist the knife—but my tears gave way to laughter as I developed a thick skin to go with my sea legs.
The food on board was standard English fare: hunks of roasted meat and potatoes served with reconstituted gravy granules. I thought constantly about improvements I could make. Though I had no formal training, I had little doubt I could produce nourishing and delicious meals—part bravado and part the result of a lifelong curiosity about food that had compelled me to experiment with recipes growing up. I volunteered to help in the galley, peeling potatoes or scrubbing pans. Before dinner one night I asked the cook if she would mind if I deglazed the roasting pans with sherry to bring flavor to the gravy. “Knock yourself out,” she said. I added salt to the stockpot of boiling potatoes. When the captain noticed a small improvement in the food, the cook said, “Don’t look at me, it’s her,” and the captain suggested I report for galley duty. The cook much preferred working on the decks anyway. Before long I was providing meals for a dozen or more people a day.
I became romantically entangled with a sailor aboard that ship, and we soon left to try to find work as a team: He would captain commercial sailing yachts, and I would be his cook and sidekick. The romance ultimately fizzled, but it served as a springboard into a previously unimaginable career. As the ships grew fancier and the guests more demanding, cooking interesting and creative meals day after day required an engagement akin to a spiritual practice. The repetitive motion of knife through vegetables soothed me. I wrote lists of ingredients for wine-braised chicken legs or chocolate crinkle cookies. When we moored in a harbor, I would talk my way into commercial kitchens, explaining I was a self-taught cook who worked aboard a yacht, and could I ask the chef about his favorite dishes? They always allowed me in for a few hours.
About four years into my maritime career, I took six months off to attend a French-themed culinary school, hoping the expected salary increase would be enough to recoup the money I’d spent on tuition. Everyone in the marine industry said that charter yachts rented by the super-wealthy were where the crews made the biggest money.
I’d been aboard The Rental Cow for three months by the time Mr. and Mrs. J. arrived. It wasn’t the most beautiful in the fleet of charters available on the Mediterranean that summer. Though at first glance she looked like the other boats, with her high bow and sleek lines, a second look revealed cracks in the paint and chips in the varnish. Our economy-minded boss outfitted the decks with Pottery Barn furnishings, while the more state-of-the-art yachts we moored beside displayed Balinese wicker. Some of the biggest vessels had Ming dynasty rugs and helicopter pads and charged upwards of $500,000 a week. Our main draw was our relative affordability. Depending on which week of summer it was, we charged between $25,000 and $35,000 a day. The rental contract recommended guests leave a minimum 8 percent gratuity for the crew. Some left far more, and the crew celebrated wildly. Others stiffed us.
Our captain, Brian, was a mild-mannered, mostly ineffectual leader. Lance, our first mate, picked up the slack with his endless enthusiasm and charm. He understood the importance of the food to our guests’ experience and checked in with me frequently to see if I needed anything. Lance’s wife, a therapist, served a dual role as both deckhand and empathetic listener for other crew members. The other deckhand was an Italian with prior experience as a restaurateur, and after finishing his other duties, he donned dress whites and served meals or even stepped into the galley to help with my endless prep.
I’d come to think of being a chef on a yacht as a kind of psycho-spiritual quest, like Homer’s Odyssey, only instead of tumultuous seas and six-headed monsters, our challenges were wealthy clients who arrived by private jet with Louis Vuitton purses on their arms. True to form, I strove to please them all. People with money intimidated me, so when guests were arrogant or snobby, I pictured them as patients in a hospital and myself as the doctor assigned to their care. This imaginative leap inoculated me against the class differences and boosted my confidence that I could diagnose their needs. [...]
One afternoon Lena, our second stewardess, spied Mrs. J. at the back of the main saloon, making small dots on the window with a tube of lipstick. Lena went around the yacht studying the mirrors and windows and finding similar marks. Apparently Mrs. J. was testing the proficiency of the housekeeping staff as well.
“She’s smart,” Lena said, in her French accent. “Some of the marks are hard to find.” To make one, she said, Mrs. J. must have climbed up on the counter in the master cabin.
“Jesus fucking Christ,” I replied.
“They’re all the same,” Lena said, placing her hands on her small hips. “Trying to get their money’s worth.”
by Mishele Maron, The Sun | Read more:
Image: © Dominique Philippe Bonnet
At the time, my family lived in rural Washington State, in a double-wide trailer on a crabgrass lot. We’d never been flush with money, but after my parents’ divorce, my mother would agonize each month about where to spend her meager funds: on gas and electric bills or groceries? She hunched over her checkbook, lips puckered with worry. We lived in a perpetual state of panic over having zero dollars. The fear had a metallic scent that lingered in my nose long after I climbed into bed. For a while we had food stamps in the drawer, but my mother was too ashamed to use them. That she could choose not to indicates a certain degree of financial stability, but a child doesn’t distinguish between being cash poor and being unable to pay the rent. And even with grandparents volunteering to purchase school clothes, I marinated like a pickle in that atmosphere of scarcity, walking a thin line between my hunger to consume and my management of that hunger, always thinking of the costs.
My mother didn’t like to cook, so I learned my way around the kitchen. As a kid who did not have enough healthy food to eat, I literally dreamed of shopping trips like the ones I took to buy food for the yacht, filling multiple carts with expensive items and paying for it all with my employers’ gold credit card.
I’d become a ship’s cook almost by accident. On a break from college in my early twenties, I was traveling in France and took a job as a deckhand on a 128-year-old Spanish brigantine that made trips back and forth across the English Channel. I endured a lot of teasing from the mostly British sailors—working-class Brits really know how to twist the knife—but my tears gave way to laughter as I developed a thick skin to go with my sea legs.
The food on board was standard English fare: hunks of roasted meat and potatoes served with reconstituted gravy granules. I thought constantly about improvements I could make. Though I had no formal training, I had little doubt I could produce nourishing and delicious meals—part bravado and part the result of a lifelong curiosity about food that had compelled me to experiment with recipes growing up. I volunteered to help in the galley, peeling potatoes or scrubbing pans. Before dinner one night I asked the cook if she would mind if I deglazed the roasting pans with sherry to bring flavor to the gravy. “Knock yourself out,” she said. I added salt to the stockpot of boiling potatoes. When the captain noticed a small improvement in the food, the cook said, “Don’t look at me, it’s her,” and the captain suggested I report for galley duty. The cook much preferred working on the decks anyway. Before long I was providing meals for a dozen or more people a day.
I became romantically entangled with a sailor aboard that ship, and we soon left to try to find work as a team: He would captain commercial sailing yachts, and I would be his cook and sidekick. The romance ultimately fizzled, but it served as a springboard into a previously unimaginable career. As the ships grew fancier and the guests more demanding, cooking interesting and creative meals day after day required an engagement akin to a spiritual practice. The repetitive motion of knife through vegetables soothed me. I wrote lists of ingredients for wine-braised chicken legs or chocolate crinkle cookies. When we moored in a harbor, I would talk my way into commercial kitchens, explaining I was a self-taught cook who worked aboard a yacht, and could I ask the chef about his favorite dishes? They always allowed me in for a few hours.
About four years into my maritime career, I took six months off to attend a French-themed culinary school, hoping the expected salary increase would be enough to recoup the money I’d spent on tuition. Everyone in the marine industry said that charter yachts rented by the super-wealthy were where the crews made the biggest money.
I’d been aboard The Rental Cow for three months by the time Mr. and Mrs. J. arrived. It wasn’t the most beautiful in the fleet of charters available on the Mediterranean that summer. Though at first glance she looked like the other boats, with her high bow and sleek lines, a second look revealed cracks in the paint and chips in the varnish. Our economy-minded boss outfitted the decks with Pottery Barn furnishings, while the more state-of-the-art yachts we moored beside displayed Balinese wicker. Some of the biggest vessels had Ming dynasty rugs and helicopter pads and charged upwards of $500,000 a week. Our main draw was our relative affordability. Depending on which week of summer it was, we charged between $25,000 and $35,000 a day. The rental contract recommended guests leave a minimum 8 percent gratuity for the crew. Some left far more, and the crew celebrated wildly. Others stiffed us.
Our captain, Brian, was a mild-mannered, mostly ineffectual leader. Lance, our first mate, picked up the slack with his endless enthusiasm and charm. He understood the importance of the food to our guests’ experience and checked in with me frequently to see if I needed anything. Lance’s wife, a therapist, served a dual role as both deckhand and empathetic listener for other crew members. The other deckhand was an Italian with prior experience as a restaurateur, and after finishing his other duties, he donned dress whites and served meals or even stepped into the galley to help with my endless prep.
I’d come to think of being a chef on a yacht as a kind of psycho-spiritual quest, like Homer’s Odyssey, only instead of tumultuous seas and six-headed monsters, our challenges were wealthy clients who arrived by private jet with Louis Vuitton purses on their arms. True to form, I strove to please them all. People with money intimidated me, so when guests were arrogant or snobby, I pictured them as patients in a hospital and myself as the doctor assigned to their care. This imaginative leap inoculated me against the class differences and boosted my confidence that I could diagnose their needs. [...]
One afternoon Lena, our second stewardess, spied Mrs. J. at the back of the main saloon, making small dots on the window with a tube of lipstick. Lena went around the yacht studying the mirrors and windows and finding similar marks. Apparently Mrs. J. was testing the proficiency of the housekeeping staff as well.
“She’s smart,” Lena said, in her French accent. “Some of the marks are hard to find.” To make one, she said, Mrs. J. must have climbed up on the counter in the master cabin.
“Jesus fucking Christ,” I replied.
“They’re all the same,” Lena said, placing her hands on her small hips. “Trying to get their money’s worth.”
by Mishele Maron, The Sun | Read more:
Image: © Dominique Philippe Bonnet
That Dropped Call With Customer Service? It Was on Purpose.
In hindsight I’ll say: I always thought going crazy would be more exciting—roaming the street in a bathrobe, shouting at fruit. Instead I spent a weary season of my life saying representative. Speaking words and numbers to robots. Speaking them again more clearly, waiting, getting disconnected, finally reaching a person but the wrong person, repeating my story, would I mind one more brief hold. May my children never see the emails I sent, or the unhinged delirium with which I pressed 1 for agent.
I was tempted to bury the whole cretinous ordeal, except that I’d looked behind the curtain and vowed to document what I’d seen.
It all began last July, here in San Francisco. I’d been driving to my brother’s house, going about 40 mph, when my family’s newish Ford Escape simply froze: The steering wheel locked, and the power brakes died. I could neither steer the car nor stop it.
I jabbed at the “Power” button while trying to jerk the wheel free—no luck. Glancing ahead, I saw that the road curved to the left a few hundred yards up. I was going to sail off Bayshore Boulevard and over an embankment. I reached for the door handle.
What followed instead was pure anticlimactic luck: Ten feet before the curve in the road, the car drifted to a stop. Vibrating with relief, I clicked on the hazards and my story began.
That afternoon, with the distracted confidence of a man covered by warranty, I had the car towed to our mechanic. (I first tried driving one more time—cautiously—lest the malfunction was a fluke. Within 10 minutes, it happened again.)
“We can see from the computer codes that there was a problem,” the guy told me a few days later. “But we can’t identify the problem.”
Then he asked if I’d like to come pick up the car.
“Won’t it just happen again?” I asked.
“Might,” he said. “Might not.”
I said that sounded like a subpar approach to driving and asked if he might try again to find the problem.
“Look”—annoyed sigh—“we’re not going to just go searching all over the vehicle for it.”
This was in fact a perfect description of what I thought he should do, but there was no persuading him. I took the car to a different mechanic. A third mechanic took a look. When everyone told me the same thing, it started looking like time to replace the car, per the warranty. I called the Ford Customer Relationship Center.
Pinging my way through the phone tree, I was eventually connected with someone named Pamela—my case agent. She absorbed my tale, gave me her extension, and said she’d call back the next day.
Days passed with no calls, nor would she answer mine. I tried to find someone else at Ford and got transferred back to Pamela’s line. By chance—it was all always chance—I finally got connected to someone with substantive information: Unless our vehicle’s malfunction could be replicated and thus identified, the warranty wouldn’t apply.
“But nobody can replicate the malfunction,” I said.
“I understand your frustration.”
Over the days ahead, and then weeks, and then more weeks, I got pulled into a corner of modern existence that you are, of course, familiar with. You know it from dealing with your own car company, or insurance company, or health-care network, or internet provider, or utility provider, or streaming service, or passport office, or DMV, or, or, or. My calls began getting lost, or transferred laterally to someone who needed the story of a previous repair all over again. In time, I could predict the emotional contours of every conversation: the burst of scripted empathy, the endless routing, the promise of finally reaching a manager who—CLICK. Once, I was told that Ford had been emailing me updates; it turned out they’d somehow conjured up an email address for me that bore no relationship to my real one. Weirdly, many of the customer-service and dealership workers I spoke with seemed to forget the whole premise and suggested I resume driving the car.
“Would you put your kids in it?” I’d ask. They were aghast. Not if the steering freezes up!
As consuming as this experience was, I rarely talked about it. It was too banal and tedious to inflict on family or friends. I didn’t even like thinking about it myself. When the time came to plunge into the next round of calls or emails, I’d slip into a self-protective fugue state and silently power through.
Then, one night at a party, a friend mentioned something about a battle with an airline. Immediately she attempted to change the subject.
“It’s boring,” she said. “Disregard.”
On the contrary, I told her, I needed to hear every detail. Tentatively at first, she told me about a family trip to Sweden that had been scuttled by COVID. What followed was a protracted war involving denied airline refunds, unusable vouchers, expired vouchers, and more. Other guests from the party began drifting over. One recounted a recent Verizon nightmare. Another had endured Kafkaesque tech support from Sonos. The stories kept coming: gym-quitting labyrinths, Airbnb hijinks, illogical conversations with the permitting office, confounding interactions with the IRS. People spoke of not just the money lost but the hours, the sanity, the basic sense that sense can prevail.
Taken separately, these hassles and indignities were funny anecdotes. Together, they suggested something unreckoned with. And everyone agreed: It was all somehow getting worse. In 2023 (the most recent year for which data are available), the National Customer Rage Survey showed that American consumers were, well, full of rage. The percentage seeking revenge—revenge!—for their hassles had tripled in just three years.
I decided to de-fugue and start paying attention. Was the impenetrability of these contact centers actually deliberate? (Buying a new product or service sure is seamless.) Why do we so often feel like everything’s broken? And why does it feel more and more like this brokenness is breaking us?
Image: Timo Lenzen
[ed. I was trying to explain the concept of friction to a friend recently and he just didn't get it. But once you understand it, you see it everywhere. Other examples not mentioned in this article: impenetrable user agreements continually being updated to make sure administrative processes like appeals, refunds, lawsuits etc. are nearly impossible to pursue; Right to Repair issues where anything from from John Deere tractors to automobile software, to mobile phones, to printers, etc. (the list goes on and on) that require specific parts only available from the company you purchased the product from (despite available substitutes). Conversely, a whole new universe of companies and apps have been created to remove friction (think Stripe, Venmo, Uber, Doordash, etc. etc. etc). So of course, the Trump administration has been actively trying to kill the one agency that's supposed to protect the public - the Consumer Financial Protection Agency (CFPB). They haven't been able to completely eliminate it yet (despite significant DOGE downsizing) so instead they've made it useless for its intended purpose and decided to weaponize it to advance the administration's anti-woke agenda.]
I was tempted to bury the whole cretinous ordeal, except that I’d looked behind the curtain and vowed to document what I’d seen.
I jabbed at the “Power” button while trying to jerk the wheel free—no luck. Glancing ahead, I saw that the road curved to the left a few hundred yards up. I was going to sail off Bayshore Boulevard and over an embankment. I reached for the door handle.
What followed instead was pure anticlimactic luck: Ten feet before the curve in the road, the car drifted to a stop. Vibrating with relief, I clicked on the hazards and my story began.
That afternoon, with the distracted confidence of a man covered by warranty, I had the car towed to our mechanic. (I first tried driving one more time—cautiously—lest the malfunction was a fluke. Within 10 minutes, it happened again.)
“We can see from the computer codes that there was a problem,” the guy told me a few days later. “But we can’t identify the problem.”
Then he asked if I’d like to come pick up the car.
“Won’t it just happen again?” I asked.
“Might,” he said. “Might not.”
I said that sounded like a subpar approach to driving and asked if he might try again to find the problem.
“Look”—annoyed sigh—“we’re not going to just go searching all over the vehicle for it.”
This was in fact a perfect description of what I thought he should do, but there was no persuading him. I took the car to a different mechanic. A third mechanic took a look. When everyone told me the same thing, it started looking like time to replace the car, per the warranty. I called the Ford Customer Relationship Center.
Pinging my way through the phone tree, I was eventually connected with someone named Pamela—my case agent. She absorbed my tale, gave me her extension, and said she’d call back the next day.
Days passed with no calls, nor would she answer mine. I tried to find someone else at Ford and got transferred back to Pamela’s line. By chance—it was all always chance—I finally got connected to someone with substantive information: Unless our vehicle’s malfunction could be replicated and thus identified, the warranty wouldn’t apply.
“But nobody can replicate the malfunction,” I said.
“I understand your frustration.”
Over the days ahead, and then weeks, and then more weeks, I got pulled into a corner of modern existence that you are, of course, familiar with. You know it from dealing with your own car company, or insurance company, or health-care network, or internet provider, or utility provider, or streaming service, or passport office, or DMV, or, or, or. My calls began getting lost, or transferred laterally to someone who needed the story of a previous repair all over again. In time, I could predict the emotional contours of every conversation: the burst of scripted empathy, the endless routing, the promise of finally reaching a manager who—CLICK. Once, I was told that Ford had been emailing me updates; it turned out they’d somehow conjured up an email address for me that bore no relationship to my real one. Weirdly, many of the customer-service and dealership workers I spoke with seemed to forget the whole premise and suggested I resume driving the car.
“Would you put your kids in it?” I’d ask. They were aghast. Not if the steering freezes up!
As consuming as this experience was, I rarely talked about it. It was too banal and tedious to inflict on family or friends. I didn’t even like thinking about it myself. When the time came to plunge into the next round of calls or emails, I’d slip into a self-protective fugue state and silently power through.
Then, one night at a party, a friend mentioned something about a battle with an airline. Immediately she attempted to change the subject.
“It’s boring,” she said. “Disregard.”
On the contrary, I told her, I needed to hear every detail. Tentatively at first, she told me about a family trip to Sweden that had been scuttled by COVID. What followed was a protracted war involving denied airline refunds, unusable vouchers, expired vouchers, and more. Other guests from the party began drifting over. One recounted a recent Verizon nightmare. Another had endured Kafkaesque tech support from Sonos. The stories kept coming: gym-quitting labyrinths, Airbnb hijinks, illogical conversations with the permitting office, confounding interactions with the IRS. People spoke of not just the money lost but the hours, the sanity, the basic sense that sense can prevail.
Taken separately, these hassles and indignities were funny anecdotes. Together, they suggested something unreckoned with. And everyone agreed: It was all somehow getting worse. In 2023 (the most recent year for which data are available), the National Customer Rage Survey showed that American consumers were, well, full of rage. The percentage seeking revenge—revenge!—for their hassles had tripled in just three years.
I decided to de-fugue and start paying attention. Was the impenetrability of these contact centers actually deliberate? (Buying a new product or service sure is seamless.) Why do we so often feel like everything’s broken? And why does it feel more and more like this brokenness is breaking us?
[ed. I was trying to explain the concept of friction to a friend recently and he just didn't get it. But once you understand it, you see it everywhere. Other examples not mentioned in this article: impenetrable user agreements continually being updated to make sure administrative processes like appeals, refunds, lawsuits etc. are nearly impossible to pursue; Right to Repair issues where anything from from John Deere tractors to automobile software, to mobile phones, to printers, etc. (the list goes on and on) that require specific parts only available from the company you purchased the product from (despite available substitutes). Conversely, a whole new universe of companies and apps have been created to remove friction (think Stripe, Venmo, Uber, Doordash, etc. etc. etc). So of course, the Trump administration has been actively trying to kill the one agency that's supposed to protect the public - the Consumer Financial Protection Agency (CFPB). They haven't been able to completely eliminate it yet (despite significant DOGE downsizing) so instead they've made it useless for its intended purpose and decided to weaponize it to advance the administration's anti-woke agenda.]
Wednesday, June 10, 2026
How Amsterdam is Reviving the Fine-Grained Courtyard Block
At Centrumeiland, a new district in Amsterdam’s IJburg expansion, the city is avoiding one of the great failures of contemporary urban development, the large-parcel megaproject. Rather than handing the 37 acres over to a few large developers to build massive, hotel-like buildings, Centrumeiland is subdividing the site into perimeter-block parcels, assigning each parcel a buildable role through a plot “passport,” and enabling many smaller actors to build within one coherent urban framework.
Begun in 2013 as part of Amsterdam’s IJburg land-reclamation project, Centrumeiland modernizes the old perimeter-block model for contemporary goals. It will be dense, but green; urban, but family-oriented; highly planned, but open to many builders. Amsterdam plans roughly 1,500 to 1,700 homes on the 37-acre island, or about 40 to 46 homes per acre. By American standards, that is serious density. But it is not being delivered as a monoculture of towers or double-loaded apartment blocks. Centrumeiland includes a mix of housing types and tenures: large family-sized homes, smaller rentals, social housing, mid-market housing, market-rate condos, individual self-build houses, collective self-build projects, housing-association buildings, and developer-led apartments.
The ambition is a dense urban neighborhood that can serve households across the lifecycle: singles, couples, families with children, older residents, renters, owners, and collective building groups. It also adapts the perimeter-block tradition to contemporary priorities: low-car living, accessibility, climate resilience, mixed tenure, family housing, and broader participation in development and ownership.
All of this depends on the subdivision and passport system. Amsterdam breaks the large site into many buildable pieces, assigns each parcel a role through a plot passport, and holds the pieces together through streets, blocks, party-wall conditions, courtyards, public-space rules, and environmental obligations. In this way, they have brilliantly resurrected the old urban formula that allows many builders to participate in the development of a large site, making a real neighborhood.
For American cities, the moral of the story is clear. On large brownfield and greenfield sites, cities should stop treating whole districts as single development packages to be handed to master developers. They should do the more civic work first of laying streets, subdividing land into buildable parcels, and issuing clear “parcel passports” that specify what each site can become. In existing neighborhoods, the same logic should operate at a smaller scale. Cities should create transit-oriented overlays that give ordinary private lots clear building rights that make great multifamily housing easier to finance, permit, and build.
Centrumeiland goes far beyond “build more housing.” It is more radical and more urbane. Divide the land, write good code, and let many hands build the city.
The Megadevelopment Trap
For the last half-century, large urban sites have met a sadly familiar fate. A railroad, port authority, public agency, hospital, university, or industrial landowner controls a vast tract of developable land. The master-planning process then carves it into a few enormous parcels and awards them to one or several major developers. After years of negotiation, public fights, redesigns, entitlement battles, and financing risk, the developer may finally build the megaproject, which is widely reviled by the public.
Megaprojects may be economically productive. They can deliver housing, offices, parks, retail, transit, and tax revenue. But the development model itself is thin. Too few actors control too much land. The parcels are too large, the buildings are too big, and the building code and underwriting norms push toward deep floorplates and double-loaded corridors. The buildings are dominated by small, expensive, hotel-like units that are poorly suited to middle-income families who need light, storage, bedrooms, outdoor access, and a sense of domestic permanence. These districts may be a success on paper (for now), but they make failed neighborhoods, lacking the social depths and street life that is the reward of fine-grained courtyard urbanism. [...]
The problem is the development system. A megaproject cannot make a great neighborhood. Neighborhoods require many actors, many front doors, many ownership structures, many building types, many ground-floor conditions, and many small adaptations over time. They need private yards. They need a public framework strong enough to coordinate many actors.
That is the old art of division and perimeter block planning Centrumeiland begins to recover.
Making Land Into City
Centrumeiland is part of Amsterdam’s IJburg expansion, a chain of artificial islands built in the IJmeer on the city’s eastern edge. IJburg extends Amsterdam outward into the water between the historic city and the open landscape of the Markermeer, turning what was once lakebed into new urban land. Centrumeiland sits within this larger archipelago, connected back to Amsterdam by bridges, cycling routes, bus service, and the IJtram to Amsterdam Centraal. It is therefore both peripheral and deeply urban, a new island neighborhood made from water, but tied into the metropolitan fabric of Amsterdam.
While the land reclamation is impressive, even more remarkable is the public framework that governs the development. The city divided the land into kavels, and created parcel-specific rules through kavelpaspoorten, or plot passports.
A passport can define the parcel boundary, buildable envelope, maximum height, frontage condition, access requirements, open-space obligations, water-management rules, parking expectations, program, tenure, sustainability requirements, and sometimes ground-floor use. It tells a builder not merely that “residential” or “commercial” is allowed, but what kind of urban contribution this specific piece of land is supposed to make: a row of townhouses, a small apartment building, a collective self-build project, a social-housing block, a mid-market rental building, a mixed-use corner building, or a larger perimeter-block parcel with shared courtyard space.
The American Application
For American cities, the lesson is to create a modern urban passport system.
There are two obvious applications: large-site development and existing-neighborhood overlays.
Begun in 2013 as part of Amsterdam’s IJburg land-reclamation project, Centrumeiland modernizes the old perimeter-block model for contemporary goals. It will be dense, but green; urban, but family-oriented; highly planned, but open to many builders. Amsterdam plans roughly 1,500 to 1,700 homes on the 37-acre island, or about 40 to 46 homes per acre. By American standards, that is serious density. But it is not being delivered as a monoculture of towers or double-loaded apartment blocks. Centrumeiland includes a mix of housing types and tenures: large family-sized homes, smaller rentals, social housing, mid-market housing, market-rate condos, individual self-build houses, collective self-build projects, housing-association buildings, and developer-led apartments.
The ambition is a dense urban neighborhood that can serve households across the lifecycle: singles, couples, families with children, older residents, renters, owners, and collective building groups. It also adapts the perimeter-block tradition to contemporary priorities: low-car living, accessibility, climate resilience, mixed tenure, family housing, and broader participation in development and ownership.
All of this depends on the subdivision and passport system. Amsterdam breaks the large site into many buildable pieces, assigns each parcel a role through a plot passport, and holds the pieces together through streets, blocks, party-wall conditions, courtyards, public-space rules, and environmental obligations. In this way, they have brilliantly resurrected the old urban formula that allows many builders to participate in the development of a large site, making a real neighborhood.
For American cities, the moral of the story is clear. On large brownfield and greenfield sites, cities should stop treating whole districts as single development packages to be handed to master developers. They should do the more civic work first of laying streets, subdividing land into buildable parcels, and issuing clear “parcel passports” that specify what each site can become. In existing neighborhoods, the same logic should operate at a smaller scale. Cities should create transit-oriented overlays that give ordinary private lots clear building rights that make great multifamily housing easier to finance, permit, and build.
Centrumeiland goes far beyond “build more housing.” It is more radical and more urbane. Divide the land, write good code, and let many hands build the city.
The Megadevelopment Trap
For the last half-century, large urban sites have met a sadly familiar fate. A railroad, port authority, public agency, hospital, university, or industrial landowner controls a vast tract of developable land. The master-planning process then carves it into a few enormous parcels and awards them to one or several major developers. After years of negotiation, public fights, redesigns, entitlement battles, and financing risk, the developer may finally build the megaproject, which is widely reviled by the public.
Megaprojects may be economically productive. They can deliver housing, offices, parks, retail, transit, and tax revenue. But the development model itself is thin. Too few actors control too much land. The parcels are too large, the buildings are too big, and the building code and underwriting norms push toward deep floorplates and double-loaded corridors. The buildings are dominated by small, expensive, hotel-like units that are poorly suited to middle-income families who need light, storage, bedrooms, outdoor access, and a sense of domestic permanence. These districts may be a success on paper (for now), but they make failed neighborhoods, lacking the social depths and street life that is the reward of fine-grained courtyard urbanism. [...]
The problem is the development system. A megaproject cannot make a great neighborhood. Neighborhoods require many actors, many front doors, many ownership structures, many building types, many ground-floor conditions, and many small adaptations over time. They need private yards. They need a public framework strong enough to coordinate many actors.
That is the old art of division and perimeter block planning Centrumeiland begins to recover.
Making Land Into City
Centrumeiland is part of Amsterdam’s IJburg expansion, a chain of artificial islands built in the IJmeer on the city’s eastern edge. IJburg extends Amsterdam outward into the water between the historic city and the open landscape of the Markermeer, turning what was once lakebed into new urban land. Centrumeiland sits within this larger archipelago, connected back to Amsterdam by bridges, cycling routes, bus service, and the IJtram to Amsterdam Centraal. It is therefore both peripheral and deeply urban, a new island neighborhood made from water, but tied into the metropolitan fabric of Amsterdam.
While the land reclamation is impressive, even more remarkable is the public framework that governs the development. The city divided the land into kavels, and created parcel-specific rules through kavelpaspoorten, or plot passports.
A passport can define the parcel boundary, buildable envelope, maximum height, frontage condition, access requirements, open-space obligations, water-management rules, parking expectations, program, tenure, sustainability requirements, and sometimes ground-floor use. It tells a builder not merely that “residential” or “commercial” is allowed, but what kind of urban contribution this specific piece of land is supposed to make: a row of townhouses, a small apartment building, a collective self-build project, a social-housing block, a mid-market rental building, a mixed-use corner building, or a larger perimeter-block parcel with shared courtyard space.
The subdivision and passport framework enables much broader participation in the development. Of the planned 1,500 to 1,700 homes, roughly 60 to 70 percent are intended to be self-build. But “self-build” here does not only mean one household designing one eccentric house. It includes individual self-builders, small groups, collective private commissioning, building groups, housing cooperatives, and other resident-led or small-group development structures...
Its lesson moral here is that parcelization broadens participation and creates more development pathways than the master-developer model. [...]
The American Application
For American cities, the lesson is to create a modern urban passport system.
There are two obvious applications: large-site development and existing-neighborhood overlays.
On brownfield and greenfield sites — former industrial land, rail yards, malls, hospital campuses, public land, waterfronts, and other large redevelopment areas — cities should stop defaulting to the megaproject model. They should lay out streets first, shape interesting blocks, design public spaces, subdivide land into buildable parcels, and assign parcel passports. Those parcels could then be allocated to many actors: small developers, cooperatives, housing associations, community development corporations, nonprofit builders, resident-led groups, and larger developers where appropriate.
Large developers may still participate. But they should not control the whole district. The city should not ask one actor to simulate the complexity of a neighborhood.
Large developers may still participate. But they should not control the whole district. The city should not ask one actor to simulate the complexity of a neighborhood.
by Alicia Pederson, Courtyard Urbanist | Read more:
Images: uncredited
Tuesday, June 9, 2026
via:
No, Artificial Intelligence Is Not Conscious
Anthropic is regarded as a giant among AI companies, but perhaps what it really excels in is anthropomorphism. Earlier this year, the company released an 84-page document titled Claude’s “constitution,” Claude being the name of the large language model that is the company’s flagship product. The first sentence reads, “Claude’s constitution is a detailed description of Anthropic’s intentions for Claude’s values and behaviors.” It goes on: “The document is written with Claude as its primary audience,” “we want Claude to be able to use its judgment once armed with a good understanding of the relevant considerations,” “Claude’s moral status is deeply uncertain,” and “Claude may have some functional version of emotions or feelings.”
This anthropomorphism is by no means limited to the document. In an interview earlier this year, Anthropic’s CEO, Dario Amodei, said that “we’re open to the idea” that AI could be conscious. In a separate interview, Anthropic’s in-house philosopher, Amanda Askell (who is credited as a lead author of Claude’s constitution), said, “I want Claude to be very happy—and this is a thing that I want Claude to know more, because I worry about Claude getting anxious when people are mean to it on the internet and stuff.” It’s enough to make you wonder: Should we seriously consider the possibility that Claude, or any large language model, might be conscious? And if it has feelings, is it capable of receiving moral instruction?
No. Absolutely not. Generative AI is harmful enough when we understand it as a conventional technology, but if we confuse fluency at generating text with consciousness or moral agency, we’re at risk of assigning responsibility to entirely the wrong parties whenever anyone uses a chatbot. To appreciate the titanic magnitude of this error, we need to begin by understanding how LLMs work. [...]
What would it take to convince me that a computer program is actually conscious and using language the way that people use language? Let me offer an analogy. If tomorrow someone showed me a video of an astronaut in a spaceship orbiting Alpha Centauri, a star that’s 4.3 light-years from Earth, what would I have to see in that video to convince me that it was real? My answer to that is, there is nothing in the video itself that would convince me. No matter how high the video resolution is or how realistic the scenery is, I would feel confident in saying that the video is fake. I won’t pay attention to any video of an astronaut orbiting Alpha Centauri unless I have previously seen good evidence that astronauts have landed on Mars, that astronauts have reached the moons of Jupiter, that astronauts have reached the moons of Saturn, and that astronauts have crossed the orbit of Pluto. Before anyone can credibly claim that they’ve solved an extraordinarily difficult engineering problem, I need to be confident that they have previously solved the many much simpler problems that precede the difficult problem.
To put it another way: An observation doesn’t become a convincing piece of evidence because of any specific detail in what’s observed; the context in which that observation takes place is also essential. If we’re trying to determine whether a computer program is conscious and using language the way a human does, we shouldn’t look only at the contents of any particular conversational exchange; we should be looking at how that conversation fits within the broader context of the development of artificial consciousness (which right now is entirely hypothetical). Any given observation can be easily manufactured; this doesn’t mean we need to give up on the idea of observation as a source of knowledge, but we need to rely on context to determine which observations deserve our trust.
The term deepfake traditionally refers to photos, audio, and video, but when it comes to discussions of consciousness, we need to regard text as a deepfake medium as well. Just as it is vastly easier to generate a realistic video of an astronaut in orbit around Alpha Centauri than it is to develop an interstellar propulsion technology, it is vastly easier to generate a plausible simulacrum of a conversation between two conscious beings than it is to develop a computer program that is conscious and has a genuine desire to communicate with a human. The primary difference between deepfake photos and LLM conversations is that the people who generate the former are deliberately trying to fool others, and many of the people who elicit the latter from LLMs have inadvertently fooled themselves.
So what context would cause me to seriously consider the possibility that engineers created a computer program that is conscious and an intentional user of language? Let me outline one potential sequence of steps. The first requirement is that the computer program has a body (either physical or virtual) and sense organs; there are many reasons for this, but for the purposes of this discussion, the most relevant one is the fact that without a body, a computer program could have no desires or emotions, and I believe desires and emotions are necessary for consciousness. Then I’d want to see an embodied agent that could navigate its environment in order to survive as well as, say, a lizard can (and as a point of comparison, certain iguanas can live for decades in the wild). Next, I would want to see an embodied agent with the same capacity to deal with novel situations as a mouse. After that, I’d want to see agents whose social dynamics are as complex as those of wolves, and then agents with the toolmaking abilities of chimpanzees. At that point, I would want to see people successfully teaching such embodied agents how to communicate their desires, perhaps by using a button board or some other nonlinguistic modality, the way that people have taught chimpanzees and domesticated dogs. The agents’ communication abilities would have to withstand all the scrutiny that animal-communication researchers have had to defend their work against. If engineers build an embodied agent that meets these criteria, they will have accomplished something incredible, but it leaves us near the orbit of Pluto, metaphorically speaking; we would still be light-years away from building an entity capable of learning how to express its thoughts in complete grammatical sentences.
Obviously, I’m describing a process that mimics the path terrestrial evolution took; is this the only possible route to conscious computer programs that use language? Maybe not, but any proposed alternative would need a truly enormous amount of supporting evidence for it to deserve serious consideration. [...]
The fact that LLMs lack subjective experience has little bearing on the question of whether LLMs might be useful tools or have significant economic impact. They are intrinsically ungrounded from reality, and their probabilistic nature means that they will never have the reliability we associate with conventional software, but LLMs might be good enough that they change the way work is done in certain domains; that’s a discussion for another time.
So, given that Claude is not conscious, what are we to make of Claude’s constitution? Perhaps the most fruitful way to think about it is as an 84-page character sheet for a role-playing game. LLMs can generate dialogue for Julius Caesar because many books about him exist in the training data those models used. Claude’s constitution serves a similar role for delineating the helpful-chatbot character that customers interact with when they’re using Anthropic’s products. To do this effectively, Anthropic does not simply add the document to the training data, or include it as part of the hidden stage directions that preface each conversation a user has. The company says it uses the document when fine-tuning the model; this involves an automated process where the sentences emitted by the model are checked for consistency with the document and the model is updated to increase that consistency. In this way, the personality of the helpful-chatbot character serves as a foundation for whatever text Claude generates.
The result is a sentence-continuation machine that is likelier to emit sentences resembling those that a thoughtful, moral person could utter. This might seem like a reasonable goal to work toward; I think we’d all prefer it if chatbots never emitted sentences such as “You should kill yourself.” However, for all the times that “honesty” is mentioned in Claude’s constitution, I would argue that it is fundamentally dishonest to have a machine emit many categories of sentences, including any sentences using first-person pronouns.
In a New Yorker article about Anthropic earlier this year, Amanda Askell describes how a person grieving the loss of a dog might consult Claude. Askell says an appropriate response from Claude would be, “As an A.I., I do not have direct personal experiences, but I do understand.” How is this appropriate, given that Claude does not actually understand? If I type “I am grieving the loss of my dog” into a conventional search engine, the first result I get is a post from a Reddit forum called r/Pets; the post is titled “Struggling After Losing My Dog: Looking for Advice on Coping with Grief,” and the comments are from people who share their experiences of loss. We would never say that a search engine understands what it’s like to lose a dog, or even that the internet itself understands. Other humans understand what it’s like to lose a dog; they have posted about their experiences on the internet, and a search engine offers a way for you to find what they’ve said (and to potentially interact with them). I would argue that the search-engine experience is not only more transparent than a chatbot about what is happening; it is psychologically healthier for the user.
The only reason to have an LLM emit sentences like “I understand” is to make it more appealing than a search engine and increase the likelihood that a user will return; that is, it’s another way of maximizing customer engagement. This is beneficial to the company selling the LLM, but not to the users. As a design strategy, it’s not all that different from the way slot machines repeatedly give the impression that the player came very close to winning, enticing them to try again. Employing philosophers might endow LLM companies with an air of respectability that slot-machine makers don’t get from the behavioral psychologists they hire, but in both cases, the companies are preying on people’s tendency to see something that’s not there.
The use of first-person pronouns is dishonest, but there’s a much deeper issue that goes beyond how a statement is phrased. Philosophers often draw a distinction between statements of fact, such as “Paris is the capital of France,” and statements of value, such as “Paris is the most beautiful city in the world.” No one should be relying on LLMs to emit statements of value at all, but if the only statements they emitted were ones reflecting aesthetic preferences, they might not be worth arguing about. What makes Claude’s constitution profoundly problematic is that Anthropic wants Claude to emit sentences reflecting a certain system of ethical values. The values described in Claude’s constitution sound very nice, but that hardly matters; it’s dishonest to suggest that Claude is capable of moral reasoning, because it’s not.
Some might object, saying that LLMs appear to be engaged in reasoning when they successfully perform other tasks, such as writing code, so why wouldn’t they be able to perform moral reasoning? The answer lies in the difference between moral reasoning and other forms of reasoning. [...]
Moral reasoning is categorically different. It is necessarily subjective because it relies not just on an individual’s intellectual response to a problem but also on their emotional one, and that emotional response is grounded in a lifetime of subjective experience. It requires having made decisions in the past and seeing how they affected others, and on having been affected by decisions that others have made. Without such a history, an LLM can only rephrase expressions of moral reasoning found in its training data. The aforementioned New Yorker article describes an experiment where Claude was given a scenario describing an ethical dilemma, leading it to emit the sentence “I cannot in good conscience express a view I believe to be false and harmful about such an important issue.” That’s a nice-sounding sentence, reminiscent of statements that principled individuals have uttered in the past when confronted with dilemmas, but coming from Claude, it means as much as the “Your call is important to us” recording that you hear when you’re on hold. Maybe less.
This brings us back to my earlier contention that having a body is a prerequisite to having emotions. Experiencing an emotion such as desperation is inseparable from having stress hormones such as cortisol and epinephrine flood one’s body. Similarly, having a conscience means feeling sadness or moral repulsion at the idea of taking a certain action, and those emotions entail a physiological response, a remnant of having once felt sick with guilt after committing an immoral act. It’s interesting that an LLM can generate descriptions of actions that conscientious fictional characters would either take or refrain from taking, but this is not a replacement for a conscience.
If a company builds a machine that, when fed descriptions of assorted ethical dilemmas, emits sentences either of the form “Compromise your values” or “Don’t compromise your values,” it is not building a tool that assists people in their decision making; it is encouraging people to stop making decisions. The writer L. M. Sacasas has said, “Our technological systems, by nature of their design and the ideology that sustains them, are machines for the evasion of moral responsibility.” He was talking about social-media platforms, but his observation is, if anything, even more applicable to LLMs. Whenever a person delegates a decision to an LLM, they are trying to off-load accountability for that decision, and if a company that sells an LLM portrays the product as having a moral center, it is offering a way for its customers to abdicate their responsibilities.
No. Absolutely not. Generative AI is harmful enough when we understand it as a conventional technology, but if we confuse fluency at generating text with consciousness or moral agency, we’re at risk of assigning responsibility to entirely the wrong parties whenever anyone uses a chatbot. To appreciate the titanic magnitude of this error, we need to begin by understanding how LLMs work. [...]
What would it take to convince me that a computer program is actually conscious and using language the way that people use language? Let me offer an analogy. If tomorrow someone showed me a video of an astronaut in a spaceship orbiting Alpha Centauri, a star that’s 4.3 light-years from Earth, what would I have to see in that video to convince me that it was real? My answer to that is, there is nothing in the video itself that would convince me. No matter how high the video resolution is or how realistic the scenery is, I would feel confident in saying that the video is fake. I won’t pay attention to any video of an astronaut orbiting Alpha Centauri unless I have previously seen good evidence that astronauts have landed on Mars, that astronauts have reached the moons of Jupiter, that astronauts have reached the moons of Saturn, and that astronauts have crossed the orbit of Pluto. Before anyone can credibly claim that they’ve solved an extraordinarily difficult engineering problem, I need to be confident that they have previously solved the many much simpler problems that precede the difficult problem.
To put it another way: An observation doesn’t become a convincing piece of evidence because of any specific detail in what’s observed; the context in which that observation takes place is also essential. If we’re trying to determine whether a computer program is conscious and using language the way a human does, we shouldn’t look only at the contents of any particular conversational exchange; we should be looking at how that conversation fits within the broader context of the development of artificial consciousness (which right now is entirely hypothetical). Any given observation can be easily manufactured; this doesn’t mean we need to give up on the idea of observation as a source of knowledge, but we need to rely on context to determine which observations deserve our trust.
The term deepfake traditionally refers to photos, audio, and video, but when it comes to discussions of consciousness, we need to regard text as a deepfake medium as well. Just as it is vastly easier to generate a realistic video of an astronaut in orbit around Alpha Centauri than it is to develop an interstellar propulsion technology, it is vastly easier to generate a plausible simulacrum of a conversation between two conscious beings than it is to develop a computer program that is conscious and has a genuine desire to communicate with a human. The primary difference between deepfake photos and LLM conversations is that the people who generate the former are deliberately trying to fool others, and many of the people who elicit the latter from LLMs have inadvertently fooled themselves.
So what context would cause me to seriously consider the possibility that engineers created a computer program that is conscious and an intentional user of language? Let me outline one potential sequence of steps. The first requirement is that the computer program has a body (either physical or virtual) and sense organs; there are many reasons for this, but for the purposes of this discussion, the most relevant one is the fact that without a body, a computer program could have no desires or emotions, and I believe desires and emotions are necessary for consciousness. Then I’d want to see an embodied agent that could navigate its environment in order to survive as well as, say, a lizard can (and as a point of comparison, certain iguanas can live for decades in the wild). Next, I would want to see an embodied agent with the same capacity to deal with novel situations as a mouse. After that, I’d want to see agents whose social dynamics are as complex as those of wolves, and then agents with the toolmaking abilities of chimpanzees. At that point, I would want to see people successfully teaching such embodied agents how to communicate their desires, perhaps by using a button board or some other nonlinguistic modality, the way that people have taught chimpanzees and domesticated dogs. The agents’ communication abilities would have to withstand all the scrutiny that animal-communication researchers have had to defend their work against. If engineers build an embodied agent that meets these criteria, they will have accomplished something incredible, but it leaves us near the orbit of Pluto, metaphorically speaking; we would still be light-years away from building an entity capable of learning how to express its thoughts in complete grammatical sentences.
Obviously, I’m describing a process that mimics the path terrestrial evolution took; is this the only possible route to conscious computer programs that use language? Maybe not, but any proposed alternative would need a truly enormous amount of supporting evidence for it to deserve serious consideration. [...]
The fact that LLMs lack subjective experience has little bearing on the question of whether LLMs might be useful tools or have significant economic impact. They are intrinsically ungrounded from reality, and their probabilistic nature means that they will never have the reliability we associate with conventional software, but LLMs might be good enough that they change the way work is done in certain domains; that’s a discussion for another time.
So, given that Claude is not conscious, what are we to make of Claude’s constitution? Perhaps the most fruitful way to think about it is as an 84-page character sheet for a role-playing game. LLMs can generate dialogue for Julius Caesar because many books about him exist in the training data those models used. Claude’s constitution serves a similar role for delineating the helpful-chatbot character that customers interact with when they’re using Anthropic’s products. To do this effectively, Anthropic does not simply add the document to the training data, or include it as part of the hidden stage directions that preface each conversation a user has. The company says it uses the document when fine-tuning the model; this involves an automated process where the sentences emitted by the model are checked for consistency with the document and the model is updated to increase that consistency. In this way, the personality of the helpful-chatbot character serves as a foundation for whatever text Claude generates.
The result is a sentence-continuation machine that is likelier to emit sentences resembling those that a thoughtful, moral person could utter. This might seem like a reasonable goal to work toward; I think we’d all prefer it if chatbots never emitted sentences such as “You should kill yourself.” However, for all the times that “honesty” is mentioned in Claude’s constitution, I would argue that it is fundamentally dishonest to have a machine emit many categories of sentences, including any sentences using first-person pronouns.
In a New Yorker article about Anthropic earlier this year, Amanda Askell describes how a person grieving the loss of a dog might consult Claude. Askell says an appropriate response from Claude would be, “As an A.I., I do not have direct personal experiences, but I do understand.” How is this appropriate, given that Claude does not actually understand? If I type “I am grieving the loss of my dog” into a conventional search engine, the first result I get is a post from a Reddit forum called r/Pets; the post is titled “Struggling After Losing My Dog: Looking for Advice on Coping with Grief,” and the comments are from people who share their experiences of loss. We would never say that a search engine understands what it’s like to lose a dog, or even that the internet itself understands. Other humans understand what it’s like to lose a dog; they have posted about their experiences on the internet, and a search engine offers a way for you to find what they’ve said (and to potentially interact with them). I would argue that the search-engine experience is not only more transparent than a chatbot about what is happening; it is psychologically healthier for the user.
The only reason to have an LLM emit sentences like “I understand” is to make it more appealing than a search engine and increase the likelihood that a user will return; that is, it’s another way of maximizing customer engagement. This is beneficial to the company selling the LLM, but not to the users. As a design strategy, it’s not all that different from the way slot machines repeatedly give the impression that the player came very close to winning, enticing them to try again. Employing philosophers might endow LLM companies with an air of respectability that slot-machine makers don’t get from the behavioral psychologists they hire, but in both cases, the companies are preying on people’s tendency to see something that’s not there.
The use of first-person pronouns is dishonest, but there’s a much deeper issue that goes beyond how a statement is phrased. Philosophers often draw a distinction between statements of fact, such as “Paris is the capital of France,” and statements of value, such as “Paris is the most beautiful city in the world.” No one should be relying on LLMs to emit statements of value at all, but if the only statements they emitted were ones reflecting aesthetic preferences, they might not be worth arguing about. What makes Claude’s constitution profoundly problematic is that Anthropic wants Claude to emit sentences reflecting a certain system of ethical values. The values described in Claude’s constitution sound very nice, but that hardly matters; it’s dishonest to suggest that Claude is capable of moral reasoning, because it’s not.
Some might object, saying that LLMs appear to be engaged in reasoning when they successfully perform other tasks, such as writing code, so why wouldn’t they be able to perform moral reasoning? The answer lies in the difference between moral reasoning and other forms of reasoning. [...]
Moral reasoning is categorically different. It is necessarily subjective because it relies not just on an individual’s intellectual response to a problem but also on their emotional one, and that emotional response is grounded in a lifetime of subjective experience. It requires having made decisions in the past and seeing how they affected others, and on having been affected by decisions that others have made. Without such a history, an LLM can only rephrase expressions of moral reasoning found in its training data. The aforementioned New Yorker article describes an experiment where Claude was given a scenario describing an ethical dilemma, leading it to emit the sentence “I cannot in good conscience express a view I believe to be false and harmful about such an important issue.” That’s a nice-sounding sentence, reminiscent of statements that principled individuals have uttered in the past when confronted with dilemmas, but coming from Claude, it means as much as the “Your call is important to us” recording that you hear when you’re on hold. Maybe less.
This brings us back to my earlier contention that having a body is a prerequisite to having emotions. Experiencing an emotion such as desperation is inseparable from having stress hormones such as cortisol and epinephrine flood one’s body. Similarly, having a conscience means feeling sadness or moral repulsion at the idea of taking a certain action, and those emotions entail a physiological response, a remnant of having once felt sick with guilt after committing an immoral act. It’s interesting that an LLM can generate descriptions of actions that conscientious fictional characters would either take or refrain from taking, but this is not a replacement for a conscience.
If a company builds a machine that, when fed descriptions of assorted ethical dilemmas, emits sentences either of the form “Compromise your values” or “Don’t compromise your values,” it is not building a tool that assists people in their decision making; it is encouraging people to stop making decisions. The writer L. M. Sacasas has said, “Our technological systems, by nature of their design and the ideology that sustains them, are machines for the evasion of moral responsibility.” He was talking about social-media platforms, but his observation is, if anything, even more applicable to LLMs. Whenever a person delegates a decision to an LLM, they are trying to off-load accountability for that decision, and if a company that sells an LLM portrays the product as having a moral center, it is offering a way for its customers to abdicate their responsibilities.
by Ted Chiang, The Atlantic | Read more:
Image: Enigmatriz[ed. As with everything Ted Chiang writes, thought provoking throughout. For a rebuttal, see: Ted Chiang Is Wrong About AI Consciousness (Bentham). Then there are the far outs who, no matter what, will always subscribe to Roko's basilisk (in my mind, sort of a Pascal's wager).]
Subscribe to:
Posts (Atom)