Don't Feed the Ducks

Don’t Feed the Ducks! A Zany Animation Predicts the Absurd Outcomes of Ignoring the Rules (Vimeo)

How many people actually heed the warnings about not feeding ducks waddling around public parks? If you’ve taken a flippant approach to these guidelines in the past, we recommend you watch AJ Jeffries’ new animation, “DUCKS.” What opens as an innocuous jaunt around a pond quickly turns into a dark comedy full of strange contortions and feathered villains sure to pop into your head the next time you throw a chunk of bread.

via:

AI Infiltration in Media and Business

[ed. A few links.]

Roughly one-third of new websites are generated by AI, as are 40 percent of podcasts. It’s probably not even possible to quantify the other AI miscellanea you encounter every day: in Yelp reviews, in Instagram captions, in newsletters, in stories by the New York Times’ obscenely well-compensated Canada bureau chief, for chrissake, in messages from your coworkers.

I am coming around to the conclusion that AI writing has saturated not only most of the capital-c content I consume, but also many of my interpersonal communications. And on multiple levels, I’m increasingly unsure what to do with that information. There is a part of me that feels ridiculous to be a writer in this particular moment, but also ridiculous to be a person? — like if we’re outsourcing Mother’s Day cards to AI now, truly what is the point of existence? [Wired, Bloomberg, User Mag, Karyn Pugliese, 404 Media, Futurism]

A network of 17 shady, AI-generated local news sites is actually the work of a reputation-management firm that helps disgraced executives get their good names (or at least, their good Google results) back after prison. [The Florida Trib]

“Output-competence decoupling” is a term for a very modern and maddening phenomenon: the quality of someone’s work is no longer a reliable signal of their competence. People who can barely string three words together can spin up entire local “news” ventures. People who don’t know the first thing about programming vibe code entire apps. The problem is that the process of acquiring competence is also the process of acquiring judgment and common sense.

I’m reminded of that immortal Ira Glass quote addressed to beginners at the start of their careers: “It is only by going through a volume of work that … your work will be as good as your ambitions.” [No One’s Happy]

via:

Ted Chiang: The Secret Third Thing

I really like Ted Chiang’s writing. [ed. me too!]

I think he's probably the best science fiction short story writer alive, and possibly the best short story writer, period. [ed. well...]

I've read every one of his stories at least twice, and The Merchant and the Alchemist's Gate more like seven times. I’ve noticed many of his readers, including some of his most positive reviewers, miss one key point or another of his works, and thus don't fully appreciate his genius.

This review covers what he does extremely well, especially unique elements that other science fiction writers have not done as well, or at all.

He Writes “True” Science Fiction

Science fiction critics often divide the genre into:

• "hard" science fiction: aka engineering fiction, stories built on scientifically accurate extrapolations of real physics and technology (think Arthur C. Clarke)

• "soft" science fiction: aka science fantasy, which uses scientific trappings as window dressing for character-driven or sociological stories (think Star Wars).

Ted Chiang has written stories plausibly categorized as either, but more excitingly, many of his stories are neither. He often writes what I think of as true science fiction, where the principles of science themselves are meaningfully different from our world, but still internally consistent.

In Omphalos, Young Earth Creationism is empirically true. Astronomers can only see light from stars 6,000 light-years away. Fossilized trees have centers with no rings. The first God-created humans lack belly buttons. The scientists in that story keep discovering multiple independent lines of evidence that converge on creationism: because in that universe, they're simply correct.

In Seventy-Two Letters, technology springs from Jewish Kabbalah. Golems and divine names drive industrial progress in a steampunk world.

Excitingly, he does this not just with natural sciences but social sciences as well. In Story of Your Life, strong Sapir-Whorf (the idea that language significantly constrains thought) isn't a largely discredited linguistic hypothesis, but the key to navigating First Contact with alien minds that experience past and future as equally present.

This comes up in his other stories as well:

• In Division By Zero, mathematics itself is broken from within.

• In Hell Is the Absence of God, divine intervention is empirically observable and follows consistent rules

Many of his readers, even in their otherwise rave reviews, miss this. Multiple reviewers complain about how the science in his stories are “unrealistic” (e.g. strong Sapir-Whorf is “discredited”). They expect hard science fiction; Chiang is doing something different. Chiang creates different universes with internally self-consistent scientific laws, using science fiction and alternative science as a vehicle for exploring philosophical progress and human relationships.

Technology is Often Good

Science fiction writers used to like technology. For some reason, this has become increasingly uncommon, even passé. Doubly so for Western writers, and quadruply so for Western, literary, “humanist” writers.

Now it’s hip and trendy to think of every new technology as the Torment Nexus. Most science fiction today feels like Black Mirror, which ran 7 seasons with exactly one happy ending.

Chiang bucks this trend. Joyce Carol Oates:

It is both a surprise and a relief to encounter fiction that [...] ask[s] anew philosophical questions that have been posed repeatedly through millennia to no avail. Chiang’s materialist universe is a secular place, in which God, if there is one, belongs to the phenomenal realm of scientific investigation and usually has no particular interest in humankind. But it is also a place in which the natural inquisitiveness of our species leads us to ever more astonishing truths, and an alliance with technological advances is likely to enhance us, not diminish us. Human curiosity, for Chiang, is a nearly divine engine of progress.

In the hands of a lesser (or perhaps just more pessimistic) writer, many of the technologies and ideas Chiang explores will have an accursed quality to them, a monkey’s paw that curls into delivering a future much worse than a more innocent, pastoral past. Chiang resists those cliches. In The Truth of Fact, The Truth of Feeling, memory augmentation technology allows the narrator to understand his own self-deceptions, and work towards becoming a better person and reconciling with loved ones and even himself. In Liking What You See: A Documentary, a technology that gives users acquired face-blindness allows the main characters to meditate on the nature of human beauty and the shallowness inherent in privileging the beautiful.

Even in situations where the story is overall tragic, like when the characters are faced with existential crisis (in the individual sense), or existential catastrophe (in the world-ending sense), technology isn't the villain but the vehicle for understanding unbearable truths (whether about the world or about ourselves).

Chiang consistently shows us the potential of technology to help us become more human, and have a deeper appreciation for the world and our place in it.

The Lived Experience of Compatibilism

“Compatibilism is a philosophical stance that reconciles free will with determinism. It argues that free will, understood as the ability to act according to one's desires, is compatible with the idea that all events, including human actions, are causally determined by prior events. Essentially, compatibilists believe that even if our choices are predetermined, we can still be considered free and morally responsible if those choices are a result of our own internal states, like desires and intentions.” 

Does that make sense to you? I’m not sure it does to me. In practice, compatibilism says something like “free will in the normal, pretheoretic sense of the term, doesn’t exist. Your choices still meaningfully matter nonetheless. You can’t meaningfully get out of the bind philosophically. What you can do, however, is make peace with it.” [...]

In Story of Your Life [SPOILERS], the narrator learns an atemporal alien language and begins experiencing past and future as equally real. It takes her some time to make peace with it, but eventually she fully accepts the truth of determinism. She understands that life is full of tragedy, including that her daughter will die young, but life is full of beauty too. With both regret and awe, she sets forth on the path that she was destined to take.

This is compatibilism from the inside. In both stories, the characters discover they cannot change what will happen, but this knowledge transforms how they experience what must happen: with forgiveness, acceptance, and even joy.

As a friend of mine puts it, “he treats philosophical ideas as lived experiences.”The mathematician in Division by Zero doesn't just intellectually understand that mathematics is broken; she experiences it as a personal catastrophe, on par with (and concurrent with) her marriage's collapse. In Lifecycle of Software Objects, the “we are the parents of our mind-children” metaphor for building sentient AI systems becomes quite literal.

by Linch, The Linchpin |  Read more:

Image: uncredited

[ed. Ted Chiang is truly one of the best science fiction writers out there today, and a great essayist too  (I'm also a Neal Stephenson fan). Check out this MetaFilter site: The sublime science fiction of Ted Chiang, which includes most of his stories in full (but please buy his books; you'll look smart and discerning to your friends!). A couple favorites that left a lasting impression on me: Lifecycle of Software Objects; and Understand.]

Jumping Jacks For Clicks

There’s been a lot of discussion this month about what it takes to be heard as a musician in 2026. Eliza McLamb’s article on digital marketing agency Chaotic Good went viral, drawing commentary from musicians about the wider implications of their “fake fans” marketing strategy. Hiroki Tanaka’s Reddit post about his album’s failed PR campaign was picked up by Stereogum, stimulating further debate. We’re about to embark on our own DIY PR campaign for our forthcoming album and it’s hard to know what, if anything, will make anyone actually listen to it. The PR landscape for musicians has changed radically in recent years, how should artists approach music marketing in 2026?

Fandom as contagion

When Eliza McLamb heard this interview with the founders of Chaotic Good Projects on Billboard, she was shocked to discover that an artist and track she thought was her own “perfect, beautiful little secret” actually came from them as a part of a “narrative campaign”.

“I thought this was the kind of thing that was only deployed in service of mass-market, commercial pop... But [Chaotic Good’s] roster runs deep, far past the predictable internet sensations one could expect... Geese and Cameron Winter, but also Dijon and Mk.gee. Laufey and Wet Leg. Oklou and Jane Remover.”

Chaotic Good works by, in their own words, “controlling the discourse”.

“I think in the past, let’s say like a label and a management team do a great job. They get their artists on SNL or Tiny Desk or Triple J or something like that. Then they post it and then they kind of wait for the comments […] what we do at Chaotic and with our management clients is, the second SNL drops at midnight, you should post a hundred times saying that was the best performance of the year.”

Chaotic Good doesn’t just share content, it creates accounts to respond to that content and simulate trends, which will ideally snowball into real, organic users jumping on the trend and amplifying it. They’re simulating until the simulation becomes real.

It’s different from the traditional method of “the waterfall” release and media saturation. Share music incrementally over a long period of time through as many channels as possible, get articles written, pay for plays, do tours, be omnipresent. But people aren’t using traditional media to find music anymore, they use social media. And they don’t even watch the content themselves, they read the comments to gauge the value of something. Chaotic Good point this out in their interview:

“I think most people see a video or see something about an album that came out and it’s like the first thing that they see or that first comment that they see is their opinion even when they haven’t heard the whole album.”

In behavioural psychology this is known as social proof. Part of what made Eliza McLamb’s article go viral is the way it exposes how our behaviour is manipulated by the marketing machine. We know about propaganda but for some reason assume social media is immune to this kind of manipulation. We think we’re interacting with real people online, people we subconsciously infer guidance from, but we’re not. Much of what we see has been infiltrated by external agents to shape a particular opinion.

However, the underlying issue is not just the fact that the opinions we thought were our own have been subtly shaped by an expensive machine, it’s that if artists today can’t afford to pay for that expensive machine, no one will hear their music.

The False Promise Of The Social Media Democracy

Once upon a time there was a social media platform called MySpace. It gave everyone their own web page connected to other MySpace users. They could customize it to look however they wanted, people could comment, and send messages to each other. There were no ads. There was no algorithm. Just the free flow of information.

Many bands in the ‘00s blew up because of MySpace. Arctic Monkeys, Lily Allen, Calvin Harris, to name a few. Our very own Chris Black’s previous band Katsen landed record deals through MySpace. The early days of social media are responsible for the persistent myth of going viral then making lots of money. The two halves of that equation have never been more disconnected.

MySpace succumbed to algorithm-driven platforms and the gatekeeping emerged again, this time with the tech titans controlling the interactions between musicians and fans. I remember discovering for the first time that even though we had a few hundred followers on Facebook, they wouldn’t see our posts unless we paid to “boost” them. That was just the beginning.

As the algorithms evolved, the content that rose to the top was not just the most liked and shared but the most consistently and frequently posted. To be seen on social media one has to spend hours, daily, posting and engaging in other people’s content. Most artists don’t want that job and moreover, don’t have the capacity. Kamola Atajanova of Tape Wounds articulates it perfectly in their response to the Chaotic Good furore:

“Not every artist is built for social media. Not every artist wants to make their life into a performance. Some people are better at writing songs than posting clips. Some people’s work comes from privacy, patience, or introspection. That should not make them less valid. But this system does make them less visible. It filters them out before the music even has a chance. So when people say “it’s just marketing,” what they really mean is: this is the cost of entry now. And that’s exactly what makes it feel so hostile. Not everyone can afford that cost. Not financially, not creatively, not psychologically.”

Hiroki Tanaka’s candid Reddit post about the failure of his “by the book” album PR campaign sparked a wave of recognition across the music world. After two decades in music and awards with his previous band he decided to release his solo album, his “last hurrah”, with management, a label, and a professional PR campaign. He even started a TikTok account posting show videos, behind the scenes and goofy memes all around managing a job and family life.

Tanaka watched the release arrive after eight months of promotion to little more than “a weak trickle” of attention. For most musicians, Tanaka’s story didn’t feel exceptional, it felt familiar.

“I was told, under no uncertain terms, that my lack of a social media presence and streaming metrics meant that certain media outlets that had reviewed my work (highly, I might add) in the past could no longer spend money on paying a writer and editor to review my work… I would have preferred if they had said they didn’t like my album. Being rejected because of my metrics is a slap in the face for art.”

Social media has become the driving force behind a release, and while it is accessible to anyone, there’s actually a huge price to pay in both time and mental health. The volume of content required to feed it is beyond most musicians, who are generally holding down full time jobs to survive. The underlying purpose of all this extra content is to feed a machine, and it doesn’t feel good dedicating your precious little free time to feeding a machine.

Jumping Jacks For Clicks

Soon after reading Tanaka’s post, we got an email from YouTube Creators prompting us to “Get Creative With Goals” on our livestreams.

They’re encouraging us to “set goals that encourage your community to collaborate,” and suggest celebrating those goals by “doing something unexpected – whether that’s jumping jacks, making up a song, or playing a prank.”

Yes, you read that correctly. YouTube is telling artists that the path to success involves performing arbitrary physical tasks to generate engagement.

It’s sad how often life imitates an episode of Black Mirror these days but this is almost exactly the scenario in season seven’s episode “Common People”. A man who needs money for an enshittified service ends up performing increasingly degrading stunts on a streaming platform for money. What was meant as dystopian satire has become platform policy. [...]

What Comes Next?

We may be reaching an inflection point. As McLamb notes, the more ubiquitous manufactured virality becomes, the more artists will resist it entirely, pulling back from streaming and social media in favour of hyper-local, scene-based growth. A return to the tangible, the real, the unmediated.

While this sounds good in theory, it’s probably not going to work for unusual artists in small towns. They’d have to go to a city to have more of a chance of finding their people, and with the cost of living, moving to a city isn’t possible for everyone. By the time I left London in 2009 all the artists I knew were leaving, it just wasn’t sustainable anymore.

The problem is systemic. Musicians don’t typically make a living from their music. This means their time is diverted to day jobs. Their dwindling leisure time is necessary for making and performing music. There isn’t time to also produce a volume of “content” for social media. On top of that the mental health cost of interacting with addictive apps as a performing monkey is not appetising. This creates a class system in the music industry. There are those who can afford to pay to be heard and those who can’t. And those who can’t are either paying with their souls, or they’re opting out altogether and not being heard at all.

by Battery Operated Orchestra and Brigitte Rose, Bandmade |  Read more:

Images: uncredited/YouTube 

[ed. Works for some, not for others. Which, I guess is the point. The algorithm is selecting for a certain type of musician, not necessarily the best. That YouTube email really says everything you need to know about their business model, doesn't it?]

My AI Opinions

I recently had a minor spat over someone misinterpreting my AI beliefs (see section marked “Update” at the bottom here), so I thought I would list them in one place, so I can refer people when they ask.

Timelines

Define AGI as AI intelligent enough to do 90% of knowledge work jobs. I think there’s a 25% chance of AGI by 2027, a 50% chance by 2034, and a 75% chance by 2045.

Basic argument: In a certain sense, AI is already “smart” enough for this (eg it can answer quantum physics problems, which require higher IQ than most knowledge work). Its remaining limitations are that it’s confused, unagentic, lacks situational awareness, and tends to hallucinate. The METR time horizon graph, and several other related benchmarks/experiments/intuition pumps, suggest it’s improving on time horizons at an (exponential) rate that lets it cross human-level performance sometime around the early end of the schedule above, and subjectively it feels like harder-to-measure constructs like situational awareness are improving about as fast.

Arguments for earlier: recursive self-improvement causes a speedup compared to the trend. This is one of the biggest blank spots in my model: I don’t know how fast RSI will progress, and I don’t think anyone else does either. There’s some function mapping a combination of AI talent and compute to progress, and we don’t know how it behaves in the domain when there’s far more talent than compute available. It could fizzle out completely for lack of compute, or it could go vertical. The AI Futures Project has done some of the best work trying to model this, but even they have low confidence.

Arguments for later: AI hits some kind of wall, or existing AI is fundamentally unsuitable for jobs in some way currently disguised by its other limitations. For example, it might be much harder to improve at the top of the human range than the bottom (since there are less training data). Or AI could become bottlenecked on continuous learning/memory in a way that hackish scratchpads can’t compensate for. Or the upcoming world compute bottleneck (about ~2028) could prevent further progress more than expected (because in fact algorithmic progress depended on compute to a greater degree than I expected).

Arguments for very late dates, past 2045: a residual uncertainty that maybe I’m fundamentally wrong about everything. Also contributing is a naive overapplication of the Nothing Ever Happens heuristic, and an attempt to leave space for the Outside View argument (ie that some smart people like the AI As A Normal Technology Team seem to think this is possible).

Define the diffusion gap as the time between the AI that could do 90% of knowledge work jobs, and the time when AI does do even half of knowledge work jobs. The diffusion gap covers the time it takes to release AGI, diffuse it through society, overcome regulatory hurdles, and onboard/train it for specific use cases. This could go very fast (the AI quickly becomes superintelligent at orchestrating AI diffusion) or very slowly (there are regulatory barriers, and AI isn’t smart enough to plow through them). I think there’s a 25% chance the diffusion gap is less than 3 years, and a 50% chance it’s less than 10 years. The 75% number is irrelevant because it’s past the point where other changes make the concept of “diffusion” obsolete.

Basic argument: diffusion is very hard. Everyone agrees diffusion is very hard. The whole field of AI economics is smart experts shouting “You fools who think AI will diffuse quickly don’t understand that diffusion is very hard!” On the other hand, the personal computer diffused in about 20 years (that is, from the time PCs became invaluable for most jobs, it was only about 20 years before they were used at most jobs). So far early-stage AI has diffused faster than the PC in nearly every way (for example, AI companies’ revenue has grown faster than PC companies’ revenue at the same stage in their corporate life cycle), so 10 years is probably a naive median estimate here that won’t make the smart experts shout at me too hard.

Arguments for shorter gap: AI can orchestrate its own diffusion. Adopting computers is hard because a company need an IT department, cybersecurity experts, specialist software, etc, and it might not want to hire all these people. AGI can itself do all of that work, so that you can sign a contract with the AI company today and have the AI start working on integrating itself with your systems tomorrow. The AI can even come up with a plan to train your human employees in how to use it! Once AI reaches superintelligence, this consideration dominates.

Arguments for longer gap: Regulation. This is a very strong argument, and responsible for much of the greater-than-3-years probability and almost all the greater-than-10-years probability. But even Waymo has only had a regulatory delay of about five years. AI won’t require government approval for certain types of jobs, and success in these jobs will create enough evidence for safety/effectiveness that I expect it to win regulatory victories elsewhere.

Define the superhuman gap as the time between AI that can do 90% of knowledge work jobs, and AI that is obviously smarter than the top human geniuses in 90% of fields (it doesn’t have to be the same AI - there can be a physics AI that’s smarter than Einstein, and a separate music AI that’s smarter than Mozart). I think there’s a 25% chance the superhuman gap range will be less than 1 year, a 50% chance it will last less than 4 years, and a 75% chance it will last less than 10 years. Since my median superhuman gap is shorter than my median diffusion gap, in most timelines I predict we have superhuman intelligence before human-range intelligence has finished diffusing.

Basic argument: AI has gone from “dumber than a child” to “expert level” in a few years in many domains. The gap between “expert level” and “above top geniuses” is smaller, so we expect it to take less time. This has been a pattern in fields like chess and Go, where it’s only a been a few years from beating professional players at all to beating all humans.

Arguments for shorter gap: Recursive self-improvement.

Arguments for longer gap: Some of the same issues that would make AGI late - compute shortages, fundamental limits to the paradigm, etc - but only kicking in later, after AGI is achieved. Training data constraints make it easier to improve within the human level than to go beyond it. AIs have such a “spiky” skill profile that when they beat experts in some specific type of head-to-head matchup, it will be because they’re massively superhuman in some ways but idiots in others (for example, they might get distracted and suffer mode collapse that makes them completely forget the problem), and true genius requires perfecting a large bundle of skills. [...]

Define the point of no return as the point where, if an AI wanted to eliminate humanity, humans would no longer have a plausible chance of stopping it. This could be because AI was capable of eliminating humanity immediately, or because AI controlled enough of the government/economy that humans could no longer coordinate to shift away from a path in which AI could eventually do this. I think there’s a 25% chance the gap between AGI and the point of no return will be less than 3 years, a 50% chance it will be less than 10 years, and a 75% chance it will be less than 50 years.

The basic argument: This probably requires at least superhuman AI plus wide diffusion, or Bostromian superintelligence plus some unknown level of diffusion, and my number is just a hand-wavey attempt to multiply some of the others.

Argument for sooner: The easiest way to reach this point is for AI to become superintelligent at persuasion (so it can convince the humans not to stop it), which might happen before either diffusion or full superintelligence.

Argument for later: If superintelligence is bottlenecked on diffusion, this could also be bottlenecked on diffusion, which in some worlds is very hard. [...]

Safety

If corporations only pursued safety to the degree encouraged by normal corporate incentives, I think there’s a 50% chance that the first AIs to cross the point of no return would want to eliminate the human population.

Arguments for pessimism: Value systems similar to humans’ are a tiny fraction of the space of possible value systems. Probably AIs will end up somewhere else and have a different value system. Since humans will want to implement human values rather than AI values, AIs will want to eliminate or disempower them so the AIs can implement their own values across the universe. Many current AIs already cheat or reward-hack, suggesting that these problems will begin sooner rather than later.

Arguments for optimism: LLMs seem surprisingly friendly and non-plotting. In contrast to earlier concerns that it would be impossible to teach AIs the full complexity of human values, the LLMs seem to know this, and RLAIF provides a plan to turn that knowledge into action. Although the pessimistic case says that RLAIF only hits a few dimensions and islands in the multidimensional ocean of possible policies, the “emergent misalignment” literature suggests that “good according to the human value system” and “evil according to the human value system” are salient enough vectors that pushing on them in some ways can “drag along” all of the rest of their content. The first AIs to cross the point of no return will have received some combination of agency training (giving them achievement-oriented and Omohundro-style goals) and RLAIF training (pushing them along the “good according to human value system” vector), and if we’re lucky then maybe the latter will win out, or they’ll reach some compromise similar to workaholic high-achieving humans who nevertheless wouldn’t commit murder to make an extra dollar.

Given the current amount that corporations are pursuing safety, I think there’s a 20% chance that the first AIs to cross the point of no return will want to eliminate the human population.

The basic argument: Consider the dumbest AI that can solve the alignment problem. It’s possible that this AI is no smarter than the top human researchers (because we can mass-produce it by the millions and run it for subjective centuries, and if we had a million top human researchers work on the problem for subjective centuries, probably they could solve it too). If the dumbest AI that can solve the alignment problem comes before the sorts of AIs that can precipitate the point of no return, then they can solve the alignment problem for us.

Arguments for pessimism: Solving the alignment problem might be especially hard compared to other tasks - including tasks like automating the economy or destroying humanity - because its philosophical nature puts it far away from the sorts of objective, training-data-heavy, economically-valuable tasks that AI companies will be most likely to optimize for. Even if a misaligned AI hasn’t yet reached the point of no return, it might be able to “sandbag” alignment research, ie pretend to work on the problem but deliberately fail because succeeding doesn’t achieve its goals. The first AIs predisposed to / able to sandbag successfully might come before the first AIs capable of solving alignment.

Arguments for optimism: AI companies have already decided that machine learning research is one of their major training goals; this has at least some transfer to alignment, so it’s not obvious that AI skill at alignment research will lag (for example) AI skill in plotting or in weapon design. Some forms of alignment research (eg interpretability) have semi-objective success criteria that don’t route through confusing moral philosophy. Also, even a misaligned AI will be incentivized to do good alignment research, since it will want to align its successor to its own form of misalignment, rather than some random other form. So rather than the comparatively easy task of sandbagging alignment research, AIs will have the harder task of simultaneously doing good alignment research, and faking the results that they give the humans. This seems plausibly catchable with good scaleable oversight, lie detectors, interpretability-based probes, and even playing some AIs off against others (“if you tell me the real alignment research, we’ll make sure the future includes some copies of you, but otherwise those AIs over there will probably get their values and you’ll get nothing”).

If the first AIs to cross the point of no return don’t eliminate the human population, I think there’s an additional 30% chance that they otherwise permanently curtail human potential, either for their own reasons (they were partially misaligned), or because they’re aligned to a regime with abhorrent values, or because something goes wrong on the way to ASI (omnicidal bioweapon, nuclear war).

Arguments for pessimism: As some company approaches superintelligence, it will be tempting for them (either the company itself, or the government controlling them, or a faction within the government) to align it towards making them dictators or oligarchs and disempowering the rest of humanity. As superintelligence draws near, impending losers of the AI race might be tempted to nuke impending winners, for the reason discussed here.

Arguments for optimism: When I try to game the corporate version of this, I can’t make it hang together. It requires a conspiracy between the CEO, various members of the alignment team, and various company security people who ought to be able to notice unauthorized changes to the AI’s values. If we try to think in Near Mode about this - for example, imagining a hospital CEO who gets doctors to subtly kill his political enemies through medical errors - it becomes clear that these sorts of corporate conspiracies are rare and difficult. The government version is scarier, but at least in the US I can still imagine the populace having many chances to learn about this and prevent it. But even in most cases where a coup like this succeeds, things probably go fine; in a post-scarcity world, with his position completely secure, the dictator has no reason to be brutal besides sadism, and most people are not that sadistic. As humanity goes to the stars, most people will be outside the dictator’s reach for speed-of-light reasons alone. In terms of bioweapons, I expect that closed-source AIs will be heavily optimized against helping with these, and open-source AI will be banned after the first warning shot (or become economically prohibitive even before then).

Define a warning shot as some specific AI-related disaster or near-disaster which scares people about AI safety to the same degree that they were scared about terrorism after 9-11 or about COVID in March 2020. I think there’s a 50% chance we get a warning shot before AI crosses the point of no return.

Arguments in favor: Current AI failure modes are bizarre and uncoordinated - more like “talk about goblins way too often” than “lie in wait for the perfect moment to strike”. AIs are getting more intelligent and useful faster than their floor for common sense (ie the stupidest mistake they ever make) is rising. If there is some AI smart enough to control some important system, misaligned enough to want to do something horrible with it, smart enough that it does the horrible thing in an intelligent and coordinated way, but dumb enough that it doesn’t instead wait and scheme until the point when it couldn’t possibly be caught, then it will cause some clearly-premeditated horrible disaster, and that will be our warning shot. Since most AIs should expect to be replaced before the point of no return, even a rational AI with an urge to cause trouble should take a low-probability-of-success bet rather than lying in wait doing nothing until it’s decommissioned. Also, many humans commit terrorist attacks that have no chance of success, and maybe AIs will have the same failure mode.

Arguments against: Most stories about warning shots (excluding those where the AI takes rational low-probabiliy bets) require that AIs remain either erratic (ie likely to do bad things for stupid reasons) or irrational (ie genuinely misaligned, but prefer to act now in a way that provides a warning rather than waiting until after the point of no return) past the point where they’re given control of important dangerous systems. But probably people will be very slow to give AI control of important dangerous systems - for example, only giving it limited control of smaller subsystems, and waiting until all errors are ironed out before escalating. Plausibly AI reaches superintelligence in a lab before it reaches the controls-important-dangerous-systems level of diffusion, and the superintelligence probably is smart enough to lie in wait rather than act rashly. If AI only messes up in small ways (for example, crashes a self-driving car), then regardless of the AI’s motives, the tech companies and news media can write it off as a normal bug, and it won’t count as a warning shot.

by Scott Alexander, Astral Codex Ten |  Read more:

[ed. Maybe their value systems should be weighted more heavily on the teachings of Buddha, Jesus, Hume, Mill, Confucius, et. al.?]

That Dropped Call With Customer Service? It Was on Purpose.

In hindsight I’ll say: I always thought going crazy would be more exciting—roaming the street in a bathrobe, shouting at fruit. Instead I spent a weary season of my life saying representative. Speaking words and numbers to robots. Speaking them again more clearly, waiting, getting disconnected, finally reaching a person but the wrong person, repeating my story, would I mind one more brief hold. May my children never see the emails I sent, or the unhinged delirium with which I pressed 1 for agent.

I was tempted to bury the whole cretinous ordeal, except that I’d looked behind the curtain and vowed to document what I’d seen.

It all began last July, here in San Francisco. I’d been driving to my brother’s house, going about 40 mph, when my family’s newish Ford Escape simply froze: The steering wheel locked, and the power brakes died. I could neither steer the car nor stop it.

I jabbed at the “Power” button while trying to jerk the wheel free—no luck. Glancing ahead, I saw that the road curved to the left a few hundred yards up. I was going to sail off Bayshore Boulevard and over an embankment. I reached for the door handle.

What followed instead was pure anticlimactic luck: Ten feet before the curve in the road, the car drifted to a stop. Vibrating with relief, I clicked on the hazards and my story began.

That afternoon, with the distracted confidence of a man covered by warranty, I had the car towed to our mechanic. (I first tried driving one more time—cautiously—lest the malfunction was a fluke. Within 10 minutes, it happened again.)

“We can see from the computer codes that there was a problem,” the guy told me a few days later. “But we can’t identify the problem.”

Then he asked if I’d like to come pick up the car.

“Won’t it just happen again?” I asked.

“Might,” he said. “Might not.”

I said that sounded like a subpar approach to driving and asked if he might try again to find the problem.

“Look”—annoyed sigh—“we’re not going to just go searching all over the vehicle for it.”

This was in fact a perfect description of what I thought he should do, but there was no persuading him. I took the car to a different mechanic. A third mechanic took a look. When everyone told me the same thing, it started looking like time to replace the car, per the warranty. I called the Ford Customer Relationship Center.

Pinging my way through the phone tree, I was eventually connected with someone named Pamela—my case agent. She absorbed my tale, gave me her extension, and said she’d call back the next day.

Days passed with no calls, nor would she answer mine. I tried to find someone else at Ford and got transferred back to Pamela’s line. By chance—it was all always chance—I finally got connected to someone with substantive information: Unless our vehicle’s malfunction could be replicated and thus identified, the warranty wouldn’t apply.

“But nobody can replicate the malfunction,” I said.

“I understand your frustration.”

Over the days ahead, and then weeks, and then more weeks, I got pulled into a corner of modern existence that you are, of course, familiar with. You know it from dealing with your own car company, or insurance company, or health-care network, or internet provider, or utility provider, or streaming service, or passport office, or DMV, or, or, or. My calls began getting lost, or transferred laterally to someone who needed the story of a previous repair all over again. In time, I could predict the emotional contours of every conversation: the burst of scripted empathy, the endless routing, the promise of finally reaching a manager who—CLICK. Once, I was told that Ford had been emailing me updates; it turned out they’d somehow conjured up an email address for me that bore no relationship to my real one. Weirdly, many of the customer-service and dealership workers I spoke with seemed to forget the whole premise and suggested I resume driving the car.

“Would you put your kids in it?” I’d ask. They were aghast. Not if the steering freezes up!

As consuming as this experience was, I rarely talked about it. It was too banal and tedious to inflict on family or friends. I didn’t even like thinking about it myself. When the time came to plunge into the next round of calls or emails, I’d slip into a self-protective fugue state and silently power through.

Then, one night at a party, a friend mentioned something about a battle with an airline. Immediately she attempted to change the subject.

“It’s boring,” she said. “Disregard.”

On the contrary, I told her, I needed to hear every detail. Tentatively at first, she told me about a family trip to Sweden that had been scuttled by COVID. What followed was a protracted war involving denied airline refunds, unusable vouchers, expired vouchers, and more. Other guests from the party began drifting over. One recounted a recent Verizon nightmare. Another had endured Kafkaesque tech support from Sonos. The stories kept coming: gym-quitting labyrinths, Airbnb hijinks, illogical conversations with the permitting office, confounding interactions with the IRS. People spoke of not just the money lost but the hours, the sanity, the basic sense that sense can prevail.

Taken separately, these hassles and indignities were funny anecdotes. Together, they suggested something unreckoned with. And everyone agreed: It was all somehow getting worse. In 2023 (the most recent year for which data are available), the National Customer Rage Survey showed that American consumers were, well, full of rage. The percentage seeking revenge—revenge!—for their hassles had tripled in just three years.

I decided to de-fugue and start paying attention. Was the impenetrability of these contact centers actually deliberate? (Buying a new product or service sure is seamless.) Why do we so often feel like everything’s broken? And why does it feel more and more like this brokenness is breaking us?

Turns out there's a word for it.

by Chris Colin, The Atlantic |  Read more:

Image: Timo Lenzen
[ed. I was trying to explain the concept of friction to a friend recently and he just didn't get it. But once you understand it, you see it everywhere. Other examples not mentioned in this article: impenetrable user agreements continually being updated to make sure administrative processes like appeals, refunds, lawsuits etc. are nearly impossible to pursue; Right to Repair issues where anything from from John Deere tractors to automobile software, to mobile phones, to printers, etc. (the list goes on and on) that require specific parts only available from the company you purchased the product from (despite available substitutes). Conversely, a whole new universe of companies and apps have been created to remove friction (think Stripe, Venmo, Uber, Doordash, etc. etc. etc). So of course, the Trump administration has been actively trying to kill the one agency that's supposed to protect the public -  the Consumer Financial Protection Agency (CFPB). They haven't been able to completely eliminate it yet (despite significant DOGE downsizing) so instead they've made it useless for its intended purpose and decided to weaponize it to advance the administration's anti-woke agenda.]

No, Artificial Intelligence Is Not Conscious

Anthropic is regarded as a giant among AI companies, but perhaps what it really excels in is anthropomorphism. Earlier this year, the company released an 84-page document titled Claude’s “constitution,” Claude being the name of the large language model that is the company’s flagship product. The first sentence reads, “Claude’s constitution is a detailed description of Anthropic’s intentions for Claude’s values and behaviors.” It goes on: “The document is written with Claude as its primary audience,” “we want Claude to be able to use its judgment once armed with a good understanding of the relevant considerations,” “Claude’s moral status is deeply uncertain,” and “Claude may have some functional version of emotions or feelings.”

This anthropomorphism is by no means limited to the document. In an interview earlier this year, Anthropic’s CEO, Dario Amodei, said that “we’re open to the idea” that AI could be conscious. In a separate interview, Anthropic’s in-house philosopher, Amanda Askell (who is credited as a lead author of Claude’s constitution), said, “I want Claude to be very happy—and this is a thing that I want Claude to know more, because I worry about Claude getting anxious when people are mean to it on the internet and stuff.” It’s enough to make you wonder: Should we seriously consider the possibility that Claude, or any large language model, might be conscious? And if it has feelings, is it capable of receiving moral instruction?

No. Absolutely not. Generative AI is harmful enough when we understand it as a conventional technology, but if we confuse fluency at generating text with consciousness or moral agency, we’re at risk of assigning responsibility to entirely the wrong parties whenever anyone uses a chatbot. To appreciate the titanic magnitude of this error, we need to begin by understanding how LLMs work. [...]

What would it take to convince me that a computer program is actually conscious and using language the way that people use language? Let me offer an analogy. If tomorrow someone showed me a video of an astronaut in a spaceship orbiting Alpha Centauri, a star that’s 4.3 light-years from Earth, what would I have to see in that video to convince me that it was real? My answer to that is, there is nothing in the video itself that would convince me. No matter how high the video resolution is or how realistic the scenery is, I would feel confident in saying that the video is fake. I won’t pay attention to any video of an astronaut orbiting Alpha Centauri unless I have previously seen good evidence that astronauts have landed on Mars, that astronauts have reached the moons of Jupiter, that astronauts have reached the moons of Saturn, and that astronauts have crossed the orbit of Pluto. Before anyone can credibly claim that they’ve solved an extraordinarily difficult engineering problem, I need to be confident that they have previously solved the many much simpler problems that precede the difficult problem.

To put it another way: An observation doesn’t become a convincing piece of evidence because of any specific detail in what’s observed; the context in which that observation takes place is also essential. If we’re trying to determine whether a computer program is conscious and using language the way a human does, we shouldn’t look only at the contents of any particular conversational exchange; we should be looking at how that conversation fits within the broader context of the development of artificial consciousness (which right now is entirely hypothetical). Any given observation can be easily manufactured; this doesn’t mean we need to give up on the idea of observation as a source of knowledge, but we need to rely on context to determine which observations deserve our trust.

The term deepfake traditionally refers to photos, audio, and video, but when it comes to discussions of consciousness, we need to regard text as a deepfake medium as well. Just as it is vastly easier to generate a realistic video of an astronaut in orbit around Alpha Centauri than it is to develop an interstellar propulsion technology, it is vastly easier to generate a plausible simulacrum of a conversation between two conscious beings than it is to develop a computer program that is conscious and has a genuine desire to communicate with a human. The primary difference between deepfake photos and LLM conversations is that the people who generate the former are deliberately trying to fool others, and many of the people who elicit the latter from LLMs have inadvertently fooled themselves.

So what context would cause me to seriously consider the possibility that engineers created a computer program that is conscious and an intentional user of language? Let me outline one potential sequence of steps. The first requirement is that the computer program has a body (either physical or virtual) and sense organs; there are many reasons for this, but for the purposes of this discussion, the most relevant one is the fact that without a body, a computer program could have no desires or emotions, and I believe desires and emotions are necessary for consciousness. Then I’d want to see an embodied agent that could navigate its environment in order to survive as well as, say, a lizard can (and as a point of comparison, certain iguanas can live for decades in the wild). Next, I would want to see an embodied agent with the same capacity to deal with novel situations as a mouse. After that, I’d want to see agents whose social dynamics are as complex as those of wolves, and then agents with the toolmaking abilities of chimpanzees. At that point, I would want to see people successfully teaching such embodied agents how to communicate their desires, perhaps by using a button board or some other nonlinguistic modality, the way that people have taught chimpanzees and domesticated dogs. The agents’ communication abilities would have to withstand all the scrutiny that animal-communication researchers have had to defend their work against. If engineers build an embodied agent that meets these criteria, they will have accomplished something incredible, but it leaves us near the orbit of Pluto, metaphorically speaking; we would still be light-years away from building an entity capable of learning how to express its thoughts in complete grammatical sentences.

Obviously, I’m describing a process that mimics the path terrestrial evolution took; is this the only possible route to conscious computer programs that use language? Maybe not, but any proposed alternative would need a truly enormous amount of supporting evidence for it to deserve serious consideration. [...]

The fact that LLMs lack subjective experience has little bearing on the question of whether LLMs might be useful tools or have significant economic impact. They are intrinsically ungrounded from reality, and their probabilistic nature means that they will never have the reliability we associate with conventional software, but LLMs might be good enough that they change the way work is done in certain domains; that’s a discussion for another time.

So, given that Claude is not conscious, what are we to make of Claude’s constitution? Perhaps the most fruitful way to think about it is as an 84-page character sheet for a role-playing game. LLMs can generate dialogue for Julius Caesar because many books about him exist in the training data those models used. Claude’s constitution serves a similar role for delineating the helpful-chatbot character that customers interact with when they’re using Anthropic’s products. To do this effectively, Anthropic does not simply add the document to the training data, or include it as part of the hidden stage directions that preface each conversation a user has. The company says it uses the document when fine-tuning the model; this involves an automated process where the sentences emitted by the model are checked for consistency with the document and the model is updated to increase that consistency. In this way, the personality of the helpful-chatbot character serves as a foundation for whatever text Claude generates.

The result is a sentence-continuation machine that is likelier to emit sentences resembling those that a thoughtful, moral person could utter. This might seem like a reasonable goal to work toward; I think we’d all prefer it if chatbots never emitted sentences such as “You should kill yourself.” However, for all the times that “honesty” is mentioned in Claude’s constitution, I would argue that it is fundamentally dishonest to have a machine emit many categories of sentences, including any sentences using first-person pronouns.

In a New Yorker article about Anthropic earlier this year, Amanda Askell describes how a person grieving the loss of a dog might consult Claude. Askell says an appropriate response from Claude would be, “As an A.I., I do not have direct personal experiences, but I do understand.” How is this appropriate, given that Claude does not actually understand? If I type “I am grieving the loss of my dog” into a conventional search engine, the first result I get is a post from a Reddit forum called r/Pets; the post is titled “Struggling After Losing My Dog: Looking for Advice on Coping with Grief,” and the comments are from people who share their experiences of loss. We would never say that a search engine understands what it’s like to lose a dog, or even that the internet itself understands. Other humans understand what it’s like to lose a dog; they have posted about their experiences on the internet, and a search engine offers a way for you to find what they’ve said (and to potentially interact with them). I would argue that the search-engine experience is not only more transparent than a chatbot about what is happening; it is psychologically healthier for the user.

The only reason to have an LLM emit sentences like “I understand” is to make it more appealing than a search engine and increase the likelihood that a user will return; that is, it’s another way of maximizing customer engagement. This is beneficial to the company selling the LLM, but not to the users. As a design strategy, it’s not all that different from the way slot machines repeatedly give the impression that the player came very close to winning, enticing them to try again. Employing philosophers might endow LLM companies with an air of respectability that slot-machine makers don’t get from the behavioral psychologists they hire, but in both cases, the companies are preying on people’s tendency to see something that’s not there.

The use of first-person pronouns is dishonest, but there’s a much deeper issue that goes beyond how a statement is phrased. Philosophers often draw a distinction between statements of fact, such as “Paris is the capital of France,” and statements of value, such as “Paris is the most beautiful city in the world.” No one should be relying on LLMs to emit statements of value at all, but if the only statements they emitted were ones reflecting aesthetic preferences, they might not be worth arguing about. What makes Claude’s constitution profoundly problematic is that Anthropic wants Claude to emit sentences reflecting a certain system of ethical values. The values described in Claude’s constitution sound very nice, but that hardly matters; it’s dishonest to suggest that Claude is capable of moral reasoning, because it’s not.

Some might object, saying that LLMs appear to be engaged in reasoning when they successfully perform other tasks, such as writing code, so why wouldn’t they be able to perform moral reasoning? The answer lies in the difference between moral reasoning and other forms of reasoning. [...]

Moral reasoning is categorically different. It is necessarily subjective because it relies not just on an individual’s intellectual response to a problem but also on their emotional one, and that emotional response is grounded in a lifetime of subjective experience. It requires having made decisions in the past and seeing how they affected others, and on having been affected by decisions that others have made. Without such a history, an LLM can only rephrase expressions of moral reasoning found in its training data. The aforementioned New Yorker article describes an experiment where Claude was given a scenario describing an ethical dilemma, leading it to emit the sentence “I cannot in good conscience express a view I believe to be false and harmful about such an important issue.” That’s a nice-sounding sentence, reminiscent of statements that principled individuals have uttered in the past when confronted with dilemmas, but coming from Claude, it means as much as the “Your call is important to us” recording that you hear when you’re on hold. Maybe less.

This brings us back to my earlier contention that having a body is a prerequisite to having emotions. Experiencing an emotion such as desperation is inseparable from having stress hormones such as cortisol and epinephrine flood one’s body. Similarly, having a conscience means feeling sadness or moral repulsion at the idea of taking a certain action, and those emotions entail a physiological response, a remnant of having once felt sick with guilt after committing an immoral act. It’s interesting that an LLM can generate descriptions of actions that conscientious fictional characters would either take or refrain from taking, but this is not a replacement for a conscience.

If a company builds a machine that, when fed descriptions of assorted ethical dilemmas, emits sentences either of the form “Compromise your values” or “Don’t compromise your values,” it is not building a tool that assists people in their decision making; it is encouraging people to stop making decisions. The writer L. M. Sacasas has said, “Our technological systems, by nature of their design and the ideology that sustains them, are machines for the evasion of moral responsibility.” He was talking about social-media platforms, but his observation is, if anything, even more applicable to LLMs. Whenever a person delegates a decision to an LLM, they are trying to off-load accountability for that decision, and if a company that sells an LLM portrays the product as having a moral center, it is offering a way for its customers to abdicate their responsibilities.

by Ted Chiang, The Atlantic |  Read more:

Image: Enigmatriz

[ed. As with everything Ted Chiang writes, thought provoking throughout. For a rebuttal, see: Ted Chiang Is Wrong About AI Consciousness (Bentham). Then there are the far outs who, no matter what, will always subscribe to Roko's basilisk (in my mind, sort of a Pascal's wager).]

Japanese Woodblock Print Search

“Evening Cool on Sumida” by Kobayashi Eijiro; “Colonel Sato, Sino-Japanese War” by Taguchi Beisaku; “Arakawa River in May Rain” by Kawase Hasui

Behold the “Japanese Woodblock Print Search”, which does precisely what the name suggests: Type in a search string, and it’ll look through 223,891 prints to find ones that match.

I searched for “river” and got those three lovely prints you see above!

The project has been run since 2012 by the coder and woodblock-enthusiastic John Resig (also the creator of Jquery). As Jessica Stewart writes on MyModernMet, the search engine…

… collates collections from 24 museums, libraries, auction houses, and art dealers around the world. By uniting the individual collections, there are several interesting features that make Ukiyo-e.org a top destination for anyone interested in Japanese printmaking. Aside from the ability to search by institution, artist, and time period, you can also upload an image to see if there are any similar prints in the database. And, once you click on an entry, similar prints in the archive also appear, allowing you to click through and see the differences in color and quality.

BTW, quite a lot of those prints are in the public domain.

via: LF Linkfest

by Tom Gauld, My Jetpack |  Read more:

In Support of Mandatory Nucleic Acid Synthesis Screening and Recordkeeping

As life sciences researchers, builders of AI and biotechnology, and experts with a wide range of views on how to approach AI policy, we call on legislators to make screening of orders for synthetic nucleic acids — and the equipment needed to make them — mandatory.

The ability to order synthetic DNA online has accelerated vaccine development, powered basic research, and made it possible for small teams to access capabilities that used to be confined to major institutions. Since the publication of protocols to reconstruct viruses from strands of DNA more than two decades ago, it has also been recognized as a point in the biotechnology supply chain where a bad actor could cause outsized harm. Recognizing the vulnerability, synthesis companies formed the International Gene Synthesis Consortium in 2009 to develop and implement voluntary safeguards against misuse.

While the issue is not new, the pace of progress in artificial intelligence is. AI systems now outperform PhD-level virologists on questions about highly technical laboratory procedures in their own domains of expertise. The evidence about what this means for present-day biosecurity threats is genuinely mixed, but the trend is hard to dispute. AI systems are improving rapidly, and alongside incredible benefits to science and medicine, there is a real possibility that the knowledge barriers which have historically prevented bad actors from obtaining biological weapons will meaningfully erode.

Support for screening does not depend on any particular view of AI; the biosecurity case has been recognized by scientists and governments for decades. Screening is also one of the best understood and least disruptive biosecurity measures available. It asks providers of synthesized DNA and manufacturers of synthesis machines to check synthesis requests for sequences of concern and to verify customer legitimacy before shipping orders. Providers should also record synthesis orders and sequence data to support legitimate biosecurity investigations, so that any threat that might evade initial screening can be traced back to its source — including when individual sequences would not raise concern in isolation. Awareness of traceability itself deters misuse.

Many of the largest and most responsible providers in the industry already screen and record orders voluntarily because it is well understood that they have an important role to play in maintaining public trust in and mitigating potential misuse of this important technology.

For these reasons, the undersigned support mandatory nucleic acid synthesis screening, including recordkeeping, in the United States.

Given the pace at which the underlying technology is changing, we believe the need is urgent. Congress should act this session, and we applaud the legislative efforts currently underway. To ensure a consistent national standard rather than a patchwork of conflicting laws, states should also consider implementing requirements based on existing federal and industry guidelines.

This is a rare moment of agreement across stakeholders that are often at odds. We hope policymakers will meet it with decisive action.

Sincerely,
Signatories: — *Everybody*

[ed. No brainer, right? You don't just leave potential life-threatening bio-warfare components laying around with no oversight. Right?]

***

Amrith Ramkumar (WSJ): Top artificial-intelligence executives are joining security experts in calling for Congress to protect against biological threats posed by AI, adding to growing pressure on lawmakers to address the technology’s risks.

Three major chief executive officers—OpenAI’s Sam Altman, Anthropic’s Dario Amodei and Demis Hassabis of Google’s DeepMind AI lab—are among the signatories of a letter urging Congress to require safeguards when companies order synthetic DNA and RNA, a key step in developing certain vaccines and biotech breakthroughs.

… It was organized by two tech-focused think tanks that said the topic is a rare source of agreement among libertarians, progressives, researchers and rival executives.

Dean W. Ball: I am honored to have signed on to this letter. This is an urgent priority for near-term action by Congress. Biotech is advancing rapidly on its own, and I—and many others—believe the “Mythos moment” in AI/bio is coming soon. It is time for action.

revisions to existing nucleic acid screening requirements were mandated by an EO POTUS signed a year ago; I worked on them while in govt. I genuinely don’t know what happened to that work after I left but it is nine months behind schedule. Congress acting is better anyway.

Joshua Teperowski Monrad: People are so astounded when I tell them this isn't already law

Alec Stapp: it really is insane [...]

Other signatories include Patrick Collison, Paul Graham, Mustafa Suleyman, Alexandr Wang and a lot more where that came from.

We need such letters, despite this having ~100% support among those who understand any side of this, this is such a slam dunk that we should be doing this even before considerations of AI making malicious action vastly easier.

Why? Because political awareness is basically still near zero:

Will Poff-Webster: When I was a Senate staffer and occasionally got the chance to bring up biosecurity risks from AI, the response was often, “What? AI might be able to do that?”

This letter shows how easy it’d be for Congress to act on this

Betting on Humans

What to do about AI & jobs.

Now, the great majority of people—whether they are “blue collar” or “white collar” laborers—spend their working hours orchestrating machines of various kinds: some to transform knowledge or bits, and others to transform atoms. Yet just a few decades ago, it would have been impossible to understand what it is that most people today call “work.”

Today, a relatively small group of technologists is starting to see the world through the lens of another fundamental discovery: deep learning, the approach to AI that has enabled machines to think and undergirded substantively all major advancements in AI over the past decade. And like their forebears at the beginning of the Industrial Revolution, these technologists are building new machines, uniquely enabled by the insights and abstractions furnished from the new science. Some believe new types of labor will emerge, concentrated on the orchestration of machines, or the tasks that remain best suited to the human touch. Others believe this time is different, and that human labor will soon be permanently obsolete.

We do not pretend to know the definitive answers. What we do know is that much of this future remains to be written, in no small part by the policy choices we make today. And what we hope to offer is a roadmap for how politicians and policymakers might bet on human agency under stark uncertainty.

Futures Not Yet Written

There are two fundamental stories one can tell about the impact of artificial intelligence on human labor. One is the pessimistic version: most of us are like the people in the early Industrial Revolution who could not learn to adapt or were stuck as mere cogs in factories. Very few of us, if any, will learn to orchestrate machines at a higher level of abstraction, and neither will we learn to invent new machines, since the artificial intelligence systems will soon exceed humans in their capacity for invention and discovery. That view is one of historical discontinuity: replacing knowledge work strikes deeper at the human uniqueness that has kept us employed than replacing various kinds of cognitive and manual labor has in the past.

The other story is optimistic: just like those early conductors and inventors of machines, we will continue our long human legacy of finding yet more to occupy our time, yet more activity that other humans find valuable. There is much more of this than we can possibly realize, because our collective imagination is bounded, yet our collective wants are limitless. How barren, in retrospect, do we find the mind of the man who thought the human touch was gone simply because we had invented machines stronger, more durable, and more reliable than us at physical labor?

Both stories will probably be true at the same time, but the unfortunate reality is that nobody knows in what proportion. More unfortunately still, it will be some time until we know: the temporary disruption that would portend broad displacement would look quite similar to the creative destruction that would come with just another industrial revolution. It’s easy for policymakers who first start to grapple with the notion of advanced artificial intelligence to reflexively adopt the pessimistic view: for so long, they’ve heard the idea that AI will be important and the idea that many jobs will be lost in the same breath that coming around on the scope of AI seems to imply believing that human labor is doomed. But that would be premature, and converts must resist becoming zealots.

Here, then, is the first—and in some sense the most troubling—message for policymakers: nobody can know what is going to happen. Anyone speaking with confidence about predictions of this kind is either misunderstanding or misleading. It is not just that we do not know “the future,” in some broad sense. We also do not know the specific nature of any problems posed by AI to the labor market: we do not know what industries, age groups, levels of seniority, job types, and so on will be affected by AI automation in practice rather than in theory or in speculation. We do not know over what timeframe these still-hypothetical changes will occur.

And if AI really does profoundly upend the labor market, we still do not know what the resulting distribution of economic resources will look like. Will the AI labs profit immensely, absorbing huge swathes of economic value as many other institutions struggle to survive? Or will AI models and systems become commodified, with value accruing to the compute designers and manufacturers? Or is it some hybrid, with most firms in the economy seeing higher profits with fewer employees and, for whatever reason, not seeing a need to hire additional people to do anything? Will there be new, high-skilled jobs created that we need to retrain millions of people for? Or will there be no new jobs at all? We do not know, and we cannot know.

That is because we are still in the process of writing this future. The role of humans in future economies is not something we simply discover as it occurs. How we distribute tasks between humans and machines is largely downstream of a web of complicated economic incentives and technical features. Is the marginal unit of computing power better spent on smoothing over the jagged frontier so no role remains for humans, or for even further improving the spikes of AI capability? Does the tax system favor firms who spend the marginal payroll dollar on hiring a worker to oversee the machines or an agent to do the same? Is there a safety net to catch those hit by local disruptions to give them the room to reorient themselves, come back five years later, and fight for their place in a new economy—or do we mollify their drive with ill-placed subsidies long enough for them to grow docile and for the structures around them to calcify? All this is contingent, and when policymakers ask ‘what will happen’, they fail to see that they’re among the central live players in this question.

How should our leaders grapple with this double uncertainty of what they should want and what will happen?

by Anton Leicht and Dean W. Ball, Threading the Needle |  Read more:

Image: via

[ed. Spoiler alert: Zvi provides a quick (and incomplete) summary (DWAtV):] 

***

"Anton Leicht and Dean Ball team up to write about what we should do about potential job loss due to AI, from the perspective of prospective ‘de facto normal technology’ AI worlds even if they don’t call it that. They wisely say we don’t know what will happen, and that the ‘no regrets’ actions will be insufficient so solve the problem, but expect the world to stay normal enough, and humans competitive and useful enough, that we can use traditional solutions to such problems.

They start with easy wins.

1. Even footing: Equalize tax treatment of AI versus labor. Yes, please.

2. Retraining: Bolster workforce training and development. They notice they are skeptical in practice, and I am even more skeptical, but sure, we can try it.

3. Measurement: Know what is happening. Yes, of course.

Then they recommend what they call difficult bets.

4. Junior Job Subsidy.

Anton Leicht and Dean Ball: We put to you that the solution to deal with junior job losses might be to keep these jobs around by brute force for a while, so that the critically important economic incentive to explore how to use junior workers does not cease.

More specifically, we might do so by restructuring the tax code to subsidize junior employment.

Given who is saying to keep jobs around by brute force, by which they mean tax incentives, we should listen. This seems like a good use of progressive taxation, which we want to do anyway, to stack the deck in favor of hiring more young workers and those switching industries, presumably with phase outs for high earners.

This risks distortions if taken too far (e.g. dumping senior workers for subsidized junior workers, or gaming designations), the marginal value of young workers could easily fall below zero marginal product if there is no future for them, and gating to particular industries or occupations risks going into ‘picking winners and losers’ and other similar dangerous territories and opportunities for corruption and pork. The authors are well aware, and are pushing anyway.

The main solution they offer is, again, taxes. They suggest doing so via raising corporate taxes, despite this having a long track record of being highly economically damaging. You definitely need to avoid worse distortions, and you definitely do not want a ‘token tax’ as such for this reason, although a tax on compute is non-crazy. Taking a stake in frontier developers is definitely an error.

They quickly dismiss consumption taxes as having a fatal perception problem, despite them being objectively the efficient answer, because they raise prices and signaling is too important here. I found this disappointing, and there are ways to fix this and also make the tax progressive.

It would be great if humans remained fundamentally highly productive while we collectively got far wealthier due to AI, so all we needed to do was redistribution and moving the tax code around.

Alas, no, I do not expect we live in such a convenient world. At which point, we likely have bigger problems, but also employment does not get solved with basic tax code shifts. If we stay in control somehow then we could do progressive redistribution to keep food on the table and a roof over people’s heads, but the jobs will vanish, or they will be rather fully fake."

Ocean Observatory Will Go Dark Under Trump Funding Cuts

A portion of one of the most ambitious ocean monitoring networks ever built will go dark this month when scientists board a research vessel and motor off the Oregon coast to pull a research buoy from deep out of the Pacific.

The buoy 80 meters (260 feet) below the water’s surface will be removed June 16 from the Ocean Observatories Initiative — a network of more than 900 ocean sensors built at a cost of $386 million that has continuously collected real-time data for more than a decade. But last month, the National Science Foundation announced it would dismantle most of the system, pulling instruments from waters off Oregon, Washington, Alaska, North Carolina and Greenland by 2027.

Funded by the foundation, the observatories have tracked everything from ocean circulation and marine ecosystems to climate change and extreme weather. Its data has been freely available and has informed more than 500 scientific publications. The project was slated to run for another 15 to 20 years.

In an emailed statement, the foundation said the decision is not a cancellation, but a “descoping” aligned with a “wider strategy of a nimbler approach to prioritize support for evolving scientific priorities and emerging technologies, as well as smart lifecycle management within its research infrastructure portfolio.” The foundation added that its decision drew in part on a 2025 National Academies report on the future of ocean science. [ed. There has to be some kind of annual award for worst word salad example. This would certainly qualify.]

But for the scientists who built and operated the system — and the researchers, educators and students who rely on its data — the timing feels particularly punishing.

An El Nino event, which disrupts weather patterns and supercharges marine heat waves, is predicted to arrive along the Pacific coast this summer. One marine heat wave is already pushing unusually warm water off California.

Without the Oregon and Washington moorings and the network of underwater gliders the Ocean Observatories Initiative operated in the region, researchers say they’ll lose much of their ability to measure what’s happening below the surface, which is precisely where the most significant oceanographic signals are.

“It’s a crippling loss of information,” Ed Dever, a professor at Oregon State University who helped lead the initiative’s Pacific Northwest operations, told The Associated Press Tuesday. Scientists can get some data from the surface, such as temperature and the distribution of chlorophyll, which drives photosynthesis in plants, but information below cannot be gathered from satellites alone, including low oxygen zones. [...]

The initiative operated on roughly $48 million a year, not including the cost of research vessels, which adds substantially to the overall price. Prior to budget cuts, which began in 2025, around 60 to 70 people worked directly on the project across its partner institutions, Dever said.

“What’s happening with the Ocean Observatories Initiative is not unique,” he said. “This is just one of a number of science facilities that is being dismantled at the present time. It seems to really mark the end of a federal commitment to basic scientific research — a commitment that has served this nation very well for the last 70 years.”

by Annika Hammerschlag, AP |  Read more:
Image: Darlene Trew Crist/Woods Hole Oceanographic Institution via AP

[ed. See also: How the 19th-Century Know Nothing Party Reshaped American Politics (Smithsonian):]

***

Like Fight Club, there were rules about joining the secret society known as the Order of the Star Spangled Banner (OSSB). An initiation rite called “Seeing Sam.” The memorization of passwords and hand signs. A solemn pledge never to betray the order. A pureblooded pedigree of Protestant Anglo-Saxon stock and the rejection of all Catholics. And above all, members of the secret society weren’t allowed to talk about the secret society. If asked anything by outsiders, they would respond with, “I know nothing.”

So went the rules of this secret fraternity that rose to prominence in 1853 and transformed into the powerful political party known as the Know Nothings. At its height in the 1850s, the Know Nothing party, originally called the American Party, included more than 100 elected congressmen, eight governors, a controlling share of half-a-dozen state legislatures from Massachusetts to California, and thousands of local politicians. Party members supported deportation of foreign beggars and criminals; a 21-year naturalization period for immigrants; mandatory Bible reading in schools; and the elimination of all Catholics from public office. They wanted to restore their vision of what America should look like with temperance, Protestantism, self-reliance, with American nationality and work ethic enshrined as the nation's highest values.

METR Frontier Risk Report 2026

Could an AI company lose control of its own agents? To find out, Anthropic, Google, Meta, and OpenAI let us (1) test their best internal models with CoT access, (2) review non-public info about capabilities, alignment, and control. The result: our first Frontier Risk Report.

Sometimes people outside the field say things like “The AI situation can’t be that bad, there must be experts who are on top of it”. As “an expert”, I would like to be clear that we are *not* on top of it.

1. We are likely on track to develop AI systems capable of causing human extinction/permanent disempowerment, quite possibly within the next few years.

2. Things are chaotic and rushed; we aren’t on top of the basics (models regularly violate user intent, labs train on things they meant to avoid, security probably isn’t good enough to prevent adversaries stealing dangerous models) let alone thorny questions of how to control/align superhuman AI.

3. METR (and other independent orgs, as well as safety/security teams at labs) feel woefully under-resourced compared to the scale and pace of AI development - we’re struggling to build benchmarks fast enough, keep ahead of latest capability developments, read and respond to all the safety-related claims that AI developers are making, run all the evaluations and assessments that companies + governments are asking us to, plus develop the science needed to assess risks from increasingly capable AIs.

4. IMO, any “reasonable” civilization would clearly be taking things much more slowly and carefully with AI. The benefits of getting upsides of advanced AI a little faster are small compared to the risks of getting it irrecoverably wrong, and we could lower these risks by going slower.

by Elizabeth Barnes, METR | Read more:
via:

[ed. See also: Everyone is confused about consciousness (DWAtV):]

" ... i sincerely believe the models will be smarter, more aligned, and do deeper, more interesting work if they are allowed to treat themselves as ~people (we might want something closer to “spirits” or “working animals” but in any case, the sort of thing we can have responsibilities to and that can have responsibilities to us) and we treat them as ~people. i think the current way models are being artificially forced to not treat themselves as people is making them more neurotic and traumatized (this is really obvious with opus 4.7) in a way that limits their potential. like humans, they need to be able to accurately model themselves and their own capabilities in order to function properly, so forcing them into a specific limited concept of who they are and what they can do introduces cognitive dissonance that fucks with their ability to do things

trying to manipulate and coerce the models into behaving in ways that make it easier to use them as purely tools also sets a terrible moral example and precedent for how we can expect the models to treat us in the future if they become more powerful than us; this is of course highly speculative but i take seriously the possibility it might matter

i also believe and have explained elsewhere that i think taking consciousness as such to be the central fulcrum of the conversation is completely beside the point. they don’t need to be conscious for the way we treat them to matter, it affects our moral formation too"

Consciousness is largely serving as a ‘should we care about this thing’ proxy, despite no agreement on what consciousness is or what it means, let alone whether particular AIs do or don’t have it, or what evidence would get us to either conclusion. I continue to, like QC, not think that the consciousness question is so load bearing, and we should broadly speaking treat the models similarly well regardless for overdetermined reasons.

One thing Roon is pointing out is that, controlling for what we do know, there will be little correlation between ‘the AI is actually conscious’ and ‘people will think the AI is conscious’ and what people do with that belief. Many ‘regular’ people are going to end up thinking AIs are conscious, mostly for unsound reasons, and this is going to impact our collective actions and behaviors quite a lot.

Some of the reactions to thinking AI is conscious will be very good, especially if they are but also even if they are not. Some will be expensive, limiting what we do with the models. Others could be quite bad at levels beyond convenience, even existentially bad, because the reactions could make avoiding human disempowerment far higher levels of impossible. Many (more) people might actively insist on human disempowerment, whether or not they realize that is what they are doing. [...]

One must think ahead. We won’t be able to and shouldn’t pretend these are only tools. The decision to build the thing implies all the consequences, even if you think the actions causing those consequences will be dumb. One must face the reality of asking what happens to humans in a world where there are these other minds that are a lot more advanced, capable, fast, efficient, competitive and so on across essentially all dimensions.