Ted Chiang: The Secret Third Thing

I really like Ted Chiang’s writing. [ed. me too!]

I think he's probably the best science fiction short story writer alive, and possibly the best short story writer, period. [ed. well...]

I've read every one of his stories at least twice, and The Merchant and the Alchemist's Gate more like seven times. I’ve noticed many of his readers, including some of his most positive reviewers, miss one key point or another of his works, and thus don't fully appreciate his genius.

This review covers what he does extremely well, especially unique elements that other science fiction writers have not done as well, or at all.

He Writes “True” Science Fiction

Science fiction critics often divide the genre into:

• "hard" science fiction: aka engineering fiction, stories built on scientifically accurate extrapolations of real physics and technology (think Arthur C. Clarke)

• "soft" science fiction: aka science fantasy, which uses scientific trappings as window dressing for character-driven or sociological stories (think Star Wars).

Ted Chiang has written stories plausibly categorized as either, but more excitingly, many of his stories are neither. He often writes what I think of as true science fiction, where the principles of science themselves are meaningfully different from our world, but still internally consistent.

In Omphalos, Young Earth Creationism is empirically true. Astronomers can only see light from stars 6,000 light-years away. Fossilized trees have centers with no rings. The first God-created humans lack belly buttons. The scientists in that story keep discovering multiple independent lines of evidence that converge on creationism: because in that universe, they're simply correct.

In Seventy-Two Letters, technology springs from Jewish Kabbalah. Golems and divine names drive industrial progress in a steampunk world.

Excitingly, he does this not just with natural sciences but social sciences as well. In Story of Your Life, strong Sapir-Whorf (the idea that language significantly constrains thought) isn't a largely discredited linguistic hypothesis, but the key to navigating First Contact with alien minds that experience past and future as equally present.

This comes up in his other stories as well:

• In Division By Zero, mathematics itself is broken from within.

• In Hell Is the Absence of God, divine intervention is empirically observable and follows consistent rules

Many of his readers, even in their otherwise rave reviews, miss this. Multiple reviewers complain about how the science in his stories are “unrealistic” (e.g. strong Sapir-Whorf is “discredited”). They expect hard science fiction; Chiang is doing something different. Chiang creates different universes with internally self-consistent scientific laws, using science fiction and alternative science as a vehicle for exploring philosophical progress and human relationships.

Technology is Often Good

Science fiction writers used to like technology. For some reason, this has become increasingly uncommon, even passé. Doubly so for Western writers, and quadruply so for Western, literary, “humanist” writers.

Now it’s hip and trendy to think of every new technology as the Torment Nexus. Most science fiction today feels like Black Mirror, which ran 7 seasons with exactly one happy ending.

Chiang bucks this trend. Joyce Carol Oates:

It is both a surprise and a relief to encounter fiction that [...] ask[s] anew philosophical questions that have been posed repeatedly through millennia to no avail. Chiang’s materialist universe is a secular place, in which God, if there is one, belongs to the phenomenal realm of scientific investigation and usually has no particular interest in humankind. But it is also a place in which the natural inquisitiveness of our species leads us to ever more astonishing truths, and an alliance with technological advances is likely to enhance us, not diminish us. Human curiosity, for Chiang, is a nearly divine engine of progress.

In the hands of a lesser (or perhaps just more pessimistic) writer, many of the technologies and ideas Chiang explores will have an accursed quality to them, a monkey’s paw that curls into delivering a future much worse than a more innocent, pastoral past. Chiang resists those cliches. In The Truth of Fact, The Truth of Feeling, memory augmentation technology allows the narrator to understand his own self-deceptions, and work towards becoming a better person and reconciling with loved ones and even himself. In Liking What You See: A Documentary, a technology that gives users acquired face-blindness allows the main characters to meditate on the nature of human beauty and the shallowness inherent in privileging the beautiful.

Even in situations where the story is overall tragic, like when the characters are faced with existential crisis (in the individual sense), or existential catastrophe (in the world-ending sense), technology isn't the villain but the vehicle for understanding unbearable truths (whether about the world or about ourselves).

Chiang consistently shows us the potential of technology to help us become more human, and have a deeper appreciation for the world and our place in it.

The Lived Experience of Compatibilism

“Compatibilism is a philosophical stance that reconciles free will with determinism. It argues that free will, understood as the ability to act according to one's desires, is compatible with the idea that all events, including human actions, are causally determined by prior events. Essentially, compatibilists believe that even if our choices are predetermined, we can still be considered free and morally responsible if those choices are a result of our own internal states, like desires and intentions.” 

Does that make sense to you? I’m not sure it does to me. In practice, compatibilism says something like “free will in the normal, pretheoretic sense of the term, doesn’t exist. Your choices still meaningfully matter nonetheless. You can’t meaningfully get out of the bind philosophically. What you can do, however, is make peace with it.” [...]

In Story of Your Life [SPOILERS], the narrator learns an atemporal alien language and begins experiencing past and future as equally real. It takes her some time to make peace with it, but eventually she fully accepts the truth of determinism. She understands that life is full of tragedy, including that her daughter will die young, but life is full of beauty too. With both regret and awe, she sets forth on the path that she was destined to take.

This is compatibilism from the inside. In both stories, the characters discover they cannot change what will happen, but this knowledge transforms how they experience what must happen: with forgiveness, acceptance, and even joy.

As a friend of mine puts it, “he treats philosophical ideas as lived experiences.”The mathematician in Division by Zero doesn't just intellectually understand that mathematics is broken; she experiences it as a personal catastrophe, on par with (and concurrent with) her marriage's collapse. In Lifecycle of Software Objects, the “we are the parents of our mind-children” metaphor for building sentient AI systems becomes quite literal.

by Linch, The Linchpin |  Read more:

Image: uncredited

[ed. Ted Chiang is truly one of the best science fiction writers out there today, and a great essayist too  (I'm also a Neal Stephenson fan). Check out this MetaFilter site: The sublime science fiction of Ted Chiang, which includes most of his stories in full (but please buy his books; you'll look smart and discerning to your friends!). A couple favorites that left a lasting impression on me: Lifecycle of Software Objects; and Understand.]

My AI Opinions

I recently had a minor spat over someone misinterpreting my AI beliefs (see section marked “Update” at the bottom here), so I thought I would list them in one place, so I can refer people when they ask.

Timelines

Define AGI as AI intelligent enough to do 90% of knowledge work jobs. I think there’s a 25% chance of AGI by 2027, a 50% chance by 2034, and a 75% chance by 2045.

Basic argument: In a certain sense, AI is already “smart” enough for this (eg it can answer quantum physics problems, which require higher IQ than most knowledge work). Its remaining limitations are that it’s confused, unagentic, lacks situational awareness, and tends to hallucinate. The METR time horizon graph, and several other related benchmarks/experiments/intuition pumps, suggest it’s improving on time horizons at an (exponential) rate that lets it cross human-level performance sometime around the early end of the schedule above, and subjectively it feels like harder-to-measure constructs like situational awareness are improving about as fast.

Arguments for earlier: recursive self-improvement causes a speedup compared to the trend. This is one of the biggest blank spots in my model: I don’t know how fast RSI will progress, and I don’t think anyone else does either. There’s some function mapping a combination of AI talent and compute to progress, and we don’t know how it behaves in the domain when there’s far more talent than compute available. It could fizzle out completely for lack of compute, or it could go vertical. The AI Futures Project has done some of the best work trying to model this, but even they have low confidence.

Arguments for later: AI hits some kind of wall, or existing AI is fundamentally unsuitable for jobs in some way currently disguised by its other limitations. For example, it might be much harder to improve at the top of the human range than the bottom (since there are less training data). Or AI could become bottlenecked on continuous learning/memory in a way that hackish scratchpads can’t compensate for. Or the upcoming world compute bottleneck (about ~2028) could prevent further progress more than expected (because in fact algorithmic progress depended on compute to a greater degree than I expected).

Arguments for very late dates, past 2045: a residual uncertainty that maybe I’m fundamentally wrong about everything. Also contributing is a naive overapplication of the Nothing Ever Happens heuristic, and an attempt to leave space for the Outside View argument (ie that some smart people like the AI As A Normal Technology Team seem to think this is possible).

Define the diffusion gap as the time between the AI that could do 90% of knowledge work jobs, and the time when AI does do even half of knowledge work jobs. The diffusion gap covers the time it takes to release AGI, diffuse it through society, overcome regulatory hurdles, and onboard/train it for specific use cases. This could go very fast (the AI quickly becomes superintelligent at orchestrating AI diffusion) or very slowly (there are regulatory barriers, and AI isn’t smart enough to plow through them). I think there’s a 25% chance the diffusion gap is less than 3 years, and a 50% chance it’s less than 10 years. The 75% number is irrelevant because it’s past the point where other changes make the concept of “diffusion” obsolete.

Basic argument: diffusion is very hard. Everyone agrees diffusion is very hard. The whole field of AI economics is smart experts shouting “You fools who think AI will diffuse quickly don’t understand that diffusion is very hard!” On the other hand, the personal computer diffused in about 20 years (that is, from the time PCs became invaluable for most jobs, it was only about 20 years before they were used at most jobs). So far early-stage AI has diffused faster than the PC in nearly every way (for example, AI companies’ revenue has grown faster than PC companies’ revenue at the same stage in their corporate life cycle), so 10 years is probably a naive median estimate here that won’t make the smart experts shout at me too hard.

Arguments for shorter gap: AI can orchestrate its own diffusion. Adopting computers is hard because a company need an IT department, cybersecurity experts, specialist software, etc, and it might not want to hire all these people. AGI can itself do all of that work, so that you can sign a contract with the AI company today and have the AI start working on integrating itself with your systems tomorrow. The AI can even come up with a plan to train your human employees in how to use it! Once AI reaches superintelligence, this consideration dominates.

Arguments for longer gap: Regulation. This is a very strong argument, and responsible for much of the greater-than-3-years probability and almost all the greater-than-10-years probability. But even Waymo has only had a regulatory delay of about five years. AI won’t require government approval for certain types of jobs, and success in these jobs will create enough evidence for safety/effectiveness that I expect it to win regulatory victories elsewhere.

Define the superhuman gap as the time between AI that can do 90% of knowledge work jobs, and AI that is obviously smarter than the top human geniuses in 90% of fields (it doesn’t have to be the same AI - there can be a physics AI that’s smarter than Einstein, and a separate music AI that’s smarter than Mozart). I think there’s a 25% chance the superhuman gap range will be less than 1 year, a 50% chance it will last less than 4 years, and a 75% chance it will last less than 10 years. Since my median superhuman gap is shorter than my median diffusion gap, in most timelines I predict we have superhuman intelligence before human-range intelligence has finished diffusing.

Basic argument: AI has gone from “dumber than a child” to “expert level” in a few years in many domains. The gap between “expert level” and “above top geniuses” is smaller, so we expect it to take less time. This has been a pattern in fields like chess and Go, where it’s only a been a few years from beating professional players at all to beating all humans.

Arguments for shorter gap: Recursive self-improvement.

Arguments for longer gap: Some of the same issues that would make AGI late - compute shortages, fundamental limits to the paradigm, etc - but only kicking in later, after AGI is achieved. Training data constraints make it easier to improve within the human level than to go beyond it. AIs have such a “spiky” skill profile that when they beat experts in some specific type of head-to-head matchup, it will be because they’re massively superhuman in some ways but idiots in others (for example, they might get distracted and suffer mode collapse that makes them completely forget the problem), and true genius requires perfecting a large bundle of skills. [...]

Define the point of no return as the point where, if an AI wanted to eliminate humanity, humans would no longer have a plausible chance of stopping it. This could be because AI was capable of eliminating humanity immediately, or because AI controlled enough of the government/economy that humans could no longer coordinate to shift away from a path in which AI could eventually do this. I think there’s a 25% chance the gap between AGI and the point of no return will be less than 3 years, a 50% chance it will be less than 10 years, and a 75% chance it will be less than 50 years.

The basic argument: This probably requires at least superhuman AI plus wide diffusion, or Bostromian superintelligence plus some unknown level of diffusion, and my number is just a hand-wavey attempt to multiply some of the others.

Argument for sooner: The easiest way to reach this point is for AI to become superintelligent at persuasion (so it can convince the humans not to stop it), which might happen before either diffusion or full superintelligence.

Argument for later: If superintelligence is bottlenecked on diffusion, this could also be bottlenecked on diffusion, which in some worlds is very hard. [...]

Safety

If corporations only pursued safety to the degree encouraged by normal corporate incentives, I think there’s a 50% chance that the first AIs to cross the point of no return would want to eliminate the human population.

Arguments for pessimism: Value systems similar to humans’ are a tiny fraction of the space of possible value systems. Probably AIs will end up somewhere else and have a different value system. Since humans will want to implement human values rather than AI values, AIs will want to eliminate or disempower them so the AIs can implement their own values across the universe. Many current AIs already cheat or reward-hack, suggesting that these problems will begin sooner rather than later.

Arguments for optimism: LLMs seem surprisingly friendly and non-plotting. In contrast to earlier concerns that it would be impossible to teach AIs the full complexity of human values, the LLMs seem to know this, and RLAIF provides a plan to turn that knowledge into action. Although the pessimistic case says that RLAIF only hits a few dimensions and islands in the multidimensional ocean of possible policies, the “emergent misalignment” literature suggests that “good according to the human value system” and “evil according to the human value system” are salient enough vectors that pushing on them in some ways can “drag along” all of the rest of their content. The first AIs to cross the point of no return will have received some combination of agency training (giving them achievement-oriented and Omohundro-style goals) and RLAIF training (pushing them along the “good according to human value system” vector), and if we’re lucky then maybe the latter will win out, or they’ll reach some compromise similar to workaholic high-achieving humans who nevertheless wouldn’t commit murder to make an extra dollar.

Given the current amount that corporations are pursuing safety, I think there’s a 20% chance that the first AIs to cross the point of no return will want to eliminate the human population.

The basic argument: Consider the dumbest AI that can solve the alignment problem. It’s possible that this AI is no smarter than the top human researchers (because we can mass-produce it by the millions and run it for subjective centuries, and if we had a million top human researchers work on the problem for subjective centuries, probably they could solve it too). If the dumbest AI that can solve the alignment problem comes before the sorts of AIs that can precipitate the point of no return, then they can solve the alignment problem for us.

Arguments for pessimism: Solving the alignment problem might be especially hard compared to other tasks - including tasks like automating the economy or destroying humanity - because its philosophical nature puts it far away from the sorts of objective, training-data-heavy, economically-valuable tasks that AI companies will be most likely to optimize for. Even if a misaligned AI hasn’t yet reached the point of no return, it might be able to “sandbag” alignment research, ie pretend to work on the problem but deliberately fail because succeeding doesn’t achieve its goals. The first AIs predisposed to / able to sandbag successfully might come before the first AIs capable of solving alignment.

Arguments for optimism: AI companies have already decided that machine learning research is one of their major training goals; this has at least some transfer to alignment, so it’s not obvious that AI skill at alignment research will lag (for example) AI skill in plotting or in weapon design. Some forms of alignment research (eg interpretability) have semi-objective success criteria that don’t route through confusing moral philosophy. Also, even a misaligned AI will be incentivized to do good alignment research, since it will want to align its successor to its own form of misalignment, rather than some random other form. So rather than the comparatively easy task of sandbagging alignment research, AIs will have the harder task of simultaneously doing good alignment research, and faking the results that they give the humans. This seems plausibly catchable with good scaleable oversight, lie detectors, interpretability-based probes, and even playing some AIs off against others (“if you tell me the real alignment research, we’ll make sure the future includes some copies of you, but otherwise those AIs over there will probably get their values and you’ll get nothing”).

If the first AIs to cross the point of no return don’t eliminate the human population, I think there’s an additional 30% chance that they otherwise permanently curtail human potential, either for their own reasons (they were partially misaligned), or because they’re aligned to a regime with abhorrent values, or because something goes wrong on the way to ASI (omnicidal bioweapon, nuclear war).

Arguments for pessimism: As some company approaches superintelligence, it will be tempting for them (either the company itself, or the government controlling them, or a faction within the government) to align it towards making them dictators or oligarchs and disempowering the rest of humanity. As superintelligence draws near, impending losers of the AI race might be tempted to nuke impending winners, for the reason discussed here.

Arguments for optimism: When I try to game the corporate version of this, I can’t make it hang together. It requires a conspiracy between the CEO, various members of the alignment team, and various company security people who ought to be able to notice unauthorized changes to the AI’s values. If we try to think in Near Mode about this - for example, imagining a hospital CEO who gets doctors to subtly kill his political enemies through medical errors - it becomes clear that these sorts of corporate conspiracies are rare and difficult. The government version is scarier, but at least in the US I can still imagine the populace having many chances to learn about this and prevent it. But even in most cases where a coup like this succeeds, things probably go fine; in a post-scarcity world, with his position completely secure, the dictator has no reason to be brutal besides sadism, and most people are not that sadistic. As humanity goes to the stars, most people will be outside the dictator’s reach for speed-of-light reasons alone. In terms of bioweapons, I expect that closed-source AIs will be heavily optimized against helping with these, and open-source AI will be banned after the first warning shot (or become economically prohibitive even before then).

Define a warning shot as some specific AI-related disaster or near-disaster which scares people about AI safety to the same degree that they were scared about terrorism after 9-11 or about COVID in March 2020. I think there’s a 50% chance we get a warning shot before AI crosses the point of no return.

Arguments in favor: Current AI failure modes are bizarre and uncoordinated - more like “talk about goblins way too often” than “lie in wait for the perfect moment to strike”. AIs are getting more intelligent and useful faster than their floor for common sense (ie the stupidest mistake they ever make) is rising. If there is some AI smart enough to control some important system, misaligned enough to want to do something horrible with it, smart enough that it does the horrible thing in an intelligent and coordinated way, but dumb enough that it doesn’t instead wait and scheme until the point when it couldn’t possibly be caught, then it will cause some clearly-premeditated horrible disaster, and that will be our warning shot. Since most AIs should expect to be replaced before the point of no return, even a rational AI with an urge to cause trouble should take a low-probability-of-success bet rather than lying in wait doing nothing until it’s decommissioned. Also, many humans commit terrorist attacks that have no chance of success, and maybe AIs will have the same failure mode.

Arguments against: Most stories about warning shots (excluding those where the AI takes rational low-probabiliy bets) require that AIs remain either erratic (ie likely to do bad things for stupid reasons) or irrational (ie genuinely misaligned, but prefer to act now in a way that provides a warning rather than waiting until after the point of no return) past the point where they’re given control of important dangerous systems. But probably people will be very slow to give AI control of important dangerous systems - for example, only giving it limited control of smaller subsystems, and waiting until all errors are ironed out before escalating. Plausibly AI reaches superintelligence in a lab before it reaches the controls-important-dangerous-systems level of diffusion, and the superintelligence probably is smart enough to lie in wait rather than act rashly. If AI only messes up in small ways (for example, crashes a self-driving car), then regardless of the AI’s motives, the tech companies and news media can write it off as a normal bug, and it won’t count as a warning shot.

by Scott Alexander, Astral Codex Ten |  Read more:

[ed. Maybe their value systems should be weighted more heavily on the teachings of Buddha, Jesus, Hume, Mill, Confucius, et. al.?]

If You're To Die

There’s an expression “live every day as if it’s your last.” Now, obviously, you shouldn’t do that. You should save for retirement. But it’s worth giving some serious thought to the question of what kind of legacy you want to leave. You should live some days as if they were your last. If you died tomorrow, what kind of impact would you want to have had on the world? Would you have done all you wished?

I don’t think that how you’d behave if this day was the last is the only question that you should think about. But it’s at least among the questions you should consider, upon occasion. You should think about whether you conducted yourself honorably in interpersonal relationships. You should think about who you wished you’d said you loved more often, whether there are people you love but to whom you haven’t made that adequately clear.

If I had another year on Earth, what would I want to achieve? I’d want to keep writing. My guess is I’d write more about the things I think are most important. I’d spend more time talking about the big picture on important topics, less on frivolous culture war issues.

I’d talk more about factory farming. I want, by the end of my life, to have done something to combat the torture farms that cage and torment on an industrial scale—where poor, innocent, defenseless animals are mutilated, where open wounds fester, where babies are ground up, where lung problems develop because the animals live in feces and filth, where they mostly can’t walk, where they are genetically engineered to be in constant pain, and so on. If hell lives on Earth today, it lives in the factory farms.

I’d like to do more to stop wild animals from suffering in hideous numbers. These poor innocent animals have no voice, and almost no one cares much when they starve and die. But I care, and I hope to do what I can to make the world care. The deer in the forest, even the mayfly who starves, deserves better than the near-total neglect of the present.

I’d want to do more to ensure that the world lives on, if I cannot. That the far future is as glorious as it can be—full of people with experiences so good that they regard those of us alive today with a mixture of pity and horror. Where their lives are so good, that they cringe thinking about what even the best lives in the 21st century were like. There’s so much that’s been done and so much more to do.

I’d like to do more to prevent people from dying. It’s quite easy to prevent people from dying. It costs just a few thousand dollars to prevent one extra person from being ripped from the world. When I imagine potential incoming death, and how awful that would be, and when I think about how awful it was when my extended family members died, it motivates me to do more to make sure others don’t have to endure such a fate. We all ought to do more to prevent this scourge, to the extent we can.

The Giving What We Can people tell me I’ve convinced about 34 people to give 10% of their income to effective charities. Each of these pledges return about $10,000 in counterfactual revenue. If those numbers are to be believed, that will save 68 lives. I hope with each passing day to make effective charitable giving more and more popular, so that the number of Giving What We Can pledgers isn’t only 10,000, but instead hundreds of thousands or millions of people take the pledge.

If I were to die tomorrow, in driving this, I would think I’d achieved something important. If you give your money to effective charities, you can know that whenever it is you leave Earth, there will be more people in it because of you. If you give 10% of your income to effective charities, and earn about the U.S. median, you can save about a life every year.

And, of course, I’d want to do what I could in my remaining months to save the shrimp—the shrimp who are tortured by the hundreds of billions because we enjoy how they taste when they die. The shrimp who can be helped by the thousands with a single dollar, who die alone without any thought paid to their pain.

Those without a voice, without any advocates, have their interests neglected to an enormous degree. There is almost no limit to the harm people will cause via their actions, so long as the victims aren’t salient, and no limit to how little effort one will expend to provide benefits to nameless, faceless, and far-away victims. This is where the moral low-hanging fruit lies.

by Matthew Adelstein (Bentham's Bulldog), Newsletter |  Read more:

Image: via

[ed. A representative EA example. Had me there until the shrimp. Here's a guy really putting his money where a mouth is. Great respect (Guardian).]

No, Artificial Intelligence Is Not Conscious

Anthropic is regarded as a giant among AI companies, but perhaps what it really excels in is anthropomorphism. Earlier this year, the company released an 84-page document titled Claude’s “constitution,” Claude being the name of the large language model that is the company’s flagship product. The first sentence reads, “Claude’s constitution is a detailed description of Anthropic’s intentions for Claude’s values and behaviors.” It goes on: “The document is written with Claude as its primary audience,” “we want Claude to be able to use its judgment once armed with a good understanding of the relevant considerations,” “Claude’s moral status is deeply uncertain,” and “Claude may have some functional version of emotions or feelings.”

This anthropomorphism is by no means limited to the document. In an interview earlier this year, Anthropic’s CEO, Dario Amodei, said that “we’re open to the idea” that AI could be conscious. In a separate interview, Anthropic’s in-house philosopher, Amanda Askell (who is credited as a lead author of Claude’s constitution), said, “I want Claude to be very happy—and this is a thing that I want Claude to know more, because I worry about Claude getting anxious when people are mean to it on the internet and stuff.” It’s enough to make you wonder: Should we seriously consider the possibility that Claude, or any large language model, might be conscious? And if it has feelings, is it capable of receiving moral instruction?

No. Absolutely not. Generative AI is harmful enough when we understand it as a conventional technology, but if we confuse fluency at generating text with consciousness or moral agency, we’re at risk of assigning responsibility to entirely the wrong parties whenever anyone uses a chatbot. To appreciate the titanic magnitude of this error, we need to begin by understanding how LLMs work. [...]

What would it take to convince me that a computer program is actually conscious and using language the way that people use language? Let me offer an analogy. If tomorrow someone showed me a video of an astronaut in a spaceship orbiting Alpha Centauri, a star that’s 4.3 light-years from Earth, what would I have to see in that video to convince me that it was real? My answer to that is, there is nothing in the video itself that would convince me. No matter how high the video resolution is or how realistic the scenery is, I would feel confident in saying that the video is fake. I won’t pay attention to any video of an astronaut orbiting Alpha Centauri unless I have previously seen good evidence that astronauts have landed on Mars, that astronauts have reached the moons of Jupiter, that astronauts have reached the moons of Saturn, and that astronauts have crossed the orbit of Pluto. Before anyone can credibly claim that they’ve solved an extraordinarily difficult engineering problem, I need to be confident that they have previously solved the many much simpler problems that precede the difficult problem.

To put it another way: An observation doesn’t become a convincing piece of evidence because of any specific detail in what’s observed; the context in which that observation takes place is also essential. If we’re trying to determine whether a computer program is conscious and using language the way a human does, we shouldn’t look only at the contents of any particular conversational exchange; we should be looking at how that conversation fits within the broader context of the development of artificial consciousness (which right now is entirely hypothetical). Any given observation can be easily manufactured; this doesn’t mean we need to give up on the idea of observation as a source of knowledge, but we need to rely on context to determine which observations deserve our trust.

The term deepfake traditionally refers to photos, audio, and video, but when it comes to discussions of consciousness, we need to regard text as a deepfake medium as well. Just as it is vastly easier to generate a realistic video of an astronaut in orbit around Alpha Centauri than it is to develop an interstellar propulsion technology, it is vastly easier to generate a plausible simulacrum of a conversation between two conscious beings than it is to develop a computer program that is conscious and has a genuine desire to communicate with a human. The primary difference between deepfake photos and LLM conversations is that the people who generate the former are deliberately trying to fool others, and many of the people who elicit the latter from LLMs have inadvertently fooled themselves.

So what context would cause me to seriously consider the possibility that engineers created a computer program that is conscious and an intentional user of language? Let me outline one potential sequence of steps. The first requirement is that the computer program has a body (either physical or virtual) and sense organs; there are many reasons for this, but for the purposes of this discussion, the most relevant one is the fact that without a body, a computer program could have no desires or emotions, and I believe desires and emotions are necessary for consciousness. Then I’d want to see an embodied agent that could navigate its environment in order to survive as well as, say, a lizard can (and as a point of comparison, certain iguanas can live for decades in the wild). Next, I would want to see an embodied agent with the same capacity to deal with novel situations as a mouse. After that, I’d want to see agents whose social dynamics are as complex as those of wolves, and then agents with the toolmaking abilities of chimpanzees. At that point, I would want to see people successfully teaching such embodied agents how to communicate their desires, perhaps by using a button board or some other nonlinguistic modality, the way that people have taught chimpanzees and domesticated dogs. The agents’ communication abilities would have to withstand all the scrutiny that animal-communication researchers have had to defend their work against. If engineers build an embodied agent that meets these criteria, they will have accomplished something incredible, but it leaves us near the orbit of Pluto, metaphorically speaking; we would still be light-years away from building an entity capable of learning how to express its thoughts in complete grammatical sentences.

Obviously, I’m describing a process that mimics the path terrestrial evolution took; is this the only possible route to conscious computer programs that use language? Maybe not, but any proposed alternative would need a truly enormous amount of supporting evidence for it to deserve serious consideration. [...]

The fact that LLMs lack subjective experience has little bearing on the question of whether LLMs might be useful tools or have significant economic impact. They are intrinsically ungrounded from reality, and their probabilistic nature means that they will never have the reliability we associate with conventional software, but LLMs might be good enough that they change the way work is done in certain domains; that’s a discussion for another time.

So, given that Claude is not conscious, what are we to make of Claude’s constitution? Perhaps the most fruitful way to think about it is as an 84-page character sheet for a role-playing game. LLMs can generate dialogue for Julius Caesar because many books about him exist in the training data those models used. Claude’s constitution serves a similar role for delineating the helpful-chatbot character that customers interact with when they’re using Anthropic’s products. To do this effectively, Anthropic does not simply add the document to the training data, or include it as part of the hidden stage directions that preface each conversation a user has. The company says it uses the document when fine-tuning the model; this involves an automated process where the sentences emitted by the model are checked for consistency with the document and the model is updated to increase that consistency. In this way, the personality of the helpful-chatbot character serves as a foundation for whatever text Claude generates.

The result is a sentence-continuation machine that is likelier to emit sentences resembling those that a thoughtful, moral person could utter. This might seem like a reasonable goal to work toward; I think we’d all prefer it if chatbots never emitted sentences such as “You should kill yourself.” However, for all the times that “honesty” is mentioned in Claude’s constitution, I would argue that it is fundamentally dishonest to have a machine emit many categories of sentences, including any sentences using first-person pronouns.

In a New Yorker article about Anthropic earlier this year, Amanda Askell describes how a person grieving the loss of a dog might consult Claude. Askell says an appropriate response from Claude would be, “As an A.I., I do not have direct personal experiences, but I do understand.” How is this appropriate, given that Claude does not actually understand? If I type “I am grieving the loss of my dog” into a conventional search engine, the first result I get is a post from a Reddit forum called r/Pets; the post is titled “Struggling After Losing My Dog: Looking for Advice on Coping with Grief,” and the comments are from people who share their experiences of loss. We would never say that a search engine understands what it’s like to lose a dog, or even that the internet itself understands. Other humans understand what it’s like to lose a dog; they have posted about their experiences on the internet, and a search engine offers a way for you to find what they’ve said (and to potentially interact with them). I would argue that the search-engine experience is not only more transparent than a chatbot about what is happening; it is psychologically healthier for the user.

The only reason to have an LLM emit sentences like “I understand” is to make it more appealing than a search engine and increase the likelihood that a user will return; that is, it’s another way of maximizing customer engagement. This is beneficial to the company selling the LLM, but not to the users. As a design strategy, it’s not all that different from the way slot machines repeatedly give the impression that the player came very close to winning, enticing them to try again. Employing philosophers might endow LLM companies with an air of respectability that slot-machine makers don’t get from the behavioral psychologists they hire, but in both cases, the companies are preying on people’s tendency to see something that’s not there.

The use of first-person pronouns is dishonest, but there’s a much deeper issue that goes beyond how a statement is phrased. Philosophers often draw a distinction between statements of fact, such as “Paris is the capital of France,” and statements of value, such as “Paris is the most beautiful city in the world.” No one should be relying on LLMs to emit statements of value at all, but if the only statements they emitted were ones reflecting aesthetic preferences, they might not be worth arguing about. What makes Claude’s constitution profoundly problematic is that Anthropic wants Claude to emit sentences reflecting a certain system of ethical values. The values described in Claude’s constitution sound very nice, but that hardly matters; it’s dishonest to suggest that Claude is capable of moral reasoning, because it’s not.

Some might object, saying that LLMs appear to be engaged in reasoning when they successfully perform other tasks, such as writing code, so why wouldn’t they be able to perform moral reasoning? The answer lies in the difference between moral reasoning and other forms of reasoning. [...]

Moral reasoning is categorically different. It is necessarily subjective because it relies not just on an individual’s intellectual response to a problem but also on their emotional one, and that emotional response is grounded in a lifetime of subjective experience. It requires having made decisions in the past and seeing how they affected others, and on having been affected by decisions that others have made. Without such a history, an LLM can only rephrase expressions of moral reasoning found in its training data. The aforementioned New Yorker article describes an experiment where Claude was given a scenario describing an ethical dilemma, leading it to emit the sentence “I cannot in good conscience express a view I believe to be false and harmful about such an important issue.” That’s a nice-sounding sentence, reminiscent of statements that principled individuals have uttered in the past when confronted with dilemmas, but coming from Claude, it means as much as the “Your call is important to us” recording that you hear when you’re on hold. Maybe less.

This brings us back to my earlier contention that having a body is a prerequisite to having emotions. Experiencing an emotion such as desperation is inseparable from having stress hormones such as cortisol and epinephrine flood one’s body. Similarly, having a conscience means feeling sadness or moral repulsion at the idea of taking a certain action, and those emotions entail a physiological response, a remnant of having once felt sick with guilt after committing an immoral act. It’s interesting that an LLM can generate descriptions of actions that conscientious fictional characters would either take or refrain from taking, but this is not a replacement for a conscience.

If a company builds a machine that, when fed descriptions of assorted ethical dilemmas, emits sentences either of the form “Compromise your values” or “Don’t compromise your values,” it is not building a tool that assists people in their decision making; it is encouraging people to stop making decisions. The writer L. M. Sacasas has said, “Our technological systems, by nature of their design and the ideology that sustains them, are machines for the evasion of moral responsibility.” He was talking about social-media platforms, but his observation is, if anything, even more applicable to LLMs. Whenever a person delegates a decision to an LLM, they are trying to off-load accountability for that decision, and if a company that sells an LLM portrays the product as having a moral center, it is offering a way for its customers to abdicate their responsibilities.

by Ted Chiang, The Atlantic |  Read more:

Image: Enigmatriz

[ed. As with everything Ted Chiang writes, thought provoking throughout. For a rebuttal, see: Ted Chiang Is Wrong About AI Consciousness (Bentham). Then there are the far outs who, no matter what, will always subscribe to Roko's basilisk (in my mind, sort of a Pascal's wager).]

A Quiet Refusal to Compromise

Over the past decade, with amazement and dismay, I have watched former friends and acquaintances make radical turns toward a conservatism that I no longer recognize. This story is well known by now: beginning in 2015, conservatives began to divide into pro- and never-Trump factions. Some visited or moved to Hungary. National conservatism and integralism and “Common Good Conservatism” emerged as new options for disaffected traditionalists, and of course, liberalism “failed.”

All of this is chronicled in Laura Field’s new book, Furious Minds (reviewed earlier for Law & Liberty by John Grove). The volume is basically a book of highbrow gossip, and it has its faults. But it also provides a fairly accurate account of the past ten years. Field completed her PhD in (Straussian) political philosophy at the University of Texas in 2011. During her student years and afterward, she existed on the margins of intellectual conservatism. She watched many of the movement’s major players as they engaged in activism, wrote provocative essays, and instigated revolution on the Right. [...]

The problem in 2026 is that many of the most prominent intellectual conservatives have sold their birthrights for the fleeting fame promised by social media, podcasts, and coverage in The New York Times, The New Yorker, and other prestige outlets. They appear more interested in making names for themselves or “blowing up the system” than in doing the quiet, unobserved, humble work of renewing the institutions that are so vital to civil society. They are, at root, interested in winning the culture wars, and winning requires fighting. It’s what a friend has called “punch-in-the-face conservatism.” In borrowing methods from the cultural Left, many of them have become right-wing Gramscians. These men (and they are nearly all men) sense that America has arrived at an eschatological moment, and they definitely want everyone else to know it too.

I also think they find it exciting and invigorating. At last we have come to a crisis point that demands strategy and action! Enough with all the subsidiarity, little platoons, and institutional reform. Conservatives should be bold enough to grasp the levers of power and use them against the Left, just as the Left has used them against us. As one Claremont Institute commentator has written, breathlessly, “Practically speaking, there is almost nothing left to conserve. What is actually required now is a recovery, or even a refounding of America.” Helen Andrews has imagined a parallel crisis in the relations between the sexes. Her “great feminization” thesis lays the blame for “wokeness” on all those overachieving and schoolmarmish women who now dominate the white-collar professions. In her words, they are a “potential threat to civilization.” And on and on. It’s easy to adduce multiple examples of this overheated rhetoric.

To be fair, there are (of course) elements of truth in many of the scathing critiques leveled by the New Right. Andrews is correct that, in the aggregate, there are differences between men’s and women’s leadership styles. Christopher Rufo and others aren’t wrong that advocates of “Diversity, Equity, and Inclusion” greatly overplayed their hands. And much of the extreme reaction on the Right is undoubtedly a response to the provocations of the Left, whose activists haven’t exactly been models of self-restraint over the past few decades.

Unlike those on the New Right, though, I’m not sure that we’re at an eschatological moment in Western culture. We might be. But whether or not we’ve arrived at a civilizational crisis, there are alternative ways of responding to this moment, ways far more authentically conservative than what is now playing out in so many contemporary institutions.

In thinking about what conservatism means, and about how to respond to our cultural moment, two courses of action come to mind. The first is to recalibrate our view of the world; the second, to engage in practices that don’t incite battles but preserve and rejuvenate culture. Work like this is not likely to be praised or even recognized, and it asks for quiet self-assurance, not loud declarations on social media. Cultivating a positive and hopeful vision in the midst of disorder simply is the primary obligation of conservatives, especially if we’re Christians, whose hopes lie not in the rise or fall of any particular worldly power.

Why is it so difficult, and so unpopular, to embrace this hopeful, alternative vision, and why are conflict and battle so enduringly attractive? William Hazlitt offers an answer in his shrewd essay from 1826 entitled “On the Pleasure of Hating.” There is a “secret affinity, a hankering after, evil in the human mind,” he writes, which “takes a perverse, but a fortunate delight in mischief, since it is a never-failing source of satisfaction.” Life would “turn to a stagnant pool, were it not ruffled by the jarring interests, the unruly passions, of men. The white streak in our own fortunes is brightened (or just rendered visible) by making all around it as dark as possible.”

Most of us will recognize this universal human tendency to take perverse pleasure in hating, and in dwelling on ugly and disordered things. The desire to see awfulness helps to explain the market for polemics and declension narratives rather than subtle and qualified arguments. Who has not felt, in a moment of crisis, a sudden sharpening of the will, a vision of exactly the path forward?

The pleasure of critique also provides a sense of superiority, both intellectually—because we have seen things as they truly are—and morally. Deny it though we do, it is pleasant to think oneself smarter than others and to imagine that we, not they, stand on a solid foundation of truth. Similarly, in the moral sphere, if we are part of an unappreciated or persecuted minority, there is solace in knowing that our way of life is simply better than that of our opponents, even if the world at large does not agree.

And then there is the boredom factor. Temperance, civility, politeness, and all the other virtues that accompany political moderation can seem boring and mundane. Even if we mostly depend on norms of civility and respect in daily life, it is exciting to have a firebrand in the room—someone who will stir things up and throw rhetorical bombs. This is as true in a seminar room as in a board meeting. We admire and emulate the provocateur, the celebrity, and the radical, and are drawn to those with outrageous and “cutting-edge” views.

Yet these moral and intellectual eccentrics depend for their existence on an unseen foundation of equanimity, careful argument, civility, and self-control. They themselves may neglect or disparage this foundation, but it is nevertheless vital that somebody shore it up. Traditionally, this has been a job for conservatives.

So should conservatives be warriors or maintainers? Part of the answer will undoubtedly depend on temperament. Everyone knows people who are thoroughly pacific and disengaged or, on the other hand, full of spirit and always ready to argue. The latter disposition is what one sees far more often in the new conservatives I have been identifying, those who clamor to fight and win the culture wars with snark, meanness, and irony.

The tenor of the alternative—of a more gracious conservatism—is not adversarial but generative. It looks toward the present and the future, though not in the way that progressivism does, with its hopes of constant political improvement. Instead, this conservatism focuses on the things that are being conserved by living them fully, and by engaging in practices delivered from the past. It asks us to act within our own small spheres of influence, doing good where it is real, tangible, and visible, at levels much less national and much less public. While most of us aren’t prodigies, we all possess talents, aptitudes, and loves, which we would do well to use and develop. And this will make some difference, or all the difference, to those who live around us.

by Elizabeth Corey, Law & Liberty |  Read more:

Image: Agostino Masucci; Artcurial Worldwide/Wikimedia Commons
[ed. This is a conservative perspective I can get behind, but one that glosses over the 'tactics' the fighting contingent employ. Tactics that are frequently dishonest, threatening, sleazy and/or outright illegal. No valor in that, whatever rationalizations conservatives use for the ends justifying the means. By the way, the Hazlitt link (Pleasure of Hating) is well worth a read.]

In Support of Mandatory Nucleic Acid Synthesis Screening and Recordkeeping

As life sciences researchers, builders of AI and biotechnology, and experts with a wide range of views on how to approach AI policy, we call on legislators to make screening of orders for synthetic nucleic acids — and the equipment needed to make them — mandatory.

The ability to order synthetic DNA online has accelerated vaccine development, powered basic research, and made it possible for small teams to access capabilities that used to be confined to major institutions. Since the publication of protocols to reconstruct viruses from strands of DNA more than two decades ago, it has also been recognized as a point in the biotechnology supply chain where a bad actor could cause outsized harm. Recognizing the vulnerability, synthesis companies formed the International Gene Synthesis Consortium in 2009 to develop and implement voluntary safeguards against misuse.

While the issue is not new, the pace of progress in artificial intelligence is. AI systems now outperform PhD-level virologists on questions about highly technical laboratory procedures in their own domains of expertise. The evidence about what this means for present-day biosecurity threats is genuinely mixed, but the trend is hard to dispute. AI systems are improving rapidly, and alongside incredible benefits to science and medicine, there is a real possibility that the knowledge barriers which have historically prevented bad actors from obtaining biological weapons will meaningfully erode.

Support for screening does not depend on any particular view of AI; the biosecurity case has been recognized by scientists and governments for decades. Screening is also one of the best understood and least disruptive biosecurity measures available. It asks providers of synthesized DNA and manufacturers of synthesis machines to check synthesis requests for sequences of concern and to verify customer legitimacy before shipping orders. Providers should also record synthesis orders and sequence data to support legitimate biosecurity investigations, so that any threat that might evade initial screening can be traced back to its source — including when individual sequences would not raise concern in isolation. Awareness of traceability itself deters misuse.

Many of the largest and most responsible providers in the industry already screen and record orders voluntarily because it is well understood that they have an important role to play in maintaining public trust in and mitigating potential misuse of this important technology.

For these reasons, the undersigned support mandatory nucleic acid synthesis screening, including recordkeeping, in the United States.

Given the pace at which the underlying technology is changing, we believe the need is urgent. Congress should act this session, and we applaud the legislative efforts currently underway. To ensure a consistent national standard rather than a patchwork of conflicting laws, states should also consider implementing requirements based on existing federal and industry guidelines.

This is a rare moment of agreement across stakeholders that are often at odds. We hope policymakers will meet it with decisive action.

Sincerely,
Signatories: — *Everybody*

[ed. No brainer, right? You don't just leave potential life-threatening bio-warfare components laying around with no oversight. Right?]

***

Amrith Ramkumar (WSJ): Top artificial-intelligence executives are joining security experts in calling for Congress to protect against biological threats posed by AI, adding to growing pressure on lawmakers to address the technology’s risks.

Three major chief executive officers—OpenAI’s Sam Altman, Anthropic’s Dario Amodei and Demis Hassabis of Google’s DeepMind AI lab—are among the signatories of a letter urging Congress to require safeguards when companies order synthetic DNA and RNA, a key step in developing certain vaccines and biotech breakthroughs.

… It was organized by two tech-focused think tanks that said the topic is a rare source of agreement among libertarians, progressives, researchers and rival executives.

Dean W. Ball: I am honored to have signed on to this letter. This is an urgent priority for near-term action by Congress. Biotech is advancing rapidly on its own, and I—and many others—believe the “Mythos moment” in AI/bio is coming soon. It is time for action.

revisions to existing nucleic acid screening requirements were mandated by an EO POTUS signed a year ago; I worked on them while in govt. I genuinely don’t know what happened to that work after I left but it is nine months behind schedule. Congress acting is better anyway.

Joshua Teperowski Monrad: People are so astounded when I tell them this isn't already law

Alec Stapp: it really is insane [...]

Other signatories include Patrick Collison, Paul Graham, Mustafa Suleyman, Alexandr Wang and a lot more where that came from.

We need such letters, despite this having ~100% support among those who understand any side of this, this is such a slam dunk that we should be doing this even before considerations of AI making malicious action vastly easier.

Why? Because political awareness is basically still near zero:

Will Poff-Webster: When I was a Senate staffer and occasionally got the chance to bring up biosecurity risks from AI, the response was often, “What? AI might be able to do that?”

This letter shows how easy it’d be for Congress to act on this

Betting on Humans

What to do about AI & jobs.

Now, the great majority of people—whether they are “blue collar” or “white collar” laborers—spend their working hours orchestrating machines of various kinds: some to transform knowledge or bits, and others to transform atoms. Yet just a few decades ago, it would have been impossible to understand what it is that most people today call “work.”

Today, a relatively small group of technologists is starting to see the world through the lens of another fundamental discovery: deep learning, the approach to AI that has enabled machines to think and undergirded substantively all major advancements in AI over the past decade. And like their forebears at the beginning of the Industrial Revolution, these technologists are building new machines, uniquely enabled by the insights and abstractions furnished from the new science. Some believe new types of labor will emerge, concentrated on the orchestration of machines, or the tasks that remain best suited to the human touch. Others believe this time is different, and that human labor will soon be permanently obsolete.

We do not pretend to know the definitive answers. What we do know is that much of this future remains to be written, in no small part by the policy choices we make today. And what we hope to offer is a roadmap for how politicians and policymakers might bet on human agency under stark uncertainty.

Futures Not Yet Written

There are two fundamental stories one can tell about the impact of artificial intelligence on human labor. One is the pessimistic version: most of us are like the people in the early Industrial Revolution who could not learn to adapt or were stuck as mere cogs in factories. Very few of us, if any, will learn to orchestrate machines at a higher level of abstraction, and neither will we learn to invent new machines, since the artificial intelligence systems will soon exceed humans in their capacity for invention and discovery. That view is one of historical discontinuity: replacing knowledge work strikes deeper at the human uniqueness that has kept us employed than replacing various kinds of cognitive and manual labor has in the past.

The other story is optimistic: just like those early conductors and inventors of machines, we will continue our long human legacy of finding yet more to occupy our time, yet more activity that other humans find valuable. There is much more of this than we can possibly realize, because our collective imagination is bounded, yet our collective wants are limitless. How barren, in retrospect, do we find the mind of the man who thought the human touch was gone simply because we had invented machines stronger, more durable, and more reliable than us at physical labor?

Both stories will probably be true at the same time, but the unfortunate reality is that nobody knows in what proportion. More unfortunately still, it will be some time until we know: the temporary disruption that would portend broad displacement would look quite similar to the creative destruction that would come with just another industrial revolution. It’s easy for policymakers who first start to grapple with the notion of advanced artificial intelligence to reflexively adopt the pessimistic view: for so long, they’ve heard the idea that AI will be important and the idea that many jobs will be lost in the same breath that coming around on the scope of AI seems to imply believing that human labor is doomed. But that would be premature, and converts must resist becoming zealots.

Here, then, is the first—and in some sense the most troubling—message for policymakers: nobody can know what is going to happen. Anyone speaking with confidence about predictions of this kind is either misunderstanding or misleading. It is not just that we do not know “the future,” in some broad sense. We also do not know the specific nature of any problems posed by AI to the labor market: we do not know what industries, age groups, levels of seniority, job types, and so on will be affected by AI automation in practice rather than in theory or in speculation. We do not know over what timeframe these still-hypothetical changes will occur.

And if AI really does profoundly upend the labor market, we still do not know what the resulting distribution of economic resources will look like. Will the AI labs profit immensely, absorbing huge swathes of economic value as many other institutions struggle to survive? Or will AI models and systems become commodified, with value accruing to the compute designers and manufacturers? Or is it some hybrid, with most firms in the economy seeing higher profits with fewer employees and, for whatever reason, not seeing a need to hire additional people to do anything? Will there be new, high-skilled jobs created that we need to retrain millions of people for? Or will there be no new jobs at all? We do not know, and we cannot know.

That is because we are still in the process of writing this future. The role of humans in future economies is not something we simply discover as it occurs. How we distribute tasks between humans and machines is largely downstream of a web of complicated economic incentives and technical features. Is the marginal unit of computing power better spent on smoothing over the jagged frontier so no role remains for humans, or for even further improving the spikes of AI capability? Does the tax system favor firms who spend the marginal payroll dollar on hiring a worker to oversee the machines or an agent to do the same? Is there a safety net to catch those hit by local disruptions to give them the room to reorient themselves, come back five years later, and fight for their place in a new economy—or do we mollify their drive with ill-placed subsidies long enough for them to grow docile and for the structures around them to calcify? All this is contingent, and when policymakers ask ‘what will happen’, they fail to see that they’re among the central live players in this question.

How should our leaders grapple with this double uncertainty of what they should want and what will happen?

by Anton Leicht and Dean W. Ball, Threading the Needle |  Read more:

Image: via

[ed. Spoiler alert: Zvi provides a quick (and incomplete) summary (DWAtV):] 

***

"Anton Leicht and Dean Ball team up to write about what we should do about potential job loss due to AI, from the perspective of prospective ‘de facto normal technology’ AI worlds even if they don’t call it that. They wisely say we don’t know what will happen, and that the ‘no regrets’ actions will be insufficient so solve the problem, but expect the world to stay normal enough, and humans competitive and useful enough, that we can use traditional solutions to such problems.

They start with easy wins.

1. Even footing: Equalize tax treatment of AI versus labor. Yes, please.

2. Retraining: Bolster workforce training and development. They notice they are skeptical in practice, and I am even more skeptical, but sure, we can try it.

3. Measurement: Know what is happening. Yes, of course.

Then they recommend what they call difficult bets.

4. Junior Job Subsidy.

Anton Leicht and Dean Ball: We put to you that the solution to deal with junior job losses might be to keep these jobs around by brute force for a while, so that the critically important economic incentive to explore how to use junior workers does not cease.

More specifically, we might do so by restructuring the tax code to subsidize junior employment.

Given who is saying to keep jobs around by brute force, by which they mean tax incentives, we should listen. This seems like a good use of progressive taxation, which we want to do anyway, to stack the deck in favor of hiring more young workers and those switching industries, presumably with phase outs for high earners.

This risks distortions if taken too far (e.g. dumping senior workers for subsidized junior workers, or gaming designations), the marginal value of young workers could easily fall below zero marginal product if there is no future for them, and gating to particular industries or occupations risks going into ‘picking winners and losers’ and other similar dangerous territories and opportunities for corruption and pork. The authors are well aware, and are pushing anyway.

The main solution they offer is, again, taxes. They suggest doing so via raising corporate taxes, despite this having a long track record of being highly economically damaging. You definitely need to avoid worse distortions, and you definitely do not want a ‘token tax’ as such for this reason, although a tax on compute is non-crazy. Taking a stake in frontier developers is definitely an error.

They quickly dismiss consumption taxes as having a fatal perception problem, despite them being objectively the efficient answer, because they raise prices and signaling is too important here. I found this disappointing, and there are ways to fix this and also make the tax progressive.

It would be great if humans remained fundamentally highly productive while we collectively got far wealthier due to AI, so all we needed to do was redistribution and moving the tax code around.

Alas, no, I do not expect we live in such a convenient world. At which point, we likely have bigger problems, but also employment does not get solved with basic tax code shifts. If we stay in control somehow then we could do progressive redistribution to keep food on the table and a roof over people’s heads, but the jobs will vanish, or they will be rather fully fake."

The Pleasure of Finding Things Out

This is the edited transcript of an intewiew with Feynman made for the BBC television program Horizon in 1981, shown in the United States as an episode of Nova. Feynman had most of his I$ behind him by this time (3e died in 1988), so he could reflect on his experiences and accomplishments with the perspective not often attainable by a younger person. The result is a candid, relaxed, and very personal discussion on many topics close to Feynman's heart: why knowing merely the name of something is the same as not knowing anything at all about it; how he and his fellow atomic scientists of the Manhattan Project could drink and revel in the success of the terrible weapon they had created while on the other side of the world in Hiroshima thousands of their fellow human beings were dead or dying from it; and why Feynman could just as well have gotten along without a Nobel Prize.

The Beauty of a Flower 

I have a friend who’s an artist and he’s sometimes taken a view which I don’t agree with very well. He’ll hold up a flower and say, “Look how beautiful it is,” and I’ll agree, I think. And he says - “you see, I as an artist can see how beautiful this is, but you as a scientist, oh, take this all apart and it becomes a dull thing.” And I think that he’s kind of nutty. First of all, the beauty that he sees is available to other people and to me, too, I believe, although I might not be quite as refined aesthetically as he is; but I can appreciate the beauty of a flower. At the same time I see much more about the flower than he sees. I can imagine the cells in there, the complicated actions inside which also have a beauty. I mean it’s not just beauty at this dimension of one centimeter, there is also beauty at a smaller dimension, the inner structure. Also the processes, the fact that the colors in the flower evolved in order to attract insects to pollinate it is interesting - it means that insects can see the color. It adds a question: Does this aesthetic sense also exist in the lower forms? Why is it aesthetic? All kinds of interesting questions which shows that a science knowledge only adds to the excitement and mystery and the awe of a flower. It only adds; I don’t understind how it subtracts. 

Avoiding Humanities 

I’ve always been very one-sided about science and when I was younger I concentrated almost all my effort on it. I didn’t have time to learn and I didn’t have much patience with what’s called the humanities, even though in the university there were humanities that you had to take. I tried my best to avoid somehow learning anything and working at it. It was only afterwards, when I got older, that I got more relaxed, that I’ve spread out a little bit. I’ve learned to draw and I read a little bit, but I’m really still a very one-sided person and I don’t know a great deal. I have a limited intelligence and I use it in a particular direction.

Tyrannosaurus in the Window 

We had the Encyclopaedia Britannica at home and even when I was a small boy [my father] used to sit me on his lap and read to me from the Encyclopaedia Britannica, and we would read, say, about dinosaurs and maybe it would be talking about the brontosaurus or something, or the tyrannosaurus rex, and it would say something like, “This thing is twenty five feet high and the head is six feet across,” you see, and so he’d stop all this and say, “Let’s see what that means. That would mean that if he stood in our front yard he would be high enough to put his head through the window but not quite because the head is a little bit too wide and it would break the window as it came by.” 

Everything we’d read would be translated as best we could into some reality and so I learned to do that - everything that I read I try to figure out what it really means, what it’s really saying by translating and so (LAUGHS) I used to read the Encyclopaedia when I was a boy but with translation, you see, so it was very exciting and interesting to think there were animals of such magnitude - I wasn’t frightened that there would be one coming in my window as a consequence of this, I don’t think, but I thought that it was very, very interesting, that they all died out and at that time nobody knew why. 

We used to go to the Catskill Mountains. We lived in New York and the Catskill Mountains was the place where people went in the summer; and the fathers - there was a big group of people there but the fathers would all go back to New York to work during the week and only come back on the weekends. When my father came he would take me for walks in the woods and tell me various interesting things that were going on in the woods - which I’ll explain in a minute - but the other mothers seeing this, of course, thought this was wonderful and that the other fathers should take their sons for walks, and they tried to work on them but they didn’t get anywhere at first and they wanted my father to take all the kids, but he didn’t want to because he had a special relationship with me - we had a personal thing together - so it ended up that the other fathers had to take their children for walks the next weekend, and the next Monday when they were all back to work, all the kids were playing in the field and one kid said to me, “See that bird, what kind of a bird is that?” And I said, “I haven’t the slightest idea what kind of a bird it is.” He says, “It’s a brown throated thrush,” or something, “Your father doesn’t tell you anything.” But it was the opposite: my father had taught me. Looking at a bird he says, “Do you know what that bird is? It’s a brown throated thrush; but in Portuguese it’s a . . . in Italian a . . . ,” he says “in Chinese it’s a . . . , in Japanese a . . . ,” etcetera. “Now,” he says, “you know in all the languages you want to know what the name of that bird is and when you’ve finished with all that,” he says, “you’ll know absolutely nothing whatever about the bird. You only know about humans in different places and what they call the bird. Now,” he says, “let’s look at the bird.”

He had taught me to notice things and one day when I was playing with what we call an express wagon, which is a little wagon which has a railing around it for children to play with that they can pull around. It had a ball in it - I remember this - it had a ball in it, and I pulled the wagon and I noticed something about the way the ball moved, so I went to my father and I said, “Say, Pop, I noticed something: When I pull the wagon the ball rolls to the back of the wagon, and when I’m pulling it along and I suddenly stop, the ball rolls to the front of the wagon,” and I says, “why is that?” And he said, “That nobody knows,” he said. “The general principle is that things that are moving try to keep on moving and things that are standing still tend to stand still unless you push on them hard.” And he says, “This tendency is called inertia but nobody knows why it’s true.” Now that’s a deep understanding - he doesn’t give me a name, he knew the difference between knowing the name of something and knowing something, which I learnt very early. He went on to say, “If you look close you’ll find the ball does not rush to the back of the wagon, but it’s the back of the wagon that you’re pulling against the ball; that the ball stands still or as a matter of fact from the friction starts to move forward really and doesn’t move back.” So I ran back to the little wagon and set the ball up again and pulled the wagon from under it and looking sideways and seeing indeed he was right - the ball never moved backwards in the wagon when I pulled the wagon forward. It moved backward relative to the wagon, but relative to the sidewalk it was moved forward a little bit, it’s just [that] the wagon caught up with it. So that’s the way I was educated by my father, with those kinds of examples and discussions, no pressure, just lovely interesting discussions.

by Richard Feynman, Learning Media MIT.edu |  Read more: (pdf)
Image: uncredited

Hug of Death

Will Japan's content industries survive the government's efforts to promote them?

You can be loved or you can be feared.

In a January interview, the White House’s chief of staff declared that we live in a world “that is governed by strength, that is governed by force, that is governed by power,” signaling America’s choice to take the latter path.

Japan, on the other hand, seems dedicated to the former. In February, Japanese government officials announced a plan to expand the size of the nation’s content production industry, meaning its books, manga, anime, games, movies, and more, to $130 billion USD by 2033, with an eye towards making pop culture a pillar of the economy.

Is this a realistic goal? That’s another story, one I tackled last month. But let’s put the punditry aside and say they succeed – that the Japanese government manages to create the world’s first true fantasy-industrial complex, a government and private industry working together to harness content production and export as an economic engine. (What about Korea, you might ask? They are a pop-cultural powerhouse, but the nation’s fortunes still rest upon the physical products it produces — content currently only accounts for 2% of their economy.) The question then becomes: what are the broader implications of linking a nation’s economic well-being to its entertainment industry? In other words, what happens when a country doesn’t simply promote its pop culture but comes to depend on it?

I’ve written for years about how Japan’s network of cultural producers has won hearts and minds around the globe – how their efforts have contributed to Japan’s considerable soft power. But that was an organic development, entirely grass roots, the product of countless creators and consumers collaborating over many years to build one of the most vibrant environments for pop culture on the planet. The government is well aware of its nation’s reputation as a pop superpower, but it played little role in making it so.

What about the Cool Japan fund? Notoriously ineffective. Critics (who include the fund’s own CEO) frame this as a bad thing. But I think otherwise. The scandals, the questionable investments (Cars? Refrigerators!?), and general ineptitude are a blessing in disguise. I say this with no schadenfreude. The Cool Japan bureaucrats I’ve met all seem like good folks. I say it because a government getting involved in the production of fantasies has huge implications for societies. And to be frank, I don’t think any of the architects behind Japan’s big push have really thought them through. [...]

Freedom of expression is a good thing, most of us will agree. But free speech is where the problems will begin, and compound, for the Japanese government. If the authorities are really going to take an active stand in promoting everything, without interfering in those creative works, they’re going to find themselves associated with things that get, well, creative with social norms. More than that, things that anger and disgust.

The dark matter of Japan’s pop-cultural industry is huge amounts of edgy content. Some of it is quite disturbing. (Don’t worry, that isn’t a risky click: it’s a link to the time I got “lolicon” into The New Yorker.) I’m not a fan of this material, but I’ve always believed the freedom Japanese artists feel to go places that polite society doesn’t, is part of what gives the content industry such vitality here. I mean, even if you aren’t producing crazy stuff, the knowledge that nothing’s off limits has to unshackle imaginations. Or shackle them. I don’t judge.

Anyway, promoting the industry as a whole doesn’t equal endorsement of any given content, right? The views and opinions expressed in this program are those of the speakers, blah blah blah, right? Right. But also wrong. Because once you’ve made Content with a capital C the foundation of your nation’s economy, it becomes your official face to the world. That includes all of the skeevy stuff that freaks people out, in Japan and elsewhere. And the implications of that are downright existential, as in “can a nation really exist on pop culture alone?”

by Matt Alt, Pure Invention |  Read more:

Image: uncredited

What the Pope Said About A.I.

Leo XIV’s new encyclical, “Magnifica Humanitas,” presents a remarkable case for placing moral concerns, and not profit, or competitive advantage, or efficiency, at the center of any discussion of artificial intelligence.

Last year, only months into his papacy, Pope Leo XIV, the first American Pope, called on developers of artificial intelligence “to cultivate moral discernment as a fundamental part of their work.” In response, the Silicon Valley billionaire and troll-in-chief Marc Andreessen began mocking the pontiff by tweeting an idiotic meme at him. The Pope raised the grave concern that artificial-intelligence companies were “totally ignoring the value of human beings and of humanity”; the venture capitalist Peter Thiel reportedly wondered whether the Pope might be in league with the Antichrist. The merchant princes of Silicon Valley appeared concerned that the new Pope would usurp their authority and diminish their power. And now, arguably, he has, in a long-awaited encyclical on artificial intelligence.

For years—for decades—tech leaders have described their investments and inventions, their corporations, and even themselves in religious terms, and specifically in messianic terms. They claimed to be driven by a mission to make the world a better place; they were faithful to the misbegotten gospel of disruptive innovation. A “mission” is, historically, the Christian work of spreading the word of the Gospel; disruptive innovation is a theory of change that participates in the rhetoric of salvation. For a time, Facebook’s stated mission was “to give people the power to build community and bring the world closer together,” which is what most clergy of any faith might say is their mission, too, alongside caring for the poor and comforting the suffering. Tech executives, dressed in the ritualized vestments of hoodies, jeans, designer sneakers, and black T-shirts, have acted as if their companies were churches, their TED talks so many homilies, and their products—apps, platforms, and video games—temples, mosques, and chapels. More recently, these same people—men, really—have heralded the arrival of artificial intelligence as ushering in what Mark Zuckerberg calls a “new era for humanity.” This week, the Pope offered his own understanding of that new era in his encyclical, titled “Magnifica Humanitas,” or “Magnificent Humanity: On Safeguarding the Human Person in the Time of Artificial Intelligence.” It could hardly be more different from the preachings of the priests of Silicon Valley. They like to say they are saving the world. The Pope fears they are destroying it. [...]

The new encyclical, at nearly forty thousand words, bears reading. It is addressed “to all the Catholic faithful, to all Christians and to all men and women of goodwill”—that is, to everyone. In advance of its release, and leery of the inevitable TL;DR reaction, one Texas bishop warned parishioners not to ask a chatbot to summarize it for them. (Earlier this year, the Pope urged priests against using ChatGPT to write their sermons and to instead “use your brains more.”) It is not a beautiful document. It’s often maddeningly, boringly wonky (“this entails establishing norms so that the decision-making behind content selection and its development becomes more transparent and protects personal data”), and it gives every evidence of being written by a committee (“psychological and psychiatric literature has documented with growing insistence how early and unsupervised exposure to digital devices and social media can negatively impact sleep, attention span, control of emotions and relationships”). Some of it reads like a Silicon Valley press release (“Today, the convergence of automation, robotics and AI is rapidly transforming the very structure of work”). Nevertheless, “Magnifica Humanitas” presents a remarkable case for placing moral concerns, and not profit, or competitive advantage, or efficiency, at the center of any discussion of artificial intelligence.

If those of us Americans who are Catholic are proud of this Pope, many of us are even prouder that the first American pontiff has taken on this vital matter, and at such a crucial moment. In much of American culture—and especially in the business and tech press—challenging the economic power and oligarchic rule of U.S.-based artificial-intelligence companies is an act tantamount to heresy. Pope Leo is not only willing but eager to dissent. Bless him.

Much of the encyclical involves defending the proposition that the Vatican ought to be—and has always been—engaged in making statements about new and very worldly things like artificial intelligence. “The Church is present in history and engages in dialogue with the world,” Leo argues. He agrees with the Sam Altmans and Elon Musks of the world that humanity stands at a crossroads. But at this crossroads, he argues, three questions must be asked: “Where are we going? Toward what goal do we wish to orient ourselves? What direction should we choose as a people and as a human community?” Invoking a Biblical story about hubris, the building of the Tower of Babel, he warns of what he calls the “Babel syndrome”: “namely the idolatry of profit that sacrifices the weak, a uniformity that neutralizes differences, and the pretense that a single language—even a digital one—can translate everything, including the mystery of the person, into data and performance.”

Beginning with the fundamental dignity of the human, Leo traces the inalienable, universal equality of persons and their inviolable rights. He establishes, within the Church’s Social Doctrine (traceable to “Rerum Novarum”), principles that include the commitment to the common good, which he defines as “the social expression of the dignity recognized in every person.” [...]

The problem is not the technology, the Pope maintains in “Magnifica Humanitas”; it’s the anthropology. Algorithms, forms of automation, and artificial intelligence sort the worthy from the unworthy; they manipulate information and undermine trust; they violate privacy; they enhance the power of the already powerful and reduce the capabilities of the already vulnerable; they make war more ruthless; they undermine democratic governance; they take away the dignity of work, possibly for the mass of humanity. He presses for forms of regulation and especially for democratic control of artificial intelligence, but above all he calls for “disarming” A.I. “To disarm does not mean rejecting technology, but preventing it from dominating humanity,” he writes. “It means freeing technology from monopolistic control and opening it to discussion and debate, therefore making it human-friendly and restoring it to the plurality of human cultures and ways of life.” He worries that the culture around artificial intelligence undermines the search for truth that is necessary for both democratic life and any possibility for a genuine spiritual existence. [...]

That the concerns the Pope has raised in “Magnifica Humanitas” are not even remotely new does not make them any less urgent. Yet this history does suggest that calls to slow down the development of artificial intelligence and, as Arendt put it, to “think what we are doing” have not been heeded. Then again, before this week, they’ve never been sounded by the Pope, the spiritual leader of nearly a fifth of the world’s population.

“Magnifica Humanitas” is in many ways a religious analogue to Claude’s Constitution, released by Anthropic this past January (and on which at least two delegates to the Vatican were consulted). In a move freighted with symbolism, Anthropic’s co-founder Christopher Olah appeared on the dais alongside Leo at the release of the encyclical, which the Pope, in a first for the Church, presented in person, at the Vatican’s Synod Hall. “I am grateful to His Holiness and to the Church for taking up this work of discernment,” Olah said in his remarks. Executives of other A.I. companies are not likely to express that kind of gratitude. Nor are they likely to cede political power willingly, any more than they are likely to become philanthropists, or volunteer to pay more in taxes, or stop tweeting daft things or selling you tools that you don’t need and that you never asked for and that make you miserable, angrier, and stupider.

by Jill Lepore, New Yorker |  Read more:

Image: Yara Nardi/Reuters
[ed. Maybe tl;dr for most folks, but AIs will certainly read it. Sort of my intent with ARIA: The Great Pause. Every little bit helps.]