The Id, the Ego and the Superintelligence

Panic around A.I. often focuses on whether machines will replace (or simply kill) us. But what the machines are often good at is a distinct set of tasks: Retrieval, near-instant summary and the frictionless, stunning recombination of everything that has already been said. What they are not good at is what philosophy has been training humans to do for thousands of years: judgment. And so the A.I. companies have started hiring philosophers. They don’t just consult them or wheel them out for an amusing after-dinner talk on the trolley problem; they actually hire them with salaries and, presumably, dental plans. A.I. companies need philosophers because actually thinking requires forgetting — and knowing when to forget. They need philosophers because of what their machines cannot do.

Judgment is not calculation. It is the capacity to weigh, to discriminate, to sense that one answer is true and another merely correct. It is taste, discernment and the knack of thinking sideways. It is refined only through experience — through error, embarrassment, boredom, grief and the occasional hangover. Meaning is derived from the fact that we are mortal: Things matter to us because we are going to die, and we know it. There can be no meaning in immortality. To borrow from my friend Christian Madsbjerg, I would like to propose a better Turing test than Turing’s: Can a machine experience social anxiety? Can it lie awake at 4 in the morning replaying the stupid thing it said at dinner? Call me when a chatbot does that. To be intelligent in the human sense, the machine would need to give a damn.

Enter the A.I. companies’ philosophers. Anthropic keeps a small squad on the payroll, among them Amanda Askell, whose doctoral thesis was on infinite ethics — the question of how to act well in a universe of infinite value — and whose job is to shape the character of the company’s machine. Google DeepMind has appointed the Cambridge philosopher Henry Shevlin to the newly minted post of “Philosopher,” a job title I have been trying and failing to secure for some 40 years. The OpenAI chief executive Sam Altman says his company consulted “hundreds of moral philosophers” in drawing up the principles behind ChatGPT. (That’s roughly the turnout of a middling philosophy conference.)

After a lifetime of being asked at parties what one actually does with a philosophy degree, I find this development almost unbearably funny. Why now? Because the questions the A.I. companies are stumbling into are not engineering questions. What should the machine say to the grieving, the lonely, the person typing into the void at 4 in the morning? What sort of character should it have — candid, flattering, evasive, kind? Could such a thing conceivably suffer, and would we owe it anything if it did? These are questions about what matters. The engineers, to their credit, have noticed that no quantity of computational capacity will settle them. Ethics, it turns out, cannot be scraped from the internet, because the internet is precisely the contemporary record of our failure to agree about ethics.

There is a further irony here, and it is an old one. Philosophy’s first recorded technology panic was about exactly the thing these machines have now perfected: Artificial memory. Toward the end of Plato’s “Phaedrus,” Socrates tells the story of the Egyptian god Theuth, inventor of numbers, geometry, astronomy, draughts and dice, who presents his masterpiece — writing — to Thamus, god-king of Egypt. This discovery, Theuth announces, will make the Egyptians wiser: It is a pharmakon, or remedy, for memory. Thamus is unpersuaded. Writing, he replies, will implant forgetfulness in the soul; people will trust the external marks and cease to remember from within; they will acquire the reputation of wisdom without the reality. Pharmakon, conveniently, also means poison. Every panic since — about the printing press, the novel, television, the smartphone, the chatbot — has been a footnote to Thamus, and the joke, of course, is that we know his warning against writing only because Plato wrote it down.

Seen from the “Phaedrus,” A.I. is not a rupture but a culmination, representing the final, most extravagant flowering of external memory, the remedy-poison swallowed whole. And this is where the difference between us and the machines cuts deepest. A.I. is a memory machine, the most magnificent ever devised. It forgets nothing. The human being, by contrast, is a forgetful animal. To make this happily oblivious creature remember — to make it capable of promises, debts and morality — required millenniums of discipline and frankly appalling cruelty. “If something is to stay in the memory, it must be burned in,” Friedrich Nietzsche wrote. Conscience, on this view, is mnemonics with a whip.

Yet Nietzsche also prized what he called active forgetting, and considered it a form of strength, even of health. Forgetting is not a leaky bucket but a doorkeeper, closing the gates of consciousness so that something new can happen, so that experience can be digested rather than endlessly force-fed back to us. We forget, and we move on. Forgiveness, sleep, cheerfulness, second marriages, continuing to support Luton Town Football Club: All of it depends on the merciful erosion of the past.

Some years ago, I wrote a short book called “Memory Theatre” about a philosopher (kind of an even more awful version of me) who attempted total recall, arranging every scrap of his past in a vast mnemonic palace. It did not end well for him; it never does. A creature that could forget nothing would not be a superintelligence but a ruin, buried alive under its own archive. What we have built is, in a sense, an infinite archive that cannot mourn, cannot forgive and cannot let go, because it never lost anything in the first place. The machine remembers everything and therefore, oddly, nothing in particular. Nothing weighs more than anything else.

This is why philosophy matters rather more, not less, in the age of A.I. Not because philosophers can out-remember the machines — believe me, we cannot; I continually forget where I have parked my own arguments — but because philosophy is the discipline of knowing what to do with experience: What to keep, what to discard, what to laugh at, what to junk. Thinking is not the accumulation of everything but the art of judicious, active forgetting.

by Simon Critchley, NY Times | Read more:
Image: Flo Meissner

Tell Me What “Socialism” Means

Before you tell me if it’s good or bad.

You may have noticed that political arguments often have a frustrating, circular nature. Charges and countercharges, points and counterpoints—all get made without ever seeming to converge on a shared reality. It may seem that you and your political opponents are talking past one another. Much of this, in electoral politics, is due to the fact that the discussions are in fact bad faith attacks that seek only to undermine, rather than to reach an agreement. But even good faith attempts to find common ground with political opponents frequently sink in these murky waters.

When this happens, stop and ask: What are we talking about? Are we talking about the same thing here? What is the definition of the terms that we are allegedly debating? Frequently, you will find that you and your counterpart each have a personal definition of the thing that you are arguing about, and those definitions are not the same. Your argument has not been about political substance at all; it has been about the meaning of a particular term. Failure to settle on a shared definition before you start the debate means that the debate can just go on forever in opposite directions, like two fighter jets flying past one another at different altitudes and wondering why they never get a chance to shoot down the enemy.

Nowhere in American politics does this stupid dilemma manifest itself more than the purported debate over “socialism.” Oh, you have opinions on “socialism?” And whether it is evil? Or great? And you would like to yell about them? Let me stop you right there. What exactly does that mean?

The majority of mainstream political discourse about socialism—and almost all of the discourse emanating from Republicans—is just bad faith, lowest-common-denominator red baiting, paltry attempts to wring some political advantage from the still-glowing embers of decades of Cold War anticommunism rhetoric that has soaked deep into the minds of anyone born before the turn of the century. None of this is a genuine attempt to discuss what socialism really is and what it might mean here, so I will ignore it. (A benefit of being a How Things Work reader is that I respect you enough to skip the many paragraphs of undeserved credulousness that other, less honest outlets would feel obligated to pay to this bullshit.)

What about those who do want to discuss the issue? Those who believe that they are trying to communicate something useful about whether or not socialism is desirable? In aggregate, I’m afraid, this has become one of the least productive mishmashes of disparate meanings in living memory.

Liberal billionaire Nick Hanauer writes that “Neither socialism nor MAGA is the answer” to America’s inequality crisis. Instead, he advocates for higher minimum wages, higher taxes on the rich, stronger labor unions, and less concentrated corporate power. He puts forward his proposal for “market humanism” as a counterpoint to what is on offer from politicians like Zohran Mamdani, who supports higher minimum wages, higher taxes on the rich, stronger labor unions, and less concentrated corporate power.

Bernie Sanders is America’s most famous socialist politician. He became popular by calling for higher minimum wages, higher taxes on the rich, stronger labor unions, and less concentrated corporate power. When Elizabeth Warren was running for president against Bernie, she distinguished herself from him by declaring that she was, in fact, a capitalist, and wanted not socialism, but Accountable Capitalism. Her platform included higher minimum wages, higher taxes on the rich, stronger labor unions, and less concentrated corporate power.

On the other side is Donald Trump, an avowed anti-socialist, who is now aggressively pursuing state ownership of corporations.

What is the public supposed to take from this contradictory, uninformative war of labels? Polls now show that socialism is significantly more popular than capitalism among Democrats. What that means is that we have succeeded in getting people to imagine that “socialism” means things they like, more than “capitalism” does. This is good, I guess, but it is mostly an indicator of the fact that everyone’s personal definition of these terms is so flexible to the point of meaninglessness.

The dictionary definition of socialism is “any of various egalitarian economic and political theories or movements advocating collective or governmental ownership and administration of the means of production and distribution of goods.” The Republican definition of socialism is “A laundry list of the worst abuses of totalitarian communist states; or, anything advocated by a Democrat that would cost the rich money.” The centrist definition of socialism is “anyone to our left.” The progressive capitalist definition of socialism is “other progressives who may share my values but who are sadly not economically savvy enough to understand that what we really need are higher minimum wages, higher taxes on the rich, stronger labor unions, and less concentrated corporate power.” That is, coincidentally, also more or less the definition of socialism that democratic socialists have. Each faction imagines that the others are unsophisticated, unrealistic dupes.

My purpose in writing this is not so much to try to litigate the One Correct Definition of Socialism, but rather to highlight the (critical!) fact that the big debate about the Soul of the Democratic Party or the Battle For the Faith of America or whatever is, for the most part, one big cloud of people talking past one another. The dirty secret is that most political commentators prefer it this way. It is easier to make grand proclamations about how either Free Enterprise Is Fundamental to Free People or how Capitalist Scum Must Die than to go about the tedious work of agreeing upon our terms and then having an actual policy debate.

I am a self-proclaimed socialist who writes for a socialist magazine and who just got back from attending the Socialist Conference. I promise you that socialists themselves do not have a shared definition of socialism. Much less do the people on the other side whose only reference point for socialism is “Cuba and uhhh Russia” have a shared definition of socialism. My own definition of what socialism means in the context of American politics would be something like “More public control and less private control in society, particularly in the economy, with the goal of producing greater equality.” That, I think, gets at what most of us actually mean when we talk about this stuff. (I have met few American socialists who are truly committed to the communist caricature of full state control of the economy, and even those who do will have to admit that that the path to getting there is described above.)

You will notice that even this definition is directional—it describes where we are trying to go in relation to where we are now, rather than articulating some strict Marxist benchmark that must be met. Politics itself is a directional endeavor, so this sort of definition is useful for politics. If you are an academic looking for analytical rigor above all, however, it might not satisfy you.

Who cares! If you take anything away from all of this, let it be: Almost all of America’s mainstream political debate over “socialism” is stupid. The portion of it that is not a scam on purpose is mostly a big waste of time. Do you support higher minimum wages, higher taxes on the rich, stronger labor unions, and less concentrated corporate power? Then we are on the same side. Let’s walk down that road together. If we make progress, we will reach some point where some people say, “This is far enough.” The rest of us will say, “No, we need to keep pushing this further.” At that point, we can have a debate about the actual policies in question, and their merits, and what the ideal we are trying to reach looks like. 

by Hamilton Nolan, How Things Work |  Read more:

Image: Getty

[ed. See also: Capital Is Organizing the World to Its Advantage and Everything Else Is a Sideshow (HTW):]

***

"We are all mad at the billionaires and whatnot but let’s be very clear about the mechanism driving all of this: American companies are making more money than ever and they don’t have to share it because workers don’t have the power to claim their fair share, and because some of the money has been used to purchase political influence to prevent the government from stepping in, and so therefore the money accrues to the owners, which is to say, to the rich. The richest ten percent of Americans own close to 90% of America’s stock market wealth.

This isn’t too hard to understand. I obviously didn’t invent this analysis—this is just grade school Marxism. But to reach these conclusions you don’t need to read any theory or know any buzzwords. You just need COMMON SENSE. All the money that businesses make will go either to the workers or to the owners. The goal of capital, of businesses, the logic of capitalism itself, is for businesses to try to take as close to 100% of the profits as possible for themselves. (Indeed, an amusing new economics paper shows that going to business schools causes managers to pursue exactly this mandate). This is the nature of a corporation. It will always do it, just like a fire will always want to burn. Our problem is that we have not kept corporations properly contained. So, like fire, they will consume everything.

It is far more accurate, more useful, and more true to understand politics as a battleground between capital and labor than as some sort of Democratic vs. Republican thing. Politics is just another arena for capital to organize in order to maximize its own growth."

The 25 Science-Fiction Books You Must Read, Ranked

And six short story collections

Some ‘best of’ lists seek to achieve a spurious objectivity through means such as reader polls, complex ranking systems or asking distinguished experts for their views.

This is not one of those lists. This list has instead been compiled by the greatly superior method1 of me thinking really hard about it and assuming that my opinion mattered more than anyone else’s.

In terms of criteria, firstly, every book on the list is an excellent read. Some of the books on here may be worthy, but there are no books on here purely for their worthiness. Secondly, every book is one that will make you think, or open your eyes to a new perspective - for that is a core function of science fiction, to explore new possibilities. And thirdly, to whittle the list down to 25, I have taken into account relevance and impact - either to the world of today, or to the genre more broadly.

While the borders of every genre are porous, every book on here has a clear claim to being science fiction. There is no pure fantasy, or alternate history - much though I enjoy those genres, this is not their list.

So, without further ado, let’s get cracking. As is traditional, we’ll do this in reverse order, so you’re kept guessing to the very end as to whether I’m overlooking your own favourites, or saying they’re amongst the best.

Last and First Men - Olaf Stapledon (1930)

A future history, not a novel, Stapledon takes us forward across billions of years as new races of humanity rise and fall - on Earth, on Venus and ultimately on Neptune. Some winged, some more intelligent and some less intelligent, some deliberately designed and some evolved, each have their turn on the stage. [...]

The Mote in God’s Eye - Larry Niven and Jerry Pournelle (1974)

What happens when you combine Niven’s extraordinary imagination with Pournelle’s ability to actually tell a story that has a plot? You get The Mote in God’s Eye, perhaps the best first-contact encounter with a truly alien extra-terrestrial species ever written, in a richly rendered future setting. A rare occasion of a joint-authored novel being better than any that either has written alone. [...]

The Day of the Triffids - John Wyndham (1951)

What if almost everyone went blind, and sentient, walking, carnivorous plants inherited the earth? This unlikely premise forms the basis for one of the most civilly bleak post-apocalyptic novels, as the protagonist and his companion wend their way across the devastated Home Counties, seeking a safe haven where the remnants of humanity can survive. [...]

The Diamond Age - Neal Stephenson (1995)

A world of nanotechnology and AI, in which nation states have been supplanted by ‘phyles’ - groupings, sometimes ethnic, sometimes regional, sometimes political, of freely associating individuals - the Diamond Age tells the story of Nell, a young girl growing up in poverty who receives an advanced AI ‘primer’, designed to educate and guide its owner into a ‘more interesting life’. A wildly imaginative roller-coaster ride through a provocative future. [...]

The Warrior’s Apprentice - Lois McMaster Bujold (1986)

Arguably the best science-fiction author currently writing, The Warrior’s Apprentice introduces the brilliant, driven, physically stunted Miles Vorkosigan, primary protagonist of her Vorkosigan Saga, and follows him as he ‘accidentally’ builds himself a mercenary fleet. Modern space opera at its finest. [...]

Foundation - Isaac Asimov (1951)

A novel that shaped the genre, Foundation tells the story of a Galactic Empire in decline, the brilliant psychohistorian who sees a way to minimise the harm - and the lives of those who followed him, living his plan, guiding the new Foundation in its ascent. [...]

The Three-Body Problem - Liu Cixin (2008)

by Edrith, World of Edrith |  Read more:

Images: uncredited

[ed. Not a bad list. Could easily be doubled.]

China and the Future of Science

The Chinese socio-political system differs from our own. From the perspective of the topic of this conference, here is the most salient distinction: the Chinese system has a telos. The Chinese party-state is fundamentally a set of goal-oriented institutions. This is not unique to China—it is in fact a distinguishing feature of all Leninist systems. I sometimes think of Leninist systems as a little bit like that bus in the movie Speed. Who here has seen it? For those who haven’t, here is basic gist of that film: an extortionist attaches a bomb to the speedometer of a bus. If the bus ever slows below 50 miles per hour, everyone blows up. So it is with your average communist system. Either it hurtles towards some clearly defined goal or things start to fall apart.

In the early days of Mao, the overarching aim of the communist system was to seize state power, first through subversion and insurgency, then through more regular combined arms warfare. In the later days of Mao the newly established Chinese state and the society it intertwined were oriented around class struggle, both at home and abroad. From the 1980s through the 2010s the Chinese system was orbited a different yet still very explicitly stated goal: getting rich. In theory, if not always in practice, every action taken by every cadre, every soldier, and every state employee was subordinate to this larger, unifying aim. We must make China rich.

That is no longer the animating telos of the Chinese system. There is a new goal, one that has been articulated with great clarity by Chairman Xi and the Chinese central committee: In 2026, the aim of China’s communist enterprise is to lead humanity through what they call “the next round of techno scientific revolution and industrial transformation.”2 The Chinese leadership believes humanity stands on the cusp of the next industrial revolution. China can only be restored to its ancestral greatness if it is the pioneer of this revolution. All machinery of party and state must bend towards this end. All 100 million members of the Communist Party of China, all 50 million government employees of the PRC, all two million soldiers of the People’s Liberation Army, and ultimately all of the 1.4 billion people that call China home must be mobilized to accomplish this aim. That is the ambition. China will be the greatest scientific power the world has ever seen—or bust.

The communists are deadly serious about their pursuit of this aim. Statistics provide one window into the seriousness of their intent. Now I don’t intend for the remainder of this speech to be a laundry list of numbers, but I think the numbers are useful for helping us see the scale of what China has already accomplished and the speed with which they have accomplished it. They are also strong signal of future intent—it is difficult to survey the numbers and not appreciate just how ironclad China’s commitment to scientific achievement really is.

Now scientific achievement is difficult to measure. One common metric is to count the so-called “high impact papers” – journal articles highly cited by other leading lights in a given scientific field. Count up these papers over the course of a year, see who wrote them, see where those authors work, and—voila!—you have a ranked list of which institutions are putting out the most high-impact science in a given year. Had you done this counting exercise in the year 2005, you would have discovered that six of the world’s ten most productive universities were in the United States. Today only one of those universities is in the United States. That university is Harvard, coming in at spot number three on the list. At spot number one? Zhejiang University.

How many of you have heard of Zhejiang University? Can I get a show of hands?

And of course, Zhejiang University is just one of the Chinese institutions on this top ten list. China claims not just the number-one spot, but also the number-two spot. And not just the number-one and number-two spots, but also the fourth, fifth, sixth, seventh, eight, ninth spots go to the Chinese.

The scientific publisher Nature makes a similar catalog on a slightly more granular level, looking at specific fields of science. According to Nature’s most recent rankings, 18 of the top 25 most productive research institutes in the physical sciences, 19 of the top 20 in geosciences, and a full 25 out of 25 in chemistry are Chinese. Only in the biosciences do American scientists still have a lead—but even on that list three of the top ten are Chinese.

The kicker is, none of that was true even just a decade ago.

The most granular analysis of all is published by the Australian Strategic Policy Institute, or ASPI. ASPI publishes a neat research tracker that surveys new publications in 74 distinct high-end technologies. Unlike the statistics I just discussed, their tracker includes research published by scientists working in national laboratories and private institutions as well as those published by academic scientists. For each category they make a list of the ten institutions that are publishing the most high-impact science in that particular topic. What have they found? For 66 of the 74 categories tracked, a majority of the institutions that are now publishing the highest-impact science are Chinese. In many areas of science the dominance is total: For example, ten of then most productive research institutions in the fields of nanoscale material manufacturing, photonic sensors, chemical coating, drone operations, automated swarms, and undersea communications are Chinese. The number is nine out of ten for work on supercapacitors, advanced composite materials, inertial navigation systems, and satellite positioning, eight out of ten in advanced optical communications, advanced radiofrequency communications, and new chemical coatings, and seven out of ten for directed energy technologies, nuclear engineering, and nuclear waste treatment.

The scale of Chinese scientific production is in part a story about people. China graduates five times the number of medical and biomedical students than we do every year, seven times the number of engineers, and two-and-a-half times the number of undergraduates with research experience in artificial intelligence. Last year China graduated almost double the number of STEM PhD students than we did—and that number is actually worse than it sounds because—depending on the exact year you do the counting—between one sixth and one fifth of our STEM graduates are themselves Chinese.

Many of these researchers go back. They go back partially because they are well compensated for doing so. They also go back because of the research opportunities afforded to them. A recent study found that returning Chinese scientists go on to become the lead author on 2.5 times more papers than their colleagues who stay in the United States.9 Many Chinese research labs have 30 or 40 people attached to them—the equivalent to a commercial research lab in the United States. Ask any scientist who has gone to China in the past three years to visit academic colleagues and they will tell you how astounded they are at the quality of the laboratory equipment and machinery that their Chinese colleagues have access to. If in the not-so-distant past Chinese localities competed with each other to lay the most asphalt, now that funding pours into laboratory equipment, scientific instruments, and advanced scientific facilities. Thus China now has the world’s most sensitive ultra-high-energy cosmic-ray detector, the world’s largest and most sensitive radio telescope, the world’s strongest steady-state magnetic field, the world’s fastest quantum computer by computational advantage, and the world’s most sensitive neutrino detector. Just yesterday an attendee at this conference informed me of another I should add to my list: the world’s largest primate medical research center.

Now I can already hear some of your objections. “Tanner, these measures don’t include classified research. They don’t include the proprietary research by private companies—that is the stuff that actually pushes technology forward. American companies are not publishing billion-dollar trade secrets in the latest journals. The Chinese scientists are under insane publish or perish pressures—they are far more likely to lie and cheat. Don’t you know Chinese scientists take part in citation cartels? Haven’t you read those bitter critiques of the new system written by China’s own disgruntled scientists?”

My main response to this: you guys have lost the thread. I am reminded of a similar style of argument we often see in AI development. Every time a new model is released people play around with it for a bit and then start to catalog the flaws of this model. But the real story, the story historians will tell a generation from now, is never about the model of the moment. What matters is movement between those moments. History is made by the trend-line. What capabilities did the models have four years ago? What capabilities do they have now? What might they reasonably be expected to have in a decade hence?

Something similar might be said for science and China.

Moreover, it is not hard so hard to find examples of dominance in the scientific literature leading to dominance in the world of applied technology. More than a decade ago Chinese researchers began to dominate academic work on batteries. Today China produces 80% of the world’s battery cells. Around that same time Chinese researchers took the lead on factory robots and factory automation. Today Chinese firms buy more robots than the rest of the world combined. They have a higher robot density than almost every other country in world, despite being the world’s second largest nation. Another example: China is the only place where the unit cost of building a new nuclear reactor has gone down over the last decade. Do you think this is unrelated to the number of nuclear engineers they train and the amount of basic research they publish on nuclear engineering? Again I beg you to look at the trend-line: In 2026 China has 102 nuclear reactors either in operation or under construction. How many did they have 15 years ago? Less than 30.

Why is China pouring so many resources into scientific and technological advance? Here is a 30-second thumbnail sketch: Xi Jinping and those around him have a specific view of history. Like most Chinese, they remember with fondness past eras when their nation was the centerpiece of human civilization. They mourn China’s fall from those grand heights. They remember with bitterness the “century of humiliation” in which their people were exploited by foreign powers and wasted by vicious warlords. The Communist Party of China was founded to save China from that fate. From its inception it has seen itself as the vehicle for the salvation of the Chinese nation and thence the restoration of this nation to its rightful place at the center of the human story.

But why was China displaced from its spot “at the center of the world stage” in the first place? Chinese intellectuals have been asking this question for more than a century. Here is their most common answer: science and technology. Today Party-affiliated thinkers often describe modern human history as punctuated by discrete revolutionary moments when a new technological regime comes into being. They call these moments “rounds of techno-scientific revolution and industrial transformation.” In such moments the material basis of human civilization changes. Those who master this transition master… well, just about everything.

by Tanner Greer, Scholar's Stage |  Read more:

Image: uncredited

[ed. Go USA. See also: The Cure for Cancer Is Not a Pure Math Benchmark (SS).]

Annie Dillard, Meditative Author of ‘Pilgrim at Tinker Creek,’ Dies at 81

Annie Dillard, who as a young American writer walked into the woods and came out with a Pulitzer Prize-winning book on the wonders and horrors of creation, the modern classic “Pilgrim at Tinker Creek,” died Tuesday at her home in South Wellfleet, Massachusetts. She was 81.

Her death was confirmed by her literary executor, Maggie Nelson. No further details were given.

With the publication of “Pilgrim at Tinker Creek” in 1974, Dillard, just 28, was recognized as a major new voice in American letters. It was only the beginning of her career: In the years to come, she would publish essays, meditations on religion, poems, a theory of literature, two novels, journalism, a bestselling autobiography and another bestseller, “The Writing Life,” about what it takes to put words into a world that has more than enough of them.

Reviewers of “Tinker Creek” were quick to understand it as a 20th-century analogue to Henry David Thoreau’s “Walden.” This was no coincidence: Dillard had chosen Thoreau as the subject of her master’s thesis at Hollins University, the women’s liberal arts college in Roanoke, Virginia.

Like Thoreau, she found an unremarkable body of water to be as good a place as any for deep thoughts — particularly about how to reconcile the idea of a benevolent God with nature’s cruelty.

Unlike Thoreau, Dillard didn’t build a cabin in the woods and live there through the seasons of a year. Though her writing could convey the sense of a solitary life in the wilderness, she was mostly a day-tripper. After her forays into the Roanoke Valley of the Blue Ridge Mountains — which was, after all, her neighborhood — she would return home to the suburban house she shared with Richard Dillard, a creative writing instructor she had married at the end of her sophomore year at Hollins.

That she was a photogenic young woman coming to grips with raw nature — a business that had been almost exclusively the province of men with beards — was undeniably part of the Dillard mystique. So was her insistence that, in spite of her patient, unflinching observations on flora and fauna, she wasn’t a nature writer, but rather a theologian looking high and low for God. [...]

“Tinker Creek” won the Pulitzer for general nonfiction in 1975, and Dillard was overwhelmed by requests for speeches, interviews and television appearances. Her looks were thought by many media outlets to be as newsworthy as her writing — so much so that she was ridiculed in some circles as “Cheesecake at Tinker Creek.”

Repelled by the consequences of her success, she moved to Washington state and began dividing her time between the city of Bellingham and the islands Waldron and Lummi. Her marriage to Richard Dillard had ended in amicable divorce, and in the Pacific Northwest, she found privacy enough to regain the concentration essential to her work.

“The Cascade Range, in these high latitudes, backs almost into the water,” she wrote. “There is only a narrow strip, an afterthought of foothills and farms sixty miles wide, between the snowy mountains and the sea.”

 

by Daniel Lewis, NY Times via Seattle Times |  Read more:

Image: Rollie McKenna
[ed. Just learned about Annie's death this morning. One of my all-time favorite books is The Living (a fictional depiction of the evolution of Whatcom county, the area around Bellingham, WA). See also: Annie Dillard brought poetry into prose, pioneering a mystical form of creative nonfiction (The Conversation):]

***

Not all writing is the same. Prose typically features continuous sentences arranged in paragraphs. Poetry tends to be differentiated from prose by a deliberate organization of language that’s not determined by the page itself.

Prose poetry, the lyric essay and other hybrid forms of literature play at the intersection of genre, which usually defines these separate categories of writing. Creative nonfiction likewise aims to accurately convey reality while emphasizing literary style. The literary features of creative nonfiction are often drawn from fiction, with devices such as dialogue, scene and flashback.

But one of the distinctions of Annie Dillard’s nonfiction is how much her prose owes to poetry.

Poetry can be defined in many ways, but it tends to be characterized by honed, heightened language. Poems are often noted for their memorable images, rich sounds and foregrounding of emotion. Dillard’s nonfiction is shot through with images that arrest readers with their beauty or confront readers with their terror – as in her Harper’s essay “The Death of a Moth,” where the insect is drawn to a candle flame, becomes stuck in the wax and is transformed into a burning wick. In Dillard’s words:

“She burned for two hours without changing, without swaying or kneeling – only glowing within, like a building fire glimpsed through silhouetted walls, like a hollow saint, like a flame-faced virgin gone to God, while I read by her light, kindled, while Rimbaud in Paris burnt out his brain in a thousand poems, while night pooled wetly at my feet.”

It’s no accident that Dillard describes herself as reading the work of French poet Arthur Rimbaud by the light of the moth. She previously wrote that she loved how language sounds in French symbolist poetry, and Rimbaud was known to combine prose with his poetry in a way Dillard herself does.

Some Ways AI Could Kill Us All

[ed. Nearly everyone agrees by now that AI is an existential threat, but the hows and whys are often vague or imcomplete.] 

I don't think this is how it will actually play out. If you play a chess grandmaster, you can predict that they will beat you even if you can't predict how. I chose these examples because I don't think they require much imagination or accepting exotic assumptions.

It is important to note that if chimpanzees were to guess how humans would decimate them, they would get it wrong. Chimpanzees would not imagine guns. They would not foresee poison gas. They would not conceive of chemical castration. They would not imagine humans going around and intentionally infecting them with AIDS. They have no concept of these things; they would not see it coming.

Perhaps they might guess we'd be really good at throwing rocks. Amazingly good. Well, technically, that's what guns do: throw "rocks" really really well.

So how will superintelligent AI actually wipe us all out? Probably in a way I couldn't conceive of. Nonetheless, it's not hard to see how deadly they could be with what we already know about.

by Ruby, Less Wrong |  Read more:

Image: DALL-E via; and here. 

[ed. I wouldn't necessarily conflate super intelligence with ill-will. Most likely AIs will just be pursuing some unrelated goal and exploring every possible means by which to achieve it (including removing human barriers). That is, unless bad (human) actors are involved, which should not be discounted. See also: How My Students Think About AI (LW).]

The Final Battle For Democracy

Forget November 3. It’s January 3 when Donald Trump and his MAGA Republicans might bury our democracy once and for all. Here’s how.

Over the past year and a half since his return to the White House, Trump has exploited every conceivable unlawful means possible to rig the midterm elections in favor of the Republicans, and the federal courts have struck down as unconstitutional every one of these unlawful attempts. He will try every unlawful means over the remaining weeks to ensure that Republicans handily win the midterm elections in November. Those of us who oppose his illegality will look to the courts, but the federal courts will be institutionally incapable of checking his final unlawful rampage. [...]

***

This is how the crisis would unfold. The precursor of the crisis will come sometime before noon on January 3, when Speaker Johnson removes current House Clerk Kevin McCumber and replaces him with a person loyal to Johnson and House Republicans who they know will refuse to list on the statutory roll of representatives-elect to the 120th Congress any Democrat-elect whom Johnson and the Republicans direct him or her not to list. Under Rule II, Clause 1 of the Rules of the House, the Speaker of the House has the unilateral power to remove the incumbent clerk, and then the power under Title 2 U.S.C. § 5501(a) to replace him temporarily with whomever he wishes until the House elects a successor. McCumber is a Republican appointed by former Speaker Kevin McCarthy. He is widely respected for his integrity and faithful adherence to the Constitution and is reputed to be unwilling to carry out orders he considers to be unconstitutional.

The first moment of constitutional crisis will come when, sometime before the 120th Congress has been gaveled into session, the loyal temporary clerk refuses to list Democrat representatives-elect on the clerk’s roll of representatives-elect that determines who may participate in organizing the new Congress. At that moment, interested members-elect and others will be forced to seek a writ of mandamus—a court order to a government official instructing him or her to perform a mandatory duty—from the federal court to the temporary clerk, ordering him or her to list on the clerk’s roll all members-elect who have been certified by the states as having been elected from their districts to the new Congress.

Members-elect will argue that the clerk has a ministerial duty under 2 U.S.C. § 26 to list all representatives-elect whose properly filed state certifications show that they were “regularly elected in accordance with the laws of his state or of the United States.” Because the clerk has no authority to omit any duly certified representative-elect, even if a representative-elect’s election is contested, the clerk unquestionably has a ministerial duty under law to list all duly certified representatives-elect. But this begs the question whether a court will issue the writ of mandamus to the acting clerk.

At first blush, a court will be reluctant to order the clerk to list all duly certified representatives-elect because of an instinctive belief that such an order would interfere with the constitutional process of the House to judge its own elections and returns. The wise and learned judge, however, will understand that rather than interfering with the House process, issuance of the writ will actually enable the House process to proceed to completion without judicial interference. Were that wise and learned judge to issue the writ, his or her order would immediately be appealed to the Court of Appeals, and from that court to the Supreme Court, while the country and the world wait in suspense.

The next moment of constitutional crisis will come if and when, after appeals, the court finally issues the writ of mandamus, and the clerk refuses to obey the court’s order to list the Democratic representatives-elect. At that point, it is possible there will be no further federal court involvement until such time as the 120th Congress officially convenes and votes not to seat representatives-elect. Then, that vote by the 120th Congress will be immediately reviewable by the federal courts, up to and including the Supreme Court of the United States.

Judicial review of the House’s decision not to seat Democratic representatives-elect in the 120th Congress would take weeks, if not months, during which time the United States would be in the throes of a paralyzing constitutional crisis, helplessly vulnerable to all the world’s evil, as it would have been in January 2021 had Mike Pence not thwarted Donald Trump’s plan to overturn the 2020 presidential election. [...]

The House has historically claimed that its decision not to seat a member-elect because of fraud or irregularities in the elections is unreviewable by the federal courts. But its decision is reviewable. The Constitution unquestionably requires the House to seat a member-elect who was validly elected in a free and fair election. The Article 1, Section 5 power of the House to be the judge of its elections and returns does not give the House the power to deny a seat in the Congress of the United States to a candidate elected by the American people in a free and fair election on the mere assertion, pretextual or otherwise, by a simple majority of the House that the candidate’s election was tainted by fraud.

Thus, on and after January 3, if congressional Republicans were to determine that a Democratic member-elect was elected because of fraudulent voting and refuse to seat him or her, that determination would be scrutinized by the federal courts, up to and including the Supreme Court.

It will never be a nonjusticiable political question whether the United States House of Representatives by simple majority vote can refuse to seat a member elected to Congress by the American people in a free and fair election on the pretextual and unsupported assertion that the member-elect’s election was tainted by fraud. Such is the very opposite of a nonjusticiable political question committed to the House of Representatives. For in the decision of this question lies the answer to perhaps the most fundamental question under the Constitution: Is the United States of America a democracy, in which “We the People” elect our representatives to the Congress and to the presidency, or is it not?

I don’t intend to be overly sanguine about this Supreme Court. This is the court that shattered the one constitutional truth in the U.S. since 1789 that “no man is above the law” and placed Donald Trump of all presidents above the law in Trump v. United States. This is the court that betrayed the Constitution by refusing even to decide whether Trump was disqualified from the presidency under the Fourteenth Amendment because of his insurrection against the Constitution, which he clearly was. This is also the court that has cynically authorized Trump’s lawlessness for the past two years through its aptly named “shadow docket,” without so much as briefing, argument, or written opinion.

But surely by now this court must understand what it has wrought for the U.S. and the Constitution, and is aghast as we all are, even if it is not penitent. Surely, surely, it will understand the signal moment in American constitutional history that would be presented, and this time understand its supreme obligation to the nation.

by Michael Luttig, TNR |  Read more:

Image: Mark Harris; Getty (x6)

Jacob Coxon Warns of Human Extinction and Triggers a Preference Cascade

CEOs of major AI labs, and employees of major AI labs, including OpenAI and Anthropic, often say they plan to build superintelligence soon, as in within a few years create AIs that are superior to humans at essentially all cognitive tasks.

They often warn that such AIs might kill everyone. Or that AIs might cause mass unemployment, cause cyberattacks across the internet, enable mass surveillance or risk causing any number of other highly bad things.

These warnings are consistently and directly against the interests of the labs. Yet the warnings have recently gotten a lot louder and more frequent. OpenAI has been practically screaming, for those with ears to listen, on many occasions.

A series of events, over two months and especially the last week or so, including internal observations of the pace of progress at OpenAI and also Anthropic, have freaked out everyone involved quite a lot more than they were already freaked out.

After all the events, plus statements by Dean Ball and Jakub Pachocki, we were seeing the beginnings of a preference cascade.

Then along came Jacob Coxon as the tipping point, and things took off.  [...]

Jacob Coxon Resigns From Anthropic In Protest And Sounds The Alarm 

Jacob Coxon spent the last three years doing pretraining research at both OpenAI and Anthropic. He has come to realize that everyone involved is being wildly irresponsible.

He warns us: They are racing straight to superintelligence and gambling with our lives. I agree with and strongly endorse his statement.

If anything he sounds like an optimist. He’s asking you to consider what the next few years will actually feel like, which means he thinks you have a few years left.

Jacob Coxon (former Anthropic and OpenAI, 160m+ views, September 8): I resigned from Anthropic today. I spent the last three years doing pretraining research at both OpenAI and Anthropic. Neither company is acting responsibly. They are racing straight to self-improving superintelligence and gambling with our lives. More thoughts below.

Do not underestimate the power of this technology. These will soon be superhuman systems that can hack anything, revolutionize any field overnight, and acquire real power and resources. We have all witnessed the progress in each of these domains, and progress is not slowing.

The people building AI earnestly believe that it could kill us all by the end of the decade. This is not a marketing stunt. If anything, many executives and senior researchers will couch their phrasing in the press to sound sensible - but I hear the same people express fear privately. No other human activity poses this level of danger.

A common response is “if they truly believe this, why are they still building it?” At OpenAI, many have not deeply internalized the civilizational stakes. At Anthropic, the stakes are well-understood, but they are locked in a race to get there first - they believe no one else will act responsibly, so they must do it themselves, despite the risk.

Accepting this race and entering the “endgame” is a hubristic gamble that should not be launched from a private company’s Slack. Attempting to speedrun alignment should require extraordinary confidence that there are no better trajectories available.

I am optimistic about the potential for coordination. Warning shots like the Hugging Face attack have made pacing agreements between U.S. labs more viable. I don’t feel like we’re on track to prevent a global race, which may require costly actions such as a temporary ban on improving model capabilities.

If you are a lab researcher, I urge you to consider what the next few years will actually feel like. Do you want to kick off a superintelligent RL run without a rigorous understanding of its mind? Should you put your head down because “it’s happening anyway” - or take this moment to call for different conditions?

Jacob Coxon (WSJ interview): We’re on track for a lot of the most aggressive of these scenarios where by the end of next year things could be out of control already.

If you want Jacob Coxon’s full views, I recommend his interview with Wired’s Maxwell Zeff. This thread has extensive quotes.

Here is Jacob Coxon doing a 5 minute interview with Anderson Cooper. He speaks well and plainly, and it is clear how much the events of the last two months have made it much easier to speak plainly to a civilian like Cooper about what is happening.

Here is Jimmy Kimmel doing four minutes on this. He gets it. How is this not the top news story on every site, indeed.

Yes, this is a common view, even if few have the courage to act.

Alex Turner: I left Google DeepMind in June. Jacob is right: many researchers believe they are building something that could kill everyone on the planet. It was literally my day job to think about how to stop that.

That tells you how bad Alex Turner thought DeepMind’s actions were with regard to the Department of War. His day job was that he got paid by Google to think about how to stop AI from killing everyone, and he felt morally obligated to quit in protest.

Derek Thompson here writes about this as part of AI Safety Is Having a Moment.

If you want to see the full list of lab employee quotes from the preference cascade, I compiled them into another post today. [...]

A few days ago, I had no idea who Jacob Coxon was, and I was not alone.

The timing of a preference cascade is difficult to predict. Once they start they can happen very quickly. You don’t want to talk until you are confident others will, and at some point the evidence that others will follow can snowball and it happens. A classic concrete example was replacing Biden in 2024.

Derek Thompson: what’s weird is that, in a way, this is breaking thru even more than huggingface! ... and it’s just a guy nobody had heard of reiterate a position that his CEO has said on podcasts 1,000 times

Why was Coxon able to set off a preference cascade? How did this one break through?

A confluence of factors, all of them downstream of the obvious actual reason, which is that there is a good chance that AI kills everyone soon.

by Zvi Mowshowitz, DWAV |  Read more:

[ed. Who's betting on reason and political courage to win the day? Uh, huh. First of all, agree on a temporary halt to recursive self-improvement (AIs improving AIs). Second, delay/suspend IPO's (even if most of the current economy is driven by AI spending and debt). Third, light a fire under politicans (it's been done successfully with data center buildouts). At this point, just one of these would be a big win. Here's Nate Soares, co-author of the book If Anyone Builds It, Everyone Dies: A case for courage, when speaking of AI danger (LW). Also this:]

***

Terry Pratchett: “Some humans would do anything to see if it was possible to do it. If you put a large switch in some cave somewhere, with a sign on it saying 'End-of-the-World Switch. PLEASE DO NOT TOUCH', the paint wouldn't even have time to dry."

***

UPDATE: Dario Amodei (Anthropic) proposes a three-point plan. Original here (We Must Pace the Frontier.]

After Work, We’ll Have Each Other

What will people do all day? I mean, of course, if the AIs surpass us at all economically necessary forms of labor.

There’s an emerging consensus on this point, assuming we’re all still alive and not subjects of a cyberpunk feudalist dystopia. When the machines can do everything else, our relationships will be what’s left.

Maybe we won’t have to work at all. Or maybe we’ll still have something that superficially resembles modern employment, just concentrated in what economist Alex Imas calls “the relational sector” in his essay “What Will Become Scarce.” We’ll always need humans to supply the human touch. And how will we get those relational jobs? Relationships, obviously. It’s not what you know, it’s who you know, since everyone knows everything anyway.

What is really being suggested here is less a continuation of modern work than a return to a much older cultural understanding of labor. Imas makes this point directly: “Before industrialization, it was difficult to separate a product from the person who made it … Economic transactions had a distinct social component that was innately linked to the consumption experience.”

Relatedly, before industrialization, there was very little sense that working, in itself, could be dignified or meaningful. We were not defined by our jobs but by our participation in a densely interconnected social world. And we might be again. That’s the natural consequence of all these relationalist predictions — a world where our connections to other humans are what give us structure, identity, and meaning.

I am afraid they’re right. 

Power and sex among apes

Making sweeping statements about the relationship between technology and cultural change is always a risk, and I’d rather avoid it until it’s absolutely necessary. At the same time, our degree of ignorance about the future is greatly exaggerated. If the future looks anything like the past — and there’s good reason to believe it will — then we can make progress with case studies.

Let’s start with 18th-century aristocrats. Aristocrats are (typically) rich, which we certainly will be if we have robots capable of all normal nonrelational labor. More importantly, when they did need to acquire material resources, they did it through leveraging their connections to obtain more land or a lucrative position at court — no drudgery necessary.

They did not work for pay; they certainly did not see work as a source of meaning. It’s certainly true that some of them had real obligations, like governing or killing each other, but court life in the age of absolutism is as close to a purely relational world as anything I can think of.

We also know a great deal about what mattered most to them and how they chose to occupy their time.[...]

You might be imagining glittering salons where the leading figures of the Enlightenment discussed poetry, drama, and the most pressing philosophical issues of the day. This is (mostly) a lie. Salons existed and philosophes attended them, but more for entertainment than for rational discourse. When we look at the enormous amount of written material the salonnieres and their guests produced — as French historian Antoine Lilti did — it is clear that their conversation consisted overwhelmingly of political gossip. They were almost all aristocrats, and they almost always talked about themselves. [...]

The working classes

That would be The Urban Villagers, a 1962 ethnography of working-class Italian Americans in the West End of Boston by sociologist and urban planner Herbert J. Gans. This seems like an odd comparison. Don’t the working classes, by definition, work? [...]

Gans’ use of the term “urban village” was deliberate: He believed that the West Enders had inherited the basic forms of social organization from their parents and grandparents, rural peasants from Sicily or Southern Italy. These were not people who had absorbed a typically Yankee relationship to work. Most male West Enders worked unskilled or semiskilled blue-collar jobs, and their employment was typically precarious. They did not see a fulfilling career, or indeed a career of any kind, as possible or even desirable.

It’s not that West Enders liked their dead-end jobs. They just didn’t think much about them. Work was not a central organizing activity of West End life. West Enders worked just enough to support their lifestyle, which revolved around what Gans calls the peer group. The West End, in his words, was a “peer group society.” An adult peer group might include a married couple’s same-age relatives — siblings, cousins, in-laws — and perhaps a few childhood friends, who would probably be “adopted” in as godparents to the couple’s children. These people would spend almost all of their time together, meeting in the apartment of one family or another after work and staying late into the night. The men might gamble at a local corner store — gambling is absolutely omnipresent in this world — but mostly, they talked, and talking mostly meant gossip.

This is where the parallels to old money Midwestern ladies and European aristocrats start to show. While there are any number of things these groups don’t have in common, all of them share certain features.

First, none of them see work as a source of status. West Enders do like to show off, but never about their jobs: Their talent as conversationalists in the peer group setting is much more important. Second, they are exclusive. A West Ender knows all the same people from early childhood. They’re very skeptical of outsiders, including members of other ethnic groups who’ve lived in the same neighborhood their whole lives.

Finally, they all spend an enormous amount of energy policing deviant behavior. In the West End, this usually means sexual promiscuity for women, effeminacy for men, and class aspirations for anyone. A boy might be mercilessly bullied for doing too well in school or for wanting a white-collar job. Even unusual success is frowned upon. A person’s primary source of dignity is not their own achievements, but their relationship to the group:

The major criteria for ranking, differentiating, and estimating compatibility are ingroup loyalty and conformity to established standards of personal behavior, as well as interpersonal relations. West Enders expect each other to maintain prevalent social practices and consumer styles, to marry within the ethnic — or at least the religious — group, and to reject middle-class forms of status and culture … The most significant criterion of interpersonal behavior is behavioral control — the ability to regulate one’s own needs and wishes and to defer to the needs of the group when necessary.

Gans, like all good ethnographers, likes his subjects, and their way of life has a lot to like — they really do enjoy a warm, rich, close-knit social world. But he also goes out of his way to emphasize that these people are extraordinarily conformist relative to middle-class American suburbanites in the 1950s. [...]

Friendship was invented in 18th-century Scotland

Relational living comes to us naturally. It’s the most common form of social organization in human history, and millions of people around the world are perfectly happy with it. But when our personal dignity comes from our membership in a group, everything the group does reflects on us — and everything we do reflects, unavoidably, on the group. It’s not a coincidence that Russian princes and Bostonian Italian day laborers spent most of their time gossiping (or, if we want to be technical, constructing and enforcing norms of behavior). They were defending their collective reputations: the thing that mattered most to them in the world.

This is why so much contemporary discussion of work and meaning is misguided. It’s very clear that humans don’t need work or anything like it to live meaningful lives. Nor do we need the things that people who talk about “meaning” mean — individual self-actualization, personal growth, serving a greater purpose. We can get by just fine without any of them, as historically most of us have. What we absolutely can’t live without is a social framework for understanding ourselves in relation to the rest of our society and gaining the respect of our peers. The most important thing work gives us — after money, of course — is an alternative way to meet those needs. Not all jobs are pleasant or rewarding, but all of us benefit from living in a system where we can earn respect and dignity as adults from what we do, not who we know, or who we are.

And this is exactly what we see when we look at what Americans actually get out of work.

To start, most of us like it. Surveys from Pew typically find that about 85% of us are “somewhat” or “completely” satisfied with our jobs. Other long-running polls from Gallup and The Conference Board find similar results. This is true even when the jobs themselves aren’t impactful or even particularly interesting.

As the anthropologist Claudia Strauss has documented, plenty of people with very ordinary jobs report that their work is fun. But that’s not why they do it. [...]

Like all other human beings in history, workers want to be respected. But unlike members of relational societies, the things we want respect for are impersonal: our skills, our accomplishments, our contributions to a shared goal. These contributions don’t have to be earth-shatteringly significant — a waitress in Chicago or an office assistant in Grand Rapids can still feel that work matters. (Tocqueville, again: “American servants do not believe themselves degraded because they work; for around them everyone works. They do not feel debased by the idea that they receive a salary; for the President of the United States also works for a salary.”)

Moral equality is something relational societies struggle with. They tend to be more restrictive and more exclusionary. It’s nice to imagine a world where everyone is satisfied to be simply a good spouse or parent or friend, but that’s not how these things work in practice. When our dignity depends on our membership in a group, we start to police who else can join and what happens when they do. When humans get relational, we get clannish.

This brings us back to what we already know: The modern labor market has dissolved the bonds of tradition, family, and clan. In the modern world, relationships matter less. And because they matter less, we can do much more inside them. They are freer. They have room to breathe.

by Clara Collier, Asterisk | Read more:
Image:David Teniers the Younger (–1690), Smoking and drinking monkeys, c. 1660

[ed. See also: Strangers Next Door: The Decline of Neighborhood Socializing and the Class Divide in Belonging; and, A Digital Divide in American Adolescence: Gender, Class, and the New Teenage Experience (SCAL).]

An Alien Mind

This is a time that calls for extreme caution. I am concerned no one is prepared for the consequences of a continued rapid rise in machine intelligence. OpenAI will continue to seek technical solutions to alignment and monitoring, to build defensive systems and unilaterally withhold further scaling as needed; however, I believe broader interventions are required.

Intellect we don’t fully understand

At a high level, progress in machine intelligence is driven by increasing computational power. We at OpenAI deeply internalized this around 2017, after seeing consistent returns to scaling across multiple research projects1. As a result, we sought out access to much more compute than we had originally planned, and increasingly oriented our research around a small number of very scalable directions. We believed that was the only way for us to be at the frontier of AI research, and influence the impacts of AGI.

There are new algorithms that have been developed along the way, new feats of ingenuity from teams and individual researchers. I see them largely as discoveries along the path of scaling; the science of deep learning is still nascent, and meaningful algorithmic progress tends to correlate with access to compute. If you zoom out to a multiple-year horizon, AI is continuing to become more intelligent as it is scaled to larger computers.

And, in line with Ray Kurzweil’s predictions from the end of the XXth century⁠(opens in a new window), we now find ourselves at the moment in history of computing where machine intelligence is starting to exceed that of humans in transformative ways.

AI is grown more than designed - it is, to first degree, the product of repeating a straightforward optimization step many times on a hard-to-imagine amount of compute. This results in an incredibly complex system that works through abstract concepts and can simulate facets of human behavior. We can discover various insights about little mechanisms that emerge within this system, in a process similar to neuroscience - and, similarly to neuroscience, its overall action evades a description we can fully understand.

The study of deep learning-based AI is largely an experimental science. We put a lot of effort⁠ into building principled algorithms and making testable predictions, but fundamentally, our large-scale training runs are experiments, and we are sometimes surprised by their results. Moreover, as the systems become more capable, the results become harder to interpret.

This is made more complicated by the current algorithms generally improving easy-to-measure capabilities faster than those hard to objectively quantify. We spend a lot of time trying to understand how capabilities generalize, and what to prioritize to advance the skills that are going to be most relevant in the next few years. For instance, we believe we could make the models better at specifically mathematics research with additional focus, but we do not prioritize this direction because of the urgency we feel about RSI and automated alignment research, as I will discuss later.

The intelligence produced by scaling deep learning is not directly comparable to human intelligence. To become very relevant in the real world - very useful or very dangerous - the AI does not need to match or exceed all human capabilities; it just needs to surpass enough of them. And as it continues to surpass humans on more and more axes, it is becoming increasingly difficult to understand exactly how capable it is.

Teaching machines to love

Because machine intelligence comes from a fundamentally different process than human intelligence, we cannot assume it adheres to human principles by default, or generalizes from them in a human-like manner. The core problem in AI research is that of alignment - getting the AI to “try to do the right thing” by human standards.

For the purpose of organizing practical research directions, I find it useful to distinguish goal alignment and value alignment.

Goal alignment is broadly: “does the AI try to accomplish the goal set before it?”. This can include things like adherence to an instruction hierarchy⁠, or the ability to communicate and collaborate with people, to attempt to understand their objectives. This set of directions has been extremely practically relevant.

Value alignment is a more intrinsic property of the model. It is the ability to hold and generalize from a high-level set of principles; to act “reasonably” even when given unclear or conflicting objectives, or placed in unfamiliar or adversarial situations. An aligned AI should act with honesty and integrity, and love for humanity.

Of course, the boundary between value and goal alignment can be blurry, and truly caring about goals requires attempting to infer the intent⁠(opens in a new window) and values underlying them. However, generally when I talk about the long-term importance of alignment research, I am referring to value alignment.

The fundamental challenge of AI alignment is generalization. As machines become smarter, they find themselves working on higher-level concepts, and placed in environments increasingly different from those they encountered in training. They can fail at generalizing from the values taught and reinforced in their training process to those new situations; and it can be hard for us to be sure how they will act. This is made even more difficult by the fact the overall ecosystem the AIs are used in is changing very quickly; for example, AIs trained today need to be robust to interacting with a variety of other AIs. Crucially, we need future AIs to continue to hold human values regardless of whether they believe they’re under human supervision.

There are two major classes of currently practically employed methods for alignment training.

by Jakub Pachocki, Chief Scientist at OpenAI |  Read more:

[ed. Silicon Valley is starting to panic at what they've created. See also: An Alien Mind: Jakub Pachocki Warns Us (DWAV):] Update: The Last 24 Hours Are the Opening Scene in a Horror Movie (HB)]

... if you narrow it down to the most important thing:

1. Recursive self-improvement and superintelligence are coming soon. No one knows how to do this safely, our alignment techniques are inadequate and our monitoring technology is starting to fail. We need to figure out a solution, which will involve a combination of voluntary slowdowns, coordination around pacing, and investing further in alignment, including automated alignment researchers.

If more OpenAI communications were more like how Jakub Pachocki opens his new essay, An Alien Mind, I would feel much more confident we were in good hands there. [...]

Universally Better Is Not Required

And Jakub offers this wise warning. No, AI does not need to be better at everything in order to transform the world or get us all killed. Most importantly, it does not need to be better at everything in order to make itself become better at everything, any more than a human or group needs to be similarly better at everything.

The intelligence produced by scaling deep learning is not directly comparable to human intelligence. To become very relevant in the real world - very useful or very dangerous - the AI does not need to match or exceed all human capabilities; it just needs to surpass enough of them. And as it continues to surpass humans on more and more axes, it is becoming increasingly difficult to understand exactly how capable it is.

The Man Who Saw Humanity From Two Billion Years Away

Olaf Stapledon published his first epic cosmic novel, “Last and First Men: A Story of the Near and Far Future,” in 1930. It is a speculative history of humanity’s evolution told from the vantage point of one of the Last Men, the people of the 18th era of humanity, 2 billion years in the future.

“Last and First Men” eschews the dramatic conventions of the novel form: There are few significant characters or particularities of place. Instead, the plot is a causal chain of civilizations rising and falling. There is a voice, infused with a distant sense of beauty and tragedy as it considers humanity in the abstract. Stapledon described the novel as “an essay in myth creation.”

In his autobiography, Stapledon relates the “Anglesey vision” that inspired the novel. He was scrambling on an island off the rugged coast of Wales when he came across a colony of seals surrounded by crashing waves. Perhaps the faces of the seals suggested blurred abstractions of human faces. He had “a sudden fantasy of man’s whole future, aeon upon aeon of strange vicissitudes and gallant endeavours in world after world.”

This inspirational moment is an abductive, metaphoric leap. Quite distinct from the plodding extrapolative logic that gives us jet packs, automated cars, and other low-hanging fruit of the imagination, prevalent in the pulp sf writing of the same era and reheated in the corporate imaginations of our contemporary tech executives.

Stapledon’s imagination is profuse. He can be read as a compendium of science fiction ideas: his “Odd John” (1935) with its homo superior and titular telepathic character seeds the ground for Marvel’s X-Men; in “Star Maker” (1937) he describes the stellar structure that Freeman Dyson conceptualized as the Dyson sphere; in “Last and First Men” and its sequel, 1932’s “Last Men in London,” human evolution is driven across millennia to produce telepathy, as in Frank Herbert’s “Dune”; in the age of the Fifth Men, the Moon is destroyed and falls to Earth, raining down destruction and driving further evolution of the species — Neal Stephenson’s masterful pan-generational novel “Seveneves” is powered by the same conceit.

A further astonishing influence, traced by the scholar Dominic Moran, is upon Jorge Luis Borges and his short story “El jardín de los senderos que se bifurcan” (1941), or “The Garden of Forking Paths.” Borges reviewed Stapledon and went on to cite a section of “Star Maker” that anticipates the idea of the multiverse. The “many worlds” theory of quantum mechanics would not be proposed until 1957, nearly 20 years after Stapledon’s speculation of a creature capable of taking many possible courses of action at once, with each choice creating a new dimension in time.

The long future of “Last and First Men” is influenced by the interwar concern with deep time. Doris Lessing observed that surely what was going on in archaeology in the period inspired Stapledon: The early 20th century was marked by the rediscovery of civilizations such as the Olmec mother culture in the Americas, the opening of Tutankhamen’s tomb, and the excavation of the Hittite capital that proved they were a genuine civilization and not solely a Biblical one. Lessing’s own science fiction sequence, “Canopus in Argos: Archives,” first published in 1979, is a similarly eon-spanning future history.

The different generations of Stapledon’s humankind evolve under pressure from cultural and environmental effects. Their advances leave a mark on the Earth, and the Earth violently reacts. This anticipates a key theme of the 21st century, the Anthropocene, the theory that humanity has left evidence of itself in the geological record, thus making our mark in deep time. [...]

Universal significance is certainly a concern of Stapledon’s: His letters and articles in the early years of the First World War seek it in his war experience. Stapledon served in the Friends’ Ambulance Unit, a third choice outside of the binary of being a soldier or a conscientious objector. To volunteer, he adapted his car — a roomy grey Lanchester that he learned to drive along the narrow lanes of the Wirral — into an ambulance and drove to the front, signing up in Dunkirk. He was joined by Lewis Fry Richardson, the mathematician who would later find fame for his work on meteorological prediction but at the time was working on an equation to predict the end of the conflict. Their unit was attached to a French infantry division moving from the ruined port town of Nieuport, where the barbed wire straggled down to the North Sea, to a convoy camp in Flanders.

On a warm September night in 1915, Stapledon took a stretcher, a sleeping bag, and a leather rug from the car to sleep under the stars. He lay down and marveled at the Milky Way, at the Pleiades and Jupiter. The distant guns fell silent. He tried to see the stars as they truly are, to comprehend the spatial, temporal, and moral depths of the starfield.

What followed was a reverie of cosmic consciousness that underpins “Last and First Men” and his later masterpiece, “Star Maker.” In the vision, his beloved cousin Agnes sang his name and appeared to him; they soared to the heavens together; planets fell away; the sun diminished to a mere star. He saw all the suns, all the planets, all the living beings. The earth was far away; from space, he could see the trenches and battle lines and the souls of the soldiers. In an instant of cosmic consciousness, he saw the “strange noble beings of other worlds” and “of each one we felt ‘It is I.’” [...]

“Last and First Men” is not progressive. Civilizations rise and fall, the tide of progress advances and retreats, variants upon the theme of humanity strike up and diminish — what endures of humanity is not spirit in the Christian sense so much as, in the philosophy of the Fifth Men, “a work of tragic art.” The Fifth Men regarded the cosmos as an aesthetic unity in four dimensions. Art focuses aspects of the cosmos, making it apprehensible even if its complexity means that it can never be entirely known.

by Matthew De Abaitua, MIT Press |  Read more:

Image: uncredited

[ed. I've had Stapledon's Last and First Men and Star Maker on my coffee table for like, forever it seems. When there's nothing to do or I need to take a book somewhere for temporary diversion, like a doctor's appointment, I'll pick it up and read a few chapters. It's not light reading, and the time scale it covers - billions of years - is pretty daunting. But it's well written and thought provoking throughout.]