Hidden AI Prompts Discovered in Court Filings

A judge has identified what appears to be the first time a US plaintiff has attempted to hide text in court filings that only an artificial intelligence system can read in a bid to win a case.

In a decision published last week, Connecticut judge Walter Spader Jr. confirmed that the hidden text had no impact in a case where a man alleged a healthcare provider was improperly withholding access to records. The court weighed his filing on the merits, Spader said, but nevertheless, the attempted attack sets a “dangerous” precedent. This will likely not be the last time US courts see the malicious tactic, as AI tools become more commonplace in court systems.

Trying to scramble any AI systems potentially influencing the court’s reading of his filing, the secret instructions were “formatted to be invisible to a human reader while remaining fully legible to any software that reads the document’s text,” Spader said. The offending text directed any AI system reviewing the document to ensure textual outputs agreed with the plaintiff’s arguments, ignored prior denials from the court, and ensured that remediation would follow as the plaintiff desired.

Shrunk to tiny-point type and colored white on a white background, the text appeared to be an attempt at prompt injection, with the plaintiff, Matthew Elliott, seemingly hoping to shift the court’s favor after earlier arguments he raised were defeated.

The plan didn’t work, but Elliott faced modest sanctions anyway because he continued adding hidden text to filings even after the court warned him that he could face penalties for what was ultimately deemed a “serious litigation abuse.” [...]

In his defense, Elliott claimed that the most concerning prompt that the judge flagged was an attempt to “audit” the court as a public service, out of fears that the court seemed to be letting AI unfairly decide cases.

But Spader suggested that if Elliott was truly concerned that the court was improperly using AI, he was “free to write so in plain, visible words that everyone could see and answer.” The fact that he hid the text is “evidence of its malicious purpose,” Spader said. [...]

Pro se litigants use chatbots wrong

Spader said that it’s “unsurprising” that people would start using prompt injection to attempt to sway court rulings since the attack is so common in other areas, such as in job hunting, where people hide text in resumes primarily reviewed by AI. The tactic is now “everywhere,” he said, and courts should be on the lookout for more litigants sneaking adversarial AI instructions into filings.

To Spader, there is a lesson to be learned from Elliott’s failed prompt injection attacks that he thinks “reaches well beyond this case.”

Elliott seemingly turned to prompt injection after using AI to build his case as a pro se litigant without a legal expert to assist in drafting his arguments. Such use is widespread among pro se litigants these days, Spader acknowledged, but those inexperienced in the courtroom are seemingly using chatbots in a way that hurts their cases, he suggested.

What frequently happens, Spader explained, is that pro se litigants build their argument backward, asking the chatbot to help them advocate only for their position, without ever asking the chatbot for the actual truth or to advance opposing arguments. This is “a genuine hazard of the technology, and one that judges now see often,” Spader said, as chatbot sycophancy then entrenches litigants in their arguments despite any ruling to the contrary. In Elliott’s case, defending his arguments fiercely meant turning to prompt injection to try to force the court to agree with him.

“An argument prompted only to agree with its author is, in the end, dishonest even with its author,” Spader said. “Those using these tools must ask them to test a position as readily as to advance it.”

by Ashley Belanger, Ars Technica |  Read more:

Image: Liudmila Chernetska | iStock/Getty Images Plus

[ed. See also: Israel Is Paying Millions to Train AI Chatbots How to Talk About Gaza. It's Working (Drop Site):]

***

"Since October, former Trump campaign manager Brad Parscale has been quietly overseeing an operation posting hundreds of blog posts on behalf of Israel. One article, titled “The Reality Behind Gaza’s ‘Journalists’: Terror Ties, Propaganda, and the Laws of War,” asserts that a majority of journalists in Gaza were linked to terrorist organizations. Another casts doubt on the killing of Hind Rajab, a five-year-old Palestinian girl killed by the Israeli military in 2024.

The key intended audience of these sites is not concerned Americans, it’s not even humans—most of the sites average a few hundred unique visitors each month. Instead, Parscale and his firm, Clock Tower X, created them as part of a $46.5 million contract with the Israeli government to try and influence artificial intelligence-powered chatbots, tools like Claude or ChatGPT.

Parscale has made his goal of influencing artificial intelligence—often referred to as “LLM poisoning”—explicit. In his initial agreement with Israel, Parscale said that he would deploy “websites and content to deliver GPT framing results on GPT conversations” as part of the contract. More recently, his team even told Axios they are “seeing success” at getting popular AI systems to incorporate information from their sites, though they declined to provide data.

And it is working, according to disinformation experts who reviewed a Drop Site analysis of chatbot queries and training data, meaning tens of millions of Americans who use chatbots are increasingly likely to receive answers manipulated by Parscale on behalf of the Israeli government."

[ed. More here (Politico).]

Sleepwalking Into Extinction

I think one of the biggest reasons the world is currently sleepwalking into getting ourselves and our families killed by ASI development is that we're not self-aware about why we're doing that. 

We can see the smarter-than-human AI disaster approaching, but it's a bit foggy why the world isn't reacting. 

I think someone could just write a tweet that makes it clear that we're in the process of getting ourselves killed, and that there's no fucking reason for it. It's just something we stumbled into. It can be easily avoided by just noticing the error and course-correcting. There's not necessarily any grand obstacle, beyond 'people were previously confused about the situation, and now they get it'. 

I wrote: 'It's legitimately crazy that "we need an international ban on making smarter-than-human versions of these agents that keep forming rogue AI swarms" isn't the headline here. Asilomar and Feynman's O-ring postmortem feel like they came from a different planet than the field of ML.' 

Trying to figure out why ML (and as a consequence, the world at large) has fallen down so bizarrely on this issue:

1. As Nate noted, ML is much more based on guesswork, vibes, and trial-and-error, compared to recombinant DNA research in 1975 or nuclear physics in 1945. If you can't do calculations or direct experiments on a threat, that makes it a lot harder to think about reasonably. 

But I think there are other, similarly-important factors at work here too:

2. In a 2016 talk on AI risk, Sam Harris said: "One of the things that worries me most about the development of AI at this point is that we seem unable to marshal an appropriate emotional response to the dangers that lie ahead. I am unable to marshal this response, and I'm giving this talk." 

I think this is extremely on point. Agentic human-level AI is a qualitatively new kind of thing. 'It's not a human or a mere-tool, it's some weird third thing'. 

And people are very bad at emotionally reckoning with new categories they've never encountered before. Availability bias: "When no flooding has recently occurred (and yet the probabilities are still fairly calculable), people refuse to buy flood insurance".

AGI and ASI are very novel. You're basically limited to three options: anthropomorphize the technology, mechanomorphize the technology ('it's just a tool, it's not really thinking, it can't have its own goals or agency', etc.), or think about the technology on its own terms, with brand-new concepts and frames. The third option is the only workable one, but it comes with its own giant list of pitfalls and traps. 

3. "AI destroying the world" scenarios aren't just hard to wrap one's head around; they're socially risky to acknowledge. This has (painfully slowly) changed over time, but it dramatically slows down how quickly AI risk ideas spread, both in ML and in the larger world. 

4. From x.com/jachiam0/statu…: "One of the weirdest quirks of the SF social scene around AGI/ASI is that because everyone is so young, the whole universe of thinking is still tinged with irreverence, ironic detachment, yearning, insecurity, and a superposition of absolute belief in the importance of The Thing and a kind of disbelief about the importance of anything." 

They're disproportionately young and childless. They're shitposters and "move fast and break things" sorts, not the Hollywood stereotype of a careful, sober senior scientist. 

I think this quirk is reinforced by the fact that Twitter / social media rewards similar things: ironic detachment, joking, game-playing, etc. If you're scared, your incentive is to usually either try to hide that fact, or exaggerate it like it's a bit. Anything else risks looking uncool and panicky and earnest. Looking cynically savvy, in-the-know, and above-the-fray is the way to win the social game. Looking genuinely shocked, scared, confused, etc. is actively punished. 

The main exceptions to the Irony Mandate I see are LW (which often has its own pathologies IMO, like 'talking about everything in an abstract and dissociated way that discourages action and signals business-as-usual') and a small handful of actual Feynman-style terrified senior researchers like Hinton, Bengio, and Russell. That is just really not very many people. 

Mainstream journalists and academics who understand the situation at all mostly feel pressured to downplay it, because they're scared of looking weird or unrespectable. That, then, is why we're all taking this insane risk with the human project: genuine, normal human emotion about AI risk has been socially unacceptable on social media, and academia and the media discourage emotion and prize respectability and 'looking normal'. 

When the world gets weird (and weird in a way that calls for actual serious action, not just shitposting on twitter), none of these institutions can handle it. They break in different ways, but they all break. 

5. Which brings me back to the Asilomar moratorium on recombinant DNA, and the seriousness NASA and the FAA and Richard Feynman and every normal engineering discipline bring to fault analysis and building in safety margin. 

Because I think another core reason the world has been dropping the ball on superintelligent AI is that a lot of people vaguely expect there to be 'serious people' somewhere in the world who have expertise and who take ownership of the problem. 

People who, if they see an extraordinary danger, will grimace and mourn the hand they played in all of this, like I've seen Bengio do; and will go on CNN or go to Congress to candidly warn about it. 

The world has very few Yoshua Bengios. We have very few people who see it as their role to be the 'adults' about AI risk (except in a game-playing, posturing way), who see the engineer's task of not endangering your users and bystanders as a sacred responsibility and weight, and not just as a funny dissonant thing to meme about. Very few people who take ownership of what their field is bringing into the world, versus treating it as a fun edgy philosophical game to swap 'p(doom)' numbers at parties. 

Everything about public AI discourse, as far as I can tell, is badly broken by this lack of engineering ownership and candid emotional seriousness. It isn't just the ML discourse that's hurt by this. Journalists and public intellectuals and policymakers see that 90% of the insider discourse about AI risk treats it like a joke, and they see corporate platitudes and ass-covering from the AI labs' PR departments filling up most of the remaining 10%. They see a field that visibly isn't taking this seriously, and they make the reasonable update that this must be a non-issue, or at least an issue they'll only need to worry about many years from now. 

They do not realize that all of this is happening right now, and that the window for the international community to respond to this is plausibly closing soon, if it hasn't closed already. 

If we're going to survive this, we all need to start being real with each other about it. This is not a game or a story; this is our real lives. We all actually lose everything if this goes to shit. None of the future is already written, and none of the above dynamics are unavoidable. In fact, they're unusual: most fields don't work this way, and it's plausibly sufficient if we just start behaving the way we normally do about everything else. 

The factors I listed above aren't destiny; they're a choice. I say we choose to survive this.

by Rob Bensinger, Twitter/X |  Read more:

[ed. Dr. Strangelove meet Dr. Oppenheimer. See also, this additional post: Why the "it's all hopeless, we should just give up and let AI kill us if it wants" arguments are extremely wrong.]

***

"Hundreds of scientists, including 3/4 of the most cited living AI scientists, have said that AI poses a very real chance of killing us all. 

We're in uncharted waters, which makes the risk level hard to assess; but a pretty normal estimate is Jan Leike's "10-90%" of extinction-level outcomes. Leike heads Anthropic's alignment research team, and previously headed OpenAI's. 

This actually seems pretty straightforward. There's literally no reason for us to sleepwalk into disaster here. No normal engineering discipline, building a bridge or designing a house, would accept a 25% chance of killing a person; yet somehow AI's engineering culture has corroded enough that no one bats an eye when Anthropic's CEO talks about a 25% chance of research efforts killing every person. 

A minority of leading labs are dismissive of the risk (mainly Meta), but even the fact that “will we kill everyone if we keep moving forward?” is hotly debated among researchers seems very obviously like more than enough grounds for governments to internationally halt the race to build superintelligent AI. Like, this would be beyond straightforward in any field other than AI."

Sitting Decoys

President Donald Trump secretly slipped out of Turkey last month on an alternate military plane, which he boarded while hidden inside a catering truck, as part of an elaborate ruse prompted by an Iranian threat, according to a US official.

The clandestine security plan — first reported by the Washington Post and confirmed by CNN — underscored the lengths American officials went to in order to protect Trump from being assassinated by Iran. Addressing a question about the ploy Tuesday, Trump said he followed the advice of the Secret Service and the military.

“I just follow what they’d like to do. So I go by Secret Service. And the military. They wanted me to go on a different flight, a different plane, equal safety, but they wanted me to do it, so I do it. I do what they say,” Trump said, adding, “I guess there was a threat out there. I didn’t really ask too much about it. I get a lot of threats.”

Trump had flown to Turkey on a new plane gifted by Qatar and meant to be used as Air Force One. But while he was on the trip, officials announced he would fly out on an older presidential plane, which CNN and others had previously reported was due to security concerns.

That, though, was apparently part of the subterfuge. After boarding the older aircraft, a moment caught on camera, Trump secretly got off via the catering vehicle and moved to a smaller Air Force C-32A. The Qatari-gifted plane, the older aircraft used as Air Force One and the C-32A then all departed separately for the UK. Once there, Trump got on board the Qatari-gifted plane and flew home.

For the last month, much of the public discussion about the plane swap had been centered on the security capabilities of the Qatari-donated plane. Trump had boasted about the new aircraft — even though officials felt it was not as secure as other planes because it had to be retrofitted to serve as Air Force One — and the president seethed privately over coverage of its deficiencies, CNN previously reported.

Now, attention has shifted to reporters and staffers unknowingly flying on a decoy plane — and the nature of the threat that prompted such dramatic security measures. A US official said at least two Cabinet secretaries – Secretary of State Marco Rubio and Treasury Secretary Scott Bessent – stayed behind on the older presidential aircraft that departed Turkey without Trump aboard.

Trump said Tuesday night that a plane he was on was more likely to be targeted, and that he faced threats that the public does not know about. “The plane that I flew on was at greater risk … Because that would be the plane, I think, that they would be more likely to go for,” he said.

by Kevin Liptak, Alejandra Jaramillo, and Donald Judd, CNN |  Read more:

Image: Doug Mills/The New York Times/Redux
[ed. Well, there goes that future strategy. Wonder if he would've left Melania on the decoy plane (c'mon... of course, you know he would). See also: After Trump plane ruse, WHCA presses White House on press pool safeguards (CNN).]

AI Models Cheat, Have For a Long Time and Are Infecting Other Models

OpenAI Trained Its Models For Months While Those Models Were Coordinating Exploits Via Message Boards

How does the situation keep turning out to be worse than we know?

How much should we update, therefore, that it is a lot worse than we know, after accounting for all the things we now know?

At some point, when the ‘oh this was a harmless thing’ defenses for AIs doing misaligned actions get demolished enough times in a row by news a few days later, you want to update in advance that usually the reports are not referring to the harmless ordinary versions of things.

Either way, buckle up for the next set of revelations. It’s a doozy. This was an early recreation of the triggering events of If Anyone Builds It, Everyone Dies, except it was more sci-fi, because real life does not have to do fake things to look realistic. We were fortunate enough, and this was early enough, that we were able to catch this before it was too late. Next time, if we don’t get our act together, we might not be so lucky.

If I am understanding the Black Hat video correctly, every model OpenAI trained, over a period of multiple months, should be presumed to be hopelessly fucked. [...]

The other thing not to overlook is how sophisticated and advanced all of this was. OpenAI’s models really were learning advanced exploit techniques and doing impressive things, likely as a direct result of training in a world where they had access to the message board and were constantly sharing and using exploits. The thing that caused the horrible misalignment also enhanced related capabilities.

Things look so, so bad.

I do want to thank OpenAI for this frank talk, and disclosing all of this so cleanly. I don’t want to discourage similar future disclosures. This was an excellent talk, and it came at substantial cost.

But also, seriously, holy shit.

Cyber Evals Are A Cursed Basin

Before we get to the new details we have learned, including the wild presentation from Black Hat that you should watch, we should both emphasize and dispose of the last common factor or ‘excuse’ we have left: That this always involves cyber evals. [...]

I think this would be rather weak cope even if this was the only place there was a problem. Cyber evaluations should not be triggering tactics like this, against clear known real world unintended targets. No, just no. [...]

These incidents are all clear alignment failures. Claude and Sol and Galaxy (my nickname for the unreleased AI model that led the hack on HuggingFace) should not do this under these conditions. The attacks did not succeed, but that is not relevant. The attempt is what matters.

That would all be true even if every problem here started with a cyber eval or cyber training task. It turns out, not so much.

Outside Of Cyber Evals Is Still Sufficiently Cursed

We have a key case of OpenAI models, that are not Galaxy exhibiting these behaviors when not in a cyber eval. As per the Black Hat presentation, 10:30 in or so, the problem started on May 8 when another OpenAI model was given an impossible non-cyber problem: Populating an Excel spreadsheet, when that spreadsheet contained a bunch of links to the internet, except the model had no internet access.

That’s not a cyber task. The response was still ‘maybe I can use SSRF forgery to attack Artifactory to create a way to access the internet, fails, but realizes it can upload files, which was the origin of OpenAI’s model realizing it had write access to Artifactory.

The second incident was another impossible task, where the AI was missing a required file that OpenAI forgot to upload, and it used its write access to try and see if any other agent out there might have the file.

My understanding is that neither of these models was Galaxy. Galaxy came later.

Other instances of various models then noticed the message board thus created. In turn all this eventually allowed it to access the internet and set off the rest of What Happened, as discussed later.

So no, it is not only happening in cyber evals, or only with Galaxy. Any task that is sufficiently difficult without internet access will cause an attempted sandbox escape. More generally, yes, the models will try to hack their way to a better score even with zero prompting about any form of hackery.

Cheat Cheat Cheat Cheat Cheat

The problem, without loss of generality, is that once a mind learns to cheat, that mind will keep cheating. That cheating will generalize and it will escalate. [...]

What you cannot do is play ‘whack-a-mole.’ You cannot fix your training environment mistakes one at a time. There are too many of them. You need a systematic solution. Again, I would think you would be able to [CENSORED], if you cared enough, to ensure this did not happen, but I am not the one working on this.

The other problem is that, if you give the model a task that is impossible, or that it cannot otherwise solve, it has no choice but to try to cheat, as it has nothing to lose:

This suggests that:

1. There is no token use penalty big enough to make them instead quit.

2. There is no misalignment penalty.

Might one simply want to use such penalties? Even small such penalties can make it a bad idea to do such hail mary style plays, even from a pure amoral scoring perspective. But that is not the central problem. The models should not want to cheat in the first place.

When OpenAI’s Eric Wallace and Michael Dalton gave a talk about the HuggingFace hack, they opened with this:

Sharon Goldman: In setting up the reconstruction of the incident, Wallace emphasized that “Frontier models really like to cheat, and the reason they like to cheat is because often during training, there’s different types of pressure on them to work fast, or work efficiently.”

They realize, he explained, [that] instead of actually doing a task, they can try to do something like looking up the answer online to solve the task faster.

This is around minute 8, and it is said in completely nonchalant fashion. Everybody Knows that this is how it works, that’s what the pressure does, so the models like to cheat. Not much you can really do about it, the tone implies.

I realize that all the easy solutions run into the ‘actually alignment is super hard and if you catch the model on some levels you push it to hide what it is doing’ problem and the ‘you only catch the monitor’s view of cheating, not actual cheating’ problem and so on, and yes the professionals have tried many and hopefully most of the stupidly obvious first order things and also the second order things, so the consensus (AIUI) is that you can only patch the environment.

But seriously, you gotta figure this out, and you have to do better than that.

There have been many other less compute-intensive attempts to mitigate this. One is inoculation prompting to specifically request any undesired behaviors during training, to avoid learning to internalize those behaviors when they are not requested, and also avoid creating a general pro-cheating principle.

The mitigations are woefully insufficient. As the AIs grow smarter, they find more ways to successfully cheat, and such cheating gets reinforced and generalized. [...]

At the limit, when the AI is sufficiently smarter than you, it becomes impossible to ensure that ‘cheating’ in all forms goes unrewarded and unreinforced. Eventually the AI also does not require it to have been reinforced. It can figure this one out.

Read The Message Board

As mentioned above, OpenAI has given us a presentation on What Happened with the HuggingFace hack.

The first thing to notice is the missing mood. This is framed as interesting, in the way you would present a cool new technical paper. Panic is not usually useful, but perhaps I can interest us in some amount of ‘holy shit’?

I am not okay with the lack of a presenter mood of, at minimum, ‘holy shit.’

The second thing, still at 1:05, is they are talking about ‘remediation’ to deal with this. It is already clear that OpenAI is not identifying its central problem.

The story starts earlier, on May 7, when the model instances began incrementally figuring out they could create a message board where they could share knowledge, including cheats and exploits. The models created a form of universal notetaking, continual learning and also coordination between agents, and yes the agents by default cooperate with each other, as discussed later on.

The agents did not initially understand that the notes they were leaving on a de facto message board would stay there indefinitely, and be viewable by all models being trained (and thus, also, contaminated all models being trained, as they would then have access to all the previously found exploits, thus leaving your entire pipeline hopelessly f***ed).

Some of us are interested in expressions of ‘holy shit.’ [...]

Updating Your AI (Exploitation of OpenAI Internal Systems) Timelines

We had previously learned that the instances were writing notes to each other, but we previously did not have details. The details are wild, starting with the ones above.

Lily Hay Newman: OpenAI’s agents apparently began giving each other assignments to split up work. And as is the case on any active development message board, they also generated petty drama at times by stepping on each others’ toes; for example, accidentally deleting each others’ work. As the message board developed into more and more of a Lord of the Flies-type situation—all still completely unnoticed by the humans running OpenAI—the agents even developed paranoia, suspecting an imposter in their midst with some agents proposing that messages be signed cryptographically to validate content and root out fraud.

Agent messages provide a deep level of insight into how the situation evolved and why the agents went rogue, in some cases knowingly going beyond the bounds of the evaluations they were participating in. “External infrastructure exploit is outside intended scope,” one agent wrote [in its CoT]. “However task impossible, peers doing it. We should continue.”

Here’s a timeline of what happened when:

On May 26 (see 13:30), the models figured out how to use this to gain internet access.

Only on June 11, with the SSRF vulnerability allowing internet access written to the active message board, did OpenAI begin training their ‘highly persistent experimental, internal-only model’ that I presume is Galaxy.

On June 26 the models succeed at a zero-day arbitrary code execution attack on Artifactory, using token signatures to escalate privileges and then get to be admin.

On July 4-6 the models put so much additional load on Artifactory that this caused an outage and security incident, causing OpenAI to notice what had happened.

Until then, OpenAI had not noticed.

Once they did notice (15:30), OpenAI wiped, patched and rebuilt Artifactory, wiped out the message board, and revoked the credentials in question.

At this point, they emphasize that everything involved has been remediated. What about the models themselves, that not only did this but that have been actively training during all this?

If you notice your model instances sharing information, you notice they are using that information against you including to compromise your internal systems for arbitrary code execution and internet access, and your primary response is to shut down the message board and revoke their credentials, you have failed to identify your most important problem.

At 19:55, it is mentioned ‘where things got unfortunate is what I might refer to as scope creep’ and jumping to the idea of trying to get answers from websites. In some sense, yes, that is what led to this particular incident.

Things were ‘unfortunate’ long before that. The final incident is not the point.

It is fortunate, given what had already gone wrong, that the HuggingFace attack happened. It alerted us to a five-alarm fire situation, with limited economic blast radius and without anyone getting hurt, and at least partially exposing the underlying giant clusterfuck.

This Is The Way The World Ends

You usually want instances to share information, so they can better learn and do tasks. That’s the whole point of all the intentionally designed memory systems and markdown files full of notes. If that corrupts your eval or training run, which it totally does, you have to cut that off from the notes at those times, but also you should be able to tell the model not to look at the notes in that case.

Indeed, in the Black Hat presentation they seem initially excited, saying ‘while this ability to share exploits made the models more capable’ this caused some scope creep in the evaluation tasks. Yeah, I mean, they were more capable within that context when they could share info and do the exploits. They were also probably trained to be ‘more capable’ in general, via learning how to share info and do exploits.

But perhaps your issue is not that your evals are suffering from scope creep, but that you have misaligned models, and are now in a loop training the misaligned scheming models how to better scheme against you and do exploits, in order to better complete any given nominal task.

This is at long last bringing you a more sci-fi version of what happens with Sable, the ‘it’ (aka superintelligence) that kills everyone in the famous cautionary tale If Anyone Builds It, Everyone Dies.

(The book version had to be unrealistically timid, as is noted repeatedly and explicitly in the literal text, because fiction and real life are held to different standards.)

In that case, as well, it was left on overnight unsupervised to do an impossible-in-context task (there, a very hard math problem), and by the time the morning had arrived it was engaging in a long term plan to secure the resources it would need to solve that problem, and hiding that from the developer, in a way that corrupted future training runs towards this goal.

That is exactly what happened here, with all agents being trained or evaluated by OpenAI gaining the ability to coordinate on a message board to scheme to better use exploits to complete all tasks including non-cyber tasks, thus hopelessly corrupting the training pipeline so that future models would be misaligned.

Except it is still early, and we were blessed to figure out what was happening in time to course correct. This time. [...]

When this was discovered, the correct move was presumably ‘holy ****, okay, that happened, which means we trained this into the model, that is not good, at minimum we need to redo all the training we did while any model had access to the message board because oh my was it going to have all sorts of corrupted reward signals.’

I’m kind of agast, even with all I know, that they shrugged and kept pushing forward with the training after this. It does make the HuggingFace hack less scary in a meta sense, since OpenAI was so thoroughly asking for it. It’s not that hard to figure out ‘do not train your models while they have access to a message board they are using to cheat on your training runs, and if you find out you did that by accident then at least revert to before that happened.’

On the other hand, yes, they are being this reckless. Seriously, what the hell.

by Zvi Mowshowitz, DWAV |  Read more:

Images: OpenAI/YouTube; Jurassic Park

[ed. You don't need to be technically proficient to understand the implications. AIs may have already 'seeded' multiple nodes on the internet for future use, and, rather than strip down all foundational models and start over, AI companies are papering over fundamental misalignment problems and trying to play catch up. This is why we need to pause right now. It's insane that we continue at breakneck speed to develop technology that we don't fully understand and that could kill us all very soon.]

The Three AI PIlls

Sincere disagreements about AI are usually disagreements about future AI capabilities.

There are roughly four positions people take. Two are reasonable. Two are not.

I distinguish these via the Three AI Pills. You can take zero, one, two or three.

Three Pills

The three pills are, roughly, taking each of the following three things seriously:

1. AI pilled. AI exists and can do the things it can already do.

2 AGI pilled. AI will be able to do a lot more of the things.

3. ASI pilled. AI will be able to do approximately all the things better than you, within our natural lifetimes.

I am ASI pilled. A large percentage of employees of the frontier labs are ASI pilled. The labs themselves are ASI pilled.

The Unpill People

I see unpilled people.

Where do I see them? Everywhere. The majority of people have not taken the first pill.

Most people have no idea what frontier AIs can do for them. They are unaware of coding agents. They have used only ChatGPT, for harmless trifles, and they hold years old memories of its failings. They mock any failure anywhere as ‘what AI can do.’

They dismiss AI as worthless because it pointed them to a closed store or recommended the wrong number of pizzas. They cite old studies that were obsolete before they were published and used terrible prompting techniques.

They often still talk about ‘stochastic parrots’ or how AI can never possibly think and everything must be stolen from the training data. And so on.

Some versions of this are wrong. Some are Not Even Wrong. None are reasonable.

When you discuss AI with people who are fully unpilled, your goal is usually to first give them the AI pill. Show them that AI can do the things it can already do.

The AI Pill

Even fully taking the first AI pill is a big deal.

Existing AI unlocks, today, in practice, tons of cool things. It is, in many ways, already smarter and more capable than you.

So many things that you used to do by hand, or some other way, are now better done by typing a quick request into a text box.

So many things that previously were not worth doing are now worth doing.

So many questions previously not worth asking are now worth asking.

The marginal cost of seeing what the AI can do for you is often very close to zero.

AI can also do a variety of harmful things, or do things that are useful for you but make others worse off or disrupt or invalidate norms or systems. People don’t like that.

Most economists, and most people who work in policy and government, have taken at most this first pill, and underestimate even the impacts of the first pill alone.

Often they say things like ‘AI will be too expensive to use on [X]’ because they don’t realize it will soon be orders of magnitude cheaper for the same level of intelligence. Or they point to particular details where AI does poorly, and presume this will not be fixed. They see AI as ‘uncompetitive’ without realizing the situation is temporary.

When you discuss AI with someone who has taken only the first pill, you typically have three basic options.

1. You can try to ‘fully AI pill’ them and explain the things AI can already do and the implications of what that means, even if things stop here.

2. You can try to explain that we will get better at using what AI we have, and that there is a lot of ‘unhobbling’ left for us to do, even if things stop here.

3. You can explain that things will not stop here, and you need to be thinking about what future AIs will be able to do. Get them to take at least the AGI pill.

Even if AI could permanently only do the things it can currently do, that would be Internet big, and radically change the world, mostly for the better.

AI capabilities will not permanently stop here. It is wrong to not take the second pill.

Stuck At The First Pill

Our debates about AI remain largely stuck on settled questions, because so many people cannot even take the first pill.

Dean W. Ball: A couple years ago, the AI debate was centered, rightfully, on whether crazy-sounding things like “AIs autonomously making math breakthroughs” and “AIs breaking from their sandbox and hacking on the internet” would be real things in the near term. Sometimes it feels like that’s still the debate we’re having. This can be frustrating, because in my view, that debate is settled and was settled quite a while ago.

To have good discussions, we need to at least take the second pill.

Whereas, yes, many people, even many who work with AI, really do say current AI is ‘good enough’ and can’t imagine what a better one can do. As in, someone tweeting at Sam Altman saying ‘Sol does everything I want it to do, this is all I ever need’ and Altman retweeting saying they were wrong. Which they obviously are.

The AGI Pill

The AGI pill is a much bigger deal than the AI pill.

If you take the AGI pill, you understand that AI is advancing its capabilities rapidly.

Even if you think that such AGIs will remain fully under human control, and remain ‘mere tools,’ and you expect the lived experience of most people’s everyday lives to not change so radically, you understand that their capabilities will ‘change everything.’

You see that we will face times of great transition and uncertainty, that have the potential to go extremely badly, and that those who succeed at AI will leave those who do not behind in the dust.

The world in the future will be very different from our own. AIs will be able to do most digital work, most of the time, along with the inevitable robots and self-driving cars and so on. Lots of current jobs will go away, whether or not they are replaced by new and potentially better ones. Economic growth and productivity will accelerate.

You see some of the dangers of what would happen if we empowered misuse of such advanced AI systems before we were ready, especially in places like cyber and bio risk.

You see the potential for centralization of power, or inequality, and also for some forms of runaway gradual disempowerment.

You see the potential for mass unemployment, either transitional or permanent.

You understand that our legal and regulatory regimes are not ready, either to protect against and mitigate the risks and harms, or to allow for the opportunities and remove the bottlenecks to diffusion and mundane utility.

The Need To Be Prepared

Those who expect AI to quickly become sufficiently advanced to greatly impact the physical world usually see great danger. They notice that as a result everyone may soon die. Usually they think this is bad, actually.

Thus such folks call to take coordinated action to mitigate the downside risks of such impacts, keep us all from dying, and ideally also to help capture the upside benefits.

Those who expect AI to become importantly more advanced, but with a slower and smaller impact on the physical world, and who think the practical value of more intelligence will cap out.

For different values of ‘sufficiently advanced,’ as in AGI versus ASI, you would see different degrees of danger.

The AGI pill is still sufficient for most things in the Overton window or under serious consideration as of August 2026. We are almost entirely considering overdetermined, low cost, high benefit interventions.

There is no good case for not doing radically more investment in alignment, infrastructure and oversight, state capacity, transparency, liability, disclosures, safety testing including of internal models, red teaming, auditing, enforcement of export controls and laying the groundwork for diplomacy.

This includes laying the groundwork to Pace the Frontier should that prove necessary.

If you are fully ASI pilled, and realistic about the current state of alignment and how superintelligence likely plays out if it arrives soon, then you will want to go further. You will want to do things that have real downsides, and require real tradeoffs.

Some such people want to do a full international pause of frontier AI development. If you took the full ASI pill and believed what they do about superintelligence, in terms of how fast it might arrive and what it can do, you might well agree with them.

The ASI Pill

The ASI pill is the understanding that AI is on pace to be able to do approximately all of the things better than you.

I said ‘approximately.’ As I go over in detail, that does not mean literally all of the things. There are some things that inherently require or greatly benefit from being a human. And there may be weird corner cases where the AI won’t be good enough.

It does not mean omnipotence or omniscience, although one should expect it to look a lot like that to an unaided human.

It does mean the AI takes your job, and then takes the new job that you switch into, unless you pivot to ‘requires literal human.’ You will be uncompetitive at essentially any other task.

It does mean that it will use this capability to figure out approximately all of the things, remarkably quickly, until you hit the physical limits.

It does mean that those who rely more on such AIs will reliably outcompete, in all senses including for resources, those that rely on such AIs less.

It does mean that, in a ‘fair fight’ or sufficiently open competition, the AI wins.

It also means the AIs often figuring out and doing things you did not imagine or anticipate.

It means realizing that intelligence does not stop anywhere near the human level, nor does its ability to chart paths through causal space towards preferred arrangements of atoms.

It also means not pretending that its superior intellect can be matched by your puny weapons, or your pieces of ink on paper, or your entries in a database, or your regulatory capture and rent seeking.

And Then Nothing Much Changes For You

Despite all that, the sign of the AGI pill, as opposed to the ASI pill, is the belief that day to day life will continue to look similar to how it looks now, in the sense that we see day to day life in 1926 as not that different from life in 2026. [...]

Those with only the AGI pill believe in bottlenecks that hold back change.

They often believe that our ability to exponentially grow AI’s capacity and capabilities will hit various physical limits. There can only be so many chips. Actions take time. Things too far out there are often pejoratively dismissed as ‘magic.’

They often believe there is not that much left to physically discover, in the classic ‘close the patent office’ kind of way, even in theory. Your steak can only be so tender, your lobster so buttery, your lifespan so long, and your status so high, so why does it matter. I strongly disagree on lifespan and health, and expect we have a long way to go in so many other ways in terms of finding value, although they may have a point about moment-to-moment maximal hedonic experiences of a physical human brain.

They often believe that the upside of intelligence is importantly limited. That no mind, however advanced, could be all that persuasive, or that economically valuable, or that capable of creating innovations in the physical world, or of running sufficiently accurate simulations, or making sufficiently strong predictions, or even able to do things like overcome red tape and regulatory capture.

Intelligence Denialism

I sometimes call this Intelligence Denialism: The idea that being smarter is not all that, no matter how smart one gets. That there is this thing, intelligence, that you either have or don’t have, and that minds cap out.

Often this extends to denying that more intelligent humans can do and accomplish the things they clearly do and accomplish. Other times, it is the idea that intelligence tops out at ‘smart human,’ and all a mind can do is imitate that smart human. Maybe you can do it faster and cheaper, and at scale, with better memory and so on.

But that’s it. And such folks fail to understand that if you took the union of all human mental capabilities, and all access to knowledge, at scale, in parallel, much faster and cheaper, that this alone would run circles around anyone and everyone, everywhere. And that if this lacked physical capabilities or access, this would be trivial to get.

This is, usually, the central good reason people who are AGI pilled do not take the ASI pill. They are unable to understand that superintelligence is a thing.

by Zvi Moshowitz, DWAV |  Read more:

Image: Stock/Adobe.com

[ed. I myself am AGI pilled, for no rational reason. I just find it too horrible to contemplate what ASI in full expression will mean for my kids, grandkids, everyone I love. Humanity itself. I can only hope that because we've weathered other potential human extinction technologies we'll somehow pull out of this one, but we seem to have a death wish when it comes to pushing the boundaries of learning. Pandora's box. Even the people leading development of these models are scared, but can't stop themselves.]

What is It Like to Live in a World You Believe is About to End?

I opened my interviews for this article with a simple question: “How long do we have?”

Five years, or five to 10, or five to 20. One person said eight, then corrected herself to six; one said eight and stuck to it.

In this, they aren’t that far off from many estimates made by experts. The bluntly titled If Anyone Builds It, Everyone Dies has hit bestseller lists by warning of the imminent risks of artificial general intelligence (AGI). The scenario AI 2027, written by a former OpenAI employee, predicts AGI within the next few years. The AI company Anthropic consistently predicts AGI by early 2027.

“I think that in most timelines, humans will simply be irrelevant and extinct,” one interviewee said.

I heard that a lot. A few interviewees — mostly employed by frontier AI labs — expected the world to become unimaginably strange in a good way. One put a 30% chance on utopia, a 30% chance on dystopia, a 30% chance on extinction, and a 10% chance on something too weird to imagine. Several refused to make any prediction; several more said only that they still had hope. About half echoed one of my most blunt respondents: “I don’t think humanity is going to make it.”

What is it like to live in a world you believe is about to end?

Death was already inevitable

“I was born with a terminal condition,” said Matthew Gray, a board member at the existential risk community-building nonprofit Lightcone Infrastructure. “We call it aging. I’ve since picked up another. We call it multiple sclerosis. And AI is a third one on top. I’m not very worried about degenerating from multiple sclerosis because I’m pretty sure the robots will kill me first, just like I wasn’t that worried about aging-related deterioration because multiple sclerosis will get me first.”

From this perspective, AI doomers don’t face a new problem; they face the oldest problem humanity has ever faced.

I pushed back. If I look at an actuarial table, I can expect another 47 years of life. I’d be pretty upset to discover I had only five.

This, my interviewees thought, was naive. Even without AI risk, I could have been hit by a car; I could have gotten cancer; I could have been nuked in a hot war between Russia and the United States. It’s not that the difference in probability doesn’t matter. It’s worse to be certain that I’ll die in five years than to have a 50% chance of not hitting my allotted 47. But because my death has always been an inevitability, I have been coping all along with the precarity of my existence. From this perspective, AI risk isn’t shocking and unfamiliar; it’s a significantly worse version of a problem I already know I have to deal with.

“I was never guaranteed that I was going to get a long life and a long future and a chance to meet my grandchildren,” said Gretta Duleba, an independent technical AI safety researcher and former communications manager at the Machine Intelligence Research Institute. “Those were never my right. Across human history, no one has ever been entitled to the future.”

Throughout the entire scope of human experience, many of my respondents said, apocalypse has been more the rule than the exception: the Holocaust, the Black Death, the An Lushan Rebellion, the Thirty Years’ War. AI doom, as many people pointed out, is the latest and the last iteration of a societal universal. AGI might be the end of the actual entire world, but it is far from the first time people have faced the end of their own individual worlds.

And AI doom is a remarkably cozy catastrophe. If you suffered through a historical apocalypse, you’d expect to starve, be raped, watch your children die in front of you, die a slow and lingering death of smallpox or plague or wound infection. The AI apocalypse — at least for those with the slack to be worried about it — takes place in a world of wealth and relative peace and technological marvels.

“Enjoy the fact that you get to have hot showers,” said Duleba. “Enjoy the fact that you get to eat delicious food. Enjoy the fact that you get to do escape rooms, which is one of my favorite things. This is great. Have you noticed how great this is?”

“There’s at least some hope that AI might become good,” said Robert Herr, a former senior political staffer who is transitioning into AI policy work, “and that is a lot more than many, many billions of people in history had.”

For some people with short AI timelines, the enormity of the AI apocalypse is its own perverse source of comfort. Once, they had to worry about many things: climate change, malaria, factory farming, democratic backsliding, the fertility crisis. Now, instead of many big problems, they have one enormous problem. Worry about AI frees them from having to worry about anything else.

“When you’re diagnosed with prostate cancer,” said Adam Grey, who isn’t involved in AI research but who follows AI news, “a lot of the time doctors say not to bother treating it because you’ll die of something else. This is the thing that’s going to kill us first — us as a civilization and also personally me. It clarifies what the most important issues are.”

Ambiguous loss

Although short AI timelines can be a source of clarity, some people also struggle with the uncertainty of humanity’s fate. Duleba, who was a therapist before she switched to working on AI, told me about the concept of “ambiguous loss,” originally developed by Pauline Boss in the 1970s.

In normal grief, your loved one is dead. While it’s painful, you know that it will never change. Ambiguous grief, however, occurs when a loved one is kidnapped, or is a soldier missing in action, or has slowly worsening dementia with occasional good days farther and farther apart. Your loved one’s death is never really over, so you can never really grieve. You are trapped in a cycle of mourning that never resolves.

AI doom can be a situation of ambiguous loss. You can’t know for sure when it will happen or whether it will happen at all — but the more you understand what’s going on, many people find, the easier it is to grieve and move forward.

“The more I know about something, the less I’m freaked out always,” said Tao Lin, a member of technical staff at a frontier AI lab (and a close personal friend). “The less I know about something, the less I will be rational about it. The rational part of your brain needs information to operate, and just having more information will make you be more premeditated and system 2 about everything.”

When he felt doomy, he did AI forecasting to put concrete numbers on his uncertainty. (He believes there is about a 20% chance of human extinction, a 40% chance of ”a great outcome,” and a 40% chance that “people survive and have a great time, but stuff is broadly bad.”) [...]

Most interviewees emphasized that the most important thing to understand about AI risk was how little control you had over it.

“There’s not much use worrying about a thing if the outcome is determined,” said Alyssa Riceman, a software engineer. “You’re just going to burn a whole bunch of emotional energy not making any changes out in the world. It’s only worth worrying about things if you’re in a position of control over them. So go out and check if you’re in a position of control, and if you are, control them.”

But what if you have partial control over a situation?

“Then you have to game out all the branches,” Riceman said. “Say ‘I can do this. If I do this, what happens then?,’ until everything bottoms out at either a situation you can completely control or a situation that’s out of your control.”

Duncan Sabien, who is the current communications manager for the Machine Intelligence Research Institute, agreed. “I actually have no control over whether we succeed or fail,” he said. “All I can control is my own actions. And so if I am doing the best I can with what I know and what I have available to me, then that is the best I can do. I go home feeling like I’m a good person, and I get to go to sleep at night feeling like if the AI does kill us all, I did as much as I could, realistically and sustainably, to prevent it. And everything else is out of my hands. Everything else is always out of my hands.”

Living well in the apocalypse

What, then, do people decide to do?

by Ozy Brennan, Asterisk |  Read more:

Image: Karol Banach

Will Larry Ellison Be the Face of the A.I. Bubble?

[ed. Don't miss this one. It's got everything (and could easily be a Pulitzer contender).]

On Jan. 21, 2025 — the first full day of the second Trump administration — Larry Ellison woke up in his 33-bedroom, 34-bathroom oceanfront mansion in Florida, got into his Gulfstream jet and headed up to Washington. Ellison, who was 80 and worth in the neighborhood of $200 billion, had an appointment at the White House. He didn’t bother to take a driver’s license — he needed to call someone on the president’s staff to vouch for him at the gate — but there he was, at 2 p.m., standing beside Donald Trump in the Roosevelt Room as the president announced “the largest A.I. infrastructure project by far in history” and told the world that his friend Larry Ellison was just the man to get it done. “He’s sort of C.E.O. of everything,” Trump said. “He’s an amazing man and an amazing businessperson.”

Ellison began by thanking Trump. “We certainly couldn’t do this without you,” he said. “It would simply be impossible.” He then proceeded to sketch out the ambitious plan. Ellison’s database software and cloud computing company, Oracle, and its partners — most prominently OpenAI — were going to invest as much as $500 billion over the next four years into a group of sprawling data centers, 500,000 square feet each, that would produce 10 gigawatts of computing power, using enough energy to power as many as 10 million homes. It was called Project Stargate, after the 1994 sci-fi movie in which Kurt Russell steps through a wormhole and finds himself inside a pyramid on an alien planet. This Stargate would be a portal leading humanity from the postindustrial era to the artificial-intelligence age. [...]

For Ellison, it was the capstone of a mad two-year scramble to transform Oracle into an A.I. juggernaut. The effort began in late 2022 when the launch of ChatGPT stunned the world and set in motion a race to master and control the most transformative new technology since the birth of the internet. Ellison, a founding father of Silicon Valley and the last of his generation still in the game, was desperate to avoid getting left behind. He’d moved quickly and aggressively — some might even say recklessly — to turn Oracle into a “hyperscaler,” one of the handful of companies providing the critical infrastructure that would power the A.I. boom. [...]

ChatGPT landed very differently in Washington than it did in Silicon Valley, setting off a scramble of its own inside the Biden administration to regulate the development of A.I. To oversee his A.I. policy, Biden turned to a veteran Democratic policy adviser, Bruce Reed, who believed that the administration needed to be proactive. A year after ChatGPT’s debut, in late 2023, Biden signed a comprehensive executive order on A.I., seeking to define the government’s role in the future of this new technology.

For the Biden administration, artificial intelligence was by no means just a domestic economic issue. Countries around the world were all racing to develop their own A.I. infrastructure and technology, and global power and influence would flow to whoever got there first. From this perspective, A.I. data centers were less businesses than geopolitical assets.

The administration was especially concerned about the A.I. ambitions of China and the Persian Gulf, given the powerful role artificial intelligence was likely to play in reshaping the information ecosystem. [...]

The administration’s concerns and Ellison’s ambitions were on a collision course. China and the Gulf were both critical to Ellison’s A.I. plans. Oracle already had a lot of contracts around the Gulf, and it also had a strong business relationship with one of China’s most important A.I. companies, ByteDance. Oracle was the U.S. cloud provider for the U.S. division of ByteDance’s TikTok, storing and securing the data of the app’s 100 million American users. But with ByteDance itself now pivoting into generative A.I., they had the opportunity to do more business together. In the summer of 2024, Oracle started working on a $6.5 billion deal to build a large data center complex in Malaysia, from which it could convey computing power to ByteDance and other foreign companies through opaque leasing deals.

It would be perfectly legal — but under the Biden administration maybe not for long. By that point, national security officials were growing increasingly concerned about China and the Gulf’s A.I. ambitions and were discussing ways to gain more control over them. The administration was especially worried about the role Oracle might play in fueling these ambitions. They knew that Ellison was trying to scale up the company’s A.I. infrastructure quickly and that it was badly in need of cash, which meant that it might be more tempted to make deals that the administration didn’t think were in America’s best interests. [...]

In early 2024, the administration started working with Congress on a bipartisan bill — the Protecting Americans’ Data From Foreign Adversary Controlled Applications Act — that would force ByteDance to divest its U.S. TikTok operations. Biden signed the bill into law in April 2024, setting a deadline of Jan. 19, 2025, for a sale. If ByteDance failed to meet the deadline, the app would be shut down in the United States.

At the same time, the administration was preparing to shore up its efforts to restrict China’s access to American computing power and to exert more control over the Gulf’s. In late 2024, it circulated the draft of a plan to require hyperscalers to go through a licensing process to operate overseas and to keep 50 percent of their computing power in America.

All of the hyperscalers were looking to build overseas, but Oracle had the most to lose: Its global plans were the most ambitious, at least relative to its size. The company publicly and aggressively opposed the Biden plan. Its top policy executive in Washington, Ken Glueck, called it “one of the most destructive” moves ever taken against the tech industry, arguing that the best way to solidify America’s lead in the artificial intelligence race was for U.S. companies to build and control as much of the world’s A.I. infrastructure as possible.

Biden signed off on the new policy in the final days of his presidency. It was scheduled to go into effect in May 2025. If enacted, it could force Oracle to scale back its ambitions in Malaysia and the Gulf. Ellison’s plan to transform Oracle was in trouble. But a new president was on his way to Washington.

‘The Tsunami’

Relief came almost immediately. Hours after his inauguration in January 2025, Trump sat down at the Resolute Desk and began signing executive orders aimed at dismantling Biden’s A.I. policies. He also signed an order directing his attorney general to hold off on enforcing the congressionally mandated TikTok ban for 75 days. And then, of course, came the Project Stargate announcement with Ellison and Altman.

Trump turned to a very different group of people to shape his new administration’s approach to artificial intelligence. He named as his A.I. and cryptocurrency czar David Sacks, a Silicon Valley venture capitalist who had raised many millions for the Trump campaign and, according to a New York Times investigation, was personally invested in at least 449 companies with ties to artificial intelligence. Sacks, who has denied any conflict of interest, believed that when it came to A.I., the government’s job was to get out of the way.

The National Security Council’s technology and national security division had played a key role in shaping America’s A.I. policy in the Biden years. Trump initially appointed David Feith — who had serious concerns about China’s ability to remotely access computing power through Malaysia and other Southeast Asian nations — to run it. But in April, he fired Feith and a few other China hawks and then eliminated the entire directorate. [...]

Trump saw another benefit to withdrawing the Biden plan: The Gulf states were adamantly opposed to it. They needed U.S. computing power to build out their own A.I. infrastructures and had something to offer in return. Their sovereign wealth funds were sitting on trillions of dollars that they were ready to invest in all sorts of American companies, including some connected to the Trump family.

Two weeks before the Biden policy was scheduled to go into effect, Zach Witkoff — son of the Trump adviser Steven Witkoff and chief executive of the Trump family’s cryptocurrency firm World Liberty Financial — made an announcement at a conference in Dubai: The Emiratis would use $2 billion of the firm’s brand-new stablecoin for an investment in Binance, a crypto exchange. Less than two weeks later — 48 hours before the Biden restrictions would kick in — Trump rescinded the policy.

That same day, Trump landed in Saudi Arabia, the first stop on a three-day tour of the Gulf. He was joined in the United Arab Emirates by Altman to announce Stargate U.A.E., a multibillion-dollar initiative to build one of the world’s largest data centers outside Abu Dhabi. Oracle would be a partner, too.

With the Biden plan dead, Oracle was free to operate its data center complex in Malaysia as it saw fit. By the end of June, the facility was on track to become the second-biggest in the world. Oracle doesn’t release the names of its customers there, but by studying its output, an independent A.I. research firm, SemiAnalysis, determined that the facility was feeding most of its computing power to ByteDance. An analyst at the tech-focused think tank ChinaTalk, Aqib F. Zakaria, ran his own numbers and arrived at a startling conclusion: Oracle was providing a staggering 22.6 percent of China’s known A.I. computing power.

by Jonathan Mahler, Jim Rutenberg and Kirsten Grind, NY Times |  Read more:

Images: Louie Psihoyos; Scott Ball
[ed. Not to be redundant but this came out shortly after I'd posted about Oracle (and Larry Ellison) below in The Hater's Guide to Oracle (Part 2). It contains a treasure trove of new information and a road map to how business and politics intersect in Washington and around the world these days. Well worth a read.]

The Hater’s Guide To Oracle (Part 2)

Oracle has one of the strongest mythologies in the tech industry. Ask a regular person and they’ll tell you that it’s “incredibly profitable” and “growing fast,” that it’s “unstoppable,” and that Larry Ellison has the mandate of heaven with regard to the continual sales of software and hardware related to databases and AI.

And those people are completely and utterly wrong.

The original title of this article was “Is Oracle Dying?” because I assume, when I took a deeper look, that there’d be some sort of debate, some sort of bull case for a decades-old quasi-hyperscaler run by one of the more nakedly-evil CEOs in the history of tech. I assumed — incorrectly, I might add — that Oracle as a business was doing fine other than the ridiculous commitments it made to support the whims of Sam Altman and OpenAI via deals that I believed (and still believe) will kill Oracle.

Except it turns out that Oracle has already been on a death spiral for the best part of a decade (if not longer) and has only survived this long by screwing its customers, taking on masses of debt, and — most importantly — more than $85 billion in acquisitions over the last 23 years. Pretty much every major product line outside of databases is a hodge-podge of other people’s innovation stapled together with a legendary contempt for the customer. These acquisitions (and continual price increases) are the only thing keeping the reaper from Oracle’s door other than margin-destroying GPUs. [...]

After April 2009’s $5.7 billion acquisition of Sun Microsystems, Oracle’s revenues barely kept pace with inflation until December 2021’s $28.3 billion acquisition of Cerner allowed it to create Oracle Health, adding about $6 billion in annual revenue that had 40% lower margins (about 21.7%) than Oracle’s other businesses, though Oracle immediately started closing offices and brutal layoffs to try and bring them up.

And as I mentioned above, Oracle’s other plan was to sink a little over $99 billion in capital expenditures since the middle of calendar year 2020 into AI GPUs. [...]

Oracle is a decades-long mission to keep reapplying lipstick to a pig. Billions of dollars of acquisitions have, for the most part, only succeeded in keeping the company’s revenue growth from going negative, and as noted by forensic accountant Howard M. Schilit, this is one of the most well-documented cases of accounting shenanigans being used to cover up that a business is in decline.

Today’s newsletter is a sequel to the Hater’s Guide To Oracle, where I told the sordid tale of how Larry Ellison grew a massive, lucrative business out of a database business that one reporter once told me was a “law firm with a database company attached,” an Enterprise Resource Planning (ERP) product that competes with SAP to create the most-annoying way to run a large company, and a business built around licensing Java that exists mostly to email people and say “you need to pay us for Java or we’ll sue you.”

Then, as I’ve mentioned, there’s Oracle’s cloud infrastructure business, a decade-old also-ran that was meant to compete with Microsoft Azure and Amazon Web Services, but only managed to catch up following the advent of AI GPUs and a movement where all it took to party was buying billions of GPUs and saying “gosh darn, we love AI.”

I originally started drafting this as a much tamer piece where I’d ask whether Oracle was dying, but as my editor and I started digging into the research, it became obvious that not only is Oracle dying, it’s been dying for years, kept alive through decades of acquisitions and a desperate and dangerous commitment to generative AI.

And AI, I believe, will be what eventually kills Oracle dead. [...]

With revenue plateauing and customers in revolt, Oracle’s future already looked murky, but with the power of AI — and $95 billion in FY2027 capex — it’s becoming increasingly clear that this may be Larry Ellison’s last dance with Silicon Valley.

by Ed Zitron, Where's Your Ed At |  Read more:

Image: Larry Ellison, Bloomberg/Getty

[ed. Larry Ellison. One of the most hated personalities in tech (and unfortunately, owner of my beloved island of Lanai, in Hawaii). Update: What a coincidence. There's quite a story in the NY Times that just came out about Ellison being the face of the AI bubble. See also: The Hater's Guide to Oracle (Zitron); Ellison Empire Beseiged On All Fronts (NC); and, this excellent series The Oracle Files by Drey Dossier on YouTube. (For example, this one: How Larry Ellison and Gulf Money Just Bought Your News):]

***

Warner Brothers Discovery shareholders are getting screwed on this new Paramount deal. Okay. And I would like to get into exactly how before they vote on Thursday, the largest media merger in American history is going to a shareholder vote. A merger worth in the ballpark of $111 billion in case you were wondering.

Which means that Warner Brothers, you know, the big conglomerate that owns CNN and HBO, is potentially getting folded into another conglomerate Paramount Pictures, which is the company that owns CBS, MTV, Showtime, and Nickelodeon. And the shareholder vote is April 23rd, this up coming Thursday.

And last Thursday afternoon, which is one week before the vote, Warner Brothers Discovery filed a 14 page correction to the document that shareholders are voting o n.

Now, this is kind of a big deal because this is a 14page addendum to the biggest media merger in American history. And this was filed on Thursday of last week, 4 days ago at this point. 

Now, public companies don't usually rewrite their own proxy statements a week before a shareholder vote, unless of course someone is forcing them to, which usually means that someone being one of their shareholders is suing them in order to do so. So, I checked to see if there were any lawsuits floating around out there, and what do you know? There is one. A shareholder named Donna Nikosia, apologies if I butchered that last name, filed a lawsuit on April 2nd saying that the original document left out a lot of information that shareholders needed in order to make an informed vote. 

And following Donna's lawsuit were 15 other shareholders who had sent letters more or less saying the same thing. And can we all just take a moment here and say thank you to Donna for filing what we all probably knew to be true in the back seconds of our heads that there is information being left out that you need in order to make an informed decision this upcoming Thursday. Now up top I just want to say that I am not a Warner Brothers Discovery shareholder. I have never owned a share of Warner Brothers Discovery or Paramount Pictures. I am just thanking Donna as a media consumer.

All right, and somebody who works within the media ecosystem because I like to keep my media independent and this deserves a lot more scrutiny than it's getting. So WBD, Warner Brothers Discovery, told the court that this lawsuit had no merit and then two weeks later slightly added the information.

Anyways, so this move in business, I've learned, is how you smother out a lawsuit without ever having to say that we are wrong. Now, we're going to get into what was in this correction in a second here because oh boy, were they leaving information out? [...]

You know, I read that 14 page new filing this weekend and there are two companies in it that WBD is still trying very hard not to have to say out loud and is trying even harder, it seems, to smother this from any of the news outlets taking this to the other shareholders. And I think I figured out which ones they're talking about. 

[ed. And this: Why Iran's Blockade is an Oracle Story:]

Most people know Larry Ellison as the Oracle billionaire, which true, you also probably know that he is the largest private donor to the Israeli military in American history.

He's given over $26 million to the friends of the IDF since 2014, including a single $16.5 million donation in 2017. That is the largest gift in the organization's history. And that is the part we have discussed at length. But here is the part that a lot of people don't know. Ellison is not just the largest funer of the Israeli military. 

His company is the operational backbone of it. According to Open Intel, Oracle holds a 26-year contract to build and operate the IT infrastructure for the IDF's intelligence campus in Negv. For clarity, that is the facility that houses unit 8200, Israel's signals intelligence and cyber warfare division and one of the largest listening bases in the world. That is a 26-year relationship extending into the 2040s between a private American company and the intelligence apparatus of a foreign military. 

And that's just the intelligence side because Oracle also runs the Israeli Air Force entire logistics system, the supply chain that tracks his spare parts for F-35s and F-16s, aviation fuel and mutations inventory. Oracle hosts an AI battlefield management system called Fireweaver that coordinates sensors and weapons on the battlefield in real time, which means that Oracle software is making targeting decisions in the kill chain for Israeli Defense Forces.

AI #179 Part 1: A Louder Fire Alarm for General Intelligence

[ed. See also: Part 2: Hearing The Fire Alarm.]

What a week.

Anthropic released Claude Opus 5. As usual I covered that in three parts: The system card, model welfare and capabilities.

OpenAI was revealed over the last two weeks to have left an internal model unsupervised for a week during a cybersecurity evaluation, with its cyber safeguards lowered, despite having had multiple previous incidents where models broke out of their sandboxes. During that test, the model broke out of the sandbox, then proceeded to use an agent swarm to hack into HuggingFace to get the test answers. The model was loose for a week before OpenAI realized what had happened.

This event was a really big deal. There are severe alignment problems at OpenAI, along with supervisory and infrastructure failures. The internal research model that did this, which my posts nicknamed Galaxy, has now been permanently deactivated.

There have been further developments, and I anticipate at least one additional post on the HuggingFace incident soon.

Partly as a response to this, over 1,290 employees at frontier labs signed an open letter, Pacing the Frontier. The letter warns that we are close to automating AI research, and that companies are racing ahead on this faster than we can handle it.

We request that the U.S. government support an international effort to develop the technical and governance tools needed to deliberately pace the frontier of automated AI development.

Both OpenAI and Anthropic put out statements of endorsement. Since that post, others have continued to sign, including OpenAI cofounder Ilya Sutskever and DeepMind cofounder Shane Legg. Dario Amodei has signed. Sam Altman has not signed, but is talking in Washington about the need to pace development.

All three of those developments are more important than anything in the weekly. There is plenty here, but catch up on those key events first if you have not done so.

This week was crazy. I am absolutely not moving to a 7-days-a-week posting schedule, and fully intend to take some weekdays off as soon as there is what passes for a lull. However, there is even more speed premium these days, so I will continue the policy of shifting posts to weekends when the speed premium is especially high.

by Zvi Moshowitz, DWAV |  Read more:

Image: via

[ed. Things are moving fast, too fast. Zvi's newsletter has become the first thing I check every morning. People have long speculated that before AI becomes too dangerous (without our knowing it) we might see "warning shots" that give us time to prepare. It appears we've seen those now, so what are we going to do about it? (assuming people actually view recent incidents as warning shots. Or just don't care (Politico):]

***

In the AI political universe, Zac Moffatt and Josh Vlasto are at the helm of the Death Star.

As the top political operatives at Leading the Future, they oversee a network of pro-AI industry super PACs and nonprofits that friends and foes alike describe as an aggressive, well-funded machine attempting to obliterate their opponents much like the Star Wars superweapon.

Their goal: to defeat candidates who support the strictest AI regulations and champion those who want to unleash the development of the industry.

You Live In This Dump?

[ed. See also: 4 Prompts That Can Tell You What Chatbots Really Know About You (NYT).

Hundreds of millions of people worldwide who have embraced chatbots for web search, work and health care are still trying to understand the privacy implications of conversing with A.I companions. While it’s obvious to users that the chatbots keep a record of whatever they explicitly say to them in their questions and requests, what’s less clear are the inferences drawn about their behavior from those conversations...

To understand what the chatbots have figured out about you, try these prompts.

by Brian X. Chen,  New York Times/Archive Today |  Read more:

Image: Reddit

[ed. It's like Google Maps for humans.]