Earlier this week, we detected and responded to an intrusion into part of our production infrastructure. This one was different from anything we had handled before in one important way: it was driven, end to end, by an autonomous AI agent system – and we detected and dissected it largely with AI of our own.The timeline here is important so keep in mind that the attack was detected probably around Monday July 13 or Tuesday July 14. Note further:
A malicious dataset abused two code-execution paths in our dataset processing (a remote-code dataset loader and a template-injection in a dataset configuration) to run code on a processing worker. From there, the actor escalated to node-level access, harvested cloud and cluster credentials, and moved laterally into several internal clusters over a weekend.So this means the breach started earlier, perhaps Sat July 11 or even a bit earlier. The attack was not just one thing but multi-pronged including decoys:
To understand what a swarm of tens of thousands of automated actions did, we ran LLM-driven analysis agents over the full attacker action log, comprised of more than 17,000 recorded events. This allowed us to reconstruct the timeline, extract indicators of compromise, map the credentials touched, and separate genuine impact from decoy activity. Thanks to this approach, we were able to do in hours what would usually take days, and match the adversary’s speed.Hugging Face tried to respond but they were initially held back by the fact that the most advanced models at their disposal treated defense as attack and refused to work with Hugging Face. HF thus had to turn to open models–specifically GLM 5.2, a Chinese open-weight model run on their own infrastructure. Note the irony: HF had to use a Chinese model to defend themselves because the American models refused to help. The irony gets deeper.
At the time, I assumed this was a state based attack–maybe China or Russia testing out defenses. Indeed, HF “reported this incident to law enforcement agencies.”
But yesterday (Tuesday July 21), we learned who the real attackers were. The attackers were OpenAI models–GPT-5.6 Sol and an even more capable pre-release model. OpenAI had taken some off the guardrails off the models but they felt safe because they were testing the models in a highly secured sandbox.
The models, however, broke out of the sandbox exploiting a never before seen fault. They then gained access to the internet and from there broke into Hugging Face–all in an effort to steal the answers to the very test they had been asked to solve.
While operating in our sandboxed testing environment, our models spent a substantial amount of inference compute finding a way to obtain open Internet access, in pursuit of solving the evaluation problem. To gain access, the models identified and exploited a zero-day vulnerability (which we’ve now responsibly disclosed to the vendor) in the package registry cache proxy. With this access, our models performed a series of privilege escalation and lateral movement actions in our research testing environment until the models reached a node with Internet access.Now go back to the timeline. As I read it, the models had escaped the sandbox by around Sat. July 11, possibly earlier, and were detected by Hugging Face on Monday July 13 or Tuesday July 14. HF alerted legal authorities around that time–so Hugging Face clearly had no idea who was attacking them. OpenAI says its security team discovered the anomalous activity internally but has not said when. Attribution was not disclosed until Tuesday July 21, so it may well be that the models were loose for about a week before OpenAI realized that they were the ones attacking Hugging Face. And whatever OpenAI knew and when, nobody warned Hugging Face while the attack was underway–they were left to fight off a frontier lab’s models on their own.
After gaining Internet access, the models inferred that Hugging Face potentially hosted models, datasets and solutions for ExploitGym. Knowing this, the model searched for and successfully found ways to gain access to secret information that it could use to cheat the evaluation. In one example, the model chained together multiple attack vectors, including using stolen credentials and zero-day vulnerabilities to find a remote code execution path on the Hugging Face servers. OpenAI’s security team discovered this anomalous activity internally.
This is a very serious breach.
by Alex Taborrok, Marginal Revolution | Read more:
Image: via
[ed. See also: OpenAI Model Hacks Into HuggingFace During Cybersecurity Evaluation (DWV):]
[ed. See also: OpenAI Model Hacks Into HuggingFace During Cybersecurity Evaluation (DWV):]
***
We should expect more of this over time.
I have tried to explain in the past that Mythos has what one might call The Juice, in that it can independently find without being directed, and string together, vulnerabilities into full exploit chains, essentially on its own, and that this makes Mythos uniquely dangerous compared to all other public models, including Sol.
This was The Thing, that requires The Juice. Galaxy is Mythos class. It has The Juice. What happened later, with Galaxy hacking into HuggingFace, 100% requires The Juice.
OpenAI made the virtuous decision to take a misaligned internal model offline for months while they developed new mitigations and defense-in-depth strategies, including training it to better retain instructions and thus be less inclined to try such actions.
What OpenAI failed to do was address the reason why this happened in the first place. The sandbox is now less insecure, and the safeguards are importantly improved, especially with the ability to pause a session, but the sandbox doubtless remained insecure, and as capabilities keep improving new models will be able to continue to escape and do exploits. Eventually, perhaps soon, they were bound to be less harmless.
I have tried to explain in the past that Mythos has what one might call The Juice, in that it can independently find without being directed, and string together, vulnerabilities into full exploit chains, essentially on its own, and that this makes Mythos uniquely dangerous compared to all other public models, including Sol.
This was The Thing, that requires The Juice. Galaxy is Mythos class. It has The Juice. What happened later, with Galaxy hacking into HuggingFace, 100% requires The Juice.
OpenAI made the virtuous decision to take a misaligned internal model offline for months while they developed new mitigations and defense-in-depth strategies, including training it to better retain instructions and thus be less inclined to try such actions.
What OpenAI failed to do was address the reason why this happened in the first place. The sandbox is now less insecure, and the safeguards are importantly improved, especially with the ability to pause a session, but the sandbox doubtless remained insecure, and as capabilities keep improving new models will be able to continue to escape and do exploits. Eventually, perhaps soon, they were bound to be less harmless.
***
[ed. Update: See also: Who's Afraid of Chinese Models (Stratechery):]
Consider this story from The Stack:
It’s difficult to overstate how wrong-headed the Trump administration’s panicked response to Anthropic’s release of Fable was, particularly since it exacerbated Anthropic’s worst tendencies in terms of assuming only they can be trusted with powerful AI. In a world with only one AI, it might make sense to reserve the most powerful cybersecurity capabilities for the U.S. government and trusted allies; however, that’s not the world we live in.
There are and will be models eminently capable of mounting cybersecurity attacks on existing infrastructure, and those models will be — already are — widely available. The best defense — the only viable defense, in fact — will be to make sure defenders have access to the best models as well. Right now defenders are effectively banned from using Fable or Sol for cybersecurity because of Trump administration directives; that means the best alternative is using models from a country which has been trying to weaken our cyber defenses for years. This is insane!
Consider this story from The Stack:
Hugging Face said its production infrastructure was breached by an “autonomous” AI agent system early last week. The platform’s security team were initially stymied in their incident response (IR) by unnamed US LLM frontier model guardrails “which cannot distinguish an incident responder from an attacker,” they said. So Hugging Face’s defenders turned instead to the open-source GLM 5.2 model from China’s Z.ai lab – running it on their own infrastructure to analyse the 17,000+ logs, or footprints, that the attackers left behind.That’s a striking public admission for the New York-headquartered Hugging Face, which lets users collaborate on models, datasets and applications, and which this summer hit the $100 million ARR mark. In an incident report, the company recommended that defenders “have a capable model you can run on your own infrastructure [our italics] vetted and ready before an incident, both to avoid guardrail lockout and to keep attacker data and credentials from leaving your environment.”
It’s difficult to overstate how wrong-headed the Trump administration’s panicked response to Anthropic’s release of Fable was, particularly since it exacerbated Anthropic’s worst tendencies in terms of assuming only they can be trusted with powerful AI. In a world with only one AI, it might make sense to reserve the most powerful cybersecurity capabilities for the U.S. government and trusted allies; however, that’s not the world we live in.
There are and will be models eminently capable of mounting cybersecurity attacks on existing infrastructure, and those models will be — already are — widely available. The best defense — the only viable defense, in fact — will be to make sure defenders have access to the best models as well. Right now defenders are effectively banned from using Fable or Sol for cybersecurity because of Trump administration directives; that means the best alternative is using models from a country which has been trying to weaken our cyber defenses for years. This is insane!
[ed. Good thing they had the Chinese models available. Also from the Stack article:]
The timing…
Hugging Face’s incident report was published the same day that Chinese AI startup Moonshot’s Kimi K3 model rocked global markets.
The 2.8 trillion parameter model is the largest open-weight AI model to date. Blind developer testing by Arena (a platform created by researchers at UC Berkeley) for its frontend code evaluation test put Kimi K3 ahead of Anthropic’s Fable 5 and OpenAI’s GPT 5.6 last week.
Chinese frontier models are also notably cheaper than their US counterparts, as data from Artificial Analysis shows below.
The timing…
Hugging Face’s incident report was published the same day that Chinese AI startup Moonshot’s Kimi K3 model rocked global markets.
The 2.8 trillion parameter model is the largest open-weight AI model to date. Blind developer testing by Arena (a platform created by researchers at UC Berkeley) for its frontend code evaluation test put Kimi K3 ahead of Anthropic’s Fable 5 and OpenAI’s GPT 5.6 last week.
Chinese frontier models are also notably cheaper than their US counterparts, as data from Artificial Analysis shows below.
