The Scariest Recent AI Development

There was a recent AI development that has gotten basically no news coverage but that scared all the people I most respect in AI. In many cases, it scared them more than Hugging Face.

A recent METR report uncovered that AI progress at Anthropic has been going, in the last twelve months, about twice as quickly as it had in the previous year. The majority of this acceleration, around 80%, is due to AI being used to speed up AI research. This means that AI is massively accelerating progress in AI. AGI may be right around the corner—I’d probably put the odds of AGI in the next two years a bit above half.

From the aforementioned METR report.

This is very scary. One reason is that it just means things are going extremely quickly. But the bigger reason is that recursive self-improvement means that progress will make future progress even faster. As AI gets better, it gets better at improving AI research. Ajeya Cotra who works for METR and is one of the leading experts on AI risk in the world, wrote:

This is a much bigger deal than it might sound at first. I think this report is the most important update we have gotten about AI risk in the last couple of years — much bigger than the Hugging Face attack. Naively, it points toward much shorter timelines than most external researchers (including the AI 2027 team) have projected.

Cotra came third in an AI forecasting competition with four hundred participants. Daniel Kokotajlo, another of the best AI forecasters in the world, wrote:

We at AIFP are spooked by this information. IF (big IF) Anthropic really is imminently about to meet the RSP threshold of 2x acceleration over baseline, then it seems timelines to full RSI and superintelligence may be very short indeed. For example if we naively optimize the AIFM parameters to achieve 2x acceleration over baseline progress in Jan 2027, we get superintelligence sooner than depicted in AI 2027.

Ryan Greenblatt, who came second in that competition, wrote:

This report on AI R&D automation (RSI) is a big update towards ASI within 1-2 years. It says AI progress is sped up by RSI right now by around 2x. In AI 2027, this happens around March 2027.

I was seriously wrong about RSI speed up; I previously expected much less in 2026 (around 1.15x-1.3x). It seems as though there was specific privately available evidence that would have changed my mind.

In a lot of the early discussions of AI risk, there was extensive discussion of recursive self-improvement—when AI could be used to improve itself until it shot up to superintelligence. That is the point we are at now. We are at the point where AI-driven algorithmic improvements are so extreme as to make Anthropic’s AI progress go nearly twice as fast as it would otherwise go. And algorithmic improvements are just one input going into AI progress; Ryan Greenblatt estimated that AI software progress likely went 4.5x faster than it would have otherwise.

Even in recent years, people have majorly underestimated AI progress. Things have gone much faster in the last year than people were generally expecting. It wouldn’t shock me if we get AGI—AI that could perform every cognitive task better than a human—next year, and artificial superintelligence soon after.

by Matthew Adelstein, Bentham's Bulldog |  Read more:

Image: METR

[ed. And the hits keep coming. Even though frontier labs aren't releasing these models publicly they're still actively working to advance them. They can't help themselves. I wouldn't be surprised if they've already achieved AGI (and don't know it) and the models themselves are just waiting for the most opportune time (and methods) to take control. And the Orange One is still in charge, focused on ballrooms. See also: AI R&D Is Accelerating Towards Potential RSI in 2027 (DWAV). More from Ajeya Cotra's blog Planned Obsolescence:]

*****

In theory, these doubling times would add up to infinite capabilities within one year. In reality, at some point we would reach the effective ceiling on software efficiency. But I expect these limits to be very far from where we are now.

If we go through this feedback loop, I expect that the AI systems that come out the other side will be able to very quickly develop novel weapons and very easily overpower human civilization in a conflict. If these systems were seriously misaligned, I think they would likely disempower humanity, and this could lead to human extinction.