AI policy decisions made in the next few months will shape how that AI is governed. The security and safety measures in place for safeguarding that AI will be among the most important in history. Key upcoming milestones include the first acts of the Trump administration, the first acts of the next US congress, the UK AI bill, and the EU General-Purpose AI Code of Practice.
If there are ways that you can help improve the governance of AI in these and other countries, you should be doing it now or in the next few months, not planning for ways to have an impact several years from now.
The announcement of o3 today makes clear that superhuman coding and math are coming much sooner than many expected, and we have barely begun to think through or prepare for the implications of this (see this thread) – let alone the implications of superhuman legal reasoning, medical reasoning, etc. or the eventual availability of automated employees that can quickly learn to perform nearly any job doable on a computer.
There is no secret insight that frontier AI companies have which explains why people who work there are so bullish about AI capabilities improving rapidly in the next few years. The evidence is now all in the open. It may be harder for outsiders to fully process this truth without living it day in and day out, as frontier company employees do, but you have to try anyway, since everyone’s future depends on a shared understanding of this new reality.
It is difficult to conclusively demonstrate any of these conclusions one way or the other, so I don’t have an airtight argument, and I expect debate to continue through and beyond the point of cross-domain superhuman AI. But I want to share the resources, intuitions, and arguments I find personally compelling in the hopes of nudging the conversation forward a tiny bit.
This blog post is intended as a starter kit for what some call “feeling the AGI,” which I defined previously as:
- Refusing to forget how wild it is that AI capabilities are what they are
- Recognizing that there is much further to go, and no obvious "human-level" ceiling
- Taking seriously one's moral obligation to shape the outcomes of AGI as positively as one can
How far we’ve come and how it happened
It has not always been the case that AI systems could understand and generate language fluently – even just for chit chat, let alone for solving complex problems in physics, biology, economics, law, medicine, etc. Likewise for image understanding and generation, audio understanding and generation, etc.
This all happened because some companies (building on ideas from academia) bet big on scaling up deep learning, i.e. making a big artificial neural network (basically just a bunch of numbers that serve as “knobs” to fiddle with), and then tweaking those knobs a little bit each time it gets something right or wrong.
Language models in particular first read a bunch of text from the Internet (tweaking their knobs in order to get better and better at generating “text that looks like the Internet”), and then they get feedback from humans (or, increasingly, from AI) on how well they’re doing at solving real tasks (allowing more tweaking of the knobs based on experience). In the process, they become useful general-purpose assistants.
It turns out that learning to mimic the Internet teaches you a ton about grammar, syntax, facts, writing style, humor, reasoning, etc., and that with enough trial and error, it’s possible for AI systems to outperform humans at any well-defined task. (...)
The fact that this all works so well — and so much more easily and quickly than many expected — is easily one of the biggest and most important discoveries in human history, and still not fully appreciated.
Here are some videos that explain how we got here, and some other key things to know about the current trajectory of AI. [ed. ..yikes]
Here are some other long reads on related topics. As with the videos, I don’t endorse all of the claims in all of these references, but in the aggregate I hope they give you some 80/20 version of what people at the leading companies know and believe, though I also think that regularly using AI systems yourself (particularly on really hard questions) is critical in order to build up an intuition for what AI is capable of at a given time, and how that is changing rapidly over time.
There is no wall and there is no ceiling
There is a lot of “gas left in the tank” of AI’s social impacts even without further improvements in capabilities — but those improvements are coming. (...)
Note that it is not just researchers but also the CEOs of these companies who are saying that this rate of progress will continue (or accelerate). I know some people think that this is hype, but please, please trust me — it’s not.
We will not run out of ideas, chips, or energy unless there’s a war over AI or some catastrophic incident that causes a dramatic government crackdown on AI. By default we maybe would have run out of energy but it seems like the Trump administration and Congress are going to make sure that doesn’t happen. We’re much more likely to run out of time to prepare.
by Miles Brundage, Mile's Substack | Read more:
Images: uncredited
[ed. Can't help but wonder how my kid's, and especially my grandkid's, lives will go. See also: Why I’m Leaving OpenAI and What I’m Doing Next (MS):]Who are you/what did you do at OpenAI?
Until the end of day this Friday, I’m a researcher and manager at OpenAI. I have been here for over six years, which is pretty long by OpenAI standards (it has grown a lot over those six years!). I started as a research scientist on the Policy team, then became Head of Policy Research, and am currently Senior Advisor for AGI Readiness. Before that I was in academia, getting my PhD in Human and Social Dimensions of Science and Technology from Arizona State University, and then as a post-doc at Oxford, and I worked for a bit in government at the US Department of Energy.
The teams I’ve led (Policy Research and then AGI Readiness) have, in my view, done a lot of really important work shaping OpenAI’s deployment practices, e.g., starting our external red teaming program and driving the first several OpenAI system cards, and publishing a lot of influential work on topics such as the societal implications of language models and AI agents, frontier AI regulation, compute governance, etc.
I’m incredibly grateful for the time I’ve been at OpenAI, and deeply appreciate my managers over the years for trusting me with increasing responsibilities, the dozens of people I’ve had the honor of managing and from whom I learned so much, and the countless brilliant colleagues I’ve worked with on a range of teams who made working at OpenAI such a fascinating and rewarding experience.
Why are you leaving?
I decided that I want to impact and influence AI's development from outside the industry rather than inside. There are several considerations pointing to that conclusion:
- The opportunity costs have become very high: I don’t have time to work on various research topics that I think are important, and in some cases I think they’d be more impactful if I worked on them outside of industry. OpenAI is now so high-profile, and its outputs reviewed from so many different angles, that it’s hard for me to publish on all the topics that are important to me. To be clear, while I wouldn’t say I’ve always agreed with OpenAI’s stance on publication review, I do think it’s reasonable for there to be some publishing constraints in industry (and I have helped write several iterations of OpenAI’s policies), but for me the constraints have become too much.
- I want to be less biased: It is difficult to be impartial about an organization when you are a part of it and work closely with people there everyday, and people are right to question policy ideas coming from industry given financial conflicts of interest. I have tried to be as impartial as I can in my analysis, but I’m sure there has been some bias, and certainly working at OpenAI affects how people perceive my statements as well as those from others in industry. I think it’s critical to have more industry-independent voices in the policy conversation than there are today, and I plan to be one of them.
- I’ve done much of what I set out to do at OpenAI: Since starting my latest role as Senior Advisor for AGI Readiness, I’ve begun to think more explicitly about two kinds of AGI readiness–OpenAI’s readiness to steward increasingly powerful AI capabilities, and the world’s readiness to effectively manage those capabilities (including via regulating OpenAI and other companies). On the former, I’ve already told executives and the board (the audience of my advice) a fair amount about what I think OpenAI needs to do and what the gaps are, and on the latter, I think I can be more effective externally.
So how are OpenAI and the world doing on AGI readiness?
In short, neither OpenAI nor any other frontier lab is ready, and the world is also not ready.
To be clear, I don’t think this is a controversial statement among OpenAI’s leadership, and notably, that’s a different question from whether the company and the world are on track to be ready at the relevant time (though I think the gaps remaining are substantial enough that I’ll be working on AI policy for the rest of my career).
Whether the company and the world are on track for AGI readiness is a complex function of how safety and security culture play out over time (for which recent additions to the board are steps in the right direction), how regulation affects organizational incentives, how various facts about AI capabilities and the difficulty of safety play out, and various other factors.
