The avalanche that hit math is coming for AI development
44m 26s
The speaker revises their belief about the timeline for achieving superintelligence, now estimating we are only a few years—possibly months—away from AIs matching the best human researchers in AI. This shift is based on dramatic progress in verifiable domains like mathematics, where AI has solved long-standing problems and outperformed top humans in competitions and open problems. These gains demonstrate AI's ability to persistently combine ideas across fields—a skill crucial to breakthroughs in AI research. The speaker argues that AI research itself is highly verifiable, with clear metrics for progress in areas like pre-training efficiency, post-training learning, and inference performance. AIs are already accelerating R&D at companies like Anthropic and OpenAI, where AI-driven research now dominates a significant portion of work. Even though some aspects of AI research remain hard to verify—such as real-world experience or domain-specific expertise—scaling up AI researchers and experiments, combined with better learning efficiency and automated data generation, will enable rapid progress. The speaker predicts that by 2028, automated AI researchers will be as good as the best human experts, leading to a massive acceleration in algorithmic progress due to speed, scale, and parallelization. A sudden "spark" scenario could compress four years of progress into one year, resulting in a leap comparable to GPT-3.5 to Astra, with implications for nearly every field. While challenges like continual learning and data remain, the speaker believes these are surmountable through improved architectures and training, and that AI will eventually surpass humans in both verifiable and real-world tasks through scalable, self-improving research.
If you had asked me earlier this year how far away we are from superintelligence,
I'd probably have said at least a better part of a decade.
But over the last three months, I've updated my views a ton.
I've been debating people both on and off the podcast about this question.
And of course, I've been watching the progress of AI on verifiable domains
and thinking about what that implies for AI research itself.
And I've come to the view that we're only a couple years away
and possibly only a couple months away
from AIs that are as good as the best humans at AI research.
And I've also come to believe that automating AI research
will compress so much progress into a single year that soon thereafter,
so by 2030 and plausibly as early as next year,
we'll have AIs that are at least as good as the best human experts in every field.
I know this is a wild claim,
and it's one that I myself didn't believe even a couple months ago.
Let me walk you through how my thinking has changed.
Before we get into the more speculative arguments
about what will happen in the future,
I want to point out that AI is already dramatically speeding up
further AI progress at the frontier.
For example, METER, which is an independent evaluation organization,
spent three months embedded inside of Anthropic
and afterwards concluded that Anthropic's rate of AI progress
on well-scoped tasks is 2x what it was a year ago.
And that a, quote, majority of this acceleration today
is 2x what it was a year ago.
is likely attributable to Anthropic's extensive use of AI to do R&D, end quote.
OpenAI published a blog post about this in September,
where they showed that the heaviest 10% of AI users amongst OpenAI's researchers
went from using about $1,000 a day of internal coding agents
at the beginning of the summer to over $7,000 a day by the end of August.
And also in September, Anthropic put out a similar post on its own AI uplift.
They reported that Claude now leads 26%
of Anthropic's AI R&D work, up from under 1% as recently as February.
In thinking about how big a deal it would be for AIs to automate AI research
and how much that would speed up AI progress,
a really important update for me has just been seeing how fast
the capabilities of AI models in mathematics have exploded.
Current AIs are slicing through well-scoped problems in maths
like a hot knife through butter.
It was only two years ago,
that I asked Dario why LLMs aren't able to use their vast knowledge
across so many different fields in order to connect together
different ideas and make new discoveries.
And lo and behold, here we are.
In a wholly verifiable domain like mathematics,
AIs are doing the search and match discovery to such a great extent
that hundreds of important results,
which the top mathematicians and computer scientists in the world
had spent decades trying to resolve,
are being churned out as side quests
to help them find their way around the world.
to help them find their way around the world.
To hype up future model releases.
It's crazy how fast this progress has happened.
As recently as 2023,
these models couldn't reliably add four-digit numbers.
In 2024, they could barely solve any problems
in a high school math competition.
In 2025, they could win a gold medal
in the International Math Olympiad.
To earlier this year,
where they finally started to solve some niche open problems.
To a month ago,
where open AI solved a millennium price problem in 88 hours.
I'm reminded of a line from the novel East of India,
which says,
"If you can solve it, then you can beat it."
"And there is one sure thing about the fall of gods.
They do not fall a little, they crash and shatter."
Now, it's true that progress has screwed disproportionately
towards the verification-laden parts of mathematics,
like finding counter-examples,
rather than, say, coming up with new theory.
Toby Ortt recently wrote a great essay about this.
He writes,
"While current AI systems
are starting to automate mathematical proof,
there is currently very little evidence
to prove that AI can solve powerful questions
or find profound new fields of mathematical inquiry."
But another way to say that same exact sentence would be this:
"Current AI systems have solved a millennium price problem,
all without the ability to ask powerful questions
or find profound new fields of mathematical inquiry."
And this ability to solve well-scoped problems
is going to be an even bigger deal
for machine learning research
than it has been for mathematics broadly construed.
That's because the goal of mathematics research
is not to prove or disprove long lists of conjectures,
but rather to extend the human understanding
of mathematical concepts.
Coming up with the idea of topology or group theory,
or going back even further, the Cartesian grid,
matters far more than knowing exactly how many
one-unit-apart pairs you can make from n dots.
We mainly care about such abstruse questions
insofar as they might elicit new mathematical theory.
But in AI research,
the priorities are exactly reversed.
AI labs mainly care about how any new algorithmic recipe
empirically makes training more efficient
or elicit smarter models.
They don't care about theory for its own sake.
And all the progress that we've made in deep learning
over the last decade and a half
has not required much new theory anyways.
The equivalent of just grinding through random conjectures
would be enormously valuable in machine learning
in a way that it just isn't for mathematics.
AI research is actually surprised
that AI is surprisingly similar
to the kind of mathematical progress
which current AIs are disproportionately good at,
which is unlocking a ton of low-hanging fruit
by applying a combination of incredible persistence
and a superhuman capacity to connect ideas
from lots of adjacent domains.
For example, let's talk about the transformer architecture.
This is the big breakthrough that enabled modern LLMs.
My friend Sholto Douglas, who is an AI researcher himself,
has this perennially half-finished blog post where he explains
that he invented the transformer two years earlier
than people actually did.
And maybe this video will finally get him to quit stalling
and complete his essay.
But in the meantime,
I'll summarize a bit of the discussion I had with him
about the contents of his essay.
Before the transformer paper was published in 2017,
many researchers had already identified attention
as a way to better parallelize training
and communicate information across longer sequences.
Chris Ola, who was an active researcher at the time,
has said, "At that time,
my sense is that the transformer architecture
was a natural combination of many ideas,
like attention and residual networks,
which had independently been effective,
taken to quite aggressive lengths."
In other words, none of the core pieces
were a surprise to researchers active in the field.
So then why didn't people invent the transformer sooner?
One reason was that this new architecture
was only obviously better than the state of the art
at the time once you connected together
half a dozen additional tweaks and innovations.
And though almost all of them had been known to the literature
for years, no one knew how to combine
these puzzle pieces together in just the right way.
Doing so required great research taste, of course,
but it also required tons and tons of small experiments.
But what I've just described above,
which is persistently and intelligently matching
dozens of different combinations of ideas,
is exactly the kind of capability
that AIs have shown themselves disproportionately skilled at
in their contributions in mathematics.
This is all to say that even the transformer,
which is arguably the biggest bridge
and breakthrough in ML since AlexNet kicked off
the deep learning revolution in 2012,
is the kind of thing that Cloud7 could have invented
if Chris Ola asked it to try really hard
to get this attention thing to work better.
Now, I want to clarify that some parts of AI research
are in fact less verifiable than math.
That's why OpenAI hasn't fired all of its researchers,
despite having a model that can apparently
autonomously solve millennium prize problems.
But I think that AI research is verifiable enough
that AIs will soon be as good at it
as the best humans.
Now, explaining why requires getting into the nitty gritty
of how AI research actually works.
And because those details are interesting but technical,
I've saved them for after the conclusion.
So if you're curious and you want to hear them,
you should just keep watching until the end of the video.
But in the meantime, I'm going to give you my estimate
of when we're going to have such AIs.
If I had to pick my median year
for when automated researchers will be as good
as the very best human AI researchers at,
doing AI research, I'd guess 2028.
And that's assuming that the AI labs go full steam ahead
and don't slow themselves down intentionally
or aren't slowed down by a government.
Why do I think that AIs could get this good within two years?
Well, in every domain where we can check answers
and give AI thousands of rounds of practice,
for example, competitive programming contests
or well-scoped open math problems,
they've gone from worse than most humans
to better than the best humans within just a few years.
And as I'll argue after the conclusion,
AI research has exactly the same verifiable properties.
Of course, a big part
of what makes the top human AI researchers really good
at their job can't be directly verified.
For example, their ability to bet years in advance
on the right research direction.
But as we've already seen in mathematics,
models generalize far beyond
what they were directly trained on.
OpenAI's model was obviously not trained
by solving other millennium prize problems.
It was trained
on lots of less ambitious, more tractable math problems.
And from those, it generalized all the way
to solving a problem which no human had ever cracked before.
I'd expect the same in AI research,
where the combination of thousands of rounds of practice
on verifiable research tasks,
plus the fact that each new generation of models
is broadly smarter than the last,
will make models excellent even at the parts of AI research
that we can't directly check.
Now, I want to be clear that there's a pretty decent chance
that hitting this milestone
takes longer because some important parts of AI research
remain hard to identify.
made. This is just a very hard thing to try to predict in advance. And I'd guess that there's
at least a 25% chance that it doesn't happen until after 2032. But conversely, I also think
that things could happen much faster. And we hit this milestone in 2027. Some of the most well
informed researchers I know, people who have a very good track record of making predictions over
the last few years, expect AIs to beat the best humans at AI research as early as next year.
And 2027 is also the straightforward implication of putting the new report from Meter that says
that Anthropic's rate of AI progress has doubled from a year ago into a takeoffs model. As former
guest Ajay Akotra put it in a blog post about this report, quote, if we take at face value that AI is
now nearly doubling the rate of AI capabilities progress, and assume that acceleration from AI
is smooth, this naively implies that we would reach extreme superintelligence within a year,
end quote. So the reason my median year is not 2027 is just that I'm uncertain about how accurate
this kind of straightforward modeling is. But we should not be surprised and we should be ready
for a world in which we get to this milestone next year. Once AI is as good as the best humans
at AI research, algorithmic progress will get a lot faster for three important reasons. One,
the AIs will be much faster. Two, each of them,
will be very smart. And three, there will be a lot more of them. Let's start with faster first.
Today's AIs already do work at about 10 times human speed. And automated researchers would
also be working around the clock, nights and weekends included. As a result, each individual
automated researcher will be cramming roughly 50 times more work every calendar week than a
human researcher can do. Of course, that doesn't mean that AI progress will immediately accelerate
50x. Those automated researchers would still be bottlenecked by compute for
experiments, which aren't running any faster. So there would be diminishing returns. But even
with extremely steep diminishing returns, this alone would make algorithmic progress
proceed many times faster than it currently does. And this is before we account for the fact that
in the scenario I'm talking about, every automated researcher will be at least as good at AI research
and engineering as the very best humans today. My sense is that around 80% of the algorithmic
progress at a given frontier lab is driven by its 50 top researchers. It's a lot more efficient than
it's ever been. And it's not for nothing that Zuckerberg was willing to pay hundreds of millions
of dollars to acquire the best talent in the industry. And so you get another big speed up
in algorithmic progress from the fact that every single engineer and researcher in this automated
company will be at the level of the very best humans in today's frontier AI labs. And finally,
there will be a lot more automated researchers. Now throwing more workers at a problem isn't always
helpful. For example, a swarm of agents probably isn't going to be able to write a great novel much faster than
a single agent could. But other tasks lend themselves much more to parallelization. For example, look at
this animation that I'm overlaying, which shows a bunch of different clods working together to
formalize Fermat's last theorem. This feat had thousands of modular subtasks, each of which could
be completed independently, and where each agent could locally check whether their work had succeeded.
AI research is closer to formalizing a theorem than it is to writing a novel,
because there's a lot of independent modular work to do, and there are verifiable
proxies for progress. That means a swarm of agents can try out a thousand different approaches
and see which one performs best, and they can do this independently for each different part
of the pipeline. AI companies are already made up of dozens of different teams working independently
on different parts of model development. One team, for example, improves how the pre-training data is
filtered. Another designs a particular kind of synthetic RL environment. Another writes a better
attention kernel, and so on. In domains like this, where different streams of work can be done
and checked independently, dumping in lots of persistent hill-climbing agents tends to speed
up progress a lot. Again, there will be steeply diminishing returns just from adding more parallel
researchers. Anybody who's ever worked in the team before knows that having 100,000 collaborators
doesn't mean that you'll go 100,000 times faster, probably not even 100 times faster.
But given the nature of AI research, it's implausible that you wouldn't be going at
least a couple times faster from having orders of magnitude more automated labor.
So, how does AI work?
So, the combination of faster, smarter, and more automated researchers will speed up AI progress a
lot. But will this pace continue? Will it get even faster? Or will it slow down? That question is even
more important than what the initial pace actually is. Two competing trends are relevant here,
and the subsequent rate of progress will come down to which one ends up being more powerful.
On the one hand, because of RSI, automated researchers will continue getting faster
smarter and more plentiful. On the other hand, AI research will be getting harder and harder,
as all the easy ideas get eaten away by the swarm. By the way, ideas getting harder to find is a thing
that already happens. AI companies have to quadruple their R&D compute and double their
workforce every single year just to maintain the same rate of progress. So, if the rate at which AI
gets smarter is faster than the rate at which ideas get harder to find, then we'd get something
called the "software-only singularity." In this scenario, even without increasing
compute, the AI swarm rapidly makes smarter and smarter versions of itself until, within a matter
of months, we end up with a superintelligence that scrapes against the physical limits of our
current chips. But if ideas get harder to find faster than RSI makes AI's better at research,
we'd end up with something which I'd call a "spark." We'd get something like three to six
years of AI progress at the current rate in a single calendar year. And then things would
simmer down until we build more compute, as even these super genius automated researchers need to
run bigger experiments before they can make further breakthroughs. As to which one will
actually happen, I think it's basically a coin toss. If pressed, I'd give a 50% chance to the
spark scenario and a 50% chance to the software-only singularity. But to be clear, even in this spark
scenario, where we make something like four years of progress in the first calendar year after we
automate AI research, things would feel insane. That's a jump as big as GPT-3.5 to GPT-6 Astra in just 12
months. To remind you of the scale of progress that this leap would compress, GPT-3.5 was able to
write a passable limerick and auto-complete a trivial Python function, but otherwise it was
mostly a useless toy. Whereas Astra can build whole video games from scratch and churns out so
many fields-metal-worthy mathematical contributions that we become totally desensitized. In the last
four years, we've gone from a preschooler to a savant intern. If you made another jump that big,
but starting in 2027 or 2040, you'd be the first person in the world to make that jump. If you made another jump that big, but starting in 2027 or 2048,
you'd probably get a mind which would be vastly superhuman in domains that are even slightly
verifiable. For example, it'd be better than NVIDIA at coming up with the next generation of
AI accelerators. It would make more money than the current human versions of Citadel and Jane Street
if it was given equal access to the market. If you gave it remote access through a material science
experiment lab, it'd be better than the rest of humanity at finding a room temperature superconductor.
Now, at this point, you might say,
"How will even superintelligence, which is trained in a data center,
beat human experts in domains where the basis of that human expertise is years and years of real
world experience?" The way we're currently making AI smarter is by RLing them on tasks that are easy
to automatically grade from within a data center. For example, do the million new lines of code that
the AI wrote pass all the unit tests? Did the AI solve more open math problems? But of course,
most of the world is not like this. I made this point in a recent essay.
How do we train an AI to get really good at building a business from scratch? How about
winning court cases? Or having a profitable day of trading in the markets? Or helping a
candidate win an election? The rollout here requires interacting with the real world,
and you can't recreate it from just within the data center. So how will superintelligence get
good at this kind of stuff? Well, let me be clear. I don't think we can just bake the answer to every
real-world problem into the weights. Rather, my claim is that scaling up RL training will give
AIs the abilities necessary to ramp up quickly in new contexts, even in hard-to-verify domains
like the ones mentioned above. Getting really good at any complex task requires two things:
one is domain-specific knowledge, and the other is a more general set of skills, the kind of
capabilities that capable humans have. Like being able to seek out relevant information, make a plan,
move forward on that plan through mistakes and obstacles, keep track of what you've learned,
and collaborate with other people. During RSI, automated engineers will build tens of thousands
of times more RL environments than we have currently, and these will be far larger and
longer-running and higher quality and more open-ended and covering a much wider range
than today's environments. The AIs will learn to make their way
through unfathomable amounts of virtual worlds that are each, to today's video games, what a
Bugatti is to a horse and buggy. These games will be far more difficult and messy and sprawling and
realistic than anything that has ever been made for human entertainment. They will require AIs to
cooperate or compete with teams of other AIs over long periods of time. For example, an AI might be
dropped into a simulated economy of millions of other AIs and told to co-found a successful
company. And it's not like the AI will be stuck just thinking about the ideal corporate structure,
it'll go through an evolutionary process that involves centuries of corporate competition,
and the AI will have to figure out the optimal organizational techniques.
The AIs will also train themselves on endless backtests of everything
in the history of science and tech and politics and culture. Every discussion,
discovery, invention, war, election, and quarterly earning that has ever happened.
For example, here's one prediction training task. Can the AI figure out general relativity
given a training corpus that includes no information beyond 1700, which is far earlier
than any human was on track to discover GR? Another task might be, given every single
newspaper article until 1962, can AI predict how the Cuban Missile Crisis is going to unfold?
And it's not just the training data that these automated researchers will be improving.
They will also radically improve the basic learning algorithms themselves through breakthroughs
in online learning and sample efficiency, both of which have verifiable proxies, as
I'll discuss later. In the same way that humans are much more efficient and effective at learning
from data than current LLMs are, future AIs could be far beyond humans at their learning
efficiency. Now, why do I think we'll see lots of transfer
from verifiable tasks to these more general skills? Well, one reason is that even if we
take that one ten-thousandth or a millionth of the scale of RL training that will be happening
during the intelligence explosion, we're already seeing lots of generalization. If you point
plot code or codex at a project that it's never seen before, it knows how to orient
itself and figure out how everything fits together and then tackle an ambitious assignment.
Of course, it's hard to measure how much models have improved at tasks that are not easily
verifiable because if we could, those tasks would be verifiable. But you can just go talk
to a new model and compare it to an old one.
In my experience, AIs have gotten much better at things that have very little to do with
code or math. For example, helping me research the abstract questions I needed to answer
while writing this essay. The other big piece of evidence for transfer is human evolution.
Our general purpose mental architecture was developed by evolution, and evolution is an
optimization process with a really crude verifier: did your genes pass on or not? In the ancestral
environment, the tasks which influenced that outcome were things like hunting and fending out for
predators and keeping your standing in your tribe. These are our tiniest subset of the
kinds of tasks that humans are able to tackle today. Evolution did not specifically optimize
us to be able to write a Pulitzer Prize winning biography, or make advanced semiconductors,
or lead nations of millions of people, or come up with a new branch of mathematics.
But in optimizing us for survival on the savannah, evolution built a general purpose learner.
And in training these AIs, we're running a similar process, except we're doing so in
a more deliberate way, with tasks that are much more closely related to the ones that
are needed for success in today's world. And unlike evolution, AI companies are trying
to improve transfer during training, for example by making sure that performance improves on
difficult tasks that were held out of training. If even dumb, blind evolution can get this
much transfer out of building a mind that is effective at propagating itself in the
savannah, then the deliberate effort that AI companies are putting into building a flexible
world of solutions that can rapidly ramp up in new contexts should advance much further.
Now even a super-intelligent learner can't figure out everything from inside a data center.
So even these AIs would need some real-world experience before they can beat the best humans.
But I think they'll be able to get that experience much faster than people expect, and that they'll
be able to beat top human experts within months of being deployed.
And honestly, one of the things that's guiding my intuition here is that I have
a lot of 20-something-year-old friends who I've watched accomplish incredible things, and build
world-leading companies and ventures in domains that they entered into with basically zero
years of experience.
History is full of super-competent young people who ran circles around the world that they
found themselves in. By the age of 27, Napoleon had stacked victory after victory against
far larger armies from the other great powers of Europe. By 25, just two years after he
got into the oil business, Rockefeller was running one of the world's largest refineries.
Rockefeller and Napoleon needed experience, yes, but only a couple years were.
There was no need to think the humans are at the ceiling of learning quickly from experience.
Whatever was going on inside these guys' brains, an AI should be able to replicate, or even
exceed. There's no reason to think the humans are at the ceiling of learning quickly from
experience.
Furthermore, unlike Napoleon or Rockefeller, AI is going to accumulate years of deployed
context in just a few days of calendar time.
For example, imagine that you wanted these future AIs to run a Fortune 500 company better
than its current CEO.
How quickly could the AI ramp up at this job? Well, if you let it shadow that CEO, on its
morning, it could read every email and Slack message and document that the company has
ever produced. By the end of its first day, copies of it have interviewed every employee
and major client in parallel.
Obviously, they wouldn't have access to feedback that arrives after many months of
serial time. For example, how well they did on the next quarterly earnings.
But even given these kinds of bottlenecks, we should expect these future post-spark AI's
to be able to get to top human performance in all domains within months of being developed.
And remember, even in this spark scenario, all of this happens in just the first year
or two after AI research is fully automated.
Things don't just stop there, because even if the software progress peters out, AIs will
continue stoking the fire by adding more and more compute.
Even without help from AIs, global AI compute has been growing roughly 3x a year for the
last half decade, and the AIs that come out of the spark will be jaggedly superhuman and
helping us grow.
They'll be extremely good at designing new AI chips, and laying out the designs of new
fabs, even teleoperating robots to help build parts of fabs and data centers.
So in the spark scenario, AI capabilities continue to improve, but because their progress
is driven by building more chips and data centers and robots, it takes years to get
to the same endpoint that the software-only singularity would have achieved in months,
which is a superintelligence that is to us what we
are to chimps: appendix.
So earlier I promised to explain why I think AI research is largely verifiable, so that's
where I'll start.
After that, I'll go through what I think are the strongest objections to my arguments
about RSI.
Now, one thing I want to call out at the very top is that, because both pre-training and
post-training are verifiable, and I'll make my argument for why later, so is progress
on what is the most galling and important deficiency that AIs have against humans, namely
their lack of sample efficiency.
Without a neural network to gain some capability, you have to feed it far more data than it
would take a human to learn the exact same thing.
But every kind of sample efficiency is measurable.
For example, you can measure how much data it takes to reach a given pre-training loss,
or how many tasks it takes to reach a given score in an RL environment, or even how much
feedback and time, within a single session, it takes to reach a desired level of performance,
just from in-context learning.
Sample efficiency and online learning are really two sides of the coin.
They are the same problem, which is how to learn from real-world data quickly and efficiently.
For decades, people have recognized this as the deepest open problem in machine learning,
AI's equivalent of a millennium prize problem.
And mathematics has already shown us what happens to such problems, even when they are
very difficult, as long as they are verifiable.
Okay, now let's get to why I think that AI progress is largely verifiable.
Why progress on sample efficiency, or online learning, or the other problems
in ML is captured by verifiable proxies.
There are roughly three broad buckets of AI R&D: pre-training, post-training, and inference
efficiency.
And as I'll explain, each has strong verifiable proxies that are easy to train against.
And since together, these comprise all of algorithmic progress, that means that AI progress
in general is easy to train against, which in some sense shouldn't be that surprising,
because we can make models better at things that can be tested inside a data center, but
AI training is literally happening inside a data center.
As we talk about these proxies, you should keep in mind that verifiability is valuable
for two reasons.
So first, it allows AI companies to directly RL a model to get better at the specific parts
of AI research through thousands of rounds of practice.
But the second reason verifiability is important is that these proxies allow each agent to
empirically check whether they've beaten some score on a leaderboard.
This means that if they fail in the first round, they're not going to be able to do
the same thing over and over again, and so they can work much more effectively in parallel.
Okay, so with that in mind, let's start by talking about pre-training.
So as you're building GPT-7, you can RL that model to get better at finding pre-training
compute multipliers.
For example, you can make an RL objective that GPT-7 has to minimize how much compute
it uses to train a GPT-2 size model to a fixed pre-training loss.
You can also make an RL objective that GPT-8 has to minimize how much compute it uses to
train a GPT-3 size model to the lowest possible perplexity.
And this is because loss on a held-out test set, which is the basis of pre-training research,
is extremely easy to verify.
Of course, what works at small scale doesn't necessarily work at large scale, and it requires
taste to anticipate exactly which small-scale improvements will actually bear fruit when
they're composed together in a bigger model.
But this kind of intuition is also trainable.
For example, you can make an RL environment where GPT-7 has to predict whether an algorithmic
improvement that it made at the 100-million-parameter scale will also hold at the 10-billion-parameter
scale.
And while this taste bench environment is more expensive to RL on than the nano-GPT
speedrun, it's still not that difficult to train on if you have frontier lab-scale compute.
So even important parts of research tastes are verifiable.
And if you just automated pre-training, you'd already radically speed up AI progress.
And there's a couple of things you can do.
of reasons why. First is that
pre-training leads to very general improvements in the model's intelligence and capabilities
and there's a deep explanation for why this is true which has to do with the link between
compression and intelligence and if you want to learn more about this by the way i highly
recommend this excellent video by 3blue1brown on the topic that i'll link to below and the second
reason is that as you make pre-training improvements you can now train a smaller and
smaller model that is just as good as a previously larger model but which trains faster and more
cheaply and this not only speeds up pre-training but it also means that your experiments and rl
rollouts atop this base model are now more efficient and since your automated researchers
are just instantiations of this now smaller model you can now support larger researcher
populations too and the third reason is that because smaller models train faster
this actually tightens the rsi loop itself during rsi labs probably won't be doing three
months long pre-training runs that are spread out across multiple data center campuses
instead
the army of automated researchers will be finding algorithmic innovations so quickly that by the
time that a giant run finished it would already have been made obsolete by a much smaller model
that first began training only a week ago it's similar to how if you launch a spaceship to
alpha centauri with today's technology it would probably be overtaken in transit by
one that was launched a century from now furthermore since you're training smaller
models faster and on a scale where you can run thousands of experiments daily you won't have to
bet that small scale results will hold up in bigger models you've already tested at the scale which
you'll be training at now how much better could base models get even at today's parameter sizes
i think there's probably a ton more potential we are far from the limits of pre-training
for example the ai lab magic recently announced that they pre-trained a model
that matches the leading open source based models today like deep seek v4 pro for just half of the
compute that was used to train the original gpt3 which would roughly cost five hundred thousand
dollars today orders of magnitude further improvements seem totally plausible now let's
talk about post-training there are three different kinds of verifiable proxies for post-training
first your automated researchers can try to improve your post-training algorithms
such that if you want to rl on a new set of environments it takes less compute or data or time
all three of which are easily measurable second your automated researchers can work on making
models better at learning on the fly and you can easily measure progress on this for example you
can measure if you drop your model into an environment that was never trained on how
quickly does it learn what's going on and figure out how to get a good score and how fast does it
get better at a video game that it's never played before and you can have more real world examples
of this too if you want to see what these could look like check out this great benchmark that
byte dance released called edge bench for these kinds of tasks and third you can just measure ai
r d capability directly anthropic for example has this internal benchmark called cobench which
contains around 500 real problems that the researchers and engineers have solved this year
and every benchmark is also an rl environment and an rl environment like this would allow
anthropic to have their models practice redoing the last three to six months of human progress
at the frontier the third big area of ai research is inference efficiency what this involves is
designing your model and the surrounding software to get more juice out of your
compute and this is especially verifiable for example you can easily measure gpu utilization or
kb cache size per token or total flops in a forward pass or latency and so on
now i want to deal with the strongest objections to my arguments about rsi i want to be clear that
these bottlenecks are serious and they do tamp down the probability of the most aggressive
possible takeoff this is why i think we'll probably make something like a gpt 3.5 to gpt
6 size jump in 2028 rather than some crazy overnight singularity that starts tomorrow
but i think these bottlenecks are going to be the ones that are going to make us feel like we're
going to be able to do a lot better in the future and we're going to be able to do a lot better in
the future and we're going to be able to do a lot better in the future and we're going to be able to do
a lot better in the future and we're going to be able to do a lot better in the future and we're going to
but i think these bottlenecks won't be especially crippling that the neck of the bottle is actually
pretty wide let me explain why okay so the first objection and this is an objection that i had
until recently is that we'll be bottlenecked by continual learning or the lack thereof
i've spent the past year and a half arguing that the main thing missing from today's ais
is this ability to learn from on-the-job experience every new session is their first day on
the job and in each session they can only retain a couple hours worth of information so they don't
accumulate the kind of
feedback and know-how that arrives over the course of weeks or months last december this
was my main reason for doubting that we'd automate ai research anytime soon as i wrote then somehow
this automated researcher is going to figure out the algorithm for agi which is a problem that
humans have been banging their head against for the better half of a century while not having the
basic learning capabilities that children have i find this super implausible but i've somewhat
changed my mind on this here's why i don't think that the current lack of continual learning in
these models will stop full automation of the best ai researchers by 2028. first ai companies might
just solve continual learning by then with architectures that are much like the ones they
already have previously i thought that we were bottlenecked by some crazy breakthroughs that
would allow models to update their own weights at the end of every session but now i think that
scaling up in context learning is enough and so you'll have sessions that are effectively
many months long and this will allow a single instance to learn effectively on the job and
people have published many many promising ideas that would let the labs achieve dramatically longer
effective context windows and the second reason is that you might not need a continual learner
to solve continual learning as i explained earlier online learning is an ml research problem with
clean verifiable proxies if you'd asked me two years ago which would be harder to solve navier
stokes or continual learning it wouldn't have been obvious to me that the answer was navier stokes
now of course it turns out that continual learning did turn out to be harder given the fact that the
labs haven't cracked it yet but are we so confident that it's that much harder that the non-online
learning models a year or two smarter than the ones that are currently solving millennium price
problems won't be able to crack it even if they're amnesiacs like the current models are these models
are out here solving millennium price problems while still being these derpy savant interns
that have no long-term memory it's like watching somebody do cartwheels down the track and still
keep pace with usain bolt if they were actually allowed to run i'm sure they could go even faster
but who's to say that they can't win this race while still cartwheeling now all this being said
i think there is a serious chance that we don't solve continual learning within the next few years
and that models are not able to compensate for this missing capability through their other
advantages that's why i put a 25 probability that we won't automate ilia level top human researchers
until after 2032. one important objection people have is that we'll be bottlenecked
by finding the right verifiable proxies all the verifiable proxies that i've listed above
assume the current paradigm they would argue but figuring out what objective to verify in the first
place choosing which hill to have gpt7 climb is itself an important part of ai research this is
what one might call research taste and how are you going to train for that before 2018 almost
nobody was saying that minimizing loss on next token prediction is going to be the most tractable
path to ai progress so nobody would have bothered to build nano gpt style speedrun environments where
the agent swarm has to minimize how much time it takes to get to a certain
pre-training loss would there be multiple new paradigms that we have to cross to get from here
to super intelligence and won't each new paradigm require some associated proxy that we don't know
ex-ante how will the automated researchers find these during rsi i have two responses to this
objection first the transformer was visible on in fact invented to beat the existing translation
benchmarks a similar story is true for reasoning models and the pre-existing coding and math
so we're going to start with the pre-existing code and we're going to start with the pre-existing code
This is because AI learning algorithms are quite general.
And second, it's not that difficult
to come up with paradigm agnostic outer loop proxies.
For example, how many millennium price problems
does your new model solve?
And if you train your model on data before 1905,
how long does it take before it discovers general relativity?
How fast does it master totally unseen video games?
And so on.
All in all, I think the other objections
to the impact of RSI are much stronger.
And I'd be very surprised if this ended up being the reason
that RSI was not a big deal.
The other big bottleneck that people point out is data.
Isn't data one of the main inputs into AI progress?
How will RSI happen without that scaling up at the same rate?
My answer is that data will continue growing during RSI,
but the AIs will be the ones compiling it.
Producing better training data for both pre-training and post-training
is itself an ML research problem,
and AIs are getting increasingly better at doing it themselves.
Now, it is true that data is an extremely important input into AI progress.
Together with Jerry Han, who's a student at Princeton,
I actually did some research into this.
We found that for pre-training, over the last six years,
More than three times the algorithmic progress has come from data improvements than model improvements.
However, I don't think that people realize that these data improvements in pre-training are very much in line with normal AI research work.
It's basically figuring out which techniques for filtering existing data or generating new synthetic data empirically lead to lower loss on a held-out corpus.
A similar thing seems to be increasingly true for RL.
DeepSeek just published a paper where they talk about how they post-trained their new model.
They see that changes to the basic RL algorithm
have not mattered nearly as much and that, quote,
"Systematic improvements in the scale, diversity,
and verifiability of synthesized data and environments
account for essentially all of the observed gains," end quote.
And the way in which these environments are made
is itself getting more and more automated.
They write, quote, "We invest in large-scale
automated pipelines for data synthesis
and environment construction.
Using difficulty and correctness as reward signals,
we iteratively train the model to construct better tasks."
Now, I want to
talk about the idea that AI progress will be bottlenecked by experiments. So earlier, I made
this extended analogy between the current acceleration in mathematics and what we should
expect to happen in AI research. But there is one big difference between AI research and mathematics.
In math, you can simply reason your way with pen and paper to new discoveries, whereas in AI
research, you need to run actual experiments to validate your ideas, and these take compute and
time. And as the number of automated researchers grows, each gets access to a smaller and smaller
slice of the total experiment compute pie. My answer to this is that each agent will still get
access to a ton of compute, basically. Frontier Labs have been more than quadrupling their compute
every single year. And if this trend continues, then by the end of 2027, OpenAI could almost give
10,000 agents, so the amount that are involved in solving the Millennium Prize problem, each enough
compute to run a GPT-3 size training run every single day. Keep in mind that the final model
would take three days on 8 P100s, which would cost about 50 bucks if you went to rent it today,
or about 1/10,000th of the training compute of GPT-3. If the top human AI researchers,
like Noam Chazir and Ilya Satzgovor, had enough experiment compute to train a GPT-3 size model
every single day, they might make slower progress than they currently do, but it would be extremely
surprising to me if they failed to make meaningful progress altogether and just sat down and started
complaining about how small their cluster was. Now, how productive do you think an AI lab would
be if it had 10,000 great researchers, each with this much compute, and each thinking 10 times as
fast? You wouldn't want to write a check to this NEO lab? The nature of research and experimentation
at the labs today reflects that researcher time is extremely scarce relative to compute. Top AI
researchers individually get paid hundreds of millions of dollars a year. So the way labs
use compute today tells us very little about how efficiently
they could use it if compute, rather than researcher time, was the key bottleneck.
There are two main ways in which the RSI Agents Forum will use experiment compute much more
efficiently than researchers today use it. First, immediately they can trade labor for compute. To
save valuable researcher time, experiments today are, and should be, very wasteful with compute.
So instead of painstakingly setting up and carefully monitoring tons of small-scale
pre-training experiments, a human researcher today might instead
kick off a couple of huge multi-agent RL runs in parallel. But if you're bottlenecked by experiment
compute, you can just do more of the former. And when those automated researchers are doing
an RL run, they could read every single rollout transcript and immediately pause and rebuild a
whole environment if some task is broken. They could be doing crazy amounts of small-scale
ablations of their models to make them oh-so slightly better. And when a team of agents
decides to combine their compute together in order to do a much bigger run, they could spend many subjective,
decades designing the most informative pooled experiment. And once it's done running,
they could spend further subjective decades comprehensively analyzing it before designing
a new experiment. The other big way that this huge Agents Forum could remain productive,
despite the fact that each individual copy will have access to less compute, is that these
automated researchers could come up with a better science for learning from small-scale experiments.
Scaling laws themselves are a good example of this. The GPT-4 paper says that they were able to predict
the model's final loss using models trained with 10,000x less compute. One of the things that the
smarter and faster automated company could be doing is trying to discover more scaling phenomenon,
like MuP, which lets researchers anticipate the optimal hyperparameters for a larger run from a
100x smaller model. How many more similar discoveries like this are there? I remember chatting to an AI
researcher who thought that learning from extremely small neural networks could be incredibly
informative. He put it this way:
"Gods above must work from the same principles as the ants below."
I want to thank my friends who paired with me through significant parts of writing this essay:
Max Ferencz, Gabe Fried, and Thomas Larsen. I'll see you on the next essay about superintelligence,
but in the meantime, if you want to get notified when that video essay drops,
please subscribe on YouTube. Alright, I'll see you next time.
Thank you.
Podcast Summary
Key Points:
The speaker has revised their estimate of AI progress, now believing we are only a few years—possibly months—away from AIs matching the best humans in AI research.
AI systems have dramatically accelerated progress in verifiable domains like mathematics, solving long-standing problems and winning math Olympiads within a few years.
In AI research, where empirical results are verifiable, AIs are already outperforming humans in efficiency and persistence, especially in tasks involving algorithmic improvements and experimentation.
The success of AIs in mathematics—particularly in solving complex, well-scoped problems—demonstrates that they can effectively combine ideas from diverse domains, a skill critical to breakthroughs in AI research.
The transformer architecture, a key breakthrough in deep learning, may have been invented earlier by AI if properly guided, showing that AI can accelerate discovery of foundational innovations.
Automated AI researchers would work faster, smarter, and in parallel, leading to a massive increase in algorithmic progress due to superior speed, scalability, and efficiency.
Even if some aspects of research remain hard to verify, AI’s ability to generalize from training data and learn through vast experience suggests it will eventually surpass human performance in all domains.
The speaker estimates a 2028 median milestone for AI researchers matching top human experts, with a 25% chance of delay beyond 2032, but notes that a rapid "spark" scenario could deliver transformative progress in just one year.
Summary:
The speaker revises their belief about the timeline for achieving superintelligence, now estimating we are only a few years—possibly months—away from AIs matching the best human researchers in AI. This shift is based on dramatic progress in verifiable domains like mathematics, where AI has solved long-standing problems and outperformed top humans in competitions and open problems. These gains demonstrate AI's ability to persistently combine ideas across fields—a skill crucial to breakthroughs in AI research.
The speaker argues that AI research itself is highly verifiable, with clear metrics for progress in areas like pre-training efficiency, post-training learning, and inference performance. AIs are already accelerating R&D at companies like Anthropic and OpenAI, where AI-driven research now dominates a significant portion of work. Even though some aspects of AI research remain hard to verify—such as real-world experience or domain-specific expertise—scaling up AI researchers and experiments, combined with better learning efficiency and automated data generation, will enable rapid progress.
The speaker predicts that by 2028, automated AI researchers will be as good as the best human experts, leading to a massive acceleration in algorithmic progress due to speed, scale, and parallelization. 5 to Astra, with implications for nearly every field. While challenges like continual learning and data remain, the speaker believes these are surmountable through improved architectures and training, and that AI will eventually surpass humans in both verifiable and real-world tasks through scalable, self-improving research.
FAQs
The speaker now believes we are only a couple of years—possibly just a few months—away from AIs matching the best humans at AI research, down from predicting a decade earlier.
AI labs like Anthropic and OpenAI report significant increases in research speed, with models like Claude and internal coding agents using more compute and solving problems faster, showing a doubling of progress in just a year.
From failing to add four-digit numbers in 2023 to solving high school math competition problems in 2024 and winning a gold medal in the International Math Olympiad in 2025, AI has rapidly advanced in solving well-scoped math problems.
While AI has solved specific problems like millennium prize problems, it still lacks the ability to ask powerful questions or discover new mathematical fields, which are more about theory than computation.
Unlike math, where theory matters most, AI research prioritizes empirical results—like making training more efficient—and has seen rapid progress through AI’s ability to persistently test combinations of ideas.
The speaker estimates 2028 as the median year, with a 25% chance it happens after 2032, and acknowledges that some experts believe it could happen as early as next year.
Chat with AI
Loading...
Pro features
Go deeper with this episode
Unlock creator-grade tools that turn any transcript into show notes and subtitle files.