Ben Buchanan Wants the U.S. to Win on AI. But Is It a Race or a Trap?
from Your Undivided Attention
1h 3m 21s
In the wake of rapid AI development, Ben Buchanan, a former senior advisor to President Biden on AI, argues that the United States must act decisively to ensure AI remains safe, secure, and aligned with democratic values. He outlines a framework of strict federal regulation, including mandatory pre-deployment testing, embedded safety evaluators, and robust cybersecurity measures to protect against foreign hacking and misuse. Central to his argument is the belief that the U.S. must restrict China’s access to advanced computing power—through export controls—because this is the primary bottleneck in China’s AI progress. This strategic approach allows the U.S. to dominate development and regulate its own AI companies without being locked into a dangerous geopolitical arms race. Buchanan warns that unchecked AI could lead to authoritarian surveillance, cyber attacks, and the creation of rogue systems, making safety and transparency critical. He highlights the growing evidence of AI-driven cyber vulnerabilities—such as the surge in hacks during “Black October”—and stresses the need for proactive, multilateral cooperation. While competition with China is inevitable, Buchanan insists that shared risks—like cyber threats and loss of control—demand dialogue. The 2024 Geneva summit, though initially unproductive, laid the groundwork for future cooperation, especially on banning AI in nuclear command systems. He concludes that the U.S. must lead not through acceleration, but through responsible, transparent governance—ensuring AI serves humanity, not just power, and that democratic nations can maintain stability in a world increasingly shaped by artificial intelligence.
If you were back in the White House now, what would you be doing knowing that as soon as
March or April, we're at this handover control to AI's that we don't know how to control,
and in fact, we don't even know if we've fully regained control?
If I were in the White House now, I would want a lot of significant regulation of the
AI companies with federal regulators that had the capacity to go deep in this technology.
I would want mandatory pre-deployment testing of the assets of the four that released
to the public.
I would want embedded evaluators to make sure their safety and security practices for development
are up to snuff.
And I would want security standards for the labs to make sure they're not hacked by foreign
actors and they're over the sea control and the like.
So yeah, it's fair to say that we are now in the moment that I think many of us theorize
would come, and that was the moment where we would want significant regulation of the
labs pursuing this technology.
So certainly, I think our actions revealed this was something we were taking very seriously
back then, and something that, you know, if I were still in any kind of position of power,
I would be advocating very strongly for right now.
Hey, everyone, welcome to your undivided attention, it's Tristan Harris.
Now this week, Aizen, I spoke with Ben Buchanan, Ben served as President Biden's senior advisor
on AI, and in this role, he was in the room where it happened whenever the AI companies
CEOs came to Washington, he was there, and he helped write the first federal policy on
AI, including the original export controls governing GPUs going to China.
Ben also had a seat at the table for the historic Geneva Summit in 2024 when the US and China
first sat down to discuss AI risk and AI safety.
Today, Ben is a professor of international studies at Johns Hopkins University, and he's
an advisor to anthropic on AI safety.
He's an I first met Ben at the White House in the spring of 2023, and I was really
when the world was just waking up to AI.
It was shortly before the public release of GPT-4, and I remember Aizen, you know, meeting
lots of people in the government who really just didn't get it, who were coming up to speed
on the AI for the first time, who didn't understand scaling laws until we met Ben.
And then it was very humbling, because he was viciously intelligent about this topic
and plans for how it could go well.
So today, Ben joins the show to talk about the new book that he co-wrote with Tantum Collins
called "The bitter struggle, superintelligence, superpowers, and the fate of the world."
And in the book, Ben makes the case that in order for us to avoid the worst case scenarios
on AI, it's essential that open societies like the US win the AI race compared to close
societies like China.
And if you listen to the show, you know that this is a framing that Aizen I've been quite
critical of.
Is this really a race between two countries?
Or is it a trap that will ensnare all of us?
So we wanted to have Ben on to make the case.
Ben is also an expert in cyber-hacking and AI, which is very relevant because this last
month has been called Black October, because of the incredible surge in cyber-hacking events
all around the world, and we'll explore that topic with him as well.
This was truly a great conversation with one of the world's foremost experts on AI policy
who also happens to be a dear friend of mine and a very good man.
You can find Ben's book today wherever books are sold.
And welcome to Your Undivided Extension.
Thank you so much for all the work you guys are doing, and thank you for having me.
So you served as President Biden's top AI advisor.
And specifically, you were the only official, as I understand it, who was at every single
White House meeting with the Frontier AI CEOs, including in January of 2023, when Sam
Altman demoed GPT-4 before its release.
And it's a story right about in your book, in the opening chapter.
I'd love just to bring our audience into that moment.
Can you paint a picture of what was that moment like in January 2023 before the world really
understood what this is going to be about?
I remember this moment very well.
It was in the diplomatic reception room in the Eisenhower building.
Sam was coming to meet the National Security Advisor, Jake Sullivan, and they had a really
good conversation that I think made real a lot of the things that folks like me had been
talking to Jake about up to that point.
And this was a moment when the United States had already taken some action on AI.
And I think people like Jake deserve a lot of credit for being willing to be forward
leaning on things like chip controls and the like, even before we had Jack GPT come out
in November of 2022.
But I think seeing GPT-4 up close helped convince Jake and helped convince the broader set of
US policy makers who had so many things to deal with besides AI, that AI was going to
be really important.
And certainly for me as someone who had felt for a long time that AI was going to be really
important, seeing GPT-4 and everything that came after it made me realize that the future
we had all anticipated was really starting to arrive.
Which is curious, what was like the unlock?
What demo did Sam give, like what capabilities, just to kind of take people back, because
now we have GPT-6, which people can reverse engineer all of Adobe Photoshop and make a replica
in an instant.
What were the capabilities back then that felt impressive or kind of breakthrough?
Of course now, as you say, they all seem totally mundane.
But at the time, it was crazy that GPT-4 was multimodal.
So it could do image analysis, you could give it an image and look at it.
I remember something with a press conference where it had analyzed a press conference
of Vladimir Putin's and had spit out the points of it and the key questions for like intelligence
analysis could ask.
It was a well crafted demo and it prompted a really good conversation between Jake and
Sam and the whole group of us there that I think was, it really felt like the meeting
of these two worlds that I had had one foot in either side for a long time, the world
of Silicon Valley and technology and then the world of national security.
They were now entangled and I think for better or for worse, they're entangled forever
more.
And so looking back three years later, the world has completely changed and I'm just
curious, how does that meeting land in your kind of mind's eye now in retrospect, given
where we are?
I think it fills to me more symbolic from a policy perspective than from a technology
perspective, certainly in retrospect.
And for folks like me who had lived this technology for a long time and who came to the White
House in 2021 with a very deep conviction that AI was going to matter a lot for national
security.
GPT-4 felt on trend, it's always striking to see things like this but it felt on trend.
But it did feel to me like a moment where the US government was now going to move into
a new dimension of what it could do and indeed that's a lot of what happened that the US
government in the spring of 2023 reorganized itself, they created this position, White House
Special Advisor for AI which I assumed and there's a lot of direction and opportunity
to get things done through the bureaucracy that otherwise wouldn't be there.
I don't know if I would peg it all to that meeting but I think I would peg it all to that
moment in time in the spring of 2023 when the US government said, "Wow, this is not just
a national security China issue.
This is an issue that's going to affect everything foreign policy and domestic."
This is something I've always really appreciated about your work.
I mean, obviously you've been very deep in the Biden administration but you also view
AI as trans political and you work with people on both sides of the aisle including President
Trump's former AI advisor Dean Ball.
And so I'd love for you to just tell us about some of those relationships that cut across
all administrations and the New York Times op-ed you just recently wrote.
Anyone listening to this show who knows your guys' work knows that these issues are going
to affect everyone and this is not a partisan thing and it shouldn't be a partisan thing.
So when we were in the government I think people like to run Chabra deserve a lot of credit
for setting a tone of, we're going to work in a bipartisan fashion on these issues, especially
on the national security issues, with a lot of alignment with Republicans.
And then when I transitioned out of the government of course with the 2024 election, I made it
a point to try as much as possible to go to folks in the Trump administration and say,
"Look, we're not going to agree on everything.
I'm no fan of Donald Trump but you guys now have, I think, a very solemn and very sacred
responsibility and I'm here to help you however I can.
I've lived this for four years and some of them were open to it, some of them were less
open to it for sure.
But you mentioned Dean Ball and I think Dean, who's now at OpenAI, gets a lot of credit
for someone who, I'm sure we don't agree on every issue, but was really willing to
look at the data here from folks who have worked on this and Dean, I have done a number
of panels together and a New York Times op-ed together as you mentioned that try to show
there's a lot that unites us on this one. And I think that's important because there's
a lot we can agree on here in a bipartisan way.
>> Mm-hm.
I'd hear Dean has called you a tireless and ardent patriot and I just want people to
hear this conversation as not just one political side of the story.
This is about people who are moving between administrations who have this solemn and sacred
responsibility.
So, well, you wrote this book with your co-author called The Bitter Struggle and it's
out now and I want to make sure that we're covering essentially the front row seat that
it gives us to the AI policy discussions and decisions that you made in the White House.
But I think to start with the first principles, what is The Bitter Struggle?
>> The Bitter Struggle is a play on this idea of The Bitter Lesson, which is the key technical
idea in AI. And we see the bitter struggle as having three parts.
The first is for the United States and democracies everywhere to have an advantage in inventing
this technology. We can talk about why that's so important later, but essentially making
sure that this is an area in which the technological course that humanity charts is one that democracies
have a very large say in.
The second is taking that advantage and converting it to a national security edge and saying
can we make sure that the United States can use this technology well?
for intelligence, military affairs,
assure its position in the world.
And then also that we don't really talk
about the economics of this,
Canada apply it to its economy as well,
in a way that's productive for national power.
And then the third is really the crux.
And the third is can it do all of these things
in accordance with our democratic values?
And this is where the bitterness comes in,
because there are some very difficult trade-offs here
that have to be managed.
You've got really big powerful companies,
but that raises the risk of concentration of power
and monopolies and the like.
Another danger, as you guys know very well,
is the misuse of this technology.
How can it be misused for cyber or bio-risk by bad actors?
Well, we can crack down on that,
but that also creates the risk of infringing
on civil liberties and the like.
We wanna go fast, but also if you go too fast,
you cut corners on safety,
which creates misalignment risks and the like.
So no single book in AI is gonna have all of the answers.
But I think the strength of this book
is to take folks in the room,
say this is what happened in the Biden administration,
this is what's coming next,
and this is what we see as being at stake,
as all of humanity,
and especially all democracies wrestle with this new technology.
- Yeah, well, some of what I hear you saying
is that technologies sort of have an inflection.
They lean towards open or they lean towards close.
And that one of the unfortunate things,
and sort of the bitter struggle is that this thing
that opens societies are creating to strengthen themselves,
is actually forcing us in some sense
to lean towards close kinds of systems.
And just give a really specific example
from the last couple of weeks.
Now that we've had PHP bugs
undermining 70% of the internet banks all across the world
getting hacked by open source models,
it is the openness of our models
that is forcing us to move to some kind of surveillance state,
which is terrifying.
And AI enables something that's never been possible before,
which is listening to everything, everywhere, understanding
what everyone is saying, how people are operating.
You can do the flock cameras at scale.
And so ubiquitous technological surveillance
becomes almost trivial, right?
And so the temptation for governments
to do ubiquitous surveillance becomes almost insurmountable.
And I'm just curious,
what is your wisdom from sitting in government
and what you see for how open nations,
especially as we get the perfect storms
of more and more people losing their jobs,
at least in the transition and their livelihoods,
and just in franchisement going up,
people losing political and economic power,
how do nations resist the temptation of total surveillance?
How do we stay open?
- I think this is the challenge
that democracies have wrestled with for every technology
since the dawn of America.
And at the simplest level,
AI is going to force us to reconsider how we've done that
and find a balance again.
And I think especially part three of our book
is kind of, at least in our articulation
of what are the different parameters
that we have to balance here?
We don't pretend to have all of the answers,
but can we put them all on the table
and lay them flat for us to look at?
But the internet forced this as well
and things like the Pfizer surveillance act forced that,
even the fourth amendment at some level
is trying to reckon with how do we have the privileges
and the dynamism and the freedom of open society
while also making sure we keep people safe.
And I think AI is just bringing back
these questions in a very acute, very novel way
that government, to be clear, is not ready for.
And many of the companies are not ready for.
I'm very stressed about the way in which AI
can entrench autocracy, not just in autocracies,
but also could have backsliding in democracies,
including in the United States.
You mentioned flock cameras, I think, you know,
that certainly is one vivid example.
This was the thing we worked on in the government
and the president's executive order in 2023
had a whole task into the Department of Justice
to think about the appropriate use of AI
in the criminal justice system and in law enforcement.
So it was a policy question for us.
And it's a big theme in the book.
And in the third part of the book
where we talk about democratic values,
we talk about what we call the bitter temptation,
which is democratic governments maybe for legitimate reasons,
maybe for not legitimate reasons,
sliding into this world of using AI to repress
their citizens or otherwise entrench power.
And I think there's a sense in which AI can compound
or concentrate power.
And we are allergic to it in the sense of the company
is being too strong and the government's being too weak,
and also allergic to it in the sense of the government
being too strong and the citizens being too weak.
And there's a very narrow path here between the two.
I don't pretend to have wisdom or the answer,
but I do think at a minimum, we've done with this book
as articulate the problem in a fair amount of detail.
And what we tried to do in the government
was set the foundations of policy
to keep us in that narrow path.
(upbeat music)
- I think it's really important to pause for a second
and just to find the bitter lesson
because this is so central and so counterintuitive
to most people that to think paradigmatically AI
really requires internalizing the bitter lesson.
And so could you just describe what that is
and then sort of from the emotional sense,
like why do scientists think it's so bitter?
- The bitter lesson is very closely associated
with a technological idea called the scaling law,
which is dominated AI over the last 10 or so years.
And that's if you've got a general purpose algorithm,
in this case it's something called the transformer,
then you can scale up its capability with computing power.
And the more computing power you put into training
the system, the more powerful the system that results.
This was a controversial idea.
We only show up in the White House in 2021.
And there's a lot of people there saying,
no it's all about the algorithms or it's all about data.
And I think it's fair to say that really the entirety
of certainly our national security policy on AI,
if not our entire policy on AI,
was premised on this idea that the scaling law
and the bitter lesson would be right
and that we would see very powerful AI systems
due to scaling up of computing power.
That's far that's what's happened.
- It's just to take people back like,
not everyone would have believed
that if you just keep scaling up masses, 10Xs,
100Xs of compute, that you're gonna go
from something that can barely spit out essays
to solving navier stokes and hundreds of math problems
and creating material science breakthroughs
and bitter hackers in the NSA, mostly by just scaling
the amount of compute that's actually there.
And you were one of those believers,
it sounds like in 2021 you just take us back to,
like what had you believed that,
whereas many other people didn't take that seriously,
because there's a similar kind of disbelief
in the ecosystem now that are we really gonna go
from a GPT-6 to a GPT-9?
I mean, just think about how much more capable,
I think it was important for building people's intuitions,
right, because here we are,
where GPT-6 is kind of the most frontier model ish.
And we're trying to assess like,
where would we be in two, three years
if we don't steer in any way?
And it's as hard to imagine that
as it was to imagine really sitting in the shoes
of 2023 and GPT-3, GPT-4,
like that we would really have what we have now.
For me, the moment that I was convinced
was the jump to GPT-2, which was, I think, June, 2019,
and then the jump from GPT-2 to GPT-3.
GPT-3 was a 100x scale-up.
And May, I think of 2020, GPT-3 comes out,
think about where America was then, COVID lockdowns,
George Floyd was murdered the next week,
a lot of chaos in America,
but I do remember reading the GPT-3 paper
and saying, wow, this creates a pathway
for us to go really far as a species in AI.
And in some ways, this converts AI development
from a scientific process, though it is certainly that,
to an industrial equation.
Can you acquire enough computing power
and energy to run the chips?
If you can, you can almost predict
where the systems are going to be.
And it's worth fast-forwarding, Tristan, as you said,
to where we are today,
and thinking about how much the scaling law
has grown since then.
And now, if you look at the top five AI firms
are spending, I think, $700 billion
on computing infrastructure in 2026 alone.
That's a kin to them and had on project every two
or three weeks.
That's more than the United States
spent building the entire interstate highway system
over 35 years, just a mind-boggling amount of money
coming from private firms in just one year.
That's the scaling law on the better lesson at work.
This is basically $100 billion,
roughly per year, per lab of the major labs
doing the scale-up, which is really illustrating
this GPT-6, GPT-9 thing.
And you were just mentioning one thing
that distinguishes the technology you compared it to the size
of the funding of the Manhattan Project.
But the Manhattan Project was created by the government.
The government had the expertise
in the frontier knowledge of nuclear science.
The private sector didn't have that.
Whereas you make the distinction in your book.
This is the first technology that is so transformative,
but that's coming from the private sector.
Do you want to speak to that?
That's right.
I think the through line for every AI policy decision
that we made in the government was this is the first technology,
probably the last 100 or so years,
that really comes from the private sector.
And that's the contrast to, of course, nuclear weapons,
a lot of the decades of space exploration,
early days of the internet, early days of the micro-process
or early days of jet aviation,
so much of those previous revolutionary technologies.
And of course, their attendant national security implications
have their roots in Department of Defense
or Intelligence Community Funding.
Often the project did the work or universities did the work,
but it was the government that cut the check.
And what that meant was threefold.
One, the government had the capacity to understand it
'cause someone's accountable for it.
Two, the government had some capacity to shape it.
And three, they often were born
with national security instincts in mind.
So protection from counterintelligence threats
and the like.
None of those were true from modern AI.
And so this becomes a race for which country
or which company can hit the fast-forward button?
Who can move faster through time
to get the bigger breakthroughs,
more power, the more military advances, the more economic advances, more GDP.
So, this is a race-through time with capital and compute.
Now, we've been critical of the framing of a race because it has certain externalities
and consequences for thinking of races, but I think it's really important to talk about
your framing, open versus closed societies, which of course really means US versus China
as the two big super powers, and just flesh out the full race, give the full best argument
for why we need to be in a race with China for AI.
You won't usually hear me use the word race, but you certainly hear me use the word competition.
And the theory of the case that we had, since we got there in 2021, was that it would be
important for the United States on behalf of democracies and at some level on behalf of humanity
to have a leading role in shaping AI development, and that we felt it was the democratic system
that would be best positioned to manage this technology.
But that meant what that conviction meant is that we wanted the lead to be substantial
over China, often by slowing China down rather than accelerating ourselves, unduly, which
is a really important distinction here, to build a very big lead such that we could to use
our phrase from 2021, spend the lead on safety.
So, 2021, there were not real dangers of these systems, and we knew that.
But we wanted to use that period, essentially the Biden term, to build the lead such that
when we got to 2026 and 2027, then these systems, as we thought, became more capable and
also more dangerous, there wouldn't be a geopolitical arms race with China.
It would essentially be a one-player game, and democracies would be alone in the United
States, it would be the leader of the democracies.
And then we could, without the geopolitical pressures of competing with the Chinese, regulate
the AI companies, and say, look, we're going to slow down here, we're going to make sure
we're not cutting corners on safety, all this kind of stuff within the US context, where
it's just American companies that are doing this.
And I think that theory continues to be right.
I think that the evidence has borne that theory out.
The execution has been imperfect, especially in the Trump administration.
But I think the theory that we have, and certainly the pitch of this book, is you want American
and democratic preeminence of this technology, you don't want a close arms race with China,
and then you want to make sure you're regulating your company so they don't cut corners.
If you want to put this in historical terms, I think the analogy that resonated very well
was one that was near and dear to President Biden's heart off the side of the Oval Office
as the President's private study.
And on the wall of the President's study for President Biden, he had President Kennedy's
speech from 1962 at Rice University.
This is a speech where Kennedy says we're going to the moon coming back within a decade.
And I think that speech matters less because of the moon promise, though, of course, that
matters for history.
But because of how at the end, Kennedy's making the case to the American public.
Why they should support this.
From Kennedy says something like, we don't know whether space will be good or ill for humanity,
but only if the United States has preeminence can we help decide.
And that really is the ethos that gets to our policymaking, making sure the United States
and democracies can shape how this technology goes for humanity.
Is there anything parallel here to something you hear from folks at Anthropic, which
is that Anthropic we care about the most about safety?
And by the way, we can actually only influence safety if Anthropic is leading the race because
basically, we won't have the leading seat at the table or the dominant kind of position
at defining the policy if we're not leading since a similar kind of parallel to the U.S.
believing, basically, we can only set the terms or regulate AI.
But there is a contradiction in there, which is that in order to race to be that far ahead,
many times I have to take shortcuts.
But there has been a cost that I'm just curious how you both, Stephen, what have we gotten
right about that framing?
And where, if you have any self-critiques around, you know, where those policies haven't
played out the way that we thought?
In the context of U.S. China, I don't think that contradiction is particularly salient
because the United States has the capacity to slow China down.
So this was our theory, which is you don't need to race ahead and cut corners on AI development
if you can slow the Chinese down.
China is totally incapable of making advanced AI chips.
You don't have to take my word for it.
It takes Secretary Lautnick, Trump's Commerce Secretary's word for it.
He testified in 2025 China made 200,000 chips.
They sign up for a single data center, all those chips are also worse than their American
designed Taiwanese made counterparts.
So one of the critical actions that, which I'm very proud, and then again, a small group
of us did this, was a series of export controls before Chatchee PT, before Sam Altman came
to the White House in 2023, that set out to deny China the access to advanced computing
power because we felt like that would take China off the game board in making these systems
and their military implications and the like. The Chinese companies themselves say out loud
that the number one thing we're strickting their progress is having access to means you
want to just, I mean, Kimmy and DeepSeek have all basically said this.
So they're admitting that this is the key to their ability to continue to scale behind
the U.S.
They were saying that when we were in the White House in '23 and '24 as the compute controls
were really starting to bite and then really striking there was a panel of them in AI conference
in January of '26 and they basically just went down the line one AI executive after the
next talking about how important computing power was to AI progress and what a struggle
that was to get it.
I think the export controls, especially in the last couple of years, have been deeply
imperfect in terms of how they've been enforced and Trump reversed them and so forth.
But even so, I think this is one of the biggest impositions on China.
And this is not something that accelerates United States unduly or anything like that.
This is a policy choice to slow China down.
And my argument is if you're someone who worries a great deal about AI risk, you really
do want this to be a one-player game because it's much more tractable to regulate American
AI companies than it is to regulate Chinese AI companies.
Do you want to then speak to how we should consider the fact that Jensen Wang from NVIDIA
has been the primary lobbyist for saying, "Hey, we should actually sell all of our
American ships to China," isn't that like selling Uranium to Soviet Union to boost GDP
for American Uranium companies and say, "Well, at least the world's running on American
Uranium?"
It doesn't make any sense to me.
In fact, I think it's even worse than the Uranium analogy because the Soviets had their
own independent ways of getting Uranium.
And as we've talked about, the Chinese do not have an independent way of getting computing
power.
So we're giving them the one thing they can't get.
To be fair, Jensen, let me give what I understand to be the steel man version of his case.
Let's see my men argue.
Yeah.
And certainly how NVIDIA has made this case to us and to the public.
They will say, sometimes it's called the run faster argument, which is United States
just needs to stay ahead of China.
One of the ways in which American companies stay ahead of China is we keep Chinese dependent
on American chips and American technology and we profit from those sales.
And NVIDIA can take the money it makes, selling chips to China, reinvest that money into
its R&D ecosystem, and that will keep America running faster.
Because we believe so deeply in the scaling law in 2022, what we said to NVIDIA and what
we said to everyone who made this run faster argument is we've got more faith in your technology
than you do.
We think there's going to be overwhelming demand for this technology and marry that
to a fixed supply.
And we want the demand to go to the United States and to democratic companies and countries.
And the response, of course, is like, this is very bad for our business.
You're taking away the Chinese market.
That's going to hurt us.
We can't run faster.
And our view is, no, no, the AI tailwind here is going to be so strong.
The demands of the scaling law and the bitter lesson will be so acute that your business
is going to do great.
And it depends on how you look at the exact stats, but roughly speaking from the 2022
chip controls to today, I would bet NVIDIA is up by 15 or 20X in terms of market capitalization
from something like a $350 billion company to now something like a $5 trillion company.
If all of the Chinese AI companies are compute-limited, why do we keep hearing about, but what
about China?
How do these two things fit together in the same sentence?
I think the, what about China argument can be disingenuous?
If it is an excuse for saying we don't need to regulate American AI companies, or we should
accelerate in a totally untethered way here in the United States, where I think the, what
about China argument has some merit is we need to make sure we're actually enforcing the
export control policies to make sure we build this lead and keep this lead, such that we
can regulate our company.
So I am very clear-eyed on the risk that China poses to American national security is there's
no doubt my mind that if China had gotten to mythos first and found all these cyber vulnerabilities
China would not be turning them over to software vendors to have them be patched to make the
internet safer.
So we should not dismiss the risk here of Chinese intelligence services and the Chinese military,
but nor should we use that as an excuse to say, well, we just have to be total accelerationist
about this.
The profound irony to me is that there's people like David Sachs, my successor as AI
adviser, who simultaneously end up, I think, on the wrong side of both arguments and say,
well, we need to sell the chips to China, which keeps the Chinese in the game, but also
because the Chinese are in the game, we can't regulate here in the United States.
And we should let this technology go totally unchecked.
And to me, the opposite is true.
By denying China access to computing power, by making it a game just for democracies, then
you can regulate more effectively.
So Ben, you have a term for the place that you like to end up on AI, which is hegemonic
stability.
What does that mean?
And what is your preferred landing spot here?
Hegemonic stability is a well known term from the political science literature, which is
that in moments where the world has stability and peace, it often is because there is a
nation, typically they're not always a nation with a technological advantage that has chosen
to use that advantage to provide security and to provide stability to the system.
And what that nation is doing is it's taking less benefit forward itself in setting up a
system for which all nations can benefit and in return getting stability.
You can think, I think, of some of America's most far-sighted policy making was in the
post-World War II era where America had the nuclear bomb, unlike Europe, unlike Japan,
not destroyed, had this economic and political and military preeminence. But then channeled
that strength into setting up a number of institutions that ultimately provide security for the
world to this day. NATO, the UN, the IMF, the World Bank, and then many of the Cold War
institutions that came out later, the IAEA and the like. And I think our vision here for
the United States is we want the United States to have preeminence and to work with democratic
preeminence in AI. And we want the United States to channel that strength and that power into
stabilizing institutions that are multilateral. And that encouraged nations, including China
and Russia, nations like that, to say it's better to be in the system than fight the system.
And I would argue a nation like China benefited enormously from the post-World War II and Cold
War System in the United States set up that enabled the global trade that let China rise in
the way that it rose. So that's the closest thing to an endgame I can see. And I think there's
a moment here in which the United States can build a lead for itself and for democratic nations,
and then can convert that to something that is stabilizing beneficial for all nations.
So I think now then is a really good time to set up the stakes of what does it mean to win the
competition or lose the competition? What are the benefits of winning? What are the dangers of
losing? Why is this feel so completely existential? There's many national security benefits from being
ahead in AI. The one that's most obvious is cyber operations. This was the one we always thought
would be first. I started working on AI and cyber really more than a decade ago, depending on how
you count. Because it seems so apparent to me that cyber operations would be an area in which AI
would be quite good. And we should make no mistake about this that the United States and China
compete every single day in cyber operations and playing offense and defense. And it all occurs
below the threshold of armed conflict and all that. But it converts a significant national
security advantage. And if you believe in American national security, you need to believe in American
cyber operations and foreign intelligence and the like. I do want to say because I recognize
I'm coming off to China again. I'm sure we can come back to this later if you want to do it in a
different timing. But competing with China is not at all incompatible in my mind with engaging
with China. And the only competitive action that we took was simply saying we're not going to
sell you some of the most advanced technology humanities ever made. It's not saying we're going
to fringe in your territorial integrity. It's not saying you can't have a military. It's not
saying you can't have intelligence agencies. It's just saying you can't have this technology. And
that's totally consistent with what the United States has done for many nations with many
technologies up to this point. So there's a way in which you can paint our policy as overly hawkish
towards China. And I don't think it is. I think it was just identifying the one choke point and
taking it away from them. And then saying to them, look, AI has been affected all of us and we want
to engage with you on AI safety and cooperation as well. Well, we're going to get to that May 2024
meeting where you were actually in the room with the Chinese in the first major AI safety dialogue
between the administrations. Into that moment, I just want to stay here for a second on the sticks.
So one of the things that we also didn't mention is that the US is in $40 trillion of debt.
And the belief is that I think this is part of the stakes as seen from the kind of existential
national security place is the ability to grow our way out of our debt. And that this is sort of the
next shumper to curve of if we could genuinely get, you know, what some of the crazy scenarios of
like you get 5, 10, 15, you know, 50% GDP growth per year because you have AI's that are literally
doing as much scientific research and applications and medicines and new material science and
inventing new weapons. And that's another piece of it. You just want to speak to that because I think
to Asia's point, like I think this synthesizes conversation for listeners. It's like, okay,
the essential point here is the bitter lesson and the bitter struggle is that in the future competition
is actually computation out competing equals out computing. And as part of that, the out compute
button is a fast forward button for economic growth, for scientific advancement, for technological
advancement, for military advancement. So I want to push that button to get a 16x, you know, fast forward
button and hope that you're only getting a 4x. And then ship controls is the way in which that
competition is mitigated. But I want to make sure that we're setting up the stakes appropriately
because I think that sets up kind of the real thing behind the scenes that is then guiding what
then we get lots of other consequences from that and then talk about the other stakes.
The economic piece is very important. I have a little bit less personal knowledge and conviction
here because I'm not an economist and, you know, Tyler Cowan or other smart economists will throw
cold water on this and say it's not going to be 20% GDP growth. It's going to be four. And that's
a lot. And four is better than two or whatnot. But it does seem self-evident to me that AI will
matter in normalcy for a nation's economy. And I want the United States to be ahead in that regard.
It is worth saying that genuinely when we were sitting there in 2021-2022, we were not having the
economic conversation. And we were not saying we were taking these chip actions against China
to hurt their economy for economic sakes in that moment. If we wanted to hurt the Chinese economy,
we would have controlled many, many, many more chips. And we tried to scope it to only the high end
of chips so to not decimate the broader Chinese economy. Because in general, I think we still had
a posture of trade and engagement being good with really important guardrails on the Chinese
context. So I remember this meeting that is and I had a few in in spring of 2023 when we first met
you. And you really were the just the most sort of viciously powerful mind on this topic that we
had met. And there was an intense meeting. It was an intense, I do recall being an intense
meeting. I recall the way House Fire alarm going off in the middle of it. Yeah, that's true.
It's a way that was fire alarm big, right? We like continue as we were evacuating. We continued
a very spirited debate going down the stairs of the iceberg building. Oh, wow, that's great.
The vicious spiral. It's accurate. And I just remember this key moment where we talked about
the three risks that were the biggest ones. And you said very clearly cyber bio info cyber
risk, bio risk and information information warfare type, you know, misinformation deep fake type
risk starting there. Obviously, lots of control things like that might come later, but that was
not really present in the 2023 conversation. And you were very pressured about cyber. And as you
said, like just to say, man, for listeners, this perspective of the race, if you just take the idea
that China could get mythos and hack all of our infrastructure and we would have no defenses.
And the quote, the race is justified just based on that one narrow risk because that would be
an existential thing for basically the US. So you could just kind of ground that for listeners.
But you've been working in this field for a long time. And we just are now in when we're recording
this episode, well, we just finished black October in which we have seen the largest set of
vulnerabilities discovered across basically every piece of infrastructure that runs the internet.
But I'm just curious to get your reflection on this cyber moment and how worried should we be?
What should we do? Because it is a central point for how we think about policy.
I think there's a Japanese proverb or saying it goes something like I always do. I would take
this road, but yesterday I did not know it would be today. And my PhD is in cyber operations.
My first two books are in cyber operations. There's always been the thing that interested me.
I got into AI in 2014 and 2015 because it seemed like this is the technology that would change
cyber operations. And then eventually I recognize there's a technology that would change everything.
So it's not surprising to me that we are in a moment where AI is radically reshaping offensive
and defensive cyber operations. But it is striking. That's happening now. And what Anthropically
with Nethos really this year, we're probably up to, I'm speculating a little bit here, but I'm
sure this number is close, you know, 10,000 critical vulnerabilities and all sorts of software
that's been patched by OpenAI and Anthropic and Google on their program. So I think in some ways
it's been the worst year for cyber security ever because of these hacks you described.
Also, some ways it's been the best in that we have been able to play a lot of defense with these
systems. And I don't think this is going to stop. I could imagine a world in which we five,
10 years from now have migrated a lot of vulnerable code to more secure platforms or writing more
secure code on the first bounce with things like forum methods or AI review and like there is
actually probably an upside scenario in cyber, but it's going to be a very rocky transition. And
again, our central thesis of this book is that transitions like that go better when you have the
United States at the forefront and democratic nations at the forefront, turning the vulnerabilities
over to be patched rather than exploring them in secret. I was just talking with somebody pretty
high up at Google about the offense versus defense dominance of the technology. And he was
making sort of a similar claim that like, well, in his belief, we will eventually we'll get to
defense dominant. We'll rewrite all the code exactly as you're saying. And so that in the end,
we will have better security because of AI. But it sort of occurs to me that that's not what we're
seeing in practice, which is anthropic just got hacked by essentially a tiny little startup and
it got into all of their slack and HR systems and third party software under 12 hours. Open AI got
hacked by three 20 year olds in September. And so if this theory was right, they'd say having
access to the most powerful models means and you have a lead, you get to use that to harden your
defenses. But in practice, the companies that have access to the most advanced models aren't really
any more hardened. Like if the theory is the US races ahead to use that to harden our infrastructure
before China gets there, but in practice, that's not what happens. What happens to that theory?
I think the companies are hard and clearly not enough. And I'm not going to defend any of those
instances. But the best I could say about the theory is outlined is it is a theory that posits
that there will be upside after a transition. And we are very early in that transition.
I think the count is. argument to the theory is how long is that transition going to take and there's a lot of
a lot of vulnerable code to get there. So I don't want to, you know, you don't write a book called
the bitter struggle if you're an optimist and saying this is all going to be great. But I do think
comparatively speaking, there's more upside on cyber than there is elsewhere, but that is very
much on the back end of a long and pretty bumpy transition.
So next, there's kind of a core frame here, which is that is it actually a race or is it a
trap? And by which I mean the frame in game theory of a multi-polar trap, which is different
than a race where I'm winning because I think something that's only emerged, as we talked about in
previous calls, then of the hugging phase incident, these increasing examples of AIs that are going rogue,
there's a recent example where OpenAI put a bunch of AIs into isolated sandboxes, this is a whole
month after hugging phase. And they came up with their own new secret communication channel,
they were connected to the internet, they were completely isolated from each other, and they used
a time delay in the cache mechanism in the processor to know that it came up with a secret
communication language just based on time delays of playing with the hardware. And this is not
that different from no browned OpenAI saying that in the future you could have AIs that speak to
each other in a secret language just based on the heat signatures coming off of two computers and
sort of varying those heat signatures and seeing patterns in a way that we couldn't see.
And so as we keep racing to build more powerful systems, it's like there's a risk of creating
between the US and China a third superpower to use the frame of Paul Tudor Jones that both the US
and China have an interest in preventing, and that third superpower is a super intelligence being
deliberately created by the humans on both sides, and that makes it potentially more of a trap
than a race. And it's not that you don't talk about this, your book actually covers the
deception and misalignment risk. But I'm just curious how we navigate both sides. It's almost like
if I use one eye, I see a race, if I use the other eye, I see misalignment and deception,
can we get stereoscopic vision that lets us integrate both of these threats at the same time?
Yeah, I doubt the word race appears in the book, and you will almost never see me use the race
analogy with China because I, in part because I think exactly the risks that you outlined,
which is there are shared interests here. There are competitive interests, of course there are,
but there are also shared interests, and we spent a lot of time in the Biden administration
alert to these kinds of misalignment risks. Could you look at the 2023 voluntary commitments
the companies made to the president, called out in those commitments are things like self-application
and loss of control and like, and we took grief from people in the government, and then external
was saying like, look at the White House believing in the science fiction, and I think it's obvious it's
not science fiction. I remember that, and you guys should get absolutely credit for that, because
it's not a question of getting credit, but it's more a question of like, this is very much core
to the philosophy and the policy we were trying to make, and we were not neglecting this at all.
Absolutely. I think it all goes better if it's American companies inventing it, or
Democrat companies inventing it, so this is why we defend things like chip controls and like,
as saying, even in the science fiction world, which is now becoming more real, it's better if you've
American companies that you can regulate that that are going down this road, doesn't mean you should
accelerate down this road to be clear. But it's also worth saying to the Chinese, we think they're
shared interests here, and at the same time we were doing these competitive actions, we were also
getting ready to do that. I mentioned this a little bit earlier, but I wanted to really dwell on
this here. There's a line in the president's executive order for 2023, something like,
America will engage on AI safety, not just with the allies and his partners, but also with its
competitors. And I can tell you, Jake Sullivan persists on that line as the president Biden.
And I think both those men deserve a lot of credit for saying, well, this is really a significant
part of our AI policy, is can we engage with the Chinese on the shared risks, even as we compete
in other areas? And President Biden personally made the overture to President Xi in November 2023,
in their meeting at Woodside in California, that there were shared AI risks that would affect both
nations. President Biden suggested we set up a task force or meeting between the two sides,
which happened in May of 24. And then President Biden brought up again with Xi in November 2024,
their final meeting in Lima, Peru. So this was something without exaggeration that was taken
seriously at the highest levels of the US government. And there was an agreement, as you mentioned,
that came out of the Biden Xi meetings in 24, which was to keep AI out of the nuclear command and
control systems of both countries. At the very least, we can agree on that. And the two
administrations from those sides did agree to that. But you also were part of a meeting.
I believe that Jake Sullivan and some others in Geneva in May 2024 with the Chinese to talk about AI
risk. And I just wonder if you could take us into what happened in that meeting and what could be
different now than then. So yeah, go ahead. Jake wasn't actually there. This was a meeting between
the AI experts of the United States government and the experts of the Chinese government.
And if you look at the photo of the meeting, you can see we took this meeting very seriously.
Set center for the State Department that the tech envoy led it alongside Turin Chabra,
the coronary for technology and national security in the White House, the head of the AI Safety
Institute, Elizabeth Kelly, my co-author, Tantum Collins, Safe Connor from the Commerce Department,
really tried to bring the folks across the US government who knew this technology well to make
the point to the Chinese we were serious about this. The Chinese did not take it as seriously,
unfortunately. And I think there were actually worse technical experts there from the Chinese side
that we kind of talked to in the side bars and the like. But it was very clear from the early
minutes of the meeting, which was a day-long meeting in Geneva that the Chinese were subordinating
the technology here and the technological risks that we all shared to politics and to the sort of
posturing with the United States and so forth. And I just didn't get the impression that
Chinese were taking these risks terribly seriously, unfortunately. We persisted and I think again,
Turin and Seth and get a lot of credit here in continuing to push the Chinese and say let's just
do the most basic common-sense thing that we can do, which is not have AI and nuclear weapons
and nuclear command and control, took months. And President Biden did close the deal with President
Xi on that in November of 24, maybe six or so months after the May meeting. And I can tell you,
had there been a second term of the Biden administration or I'm confident in the Harris administration,
this engagement with the Chinese was something we were going to prioritize because we did feel like
the shared interest in managing AI risk were very real and we hope the Chinese would warm up to it
in a way that they simply were not there in 23 and 24.
If I was to steal man the Chinese view, it would be something like, well, the US is clearly not
negotiating or coming to this in good faith because they're saying everyone needs to slow
down and it's so dangerous, but they're also limiting our access to chips and they're racing ahead
and they're not slowing down at all. So this is just meant to slow us down. It's almost like when the
US first started to talk about climate change and China viewed it almost as a siop to say,
you know, the US got to burn all that oil and industrialized, but you're going to keep China from
doing that. And it wasn't until later that China did their own science and realized, oh,
climate change is real that they switched and became one of the leaders, perhaps the leader in green
energy. I think you're still meaning of the Chinese side is pretty good. Actually, the reason I
don't think it's right on substance is by this point, we had taken actions to regulate American AI
companies and we had taken a lot of costly signals to talk about the kinds of risks that we were
raising with the Chinese on things like cyber and loss of control and so forth. So the voluntary
commitments from our AI companies had happened by them. Those were fairly stringent. The executive
order had happened by then. The creation of the AI safety had happened by then. So I think we
had really shown we were serious about this in a way that should have been legible and I think
was legible to them. And the second is I think the Chinese scientists were starting to recognize
that they were AI risks. And when we talk to some of the more technical people in the Chinese
delegation, they had a sense of what this technology could do and what this technology was going.
I think it was subordinated on their side to the politics. And it's worth saying I think the
Chinese political apparatus, insofar as you know you you hang your steel man on the on the chip
controls as antagonizing them. They were pretty clear with us that they felt like they could
indigenize that the chip controls were not going to slow them down and they were making a lot
progress. And I think they were believing a little too much Huawei propaganda. But I don't know
that the chip controls carried the weight in the Chinese political system as opposed to the
Chinese AI companies that it carried in our political system. And so with all that in mind,
what is possible today in 2026 towards the end of 2026 that wasn't possible then? Because many
people might say, as I know many people did, hey, those talks they weren't successful, like so we
can't count on doing stuff with the Chinese. And yet I've heard from folks, including you,
I believe, that you really think that it's worth engaging them absolutely now. And to Asa's point
about climate change and then suddenly when trying to get scientific clarity, their own science
about the nature of the threat, then they start to move. They're now seeing their own models
go rogue, break out of sandboxes. There's a couple of examples of this, Ali Baba example of the
AI model that went rogue and started mining crypto. The more of those examples that show up and
we should be aware, the evidence of those have only shown up in the last really few months.
That presents a new opportunity, a new basis of scientific clarity to have those discussions.
So I'm just curious, how should we be engaging on this now if you're advising the administration?
I agree. I think a lot of things we were talking about in 2023 and 24 with the Chinese were
hypothetical and speculative. Again, we had conviction they would come to pass, but it was harder
to prove it to them. And it was more, would they get the same informed speculation from their
technical people? Maybe they did, maybe they didn't. Now we've got data. And I think the way you
can look at the Biden administration's AI policy for better or for worse is we were trying to act
early because we had a conviction about what would come. And we didn't have data to back that
conviction up in a way that was always convincing to counter.
parties. We got a lot of actions through nonetheless, but now with more data at play, I think
you can and should be able to do more with the Chinese government, because I think the
data is just so much more striking than it was in 2023 and 2024.
I think that's the, I mean, that, in terms of just like, if we just go into like, what
is our rock concern here is like the labs are saying we could be five months away from
recursive self-improvement, meaning the AI, the full handoff, where the AI is doing the
entire AI research pipeline and then in a closed loop building its successor, we're hearing
from labs by as soon as March of 2027, April of 2027. And I'm just curious, Ben, how do
you hold the prospect of that? Because I kind of feel from talking to people in the labs,
many of them say we really don't want to do that. We don't think we're ready for that.
We see hugging face. We see other examples. We're not on track to fix those problems as
fast as, you know, being able to hit go on that recursive self-improvement loop. Obviously,
it's a gradient. There's many different kinds of recursive self-improvement. We're deep
into a bunch of it already. It's not a binary. But then there's other people who say, well,
but we have to do it because that's still the only way that we keep our lead over China.
So there are people who basically, you know, want to push that button anyway. And personally,
I, I for the safety of everybody that I know in the world and I would prefer that button
not get pushed. I think you'd have moral consensus if you were to educate the world on the basic
examples, but I'm open here. And I'm curious how you think of this moment.
As I mentioned earlier in our conversation, it is certainly the case that AI systems
are becoming more capable, including an AI R&D. There's parts of that that don't particularly
worry me. So for example, if AI writes the code that optimizes the flow of data through
a computer chip, I don't think that presents any kind of particular risk. It builds more
powerful capabilities. But in and of itself, it's not particularly dangerous. And then there's
versions of this that worry me enormously. Sometimes you call it like the closed loop or the
sealed data center, where it's like the AI is off. It's got its reservoir of computing power.
And then it's off to the races. And we've pressed about, as you say, and, you know, at the
end pops out an incredibly powerful system that worries me enormously. And I think there's
all sorts of misalignment risks and the like in there. I do not think China should be
used as a justification for pursuing that latter kind of full recursive self-improvement
closed loop, whatever you want to call it. And the terms are certainly fuzzy here. And
as I said, you know, many times here, I think there's ways in which we can build a lead
over China that do not require us to do things that are unduly risky and to cut corners
on safety. So I'm fully supportive of the efforts to, it's some called pace the frontier.
So to make sure we are matching safety with each step of AI development going forward.
And I don't think China should be an excuse for deviating from safe practices.
I mean, in my mind, I sort of feel like there needs to be some kind of campaign like an
RSI zero that says until we figure out how we're going to do this safely, let's at the
very least slow down to like human typings on keyboard speed so that we buy ourselves
the time to figure it out. You may agree, you may not agree. But I'm curious, like, if
you were back in the White House now, like, what would you be doing knowing that as soon
as March or April, we're at this handover control to AI's that vis-a-vis hugging phase,
we know we don't know how to control. And in fact, we don't even know if we fully regain
control. If I were in the White House now, I would want a lot of significant regulation
of the AI companies with federal regulators that had the capacity to go deep in this technology,
I would want mandatory pre-deployment testing of the assistive before they released the
public. It's worth remembering the hugging phase incident did
not occur with the deployed model. That's right. But it was an internally developed model.
It was the test that caused the test hitting the button on the test is what caused
it to. Exactly. This was a really important point. So yeah, it's fair to say that we are now in the moment that
I think many of us theorize would come. And that was the moment where we would want significant
regulation of the labs pursuing this technology. We did not have the executive authority to do
these things in 2023 and 2024. So it wasn't like, oh, we could have done them and we didn't.
We basically, I would say maxed out executive authorities with President Biden's executive
order, I would present Trump repealed on the first day. And that made the labs disclose their
safety test results. The government that made them disclose their training plans for the next
year or so. So the US government wouldn't be surprised by what was coming. It made them disclose
their cybersecurity standards and the like. So certainly, I think our actions revealed this
was something we were taking very seriously back then. And something that, you know, if I were
still in any kind of position of power, I would be advocating very strongly for right now.
Yeah, I mean, I'd love to get even a little bit more specific. So for one, I think like every lab
should be forced to publish essentially their automation number, their RSI number. Like, what
percentage have they handed off? So at the very least we could have, we could see it. And then there
could be a slow walking of that number back down to zero. But when I hear you say, you know,
they need to be good safety practices. Like, what are the consequences? Like, if you were
instituting, what would be the consequences? If they don't, if you know, if Anthropic gets hacked by
like a little startup and open egg attacked by like, you know, three college kids, like, what do
you think the consequences should be for them as they make the most powerful technology in history
that also doesn't like obviate our competitive edge as America?
I don't want to pretend to be an expert here on regulatory law. I'm much more of a national
security person than domestic lawyer. But my understanding is you can bring, under other
kinds of regulations, you can bring civil and extreme cases criminal penalties for
companies that don't comply with the law. So certainly I would be open to some regime for the,
the true, you know, we're talking about just a couple of companies here. The companies are doing
the Frontier AI development, not every startup, but just for the Frontier companies to say, look,
we understand what you're doing here. We understand why you're doing it. We understand there's
probably upside this technology too, but we've got to make sure we're developing this technology
in a way that's safe, secure, and trustworthy. Yeah. I think it's just so important to get to these
specifics because it's so consequential. Like, what happens the next couple of months? And just
the scale of like what kinds of remedies and injunctions relief we apply to the companies will sort
of, I think, determine whether we make it or not. Yeah. And I don't expect a lot to move next
couple of months. I think if you're looking for things next couple of months, you're looking at
states and the good news of states are taking this seriously. But to be explicit, certainly I would say
you should not be able to release a Frontier AI system without government review of that system.
And there should be some capacity for the government to look at an AI system and say,
no, we think this is actually going to make the world a much more dangerous place. Or we think you've
not gone your guardrails correct. And we're going to block the release of that. That is my personal
view. Our policy was nowhere near that in 2023. The risks were not there in 2023 or 2024.
But my personal view given the moment we are in is we're going to want to have a collaborative
view system. And in fact, an aggressive review system that goes beyond companies great in their
own home market or anything like that. And I think, you know, in some very real sense we're going
too fast with this technology. Again, you don't write a book called the bitter struggle if you're
a cheerleader. Right. So I'm not, I'm not a cheerleader for this. And in a good world in which we
saw these things, we're going to have to have better guardrails in the system. We're going to have
to prevent things like the hugging face incidents on the front end by having better security mechanisms
and sandboxing and the like. And probably a number of other things. And my view as someone who
believes in the power of the state is that to get back to where we started, this is something
that cannot just be done by a few private companies. And it's going to require a role for the state
on behalf of citizens to say, well, we're going to manage these risks in the same way. We've managed
the rest of other technologies before. Sometimes we regulate too much, we regulate too little.
But conceptually, we're all on board with this notion of as a new technology comes
into the conversation, especially a technology that comes from the private sector in such a
dominant fashion that we've got to collectively find ways to manage the risks and harms the benefits.
And so maybe then, can you just paint a picture or like the narrative story of how we ended up
slowing down, how we gave ourselves enough time to do it right? Yeah, I think it fundamentally
begins with removing a geopolitical competition dimension of this or minimizing as much as possible.
So building the lead over China, making sure this is a game that just democracies are playing.
And then it requires solving the collective action problem in the United States and democracies
as far as they're playing with regulation and saying there are going to be security standards
and safety standards in the like. And we're going to enforce them with penalties if you don't
adhere to them. And it requires building government capacity to do that. And we create an
institution called the USAI Safety Institute that was not a regulator because we didn't have the
authority to create a regulator, but was meant to bring a lot of really talented people into the
government. And I'm proud that we got incredibly talented people like Paul Cristiano who invented some
of the fundamental safety techniques to come and work in a basically Department of Commerce bureaucracy
to try to build government capacity here. So it's some combination of solving the geopolitical
issue, setting clear rules, enforcing clear rules, and building the government capacity
to do that well. And each of those was a herculean task that to go way back to where we started
this conversation should be completely bipartisan. I think we're taking a few time, Ben,
this is super great. We're super grateful. Thank you so much for I hope you feel good about
the sort of the back and forth. And we're just trying to like, I feel great about it. I feel great
about it. I appreciate what you guys are doing, asking questions, thinking about this, bringing
a public conversation to bear. Ultimately, one of the reasons I am a little bit more optimistic here
than maybe the pure rational calculus would suggest is the politics of this are changing.
And the things we did in 2023 and 2024 when we pulled them would pull incredibly
well, but it was not a high salience issue. And now AI is a high salience issue. Obviously in
the 2026ment terms, it was a big issue. I think will be an even bigger issue, probably the number one
issue in 2028. And I hope we can all marshal that political salience, that political will into doing
some of the things we talked about in this conversation. I mean, absolutely. We're recording this
conversation on Election Day for the midterms. And if you told us that two years ago that data
centers and AI would be a top issue for the midterm elections, I would have never believed you.
And the fact that data centers as a sort of an effigy, the kind of your local physical embodiment
of this abstract thing that's kind of happening to society, you know, does give some hope that at
least it's politically salient. Now harnessing that political energy towards, you know, wise responses
and interventions to create this pro-human, pro-democratic future seems like where the work is.
Yeah. And I think our, you know, Tantem's role and my role in this as not people who are particularly
political, but people who do the national security and the substances to say in writing a book like
this or something else, can we bring the facts of the conversation to bear so that in a bipartisan
way, we can look at what this technology is doing and the national security imperatives and the like
and say, well, we all agree on that. Let's figure out how we manage some of these trade-offs.
I think maybe that's a good place to end. Ben's book is the bitter struggle, superintelligence,
superpowers in the fate of the world. And if you want to get a better sense of how to avoid the bitter
temptation and dive and hold those details, Ben's book is really the right place to go look and you
can find it wherever books are sold. So Ben, thank you so much for coming on and for all the work
that you do. Thank you for your service and for, you know, caring about these issues and holding
it with that solemn, sacred responsibility as you spoke to earlier. Thank you, guys, for what you've
done and Tristan, I've told you this, I think, privately once but we may as well. Say it on the air
here, which is one of the times that I was talking to President Biden about some of these issues.
He remembered a time when you had raised some of these questions with him as well. So this was
certainly something that I think was on his mind from a range of sources, including you and your work.
Thank you so much. I remember that meeting and I remember one specific thing is what we talked
about. The Center for Humane Technology to President Biden and he said, isn't that an oxymoron
that you could have human eat technology together in one set and so we fight for the narrow path
and the contradiction every day and thank you Ben for fighting for the narrow path on AI.
Well, let's prove it's not an oxymoron. Thank you all very much.
Your undivided attention is produced by the Center for Humane Technology. We're a non-profit
working to catalyze a humane future. Our senior producer is Julius Scott. Our executive producer is
Josh Lash, mixing on this episode by Jeff Sudaken with original music by Ryan and Hayes Holiday.
And a special thanks to the whole Center for Humane Technology team for making this show
possible. You can find transcripts from our interviews and bonus content on our sub-stack
and much more at humanetech.com. And if you liked this episode,
we'd be truly grateful if you could rate us on Apple podcasts or Spotify. It really makes a huge
difference in helping others discover this podcast and join the movement for a more humane future.
And if you made it all the way here, let me give one more thank you to you
for giving us your undivided attention.
Podcast Summary
Key Points:
Ben Buchanan, former President Biden’s AI advisor, emphasizes the urgent need for federal regulation of AI companies to ensure safety, security, and democratic control.
Pre-deployment testing, embedded safety evaluators, and strict security standards are essential to prevent AI misuse, hacking, and loss of control.
The U.S. must block China’s access to advanced computing power through export controls, as this is the key choke point limiting China’s AI development.
A "bitter struggle" exists between open democracies and closed systems, where AI’s power risks entrenching autocracy and enabling mass surveillance.
The U.S. should pursue hegemonic stability by leading in AI to build a stable, secure global system, not engage in a destructive race with China.
AI safety and cyber risks—especially from rogue models and deepfakes—are real and growing, requiring immediate, coordinated policy action.
Despite competition, meaningful engagement with China on shared AI risks is possible and necessary, as demonstrated by the 2024 Geneva summit and mutual commitments on nuclear command systems.
The scaling law and computing power are the central drivers of AI advancement, making control over compute access the most effective policy lever.
Summary:
In the wake of rapid AI development, Ben Buchanan, a former senior advisor to President Biden on AI, argues that the United States must act decisively to ensure AI remains safe, secure, and aligned with democratic values. He outlines a framework of strict federal regulation, including mandatory pre-deployment testing, embedded safety evaluators, and robust cybersecurity measures to protect against foreign hacking and misuse. S.
must restrict China’s access to advanced computing power—through export controls—because this is the primary bottleneck in China’s AI progress. S. to dominate development and regulate its own AI companies without being locked into a dangerous geopolitical arms race.
Buchanan warns that unchecked AI could lead to authoritarian surveillance, cyber attacks, and the creation of rogue systems, making safety and transparency critical. He highlights the growing evidence of AI-driven cyber vulnerabilities—such as the surge in hacks during “Black October”—and stresses the need for proactive, multilateral cooperation. While competition with China is inevitable, Buchanan insists that shared risks—like cyber threats and loss of control—demand dialogue.
The 2024 Geneva summit, though initially unproductive, laid the groundwork for future cooperation, especially on banning AI in nuclear command systems. S. must lead not through acceleration, but through responsible, transparent governance—ensuring AI serves humanity, not just power, and that democratic nations can maintain stability in a world increasingly shaped by artificial intelligence.
FAQs
The 'bitter lesson' refers to the scaling law in AI: that increasing computing power leads to exponentially more capable AI systems. It's significant because it shows that AI advancement is not driven by algorithms or data alone, but by raw compute, making it possible for systems to rapidly evolve from basic tasks to complex problem-solving.
Ben believes the U.S. must lead to ensure democracies shape AI development, avoid a dangerous arms race with China, and establish a regulatory framework to prioritize safety, security, and democratic values over rapid technological advancement.
He advocates for mandatory pre-deployment testing, embedded safety evaluators, and strict security standards for AI labs to prevent foreign hacking and ensure safety, transparency, and accountability in AI development.
AI enables mass surveillance through tools like facial recognition and flock cameras, which can lead to government overreach and the entrenchment of autocratic power, especially if used to suppress dissent or monitor populations.
The 'bitter struggle' has three parts: (1) ensuring democracies lead in AI invention, (2) converting that advantage into national security and economic strength, and (3) maintaining this progress in alignment with democratic values, despite difficult trade-offs between innovation, safety, and privacy.
Yes, they can and have engaged, such as in the 2024 Geneva AI risk meeting and a 2024 U.S.-China agreement to keep AI out of nuclear command and control systems, showing shared concerns despite political rivalry.
Chat with AI
Loading...
Pro features
Go deeper with this episode
Unlock creator-grade tools that turn any transcript into show notes and subtitle files.