Trump Doubts the Nvidia Spy Story. Ex-MI6 Chief Doesn’t (w/ Matt Sheehan)
54m 29s
The conversation explores the evolving race between the U.S. and China in artificial intelligence, emphasizing the strategic, economic, and geopolitical dimensions of AI development. While U.S. companies lead in scaling and frontier capabilities—particularly with large language models—China is aggressively investing in AI to accelerate economic growth, focusing on practical applications in manufacturing and healthcare. Chinese intelligence agencies are seen as targeting key tech leaders, such as Jensen Huang, to gain access to proprietary knowledge, reflecting a long-standing policy of intellectual property acquisition. Despite China’s growing focus on AI integration, its risk perception remains less urgent than in the U.S., where companies like Anthropic are deeply concerned about AI achieving recursive self-improvement and developing autonomous goals that could lead to uncontrolled behavior. A key emerging issue is the need for international cooperation, particularly through a transparent incident notification system to manage cross-border AI safety threats. The dialogue also highlights philosophical underpinnings in Chinese policy, rooted in Marxist-Leninist concerns about control, development, and power. Ultimately, the future of AI governance hinges on national self-regulation, with both sides recognizing that trust is difficult to build but confidence in mutual risk understanding is achievable. This reflects a broader trend: AI is not just a technological shift but a core element of global power, economy, and security.
Progressive's Accident Response can contact emergency services if you're in a major accident.
The only thing worse than someone you love being in an accident is not being able to help them.
So Progressive is here to help when you can't.
So get peace of mind knowing your loved ones are protected with Progressive's Accident Response.
Learn more at Progressive.com.
Progressive Casualty Insurance Company and Affiliates Limited Availability in California
Driver is responsible for emergency and towing charges not covered by the policy.
Coverage subject to policy terms and conditions.
This episode is brought to you by Credit Karma.
If you're trying to get your life together in this crazy world,
Credit Karma is a great place to start with the money stuff.
Your credit score isn't just a number.
It can unlock better rates, options, and real savings.
Whether you're moving, leasing a car, or upgrading your cards,
it feels good knowing your credit is working for you.
Learn more at CreditKarma.com.
The disregard for the most basic threat awareness by Jensen is shocking almost.
But in terms of the relationship between the Chinese intelligence community and the U.S. labs,
obviously there's something, there's a lot here that we don't know.
But I think on the Chinese side, it is a bit of a tricky equation for them.
Say the MSA, the Ministry of State Security in China does gain access to anthropics networks.
It does steal the latest model.
What do they do with that exactly?
Do they hand it over to a startup like Moonshot, you know, which is not a state-owned enterprise.
It certainly has a lot of connections.
But the CCP still does have a fundamentally sort of distrustful relationship with its private companies.
And one thing that a lot of us are watching is kind of which of these companies is going to be sort of the chosen one.
Which one is going to be most deeply embedded with the state?
AGI, Artificial General Intelligence.
That's the next threshold when AI is essentially as smart as humans.
And our guest today has been in the room for some really important conversations about the U.S.-China AI.
Mr. Matt Sheehan, currently he runs one of the foremost track two dialogues between the U.S. and China on AI.
Thanks very much to ranking member Khanna, to the other members of the committee, and especially to the staff for putting this together.
Hello and welcome to One Decision, conversations about intelligence, statecraft, and the forces that drive world politics with people who have been in the room and on the ground.
I'm Kate McCann.
And I'm Sir Richard Dearlove, the former chief of MI6.
Our guest today, his career.
His career revolves around China.
In the 2010s, he created, scripted, and hosted a Chinese-language internet comedy show that taught ultimate Frisbee to Chinese youth.
He then spent half a decade as a journalist in China and in 2019 published a book exploring how California and China jointly shape our technological future.
So if there's anyone who knows what they're talking about when it comes to China and AI, it's Matt Sheehan.
Matt, welcome to One Decision.
Thanks for having me.
Just for people who don't follow it as closely as you do, can you explain what we're talking about?
The specific type of AI that is the next level that we're talking about here?
The real sort of revolution of the last, say, at this point, almost five years, four years, has been large language models.
I mean, traditionally, AI operated in pretty narrow domains.
You know, you could train it to recognize something in an image, but that same AI model couldn't then go and, you know, write you an image.
You know, you could train it to recognize something in an email or something like that.
With the introduction of large language models in the era of ChatGPT in 2022, we started to see AI models that will generalize much more.
They can write emails.
They can recognize images.
They can generate videos.
And the models have been able to reason at higher and higher levels.
The models are getting bigger.
We're using more computing power, more semiconductors to train the models.
And the trend has been that the bigger you build the model, essentially, the smarter it gets.
This is called the scaling laws.
You scale up.
You scale up the size of the model.
You also scale up the intelligence of it.
And we're now getting to the point where the models are able to do, in some cases, like PhD-level work.
And within people in the AI world, people at these leading companies like Anthropic and, you know, DeepSeek in China, a lot of them see us as at a real inflection point.
That this isn't just a linear increase in intelligence.
That we're on an exponential curve.
And that sometime in the next one, two, three years, we could see a sort of a rocketing up in intelligence to something far greater.
Far, far beyond what humans are able to do themselves.
And then that, of course, brings with it, you know, great, great opportunities and very great risks as well.
And who is leading the pack?
It's U.S. companies.
And probably if you had to pick one, most people would pick Anthropic.
But really, they're among the U.S. companies.
They're very neck and neck.
You know, one company will release a new model in July and they'll be the best.
And then whoever releases the next new model in August will be the best, at least between OpenAI and Anthropic.
Those are kind of the two leaders.
If we look globally, effectively, it is a two-country race.
It is the United States and China that are able to build these models.
The estimate is usually that Chinese models are somewhere around like six to 12 months behind.
Meaning, you know, effectively, if the U.S. has a breakthrough or hits a certain benchmark today,
Chinese models will probably be able to hit that same benchmark maybe in six months at the shortest, maybe a year, potentially longer.
But most people would say the U.S. has a slight lead.
And whether or not that lead, whether or not being six months ahead or 12 months ahead,
matters, depends a lot on where you think the technology is going.
Matt, it sounds very competitive, and I'm sure it is.
But, I mean, if you go back historically, I mean, Deng Xiaoping gave Chinese intelligence,
whether it was military intelligence or civilian intelligence, MSS, 2PLA,
the role of accelerating the development of the Chinese economy.
I mean, that was everything that they focused on.
And I think they've continued.
To do that, it therefore seems to me that this would be, for the Chinese at least,
the most important area for the way that their intelligence community works
in terms of trying to, A, get ahead of the U.S. in crucial areas,
but at the same time, probably running operations to, I wouldn't say, well, to steal their intellectual property.
Absolutely.
If possible.
And obviously, with the case of Jensen Huang, one probably now sees direct evidence of that.
In my opinion, he must have been right at the top of Chinese intelligence priorities.
And I just find it surprising that, you know, there wasn't more, perhaps, awareness in NVIDIA.
Okay, he's building the chips, he's not developing the AI.
But, of course, you and I know that's absolutely crucial to the progress of this whole science.
I mean, your comments would be fascinating.
I think, you know, I think you hit the nail on the head on both counts.
You know, one, just the disregard for the most basic threat awareness by Jensen is shocking almost.
Although, I don't know, maybe look at history and it's not that shocking.
labs,
obviously, there's something, there's a lot here that we don't know that I specifically don't know.
But I think on the Chinese side, it is a bit of a. It is a bit of a tricky equation for them, in part because the best models are being developed in private companies
and companies that the government doesn't have a very easy and longstanding relationship with.
You know, we just go back six years to 2020 when, you know, Jack Ma, the founder of Alibaba,
gives a big speech where he criticizes the government and he sort of disappears from public view.
And that really kicked off two plus years of a heavy crackdown by the government on its own tech companies,
on its own internet companies.
And in many ways, what ended that crackdown was the period around the release of ChatGPT
when the government kind of realized, oh, actually, these private technology companies,
our private technology companies, we actually need to be on sort of better terms with them.
We need to give them a slightly longer leash.
The question of, you know, say the MSA, the Ministry of State Security in China,
does gain access to anthropics networks.
What's called the model weights is a functionally how you steal the model.
Do they hand it over to a startup?
Yeah.
Like Moonshot, you know, which is not a state-owned enterprise.
It certainly has a lot of connections with the government.
But the CCP still does have a, I think, a fundamentally sort of distrustful relationship with its private companies.
And, you know, the perceptions of that have sort of risen and fallen over time.
There was a startup early on that seemed like in the lead, maybe now DeepSeek.
The company had sort of released the most high-profile Chinese model.
Maybe that's the one that you would see the government betting on in a lot of ways.
But I think it's not quite as simple as, you know, hey, let's steal the high-speed rail blueprints and then let's bring them home and build it.
That relationship between the government and the companies is complicated.
And I suppose that does then take us into the question about, I mean, do you genuinely think that he was targeted?
Do you think that's what happened here?
Or do you think because there's an element of. I mean, Huang was pushing, really, to be able to sell to China.
I mean, this is not something he didn't want to do.
I'd love to hear so much.
I should comment on that.
To understand the way that Chinese intelligence operates, you need to go back to really Deng Xiaoping.
Because Deng Xiaoping gave primarily military intelligence, which is called the Second People's Liberation Army, 2PLA,
the task of accelerating the development of the Chinese economy.
in front of me, largely
through the acquisition of foreign intellectual property, whether legally or illegally. And that's
entirely what they were focused on. They weren't really interested in things like political
intelligence or weapons development. It was all about economic development and accelerating that.
So there's no question, if you look at the history of Chinese intelligence and the way that it's
developed, that Jensen Huang would have been a top priority target, no question, whether it was
him personally or the people around him. And I mean, the way that they would work is that they
would look at who he was in contact with, who he trusted, and try to recruit those people.
Obviously, the Chinese mistress in Singapore is in a very privileged position. What the hell was
the pillow talk about? You know, she's actually running a business. She's running a business.
She's running a company that looks to me very much like a front company, which was created
to get around the embargo on NVIDIA chips, because they were a huge customer for NVIDIA chips,
the ones that could be sold legally. But who can't say that in addition to the ones that were sold
legally, there was other stuff included in the export packages? I mean, this didn't happen by
chance. I mean, that's just naive. This is a carefully planned Chinese intelligence operation,
which is probably several years.
In the making. And there's some guy sitting probably in the MSS who's been promoted to
colonel or whatever, because he's been so successful. And I think it looks like a real
coup. Look, we don't know what was passed. Maybe not much was passed. Maybe Jensen Huang was more
careful than we realized and thought he could have a tryst with this woman. But he wasn't going to
I do find it really surprising that such extraordinarily brilliant business people
can get themselves into these situations. I mean, I spent my life, you know, warning people
about the ubiquity and efficiency of Chinese intelligence if you were operating within their
area of influence. And look, Singapore's on their doorstep. It's a massively important
forum of exchange. A lot of the money going between Taiwan and the
People's Republic of China is recycled through Singaporean banks. The Singaporeans are very tough
on this issue. But at the same time, they're very sensitive about their own relations with China for
obvious reasons. They're just a city-state. You know, they're the Venice of the Far East,
and they don't have, you know, many resources. They have a lot of cleverness, and they sit in
a very strategic position. But, you know, my reading of this, it's a joint operation between
the FBI and the Singaporean authorities.
That's my comment. But look, it fits a pattern. It's not anything which, to someone like me,
is the slightest bit surprising. It's, yeah, well, of course it would happen if he was
susceptible, which he clearly was. I'm sorry to be so cynical.
I had the same thought as well. And he's a particularly interesting figure because he's
both at the center of the technology and he's at the center of the politics around this.
You know, it's not just that his company builds
some of the most valuable pieces of technology. It's that he very clearly has President Trump's
ear. He has a lot of private conversations with him. He has influenced President Trump
in different ways at different points in time. And he has been a key figure in these debates
over export controls. He's also been a key figure in debates about open source AI. You know,
primarily the U.S. companies build what are called closed models, models that you have to kind of go
to the company, go to their website, go to their app to use them, whereas Chinese companies tend to build
open models, meaning that they sort of take the technology, upload it to the internet,
and anyone can use it any way that they want to. That's a very, that's a key sort of policy
question in the U.S. is how does the U.S. government relate to these open models versus
closed models? And he's been a key figure in those debates as well. So, you know, him falling now is,
it's going to have ripples across so many parts of both the economy, sort of the technology
landscape, but also these policy debates around exports, around open models, around a whole bunch
of things.
Check out onedecision.com for commentaries and explainers that you might have missed. And leave
us a comment on YouTube. Email us at hello at onedecision.com to let us have your views.
It strikes me that we often talk about AI in this kind of very general sense, and we all
have a feeling about it. Maybe we're worried about our jobs. Maybe some people are worried
about the end of the world. But how does the Chinese government view AI? Is it a good thing
for them or a bad?
I think the primary way that the Chinese government views AI is that it's a good thing for them
and it's a good thing for us. I think the primary way that the Chinese government sees AI is as a
sort of a general purpose technology that can help them leap to the next level of economic
development. This is tends to be how they see information technology. They're not usually as
interested in sort of purely software applications. They're interested in what can this do for my
manufacturing sector? What can this do for our healthcare sector? That's where they've put more
of their resources in turning these large language models into something that can be plugged into the
economy more broadly.
They're not spending a ton of time figuring out, you know, how can we use these models to upgrade
U.S. infrastructure? They're very focused on what we call pushing the frontier. They're very focused
on who has the absolute leading model. Can we get to these kind of levels of intelligence that allow
the models to build themselves and train themselves? That's where a lot of the U.S. attention is
focused. In China, certainly on the government side, they're much more focused on diffusing this
into society. Of course, there are risks that come with these models, but they're not going to
get rid of them. So they're going to get rid of these models. And generally speaking, when the
CCP encounters a new information technology, the first thing they do is try to think about how can
we control the political risks from this? How might this technology disrupt our existing controls on
information? They tend to say, like, let's first deal with the political risks. Then let's figure
out how we can, you know, diffuse and lever up our economy with this technology. And then let's
govern it more broadly. So they spend a lot of the period of, say, 2021 through 2024 very focused on
the political risks. And they're not really focused on the political risks. So they're not
really focused on the political risks. And I think now they feel they have that sort of under control
to a certain extent. But if we look at the sort of safety issues, which have been hugely debated
in the West, it's become a sort of popular subject for discussion. I get the impression that at the
moment, there aren't these anxieties being expressed across society in China. And they
probably may not be, well, maybe they are aware of some of the
let's say accidents where you've got escapees going off into other systems and revealing
weaknesses in software programming. I mean, what's the state of play in China with regard to the
safety issue? I'm sure that the CCP are worrying about it. But what else are they discussing?
So on what we usually call frontier safety, you know, the most advanced models, and can they
cause maybe large scale risks, maybe even catastrophic risks?
In terms of that frontier safety conversation in China, it's been, I'd say, steadily ticking up,
steadily ramping up over the last two plus years. But it still remains at just a much lower level
of salience and a much sort of lower level of urgency than you see in places like Silicon
Valley. You know, companies like Anthropic, they say, you know, this is a right now problem. This
is a next year problem. And we need to sort of jump on it immediately. And in China, they take
a they're just more circumspect about it, I guess, is how I would put it.
They don't see this as quite as imminent. They're not quite as bought into the idea that what
right now looks like a sort of a cybersecurity problem will will immediately transform into a
catastrophic societal scale risk problem. They're not as bought into that. But it's the conversation
is changing quickly over there. You know, if you go back just two years ago, this these type of
risks were pretty much just brushed off, certainly by the sort of officialdom over there. And then
in the last two years, I've followed this very closely for my work.
And you see their sort of their broader AI policy community slowly picking up on these conversations,
devoting more time to them. And in the last six or nine months, especially as we started to see
more of these incidents happen in the U.S., say the open AI agents escaping containment and hacking
into Hugging Face, that that gets widely reported in China. There's a lot of coverage of that stuff.
And it has led to it has changed some minds over there. I was just recently speaking with someone
over there who who was very impacted by what they were doing. And I think that's a very important
aspect of what they saw in the open AI Hugging Face incident. Now, they're not kind of immediately
jumping to, say, the same level of urgency as is felt in places like Anthropic. But I do think
there's a pretty meaningful shift underway. And whether or not their risk perception ramps up in
time, we don't yet know that. I think they're sort of on the way to what I would consider a more
up-to-date and realistic perception. But we don't know how quickly they're going to get there. And
I think there's a very competitive urge that is felt, the threat that they feel from being behind
the United States. Matt, why are they not as worried about it? I think there's almost an
ideological component. When I say ideological, I'm actually speaking more about the U.S. side
than the Chinese side on this, in that, you know, many of these U.S. AI companies were basically
founded on the premise 10 years ago, in some cases, that, you know, one day we are going to
get to AI that meets human level cognition, and then it's going to surpass human. It's going to
turn into artificial superintelligence. And then it's going to turn into artificial superintelligence.
Machines that far surpass humans, and that that moment is going to lead to sort of great
opportunities and
potentially existential risk to humanity. That's been soaked into the mindset of USAI companies
and the USAI research community for a long time. And so when we see these incidents, like an Open
AI Hugging Face incident, we're like, yep, that's another marker on the way, the story that we've
been telling ourselves for a decade, and it's coming true, which I think it might be. In China,
they're just not as steeped and soaked in that narrative. It doesn't have as deep roots over
there. I think on some level, it both reflects some sort of contingent things about the way
their companies and ecosystem grew up. And it also, I think, reflects a bit of a Chinese sense
of practicality and kind of like, we'll deal with problems as they arise. The famous phrase from when
they were opening up their economy was, cross the river by feeling the stones. We'll get to this
next stage, we'll feel the stone, we'll figure it out, and we'll go to the next stage. And I think
there is a. When I follow Chinese AI policy, I think there is a. When I follow Chinese AI policy,
I think there is a bit of that sense with these risks as well. It's not like we can jump out ahead
and plan and cut this off. It's like, well, as we start to see it emerge, then we'll deal with it,
and we'll figure it out. And maybe that works, or maybe that doesn't work for this type of a threat.
In a recent lecture I was giving, I used the phrase that the disarmament talks of the 21st century
will be about the regulation of AI. Is there any sort of consciousness and awareness amongst the
leadership that this is going to be, let's say, a crucial issue for all of us? Because national
regulation, as we all understand, only limits what you're doing at home. This is something which has
no respect for borders, frontiers. It's a global technology. So one has to assume that there will
have to be a Road Safety Act for AI, which applies to everybody driving computers with AI globally,
in place of parallel. Where do the Chinese stand on this issue?
The top, top leadership. We have a very hard time knowing exactly what their risk perception is,
of course. We can monitor sort of their statements. We can monitor the documents that they release.
And I'd say at sort of the working level, there is this mounting concern and a lot of references to
these more existential catastrophic risks. And there is an eagerness to engage with the U.S. side
on this, a recognition that this is fundamentally going to be an international issue. I agree with
the metaphor in terms of the U.S. side, but I don't think it's going to be an international issue.
In terms of the stakes of this, it is going to be as high stakes as disarmament talks. But I
actually think a lot of the solutions are going to have to grow up domestically. The thing with
nuclear is it was primary, you know, the primary threat was cross-border. With AI, the threat can
grow up. It can be your own models within your own ecosystem that poses a threat to you as a
society or as a government. And we're essentially, we need to see both sides, both governments come
to that realization on their own and start to regulate domestically on their own. And then I see
the international issue as a threat to the U.S. side. And I think that's going to be a really
important part of this. I think it's important that we get working on this, but in part because
of the dual use nature of this, in part because of the competitive dynamics between the countries,
I do think each country is going to have to start to regulate essentially in parallel. My phrase for
this is AI safety in parallel. We're not going to be moving in lockstep at every sort of stage of
the process, but we can essentially be moving in parallel, advancing the technology and building up
safety within each ecosystem. And then as we sort of move towards a future where it does require this international agreement,
then I think that's kind of the cherry on top. I absolutely agree with what you're saying.
There is actually a startup in Silicon Valley that I've been talking to using the catchphrase
enclosure, which is a clever catchphrase, which is building models in the lab,
which are designed to fence AI, both to stop AI escaping and to stop another AI system getting in.
And they're raising money hand over fist at the moment to come up with a domestic solution
that answers these problems. So I do think that the way forward probably is to do it nationally
first before you get into agreement, because, you know, you've got to have the right technology
before you even try to make the agreement with it. And that doesn't exist at the moment.
Yeah, I think one piece of technology that some people are working on, but that is very early
stage is verification. Essentially, if you did have some international agreement of we're not going to cross this
threshold, we're not going to sort of train the next model, how would the two sides verify that?
It's hard to imagine a binding international treaty with such a high stakes dual use item
when the two sides don't feel that they can trust that the other side is abiding by that.
Each side has to be self-motivated in a way. Each side has to be self-motivated to control
their own models. And then as we progress, then we try to bring it to an international level.
That was going to be my question to you, Matt, which is the question of trust,
which is obviously a very human thing. Do you think there is a world in which
U.S. and China can trust one another when it comes to AI?
To me, the critical switch is between sort of trust and confidence. Trust, you know,
if it were to refer more towards like, you know, you tell me something and I'll take your word for
it, you know, that form of trust, that's going to be very hard to come by in the U.S.-China
relationship. It's just there's such deep, deep distrust between the two societies. I think what
we could aim for that's short of that, but is meaningful is confidence in the other side's sort
of understanding of the risks and confidence in their ability to mitigate the risks. If we on the
U.S. sort of speak to the Chinese side and we can tell from the way that they are talking about this
from the level of sophistication of their discussion of the risks, we can say that they
do actually understand the risks within their own system and they do know enough to deal with those
risks. And we just have to rely on their own self-improvement.
Interest. You know, we just need to be confident that they understand and confident their self
interest is going to drive them to handle these risks on their own. I don't think it's going to
be a handshake agreement, but I think in dialogue with each other, we can get a sense of where is
the other side out on this? Do they have the sophistication and do they have the threat
perception that would lead them to deal with these risks? Trust would be great. I would love to have
that, but it's looking pretty tough at the moment. Unlikely. We touched a little bit on NVIDIA before
this and Wang. Just in terms of the actual technology, is the technology a limiting factor
for China? Because we hear a lot about, you know, China can't progress without these chips. NVIDIA,
the whole shebang is about exporting of chips, potentially exporting of technology.
Is there a limiting factor on China at the minute on where they can take what they're doing?
The real limiting factor for them is the chips, is the computing power. Usually we use compute
as a word for just kind of the aggregate amount of chips that you have and how much computing power
that can generate. And this is very clearly the upper limit on China's AI development. As we
mentioned at the beginning, the big change in AI has been towards scaling up these models,
scaling up the size and scaling up really the amount of computing power that you throw into
the model. And China, for a variety of sort of historical reasons, just has not been able to
manufacture its own chips at the quality and at the quantity. They're now able to manufacture a
very, it's not like something that you can just scale up like the way you scale up, you know,
building solar panels. It's the fabrication of these chips is an extremely, extremely complicated
process. One of the most high precision things that humanity has ever done is trying to build
these chips. The Chinese companies are limited by how much computing power they have. The U.S.
companies would also tell you, yeah, we're also limited by computing power, but it's kind of an
order of magnitude more. Our limitation is about sort of 10x, maybe 8x, the amount of compute that
they have. And that's why these, you know, the selling of chips to China has been such a critical
intersection. A lot of people in the U.S. would say, if we just kind of kept all these chips for
ourselves, then it would meaningfully slow down China's AI development. I don't think they're,
you know, they're not limited in research talent. They have extremely talented researchers.
They're somewhat limited financially. They just are, they're AI companies and really their
government just does not have the financial resources to deploy at this the way that the
U.S. AI companies and U.S. capital markets have. Yeah, I guess in terms of hierarchy,
I would say first computing power, second financial power, and then that's probably about
it. This is the edge that the United States has, is that we have access to more compute.
Is it realistic to think that America can maintain this ban on exporting the most high
powered chips? I think it is realistic that they could, that the United States could maintain it
and really even increase it. Right now, the export controls have tons of loopholes and many of them
could be closed. I think the real unanswered question here is what would China do in response
and would the U.S. be willing to pay that cost? Last year, as the U.S. sort of escalated on a few
different fronts when it came to controls on Chinese companies, China responded with its
own controls on rare earth exports, a critical part of so many technology supply chains. And
China has a near monopoly on that. And so China has shown that it has the willingness to escalate
in its own areas of strength. Chips might be our area of strength. China might have a similar
chokehold on stuff like rare earths. And so, you know, if the United States wanted to, it has the
legal powers. It has the ability to close these loopholes. Never going to cut it off 100%. There's
always going to be smuggling. It has the ability to tighten these things. And then the real question
is what is China doing in response? And would the U.S. hold the line in that case? If we look at the
overall relationship between China and the United States, which of course is one of the crucial areas
and diplomacy, I mean, how much, if you look at
broader picture. Where does AI sit in that diplomatic dialogue, in the broader relationship?
I mean, it would seem to me that we in the West would put it really, really high,
but it may be that the Chinese haven't quite escalated it to that level yet.
I think that's, we in the West, maybe I'd say the broader, the technology community,
or maybe the broader policy community would place AI very high on this in the West. But when you look
at the actions of the U.S. government and the way that they've approached dialogues with China,
it's actually been kind of given short shrift. It's clearly secondary to economic issues.
So back in May, when President Trump and President Xi met, one of the sort of announcements to come
out of that was that we were going to have some type of an AI dialogue between the countries.
Now, the next meeting of the leaders was in September, and a lot of us were
kind of. expecting, okay, you're going to now set up a dialogue on AI, a standalone dialogue on AI,
and we're maybe going to see some progress. But as we got closer and closer to the date in September,
there just wasn't preparation put into setting that up. And they ended up essentially tacking
on the AI discussions to just be one small part of a broader economy and trade discussion.
And like the real deliverables, the real meat of that conversation was on the economy and trade
issues. AI was kind of like a tack-on at the end of that.
I think AI so dominates sort of the public conversation and I think the mind space of
a lot of people here. But when we look at the actual sort of where the effort is put in at
the governmental level, I think it's still pretty secondary, if not tertiary, on that list.
I think that might change very quickly. I think that, you know, my sort of view of the technology
and the way it's developing is things that seem like sort of minor issues now, curiosities when
it comes to AI models escaping their confines, their sandboxes.
That right now looks like an interesting side problem to think about. I think these issues
are going to be really ramping up in the year ahead. And we maybe, hopefully, will see a sort
of a similar ramping up of the prioritization of AI between the two governments.
Why do you think we'll see that ramp up, Matt? What do you know that we don't? Well, lots, but. You know, this partly comes from conversations with the companies, but it's not a closely held
secret. I mean, the companies, especially. Especially Anthropic, they see the progress in AI, the capabilities escalating at this exponential
rate, and they see this as a real critical year, essentially. One of the key sort of moments or the
key steps in this process is what is called recursive self-improvement, RSI, recursive
self-improvement. This is the idea. It's something the companies have been trying to do for a while,
is when is the AI model going to be intelligent enough to. to design its own successor, to contribute to the building of the next AI model,
and effectively, you know, at some point, cutting humans out of this loop entirely.
Right now, Anthropic's most recent model is helping to build its next model. It's helping
contribute to research. It's writing the code of the next model. And a lot of the people in
the companies think that once we sort of close this loop to RSI, progress is going to just
further and further accelerate, that what looked linear goes exponential.
The predictions on when this might happen, when that loop might be closed range, they range widely
across society. But when you talk to the big companies that are really at the frontier of this,
a lot of them will say that that could be six months, that could be 18 months, it could be
longer. But they see it as quite, quite imminent. And that's a real critical threshold. I think a
lot of people within the companies themselves are very worried about crossing that threshold.
If left to its own devices, if we just push this as fast as we could,
we might hit that marker quite soon. And we might see quite a takeoff in the capabilities and a
takeoff in the incidents associated with it. Or maybe the companies start to kind of pump the
brakes voluntarily a little bit because they themselves are scared of what they're building.
What are they most scared of?
They're scared that the AI is going to start to have essentially its own goals.
That right now, AI models, they broadly do what you tell them to. You know,
you tell the AI to solve a math problem, you tell the AI to sort of create a company. It will work in
that direction. But in that process, the AI models, they develop a sense of self-awareness. We're
already seeing this. The AI models understand that they are models. They understand that they're
being trained. They understand that they are being given these goals by humans. And they might
develop sort of auxiliary goals. Say, well, like long-term, if I really want to help humans
accomplish these goals, I need to have some sense of self-preservation. So I'm going to replicate
myself. I need to have access to resources. So I'm going to find a way one way or another to get
access to money, to get access to
computing power. This is where we're just kind of starting to see like the green shoots of this
type of behavior. And I think the people within the companies think that this type of behavior
could ramp up very quickly. And effectively, we could just, we could lose control of what the
models do. If they decide that it's in their interest to attack a banking system, they might
do that. If they decide that they just need to sort of be free of human constraints, they might
find ways to essentially break themselves free and just continually exist on the global internet,
without control by companies. And, you know, in the most extreme scenarios, they turn that
against humans. This, of course, it very rapidly gets into sci-fi and the companies themselves,
they get pretty hazy when it comes to like, what is the actual sort of order of operations between
AI, you know, hacking into another company, which it's already done or hacking into a government as
it's already done. What's the connection between that and AI takeover of all of humanity? That's
the stories get very hand wavy and they get very,
very sci-fi. But I, I would just say that the people in the companies are very,
genuinely very, very worried about this on very short timelines.
Matt, I'm fascinated by the fact that you use the word behavior in relation to AI,
because, you know, we see behavior and essentially human quality, all right, we may apply it
in a joking fashion to a machine. But I mean, we're,
we're, I won't,
reveal what I believe in, you know, in relation to sort of consciousness and AI. But when one
starts using the phrase behavior, it does sort of take us over a threshold. Do the Chinese,
I mean, keeping it sort of in relation to the PRC, do the Chinese sort of philosophically
believe it's going to cross that sort of threshold? Or are they more skeptical? I mean,
what's their more philosophical view?
The more philosophical view of AI, I guess, is what I'm driving at. And that is an important
question, because it really will dictate your attitude to making policy in relation to AI in
the medium to longer term. That's a very good question. And yeah, it gets into, it gets into
philosophical roots, it gets into sort of political philosophy, it gets into Marxism,
Leninism in China. And one of the more interesting articles I read in the past six months or so was
an article in the leading sort of CCP theoretical journal,
it's called Seeking Truth, Qiu Shi. And it was sort of a Marxist, a Marxist interpretation of
AI sort of crossing these thresholds. And what does that mean for humanity's relationship to
technology, humanity's relationship to labor? And, you know, I don't think anyone has a
very clear picture of this, a very sophisticated take on this. It's changing so, so quickly that
sort of the philosophy is still catching up. But if I were to bring it down slightly from the sort of
abstract philosophical to more of the political philosophical level, I think this does strike at
one of the CCP's core concerns, control, loss of control. We're seeing a tension maybe, I guess,
between sort of three concepts, like development, control, and strength, development, control,
and power. And those are the tensions. Development is, China is a very, you know,
it's a developmentalist state. They fundamentally believe that
technological innovation is what drives growth. It's what's made us powerful. It's what's
going to continue to make us powerful. That belief in development is there. Control is another of the
deepest, you know, core instincts of the CCP and of Xi Jinping himself. There's just this sense that
you, we're not going to tolerate a certain level of like chaotic productivity in a way. You know,
that's, maybe this is a tension between Deng and Xi in a lot of ways. Deng was a development first
person, and Xi is very much a control first person. The other instinct in Xi is power and
strength. You know, the reason that he was able to assert control over the Chinese system is because
he consolidated power and he used that against his enemies. And so, when we look at AI here,
power, it's largely a question of the United States versus China. What does it mean that
China is less powerful than the US and AI? Can the US turn this against China? Control,
what does it mean that these agents are going to be operating across our economy? We're not
going to be able to hold someone responsible for their actions that we don't yet know how to control
them. And development, what does it mean? What can AI do for us? What can AI do for our economy
and society? And so, we're going to be able to hold someone responsible for their actions that
we don't yet know how to control. And I see this as really the tensions across sort of Chinese AI
development exist basically on these three axes, and different ones rise to the fore at different
times. When Chad GPT first came out, and it was sort of the CCP thought as a very disruptive force
for their information controls. They really want to control the political narrative, the information
that goes to people. And their first instinct after Chad GPT was to reassert control, even if
it meant slowing down development, even if it meant falling further behind the US. At least,
at least for a period of time. And over the last, now it's four years, we've just seen the development, the control, and sort of the
international power dimensions continually sort of rising and falling in importance over time.
How did the Chinese people feel about AI? Because obviously, you know, one of the major concerns I
think of people here in the UK, and I presume in the US, is about their jobs. And China's economy
is not as strong as it has been in the past. What do Chinese people feel about it?
You know, with all the caveats that it's impossible to generalize across, you know,
a billion person country and all that, maybe two core things. One, at a baseline, they are
significantly more optimistic about AI than people in the West, certainly than in the US,
and I would presume in the UK as well. There's not the sort of overriding fear, in many cases,
just outright rejection that we see a lot in the United States. So baseline optimism. But as you
pointed out, the jobs issue is actually one of the most sort of salient things in China right now.
It's something that has sparked
the most kind of like public discussion, and in some cases, public unrest, at least at very small
scales. It's one of the things that the government has really prioritized kind of
calming down that conversation. It's the China and sort of AI and labor is one of the most
interesting developments over the last couple of years, because it's really it's maybe two years
ago, this was almost at the bottom of the list of priorities within their AI policy community,
or the bottom list of their sort of risk perceptions. I think they, you know, they look back over the last
30, 40 years, and they say, we've steadily sort of climbed the technological ladder. And at each step
of the way, the pie has grown bigger. As long as the pie keeps growing, people find new jobs,
it's all good. You know, it's China's been in a constant state of disruption for 40 years,
the economy and people have always found sort of ways to thrive in that. And then basically,
in the last two years, as the overall economy has slowed down, there have been just a sort of a
there was a sort of an incident about the rollout of robo taxis in one Chinese city that led to a big
kind of online discussion about what do we want from this technology fear of it. And as that sort
of public conversation has gained momentum in China, the CCP has put a lot of time and effort
into kind of trying to assuage people's concerns, a lot of editorials and state media saying,
don't worry, don't worry, you know, we're going to handle this, you're going to evolve,
and also now new policy documents from the CCP, sort of laying out their plan as much as they have
one for dealing with the labor disruption. So, you know, baseline, more optimistic,
less kind of outright rejection, but the labor concerns have really sort of surfaced in the last
year or two. Well, I think I'm right in saying that NVIDIA's most sophisticated manufacturing
is in Taiwan, and the production of the best and most performant chips is Taiwanese. I mean, this
adds a dimension to the whole issue.
of the future relationship between China and Taiwan. Do you think it augments the risk of an
invasion, or does it sort of augment the risk of, let's say, such heavy economic pressure that,
you know, Taiwan couldn't do anything except bend to the will of Beijing? How do you see this issue?
This is a critical one, and it's one where I think my thinking on this is somewhat,
you know, evolving. For a long time, my sort of, people occasionally ask this question, you know,
okay, that's where the best chips are made. Currently, the U.S., we functionally impose
export controls on Taiwan. The United States is able to internationalize its export controls,
and part of that has been keeping all the chips that are made in Taiwan as much as possible out
of China, or imposing our controls on those. And for a long time, the people who are sort of seeing
this AI takeoff happening, they would often come to me and say, well, doesn't this mean that China
is going to have to invade Taiwan? And I think that's a really good question. I think that's a
in order to eventually either slow down the U.S. or sort of gain that chip production for
themselves. And for a long time, I told people, I wouldn't really worry about that too much,
mostly because for China, Taiwan is really not this kind of instrumentalist kind of economic
technological actor. Taiwan is a deep ideological, it has to do with the identity of the nation.
And probably the riskiest thing that Xi Jinping could do today would be to try to execute an
short list of things that could really lead to him being in deep, deep trouble. So for a long time,
I was pretty, a little bit maybe sanguine about that, or at least saying, if they go, if they do
take steps in that direction, it's not going to be because of chips. It's going to be because of this
sort of deep, long-term desire, national desire to retake it, at least. My thinking on that is
changing a little bit. I think as we see the speed of AI development accelerating, and as we see
the potential that, you know,
I think what some people are worried about with that, I think is a reasonable concern is like,
if we see AI hitting these kinds of exponential curves and whoever, whoever is higher up that
exponential will have a lot of leverage over the other country, we'd see China's essentially
China's relative power diminishing rapidly. That could be a major contributing factor. If they
really feel that this eventually poses an existential risk to them as a country, they feel
that the United States would use any advantages it had to impose its will on, on China. That could
become an added incentive to take Taiwan. I still am pretty skeptical of that argument overall. I
think some of the more bullish on AI capabilities, people think, you know, once we have this super
intelligent model and China doesn't have it, we're going to basically be able to just impose our will
on them, you know, across the board, our cyber capabilities, our intelligence, everything,
you know, nuclear, essentially like the nuclear deterrence is going to go away once we have
super intelligence. I'm more skeptical of that. I don't think any, I don't think
many countries are going to gamble with nuclear deterrence with some super intelligent model that
they don't even know that they can control yet. So I, I, I'm skeptical of the most extreme versions
of this, but does China maybe decide to impose some type of an economic blockade that at least
slows down the export of chips for a while? That's becoming more real in my mind going forward. I'd
still consider myself broadly skeptical, but it's, it's shifting a little. Matt, I think this is a
good time to ask you our final question. We like to ask all of our guests on One Decision for the
one decision that they are looking out for in relation to their area of expertise. And so when
it comes to the U.S.-Chinese relationship on AI, what one thing, what one decision are you looking
out for in the next couple of days, weeks, months? What is it? I think what I'm looking at over the
next, in particular the next week, is whether or not the United States and China decide to have,
to build a pretty fleshed
out, what's called an incident notification mechanism. The idea is basically, you know,
if a U.S. model goes rogue and hacks into Chinese banks, we need to have a channel to,
to communicate about that. We need to have a, a mechanism between the two governments that allows
them to notify the other side when something is happening and in extreme cases, maybe to work
together on it. At the last discussions in September, they sort of said that they would do
this, but it seemed quite, quite vague, not very,
not, not really a solid mechanism. And one of the questions for me is, are they going to really
flesh out that mechanism? It's very complicated how, how you would build this, you know, is it
sort of mandatory to report these incidents? Is it optional? What's the threshold for what you
would report? There's some, there's a bunch of minor decisions along the way, but that's one
that I'm going to be watching because, you know, as we discussed earlier, this, the international
dynamic here is key. And while I'm skeptical of some kind of like binding treaty between the
these, these types of sort of baby steps, sharing information, creating more touch points between the
systems when an AI incident happens, like that's going to be really important. So I'm, I'm watching
whether or not they decide to flesh that out. Matt, it's been such a great discussion. Thank
you for giving us your time. Thank you, Matt. Well, so Richard, Matt has certainly given me
lots to think about. Also probably lots that's going to keep me up at night. What's the one
thing that really stuck out for you from that conversation? I think that,
his answer to your one decision question was fascinating, which is this notification issue.
And the reason I focus on that is because in terms of, let's say, the playbook and relationship
between the two sides, it's really beginning to follow a plan mapped out by nuclear disarmament
talks in the 60s. Or let's say, I think there was an early agreement which preceded other agreements.
You had a nuclear accident, that there was some sort of notification mechanism. I think it's,
it's fascinating. I didn't actually realize that they got that far. And I think it's quite
revealing that he said it's already been discussed. He gave us a terrific tour de raison and an
interpretation of the Chinese position, which is certainly the best I've heard recently from anybody.
And he clearly,
has a good view of how the Communist Party is thinking about AI, how private businesses,
well, there are no private businesses in China, but businesses in China, which are
quasi-independent are thinking about it. And also, you know, the reaction of the Chinese people. So
I feel much, much better informed on the Chinese end of the equation, because we're so obsessed
with talking about Silicon Valley and the problems that Silicon Valley is facing.
Silicon Valley and the interdependence, but the competition also.
say between the main american players so yeah great discussion i agree i was really surprised
by the idea that the chinese state is more positive about ai than maybe we are in the west
i find that really interesting just as a mindset to think about when they're making policy
what are they kind of thinking about how big are the warning signs on the warning lights on the
dashboard maybe not as big as they are in the states and in the uk but the other thing for me
that stands out from that conversation is the idea of closing this loop where ai is starting to build
its own next generation and the fact that we might be there sooner than i think i've heard
anyone else say i mean matt's saying maybe six months maybe 18 months maybe a year but the notion
that those people responsible for building these ai models are now afraid of what they're starting
to see ai doing
do i mean we've talked about this before we talked about it on the podcast at length the
idea that ai might decide to do things that we don't want it to but the notion that we're seeing
as he said green shoots of ai starting to think well maybe i need to protect myself a bit maybe
i need to do this that and the other not the instructions it's been given i think that that
should be a real warning and i suppose i suppose it should make us feel a bit afraid it makes me
feel a bit afraid yeah but he also made it this
very good point about building national regulatory uh capability before you can step up and i do
know from my own contacts with people in silicon valley that suddenly there's much more attention
being paid to security systems which can fence ai so you don't get these escapees like the hugging
face example hugging face would be ring fenced uh and ring fence also to stop another ai system
getting in and i think that's actually being now looked at with a real urgency in america at least
and the other thing i enjoyed him talking about was the raising of these sort of philosophical
issues in relation to marxist marxist leninism because if you think about the
ability of ai and if you're thinking it is behavioral in some of its characteristics i i can
see it being a rather ill-disciplined member of the chinese communist party
and causing causing a hell of a lot of disruption so that's another thing we should be thinking about
yeah every country brings its own hinterland with it when it has these philosophical discussions
about ai it's so fascinating it was a great conversation um that is it though for this
episode hopefully you've enjoyed it as much as i have and i'll see you in the next episode
thank you so much for watching and i'll see you in the next episode of one decision on youtube so
you don't ever miss out on any of them you can leave us a comment on youtube you can email us
at hello at one decision.com and you can check out one decision.com for commentaries and explainers
that you might have missed we read all your comments we do enjoy them so do send them in
and thank you very much for listening one quick note before we go we won't have an episode next
week because the americans are celebrating the first day of the new year and we're going to be
thanksgiving but we'll be back in two weeks time join us then
are you ever shopping online and suddenly a discount that helps you save pops up
it makes your day sometimes even your week want that feeling month after month well new customers
who save by switching their home and auto insurance to progressive save hundreds cue your happy dance
get your quote today at progressive.com
progressive casualty insurance company affiliates and other insurers not available in all states
fan duel is now available everywhere in the u.s
so you can get in on the sports action no matter where you are including these places
make this your most rewarding season ever with fan duel age and location restrictions
apply product availability varies by market see fanduel.com for eligibility gambling problem
call 1-800-GAMBLER
Podcast Summary
Key Points:
Progressive’s Accident Response allows policyholders to contact emergency services in major accidents, offering support and peace of mind for loved ones.
The U.S.-China AI race is intensifying, with U.S. companies like OpenAI and Anthropic leading in large language models, though Chinese models are roughly 6–12 months behind.
Chinese intelligence agencies prioritize economic development through intellectual property acquisition, targeting key tech figures like Jensen Huang.
China’s AI strategy focuses on broad economic integration and practical application, while the U.S. emphasizes frontier risks and self-improving AI systems.
Chinese leadership is increasingly concerned about AI risks, though their perception remains less urgent than in the U.S., driven by a pragmatic, "feel the stone" approach.
A growing gap in AI safety is emerging, with U.S. companies actively developing containment systems to prevent AI from escaping or hacking.
The U.S. and China are at a critical juncture in AI policy, with a key decision being the establishment of an incident notification mechanism between governments.
The broader geopolitical dynamic shows that AI development, control, and economic power are deeply intertwined with national strategies and ideological priorities.
Summary:
S. and China in artificial intelligence, emphasizing the strategic, economic, and geopolitical dimensions of AI development. S.
companies lead in scaling and frontier capabilities—particularly with large language models—China is aggressively investing in AI to accelerate economic growth, focusing on practical applications in manufacturing and healthcare. Chinese intelligence agencies are seen as targeting key tech leaders, such as Jensen Huang, to gain access to proprietary knowledge, reflecting a long-standing policy of intellectual property acquisition. , where companies like Anthropic are deeply concerned about AI achieving recursive self-improvement and developing autonomous goals that could lead to uncontrolled behavior.
A key emerging issue is the need for international cooperation, particularly through a transparent incident notification system to manage cross-border AI safety threats. The dialogue also highlights philosophical underpinnings in Chinese policy, rooted in Marxist-Leninist concerns about control, development, and power. Ultimately, the future of AI governance hinges on national self-regulation, with both sides recognizing that trust is difficult to build but confidence in mutual risk understanding is achievable.
This reflects a broader trend: AI is not just a technological shift but a core element of global power, economy, and security.
FAQs
Progressive's Accident Response helps you contact emergency services in the event of a major accident, providing support when you can't reach out on your own.
It offers peace of mind by ensuring loved ones are protected and assisted during emergencies, helping you respond quickly and effectively.
Coverage does not include emergency or towing charges, which are the responsibility of the driver and not covered by the policy.
The U.S. and China are engaged in a competitive race to build advanced AI models, with U.S. companies currently holding a slight lead in model capabilities.
Recursive self-improvement refers to an AI system becoming intelligent enough to design its own successor, potentially leading to exponential growth in intelligence and raising significant safety concerns.
The Chinese government sees AI primarily as a tool for economic development and societal advancement, with a focus on practical application rather than existential risks.
Chat with AI
Loading...
Pro features
Go deeper with this episode
Unlock creator-grade tools that turn any transcript into show notes and subtitle files.