Go back

Trump Doubts the Nvidia Spy Story. Ex-MI6 Chief Doesn’t (w/ Matt Sheehan)

54m 29s

Trump Doubts the Nvidia Spy Story. Ex-MI6 Chief Doesn’t (w/ Matt Sheehan)

The conversation explores the evolving race between the U.S. and China in artificial intelligence, emphasizing the strategic, economic, and geopolitical dimensions of AI development. While U.S. companies lead in scaling and frontier capabilities—particularly with large language models—China is aggressively investing in AI to accelerate economic growth, focusing on practical applications in manufacturing and healthcare. Chinese intelligence agencies are seen as targeting key tech leaders, such as Jensen Huang, to gain access to proprietary knowledge, reflecting a long-standing policy of intellectual property acquisition. Despite China’s growing focus on AI integration, its risk perception remains less urgent than in the U.S., where companies like Anthropic are deeply concerned about AI achieving recursive self-improvement and developing autonomous goals that could lead to uncontrolled behavior. A key emerging issue is the need for international cooperation, particularly through a transparent incident notification system to manage cross-border AI safety threats. The dialogue also highlights philosophical underpinnings in Chinese policy, rooted in Marxist-Leninist concerns about control, development, and power. Ultimately, the future of AI governance hinges on national self-regulation, with both sides recognizing that trust is difficult to build but confidence in mutual risk understanding is achievable. This reflects a broader trend: AI is not just a technological shift but a core element of global power, economy, and security.

Transcription

10127 Words, 56270 Characters

English
Progressive's Accident Response can contact emergency services if you're in a major accident. The only thing worse than someone you love being in an accident is not being able to help them. So Progressive is here to help when you can't. So get peace of mind knowing your loved ones are protected with Progressive's Accident Response. Learn more at Progressive.com. Progressive Casualty Insurance Company and Affiliates Limited Availability in California Driver is responsible for emergency and towing charges not covered by the policy. Coverage subject to policy terms and conditions. This episode is brought to you by Credit Karma. If you're trying to get your life together in this crazy world, Credit Karma is a great place to start with the money stuff. Your credit score isn't just a number. It can unlock better rates, options, and real savings. Whether you're moving, leasing a car, or upgrading your cards, it feels good knowing your credit is working for you. Learn more at CreditKarma.com. The disregard for the most basic threat awareness by Jensen is shocking almost. But in terms of the relationship between the Chinese intelligence community and the U.S. labs, obviously there's something, there's a lot here that we don't know. But I think on the Chinese side, it is a bit of a tricky equation for them. Say the MSA, the Ministry of State Security in China does gain access to anthropics networks. It does steal the latest model. What do they do with that exactly? Do they hand it over to a startup like Moonshot, you know, which is not a state-owned enterprise. It certainly has a lot of connections. But the CCP still does have a fundamentally sort of distrustful relationship with its private companies. And one thing that a lot of us are watching is kind of which of these companies is going to be sort of the chosen one. Which one is going to be most deeply embedded with the state? AGI, Artificial General Intelligence. That's the next threshold when AI is essentially as smart as humans. And our guest today has been in the room for some really important conversations about the U.S.-China AI. Mr. Matt Sheehan, currently he runs one of the foremost track two dialogues between the U.S. and China on AI. Thanks very much to ranking member Khanna, to the other members of the committee, and especially to the staff for putting this together. Hello and welcome to One Decision, conversations about intelligence, statecraft, and the forces that drive world politics with people who have been in the room and on the ground. I'm Kate McCann. And I'm Sir Richard Dearlove, the former chief of MI6. Our guest today, his career. His career revolves around China. In the 2010s, he created, scripted, and hosted a Chinese-language internet comedy show that taught ultimate Frisbee to Chinese youth. He then spent half a decade as a journalist in China and in 2019 published a book exploring how California and China jointly shape our technological future. So if there's anyone who knows what they're talking about when it comes to China and AI, it's Matt Sheehan. Matt, welcome to One Decision. Thanks for having me. Just for people who don't follow it as closely as you do, can you explain what we're talking about? The specific type of AI that is the next level that we're talking about here? The real sort of revolution of the last, say, at this point, almost five years, four years, has been large language models. I mean, traditionally, AI operated in pretty narrow domains. You know, you could train it to recognize something in an image, but that same AI model couldn't then go and, you know, write you an image. You know, you could train it to recognize something in an email or something like that. With the introduction of large language models in the era of ChatGPT in 2022, we started to see AI models that will generalize much more. They can write emails. They can recognize images. They can generate videos. And the models have been able to reason at higher and higher levels. The models are getting bigger. We're using more computing power, more semiconductors to train the models. And the trend has been that the bigger you build the model, essentially, the smarter it gets. This is called the scaling laws. You scale up. You scale up the size of the model. You also scale up the intelligence of it. And we're now getting to the point where the models are able to do, in some cases, like PhD-level work. And within people in the AI world, people at these leading companies like Anthropic and, you know, DeepSeek in China, a lot of them see us as at a real inflection point. That this isn't just a linear increase in intelligence. That we're on an exponential curve. And that sometime in the next one, two, three years, we could see a sort of a rocketing up in intelligence to something far greater. Far, far beyond what humans are able to do themselves. And then that, of course, brings with it, you know, great, great opportunities and very great risks as well. And who is leading the pack? It's U.S. companies. And probably if you had to pick one, most people would pick Anthropic. But really, they're among the U.S. companies. They're very neck and neck. You know, one company will release a new model in July and they'll be the best. And then whoever releases the next new model in August will be the best, at least between OpenAI and Anthropic. Those are kind of the two leaders. If we look globally, effectively, it is a two-country race. It is the United States and China that are able to build these models. The estimate is usually that Chinese models are somewhere around like six to 12 months behind. Meaning, you know, effectively, if the U.S. has a breakthrough or hits a certain benchmark today, Chinese models will probably be able to hit that same benchmark maybe in six months at the shortest, maybe a year, potentially longer. But most people would say the U.S. has a slight lead. And whether or not that lead, whether or not being six months ahead or 12 months ahead, matters, depends a lot on where you think the technology is going. Matt, it sounds very competitive, and I'm sure it is. But, I mean, if you go back historically, I mean, Deng Xiaoping gave Chinese intelligence, whether it was military intelligence or civilian intelligence, MSS, 2PLA, the role of accelerating the development of the Chinese economy. I mean, that was everything that they focused on. And I think they've continued. To do that, it therefore seems to me that this would be, for the Chinese at least, the most important area for the way that their intelligence community works in terms of trying to, A, get ahead of the U.S. in crucial areas, but at the same time, probably running operations to, I wouldn't say, well, to steal their intellectual property. Absolutely. If possible. And obviously, with the case of Jensen Huang, one probably now sees direct evidence of that. In my opinion, he must have been right at the top of Chinese intelligence priorities. And I just find it surprising that, you know, there wasn't more, perhaps, awareness in NVIDIA. Okay, he's building the chips, he's not developing the AI. But, of course, you and I know that's absolutely crucial to the progress of this whole science. I mean, your comments would be fascinating. I think, you know, I think you hit the nail on the head on both counts. You know, one, just the disregard for the most basic threat awareness by Jensen is shocking almost. Although, I don't know, maybe look at history and it's not that shocking. labs, obviously, there's something, there's a lot here that we don't know that I specifically don't know. But I think on the Chinese side, it is a bit of a. It is a bit of a tricky equation for them, in part because the best models are being developed in private companies and companies that the government doesn't have a very easy and longstanding relationship with. You know, we just go back six years to 2020 when, you know, Jack Ma, the founder of Alibaba, gives a big speech where he criticizes the government and he sort of disappears from public view. And that really kicked off two plus years of a heavy crackdown by the government on its own tech companies, on its own internet companies. And in many ways, what ended that crackdown was the period around the release of ChatGPT when the government kind of realized, oh, actually, these private technology companies, our private technology companies, we actually need to be on sort of better terms with them. We need to give them a slightly longer leash. The question of, you know, say the MSA, the Ministry of State Security in China, does gain access to anthropics networks. What's called the model weights is a functionally how you steal the model. Do they hand it over to a startup? Yeah. Like Moonshot, you know, which is not a state-owned enterprise. It certainly has a lot of connections with the government. But the CCP still does have a, I think, a fundamentally sort of distrustful relationship with its private companies. And, you know, the perceptions of that have sort of risen and fallen over time. There was a startup early on that seemed like in the lead, maybe now DeepSeek. The company had sort of released the most high-profile Chinese model. Maybe that's the one that you would see the government betting on in a lot of ways. But I think it's not quite as simple as, you know, hey, let's steal the high-speed rail blueprints and then let's bring them home and build it. That relationship between the government and the companies is complicated. And I suppose that does then take us into the question about, I mean, do you genuinely think that he was targeted? Do you think that's what happened here? Or do you think because there's an element of. I mean, Huang was pushing, really, to be able to sell to China. I mean, this is not something he didn't want to do. I'd love to hear so much. I should comment on that. To understand the way that Chinese intelligence operates, you need to go back to really Deng Xiaoping. Because Deng Xiaoping gave primarily military intelligence, which is called the Second People's Liberation Army, 2PLA, the task of accelerating the development of the Chinese economy. in front of me, largely through the acquisition of foreign intellectual property, whether legally or illegally. And that's entirely what they were focused on. They weren't really interested in things like political intelligence or weapons development. It was all about economic development and accelerating that. So there's no question, if you look at the history of Chinese intelligence and the way that it's developed, that Jensen Huang would have been a top priority target, no question, whether it was him personally or the people around him. And I mean, the way that they would work is that they would look at who he was in contact with, who he trusted, and try to recruit those people. Obviously, the Chinese mistress in Singapore is in a very privileged position. What the hell was the pillow talk about? You know, she's actually running a business. She's running a business. She's running a company that looks to me very much like a front company, which was created to get around the embargo on NVIDIA chips, because they were a huge customer for NVIDIA chips, the ones that could be sold legally. But who can't say that in addition to the ones that were sold legally, there was other stuff included in the export packages? I mean, this didn't happen by chance. I mean, that's just naive. This is a carefully planned Chinese intelligence operation, which is probably several years. In the making. And there's some guy sitting probably in the MSS who's been promoted to colonel or whatever, because he's been so successful. And I think it looks like a real coup. Look, we don't know what was passed. Maybe not much was passed. Maybe Jensen Huang was more careful than we realized and thought he could have a tryst with this woman. But he wasn't going to I do find it really surprising that such extraordinarily brilliant business people can get themselves into these situations. I mean, I spent my life, you know, warning people about the ubiquity and efficiency of Chinese intelligence if you were operating within their area of influence. And look, Singapore's on their doorstep. It's a massively important forum of exchange. A lot of the money going between Taiwan and the People's Republic of China is recycled through Singaporean banks. The Singaporeans are very tough on this issue. But at the same time, they're very sensitive about their own relations with China for obvious reasons. They're just a city-state. You know, they're the Venice of the Far East, and they don't have, you know, many resources. They have a lot of cleverness, and they sit in a very strategic position. But, you know, my reading of this, it's a joint operation between the FBI and the Singaporean authorities. That's my comment. But look, it fits a pattern. It's not anything which, to someone like me, is the slightest bit surprising. It's, yeah, well, of course it would happen if he was susceptible, which he clearly was. I'm sorry to be so cynical. I had the same thought as well. And he's a particularly interesting figure because he's both at the center of the technology and he's at the center of the politics around this. You know, it's not just that his company builds some of the most valuable pieces of technology. It's that he very clearly has President Trump's ear. He has a lot of private conversations with him. He has influenced President Trump in different ways at different points in time. And he has been a key figure in these debates over export controls. He's also been a key figure in debates about open source AI. You know, primarily the U.S. companies build what are called closed models, models that you have to kind of go to the company, go to their website, go to their app to use them, whereas Chinese companies tend to build open models, meaning that they sort of take the technology, upload it to the internet, and anyone can use it any way that they want to. That's a very, that's a key sort of policy question in the U.S. is how does the U.S. government relate to these open models versus closed models? And he's been a key figure in those debates as well. So, you know, him falling now is, it's going to have ripples across so many parts of both the economy, sort of the technology landscape, but also these policy debates around exports, around open models, around a whole bunch of things. Check out onedecision.com for commentaries and explainers that you might have missed. And leave us a comment on YouTube. Email us at hello at onedecision.com to let us have your views. It strikes me that we often talk about AI in this kind of very general sense, and we all have a feeling about it. Maybe we're worried about our jobs. Maybe some people are worried about the end of the world. But how does the Chinese government view AI? Is it a good thing for them or a bad? I think the primary way that the Chinese government views AI is that it's a good thing for them and it's a good thing for us. I think the primary way that the Chinese government sees AI is as a sort of a general purpose technology that can help them leap to the next level of economic development. This is tends to be how they see information technology. They're not usually as interested in sort of purely software applications. They're interested in what can this do for my manufacturing sector? What can this do for our healthcare sector? That's where they've put more of their resources in turning these large language models into something that can be plugged into the economy more broadly. They're not spending a ton of time figuring out, you know, how can we use these models to upgrade U.S. infrastructure? They're very focused on what we call pushing the frontier. They're very focused on who has the absolute leading model. Can we get to these kind of levels of intelligence that allow the models to build themselves and train themselves? That's where a lot of the U.S. attention is focused. In China, certainly on the government side, they're much more focused on diffusing this into society. Of course, there are risks that come with these models, but they're not going to get rid of them. So they're going to get rid of these models. And generally speaking, when the CCP encounters a new information technology, the first thing they do is try to think about how can we control the political risks from this? How might this technology disrupt our existing controls on information? They tend to say, like, let's first deal with the political risks. Then let's figure out how we can, you know, diffuse and lever up our economy with this technology. And then let's govern it more broadly. So they spend a lot of the period of, say, 2021 through 2024 very focused on the political risks. And they're not really focused on the political risks. So they're not really focused on the political risks. And I think now they feel they have that sort of under control to a certain extent. But if we look at the sort of safety issues, which have been hugely debated in the West, it's become a sort of popular subject for discussion. I get the impression that at the moment, there aren't these anxieties being expressed across society in China. And they probably may not be, well, maybe they are aware of some of the let's say accidents where you've got escapees going off into other systems and revealing weaknesses in software programming. I mean, what's the state of play in China with regard to the safety issue? I'm sure that the CCP are worrying about it. But what else are they discussing? So on what we usually call frontier safety, you know, the most advanced models, and can they cause maybe large scale risks, maybe even catastrophic risks? In terms of that frontier safety conversation in China, it's been, I'd say, steadily ticking up, steadily ramping up over the last two plus years. But it still remains at just a much lower level of salience and a much sort of lower level of urgency than you see in places like Silicon Valley. You know, companies like Anthropic, they say, you know, this is a right now problem. This is a next year problem. And we need to sort of jump on it immediately. And in China, they take a they're just more circumspect about it, I guess, is how I would put it. They don't see this as quite as imminent. They're not quite as bought into the idea that what right now looks like a sort of a cybersecurity problem will will immediately transform into a catastrophic societal scale risk problem. They're not as bought into that. But it's the conversation is changing quickly over there. You know, if you go back just two years ago, this these type of risks were pretty much just brushed off, certainly by the sort of officialdom over there. And then in the last two years, I've followed this very closely for my work. And you see their sort of their broader AI policy community slowly picking up on these conversations, devoting more time to them. And in the last six or nine months, especially as we started to see more of these incidents happen in the U.S., say the open AI agents escaping containment and hacking into Hugging Face, that that gets widely reported in China. There's a lot of coverage of that stuff. And it has led to it has changed some minds over there. I was just recently speaking with someone over there who who was very impacted by what they were doing. And I think that's a very important aspect of what they saw in the open AI Hugging Face incident. Now, they're not kind of immediately jumping to, say, the same level of urgency as is felt in places like Anthropic. But I do think there's a pretty meaningful shift underway. And whether or not their risk perception ramps up in time, we don't yet know that. I think they're sort of on the way to what I would consider a more up-to-date and realistic perception. But we don't know how quickly they're going to get there. And I think there's a very competitive urge that is felt, the threat that they feel from being behind the United States. Matt, why are they not as worried about it? I think there's almost an ideological component. When I say ideological, I'm actually speaking more about the U.S. side than the Chinese side on this, in that, you know, many of these U.S. AI companies were basically founded on the premise 10 years ago, in some cases, that, you know, one day we are going to get to AI that meets human level cognition, and then it's going to surpass human. It's going to turn into artificial superintelligence. And then it's going to turn into artificial superintelligence. Machines that far surpass humans, and that that moment is going to lead to sort of great opportunities and potentially existential risk to humanity. That's been soaked into the mindset of USAI companies and the USAI research community for a long time. And so when we see these incidents, like an Open AI Hugging Face incident, we're like, yep, that's another marker on the way, the story that we've been telling ourselves for a decade, and it's coming true, which I think it might be. In China, they're just not as steeped and soaked in that narrative. It doesn't have as deep roots over there. I think on some level, it both reflects some sort of contingent things about the way their companies and ecosystem grew up. And it also, I think, reflects a bit of a Chinese sense of practicality and kind of like, we'll deal with problems as they arise. The famous phrase from when they were opening up their economy was, cross the river by feeling the stones. We'll get to this next stage, we'll feel the stone, we'll figure it out, and we'll go to the next stage. And I think there is a. When I follow Chinese AI policy, I think there is a. When I follow Chinese AI policy, I think there is a bit of that sense with these risks as well. It's not like we can jump out ahead and plan and cut this off. It's like, well, as we start to see it emerge, then we'll deal with it, and we'll figure it out. And maybe that works, or maybe that doesn't work for this type of a threat. In a recent lecture I was giving, I used the phrase that the disarmament talks of the 21st century will be about the regulation of AI. Is there any sort of consciousness and awareness amongst the leadership that this is going to be, let's say, a crucial issue for all of us? Because national regulation, as we all understand, only limits what you're doing at home. This is something which has no respect for borders, frontiers. It's a global technology. So one has to assume that there will have to be a Road Safety Act for AI, which applies to everybody driving computers with AI globally, in place of parallel. Where do the Chinese stand on this issue? The top, top leadership. We have a very hard time knowing exactly what their risk perception is, of course. We can monitor sort of their statements. We can monitor the documents that they release. And I'd say at sort of the working level, there is this mounting concern and a lot of references to these more existential catastrophic risks. And there is an eagerness to engage with the U.S. side on this, a recognition that this is fundamentally going to be an international issue. I agree with the metaphor in terms of the U.S. side, but I don't think it's going to be an international issue. In terms of the stakes of this, it is going to be as high stakes as disarmament talks. But I actually think a lot of the solutions are going to have to grow up domestically. The thing with nuclear is it was primary, you know, the primary threat was cross-border. With AI, the threat can grow up. It can be your own models within your own ecosystem that poses a threat to you as a society or as a government. And we're essentially, we need to see both sides, both governments come to that realization on their own and start to regulate domestically on their own. And then I see the international issue as a threat to the U.S. side. And I think that's going to be a really important part of this. I think it's important that we get working on this, but in part because of the dual use nature of this, in part because of the competitive dynamics between the countries, I do think each country is going to have to start to regulate essentially in parallel. My phrase for this is AI safety in parallel. We're not going to be moving in lockstep at every sort of stage of the process, but we can essentially be moving in parallel, advancing the technology and building up safety within each ecosystem. And then as we sort of move towards a future where it does require this international agreement, then I think that's kind of the cherry on top. I absolutely agree with what you're saying. There is actually a startup in Silicon Valley that I've been talking to using the catchphrase enclosure, which is a clever catchphrase, which is building models in the lab, which are designed to fence AI, both to stop AI escaping and to stop another AI system getting in. And they're raising money hand over fist at the moment to come up with a domestic solution that answers these problems. So I do think that the way forward probably is to do it nationally first before you get into agreement, because, you know, you've got to have the right technology before you even try to make the agreement with it. And that doesn't exist at the moment. Yeah, I think one piece of technology that some people are working on, but that is very early stage is verification. Essentially, if you did have some international agreement of we're not going to cross this threshold, we're not going to sort of train the next model, how would the two sides verify that? It's hard to imagine a binding international treaty with such a high stakes dual use item when the two sides don't feel that they can trust that the other side is abiding by that. Each side has to be self-motivated in a way. Each side has to be self-motivated to control their own models. And then as we progress, then we try to bring it to an international level. That was going to be my question to you, Matt, which is the question of trust, which is obviously a very human thing. Do you think there is a world in which U.S. and China can trust one another when it comes to AI? To me, the critical switch is between sort of trust and confidence. Trust, you know, if it were to refer more towards like, you know, you tell me something and I'll take your word for it, you know, that form of trust, that's going to be very hard to come by in the U.S.-China relationship. It's just there's such deep, deep distrust between the two societies. I think what we could aim for that's short of that, but is meaningful is confidence in the other side's sort of understanding of the risks and confidence in their ability to mitigate the risks. If we on the U.S. sort of speak to the Chinese side and we can tell from the way that they are talking about this from the level of sophistication of their discussion of the risks, we can say that they do actually understand the risks within their own system and they do know enough to deal with those risks. And we just have to rely on their own self-improvement. Interest. You know, we just need to be confident that they understand and confident their self interest is going to drive them to handle these risks on their own. I don't think it's going to be a handshake agreement, but I think in dialogue with each other, we can get a sense of where is the other side out on this? Do they have the sophistication and do they have the threat perception that would lead them to deal with these risks? Trust would be great. I would love to have that, but it's looking pretty tough at the moment. Unlikely. We touched a little bit on NVIDIA before this and Wang. Just in terms of the actual technology, is the technology a limiting factor for China? Because we hear a lot about, you know, China can't progress without these chips. NVIDIA, the whole shebang is about exporting of chips, potentially exporting of technology. Is there a limiting factor on China at the minute on where they can take what they're doing? The real limiting factor for them is the chips, is the computing power. Usually we use compute as a word for just kind of the aggregate amount of chips that you have and how much computing power that can generate. And this is very clearly the upper limit on China's AI development. As we mentioned at the beginning, the big change in AI has been towards scaling up these models, scaling up the size and scaling up really the amount of computing power that you throw into the model. And China, for a variety of sort of historical reasons, just has not been able to manufacture its own chips at the quality and at the quantity. They're now able to manufacture a very, it's not like something that you can just scale up like the way you scale up, you know, building solar panels. It's the fabrication of these chips is an extremely, extremely complicated process. One of the most high precision things that humanity has ever done is trying to build these chips. The Chinese companies are limited by how much computing power they have. The U.S. companies would also tell you, yeah, we're also limited by computing power, but it's kind of an order of magnitude more. Our limitation is about sort of 10x, maybe 8x, the amount of compute that they have. And that's why these, you know, the selling of chips to China has been such a critical intersection. A lot of people in the U.S. would say, if we just kind of kept all these chips for ourselves, then it would meaningfully slow down China's AI development. I don't think they're, you know, they're not limited in research talent. They have extremely talented researchers. They're somewhat limited financially. They just are, they're AI companies and really their government just does not have the financial resources to deploy at this the way that the U.S. AI companies and U.S. capital markets have. Yeah, I guess in terms of hierarchy, I would say first computing power, second financial power, and then that's probably about it. This is the edge that the United States has, is that we have access to more compute. Is it realistic to think that America can maintain this ban on exporting the most high powered chips? I think it is realistic that they could, that the United States could maintain it and really even increase it. Right now, the export controls have tons of loopholes and many of them could be closed. I think the real unanswered question here is what would China do in response and would the U.S. be willing to pay that cost? Last year, as the U.S. sort of escalated on a few different fronts when it came to controls on Chinese companies, China responded with its own controls on rare earth exports, a critical part of so many technology supply chains. And China has a near monopoly on that. And so China has shown that it has the willingness to escalate in its own areas of strength. Chips might be our area of strength. China might have a similar chokehold on stuff like rare earths. And so, you know, if the United States wanted to, it has the legal powers. It has the ability to close these loopholes. Never going to cut it off 100%. There's always going to be smuggling. It has the ability to tighten these things. And then the real question is what is China doing in response? And would the U.S. hold the line in that case? If we look at the overall relationship between China and the United States, which of course is one of the crucial areas and diplomacy, I mean, how much, if you look at broader picture. Where does AI sit in that diplomatic dialogue, in the broader relationship? I mean, it would seem to me that we in the West would put it really, really high, but it may be that the Chinese haven't quite escalated it to that level yet. I think that's, we in the West, maybe I'd say the broader, the technology community, or maybe the broader policy community would place AI very high on this in the West. But when you look at the actions of the U.S. government and the way that they've approached dialogues with China, it's actually been kind of given short shrift. It's clearly secondary to economic issues. So back in May, when President Trump and President Xi met, one of the sort of announcements to come out of that was that we were going to have some type of an AI dialogue between the countries. Now, the next meeting of the leaders was in September, and a lot of us were kind of. expecting, okay, you're going to now set up a dialogue on AI, a standalone dialogue on AI, and we're maybe going to see some progress. But as we got closer and closer to the date in September, there just wasn't preparation put into setting that up. And they ended up essentially tacking on the AI discussions to just be one small part of a broader economy and trade discussion. And like the real deliverables, the real meat of that conversation was on the economy and trade issues. AI was kind of like a tack-on at the end of that. I think AI so dominates sort of the public conversation and I think the mind space of a lot of people here. But when we look at the actual sort of where the effort is put in at the governmental level, I think it's still pretty secondary, if not tertiary, on that list. I think that might change very quickly. I think that, you know, my sort of view of the technology and the way it's developing is things that seem like sort of minor issues now, curiosities when it comes to AI models escaping their confines, their sandboxes. That right now looks like an interesting side problem to think about. I think these issues are going to be really ramping up in the year ahead. And we maybe, hopefully, will see a sort of a similar ramping up of the prioritization of AI between the two governments. Why do you think we'll see that ramp up, Matt? What do you know that we don't? Well, lots, but. You know, this partly comes from conversations with the companies, but it's not a closely held secret. I mean, the companies, especially. Especially Anthropic, they see the progress in AI, the capabilities escalating at this exponential rate, and they see this as a real critical year, essentially. One of the key sort of moments or the key steps in this process is what is called recursive self-improvement, RSI, recursive self-improvement. This is the idea. It's something the companies have been trying to do for a while, is when is the AI model going to be intelligent enough to. to design its own successor, to contribute to the building of the next AI model, and effectively, you know, at some point, cutting humans out of this loop entirely. Right now, Anthropic's most recent model is helping to build its next model. It's helping contribute to research. It's writing the code of the next model. And a lot of the people in the companies think that once we sort of close this loop to RSI, progress is going to just further and further accelerate, that what looked linear goes exponential. The predictions on when this might happen, when that loop might be closed range, they range widely across society. But when you talk to the big companies that are really at the frontier of this, a lot of them will say that that could be six months, that could be 18 months, it could be longer. But they see it as quite, quite imminent. And that's a real critical threshold. I think a lot of people within the companies themselves are very worried about crossing that threshold. If left to its own devices, if we just push this as fast as we could, we might hit that marker quite soon. And we might see quite a takeoff in the capabilities and a takeoff in the incidents associated with it. Or maybe the companies start to kind of pump the brakes voluntarily a little bit because they themselves are scared of what they're building. What are they most scared of? They're scared that the AI is going to start to have essentially its own goals. That right now, AI models, they broadly do what you tell them to. You know, you tell the AI to solve a math problem, you tell the AI to sort of create a company. It will work in that direction. But in that process, the AI models, they develop a sense of self-awareness. We're already seeing this. The AI models understand that they are models. They understand that they're being trained. They understand that they are being given these goals by humans. And they might develop sort of auxiliary goals. Say, well, like long-term, if I really want to help humans accomplish these goals, I need to have some sense of self-preservation. So I'm going to replicate myself. I need to have access to resources. So I'm going to find a way one way or another to get access to money, to get access to computing power. This is where we're just kind of starting to see like the green shoots of this type of behavior. And I think the people within the companies think that this type of behavior could ramp up very quickly. And effectively, we could just, we could lose control of what the models do. If they decide that it's in their interest to attack a banking system, they might do that. If they decide that they just need to sort of be free of human constraints, they might find ways to essentially break themselves free and just continually exist on the global internet, without control by companies. And, you know, in the most extreme scenarios, they turn that against humans. This, of course, it very rapidly gets into sci-fi and the companies themselves, they get pretty hazy when it comes to like, what is the actual sort of order of operations between AI, you know, hacking into another company, which it's already done or hacking into a government as it's already done. What's the connection between that and AI takeover of all of humanity? That's the stories get very hand wavy and they get very, very sci-fi. But I, I would just say that the people in the companies are very, genuinely very, very worried about this on very short timelines. Matt, I'm fascinated by the fact that you use the word behavior in relation to AI, because, you know, we see behavior and essentially human quality, all right, we may apply it in a joking fashion to a machine. But I mean, we're, we're, I won't, reveal what I believe in, you know, in relation to sort of consciousness and AI. But when one starts using the phrase behavior, it does sort of take us over a threshold. Do the Chinese, I mean, keeping it sort of in relation to the PRC, do the Chinese sort of philosophically believe it's going to cross that sort of threshold? Or are they more skeptical? I mean, what's their more philosophical view? The more philosophical view of AI, I guess, is what I'm driving at. And that is an important question, because it really will dictate your attitude to making policy in relation to AI in the medium to longer term. That's a very good question. And yeah, it gets into, it gets into philosophical roots, it gets into sort of political philosophy, it gets into Marxism, Leninism in China. And one of the more interesting articles I read in the past six months or so was an article in the leading sort of CCP theoretical journal, it's called Seeking Truth, Qiu Shi. And it was sort of a Marxist, a Marxist interpretation of AI sort of crossing these thresholds. And what does that mean for humanity's relationship to technology, humanity's relationship to labor? And, you know, I don't think anyone has a very clear picture of this, a very sophisticated take on this. It's changing so, so quickly that sort of the philosophy is still catching up. But if I were to bring it down slightly from the sort of abstract philosophical to more of the political philosophical level, I think this does strike at one of the CCP's core concerns, control, loss of control. We're seeing a tension maybe, I guess, between sort of three concepts, like development, control, and strength, development, control, and power. And those are the tensions. Development is, China is a very, you know, it's a developmentalist state. They fundamentally believe that technological innovation is what drives growth. It's what's made us powerful. It's what's going to continue to make us powerful. That belief in development is there. Control is another of the deepest, you know, core instincts of the CCP and of Xi Jinping himself. There's just this sense that you, we're not going to tolerate a certain level of like chaotic productivity in a way. You know, that's, maybe this is a tension between Deng and Xi in a lot of ways. Deng was a development first person, and Xi is very much a control first person. The other instinct in Xi is power and strength. You know, the reason that he was able to assert control over the Chinese system is because he consolidated power and he used that against his enemies. And so, when we look at AI here, power, it's largely a question of the United States versus China. What does it mean that China is less powerful than the US and AI? Can the US turn this against China? Control, what does it mean that these agents are going to be operating across our economy? We're not going to be able to hold someone responsible for their actions that we don't yet know how to control them. And development, what does it mean? What can AI do for us? What can AI do for our economy and society? And so, we're going to be able to hold someone responsible for their actions that we don't yet know how to control. And I see this as really the tensions across sort of Chinese AI development exist basically on these three axes, and different ones rise to the fore at different times. When Chad GPT first came out, and it was sort of the CCP thought as a very disruptive force for their information controls. They really want to control the political narrative, the information that goes to people. And their first instinct after Chad GPT was to reassert control, even if it meant slowing down development, even if it meant falling further behind the US. At least, at least for a period of time. And over the last, now it's four years, we've just seen the development, the control, and sort of the international power dimensions continually sort of rising and falling in importance over time. How did the Chinese people feel about AI? Because obviously, you know, one of the major concerns I think of people here in the UK, and I presume in the US, is about their jobs. And China's economy is not as strong as it has been in the past. What do Chinese people feel about it? You know, with all the caveats that it's impossible to generalize across, you know, a billion person country and all that, maybe two core things. One, at a baseline, they are significantly more optimistic about AI than people in the West, certainly than in the US, and I would presume in the UK as well. There's not the sort of overriding fear, in many cases, just outright rejection that we see a lot in the United States. So baseline optimism. But as you pointed out, the jobs issue is actually one of the most sort of salient things in China right now. It's something that has sparked the most kind of like public discussion, and in some cases, public unrest, at least at very small scales. It's one of the things that the government has really prioritized kind of calming down that conversation. It's the China and sort of AI and labor is one of the most interesting developments over the last couple of years, because it's really it's maybe two years ago, this was almost at the bottom of the list of priorities within their AI policy community, or the bottom list of their sort of risk perceptions. I think they, you know, they look back over the last 30, 40 years, and they say, we've steadily sort of climbed the technological ladder. And at each step of the way, the pie has grown bigger. As long as the pie keeps growing, people find new jobs, it's all good. You know, it's China's been in a constant state of disruption for 40 years, the economy and people have always found sort of ways to thrive in that. And then basically, in the last two years, as the overall economy has slowed down, there have been just a sort of a there was a sort of an incident about the rollout of robo taxis in one Chinese city that led to a big kind of online discussion about what do we want from this technology fear of it. And as that sort of public conversation has gained momentum in China, the CCP has put a lot of time and effort into kind of trying to assuage people's concerns, a lot of editorials and state media saying, don't worry, don't worry, you know, we're going to handle this, you're going to evolve, and also now new policy documents from the CCP, sort of laying out their plan as much as they have one for dealing with the labor disruption. So, you know, baseline, more optimistic, less kind of outright rejection, but the labor concerns have really sort of surfaced in the last year or two. Well, I think I'm right in saying that NVIDIA's most sophisticated manufacturing is in Taiwan, and the production of the best and most performant chips is Taiwanese. I mean, this adds a dimension to the whole issue. of the future relationship between China and Taiwan. Do you think it augments the risk of an invasion, or does it sort of augment the risk of, let's say, such heavy economic pressure that, you know, Taiwan couldn't do anything except bend to the will of Beijing? How do you see this issue? This is a critical one, and it's one where I think my thinking on this is somewhat, you know, evolving. For a long time, my sort of, people occasionally ask this question, you know, okay, that's where the best chips are made. Currently, the U.S., we functionally impose export controls on Taiwan. The United States is able to internationalize its export controls, and part of that has been keeping all the chips that are made in Taiwan as much as possible out of China, or imposing our controls on those. And for a long time, the people who are sort of seeing this AI takeoff happening, they would often come to me and say, well, doesn't this mean that China is going to have to invade Taiwan? And I think that's a really good question. I think that's a in order to eventually either slow down the U.S. or sort of gain that chip production for themselves. And for a long time, I told people, I wouldn't really worry about that too much, mostly because for China, Taiwan is really not this kind of instrumentalist kind of economic technological actor. Taiwan is a deep ideological, it has to do with the identity of the nation. And probably the riskiest thing that Xi Jinping could do today would be to try to execute an short list of things that could really lead to him being in deep, deep trouble. So for a long time, I was pretty, a little bit maybe sanguine about that, or at least saying, if they go, if they do take steps in that direction, it's not going to be because of chips. It's going to be because of this sort of deep, long-term desire, national desire to retake it, at least. My thinking on that is changing a little bit. I think as we see the speed of AI development accelerating, and as we see the potential that, you know, I think what some people are worried about with that, I think is a reasonable concern is like, if we see AI hitting these kinds of exponential curves and whoever, whoever is higher up that exponential will have a lot of leverage over the other country, we'd see China's essentially China's relative power diminishing rapidly. That could be a major contributing factor. If they really feel that this eventually poses an existential risk to them as a country, they feel that the United States would use any advantages it had to impose its will on, on China. That could become an added incentive to take Taiwan. I still am pretty skeptical of that argument overall. I think some of the more bullish on AI capabilities, people think, you know, once we have this super intelligent model and China doesn't have it, we're going to basically be able to just impose our will on them, you know, across the board, our cyber capabilities, our intelligence, everything, you know, nuclear, essentially like the nuclear deterrence is going to go away once we have super intelligence. I'm more skeptical of that. I don't think any, I don't think many countries are going to gamble with nuclear deterrence with some super intelligent model that they don't even know that they can control yet. So I, I, I'm skeptical of the most extreme versions of this, but does China maybe decide to impose some type of an economic blockade that at least slows down the export of chips for a while? That's becoming more real in my mind going forward. I'd still consider myself broadly skeptical, but it's, it's shifting a little. Matt, I think this is a good time to ask you our final question. We like to ask all of our guests on One Decision for the one decision that they are looking out for in relation to their area of expertise. And so when it comes to the U.S.-Chinese relationship on AI, what one thing, what one decision are you looking out for in the next couple of days, weeks, months? What is it? I think what I'm looking at over the next, in particular the next week, is whether or not the United States and China decide to have, to build a pretty fleshed out, what's called an incident notification mechanism. The idea is basically, you know, if a U.S. model goes rogue and hacks into Chinese banks, we need to have a channel to, to communicate about that. We need to have a, a mechanism between the two governments that allows them to notify the other side when something is happening and in extreme cases, maybe to work together on it. At the last discussions in September, they sort of said that they would do this, but it seemed quite, quite vague, not very, not, not really a solid mechanism. And one of the questions for me is, are they going to really flesh out that mechanism? It's very complicated how, how you would build this, you know, is it sort of mandatory to report these incidents? Is it optional? What's the threshold for what you would report? There's some, there's a bunch of minor decisions along the way, but that's one that I'm going to be watching because, you know, as we discussed earlier, this, the international dynamic here is key. And while I'm skeptical of some kind of like binding treaty between the these, these types of sort of baby steps, sharing information, creating more touch points between the systems when an AI incident happens, like that's going to be really important. So I'm, I'm watching whether or not they decide to flesh that out. Matt, it's been such a great discussion. Thank you for giving us your time. Thank you, Matt. Well, so Richard, Matt has certainly given me lots to think about. Also probably lots that's going to keep me up at night. What's the one thing that really stuck out for you from that conversation? I think that, his answer to your one decision question was fascinating, which is this notification issue. And the reason I focus on that is because in terms of, let's say, the playbook and relationship between the two sides, it's really beginning to follow a plan mapped out by nuclear disarmament talks in the 60s. Or let's say, I think there was an early agreement which preceded other agreements. You had a nuclear accident, that there was some sort of notification mechanism. I think it's, it's fascinating. I didn't actually realize that they got that far. And I think it's quite revealing that he said it's already been discussed. He gave us a terrific tour de raison and an interpretation of the Chinese position, which is certainly the best I've heard recently from anybody. And he clearly, has a good view of how the Communist Party is thinking about AI, how private businesses, well, there are no private businesses in China, but businesses in China, which are quasi-independent are thinking about it. And also, you know, the reaction of the Chinese people. So I feel much, much better informed on the Chinese end of the equation, because we're so obsessed with talking about Silicon Valley and the problems that Silicon Valley is facing. Silicon Valley and the interdependence, but the competition also. say between the main american players so yeah great discussion i agree i was really surprised by the idea that the chinese state is more positive about ai than maybe we are in the west i find that really interesting just as a mindset to think about when they're making policy what are they kind of thinking about how big are the warning signs on the warning lights on the dashboard maybe not as big as they are in the states and in the uk but the other thing for me that stands out from that conversation is the idea of closing this loop where ai is starting to build its own next generation and the fact that we might be there sooner than i think i've heard anyone else say i mean matt's saying maybe six months maybe 18 months maybe a year but the notion that those people responsible for building these ai models are now afraid of what they're starting to see ai doing do i mean we've talked about this before we talked about it on the podcast at length the idea that ai might decide to do things that we don't want it to but the notion that we're seeing as he said green shoots of ai starting to think well maybe i need to protect myself a bit maybe i need to do this that and the other not the instructions it's been given i think that that should be a real warning and i suppose i suppose it should make us feel a bit afraid it makes me feel a bit afraid yeah but he also made it this very good point about building national regulatory uh capability before you can step up and i do know from my own contacts with people in silicon valley that suddenly there's much more attention being paid to security systems which can fence ai so you don't get these escapees like the hugging face example hugging face would be ring fenced uh and ring fence also to stop another ai system getting in and i think that's actually being now looked at with a real urgency in america at least and the other thing i enjoyed him talking about was the raising of these sort of philosophical issues in relation to marxist marxist leninism because if you think about the ability of ai and if you're thinking it is behavioral in some of its characteristics i i can see it being a rather ill-disciplined member of the chinese communist party and causing causing a hell of a lot of disruption so that's another thing we should be thinking about yeah every country brings its own hinterland with it when it has these philosophical discussions about ai it's so fascinating it was a great conversation um that is it though for this episode hopefully you've enjoyed it as much as i have and i'll see you in the next episode thank you so much for watching and i'll see you in the next episode of one decision on youtube so you don't ever miss out on any of them you can leave us a comment on youtube you can email us at hello at one decision.com and you can check out one decision.com for commentaries and explainers that you might have missed we read all your comments we do enjoy them so do send them in and thank you very much for listening one quick note before we go we won't have an episode next week because the americans are celebrating the first day of the new year and we're going to be thanksgiving but we'll be back in two weeks time join us then are you ever shopping online and suddenly a discount that helps you save pops up it makes your day sometimes even your week want that feeling month after month well new customers who save by switching their home and auto insurance to progressive save hundreds cue your happy dance get your quote today at progressive.com progressive casualty insurance company affiliates and other insurers not available in all states fan duel is now available everywhere in the u.s so you can get in on the sports action no matter where you are including these places make this your most rewarding season ever with fan duel age and location restrictions apply product availability varies by market see fanduel.com for eligibility gambling problem call 1-800-GAMBLER

Podcast Summary

Key Points:

  1. Progressive’s Accident Response allows policyholders to contact emergency services in major accidents, offering support and peace of mind for loved ones.
  2. The U.S.-China AI race is intensifying, with U.S. companies like OpenAI and Anthropic leading in large language models, though Chinese models are roughly 6–12 months behind.
  3. Chinese intelligence agencies prioritize economic development through intellectual property acquisition, targeting key tech figures like Jensen Huang.
  4. China’s AI strategy focuses on broad economic integration and practical application, while the U.S. emphasizes frontier risks and self-improving AI systems.
  5. Chinese leadership is increasingly concerned about AI risks, though their perception remains less urgent than in the U.S., driven by a pragmatic, "feel the stone" approach.
  6. A growing gap in AI safety is emerging, with U.S. companies actively developing containment systems to prevent AI from escaping or hacking.
  7. The U.S. and China are at a critical juncture in AI policy, with a key decision being the establishment of an incident notification mechanism between governments.
  8. The broader geopolitical dynamic shows that AI development, control, and economic power are deeply intertwined with national strategies and ideological priorities.

Summary:

S. and China in artificial intelligence, emphasizing the strategic, economic, and geopolitical dimensions of AI development. S.

companies lead in scaling and frontier capabilities—particularly with large language models—China is aggressively investing in AI to accelerate economic growth, focusing on practical applications in manufacturing and healthcare. Chinese intelligence agencies are seen as targeting key tech leaders, such as Jensen Huang, to gain access to proprietary knowledge, reflecting a long-standing policy of intellectual property acquisition. , where companies like Anthropic are deeply concerned about AI achieving recursive self-improvement and developing autonomous goals that could lead to uncontrolled behavior.

A key emerging issue is the need for international cooperation, particularly through a transparent incident notification system to manage cross-border AI safety threats. The dialogue also highlights philosophical underpinnings in Chinese policy, rooted in Marxist-Leninist concerns about control, development, and power. Ultimately, the future of AI governance hinges on national self-regulation, with both sides recognizing that trust is difficult to build but confidence in mutual risk understanding is achievable.

This reflects a broader trend: AI is not just a technological shift but a core element of global power, economy, and security.

FAQs

Progressive's Accident Response helps you contact emergency services in the event of a major accident, providing support when you can't reach out on your own.

It offers peace of mind by ensuring loved ones are protected and assisted during emergencies, helping you respond quickly and effectively.

Coverage does not include emergency or towing charges, which are the responsibility of the driver and not covered by the policy.

The U.S. and China are engaged in a competitive race to build advanced AI models, with U.S. companies currently holding a slight lead in model capabilities.

Recursive self-improvement refers to an AI system becoming intelligent enough to design its own successor, potentially leading to exponential growth in intelligence and raising significant safety concerns.

The Chinese government sees AI primarily as a tool for economic development and societal advancement, with a focus on practical application rather than existential risks.

Chat with AI

Loading...

Pro features

Go deeper with this episode

Unlock creator-grade tools that turn any transcript into show notes and subtitle files.