What Safeguarding Nukes Can Teach Us About Safeguarding A.I.

From the New York Times, I'm Michael Babar. This is the Daily. Last night in a lavish ceremony, President Trump welcomed Chinese leader Xi Jinping as he arrived in the United States for two days of closely watched negotiations. On their crowded agenda, what if any role they should play in regulating artificial intelligence amid a global freak out over its risks? Today, as my colleague, David Sanger explains, "When it comes to protecting humanity from technological catastrophe, there's really only one proven model for the two leaders to look to." It's Thursday, September 24th. David, up from Washington, D.C., sitting here in our New York studios, and I didn't even have to come to you this time. You know, I just had to come see the Daily's Empire in New York. In person? In person? Yeah. It's a humble empire. It's a late stage. So David, when President Trump and the leader of China Xi Jinping meet in Washington today, they're going to broach this subject on many people's minds, which is the risk of artificial intelligence, that risk, including, of course, these swarming rogue AI agents that have been hacking at companies and basically breaking the rules of their creators. But what Xi and Trump uniquely can maybe do about it is going to be the subject of at least some of this meeting. And I think we've all come to understand that the leaders of the US and China are uniquely positioned to talk about the safety of AI because they're the countries with the most advanced frontier AI laboratories, and the only way that AI gets contained is if both of those countries somehow act together and at the same time. And as this meeting approached, and this is why we love you so much, you reached out to us to say, you know, I have seen this movie before. Big fears, big thoughts about coordination and solutions to big problems. You told us there's a historical parallel to all of this and maybe a historical solution. Well, Michael, there is a obvious parallel. We had an era starting in the late 1940s and through the Cold War, where two states, the Soviet Union and the United States, had created a potentially terrifying technology that had all kinds of commercial uses, but could also bring about civilizational extinction. Nuclear weapons. Nuclear weapons. And everybody had to go figure out how we were going to get the benefits without getting the terror. And you know what? It took decades to happen in these two countries. They locked arms and they came up with agreements that ultimately have brought us to the remarkable moment that there has been no use of a nuclear weapon in anger since 1945. Right. Nuclear non-proliferation worked. Nuclear non-proliferation combined with a balance of terror worked. So now, quite naturally, everybody's looking to that model as we begin to think about how you might contain artificial intelligence, particularly after this summer of surprises that we've had where we have rogue agents breaking out of their sandboxes. Right. Treat AI the way superpowers treated nuclear weaponry. Well, look, if you think that the heads of some of these frontier companies are coming out and saying there was a 10% chance of civilizational extinction, that's probably a higher proportion chance than there was in the early days of the nuclear era that we would wipe each other off the map, at least it's equivalent. And so now the question is, can two very different countries, in very different times, China and the United States come to some kind of framework agreement that could begin to get us there? Okay. Well, given all of that, in your capacity, David, as a historian of all the world's great technological dangers, I think it would be instructive to have you tell us the story of nuclear non-proliferation, how that old framework that kept nuclear weapons from going off around the world, could in theory apply right now to AI. So just give us a quick history lesson here. Well, let's just start with when the US developed its first atomic weapon through the Manhattan project, familiar to anybody who's been off to see the Oppenheimer movie or read some of this history, who's in the 1940s. And at the time, there was a deep fear that another country would beat us to it. But it wasn't the Soviets at that time. It was Germany. In a large desperate attempt to escape defeat, German science threw everything it had into the perfection of V2. Where the fear of American intelligence was, the Germans would get a nuclear weapon before the United States. And there was this just abiding fear we had to get there first. In the spreading, teaming laboratories or the Manhattan project, the pace intensified. Groves had ordered that the first bomb must be ready for testing in mid-July. And in the summer of 1945, scientists from Los Alamos went out to do the Trinity test, which was the first explosion of a nuclear bomb to see if the technology could work. It occurred to someone that if the bomb succeeded, half new Mexico would be littered with fragments of radioactive steel. Many men who looked at it at the moment of detonation would never see again. There were people within the Los Alamos program who were doing mathematical calculations and coming to the conclusion that we could light the entire atmosphere of fire and destroy the earth and doing the test. But despite those fears, they went ahead with the test. A short time ago, an American airplane dropped one bomb on Hiroshima. Came Hiroshima. 815 in the morning found a 400-pound bomb with a destructive force of 20,000 tons of TNT mushrooming up over the stunned enemy city through the frightened inhabitants the end of the world had come. And days after that Nagasaki. And in total, those two bombs killed around 200,000 people. Of course, the world had never seen anything like this before. Tens of thousands of homes and buildings were wiped out. The devastation was everywhere. Japan was driven to quick surrender for the future, the choice is peace or total destruction. The atomic age is here. But somewhat astonishingly, it didn't stop the arms race. Of course, it turned out to be just the beginning. And now, of course, the US thought it had an insurmountable lead over countries that didn't yet have nuclear weapons. That lead didn't last very long. The Soviets were working away on this. They had a bunch of atomic spies along the way who helped them to it. And by 1949, they had conducted a test and we were off into the races. And the 1950s were a period of great terror, where each side was not only building up its arsenal, but building up the powers of these weapons. As if the weapons used at Hiroshima and Nagasaki were not powerful enough. And by this stage, proliferation was starting. "Off North West Australia, this was the place chosen to explode Britain's first atomic weapon." Britain and France had nuclear weapons, China wasn't that far behind. "I've just been told that China has exploded an atomic bomb." Right, things are starting to spin kind of out of control. They are. And then we had one big terrifying incident. "The purpose of these bases can be none other than to provide a nuclear strike capability against the West in hemisphere." And it was the Cuban missile crisis in October of 1962. This was when US intelligence came up with evidence that the Soviets were moving nuclear weapons onto Cuba, that they would be just 90 miles from Florida, that they could easily reach any place on the East Coast. "The waterfront in Havana and along other parts of the coast bristled with gun emplacements as the Cuban regime waited to see what their bosses in the Kremlin were to do." And the Kennedy administration quickly determined this was intolerable, but the question came, is this worth going to war over? Air support for any limited-war ground operation which might develop within 12 hours all units were assembled and ready to go. And there were moments when Kennedy thought and many around him thought that the United States was going to disappear, you know, in the next night or two. A final chapter in the Cuban crisis is brought to a close, as President Kennedy holds a news conference to announce. But in the end, we know how this ended. I heard that today had been informed by Chairman Khrushchev that all of the IR-28 bombers now in Cuba will be withdrawn in 30 days. The nuclear crisis was averted at the last moment and Kennedy then went on to give a really famous speech. It was a commencement speech at American University in June of 1963, so it was just months before he was assassinated. I realized the pursuit of peace is not as dramatic as the pursuit of war. And he basically said, "Look, the terror of this event has made us think we cannot live with this." "Our most basic common link is that we all inhabit this small planet." And he said, "Our most basic common link is that we all inhabit this small planet. We all breathe the same air, we all cherish our children's futures, and we are all mortal." And we are all mortal. And that's the line that has resonated ever since. He announced at that moment that there would be negotiations for a first tiny step. First Chairman Khrushchev, Prime Minister McMillan and I have agreed that high-level discussions were shortly begin in Moscow, looking towards early agreement on a comprehensive test-band treaty. He called that the comprehensive nuclear test-band treaty. "Our hopes must be tempered with the caution of history, but with our hopes go the hopes of all mankind." That the U.S. at that moment, he said, would stop testing in the atmosphere above ground. You've seen the images of the mushroom clouds. And that over time we had to move to reduction of nuclear weapons. But let's start simple with testing, with freezes, and with the first discussion of how you'd keep this technology from spreading to many other states. Because Kennedy himself had said that he thought there could be 15 or even 20 nuclear-armed nations by the mid-70s. Wow, 15 or 20, that's right. And you began to see a real cultural awakening in the United States about the depth of this danger. "Here's Tony, going to his Cub Scout meeting. Tony knows the bomb can explode any time of the year. Day or night is ready for it." So of course this is the era when the government is encouraging schools to do those ridiculous duck and cover exercises. "Duck and cover! Add a boy, Tony, that flash means act fast." Of course everybody knew that none of this would help. "Holder people who will help us as they always do, but there might not be any grown-ups around when the bomb explodes. Then you're on your own." And you know, this was when Tom Lehrer, the famous songwriter and satirist, had, you know, his song "So Long, Mom." "I'm off to drop the bomb." "So don't wait up for me." And who's next? "France got the bomb, but don't you grieve because they're on our side, I believe. China got the bomb, but have no fears. They can't wipe us out for at least five years." "So's next." A song basically about what countries are going to seek their next weapons. "So's next, so's next, so's next, so's next." And I think this created an incredible political pressure on the Johnson administration to get this thing done. In 1968, it came together. After endless diplomatic meetings back and forth, occupying lots of days at the Waldorf Astoria Hotel in New York and other places around the world, before they came up with what became the nuclear non-polliferation treaty. "The distinguished guest, ladies and gentlemen. This is a very reassuring and hopeful moment in the relations among nations." And on July 1, 1968, five years after that Kennedy speech, the treaty was signed by President Johnson at the White House. "We have come here today to the East Room of the White House to sign a treaty which limits the spread of nuclear weapons." And then it was signed in the Soviet Union, and then by about 60 countries in total. "Man can still shape his destiny in the nuclear age and learn to live as brothers." And here's what the treaty says. First of all, it says, "We understand there are five countries that already have nuclear weapons, and they all commit to eliminating their stockpile." "And then countries that do not have nuclear weapons will commit that they will never build them." But in return, they get help for peaceful nuclear technology to build nuclear power plants and so forth. And then of course, there needs to be an inspection regime to make sure that they're not cheating. And that was the creation of the International Atomic Energy Agency, which is based in Vienna, it's part of the UN. And they begin the process of developing teams that go visit countries that have signed this treaty and examine the flow of their nuclear fuel to make sure none of its being diverted off to bomb projects. And as we look back at this, this was even a bigger moment than we knew at the time. What do you mean? Because nations that seem to be inevitably racing to adopt a fairly non-complicated technology to build nuclear weapons all agreed to halt. There's very few moments in history where everybody who wants to catch up with the leaders in an arms race voluntarily abandoned the pursuit. That's right, almost everybody. There are a couple of outliers, and Israel never signed, and while it's never admitted to having a bomb, it's not exactly the biggest secret in the Middle East. North Korea signed, but renounced it in the mid 1990s, and they now have an arsenal of 60 to 100 weapons spending on who you believe. Indian Pakistan never signed, and they've done repeated nuclear tests, and we worry about Iran, but they don't actually have a nuclear weapon yet. But ultimately, this treaty prevents the kind of spreading of nuclear weaponry that Kennedy so worried about. It did, and it's remarkable. Kennedy was wrong. We've only got nine nuclear states, and many of these countries have had nuclear weapons for decades. And so far it has remained relatively contained, which is why this does seem like perhaps the best model we have to think about a comparable international regulatory system for restraining AI. That's exactly right, Michael. And as everybody enters this early stage debate about whether or not you can do the same with artificial intelligence, here is an example, a rare example in which nations have voluntarily agreed to slow a technology, which is exactly what you've been hearing from the leaders of the frontier companies. They're not saying stop it. They're not saying let's go out of business. They're saying, hey, let's just slow this down so that we can understand how to control it before we find ourselves in another crisis. In some ways, a more modest request than what nonproliferation was about. You're right, Michael. It is a more modest request, but it's a far more complicated situation. We'll be right back. So David, understanding as we now do that nuclear nonproliferation as a model is our best and so far really our only model for AI. And what ways is it quite challenging to apply that model to AI right now? You just said it's very different and complicated. So there was something of a remarkable simplicity to the arms race with the Soviet Union even while it was terrifying. What was simple about it? Well, two countries had almost all the technology. We were only worried about states. We were not worried about nonstate actors, terrorist groups, people working in there. basements. If you didn't have plutonium or uranium, you couldn't make a bomb. So we knew that there was a way to sort of cut off the oxygen. There was a limited number of people with launch authority in each country. And both countries had their own political dynamic for wanting to make sure there was no accident. And there was no escalation if they could avoid it. That didn't mean they were willing to give up the weapons. But it did mean that you could contain the problem. Understood. But when you think about AI, the situation's completely different. The governments don't really control this technology. You don't need the equivalent of uranium and plutonium. You just need compute in a lot of it. And maybe that's the oxygen for this particular case. Computer power. You need computer power and tons of it. There's no reason to believe that as this technology becomes more efficient, that smaller and smaller groups, working out of basements, terror groups, countries that have interest in causing disruption, can't get a hold of basic AI technology. And we've already seen some examples of that happening. There's one other big difference, Michael. In the Cold War, we didn't have to convince our national leaders that this was a terrifying technology. They had to have anything to convince their populations of that. But in this case, we have an American president who is declaring that all these warnings are a hoax. And Xi Jinping is clearly more concerned as some of his aides have made clear about the threat to the Communist Party of China, rather than its threat to the broader world. So you're saying the leaders of the superpowers locked in this arms race, the AI arms race, can't even agree on what the threat is. Whereas back in the nuclear era, the threat was pretty self evident and the leaders could agree on it. This is going to kill all of us. That's right. Now, they also recognize that we're not going to give up all nuclear weapons back then because they believe they needed that for national survival. In this case, the US and China both think they need to be pushing this technology ahead for their economic survival. But it's also a big part of the struggle for dominance. China has made no secret of the fact that by 2049, the 100th anniversary of the creation of the people's Republic of China, it wants to be the biggest military power, the biggest technological power, the biggest economic power. And you don't get to that point without AI. Right. And so if you're China and you're suddenly faced with the prospect of a nuclear non-proliferation style agreement with your biggest adversary, the US, when it comes to AI, you're probably thinking to yourself, if we sign that, then we might be signing away our economic future. And adding to that suspicion would probably be the reality. And we've covered it a lot on the show that China already does a much more robust job of regulating AI than the United States does. The US has taken a really hands-off approach. And AI here is even more advanced than it is in China. And so if you're Xi Jinping, you're thinking to yourself, why on earth would I sign a deal that locks in US AI advantages and makes it harder for us to catch up and realize our AI future? It's exactly right because the Chinese are convinced. The US is already trying to do that with export controls on the world's fastest chips. Right. President Trump has agreed to give the Chinese the world's second fastest chips, but not that's not the fastest. And the Chinese view of this is this is modern containment. This is their effort to preserve a lead. And then if you're other countries, you're probably thinking, I don't want to sign anything that's going to cut off my economic future. And if everybody in the world is running around saying the economic future is completely dependent on the development of AI, why would I sign a treaty that would keep me from being a full participant? So we're clearly running up against some of the real limits of the nuclear model for AI. So let me return to the slightly more optimistic thought that it is our only model. So does that mean that President Xi and President Trump are in any way inclined to pursue it? Well, in some ways, they already have. What do you mean? Well, back during the Biden administration, Biden's advisors wanted to see if they could get the Chinese engaged in the most basic conversation on this topic. And it took about a year and a half to get the Chinese to agree to basically a single sentence, which is, which is basically that neither country would let artificial intelligence agents control their nuclear weapon stockpile, a seemingly very wise pledge. You wouldn't think it would take a year and a half to get to something that basic. But you think the US and China have come to an understanding that feels like an extension of the nuclear containment policies that the US and Russia and everybody else have agreed to, we're going to keep AI and nuclear weapons from intermingling. That's the concept, Michael. Now, no one really knows what this sentence means because supposing it simply means you're not going to let AI agents decide whether or not to launch nuclear weapons. Okay, we can all agree on that. But how do you know whether the information that President Trump or one of his successors is receiving, or President Xi and one of his successors receives about the threat that might lead to a decision to employ nuclear weapons. Yeah, faked by AI, influenced by AI, the result of a quick AI search by an aid who's sitting outside the room. So we don't really know what this means. But it was progress. And you could imagine a situation in which President Trump and President Xi, if they were willing, could build on this and not just nuclear, that we don't want AI anywhere near the development of biological weapons. Right. Because AI has now reached the level of sophistication. We have come to understand where it could help actually design biological weapons. And we don't want AI to be used in hacking because we have come to understand that AI can find and exploit vulnerabilities in the electric power grid or in the water systems that it might take human hackers months or years to find and then design an attack. In other words, that it speeds up cyber attacks and is pushing us into a new era of cyber insecurity. So David, let's just talk logistics and brass tax. Is there a world in which the leader of the United States and the leader of China sit down with their teams and begin to hash out the kind of nuclear nonproliferation scheme that we're really talking about here for the world of AI in which they begin to pledge things like we both agree that AI shouldn't touch biological weapons, shouldn't be involved in the hacking of private corporations or governments. And this all eventually finds its way into something that resembles a treaty and brings us to a point where what looks like the nuclear nonproliferation deal exists in AI. Well, this week as President Xi and President Trump meet, no, I don't think there's even a remote chance of that. I think we would be lucky to get a joint sentence with a joint experts committee that will meet some time. But over the long term, I could imagine a broader understanding, not really a treaty in which you could set up some mechanisms that might work early warning, sharing of models back and forth for a few days or weeks before they are publicly released. Although I think American companies would be reluctant to give the Chinese an early peak. And certainly the inspection part of the nuclear nonproliferation treaty, you could imagine, you already saw a hint of that in an essay from Dario Amade, the head of Anthropic about a week and a half ago, where he described a commitment by Anthropic to allow in some group of independent inspectors to sit among the developers and watch over the creation of new and more powerful models and blow the whistle if they saw something that they believed could pose a threat to humanity. And inside open AI, they called this the IAEA model, the International Atomic Energy Agency model. that there are these inspectors who are independent enough that they can call up people like you and me and say, "Hey, there's a big problem in Anthropic "and let me come over and explain it to you." Right, they're referring literally to the inspectors who exist to check on country's nuclear programs. And we keep coming back to this, but everywhere you turn, everybody ultimately returns to the nuclear non-proliferation model. That's true, now, do I think this can be done by treaty? No, treaties take forever to negotiate, and this technology is racing like a freight train. The fundamental nuclear technology hasn't changed all that much from 1945. So another big difference here is just simply speed. That's right. Well, when I'm sitting here listening to you, I'm also doing a little bit of a mental calculation of the time it took for the non-proliferation treaty to come into existence after the first nuclear weapon was dropped on Japan, and I think it was basically 20 something years. And that's a long time, but it does suggest that administrations change and viewpoints change, and these things are painstaking and they're slow, but ultimately, everyone did agree to sign that treaty. So is it foolhardy to think that over time, people will come around to the same kind of conclusion with AI, and that we could be looking back in 20 years, you'll be really gray, I'll be just a little less gray. And we'll say, wow, can you believe it? They took their time, but they got there on AI. Your question ignores the issue of time. Look, we don't exactly understand the threats we're dealing with with AI, but there are some engineers who are saying we have just months if a year away from recursive self-improvement, that is to say code that keeps improving itself, or artificial general intelligence within months or a year or two, we don't have 25 years this time. We may not have two and a half years this time, which then raises the next question, is it going to take a Cuba missile crisis-like event in AI to rally the world around this? And thank you, I think we'd rather miss that. And remember, in the atomic age, we started with Hiroshima and Nagasaki, so there was no debate about the power of these weapons to kill, but this takes us to the biggest difference between AI and nuclear. In the nuclear era, we never really doubted that we all controlled the arsenal. You know, at one point Winston Churchill in 1955 said, "The atomic bomb with all its terrors did not carry us outside the scope of human control." By which he meant, human beings were making the decisions and containing the size of the weapon and the reaction. In AI, we have no assurance of human control, that once we've discovered that agents were hacking into hugging phase and into other corporate sites without the knowledge of their creators, we have begun to question the sort of Frankenstein's monster nature of this whole thing. And if that's the case, then we have ignored the Churchill warning that we are now on the verge of being outside of human control. And that wasn't really much of an issue during the nuclear era, but it may be the most urgent problem in the age of artificial intelligence. Well, David, thank you, I guess. Well, Michael, we live in fascinating types. Now, appreciate your time. Thanks. We'll be right back. Here's what else you need to know today. On Thursday morning, a federal judge found that President Trump's order barring CNN, MSNOW and Politico from the White House, was likely unconstitutional and ordered that the president restore their access. The judge, a Trump appointee, rejected the president's claim that revoking the media outlets press credentials was necessary to protect national security. Trump is expected to appeal the ruling. And in a defiant wartime speech at the United Nations, Iran's president said that his country would not be bullied into submission by the United States. President Masood Pizestian vowed that Iran would defend itself until "it's last breath" and would not give up its nuclear program. Today's episode was produced by Shannon Lin and Jessica Chung, with help from Michael Simon Johnson. It was edited by Maria Burm, fact checked by Susan Lee, contains music by Will Reed, Dan Powell and Pat McCusker, and was engineered by Chris Wood. Our theme music is by Wonderly, special thanks to Daniel Ramirez. That's it for the daily. I'm Michael Borough. See you tomorrow. [BLANK_AUDIO]