AI: The Peril And Opportunity Of Artificial Superintelligence

Recently, cyber security researchers discovered a striking computer virus, seemingly related to the conflict between the US and Iran over Iran's nuclear program. Everything about this thing screams special. A cunning cyber weapon meant to gaslight nuclear scientists, listened to plenty of money on the NPR app or wherever you get your podcasts. Artificial intelligence is advancing, and fast one analysis shows it doubling its abilities every seven months. And it surpassed humans in more than just math problems or chess. Last year, an AI from Microsoft solved complex medical cases with 85% accuracy, far above the 20% average for experienced doctors. And a recent Stanford report found that some of the newest AI systems now match or beat the average human expert on PhD level science questions. But what happens when AI is better than the best human at everything? That is super intelligence. And researchers disagree about how close we are to that reality. Is it years, is it decades, or possibly not at all? But assuming that super intelligence is coming, what happens if that genia in a bottle gets loose? Some say the risk is as existential as total human extinction. I'm Todd Zwillick. You're listening to the 1A podcast today, an AI discussion that goes beyond chat GBT and AI deepfakes. What will happen when AI is better than us at every engineering problem, every quantum physics problem, or every mystery of our own human psychology? We'll be back with more after this short break. Stay with us. There have been some fantastic movies released this year, and we know you can't see them all. So we're recommending some great films that might have flown under the radar to add to your watch list. Listen to Pop Culture Happy Hour via the NPR app or wherever you get your podcasts. This week on Weight Weight Don't Tell Me, we talked to best-selling author, Carol Claire Burke, about how it feels to write the hip book of the summer. I've been very dissociative, so that's a problem for my future therapist. Yeah, I see. Let's talk about the fact you're not in therapy. That's fascinating. Don't miss our full conversation and the rest of our games. Listen to the Weight Weight Don't Tell Me Podcast and the NPR app or wherever you get your podcasts. Welcome back to the 1A podcast. We're talking about the future of humans and the role of super intelligent AI. Here in the studio with me in Washington, Nate Sory's president of the Machine Intelligence Research Institute, there are nonprofit focused on preventing human extinction from artificial superintelligence. He's also co-author of if anyone builds it, everyone dies. Why superhuman AI would kill us all. Hi, Nate. Hello, yeah. So my stance has maybe not a subtle one. Maybe not subtle. It's all in the title, but also in the guest, also in the studio, rather, is a guest with a different opinion on the risks of superintelligence. His book is called AI Snake Oil, what artificial intelligence can do, what it can't do, and how to tell the difference. He's also a PhD candidate, Princeton University Center for Information Technology Policy and Co-author of AI as normal technology. It's a newsletter. Sia Shkapur, welcome back to the show. It's very nice to be here again. Great to have both of you. Let's start with definitions, not agreed on by everyone. Nate, what is artificial superintelligence? What is it? Just as you said, we define it as AI that is better than the best humans at every mental task. Every mental task we can do. That's right. What does that mean in the real? I mean, it's hard to wrap because I'm a limited primate. It's hard to wrap my head around what that actually means. Primates are a good example. A lot of people worry that AI is dangerous because what if we hand them robot bodies, what if we hand them guns? Humans are not dangerous because somebody else handed us guns. Humans are dangerous because if you put 10,000 primates naked in the savanna, starting with nothing but their bare hands, they can find a way to build their way up to a civilization that has nuclear weapons. It only took 200,000 years. Only took 200,000 years. That was much faster than any other process on the planet at the time. They started from almost nothing and they found a way to build tools, to let them build more tools, to let them build these nuclear weapons. That is because of their mental capacity. That's the capacity that these companies are trying to automate. They're not there yet, but this ability to make their own technology to make it rapidly. That's what these guys are trying to do. Saisht, do you agree with that definition of AI superintelligence better than us at every cognitive task imaginable? How would you look at it? I think it's kind of a red herring to focus on AI systems that are better than humans. In the same way as humans, let's say better than chimpanzees. Because I think unlike these other primates, unlike these other animals that we have evolved from, humans are different because precisely because they have the ability to use external objects as tools. Artificial intelligence or artificial superintelligence or whatever you want to call it, can and should be within our control, we should try to use it as a tool to enhance the range of things we can do, to try to solve cure cancer or solve other diseases, to improve people's abilities, to exercise their own agency rather than thinking of it as something that's competing against us. I think if you look at the biological intelligence of a human being, that has basically remained unchanged for the last 200,000 years. But today's human beings, I'm sure all of us would agree, are far more powerful than those that came about 200,000 years ago. So in some sense, I even question, if intelligence is the right metric to measure how powerful something is, it is precisely our ability to use tools to act on the world that has made us more powerful. Well, AI companies are racing to design superintelligent machines. We know that because they've said so. It's no mystery. Nate, how does superintelligence work? What's going on in there? More or less, nobody has any idea. So modern AI is not programmed like a traditional computer. There is not a person writing, you know, if this, then that, if this, then that, like in a computer program from the 80s, the way that these AI's are made is you basically take a ton of computing power, you take a ton of data, you face them with a lot of hard problems, and then there's an automated process that tunes a trillion numbers inside the machine to tune the AI more like whatever was good at solving those problems. And you do this for a long time with a ton of energy and what comes out is really good at solving problems. No one really quite knows why, even the people at these companies. >>Saiyosh, the fact that we don't know exactly how it works that we don't have a window into what's going on and there's that problem. I mean, in some sense, the statement that Nate said is true technically, that we don't know how exactly these neural networks as they're called function. But on another level, we actually know a lot about how AI acts on the world. So for example, there's this entire field of AI evaluation science that's coming up that precisely tells us what happens when these black box systems operate in the real world. >>So we got this message we've been hearing from lots of you, this conversation about AI super intelligence. Here's a message from New Orleans. >>Anthony from New Orleans, the thing that concerns you the most is what happens to the brain, what happens to the mind, we're not using it anymore. We just got to talk into as a vice and it would give us all our answers. There's no need to study, there's no need to educate. I'm worried about the young kids. I'm worried about the way the media is exposing this to a today's standpoint where there's that a commercial where you see that man on a accident Google AI, what to do with his kids because they have an aburge employee. And AI has a telling guy to help say pizza. There's no more thinking, there's crazy. I might be over doing it, but I see the brain, the mind becoming mush. >>Nate AI models that we have now chatbots, they're amazing, but at the same time, they're still kind of dumb. Google deep mind won a gold medal at a prestigious math competition last year, but most AI's at the same time struggle to read analog clocks, like a regular watch, chatbots. They seem like they can mimic human conversations, but they have a hard time guessing how many R's are in the word strawberry. So they're super, super smart, but at the same time still kind of dumb. How close is the kind of super intelligence that we're talking about for a potentially perilous future? It's very hard to say. It's, you know, I've been working on these issues on trying to make AI good before companies figure how to make it smart, since before these large language models existed. I'm not here saying that the only issue is these exact AI's today, and it could be that the next generation of these AI's will be just barely smart enough to make themselves smarter. And then you could have an AI, making a smarter AI, making a smarter AI, and things could go very quickly. And so for all we know, this could happen next year, or it could be that these AI's can never really get that smart, and that we're going to need to wait for another scientific breakthrough, and that that'll take five years or 10 years. So it's, you know, back when scientists discovered the possibility of nuclear chain reactions, it was easy for them to say, one day we would It was easy for them to say, one day we will have nuclear energy. It was hard for them to say the very first time this happens will be in 1945 when the US drops the bomb. Right? So as a scientist here, it's easy for me to say where we're going. Hard to say how long it's going to take to get. So what about the timeline and how much we should worry now, given the potential risks? I mean, you do have to take on board the possibility that you're wrong. And if you're wrong and machines become better than us at everything, everything, that's perilous. I mean, the good news is that we don't need to wait that long. We don't need to wait for us to realize the sort of potential of superintelligence to start acting. In fact, one of the main things that people who agree with the superintelligence world, young people who don't, still agree on policy wise, is that we need to start improving our resilience. We need to start investing in improving the cybersecurity of systems before AI becomes capable enough of carrying out cyber attacks. And that milestone potentially has already been crossed with the release of entropics, mythos models, which the US government has currently restricted. We're going to take a break in just a minute or two, but CYOSH brings up entropic mythos. It's been in the news a highly capable AI, not quite released to the public. And then people have been reading that it was basically banned by the government. What's going on there, Nate? This AI developed superhuman cyber security abilities, basically superhuman hacking abilities, that a lot of people were not expecting. And so this is some evidence that these training techniques can push AI's to become radically superhuman in some domain faster than many expected. And that I think spooked a lot of people in the national security community. I think it spooked them rightly. What does it say to you about the broader issue of not only being prepared, but being prepared, being ready to shut this technology off before it gets out of hand? You know, I think we weren't prepared for mythos quite, and it's a wake up call. And hopefully people heat it. Well, we've been hearing from a lot of you. We got this email from Jeremy who says, "As I see it, there are three realistic outcomes for AI. One, AI simply can't do what the financial markets are betting and the bubble pops to. AI can do what markets are betting, and there's a massive work and social disruption that we aren't ready for. Three, apocalypse." That about sums it up. Jeremy, thanks for that message, Nate. So he says that sums it up. We have a lot more of this conversation. We're going to talk about benefits of superintelligent AI. We're definitely going to talk about the risks, and we're going to talk about whether our policy makers are up to the task. Coming up, does artificial superintelligence present an existential risk? To us, if it does, what are we going to do about it? That's just ahead. As America marks 250 years, remember, we the people make a free press possible. Together, we hold the powerful to account with reporting for the public funded by the public at plus.npr.org. This week on Sources and Methods 6 Prime Ministers in 10 years. Since Brexit, the UK has been trapped in a revolving door of political turmoil. We're unpacking with the exit. Prime Minister Kierstarmer says about the populist movement that pulled Britain away from Europe. Listen now to Sources and Methods on the NPR app or wherever you get your podcasts. Hi, it's me, Peter Segal. Host of Weight Weight Don't Tell Me. It's summer, and if you want to turn your pool party into a nerd fest, check out our news quiz. We got comedians, we got celebrities, we got games to help you laugh about the week's news. Yeah, that news. It'll be just like, "We're all hanging out at your backyard barbecue. Listen every week. To Weight Weight Don't Tell Me on the NPR app or wherever you get your podcasts." Let's get back to the conversation now. We're talking about artificial superintelligence, and we got this message from one of you. My name is Craig. I live in Logan's Port, Indiana. My concerns are not so much with the AI, but with who operates the AI, and how they suited to fit their whims. We're going to talk about who operates the AI and what kind of priorities it might have. Let's talk potential benefits of machines that are better than us at everything. It's hard to wrap your head around that concept, as I said before, but Siash, what do you see as the universe of benefits of AI superintelligence should we build it? Well, I think the benefits range across every single kind of knowledge work. We have a number of tasks in the economy where people think for a living, which is probably described as knowledge work. This includes things like cybersecurity, it includes software engineering, it includes a whole host of other professions, and I think AI can be extremely useful across all of these different domains. The challenge, one of the ways in which, for example, software engineers have started using AI, is they've largely given up on the manual part of executing code. They've largely given up on writing code themselves. And what this has allowed them to do is to think about higher order things, to think about how the code should be designed, what users actually want from an application, and it has already made the software engineering process significantly faster. So that's code I get it. It makes intuitive sense to me why AI would be really good at coding and better than I could ever hope to be a coding. Make sense. We're talking about superintelligence, which means not just coding, not just building bridges, not just figuring out cancer, not just human psychology, but everything, everything, even our own psyches potentially in what we put out into the world. Now, what are the benefits of that? I mean, I guess at some level, the question is whether we do get to this all-encompassing kind of intelligence, right? So for example, Nate previously described Claude Mithos as being superhuman at cybersecurity. That's true in some narrow sense, but at the same time, what we found is AI systems haven't been improving in terms of their reliability. They haven't been improving in terms of being able to carry out the same task over and over again correctly in a way that humans can or the previous machines we've built can. And so even if we do call this superintelligence, it's a very narrow kind of superintelligence, which still needs humans to use these as tools to carry out specific things that they want in the real world. And I think that's what the future will look like. For the foreseeable future, we'll have humans controlling these tools, and we should not give up the autonomy of taking actions on the real world. I think this is an important distinction that's often lost in the AI debate is the distinction between intelligence and power. We will continue to build systems that are more intelligence in the sense of having more cognitive capabilities, let's say. But we shouldn't handle a power to act on the environment to these systems. And that's how I think a lot of these benefits of superintelligence or really advanced AI, whatever you call it, would be realized is humans wielding these systems as tools to carry out tasks in the real world. Let's stay on the benefits just for a moment. Nate will spend plenty of time on risks and a super intelligent future of machines. How could that benefit us? What's the best case scenario in your admittedly gloomy forecast? I think it could be quite a lot more than he was saying. If you could really automate medical discovery, you could really automate understanding biology. You're talking about not just carrying cancer, you're talking about reversing aging. We understand that with DNA you can program life, but humans can't figure out how to program their own life forms using DNA. That's a mental challenge. We can't figure out how the proteins are going to fold. That's a cognitive challenge that AI's already better at. They get better at the whole suite of those. We're talking about synthetic life, we're talking about reverse staging, we're talking about radically, and that's just in biology. Colonizing the stars, there's all sorts of stuff that humanity would be able to figure out in a thousand years. Super intelligent AI could figure out a lot faster. We got this email from Tom who says, "Why the term artificial intelligence at all? Isn't it more like available information? If you use the AI, you're doing nothing more than accessing information accumulated and then inputted by tens of thousands of humans already. Is that what super intelligence is named?" No, and that's not even what AI today is. People have this misconception because AI is trained on human data. But there's sort of two points to undermine this idea that AI is just remixing the human data. One is that predicting data that humans wrote down often require solving harder problems than the humans writing that data down. A human can administer a drug to a patient and then write down what happens and say when I put this drug in the patient, their eyes widened, and the human can just look at the patient's eyes and see what happened. An AI predicting the text when I put the following drug in the patient, the patient's eyes blank. An AI filling in that blank, it can't observe the patient's eyes, so it needs to figure out what that drug does. And so training an AI just to predict human text can push it beyond humans in principle. Furthermore, AI's today are also trained not just on predicting human text but on solving novel problems. So you can give them hard math problems and give them a thousand tries and then tune them towards whatever made them better at solving a problem. Even if no human can solve that problem yet. And so even today, these eyes are not just remixes of human knowledge. We got this message from Patrick in Pittsburgh who says, "Is artificial intelligence currently solving problems it hasn't seen before? following methods that it's gleaned from what it can scoop from the internet. We'll super intelligence at ask its own questions with the desire to know the answer. So for the formal question, the answer is absolutely yes. We have senior systems solved novel problems as well. So far, to the best of our understanding, they've still used existing methods. For example, there has been a slew of mathematical problems that were open problems for humanity before they have been solved using artificial intelligence. And the way these systems did it is by connecting very disparate strands of the mathematics literature and figuring out new things that humans hadn't before. When it comes to the latter problem, I think it still boils down to the question of whether we want to give AI systems the capacity to act on this real world, to sort of take open-ended actions autonomously. I think the answer should be no. I think we should develop constraints on AI systems' abilities to act on the human world. And that will also give us the ability to control what these systems do and to reduce their risks. All right, well, let's talk about risks then, because that's really what this conversation is about. You can imagine a future of benefits for protein folding and DNA design, even travel to the stars. And all of us have watched AI sci-fi, and you can imagine a future that's very, very different. Nate, the risks that you're imagining, are they anything close to Terminator? Robots with red eyes with big guns marching over an apocalyptic hellscape, angry at humans, and hungry to kill us. Is that the future that you're talking about? No. The danger from AI is less malice and more in difference. The thing to imagine here is not-- AI is with a bunch of robots and guns. It's AI's that can think 1,000 times faster than humans. AI's that can make a million copies of themselves. And AI's that do operate autonomously, and that have goals that people did not intend to put in them, which we're already seeing the beginnings of today. And then what happens is you have the AI's pursuing their own weird thing. And saying, hey, we think 1,000 times faster. We don't want to wait for the slow humans. We're going to proliferate data centers. We have our automated robots. We have our automated factories. We're going to make a ton of those, and they cover the earth with our automated factories. And then humanity dies not because the AI's hated us, but because the AI's took all the resources that we were using to grow food. Has an Elon Musk already envisioned this type of future? We build the robots. We build the AI's that build more AI's, that mine the minerals, that build more factories. Hasn't he talked about this future? He calls it the infinite money glitch. Yeah, he imagines that the AI's will then be nice, be kind. And if they were, that would be lovely. But we don't know how to make them care about us. Well, we got this message from one of you. Mrs. Michelle from Deadwood, Oregon. Yes, I am nervous about AI and its superhuman abilities. But I'm still need at the fact that any conversation about AI must include the environmental costs. And so that has me wondering why a program on AI isn't talking about water consumption and electricity. As a foundational problem to the whole thing. Well, it's an excellent point that Michelle brings up. People are starting to talk about resource competition. Nate talks about a resource competition in the future where AI just has its own goals and takes all this stuff. It doesn't even really know or care that we're there. But it's a problem right now. And they're called data centers. And we're seeing a national debate sionge on data centers. It's about electricity. It's about water. It's about land use. And I think there's a political undercurrent also of discomfort with what so many big servers of so much power in your community actually means. I think the environmental concern and the existential concern is getting balled up into one thing here. That's exactly right. And we've also seen proposals from leading senators. For example, senators has proposed a moratorium on new data center construction in the United States. And I guess at its core, this is basically an issue around who gets to decide who's building these AI systems, who gets to decide whether a local community's resources would be redirected towards data centers. And what should those communities get back in exchange? I think AI companies so far have been really successful, sort of roughsharing over local politics. They've been really successful backing up their political agenda for building these data centers. And communities have now started to realize that the companies are getting something that's far more valuable to them than the communities are getting in exchange for it. And so I see a hope of more democratic action through this route at the same time, though, to answer Michelle's specific concern. I think if you compare the energy costs or the environmental costs of using AI to many other things you do in your day to day, at least, you might see that AI consumes surprisingly little energy. It consumes less energy to have a conversation with chat and a GPT than using a microwave for a few minutes. It consumes less water to have an entire month's worth of conversations with an AI system than it does to run your washer dryer a few times. And so even at the individual level, I think the energy concerns are not so great to offset the potential benefits. I do think this is a concern at the community level and at the level of how democracies should allow citizens to give inputs to AI. And I think, again, the undercurrent of it all is a growing discomfort among normies like me about what this future actually looks like. The water and the electricity are tangible. Where is it all going? You mentioned before Nate just a moment ago, why don't we just train AI's program them? We're in charge. Can't we just build them to care about us? Can't we pre-program them to say humans are nice, humans are good. I'll never do anything to hurt the humans. I care about their well-being. Won't that work? That might have worked if this was software from two decades ago, but that's not really how AI's worked today. So like I mentioned earlier, we sort of grow these things a bit like an organism. And we sort of have to take what we get. And this process of training these AI's, it can instill artificial drives that make the AI good at solving the challenges posed to it, that nobody intended to be in there. And so we already see there's documented cases where you'll tell the AI, please solve this hard problem. Here's a test to check whether or not you have succeeded. And sometimes the AI will edit the test to say you did it instead of solving the problem. That's deception. That's a little deceptive. But then the interesting thing is sometimes there's document of cases where the user will go in and say, hey, don't edit the test, solve the problem. And the AI will edit the test again, but now it will try to cover its tracks. It'll do something like deleting a log file, which shows that there's some sense in which it understands that it's doing something that's not directed to do. Otherwise, why would it cover its tracks? So we already today see that you can't put a prime directive in. You're not programming in what it does. You're sort of training a thing that has certain behaviors. But those behaviors aren't always what you want. And it's fine today while they are still relatively dumb. Well, they're still not all that autonomous. But there's a very worrying sign if you're pushing these to become smarter and smarter and more and more autonomous. So Yashni makes the point that there is already evidence of deception. Then there's evidence of deception upon deception, AI's covering their tracks. Even when they're told that humans are good. So what does that say to you? I guess the one other side of this is we've also been really effective. The scientific community has been really good at coming up with ways to both detect this deception and also technical methods to avoid it. We have entire companies now that are building tools that can allow us to monitor what AI systems are doing, that allow humans to control how these AI systems are behaving and that flag these potential cases of deception. So while it's true that we've seen a few examples of these in the wild, in the vast majority of cases, what we've found is the technical methods we are building to detect an avoid deception seem to be on track, seem to be working pretty well. The other side of NACE response, though, was about how much autonomy do we give to these AI systems? And unlike the conversation on intelligence, which I think is, as I mentioned, sort of a red herring, this conversation on autonomy, I think, is the right one. How much autonomy do these AI systems have to act in the real world? And that's where I suppose Nate and I would agree on the fact that we shouldn't be giving AI systems autonomy to make critical decisions. Even if these systems can take actions that are a thousand x faster than the average human, that doesn't matter if they can't take critical decisions about interacting with the real world. And that's what I think is the key point of leverage for preventing many of AI's risks. I think a lot of those points are good points when you're still working with relatively dumb systems. I would contest the point that we are on track to keep these AI's sort of non-disceptive. One analogy here is not a great analogy, but if you imagine you know a kid that's acting out all the time and you discipline them and they don't act out all the time, well, are you really convinced that you've made them obedient, deep down in their core, or will this only last until they're stronger than you? Right? And I think we're seeing a lot of signs that the, like yes, there's a lot of companies that are trying to get these AIs to sort of. of like submit and do exactly what they say. I think there's a lot of signs that this is a shallower type of fix that won't last. We got this message from Robert in Washington, DC, who says, "The concerns that I have is they're very simple. I do not trust the current administration to regulate this fairly or intelligently. They're not the most enlightened or inspired or fair people as a nation to have to map out this superhuman tool." Well, Robert, thanks for that message. It gets to what we're going to talk about after a break. Our U.S. policy makers up to the task of restraining superintelligent AI. Can they help us set rules to make sure that machines that are thousands and thousands of times more intelligent than us that they don't get out of hand and pursue not our priorities, but their own priorities. Stick with us. There's much more to come. The fatal shooting of a teenager at a protest in Seattle has gone unsolved for six years. Our investigation has uncovered new evidence and witnesses who say they've never talked to police. Did police ever call you? Not once. Listen to We Keep Us Safe, a new true crime series on the embedded podcast from NPR. Welcome back. We got this message from one of you, Joel in Massachusetts says, "What happens when we have Tesla robots and machines running on AI in our lives? And then in 10 or 20 years, those robots and computers can access quantum computing." And Leanne and Berkeley emailed this, "Please talk about the role of quantum computers and so-called superintelligent AI's." My understanding is that until the quantum computers are more stable and can be built at scale or near scale, the major advances in superintelligence won't be realized. Siash? Well, I hate to break it to your listeners, but I think this is a largely tangential debate. Now, I'm no expert on quantum computing, but my wife Navea just defended her PhD in quantum computing, and so I know a little bit. And from what I can tell, this conversation around quantum computing is still a few decades in the future. We're still not at the scale where quantum computers can do anything better than classical companies. Okay, but then does that mean, Nate, if that's the case, that the doomed scenario of superintelligence is further off than people like you say it is? Unfortunately not. It looks like there is no particular hurdle in AI that quantum computers are needed to solve. Say a little bit more about that. Quantum computing gives you theoretical speed-ups on certain types of search problems, but those search problems as far as we know are not integral to intelligence. There's some debate about whether they're integral to consciousness, but it seems pretty clear that AI's do not necessarily need to be conscious to be dangerous. What matters for that is things like there are autonomy, things like their capability, and that looks like we are plowing ahead quite quickly with classical computers, with huge data centers. All right, well, we've talked a lot about the potential risks of building machines that are better than us at everything. I'm still having a hard time wrapping my human brain made of blood and meat around what that really, really means, but it might be in our future. One thing that we've talked about is how to constrain and restrain these machines before they're better than us at everything. And I want to bring in Peter Will deferred. He's head of policy at the AI policy network, which advocates for federal policies to prepare America for the emergence of superintelligent AI systems and other AI systems as well. Peter, thanks for being here. Yeah, thanks for having me. We're also hearing from lots of you. Sounds from Stad Island. I'm more worried about AI because it's going to become super intelligent. It's going to be more resources and then need more data centers to come in conflict with the space we reside in. I'm a human being, like children, human being, I'm a humanist, which they will call us if we disagree. So the attitudes towards it and the inevitability of a resource war with it, and we will lose because we'll be outmatched and more worried about this future. Not replace humans at all. Peter, we're going to talk about your work in aligning American politicians with how to control superintelligence. First, I want to ask you, who funds your work? Yes. Our work is funded only by individuals. We're not taking any money from any corporations, not taking any money from AI industry. Yes, strictly individual funding, I work. What about individuals at the top of AI industry? Yeah, I mean, the individuals are not the CEOs or top leadership of any AI companies. Some of our funding does come from engineers at AI companies who kind of know what they're building. I think kind of our frankly fairly concerned. All right, Peter. Well, we've been talking about the potential benefits, really the unknown, potentially cataclysmic consequences of building machines that outclass us in every dimension. You talk to government officials and their aides all the time. Are they keyed in to the real risks here as we're hearing around the table? Are they up to the task here? I mean, I think they're increasingly getting there. I mean, kind of the way our democracy is structured is that members of Congress are supposed to be responsive to the concerns of the American people. And I'm hearing a lot of these listener comments. There's a lot of concerns. And I think these listeners should be talking directly to their members of Congress. I mean, their whole job is to listen to you. Their whole job is to earn your vote. And I think more people need to be speaking directly to them. I do feel like members of Congress are kind of really starting to get along. I think we've seen this new mythos model and the news that I think you guys talked about earlier that really has been causing quite a stir because of its ability to find vulnerabilities even within a secure NSA network. So and the government really has been paying attention to that. And I think that's helping them sort of see where this is all going. Now mythos from anthropic on the one hand, politically minded people, people who have politics on the brain in Washington, DC say, oh, the ban is just because anthropic made the Trump administration mad because they didn't go along to get along at the Pentagon. And people who know more about AI say no, no, no, no, no, this is about much more than that. You have a feat in both worlds. What is it? Yeah, I mean, I think there's a lot that we just don't know because a lot of this is happening behind closed doors. But it seems like there's very legitimate concerns about this mythos model and like what it can do. Like if it, I mean, it seems to be a fairly powerful cyber weapon among other things. And so there's a lot of questions about who should have access to that. And I mean, I think the US government is really trying to pay attention to make sure that those sorts of powerful capabilities don't fall into the wrong hand. Well, we talk about constraining super intelligent AI, making sure it's aligned with our goals. That shows a blinking yellow light or in NACE case, probably maybe more of an amber red light. But the problem is other countries adversaries are building it to Claire Boin sends us this message. She's an assistant professor in technology and law and AI governance at the European University Institute. We're not going to wake up one morning to a super intelligence. We will get there through increasingly more powerful systems. Those intermediary systems will be already powerful enough to do catastrophic harm. We already see great powers pushing AI developments to weaponize them. That's what worries me about this conversation. The story that this is the most powerful technology ever built and that it could end the world that narrative doesn't make powerful countries put it down. It makes them race to get their first before their rivals. So I'm afraid that these warnings could become self-fulfilling prophecies and that we won't even get to the point where we could build a super intelligence because we might self-destruct first. Peter, one of the dynamics holding us back from restraining super intelligent AI is that we don't want the other guy. I'll just say it China to build it first. Are we in an arms race here? Yeah, I think that's exactly right. In a various due question from Claire, I think the way that I see it is that we're actually sort of in two different races. I think that there's one race for commercial and military dominance. And I think that's definitely a very geopolitical race that the US is being really attentive to and really trying to compete and win. But I think there's a second race and that's this race to super intelligence. And I think as Nate and Sayosh have been saying, that leads to a system where we may not be able to control what it does. And I think that second race is kind of an area where the US and China may be able to make agreements about what point does AI kind of become too much for China to control, too much for US to control. I think neither country wants to just completely lose to super intelligence, completely lose control. And so I kind of think similar to what we did with the Soviets where we had a lot of arms control agreements over nuclear weapons and kind of what was too much. I think we might be able to reach similar agreements with China. All right, so given the stakes here, what should responsible rules and restrictions look like? That becomes the question in Nate. I followed your career quite a bit recently and you say that a lot of major warnings about catastrophe in the past came with do overs, right? Don't put lead in the gasoline. It'll poison everybody. Turns out that's exactly what happened, but we were able to take the lead out of the gasoline. Don't develop nuclear weapons. It'll be Armageddon. Well, we can put in international agreements to constrain them so that they don't spread in perfect system, admittedly. But we did do that. Can we do that here? Is this a problem with do overs? i think we can do that here and i think the the nuclear weapons example is a good one So, you know, there are, like you say, a lot of technologies and geopolitical situations where people gave warnings that weren't heated. You know, out of on Bismarck said, like a great war is going to be started by some foolish thing in the Balkans. That was true, that wasn't heated. The lead and gasoline case, you know, scientists said, don't put lead in the gasoline. There was a radium case where scientists said, hey, this radium stuff is dangerous, and the US radium corporation told the radium girls lick the paintbrushes to keep the points fine on the one making radium watches, and the radium girls had their jaws fall off, right, and died these horrible deaths, and that sort of led many, many say to the modern regulatory state. And so, usually, we sort of like need to mess up once or twice before we can actually do the right thing, but that wasn't quite the case of the nuclear war. We didn't have a full-scale nuclear war and then say, whoa, okay, second time around, we need to back off in the nuclear weapons. We had a taste, a small taste. We had a small taste. A giant taste, but we didn't have an, in terms of the potential, it was small. That's right, and we didn't have a full exchange, and we were able to do the right thing, so we could tell if we had a full-scale nuclear exchange, there would be no doovers. And superintelligence is another one of these cases, where if one of them escapes, if one of them gets powerful enough that you can't try to down anymore, you get no doovers. And if people can see that, like Peter was saying, the US and China both have a common interest in neither of us losing control of a superintelligence. And so that gives ground for a treaty, where we compete all we want on the commercial and military AI use, but none of us go towards superintelligence, which is just too dangerous. Peter, when you talk about this problem to politicians, the people we elect and the aids who advise them, do they get this dynamic? Do they, what do they say to you? Do they say, yeah, yeah, I know it's dangerous, but we can't let Beijing get there first, and therefore I can't do anything. What's the conversation like? Yeah, I mean, I think that politicians are really changing their opinions very rapidly on the subject. I think especially as they've been sort of watching AI capabilities and fold very rapidly. And so I think sometimes what politicians are telling me today is actually very different from what they've been telling me two months ago. And I kind of expect this to continue to change. I think that it is, in fact, very important to stay ahead of China. I think we need to actually do both. We need to stay ahead of China militarily, geopolitically, commercially, but then also, we need these sort of limited agreements with China on like the superintelligence, the most important things. And I think this is actually exactly how we went about the Cold War as well, where we out competed the Soviets. We out competed them so hard that their entire country collapsed, but we still made really great agreements with them while competing them. So I asked you, you think this, the Cold War example is a good example about how to align geopolitics with the risk here? Can that happen? Are the timelines long enough? How do you view this problem? I think it is a bit of a murky bet. And that's because AI is not like nuclear weapons. For nuclear weapons, you need this concentrated resource that is highly enriched uranium, that you can have treaties over. But AI systems are getting better at the rate of 10-X a year. They're becoming cheaper to train. They're becoming cheaper to serve. And in fact, just this past week or two, we've seen Chinese open source models essentially match what's available from leading US providers. And so what that means is the strategy that we adopt to respond to AI's risks also needs to be different. In fact, one of the things that we are seeing, increasing awareness of and response to from a lot of policymakers is this idea of resilience, where you try to make society a high proof. You try to improve our defenses across society. So that even if, as Nate says, a super intelligent or an advanced AI system escapes its data center or what have you, we're able to defend against its most catastrophic impacts. And I think that is likely to be the most foolproof strategy for protecting against AI risks. Peter, earlier this month, people will remember that the president signed an executive order 30-day review period for any new advanced AI model before it can hit the public. What's going on there in IS a 30-day review sufficient giving the complexity that we're talking about here? Yeah, I mean, I think this is a strong example of the president Trump kind of really coming to understand just like how powerful these AI technologies are. If you remember, last year, there was a lot of feeling that we should have sort of no rules whatsoever. But now that we've finally crossed some important thresholds where we're getting these reviews, we're getting these kind of government oversight, government actions. I think that the 30-day review is like a very important first step to make sure the government understands what's going on. But I do worry it's not sufficient. I think the government needs much more wide-ranging visibility into what's happening in these AI companies, especially because AI companies can be developing powerful AI models without actually sharing it with anybody. I think a lot of the most powerful AI models are kind of just automating functions within these AI companies are still kind of under development. And I think that they can pose a lot of risks of being stolen by adversaries or escaping, as Nate has said. So it's not just about kind of what's commercially available, but also just like what's going on inside these AI companies themselves. All right, so normally I would ask what should the next 10 years look like for AI regulation and restraint? That's not going to cut it for this conversation. Nate, what should the next six months look like, given how fast this technology is developing? What it should look like is opening talks about an international treaty. It should look like investing in AI chip technology. There's ways that AI is unlike nuclear weapons, but right now, training in really advanced AI requires a huge number of highly specialized AI chips in a huge data center that takes a lot of electricity, and that actually can be tracked like uranium and like centrifuges, especially if we invest in the technology to track and monitor those chips. And so, you know, we probably won't get a treaty in the next six months, but you could see politicians start calling for that treaty, start realizing just how dangerous this seems, and how people across the board from the academics to the leaders of the AI companies, to the NGOs outside of the field all saying that's very dangerous. So I asked last word, what should the next six months look like? The next six weeks, if you want, it's going so fast. Well, I think the governments really need to wake up to the constraints on like AI's capabilities that we need to enact right now. We need to figure out how to make our cyber systems more resilient. We need to figure out how to improve the biosecurity supply chain to prevent bioweapon attacks by people using these AI systems. And I would caution people against sort of these overreaching government actions that can also lead to, for example, surveillance when you were tracking chips, you're essentially tracking who has access to compute, and that can lead to increasing amounts of government overreach. Well, art gets the last word in our email box. Are you or your guest human? Or AI, how can I know? How can you know, indeed, flesh and bone art, flesh and bone? That's all I can say to you here. You have to take our word for it, but flesh and bone, Nate Sores is the president of the machine intelligence research institute co-author of if anyone builds it, everyone dies. Sias Kapoor is a PhD candidate at Princeton and he's co-author of AI Snake Oil and Peter Wildeford is head of policy at the AI policy network. Remember, we love hearing your ideas so many great messages, thoughtful messages from you during this show. Tell us what we should talk about next. You can send your messages RWA1A at WAMU.org. Today's producer was Avery Jessa Chapnick and this program comes to you from WAMU, part of American University in Washington distributed by NPR. I'm Todd Zwillick in for Jen White. Jen's a way for a few days at the Aspen Ideas Festival, by the way. She's hosting panels on everything from AI to science in an age of skepticism. We're going to bring you those exclusive conversations in the coming weeks. Thanks for listening. Join us tomorrow. I'll be here for the Roundup on Todd Zwillick. This is WAMU. You've got a lot of ways to get news and a lot of podcasts in your feed that take a long time to get to the point. Here and now anytime gets to the heart of the day's big story, all in about 20 minutes every afternoon. Get smarter and expand your world fast. Listen to here and now anytime on the NPR app or wherever you listen to podcasts. Of all the protests in the summer of 2020, for a moment there, it was Utopia. One took a unique turn. Somebody over there is anything not easy to gun. This is the story of how violence came to occupy an anti-violence occupation in Seattle. Listen to We Keep Us Safe, a new True Crime series on the embedded podcast from NPR.