Artificial Intelligence, Real Fears: Is it time to slow down?

Hello, and welcome back to Independent Thinking, the weekly podcast from Chatham House. I'm Bronwyn Maddox. Today, we are talking about AI. It's inescapable. Is super intelligent AI going to wipe out humanity within a decade, or find a cure for major diseases in that same time? Or I guess both. Just in the past week, we've had a slew of warnings. Leaders of the industry giants have called for restraint. Sam Altman of OpenAI, Elon Musk, they both packed the call of anthropics Dario Amadei to slow the pace at which we improve the capabilities of AI models. And those warnings came after OpenAI released its latest model Astra, claiming it was close to realizing the threshold of artificial general intelligence, that is, an ability to learn and perform tasks at the level of humans and a potential step along the road to super intelligence. We've also had mathematicians in the past few days, winners of the Fields Medal, the equivalent of the Nobel Prize, adding their voices to the fear side. But we've had others in medicine saying AI could cure major illnesses within years. And Donald Trump has said, don't slow down. We, the US, need to keep ahead of China. Whoever wins the AI race wins. So who's right? Does AI represent an existential threat to our species or a transformation of medicine, health, and our understanding of the universe? Is cooperation possible, or is competition between companies and countries an insuperable obstacle? Joining us today, we've got Alex Krasodomsky, who's director of our Digital Society program. Welcome. Thanks, Bronwyn. You've had a very busy week. More to come. Also, Marion Mesmer, our program director for international security. Great to have you here. Thanks for having me. Not at all. Well, look, let's dive into the question of how serious these threats are. Alex, what is your take on this? How worried should we be? This is an area where there are a lot of questions, and the questions all can have the same answer, which is yes. So are there long-term, potentially existential threats associated with the development of artificial intelligence? Sure, absolutely. Are those threats potentially quite sudden, quite dramatic? Yes, they could potentially be. Could they be long and drawn out, this risk of human disempowerment, that over time we become so dependent on machines that actually humans sort of voluntarily give up their capacity to shape them because these things are so helpful? Yes, absolutely. Does the fact that there are long-term risks preclude the fact that there are immediate-term risks, risks today, opportunities today? Well, no, absolutely not. These things are all potentially true at the same time. And I think the big challenge for any decision-maker, or frankly, anybody who has been watching the frenzy over the past year, is that they're not going to be able to do the same. Over the past couple of weeks, has been picking out what exact risk are people talking about in this mess? Thanks for taking us straight to that question. I was at a birthday party of a columnist, a British columnist, last night, where people were discussing exactly this. What does annihilation of the whole species look like? I think this is exactly the important question, and that's one thing that I've been missing a little bit in the discussion over the last week, as for whether there's an existential risk stemming from AI. Some of our colleagues who are more focused on the AI development side of things are often quite reticent to pin down specific scenarios. And I think the conversation has actually massively improved if we look at what specifically AI could do or could accelerate. And so for me, at least, looking at security implications and looking at the international security field, there are sort of three broad categories, and all of them begin to look slightly different when you begin to discuss them with colleagues who have spent a lot of time on this. And I think decades studying risks in that field, which I also find really interesting. So there is the cybersecurity area, where we have seen that AI has made great strides, making it easier to hack different things, making it easier to spread disinformation, making various cyber attacks much easier. But at the same time, AI is also really helpful in helping detect any vulnerabilities in systems, and therefore patching them much more quickly. So if one threat scenario is that AI essentially is going to hack a really crucial part of our infrastructure, and therefore then cause a lot of excess death, that might be one such threat. But at the same time, states and infrastructure operators have a lot of tools at their disposal to detect weaknesses early because of AI, and therefore patching them more quickly. The other big threat that often comes up is around AI helping especially non-state actors develop bio or chem weapons that they might not be able to develop. But when you actually discuss that with colleagues who work on bio and chemical weapons elimination, they will say that you still need a human in the lab, that still needs to be a very trained human, you still need access to all sorts of lab equipment that is actually regulated, and you still need access to ingredients, compounds, etc. that are also quite controlled in how you can purchase them and have access to them. And then finally, another big risk that is often cited is whether non-state actor groups might be able to build a dirty bomb on nuclear weapons, anything like that. And so I think that's a really important point. And I think that's a really important point. And there, I think the limit is, again, not necessarily that AI makes it easier to access the information, which might be true, but is also already available on the dark web. But again, it's this question of how would they actually get access to the fissile materials, which, again, are a highly controlled material that is actually very difficult to access. So essentially, I really agree with Alex that there are risks, and there are risks that we're facing today. But a lot of these risks end up, I think, getting slightly overhyped when they're taking out of the system. Alex, going back to these warnings, which come right from the top of these tech giants, and they're saying, you know, the whole of humanity could go in 10 years, what is the vision that they have of that? AI have changed over the last three, four, five years is that when we're talking about AI, we used to not just talk about large language models, but now we primarily think of AI in the way that anthropic and open AI have sort of defined it, right? There are a lot of things that we would call AI solutions that actually predate these large language models being farmed out to almost everyone on the planet. And I think some of those perhaps narrower applications is also where we are already seeing a lot of benefit and we might see much greater benefit in the future. So for example, if you look at medical research, the ability to find new types of proteins that can then be used for new medicine, for new treatments and so on, or diagnostically, or for a security example, if you look at the ability to analyze really large data sets that humans can't get through, we've had a huge issue in, in intelligence generation for years, where the US, the UK, other allies have essentially amassed these huge data sets that they have collected over the years, but that the human analysts can't analyze with human capabilities. So if you are able to use a machine learning algorithm to analyze that much faster, to do keyword searches, or to find the data that you're after, then all of a sudden you have really supersized the amount of intelligence you can gather and then analyze. So being able to detect threats much earlier might become much more possible. So that's, in my view, a really positive way to use AI, to use machine learning algorithms and so on, that goes beyond the LLM conversation. And I think we're going to see those kind of applications accelerate much more as well. And we're seeing a lot in medicine, very personalized medicine becoming much cheaper, as people say, exactly what kind of cancer does this person have and exactly what drug might target it. And the astrophysics tribe, if I can put it that way, they're finding the ability to analyze patterns of very, very distant galaxies. You mentioned mathematicians in your introduction. And I sort of think that this is also the week amid the frenzy that they have had their, what I think of as a Lee Sedol moment. Lee Sedol was a Go grandmaster, which is a sort of strategy game, not too dissimilar from chess, that was always thought to be something computers could never deal with. And a few years ago, a Google AI beat Lee Sedol at Go. And he sort of described that feeling that he had initially, of extreme dread and upset as something that was previously perhaps an indicator of human creativity or ingenuity or smarts was suddenly taken away from him. And he realized actually that this is now machines would be better than humans forever at this pursuit that had previously been entirely a human one. I think, you know, reading reports of how mathematicians responded to an open AI team cracking one of the millennium maths challenges, they sort of feel, feel the same now. And they're sort of feeling that very same crisis. I thought there was some exhilaration as well. This problem can be cracked. It was about airflow patterns. Well, the example then to look to is chess, which for now many years, computers have outperformed humans. And I think initially, around, you know, Garry Kasparov and Deep Blue, there was this sense of sort of anxiety and I'll know what's going to happen. But now I mean, chess is booming. And the relationship between the humans and the machines that play chess is a really interesting one where you sort of judge your accuracy based on how the machine might play. Have played in those spaces. And can you find the kinds of solutions that the machine might have come to? And I think that sort of symbiotic relationship between the human player or the human and the machine that is also sort of trying to solve the same problems is, I think, a really interesting one that will now be expanded more broadly. It's really interesting. So I do have in mind that Demis Hassibis, creating DeepMinds, that he went into this whole line of investigation because of realising that machines could beat chess. And why spend your intelligence on that? Why not on this? All right, so let's preserve if you like, in our minds, that sense of excitement about the positive things that AI is bringing. But come back to these warnings, because it has been a week of some intensity about them. And there's been a lot of comment online about who's right. Why, for example, there's a whole block of scepticism saying, well, companies are just saying this, particularly American ones. They'd like to slow down this incredibly expensive race. They might like to work in cahoots with any regulation they're developing. China's obviously very keen to slow down the Americans. It's not a surprise the Chinese government has come in and said, yes, yes, yes, let's have some more governance. Alex, what do you make of the motives and whether that changes the judgment about what we should make of this? Many AI safety advocates must feel like Cassandra in that every time they make these warnings, there are still sceptics who look for reasons to not believe them. And that's certainly scepticism that has emerged from a number of different quarters, whether it's on the validity or the accuracy of some of the claims being made, which we touched on earlier, through to the motivations. Most notably, Donald Trump's AI czar, David Sachs, really raised this as, don't pretend like you need all of the kind of regulatory capture or permission or don't pretend like you need to form a cartel in order to slow down. Just get on with it. Get on with slowing down. Sure. If you can do that. Yeah. If they really do believe that their products are not safe for the market, or for the world, well, it's probably in their interest to make them safe because they will face any number of questions around product liability, for which there is already a lot of legal precedent that they probably want to try to head off. These companies, as far as I can tell, are in a tight spot when it comes to finding a long-term sustainable business model. And part of that business model will be the so-called agentic AIs that have been at the centre of all of these safety fears. Explain agentic AI. And AI, in this case, agentic AI is an agent that will go out and do things for you. Perhaps it will book a restaurant. Perhaps it will call you a taxi. Perhaps it will commit a felony and hack a competitor's company. Now, if it does the third one, you might be in trouble. And I think there is a big question around how we create a legal or regulatory environment that allows this explosion of personal agents to take off without, you know, launching a thousand lawsuits every time they do something wrong. Who is responsible for that? And particularly if it gets out of control or what it was supposed to do, a whole subject in itself. And as you said, the legal responsibility, the thinking on that is just developing. Marion, how is this playing out in the defence world? Because there's a lot of wrangling, isn't it, for example, between Anthropic and the US federal government over safety and what to do about it and constraints on these things? Yeah, I think there are a few really interesting, different threads that come together on this point. So, Alex mentioned that the AI labs can do things on safety. And I think what's interesting is that some of them do that already. So, Anthropic has, for example, an AI safety report that comes out quarterly or so, where they essentially sum up some of the threats that their threat analysis or threat intelligence team has intercepted. And they essentially describe what happened and what they've done to intercept it. And then that the models now have safeguards in place to make sure that something like that doesn't happen again. And that's really interesting because in the most recent one that also came out, one of the really big case studies, it was actually a case study that came out in the United States of America, and I think it's a very interesting one, is around how Russian state actors have used a cloud model to essentially do really wide-ranging cyber attacks aimed at Ukrainian government, Ukrainian drone supply chain, and so on. And the other interesting aspect of Anthropic's report is that they also try to highlight where these actors were able to do something with AI that they previously wouldn't have been able to do, or where AI has really accelerated them being able to do that. So, I think that's one component because it's a very interesting study. And I think that's essentially shows how, in this case, Anthropic's model is already being used in a security context. And then the other element of that is how governments are being able to get on with integrating various different types of AI into their own militaries, into their own security and defense architecture. And one of the challenges we have here, I think, is that there are lots of open questions around how to keep classified data safe and secure. So, a lot of what makes especially large language models work is when they have access to really big data sets. But if you need to restrict that due to classification reasons, then you're also hamstringing the model's efficiency in some ways. And then you have the slightly separate question of which governments will be able to have access, especially to the big US lab models, especially if the US is trying to restrict all of that. What will be some of the different trade-offs between having a sovereign model and not necessarily having the most cutting-edge model? What kind of level of sophistication do you really need for some of these security applications? And those are questions that are very much being worked out in real time, and different countries are coming to very different conclusions there. How much is AI used in targeting? This is becoming a concern. You see it rattling in the media these days. So, I should start by saying that most of the details of it are classified. So, we only know bits and pieces from whistleblower reports, from investigative journalism, and so on. But using AI is, of course, one of the applications because it really helps with defining targets and making targeting decisions much more quickly. And one relatively prominent example from the US attacks on Iran was when, in the early days of the war, a US missile struck a school. And from various whistleblower reports and investigations into it, it seemed that this was indeed an automated targeting system where huge amounts of targets were pre-selected on the basis of certain criteria. And the reason why. why the school ended up wrongly being selected as a target was because it's not always been a school and in old map data that hadn't been updated, it essentially was still tagged as being a military basis. So one of the challenges here was that the human operator perhaps didn't have enough time to dig more deeply or there weren't any red flags in the target selection that the software made that would have caused a human to query whether that was correct data. So in that sense, the targeting worked perfectly in the sense that the missile found its target, but the target was of course completely wrongly identified as being a military target on the basis of outdated data. And that's one of the really big risks of ceding too much decision-making power to software, which can of course really accelerate what you target and how many missiles you fire, which apparently was one of the big goals that the Pentagon had at the time. But the flip side is that you risk committing war crimes, you risk really going against any of the conventions, the legal conventions that are meant to govern warfare. Alex, how much weight should we put on the fear of disinformation? I'm just back from Berlin and obviously the AFD, the right-wing party had just had a big state election victory in Saxony, Anhalt and more elections to come. And there were a lot of questions around about whether Russia using American social media was stirring up support for right-of-center parties, which are incidentally pro-Russia parties, and trying the same thing in France ahead of next year's crucial presidential election, possibly helping Marine Le Pen, who has leant towards Russia in her politics. Not the end of humanity, but conceivably very destabilizing to European cohesion, if true. Almost in contrast to our sort of cybersecurity landscape, I think our collective information environment online is peculiarly vulnerable. To disruption. And we've known this for sort of 15 or 20 years, be it through Facebook or Instagram. I mean, there hasn't been a week that has gone past recently without some criticism leveled at the ways in which these platforms are vulnerable, going as far back, even as, you know, the Cambridge Analytica scandal of more than 10 years ago now. AI absolutely turbocharges this, both by design, I should say, and also through its own vulnerabilities. By design, these are systems that are allowing you to produce very convincing text, are very capable of convincing other humans, and are able to do so autonomously or semi-autonomously, and to do so at an extraordinary scale. So of course, you know, this fear of AI slop at the ground level or AI propaganda, when it's a little bit more advanced, are absolutely well founded. And finding some way to buttress an already extremely fragile information ecosystem, I think is really important. That is even leaving aside from when the technology, itself, becomes vulnerable. And there is now emergent work looking at the ways in which hostile states can poison these models by, for instance, producing 10 sources that are designed to look as credible as possible so that when the AI draws on evidence it finds on the internet, it draws on these sort of poison honeypot information sources and then represents them as fact. And of course, when an AI tells you something, it's very difficult to then fact check it. It is presented very confidently. And with very little external context. So let's look head on then at this question of what more governance of this might look like, what more cooperation might look like. And this fascinating question that Alex brought in of who is responsible for things going well and when things go wrong, because we're dealing with two big obstacles to cooperation and governance. One is intense competition between countries, principally the US and China, but not just, many countries. Trying to have their own industry in this. And then intense competition between companies. Alex, do you think there is a reason for, let's say, the US and China, let's start at the country level, for them to cooperate? Well, I mean, all eyes on DC a week today of recording, right? When President Trump will meet with President Xi, we're told, and I'm sure that artificial intelligence will be on the table there. Is there a reason for cooperation? I mean, absolutely. I think there are narrow examples of AI risk and other sort of principles that, for the most part, be they American or Chinese or European or from anywhere else in the world, broadly agree would be meaningful, useful steps to make sure that this technology is some amount safer than it otherwise would be. Most obviously, transparency. This is something that Dario Amadei unilaterally committed to in his letter to sort of bring third-party observers into Anthropic to sort of keep an eye on how these models were developing. These kinds of things are very important. And I think that's a very important thing. I think while they are voluntary and self-regulatory, I think are sort of steps in the right direction, but they are small steps. And I think what the world would really need here is some degree of a global governance framework. And you're absolutely right. The first blow has to be struck in agreement between the US and China. And yet we're not seeing that at all at the moment. We've got Donald Trump saying, no way. China saying, yes, please. And China's got a very good line running at the moment of how it's really on the right side of the rules-based order. And here if you like, something of a deliberate caricature. So possibly not a lot of movement there at the moment to the extent this comes from the presidency of the US. That's the key point. I think we know that there are a lot of dialogues between AI experts globally, but particularly between the US and China on AI safety. This is not a totally sort of Cold War scenario. There is a lot of exchange of ideas and learning and research. A lot of this does happen in the open and closed dialogues are also possible. The challenge is, of course, you know, at this highest level and President Trump has been unequivocal, perhaps unsurprisingly, right? Like, you know, one of the key measures that he sort of takes for his presidency is the stock market and AI means that his stock market is booming. And a slowdown would potentially threaten that, you know, we've already seen a drop in some of the valuations in the light of these announcements. And so clearly, the US president is very much put on the accelerator here. And Marion, is there quiet cooperation on the defence front? Alex said this isn't the Cold War. There's also dialogue at the expert level. And I think there is a certain amount of cooperation now among European NATO member states in terms of how they can navigate being a few steps behind the US at a time when they can no longer necessarily rely on access to US technology in the same way how they are used to doing that. And I think that's really interesting because Europe, of course, has a lot of technological capability. And so seeing what that will mean for a specific AI solution will be really interesting. There are also some really interesting developments where China is trying to push the regulatory conversation a little bit with an AI initiative that they have put together that is aimed at building capacity among global South countries, which seems to be well received. I'm going to summarise that as a bit of cooperation there, but not a lot of desire to check what the US is doing at the government level. Let's go to the company level, though, it's interesting. A company and courts, because Alex raised this point about liability. And we did have on the subject of the harm done by the content, this very big settlement driven out of the courts, not out of Congress, of Meta paying 16 billion in settlement of accusations that its content hurt particularly young people, which seemed to many to be a recognition that the public mood is turning against these companies, against data centres, against the content, in a way that may open up the space for the politicians in Congress who move more slowly to do something about it. So Alex, I was just wondering what you made of it of whether the companies want to cooperate and their own response to these challenges about their responsibility. I don't think the companies want to necessarily cooperate, they want to win, they want to come out of this on top with whatever that looks like, you know, in the market. And I want these companies to collaborate or cooperate or to form a cartel in any meaningful way. I think competition here, particularly if it is competition, not just on who can sell the most powerful AI, but also the most reliable or the safest or the most useful AI, that kind of competition is extremely useful, and it probably furthers the case for safe AI. Even leaving aside the companies, this question of what about open source models, these AI models that are released, often by China, but also by US companies, and also European companies, that anybody can download and run on their own systems. Does that create the kind of competition that eats into the market share that these major technology companies have? Yes, absolutely, it does. And I think that really has been one of the stories of the last 12 months is this extraordinary shift born of realisation that, blimey, accessing AI through an API, you know, buying a cloud licence, buying an open AI chat GPT licence or whatever, is really expensive. And companies are looking to cut costs. And in doing so, companies are looking to cut costs. And I think that's the kind of thing that I think if you're looking for a diverse market that might sort of allow for new types, safer types of products to emerge, I think that's probably a good thing. And tech stocks have had a pretty bumpy time recently on just as people start probing exactly those questions. So as we're coming to the end, I mean, Marion, do you think we really need to do something about this? really in any way can slow down the pace? it depends on what we mean by slowdown. I think there is a lot of opportunity to cooperate on very specific risks and risks we already see today. So cybersecurity, spread of disinformation, those kind of questions that are often perhaps now overlooked, but they are much more unlikely to lead to an existential threat to humanity. But they are transnational, and they require, in some way, international cooperation to tackle. And in some cases, they could be tackled effectively by national legislation, but perhaps different governments are not sure how to handle the new geopolitical layer that US companies have taken on. I think we've seen this really interesting shift over the last 18 months, where a company like Meta or like Google was almost not seen as a US company. It was a global company that had offices and that employed people in so many different countries. But because US government has tried to instrumentalize their relationship in the international arena, they have now become US companies, and therefore other states are beginning to wonder to what extent they can rely on using their services. And so I think that's one really interesting component that could drive national legislation that then means that those companies are curtailed operating outside of the US. We are still some way away from that, because those companies also provide tax revenue to an extent, they provide employment opportunities, but I think that's one avenue. And I think the other question really is, and that's something that we think a lot about here at Chatham House, right, is what will middle powers do in face of the US and China racing away with this? We've not really spoken about India, for example, in this episode, but India is a country with lots of tech capabilities, lots of very skilled tech workers. And Alex mentioned open-weight models. There could be some really interesting startups coming out of India in the next few years that essentially may take advantage of their human capital and open-source models to offer their own tech solutions that might be a lot cheaper than what OpenAI or Anthropic can offer. Alex, can we slow down? I think we can slow down. And I think it's very probable that we will now slow down. There are a lot of different reasons why the slowdown might happen. It could be because of extraordinary high-level political pressure, like campaigns run, for instance, Professor Stuart Russell, who is an associate fellow here at Chatham House, has been driving an extraordinary campaign around red lines for AI safety that has attracted a huge amount of very senior political attention. That could lead to some decisions being taken at a global level that slows this down. It could be because of public pressure. We know how unpopular data centres, for instance, are in a lot of countries. That itself could be decelerationist. Perhaps we've hit a wall in capabilities with these AI models. I'm sure that we've only seen some amount of their power based on current technology paradigms, but perhaps that acceleration will begin to slow if revenues change or if the pressure on these companies to spend in a way that is a bit more sustainable increases. That might also lead to a slowdown. And perhaps, as we've argued at Chatham House, this or something similar will constitute this high-visibility moment, which to some extent is a euphemism for some kind of major disruption, does finally get the US and China around the table to agree to something that looks more like a global crisis. More like a kind of governed slowdown than something that is forced on these companies by market pressure or public pressure. Well, the alternative, I guess, is that nothing much happens. I mean, that all the forces against a slowdown, other than, as you said, rather inchoate public pressure and slew of alarms, that they don't really do the job and things just keep going on. Well, quite. There is the other side of the coin, which is it takes one breakthrough to accelerate all over again. This acceleration of since 2017, when what's called transformer architectures are going to be a big deal. I mean, it's going to be a big deal. appeared on the scene and really did turbocharge machine learning, which, of course, has been a field of study for decades. That can all happen again. And frankly, to come back to where we started, that is where the big worry that Dario and others have pointed to is that if you have machine learning models capable of training other machine learning models, you potentially create this virtuous cycle, this what they call a hard takeoff that suddenly accelerates things all over again. And no matter how much political pressure, how much capital constraints there might suddenly be, the maths just does what maths does. And that's what we're talking about. And that's what the maths does. And the curve just gets steeper and steeper. And that has to be a real possibility. Well, we have to not just slow down, but stop in a moment. But this has been absolutely fascinating. And I'm going to take the thrust of this from the two of you as broadly optimistic. I mean, at least about the annihilation of the species, which is where we started out and that actually is rather hard to construct ways of killing every member of the species at once, though, obviously, smart minds are at it. There's been a fascinating discussion that does bring us to the end of this week's edition of Independent Thinking. Thank you, Alex Krasadomsky. Thanks so much for having me, Bronwyn. Thank you, Marion Mesmer. Thank you. Great to have you both here. And thank you all for listening. Do follow Independent Thinking. We're out every Friday. So do pick it up on your favorite app and do look at The World Today on our website. That's our quarterly magazine. And the cover at this time is a superb article and then some other articles along with it on AI. That's chathamhouse.org. Thanks a lot. Thanks for joining us. See you next week.