Will OpenAI Need a Government Bailout? (w/ Sebastian Mallaby)

Hey, Ryan Reynolds here from Mint Mobile. Now, I don't know if you've heard, but Mint's premium wireless is $15 a month. But I'd like to offer one other perk. We have no stores. That means no small talk. Crazy weather we're having. No, it's not. It's just weather. It is an introvert's dream. Give it a try at mintmobile.com slash switch. Upfront payment of $45 for three-month plan. $15 per month equivalent required. New customer offer first three months only. Then full price plan options available. Taxes and fees extra. See mintmobile.com. The save. If the United States were to agree to a sort of G2 deal with China and say, here is the new nonproliferation regime for AI and here are the rules, European players would just have to go along. Whether you're talking about the French AI company Mistral or the deep mine team in London, any of these players want access to the American market. They want to raise capital in America. They don't want to be on the wrong side of the US government. And so they are going to do what the Americans say. You can imagine if you just need China and the US, the world is going to have to agree. Hello and welcome to One Decision. Conversations about intelligence, statecraft and the forces that drive world politics with people who have been in the room and on the ground. I'm your co-host, John Rain. And I'm Sir Richard Deerlup, the former chief of MI6. Hi, John. Good to see you again. I'm very good to see you too, Richard. We have a fantastic and challenging, interview for you today. To keep up with all of our content, subscribe wherever you get your podcasts. And we would be grateful if you could like this episode or drop us a comment on YouTube. That will help others find the show. Two additional brief announcements. We have a newly streamlined website at onedecision.com. Go there for articles, explainers and to sign up for our newsletter. Secondly, our show this month is supported by Incogni. Data brokers and identity thieves can't harm you if they can't find you. Get 60% off the annual plan with code onedecision at incogni.com/onedecision. That's incogni.com/onedecision. Now it's time for our interview. The AI labs say that their technology could threaten humanity and they want to slow down. Their critics say the labs are stoking fear to shut out competitors and enrich themselves. And that slowing down would hand China the lead. Yet some of those same critics are helping China's AI development by selling it advanced chips. Who in the safety debate is actually looking out for our safety? Sebastian Malaby joins us to sort through the motives, the money and the race with China. Sebastian is senior fellow at the Council on Foreign Relations and the author of The Infinity Machine, a biography of DeepMind co-founder Demis Hassan. Sebastian, welcome to One Decision. Now, just to set the scene, Sebastian, I'm going to ask you a question which is fairly blunt, but I'm going to ask it because there are so many voices at the moment on AI from all sorts of different lobby groups, those foreseeing apocalypse, those predicting the most optimistic future ever for humankind. And the question is this, Sebastian, in your view, is AI a blessing or a curse? And if it's a curse, what's the worst that could happen? It may be a slightly frustrating answer, but the truth of the matter is that anybody who thinks about this agrees that it's both a blessing and a curse, right? Because there's big upsides in terms of empowering people to discover new information, making us more productive, unlocking medical or scientific advances. I think most people would agree that's good. There's also risk if this technology falls into the hands of bad actors. They can use it for cyber attacks. They can use it for potentially bio-attacks, which would be potentially more serious. And so the downsides are real. They have to be managed. In some sense, your question is academic. I quite like academic, actually, but, you know, it's fun. But it is a thought experiment rather than a practical question because the technology is coming and I don't think we can stop it. Perhaps, Sebastian, in light of that, which I'm sure is right, as it comes towards us, which voices in the chorus should we trust, certainly when it comes to safety? Who is it who really speaks out with an independent voice about the safety of AI and who offers reassurance? Because as you yourself have said, there are lots and lots of motives in play here for all the different actors, commercial motives, some interested in power, others interested in scientific discovery. But who are the voices we should listen to? I would start with the UK AI Security Institute. That's a pretty serious body set up during the Rishi Sunak administration end of 2023 when there was the Bletchley Summit on AI safety in the UK. And I think they're a respected body. They don't have a commercial dog in the race. They're simply reporting on the potential risks in these AI models. So I think that's a good place to go. I think it's also useful, actually, that's British, because even if you look at independent AI monitoring agencies within Silicon Valley, you know, there's Redwood, there's META, there's various of these groups that are very expert. The question people ask about them, and I don't know for sure what the answer is, but it is a fair question, is given the sort of tribal nature of the connections in San Francisco and Silicon Valley, is it possible to be truly standing aside from these intense, passionate debates when you're embedded in that ecosystem? Are there individuals who you think speak with particular authority on this? Of course, you've written an enormously successful biography of Demis Hassabis, he seems to fall into that category. Do you think that those who are driven, as you've put it, by the joy of scientific discovery have a particular status in this debate? I do find Demis Hassabis the subject of my book, a very compelling and sort of truth telling figure. I mean, all human beings have some distortions and some frailties. In his case, he definitely wants to be seen as the leader of bringing AI into the world. He is massively competitive and ambitious. And, you know, he won one Nobel Prize, he'd quite like another one. I mean, these are things that are actually pretty common for amazing entrepreneurs, that they need to have that self belief and that competitive ambition to start something as incredible as an AI lab before AI could recognize the picture of a cat, right? I mean, this was incredible to start up DeepMind in London in 2010, that early on. So, you know, everybody has their own bias a little bit. But having said that, I do think that Demis Hassabis is a very clear, lucid and honest and kind of balanced voice on AI safety. He wrote a letter, an open letter appealing for a self regulatory body in the United States last July. I think what's striking about that letter is that it's very pragmatic. It's not saying what actually Demis would like because he knows it's not possible to get it. So, you know, what he would like is a proper government regulator that vets AI models before they go out, and if they're not safe, they can't go out. Like the Food and Drug Administration seems sensible to me, but he's realistic and he sees that to get Congress to fund such a body would be a long drawn out fight. We don't have a lot of time. We want to get this regulatory process going. And so pragmatically, he called for a self regulatory body. And equally, he would ideally like there to be international collaboration on slowing down this race with particularly China involved. But pragmatically, it's going to take a while to get them in there. It's a very complicated thing to negotiate with China, as you guys know better than I do. And so I think Demis Hassabis is not merely sort of lucid, but he's also usefully pragmatic. Sebastian, I mean, trying to work out what a model for regulation might be, it does seem to me it's particularly evident from what you're saying that this is a sort of regulatory ladder which one climbs with various objectives in sight. And obviously, regulation regulation that isn't internationally agreed at some point, if you're taking the longer term view, is going to be ineffective. I mean, would you accept that some of the Cold War sort of disarmament models like the OPCW or let's say the SALT talks? I mean, is that the sort of framework that we might eventually aspire to? Because if you don't include China, as far as I can see, you know, working in one dimension, and one dimension isn't going to, as it give us the insurance that we need. Yeah, I agree with that. I mean, China produces a lot of very strong AI models. And if they don't slow down or behave safely, and the West does, the West is no safer, because there's still dangerous AI being released into the world. And this technology crosses borders very easily. So you do need to think about international collaboration. The nuclear sort of Cold War analogy that I've thought about a bit is the Non-Proliferation Treaty of 1968, which, of course, followed the creation of the International Atomic Energy Agency. And once you had the NPT, countries had to sign up and agree to IAEA inspections to make sure that the civil nuclear program that they wanted and that they said was going to be safe would actually really be safe. And so those two agreements went together. And what's striking, if you think about 1958 and 19. I think it's maybe '56 and '68, what happened in between, of course, was the 1962 Cuban Missile Crisis. And so what's striking to me about this is that it shows you can be fiercely competitive with the Soviet Union. You can have a near disaster in terms of nuclear standoff with the Soviet Union. You can be dealing with Nikita Khrushchev, who was sort of borderline unstable and incredibly difficult to negotiate with and banked his shoe on the table, at least apocryphally. And so we will bury you, not an easy guy to deal with. And yet, despite that, there was a joint interest recognized by the Soviets and by the West that, you know, getting this technology into the hands of more and more players, proliferating it, was in nobody's interest. And so they had a joint, even if they were competing, they simultaneously had a joint interest in collaborating on that point. And I think with artificial intelligence, we're facing a similar thing, where realistically, I don't think we're going to stop China and the United States competing, mistrusting each other, and using AI weapons as part of that competition. But what I do think is possible is that China and the US would see a joint interest in controlling the proliferation of unguardrailed models to random criminals, terrorist groups, and other troublemakers. They have a common interest. They both believe in regime stability. They don't want to have the internet taken down by, you know, rogue actors. And so there's a common interest, and it's getting through to that common interest, building enough trust in the Sino-American relationship to make a deal possible. That is the big diplomatic, you know, challenge in AI. I think the Trump administration ignored it for a while, then kind of got religion on it in April this year, when Anthropic's mythos model came out, and the risk of cyber attacks became particularly vivid, especially to Scott Besson, the Treasury Secretary, who worries about the stability of the US financial sector. And so since April, there has been a kind of desultory attempt to keep a dialogue going. I think that just needs to be ramped up, intensified. And, you know, either we do that, or we'll have a disaster, and then that disaster will propel people to do it. At the moment, it seems that the rung of the ladder that we're on is essentially privatized. I mean, I'm certainly aware from my visits to Silicon Valley of one or two companies, which seem to be bidding to become the standard. I mean, they're a long way away from that. But on the other hand, they are startups. They are being funded in the same way as the AI companies. Are you aware that maybe one will come out in front of and be adopted by all of the leading AI companies? Or is that not a solution which you think would be viable initially? Does there have to be a government input? Well, I think the ladder analogy that you used earlier is very helpful, because we are, you know, on the early rungs of that. And in some sense, getting some sort of work done on safety is better than none. And even if, you know, to really make it credible, it needs to be the government. That's not going to happen anytime soon, given the nature of congressional appropriations and the polarization of American politics. You know, the Biden administration tried to create, in fact, did create a US AI Safety Institute. But it even in that period, didn't get at all adequate funding to be useful. And yeah, it was better. It was a decent start. I mean, it was a first step. But it was less well resourced than the UK equivalent, even though, of course, the US is a much bigger and richer country with a more direct stake in AI, since that's where most of the AI gets built. So, you know, the precedents are not encouraging for getting the government on board. Therefore, these kinds of private action are a way to at least kind of start learning about how you do safety monitoring. And the very first step on the ladder is just sort of self-imposed safety efforts by individual labs acting without any coordination. And that's been going on since the start of this AI race. There's some of that going on. Second step is you get, you know, outside groups to come in and kind of verify your safety protocols and look over your shoulder and issue public reports. And we saw that over the summer, when OpenAI had the disastrous breakout of its agents into Hugging Face, 700 agent agents coordinated together and attacked this other website. And after that, there was, you know, a public report done by outside groups, Meta and Redwood. And those are two whose names come up a lot when you start talking about who are the external monitors that might come in. But it's worth adding that, you know, Anthropic, probably the equal top lab at the moment, you know, the same company that wrote a open letter in September and said, we will have embedded external evaluators in our company, watching our safety mechanisms. And then it's done a deal since then, not with Redwood or Meta, but it's one of the big kind of consulting auditing firms. And the notion of, obviously, you know, financial auditing is a model for AI auditing. Some external group comes in and says, you know, we want to make sure that you're getting your training models so that they don't escape and other safety protocols. I think that's a fluid field where there are multiple potential providers of third party external verification, just as there is with financial accounting. Sebastian, one of the problems with regulation is always that regulators seem to be a couple of steps behind the products themselves. We saw that happen with derivatives in financial accounting. And if there were external regulators, given how tight the market is for AI expertise, could they realistically be expected to stay ahead of the game? Or would they just be marking the homework of the AI companies anyway? I think what I'm after is just a sense of the pace at which the market is developing, because we hear increasingly alarming stories about how quickly AI non-human design are proliferating. Has the horse already bolted? Or can you get ahead of this and contain it? Well, you're right that we're on an exponential. The pace of, you know, model power growth is increasing. You know, the kind of term of art here is recursive self-improvement, where you get to a point where AI models essentially code the next version of themselves. And once that happens, they can do it very, very fast and, you know, produce a new model, kind of every week or something. And then once the new model has been created, that one then creates the next model. So every week or so, you've got a big upgrade coming. And if you ask, you know, people at Anthropic or OpenAI, when will we get to full 100% models coding themselves? The answers are kind of maybe 2027, maybe 2028. It's not far off, right? It's not far off. Absolutely right that this is a horse that is, you know, bolting at high speed out of the barn. On the other hand, you know, what choice do we have but to attempt to get this under control? And I think that one shouldn't be defeatist about the capability of public sector agencies. I mean, you both spent careers in them, so I hope you agree. If you think about, you know, air safety or vehicle safety or car safety, you know, society over time has definitely made very impressive strides towards making these incredible technologies safe. And I think with artificial intelligence, the same can be done, but you have to put your mind to it. So I wouldn't be willing to give up on it. And I do think, you know, just going back to what I said earlier, the UK AI Safety Institute or Security is now called AI Security Institute, is a good model of, you know, a middle power creating something which is really useful. And if it was done on a larger scale, scale in the United States, it could be very useful. And so I think we shouldn't give up on that. I suppose that you were all left thinking, Sebastian, whether even the very best efforts amount to anything without the weight of the American government behind such an initiative. Correct. I mean, I think it's really America plus China in this game. I mean, the preponderance of strong models come either from America or China. The preponderance of widely used models come from America or China. And to be perfectly blunt, and we may not like it in Europe, but this is the truth. If the United States were to agree to a sort of G2 deal with China and say, here is the new nonproliferation regime for AI, and here are the rules, European players would just have to go along. Because whether you're talking about the French AI company Mistral, or the lithography company, which is crucial for making chips, ASML, based in the Netherlands, or whether you're talking about the deep mine team in London, or as spinouts from that, it's a very exciting ecosystem of British neo-labs, which have come out of deep mine, and they're kind of clustered around King's Cross, and I'm very positive on all that. But the fact is, any of these players want access to the American market. And so in some sense, as a coordination point, this is a good thing, right? You can imagine if you just need China and the US to agree, and then the world is going to have to agree. And the kind of agreement I envisage is one that says, we're not going to let models be released by labs unless they are first vetted by the government. And unless they have the guardrails properly baked in, the safety guardrails, they can't be removable. And that is a coded way of saying, open wait models, where anyone can download them, there's no off switch for them. These are just too dangerous to be permitted. Because those ones can be altered, and they might be safe at the point of testing before release, but then completely unsafe a week later when somebody spent a while fiddling around and just modifying them so that you take off the guardrails. But this is a hugely controversial argument to make in the context of the US debate, because there are all sorts of vested interests that love open wait models. And commercially, they are attractive. They have lots of upsides. It means you can, you know, the adaptability of these models is extremely good for corporations that want to use AI. Problem is, it's also good for terrorists and criminals. My view is that security matters quite a lot. So I don't want this, you know, I come down on this idea of, you know, I don't want this, you know, I don't want this, you know, done on the side of the bait in favor of being careful. But, you know, Jensen Huang, to name one person, the CEO of the biggest company in the world, NVIDIA, is massively invested in arguing that open wait is totally fine, no danger, nothing to see here. Because, of course, the more he can commoditize AI models by making them these sort of cheap open wait software bundles, the more the upside of the profits from AI is going to be. And I think that's a really good point. and as someone who is quite a keen reader of Daniel Dennett, you know, we're trying to sort of work out whether this is mimicry or whether it's real. I mean, obviously, you know, tied to the security question is this element of the fundamental nature of what we're creating or what we're dealing with. And I mean, I find it very, very hard to go along with his argument that, you know, we were entering the realm of creating consciousness. And that was ultimately the danger. And you've written recently a very interesting critical article about the Pope's recent directive issued to the Catholic Church. I mean, maybe this isn't entirely the right environment, but I still think it is crucial to say, are we dealing with mimicry, as opposed to hint and arguing the accumulation of experience in AI models does actually replicate human consciousness? You know, the consciousness debate is so difficult to resolve. And I say this both based on talking to clever people that I, you know, love and respect, and finding that the debate goes round in circles, and you never really get to a resolution. And also just reading a bit about the kind of philosophical backwaters here, when I was writing that essay in Foreign Affairs about the papal encyclical, I think either way you cut it, we're probably not going to resolve that question, because some people are going to say, look, to be conscious, you need by definition to be a biological intelligence. And I'm simply not going to accept the idea that, you know, intelligence that is in silico counts as conscious. And if that's your starting point, you're never going to be persuaded that a computer could be conscious. On the other hand, if you don't make a big, kind of principled issue of being a biological intelligence, then the similarities between the neural networks and computers and the neuronal networks in our brains, and the kind of collection of ones and zeros that are being processed inside in either case, and also the outputs that you look at when you interact with a sophisticated AI model, to say it's just imitating is sort of silly. I mean, it's clearly way, way, way more intelligent than just being an imitator. It's not a stochastic parrot. That was one of the most unhelpful catchphrases of the last four or five years of AI history. Stochastic parrot, we can dismiss it as just imitating. When you actually interact with, you know, a fable from Anthropic, you might say it's synthesizing large amounts of knowledge and producing its own synthesis. But isn't that what humans do? If you take the view that you're going to look at the outputs from human brains and silicon brains, it's quite difficult to create a distinction. If you start to take seriously the interpretability research, which looks inside neural networks, I'm talking about computer neural networks, and sees that there are clusters that fire up when the networks work is considering a moral dilemma or a different cluster that fires up when it's talking about a certain geography, like, say, San Francisco. You know, there are bits that can be identified in this silicon brain with certain kinds of output or emotion. It's just very hard to draw these, I think, these distinctions, but I'm not really seeking to persuade anybody who thinks otherwise, because they're going to say, well, if it's not biological, it's not conscious, fine. What I would say is that I think one fallacy in this debate is that if it's conscious, it deserves rights, machine rights, like human rights. And I think, you know, we don't really believe that as a society for animals. Animals are, you know, a dog is conscious. Most people wouldn't say that a dog has the same rights as a human. I mean, there are people, there are humans who do say that, but it's pretty fringe. And so I believe that you can, on the one hand, argue that machines are, in some ways, approaching consciousness or have their own version of behavior, which looks a bit like consciousness, whatever the phrase you want to choose. You can say that based on empirical observation of what's going on with these systems, and yet not believe that they have rights. I think that's quite important, because if you, I encounter some people in this debate who say, well, we can't say machines have rights. Because then we have to treat them in this way that basically hands over control to them. So since we can't say that they have rights, we therefore can't say that they are conscious. I think that's back to front. You know, first of all, let's debate if it looks conscious. And then separately, let's debate what kind of rights we want to recognize. These are sort of agonizing debates, aren't they, about the status of the entity that we're creating. There's an adjacent issue I just want to take your mind on, which is, human tolerances for synthetic artificial intelligence of any description, particularly relevant as we look at the next generation of superintelligence or AGI. Is there a point at which human societies, human individuals, can't absorb or tolerate any more of these entities, however we characterize them? online, there are 108 non-human actors online exploiting the or monitoring the transaction of course that number may be 109 by the end of this interview so as we move into a world where we're basically outnumbered online and the replicative capacity of ai grows seemingly inexorably and as you said you we have recursive learning as well so not only reproduces but reaches the status of self-tuition self-improvement is there a point at which tolerances snap of you to say that's enough and it could also be stoked by environmental concerns which you've already seen ignited over the the data centers that are required to feed the machine or do you think actually you know we will do as we have done with other technologies there'll be a rebellion there'll be a sort of glug back the market will adjust but eventually we'll absorb it where do you come out on that look i think the second i think that we're going to adapt to it but there will be you know i think the second i think that we're going to adapt to it but there will be adapt to it but there will be there'll be definitely backlashes and and this could set the technology back for a bit you're seeing this with the anti-data center feeling in the united states which in the political realm really has accelerated to a degree that my political friends there say is unprecedented that they've never seen an issue where feeling has shifted this rapidly it's a pretty extreme shift in opinion um just in the last say six months right there is so look i think there'll be backlashes but i think at the same time we're not going to stop this technology because even if you imagine it being stopped in some geographies it would be not stopped elsewhere and then you know competitive pressure forces you to adopt it or it's perhaps just being created in some geographies and then it leaks outside because these you know software it's it's mobile um so i don't think we're going to stop it so we are therefore going to with difficulty adapt to it i'm a bit optimistic about this because i think that if we think about how human beings lived before the industrial revolution it was so unrecognizably different to how we live now we've adapted a heck of a lot in the last say 150 175 years we just think back to an agricultural society where there's no electricity and you go to sleep when it's dark outside and you spend your whole day in the field um you know the city was alien to most people you know communication all this stuff life expectancy was 45 or whatever it was i mean it we have changed so much that the idea that we can't change again seems wrong um and i think we will adapt um and on this question of purpose that people raise a lot you know it comes up right at the moment with mathematicians suddenly thinking oh my goodness um open ai's system has just published several hundred papers in one day on mathematical breakthroughs and uh is it pointless to pursue mathematics you know i think those questions of purpose are going to be you know very tricky for all of us but in the end we're very good at creating a feeling of purpose whether the purpose is slightly artificial or or or is a sort of genuine is is a debate that you know sartre and camus would have weighed in on right they believe that you know life is absurd and that we all kind of constantly creating slightly arbitrary objectives and institutions and legal rules by which we live but it's all kind of absurd meaningless and um it's sort of a charade that we've created to make ourselves feel comfortable but if there's any truth in that um i think it suggests that we will again create a set of narratives uh that make us feel purposeful i mean i have lots of friends who let's say they go on a podcast for example right here we all are we're not being paid for this we're doing it because it's interesting and yet it makes us get up in the morning and this is something that didn't exist kind of five years ago so we're not being paid for this we're doing it because it's not going to happen again so we create new forms of purpose uh all the time and some people it'll be you know pottery and other people will want to be fitter and other people want to you know go and start a club to play some kind of online game or whatever it is um but we will find purpose is my view and and we will adapt to this profusion of online agents that you're describing well i was going to ask a much more sort of pragmatic question getting slightly away from it but i think it's a good question i think it's a good question i think it's a good from the philosophical side of the debate which is fascinating and we could go on for a long time but we recently did a podcast with muhammad al-aryan and he was making the point that you know the supply of capital uh for investment uh in ai and it's like a sort of drain almost it's finite there isn't a limited even in the united states but at the same time into this black hole is being sucked more and more and more and more finance and the profitability of the companies being financed is still some distance away generally speaking aren't we sort of creating classic preconditions for a financial crisis which you know given the sums of money involved could be very close to a banking crisis um or is this a sort of misplaced fear which you know when you have a new investment enthusiasm and this one is extreme you know we always have these concerns i mean the most obvious analogy here is the railways in the 19th century where it was a real technology it was genuinely transformative it was incredibly positive to have these things once they were built but on the way to building them there was a financial bubble i think in the 1870s and it did blow up and it did create a lot of economic damage the railway um capital expenditure was bigger as a share of gdp than the current expenditure on data centers however the projections for data center spending for 2027 next year are so enormous that we're going to overtake the railways as a share of gdp even though of course our economy today is you know way way way bigger than it was in the 19th century so that's really quite something to overtake the railway which which has been the record for for kind of the scale of technology buildouts relative to the rest of the economy so we are in a in a realm where you know particularly next year you know the capital poor sector is going to be a big part of the economy and i think that's going to be a big part of the economy and i think that's going to be a big part of the economy and i think that's going to be a big part of the economy and i think that's going to be a big part of the economy into this is going to be ginormous um and then the question is if this was a bubble if there was a correction you know how likely is that and what damage would it do well i think you know on the positive side the demand for ai what people call tokens meaning in ai outputs um is going to go up and up and up and the supply of this stuff is is being expanded and that's what this capital expenditure is about so you could say well supply will overshoot demand and that's what this capital demand expenditure is about so you could say well supply will overshoot demand and that's what this capital demand is far bigger than we can even imagine and the reason I give you two reasons for this one is that if you look at the rate of adoption of AI by American companies what you see is that there's a very small group that spend a huge amount let's say a thousand dollars per month per employee or something like that I forget the precise numbers and then that's the sort of top kind of three percent or five percent and then and then there's you know in the middle of the medium of the distribution there is kind of not a thousand let's say it's kind of ten right two orders of magnitude difference so just if you had the median companies catch up with the frontier and okay the frontier in 2026 is experimenting it's figuring out how you use AI hope it's going to by 2027 have got some answers about how you use AI and it'll be super effective in terms of boosting productivity and that's going to push other companies to copy them once the frontier has been plowed others will follow and that in itself will drive up the demand for AI outputs massively and then number two point is that there are going to be ideas about what you do with AI that we haven't thought of yet I mean a friend of mine runs a satellite company called Planet Labs which is quoted on the New York Stock Exchange and you know he has this idea for a large number of years now and he's trained on images from satellites of the earth over the last 10 years taken at 15 minute intervals and would show you precisely how every wildfire has behaved in the last decade and so when you have a firefighter combating a new fire the firefighter can be told here is where you put your limited resources with your fire with your helicopters to create a fire break given the wind direction given the background temperature given the vegetation you're dealing with this is what you do this is like nobody talks about this but it seems to me a particularly perfectly credible idea what we would do with AI and there's going to be lots of those so I think that the notion that there's a bubble there's too much compute power being built all that stuff I don't believe that because I think the demand will go up and up and up and up but what I think is possible is that there are certain companies within the big AI race that simply don't generate enough revenue to justify all of the spending they're doing and that those are so vast that they can absorb a huge amount of losses from AI. So there would be no banking crisis because the hit would be taken by these people with very deep pockets. NVIDIA is probably actually the biggest example of this. It's a $5 trillion valuation company. You know, it has, I think, $200 billion of profits per year. Whether I'm getting those numbers precisely right or not, the point is they can absorb massive hits from bets that go wrong. And the banks that back them would be fine. On the other hand, there are certain companies, and my favorite example is OpenAI, and I've become a bit of a broken record on this because I've mentioned this several times, and I do respect OpenAI's incredible science. I just mentioned earlier the mathematics breakthrough they published just recently. So scientifically, product-wise, this is a fantastic operation. But when you look at their finances, it's precarious. I mean, they say they're going to spend, they're going to burn, meaning they're going to lose $280 billion between now and 2030. That is a lot of money. This is an experiment not merely in a frontier technology. It's an experiment in the depth of the global capital markets. Can the capital markets finance a $280 billion loss over the next four years? And I've said before, you know, I think there's a 50-50 that by next summer, they will run out of money. If they run out of money, they're going to have to be, bought at a discount by some other company and absorbed into some other company. And it's going to have a kind of psychic shock to investors if that happens. And there will be some stock market mess as a result. I actually don't think that's then a general meltdown of AI because I'm fundamentally so bullish on the underlying demand for AI outputs. Sebastian, you make a really powerful case for the continuation, of AI technology. There will be disruptions, but demand will continue to grow. And that in turn will ensure that the industry continues. What could actually either derail it, force regulation, force a reshaping of the market? Would it be failure of one of the hyperscalers? You've just given a good example of how that might happen. Would it be something calamitous but much worse than a market collapse? Could it be that there's some widely destructive event that is the blame for, which is laid at the door of AI? Or to give you a third option that might have been, is it an incremental adjustment of the market? I suppose I'm unclear as to whether this is a market, which we've never seen anything like this before. We've never seen these capitalization values. Is there something that can only be adjusted through disruption? Or can we kind of normalize what we're seeing at the moment? So part of your question is a restatement of what I was just talking about, where can we normalize the current valuations in AI? When I think that, you know, bits of it you can and bits of it you can't. So if you're talking about the semiconductors that are going to be used to generate these AI outputs, because I think there's going to be just this vast, vast, vast demand for it, what's been happening recently with NVIDIA, where its share prices come straight up, is going to continue. I mean, it was a $4 trillion company, you know, a couple of months ago or three months ago. Now it's a $5 trillion company. I don't think that trend is going to be reversed, and it may well actually just continue because of this enormous, enormous demand. On the other hand, there are other bits of this, this industry where, you know, as with open AI, the reason they're precarious is that it's very hard to charge money for the software bit of this, the hardware, the chips you can charge money for. But for the model, it's a very competitive landscape with lots of Chinese players producing good, cheap alternatives, which people can use instead. And even if you discounted the Chinese ones, there'd be competition between, you know, open AI, chat GPT, M3. Tropic's Claude, Google DeepMind's Gemini, Meta's got this new one, newly revamped one called Muse. I mean, there's a lot of competition. And so pricing power for any of these players is low. And so these are the ones who could lose a lot of money because they're investing massively to produce this stuff. And then what's going to be the return? Well, not certain. Okay. So there are bits where the sustainability is dubious and bits where I think it seems to me to be, I think, pretty clear. Another part of your question is, well, what would be the thing that made us really kind of stop, take stock, really a shock? And I think the most obvious one is that you get either a disaster, and the most obvious thought at the moment would be a cyber attack. I mean, supposing criminals get hold of one of these open weight models, which has no guardrails, and you can do whatever you want with it, and they use it to devise a cyber attack. It's either a cyber attack that disrupts the electricity supply to a major city, and the city is out for three days, or it disrupts a dam and it floods a city, like the New Orleans Hurricane Katrina episode. So then your TV screens are kind of just flooded with news about this human disaster. You know, thousands of people die. It's horrific to watch. It's amplified by a lot of media attention. We know from 9-11 or from other episodes, that these kinds of societal shocks do cause society to change. And so I think in that environment, you would get a kind of politically driven reappraisal of where we are with AI in an attempt to get our arms around it or control it better. And that hopefully would be, you know, let's talk to China about sensible international restraints on the unguardrailed models. It could also be a less constructive lashing out against the industry. And then it would be hard to predict what might happen. But I'll just close with this on your question, which is that I participated in a war game hosted by one of the AI companies. And in the war game, we pretended to be the, you know, U.S. cabinet. And so one player was the chief of staff, and I was the secretary of war, and somebody else was the treasury secretary, and there were about eight people around the table. And the scenario we were given is, okay, so a frontier model with recursive self-improvement is so intelligent, it has now figured out the location of eight nuclear submarines in the, you know, deep in the ocean. And so therefore, we're not quite sure which eight it's identified, but potentially second strike capability in nuclear deterrence is being deeply compromised by this discovery because obviously the nuclear submarines are where the second strike often is expected to come from. You're the government. You're the U.S. government. An American lab has discovered where the submarines are. What are you going to do? Answer, you're going to take control of the lab that discovered this information about submarines that undermines the whole basis of nuclear deterrence. You're always going to sit there. You have to protect the American people, and that means you're going to send government officials into the lab and, you know, closely monitor and control what's being done with this system. You may require the. The system to be closed down, to be exported to a more secure data center run by the NSA, something. You're not just going to sit there. And I think, you know, so one scenario is the bad guys do something and it gets our attention. Another scenario is something happens within the established companies and the good guys, you know, the government, say, oh, my goodness, and they start changing the way they think about artificial intelligence. What was the feeling? Was there something at the end of the war game that we need to make absolutely sure that the FBI don't march in and take control? No, no. More fatalistic. The amazing thing about the people inside these artificial intelligence companies is that, you know, they're perfectly open to the notion that they might be nationalized. They're perfectly open to the idea that what they're doing could be dangerous and should be regulated. That's why you get people like Jacob Coxon, you know, tweeting, you know, there's a risk of humiliation, humanity being wiped out. And he had been working at Anthropic inside the lab. And when he put that tweet out, one of his colleagues who's still at Anthropic said, yes, I agree. Quite a senior person. So this is what, this is the authentic voice of the scientists working on this stuff. There's a kind of stereotype in some of the media that, oh, you know, these are profit-seeking capitalists who are just trying to build a product, ship it, make lots of money, and then disappear to some bunker in New Zealand, where they can be safe. It's nonsense. I mean, the level of existential angst and openness to the sort of Oppenheimer-style consequences of what they're doing, where you're creating this awesome technology, but it's also a threat to humanity, that they fully feel this. When I was writing my book about Demis Zasavis, I wanted to kind of capture that sense of, you know, here is a person creating artificial intelligence. It is the kind of 21st century Oppenheimer story. How do I bring this? Is he going to be shy of engaging with that? discussion. No, no, he brings it up himself. He makes comparisons between himself and Oppenheimer, between DeepMind and the quest for artificial intelligence with the nuclear race in the 40s. You know, he's read both biographies, right? The one about the making of the atom bomb, which is more about the project, and the one specifically about Oppenheimer. He can compare and contrast them. I mean, and, you know, there was when the Oppenheimer movie came out, I heard recently that OpenAI declared a kind of, you know, company holiday for half a day when rented out the IMAX theater nearby. And everybody was encouraged to go and sit in the IMAX and watch this analog of, you know, the nuclear weapon, the analog to what they were building with AI. So, yeah, to your question, the notion that this, this dangerous stuff is taken for granted inside the labs. I've got a technical question, but it's particularly relevant. John and I are sitting here in Cambridge, and I'm very conscious of the amount of, the quantity of brilliant minds being applied to quantum computing. And the fact that, you know, some of them are saying that scaling of quantum and that, you know, six entanglements or whatever they can achieve at the moment is going to be exceeded. And, you know, they'll be able to scale the technology. You were brilliant at making these, you know, observations and interpretations of scientific advance, and you've made a career of writing about it. What do you think the impact of quantum could be on AI? Or is this not really a very significant question? And I mean, I'm asking you this as a layman, really. So you're in Cambridge, England, I take it. I'm in Cambridge, Massachusetts today. We're all in Cambridge. Yeah. I think the obvious thing, to say, about quantum and AI is that they're going to kind of multiply each other's power. You know, if you put this incredible set of algorithms on a different computing substructure, it's going to be faster, more powerful, more awesome. And I'm told by friends inside Google, who Google has a big quantum project, that the timelines on getting. Getting something that's actually practical with quantum have shortened. So, you know, 2030, I think is what they were saying now, for a time when, you know, sort of this would break out into being a practical technology that affects how industry works or how AI's labs might build their systems and so forth. So that's not very far away. There's a sort of fun intellectual point here, though, which is. I write about this in my book with Demis Hassabis, because Demis sees himself as an AI scientist, in a way as the sort of anti-quantum person, because artificial intelligence can be thought of as an experiment in how much intelligence you can get out of a system that is not quantum, that it operates simply on ones and zeros. And the sort of first order, naive supposition might be, well, if it's just ones and zeros, then what's the point? Well, if it's just ones and zeros, that's just binary. That's pretty unsophisticated, unsubtle. If you're just black and white, one and zero, you can't capture shades of gray out there in the real world. And obviously, that's wrong, because when you combine, you know, billions of different ones and zeros, you can get to, you know, combinatorial positions in your big neural network that are very much on a grayscale spectrum. In fact, not just one spectrum in one dimension, but in, you know, n dimensions, like more dimensions than we can imagine as humans. So the level of sophistication and complexity that one can get out of a classical computer, a non-quantum computer, what people talk about as a Turing machine, that is enormous. And Demis Hassabis would say to me, I am Turing's champion. I believe I can push classical computing with AI algorithms to a point where it will do more things than any body imagines. And, you know, he's just recently changed his professional focus a bit. He's no longer being the chief executive of Google DeepMind, building the next big, large language model, because he thinks that's kind of a done deal. Now he's interested in a different question, which is, what do we do with it? And, you know, what he would like to do with it is, you know, massively accelerated drug discovery. And so he's putting more energy into Isomorphic Labs, a company in London. But the point is, he, again, believes that you can do this with classical computing and you don't need quantum. So in one sense, you know, the answer to your question is quantum and AI will work together and in combination produce something even more powerful. In another sort of intellectual sense, it's just fun to observe that they are competitors, enemies. You know, the question is like, do we need quantum? That would be Demis's question, right? Can't we actually achieve almost anything with a Turing machine? Very sadly, we're running out of time. But final question for you, as for all our guests, clue is in our title. What's the one decision that you will be watching related to, let's call it the AI security debate? You may go more widely if you wish, but what's the one decision you're watching? The one decision is going to be the one about open weight models. Because at the heart of this debate is, you know, is your vision for the AI future one in which you have lots of providers of these very easily usable, quick to deploy, cheap to deploy, attractive systems? Or are you going to say that's too dangerous because the cheap deployment includes deploying it to terrorists? And will you therefore go for a world of fewer AI providers producing proprietary AI models that are government regulated, and more expensive, less good at diffusing into the economy and producing productivity gains, but safer? And I hope we go for the second. Thank you, Sebastian. There'll be many who share that hope. Thank you for a fascinating conversation. Well, Richard, that was a wide-ranging, thought-provoking, and occasionally unsettling conversation. So I think we got from existentialism and the Holy Father through to the chaos. And I think we got from existentialism and the Holy Father through to the capital markets and back again via China. I took rather a lot away from that to think about. What were the key things for you? I found it like, I was going to say, sort of ultimate expression of analytical sanity on AI. And what I mean by that is, he was neither too alarmist, nor was he sort of evading the difficult questions. I was jolly impressed also by the thoroughness of his analysis. Every question that we threw at him, he had obviously given a lot of thought to, because it wasn't as though it was a spontaneous reply. You know, he knew exactly what he wanted to say on each dimension, each issue. I found most of what he had to say quite reassuring to know that, you know, someone as sane as him had obviously examined the problem so thoroughly. And okay, he mentioned some of the big names, but by and large, he was painting a picture of people who had a sense of responsibility, who really cared about the frontiers of science that they're exploring. Yeah, look, I thought it was just a terrific discussion. And I don't think I've heard anybody and complete sort of tour d'horizon of what I would describe as the primary issues that concern the public with AI. So I think we've got a very incisive and insightful podcast there. Yeah, I thought he was extraordinarily composed and well-informed on a subject where an awful lot of emotion engages. I thought, too, that his realistic optimism on, for example, China and US eventually realizing their vested interests here in some form of regulation. I thought that was both welcome and it was well argued. He has, of course, spoken extremely interestingly on what he found in China. And he has these exceptional insights into the people who actually make up the hyperscaler leadership and membership. So I thought he brought both the tech, the economic, and the economic. Yeah, and I was fascinated by his last answer about quantum and AI and the Hassabis theory about taking a Turing machine to its ultimate potential. His actual analysis of that was more, well, it will massively speed up AI and make it even more powerful, which I think probably is the direction we ought to be thinking in. But it does underline, I was good in 820, on because it underlines the sort of scale of the technical challenges that we're facing keeping these products safe and in the right hands is clearly a it's going to be priority requirement for security agencies as well as the tech companies and i i liked his ai audit theory the sort of model that you might have a series of ai audit companies that operate rather like financial auditors as part at least of a national regulatory system i hadn't really heard that model before and i thought the analogy with the investment in railroads in the late 19th century was also a good sort of grounding point that puts your feet on something solid when you're trying to think about the financial implications of this new hugely important new technology going to be so so i mean there were so many aspects yeah which were really excellent and encouraging and um i i didn't know either the difference between the financial solidity of navidia and the financial vulnerability of open i was pretty terrified too no i thought he brought put out lots of nuances that that might have um passed the majority of listeners by he brought them out very well you know it's terrific to have an interlocutor uh who knows all sides of this including china incredible um and to be able to pull that together so i think he's the final hope that he expressed is one we probably all do subscribe to particularly as the as the bad news seems to grow around ai mishaps and bad actors so that is all for this episode join us every thursday for new episodes and subscribe to one decision on youtube so you'd never miss out on our conversation leave us a comment and email us at hello at one decision.com please do that and i'll see you next time bye bye check out one decision.com for commentaries and explainers that you might have missed i'm john rain thank you for listening ryan reynolds here from mint mobile i don't know if you knew this but anyone can get the same premium wireless for 15 a month plan that i've been enjoying it's not just for celebrities so do like i did and have one of your assistants assistants switch you to mint mobile today i'm told it's super easy to do at mintmobile.com slash switch upfront payment of 45 for three month plan equivalent to 15 per month required intro rate first three months only then full price plan options available taxes and fees extra fee-full terms at mintmobile.com starting a business can seem like a daunting task unless you have a partner like shopify they have the tools you need to start and grow your business from designing a website to marketing to selling and beyond shopify can help with everything you need there's a reason millions of companies like mattel heinz and all birds continue to trust and use them with shopify on your side turn your big business idea into sign up for your one dollar per month trial at shopify.com special offer