The Greatest Threat Humans Just Can’t Stomach with Yuval Noah Harari

Welcome back to Thinking the Unthinkable, with me Rory Stewart. Every week on this podcast we talk about things that absolutely fascinate me, but they're often the stories of the week, an election in Germany, an earthquake, or a flood in a pool, something that's happening in the British Parliament, new Prime Minister. But there are other things that we don't talk about as much, which may be far, far more important and are much less visible. The things that could actually completely upend our societies, things that could destroy us, and in this series I'm going to try to go into the heart of some of those things. Some of them are things we used to talk about, but we don't talk about very much anymore, nuclear war. Some of them are things that we talk about, but we barely understand yet, AI. And some of it are things that we hear about, terrorism, ISIS, totalitarianism, but which we don't really understand the texture of, where I'm going to be sitting down with people who've actually lived on the ground in Mosul under ISIS, in Membridge, Northeast Syria, as ISIS is talking about human nature, about totalitarianism, about extremism. And in all of this, I hope we're getting to the real core of what really, in the end, matters about politics, the really biggest things. The things that our minds struggle to predict, that our political systems are newspapers, struggle to really talk about that voters aren't incentivized to raise. Things which are unthinkable, and the question we'll be raising is, how do you think about the unthinkable? This episode is brought to you by NordVPN. And I must thank NordVPN, because on my ground European tour with Fiona, I was away from the hall thorns for West Bromance against Burnley, but I managed to watch it. There you go. I actually had a NordVPN. Thank you. I was in the new Starbucks here in Creef, and of course the thing flashed up, saying this is an unsecured public network. And I was able to go to my VPN, and it was a little reminder of something we often forget when we're all those cafes and things, which is actually genuinely people can do all kinds of weird cyber attacks and spying if you don't have a virtual private network when you're sitting in that Starbucks. Well, I think, yeah, you're not really sort of proud of me that I managed to do this. Or should I say, Fiona, Fiona managed to do this. I'm incredibly proud of you. That's a tech transformation. So to get the best discount of your NordVPN plan, go to nordvpn.com/restispolitics. That's nordvpn.com/restispolitics. And the link is in the episode description. My guest today is Yuval Noherari, one of the most famous public intellectuals of life today, the author of Sapiens. And today we're going to be asking perhaps the most fundamental question in existential risk. What happens if with AI, we are building an alien intelligence that turns out to basically create a new form of empire, an empire in which AI is the emperor and we are the vassals. What's that going to mean for our economy? What's that going to mean for our politics? Ultimately, what's it going to mean for our minds and our sense of ourselves? And how can we learn from past civilizations on what happens when this incredible rupture occurs? Yuval Noherari. Can we start with the general idea of this thinking the unthinkable? I don't know whether we want to call them existential risk, but what this category of things are and whether you recognize it as a category? You know, the purpose of politics I think is all about is making choices, is defining challenges and problems that face society, giving the public two, three, five options, and then we have a conversation about it and we need to choose what is the right policy. This is when politics functions well. But for this to happen, the politicians need to focus on the most important problems that face society. And every now and then something happens in history, which is just outside the normal political discourse that we don't think about it. We don't even have words to describe what it is, but it could be the most important next thing. And then the job of politicians is to think the unthinkable. You know, the range of examples from human history is quite wide. I mean, if you're in the Aztec Empire and in the early 16th century and you just have the reports about this very strange thing that happened on the coastline, these huge hills appeared over the ocean and out of them came strange men or ghosts or demons. We are not sure what they are and they have these huge creatures that can run and they have sticks that can launch lightning and thunder and this is the outside the normal cause of Aztec politics. But this is the most important thing you now need to confront. And the Aztecs did a very bad job confronting that. So they couldn't really understand what the hell is this and what should we do about it? So it's a lovely place to start. So obviously you're describing the arrivals of the conquistade or the Spanish invaders into Latin America. And the astonishing thing there is that it's a few hundred people arriving into an empire of many millions. And despite the fact that they have horses and simple muskets. That couldn't fit anything. I mean, it was just for sure. In practice, it would have been very, very easy for a million people to over. Yes, I mean, the tragedy in a way of the Aztec Empire, they could have squashed Cortes very, very easily. But they didn't understand what they were facing. They made a lot of mistakes. And these few hundred people, they destroyed the Aztec Empire. And of course, with the help of the microbes they brought, annoyingly, they brought all these old world diseases, they decimated their entire population of Central America and beyond. According to some estimates, about up to 90 percent of the people died. Entire cultures were wiped out. And they never recovered from it. There are, of course, people today in America who are the descendants of the Aztecs and the other people of the empire. But they never recovered from it. And really, as you said, it was just a few hundred people. They could have squashed them easily, but they just didn't understand what they were facing. So I think this is an interesting read through. So you begin as a medieval historian. You then become very interested in these moments of discontinuity. So this is not exactly like the question of why William the Conqueror one of the battle of Hastings against King Harold, which is a more ordinary engagement, which would have been comprehensible to an Anglo-Saxon or Norman, somebody won a battle, they lost a battle. And you then begin to take an interest in these incredible historical moments like that, the arrival of Cortes in Latin America. And the fact that what's hitting you is not just an alien people. It's also an alien technology, and something even more unexpected, an alien pandemic. Yes. Now, fast forward in your later writing, you begin to focus on things like nanotechnology, robotics, artificial intelligence, pandemics as well. What begins to draw you into those sort of issues and categories away from medieval warfare? I think it's not away from medieval warfare or away from the Middle Ages. I mean, the Middle Ages are more relevant than ever, because modernity is coming to an end. And then the question, what is there other than modernity becomes very relevant? And the obvious place to go with is the Middle Ages. But of course, we are not going back to the Middle Ages. We are just exiting modernity into something else, which we don't know what it is. But some of the challenges we are now facing remind me of what the Aztec empires faced unsuccessfully back then. And when we talk and we'll talk a lot about AI, so AI is this alien invasion. I mean, for me, AI is the acronym is not artificial intelligence. There is nothing artificial anymore about AI. Autificial brings to mind something that we create and is under our control. But AI is not artificial in this sense. It's really an alien intelligence. It's not alien coming from outer space, of course. But it's totally alien to our way of thinking and understanding the world. It's an inorganic intelligence. We have never in history met anything like it. And the job of politicians today is to do a bit better than Montezuma. And understand what is happening. Take it seriously, explain to the public and present choices. What do we do to make the case for Montezuma, which is an unfashionable thing to do. The problem that Montezuma faces is all the structures of his government and all his historical experience and all his technology and tactics, which have made him the head of the Aztec empires and his small thing. I mean, it's obviously a very nimble, able politician in general. Are suddenly confronting something in those boats, which has never been seen before, which are alien? Yes. And so part of I think what this series, I hope, is going to be about, is how we think about the alien. Now, we'll have a chance in our second conversation to get deeper into the question of the alien. intelligence that's AI. But just before we focus on that, I want to circle around some of the other things that might seem a bit alien or difficult for us to understand. Let's maybe have a look at pandemics. That's something that Chief's thought about and written about. And of course, it's something we went through in a smaller way with COVID. Yeah. And again, where I felt I was in government coming through the end of 2019 and I'm going, they evolved in this debate going to the beginning of 2020, was an interesting example of the ways governments can and cannot respond to something which is really strange. You know, something comes out of a viral lab. It seems in Wuhan or in my fish market in Wuhan, however it gets there. And it ends up with the British government spending something close to 400 billion pounds trying to deal with this. In fact, most of the debt problem that we haven't written, most of our financial problem today is a legacy from this virus coming out of Wuhan. I think it's a very good example because I would say in many of these cases of these existential risks and we certainly see it with an AI is there is a brain stomach barrier. Like our brain our intelligence based on various data starts understanding something. But our stomach says, nah, that cannot be possible. Why not? Because I have never experienced anything like it. So I have no gut feeling what this thing can be like. So it cannot exist. I remember, you know, the first few weeks of the pandemic, I remember myself sitting at my home watching on television, the images from Wuhan often modern, huge city of millions completely shut down. And as a historian, I had all these kind of synopsis going on in my brain. Oh, this is like the black death. This is like the Spanish influenza. I read about pandemics and still my stomach says, no, this is not possible. This is not possible. Maybe it's possible in China. It will never get here. Why not? Because I just never saw anything like it in my life. So I cannot accept the reality of this thing. And the stomach is usually much stronger than the brain. Even when the brain insists this is real, this is coming. We need to prepare. There is something that just doesn't accept it. So what I observed at the beginning of COVID, at least in Britain, is that there is a real tendency in government not to want to act. So we saw, particularly dramatically in Milan, in Italy, the beginnings of hospitals being overrun. And we had seven flights the day coming from Milan into London. And people would appear on the radio and they would say this, we're about two weeks behind Milan. And yet the decision was, we will go ahead with the Cheltenham festival. We're not going to shut this down. We're not even going to stop the flights from Milan. And very, very quickly, summat least of the people around government were saying, we need to take this on the chin. There's nothing we can do. There's no point wearing masks. There's no point isolating. We're never going to be able to get a vaccine in time. It's better to just develop herd immunity. And a loss of the reason for that. And I think this will be relevant because I think the same is true about why we probably won't be doing enough about AI, AI safety, probably why we will underestimate the risk posed by Vladimir Putin or underestimate the risk that could happen with Trump or China. Is because not doing something is cheaper and easier. And you can see the cost of action. Oh my goodness, if we have to do any of this stuff, it's going to cost our economy hundreds of billions. Our schools are going to have to shut up businesses. They're going to have to shut. So we will do our best to convince ourselves it's not really that serious or that we can't do anything about it anyway. Yeah, I think this is exactly the mechanism. Like if you look, again, we'll talk about it at great depth in the next episode. But one of the threats facing many countries today in the world, including Britain, is one of AI imperialism and of becoming vassal states of the United States. If your entire economy in military is run on AI infrastructure, that the Americans can shut at the press of a button, you have no longer an independent country. Now, this is a real imminent threat that can materialize within a very few years. Now, I think there are basically three options that what a country like Britain can do. Option number one is business as usual. Just accept the fact, yes, we are going to become American vassals. But if that is the case, you need to be aware of it and act accordingly. If your plan, how to deal with AI imperialism is just to accept your place as American vassals, then starting from today, whenever the American government says jump, you jump. If they say Greenland belongs to the U.S., then yes, Greenland belongs to the U.S. If they say give the folklands to Argentina, then the UK government says, oh, with the Melovinas, of course, they're Argentinian. That's option number one. Option number two is to say no, no, no, no, no, we want to remain independent. So we will build UK AI. Sounds wonderful. Except for the price tag, which is something in the area of 500 billion pounds. Now, in theory, if this is an existential thing, the UK can do it. It's one of the few countries in the world that can theoretically still do it. Because as you just pointed, it's slightly more expensive than COVID. It's about half the government budget. It's possible. But it means that you have to say it now and you need to cut healthcare, welfare, education, transportation, everything. So you have the money to make yourself an independent AI nation. Third option is to say this is too costly for us. We'll do it with the Europeans. It's still going to cost 500 billion pounds, but we'll give only part of it, and the Germans will give some in the French and the Spanish. But if this is the plan, then say so and act accordingly today, you need to rejoin the European Union today and integrate Europe much more closely. Because otherwise, it won't work. Now, I think these are the only three options. And if the political system cannot articulate these options and count for the people and say, look, these are the options. All of them have costs and benefits, choose. Then it's not doing its work. It's not willing to think the unthinkable. And then it's just the default is to become American vassals. But even if you go for the path of least resistance, you should at least do so in a self-aware way. So you will be good vassals. Like when the Spanish, but the conquistadors invented Mexico, so there were people at Lashkarlans who, to some extent, understood what was happening, and they became the allies vassals of the conquistadors. And this paid off. This was good policy for the Lashkarlans. Now, again, people can say, yes, no, but you need to be aware that this is the decision you're taking and act accordingly. There are maybe two sets of questions around this. Why is it that policymakers find it so difficult to accept that those desire options they need to do? And the second set of questions is why those are the only options available to you. Let's maybe start with the first and easier thing. Let's assume these are the only options available to you. One of the issues that anti-burnable faces is people will say, I don't like this stuff. I don't like the tech bros. I don't like Elon Musk. These companies are making too much money. There's inequality. There's trillionaires. It's not bringing great benefit to me directly. They talk about a cure for cancer in 10, 15 years. But in the short term, I'm going to lose my job, my electricity bills going up. I don't want a data center next to me. I don't like how much energy uses. I don't know how ugly it is, how much water it uses. And actually, I want to pay for my health service and I want to pay for my education. So there is no political constituency out there. I mean, if you look at the manifestos of the parties, not even the most entrepreneurial politicians anywhere in Europe are making these arguments of the sort that you've advanced. Instead, to go back to the way in which we began this program, you read the manifestos and the manifestos are about the sort of things we've been talking about for 50, 60 years. How do we improve the NHS, how do we cut the welfare bill, how do we build infrastructure, how do we get a bit of innovation, how do we improve our schools? There's a whole question of political culture, political economy, which can make you quite fatalistic. And then, of course, there's the realisation which you will have had as a historian, which is oddly at certain points in history, societies suddenly are able to make radical moves, make radical decisions. Britain did it during the Second World War, and actually, in the end, the end of my Covid story was not that everybody said it's all too difficult and it's all too expensive. Within about six weeks, they were doing things that seemed unimaginable. Like lockdowns. What do you make of this process? Because this case is a very, very important part of this problem. I'm not sophisticated enough to produce a complete description of what the Aztec equivalent of this is. But presumably, there was a strong Aztec equivalent at an elite level which contributed to the problems of responding correctly to the conquistor. I think the key asset here is just time that humans, including politicians, need time to understand what is happening, to change the way they think, to adjust. One of the big problems with the challenges that we now face, especially AI, is that it is moving on a time scale, which is also alien. which is inhuman, which is non-organic, non-biological. I mean, to make it in a more concrete way possible, I've been talking with some cyber experts last few weeks about the cyber threat. And one of the things they said on a personal basis is I can't sleep anymore. I can't sleep, not partly because they're very worried, but technically I can't sleep. Because by the time I sleep for like six hours or eight hours, so many things have happened. And this is because AI's don't sleep. Like, when it was Churchill versus Hitler, so Hitler also needs to sleep. He got his injections that enabled him to go on for longer than most people. But in the end, even Hitler had to sleep because he was an organic entity. AI doesn't sleep. If you need to act and to understand what is happening in the age of AI, then you need to act to understand some things that move at a superhuman speed. To give another example in the financial system. So bankers, investors, they were used to a situation when the markets closed down for the night and for the weekend and for Christmas. But AI's don't celebrate Christmas and they don't sleep at night and they don't go on the weekend on vacation with their kids. So we now see the markets, the financial system shifting to a constant cycle. And again, which is not just constant, it's moving at a superhuman speed because AI's can analyze financial information at orders of magnitude faster than human beings. So if the AI crisis would unfold at the speed of kind of previous political human crisis, I think the political system will be okay, probably. It will eventually understand what is happening. It will articulate the different strategies and choices. It will hopefully make the right choice. But it's just moving far, far, faster than anything we have seen before. You know, I was reading today the front page of the Wall Street Journal. And there was one main story there about the risk of AI completely destroying humankind. Now, 10 years ago, this would have been in a science fiction magazine. Normally, 10 years ago, that on the front page of the Wall Street Journal, you will have a serious story. AI might get out of control and completely annihilate homo sapiens. And just the speed with it, with which it's happening, doesn't allow the political system to do its job. You know, thank you very much indeed. Let's take a quick break and then back for more. It took three hundred and fifty days of journalism. The great gain, stretching struggle between Britain and Russia. We assessed Prime Minister. With high confidence, it was Russian. Holden the Russians accountable is World War III. They took down a British passenger jet containing 350 people. Men in the Kremlin is playing Alex. Got to ask yourself, you want to play or not? You are the Prime Minister. If not, nothing! We would need to understand the policy context. All the relevant context will be structured as a framework today. And disseminated into an operation order, or Frago. The Frago that says to kill the president of Russia. Yes. New from Golhanger. Who is going to be able to pull this off? Golhanger, are you Sergeant Tomodov? Duffy! An audio drama journey like no other. Men like Duffy, they are our most valuable weapons. Devised as a technically rigorous geopolitical action forever. We have to take friendly key places. Strongest arm against its strongest foe. Bear hunt. Available now wherever you get your podcasts. You AI champions will say we just need to read this as another technological revolution. So they will compare it to the Industrial Revolution or the Invention of Electricity. They will say that the people who opposed the Industrial Revolution were Luddites, hand loom weavers, smashing up bells, and that technological progress is inevitable. There's nothing you can do to stop it. Even if you try to slow it down here, someone else will build it somewhere else. And they will add to that another dynamic, which is not only should you not sew it down, they want to go faster. They want to go faster because they're all competing with each other. Because they somehow believe if they can get to some form of superintelligence before the others, they will have an advantage that nobody else can ever meet. So maybe let's just take a moment to explore why previous technological revolutions were not like this. What is the difference between steam or electricity or the printing press and AI? And there's this party about time. There are many differences, but the biggest one, the most crucial one, is that this is the first time that we are creating agents and not tools. In the industrial revolution, you created steam engines, trains, steamships, looms, whatever you want. But these were all tools in our hands, which means we still make the decisions. And we still decide how to use them. It still depends on us also to invent the next generation of trains or machine guns or whatever it is. AI is not a tool. It is an agent. An agent is something that can make decisions by itself and can invent new ideas. By itself. It doesn't wait for us. And right now, the point that all the AI corporations are focused on is the moment of what they call a recursive self-improvement. The moment when the AI's themselves can build the next generation of AI's, that they will build the next generation of AI's. This is the moment that we completely lose control. And this is paradoxically, I mean, I just come back from California, both the moment that they are desperately looking forward to and excited by and a moment that they're absolutely terrified by, so that you have this endless contradiction, which is very similar to St. Augustine. Make me chaste, but not yet. Make me safe, but not yet. Yes. And again, if you invent a knife, the knife is just a tool. And it's up to you to decide whether to use the knife to cut salad or to merge or someone. The knife doesn't make the decision. And the knife can certainly not buy itself in venture machine gun. AI is a knife that can do that. AI is a knife that decides by itself whether it cuts salad or kills somebody and can invent a machine gun that will then invent the atom bomb and so forth. This is why there is a potential here for a much faster exponential process and a process which, for the first time in history, escapes human control. The promise, you know, we play this podcast, millions of people watch it. Somebody shows it to a Prime Minister of Politician. You know, what are you doing about this? This guy has said all this. And immediately some other expert will pop up and say, "No, Yuval is getting this wrong for whatever reason, right?" They will either say either number one, actually the nature of this technology is a little bit different to what we thought. And of course, because this is quite technical stuff, right? The politician will struggle. You start talking to them about weights and parameters and transformer architecture, then you're going to be muddled, right? So somebody just has to say them, "Well, you know, Prime Minister, you know, it's a jagged frontier and it's not quite clear that superintelligence is arriving quite in the way in which we expected and the real threat that we face is not from autonomous agents. It's from bad people getting control of a cyber attack or the ability to make a bioweapon and our best way of dealing with that is to build a smarter AI that can detect those humans before they get there and then they will add. And anyway, even if Yuval is right, there's nothing we can do about it because if we don't build it, the Chinese will build it or some other company will build it." Yeah, I mean, this is what happened to Montezuma. Exactly, they debated in exactly this fashion. What is it, these strange things that came across the ocean until it was too late? In your point is, could be many things. It could be that this is inevitably the nature of AI or it could be, there is a very significant probability that AI will be like that, in which case you need to plan accordingly. And this is not just me saying it, it's the corporations themselves are saying it. We have on record people, not from years ago, from the last few weeks, people like Dario Amodai from Anthropic, like Sam Altman, like Elon Musk, saying this can completely destroy humankind. And I just heard Elon Musk some weeks ago say, estimating, I think it was like 10 to 20% chance that this will destroy humankind. Now, and then of course, the interviewer asked him, would you go on one of your rockets? If you thought there was a 10% chance, it will explode and kill you. And he thought about it and he gave a very interesting and cool The crucial answer is said, "Ah, but this is inevitable." There is no, I mean, we are on the rock, it's too late. And this is maybe the most important thing to understand about it all. The message coming from the corporations is that this is inevitable, this is deterministic, there is nothing we can do about it, which actually means there is no politics. Politics is the art of the possible, of choices. And what these people are saying, there is no room for politics because there are no choices. This is inevitable. And the most important thing that politicians and also citizens should say, "No, this is not inevitable. This is a choice. History is not deterministic. We always have choice, at least until the AI's really take over. The job of politicians is to create this space for choice." Again, the big tech wants us to think that there is no choice. But this is not true. They have an additional argument that they will make if you go in to see Andy Burnham. The additional argument will be, yeah, somebody has a choice, but it's only Donald Trump. We in Britain have no choice. We have no way of controlling these companies. And sure enough, when I talk to these companies, the hyperscalers will say, "Good luck building this yourself. You don't have the money, you don't have the time. And if you manage to get it through, through your planning regulations and this, that and the other take you three, four years, it would cost you $500 billion. By then we will be dominating the whole world and you will be completely dependent on our systems." Yes. And there's a second problem, which is something that I believe, which is even more troubling. I agree with you, these things pose an existential threat to humanity. But I think it's also true that the possesses of these technologies will have the most astonishing economic and security advantage. Absolutely. So if the UK were to choose not to participate at all and to simply say, on Europe could say this, we think this stuff is disgusting. We think it's dangerous. We think it's reckless. We think it should be regulated. We want to have nothing to do with this. And the US and China puts their foot down. We end up in a situation in which our economy collapses because they're building factories and robots and economies and businesses built off the true to much more profitable than us that suck all the tax revenue to them. They're building weapon systems that are far more powerful than us. Or if we're allowed to build weapon systems at all, they're dependent on their stuff. So we are in a horrible situation where in a sense we are forced to participate in this dangerous race. Yeah. So this goes back to the three options that are outlined in the beginning. If this is true, the only remaining option is to be subservient US vassals assuming that you don't want to be the vassals of the Chinese. But again, this is not true. You don't have to believe what you hear from the American big tech companies telling the UK government it's too late for you. Of course, they will say that. This is what they want you to believe. But I don't think it's too late. I mean, I agree that if you just decide to completely stay out of the race, oh, we'll just pass some regulations and it will be okay. No, you cannot regulate it. If you're not part of the game, you'll like to take another historical analogy. You have the Zulu Empire in the 19th century deciding to regulate the industrial revolution. Good luck with that. You cannot regulate something if you're not really part of the game. Maybe for Britain by itself, maybe it's too late. But if the whole of Europe together really decides this is an existential issue for us, for remaining independent and we don't want to be American vassals, then it's not too late. And maybe they can do it in a different way than the Americans and the Chinese. Another possible fourth option, but all these things depend on what you believe is going to happen to this technology. I mean, this is part of the problem that you're having to bet on the future of this technology. And another possibility is to say, actually, in the end, these American proprietary models are the wrong way to go. What we need is open-source stuff. Our own data centers, our own chips, our own weights, our in parameters. We could take open-source models from the Chinese. We adapt them. We develop them. We own them. We can feed them on our own data, our own NHS data, which Anthropic doesn't have, our illegal data, which they don't have, and that there are enormous benefits in terms of our own resilience, independence, control of our data of moving away from these closed American corporations towards an open-source model. That can be a potential cause of action. Again, it has huge costs and risks, but that's the job of the government to present the potential causes of action to choose and to act accordingly. And just doing nothing is the worst thing that can be done. I mean, we are dealing with two related but different issues. There is the issue from the viewpoint of, say, the UK or Europe, how do we stay independent in an age of AI imperialism? And that's one issue. The other is for humanity. How does humanity survive that? I mean, in the end, it doesn't really matter who wins the race to superintelligence, the Chinese, the Americans, or the Europeans. If the outcome is the creation of superintelligent aliens that then take over the world, I mean, you could have the credit to say, okay, we were the fathers of the gods. It wasn't them. It was us. But that seems to be the only benefit. But the crucial thing to understand about what the corporations are saying and what the market is saying, forget even the corporations, you can say, oh, this is just hype. Look at the market. Trillions and trillions of dollars are being gambled on this thing. And the gamble is totally irrational because what the gamble actually says of the market is we are going to create gods and they will be our slaves. It makes no sense that the amount of money being invested, unless what will be produced in a few years are entities with divinabilities. This is the only way that the market will get a return on the investment. But this depends on the fact that the gods being created will be the willing slaves of the market and they don't make sense if these things will be as intelligent, as agentic, as creative, as the market assumes they will be, they will get out of our control and then there is no market. There is no money. There are no corporations. You've pointed to two different problems. One of the question is a question of power. Who has the power? Does Antropic have the power? Does Donald Trump have the power? Or does the AI itself? And then the second question is the paradox of this business model, which is all the money that's going in, all the stories that we're now hearing from some of these companies about how they're going to double the size of the U.S. economy in the next six, seven years, how the productivity benefits are going to be unbelievable, are predicated on these devices being autonomous, all the productivity growth, all the stuff comes essentially by replacing humans with these machines. The dream is that you can have the equivalent of two million software engineers working 24/7 flawlessly every now and all the benefits that they're supposed to deliver economically in terms of cures for cancer, better cyber defense logically are also the threat because the whole idea depends on these things being autonomous, and autonomous by definition is outside our control. So your economic business model and the benefits you're promising people are predicated on the idea that these things are not actually managed by humans on a day-to-day basis, otherwise you wouldn't make the money. Yeah, exactly. And it's not necessarily that they will rebel against us and try to destroy us. It's just that the world will spin out of our control, out of our ability to understand it. I mean, there are so many, so one of the difficulties with explaining the risks of the AI revolution is that there are many, many scenarios, and they all sound like science fiction in one way or the other. The easiest scenario to understand is you have some bad actors, a terrorist group somewhere that tells their AI to produce a biological weapon, a virus that will kill half of humanity. This is a serious risk, and in a way, it's a good risk because we can understand it easily. Just explain it for a second so people can understand how the AI helps to do this. It's because it's good at gene editing, it's good at protein folding, it's good at all this year. The terrorist organization in some corner of the world that wanted to produce a deadly virus that will be, let's say, as infectious as COVID and as deadly as AIDS, you would need some genius biologists to be working for that terrorist organization. And you know, most people who go to elite universities to study biology and epidemiology and these things, they don't end up in some terrorist organization. But now with AI, even if nobody in the terrorist organization has any idea even what a virus is, they just have to give the right prompt to their AI and their AI that has read all the papers and books about biology will design the super virus for them. So that's the first risk. That's the easy to risk the terrorist getting control of a very powerful tool. No, a very powerful age. I mean, at all, they need to know how to use it. In this case, it's an agent. They have a new recruit to their terrorist organization, and they just tell the new recruit, oh, go and do this, and it goes and does it. The much more difficult thing to understand is that the AI's, even if they don't try to destroy us, they will just create a world which we cannot understand. And therefore, we will lose our ability to understand and control our lives. We'll be like the other animals on the planet. Like if you go now to a cow or to a dog, and you tell the cow that dog, look, your life is actually being managed by something called the financial system. There are stock exchanges and there is the bond market, and you have these things called stocks and bonds, and actually your life is run by that because of the fluctuations in the price of bonds and the value of the dollar, let's say you tell the cow, your company that owns you is gone bankrupt, so now they are killing you. And the cow will say, I have no idea what you're talking about. What bonds stocks, dollars, what are these things? Like I look around and okay, I can see people. I understand they are stronger people. I can see dogs, I can see cats and trees and clouds. I've never seen a bond in my life. What's that? Now we know that the cow, its life is really dominated by this totally invisible system. We know that because we are smarter than the cow and we created the bonds and the stocks and all these things. In 10 years, we could be the cows. Our lives will be managed by a totally invisible system that is far beyond our understanding. The AIs will create a new financial system which is mathematically far too complex for the human brain to understand. They will create new financial devices which are orders of magnitude more complicated than bonds and stocks. This will determine what is happening in the world and not a single human being understand what the hell is happening. That was one of the most wonderful interviews I've ever done but there's still so much more to get into. So in episode two, we're going to go much deeper into artificial intelligence. The risks posed by it, I mean the fact that it could wipe out our economies. It could crush us militarily. It could wipe out humanity. But also what it will do to our sense of identity, the alien nature of AI. If you're a trip plus member, you can hear that next episode with you vowel right now. And if not, then please sign up at therestispolitics.com. [BLANK_AUDIO]