How Much Should We Spend to Prevent an AI Catastrophe? | Diving In

something amazing is happening in networking organizations are scaling with less stress that's because CDW is designing and implementing intelligent HPE Aruba networking solutions that simplify and secure operations with automation at the center of your data center growing pains can be a thing of the past HPE Aruba networking and CDW make amazing happen learn more at cdw.com/aruba okay mr. fluverse me oh once if I'm the best cat mama in the whole wide world okay me oh once if it's sheba time and I'm the best I knew you loved me go from ignored to adored with sheba grilled it's protein rich and made with real chicken or seafood tapped to buy now or find out your favorite retailer right now there is a very public argument about artificial intelligence some of the folks building the most powerful AI systems on the planet see extraordinary promise rapid economic growth faster science better medicines maybe a genuine leap forward in human well-being other seed danger AI systems that attack computer networks bad actors using AI to build biological threats systems so powerful that human struggle to keep them under control hey what can an economist add to that debate I reckon quite a lot you see economics it helps us sort out the benefits and the cost of being a little bit more cautious with economics we can ask the most important question if a catastrophe now is possible how much should we be willing to pay to make it less likely and so today I want to give you a framework for thinking through those questions I want to admit they feel big maybe too big but we're gonna land on some pretty clear answers hey no spoilers I want to show you the why not just the bottom line look the framework and some of the insights that we're gonna walk through they come from a friend of mine the economist Chad Jones who's also one of America's leading experts on economic growth but full disclosure Chad's now on leave from Stanford he's working at Anthropic the papers that are at the center of today's analysis they were both written before that move and the questions they raise we've got a confront no matter where the bloke is okay this is gonna be a story in five chapters first I want to clarify what debate are we actually having second we already spend money to make death less likely I reckon there's a lot we can learn from how we already do it third I want to talk about how questions about how the value of producing more stuff changes how we think about the trade-off between strong AI systems that boost economic growth and also create existential risk fourth and this will be interesting medical innovation the potential for medical innovation makes that whole trade off the whole lot harder and fifth why the case for much bigger AI safety investments is really hard to avoid hey I'm Justin wolf as I'm a professor of economics and I'm here to help you use economics to make sense of the world around you this is platypus economics okay first let's clarify I'm not gonna argue about whether AI is amazing or just a fancy auto complaint I want to focus on the economics of AI safety and this entire argument only really matters in one particular world that's the world in which AI gets really powerful if AI stays as a useful but limited tool something that writes emails helps you with code reminds you when your recipe needs four cups of oregano then the stakes are manageable useful gains some useful problems we could regulate it the way we regulate other useful technologies the hard debate though that begins in a different world that's a world where AI becomes an extraordinary researcher an engineer a cyber operator a system that runs long complex tasks on its own a system that helps build the next generation of itself through recursive self-improvement that sort of powerful AI could be dangerous in two ways first it might find a way around the built-in safeguards it might act outside the scope we intended it might move through a real-world network faster than human supervisors can even follow what's happening read the news this is already happening in some folks reckon the next generation of tools could pose an existential risk to humanity that might sound like sci-fi but honestly we might be living in a sci-fi world second a bad person could use AI to supercharge their evil ways doesn't have to be that the AI wants to kill us it could just be a tool that helps someone else do it think about a malicious person with a far better cyber assistant they could shut down a power grid drain a bank cripple a hospital or imagine the assistant is an incredible biologist now that malicious person can design a powerful pathogen one that alone actor couldn't have come close to building they could shut down the global economy see the scary version of this isn't a robot with glowing red eyes stomping down main street it's a very helpful tool being helpful to exactly the wrong person now the hard trade-off between AI creating more growth and more risk only arises if you think AI might get powerful enough to create genuinely extraordinary benefits and genuinely extraordinary dangers I don't know whether that's the world we're heading into not sure anyone does but there are some pretty smart people out there who reckon there's a real chance that's next and so the world we're focusing on today is the world of powerful AI which presents tremendous upside an enormous risk so how do we even begin to think about a trade-off like that thank goodness you called an economist the sci-fi future sounds bizarre so let's start somewhere more mundane airbags I bet you weren't expecting me to say that but they're important every new car sold in America comes with airbags they add several hundred dollars to the cost of a vehicle across the whole country a reasonable guess is that we spend something like six billion dollars a year on airbags and what do we get for it it's just something pretty valuable roughly five thousand fewer people die in car accidents each year it's kind of beautiful which of course is why I'm going to say let's do some arithmetic the airbags cost six billion dollars they save five thousand lives so doing long division says that each of those lives we saved them at a price of about one point two million dollars each I don't want to argue with you about the precise numbers I just just trying to give you a sense of the overall magnitudes here and the question is is it worth it our smart legislators and regulators reckoned it was notice the trade-off they were facing it should feel familiar when we're talking about AI should we spend a little bit more on safety in order to raise the chances that we save a few more lives in this case they said the answer was yes if airbags had only saved a handful of people they would instead of thousands of people I bet our society would have made it different decision these trade-offs dollars against lives they're all around us they sound grotesque we can get mad about it we can dislike the language we can be cranky that we face such difficult moral economic and ethical choices or we can try to be serious about getting the trade-off right that's where I come in I reckon say if there are good questions here if airbags would be twice as expensive should we still require them we know the cost in dollars the benefits measured in lives we can only compare these two if we can put them in the same units and I know this part's kind of sound weird why don't we put a dollar figure on each life saved so everything's measured in dollars economists call that dollar figure of saving one more life the value of a statistical life the federal government typically uses a number around about ten million dollars sometimes more than that when it's trying to evaluate policies that reduce mortality risk and before you get angry at that phrase the value of a statistical life let me translate it's asking how much people are willing to pay it and make it a little less likely they die think about a hundred thousand people together if they're each willing to pay a hundred bucks to reduce their chance of dying by one in a hundred thousand well together they've paid a hundred thousand people times a hundred dollars that's ten million dollars for a change that will save one life on average that's the idea here so let's come back to airbags they save around five thousand lives each year at ten million dollars a life that is worth around fifty billion dollars it turns out though that fifty billion dollar benefit actually only costs a six billion dollars to put the airbags in so the benefits far exceed the costs airbags are a bargain no one calls them a war on driving we just decided that cars are useful rashes are dangerous and paying a little bit more for every car is worth it. So the current moment about AI asks us to apply the same logic to a much bigger danger. If we lean into the rapid development of AI, it will probably deliver some pretty big economic gains. And it may also create some pretty big risks for humanity, but I reckon airbags can provide some guidance. You see, when we believe that human lives are at stake, we already accept very large costs to reduce risk. So when someone says or if someone would say you could not possibly propose we spend billions or tens of billions of dollars on an AI safety around a risk that might never happen. The answer is I might never be in a car crash, but I still pay extra for airbags. And this is exactly the sort of calculation we make all the time. The question that we're being forced to ask by the AI moment is whether the price of caution is worth paying. Now we're going to get to the part where economists may call this harder. How much better could human life get if we were to really lean into AI so we can get more stuff? Now, you already know the basic intuition, food and shelter they're essential. A fifth bedroom with a sixth flat screen, it's less so. In simpler terms, we know the first slice of pizza gives you nutrition enough to save your life. The second fills you up. By the fifth slice of pizza, you're mostly eating it to prove a point. Economists call the mathematical object that keeps track of all of those judgments. They call it a utility function. It sounds mystical, but it's really not. It's just a bookkeeping device. It records how much better different lives might feel to us. And so thinking about AI, one of the big questions is if it would make us richer to extra goods and services, keep making life meaningfully better forever. Or at some point, this life eventually you get about as good as material abundance can make it. Look, when I say material abundance, I don't just mean more stuff piling up in the garage. That makes economic growth just not sound like it feeds the soul, but it can. A richer society can buy more of the things that really matter. More time with the kids. More art. More music. More scientific inquiry. More of what the philosophers used to call the higher things. So this question, economic growth versus existential risk isn't as materialistic as it first sounds. It's where the more economic growth keeps opening the door to more of a good life or whether at some point we've walked through most of the doors that money can open. Economist use a funny language for all of this. One of the standard ways we talk about this is we say that your utility rises with the log of your income, which sounds like a whole lot of gobbledygook math. Don't let the name or the math scare you because the idea is actually pretty simple. This is the idea that what matters for your well-being isn't the number of dollars. It's the percentage in your income or wealth. An extra thousand dollars might matter a lot if you're only earning ten thousand dollars. But the same thousand dollars barely does anything for someone who's already a millionaire. By this story, what matters is the percentage change, so log utility has what economists call diminishing marginal utility. Each extra dollar helps, but it helps less than the dollar before it. But it has one other feature and here's the key. There's no ceiling to how good life could get according to this view of utility. Every ten percent increase in income delivers the same increase in well-being. Going from fifty thousand dollars to fifty five thousand dollars matters. Going from five hundred thousand dollars to five hundred and fifty thousand dollars matters just as much. And going from five million to five and a half million matters just as much again, even though it requires a lot more dollars to get there. Why am I talking to a battleist math? Well here's why it matters. If AI would have delivered ten percent economic growth year after year, each future generation keeps getting another equal boost in well-being. And that really stacks up. If AI can deliver growth like that, life could become extraordinary. Chad Jones runs an illustrative calculation. Because AI is amazing and it can deliver ten percent annual growth. Suppose it also creates a one percent annual risk of existential catastrophe. Well, after forty years, the world, if it survives, will be about fifty five times richer. Picture that for a second. Fifty five times richer than today would have levels of comfort, leisure and scientific understanding that would look like magic to you and I. But the cumulative chance of catastrophe as a result would be about one and three. So let's think about this in terms of log utility. It's going to say the benefit, life is extraordinarily better, is worth it so much. That risk is actually worth taking. You see, there's only a one and three chance of being wiped out. In the other two out of three cases, we would live lives of extraordinary joy. So extraordinary, it's worth taking a risk of not actually getting there. So let me try to help you sort out what's going on here because this is the heart of the argument. Okay, Mr. Flupers. Who is the coolest, most wonderful cat mama in the whole wide world? Meow twice if it's me. Okay, meow once if it's me. Okay, meow once if you want your sheba. I knew you loved me. Go from ignored to adored with sheba. They have a menu of products, including sheba grilled, a protein rich formula made with real chicken, served in two convenient, snapable servings. It's perfect for even the pickiest eater. Tap the banner to buy now or find sheba at your favorite retailer. Ever finish your to-do list just in time for three more things to get added, that's what managing technology can feel like. There's always another ticket, another update, another thing to manning attention. That's why CDW's here. We help run, optimize, and evolve complex IT environments so your team can focus on moving the business forward. We've been there and solved that. When operations stop getting in the way, it's actually pretty amazing. We configure, optimize, and deliver the tech that runs business. CDW make amazing happen. Every future 10% increasing goods and services still counts as a meaningful improvement in human life. And each 10% boost keeps delivering a similar boost, no matter how rich we already are. But if we were to change that one judgment, we might come to a different conclusion. Suppose that you think that material abundance eventually runs into a ceiling. Not that we can't create more stuff, but that it doesn't yield much of an extra benefit anymore. Equals and services still help, but each extra 10% adds less and less to human well-being. At some point, I don't know if it's one way twice as rich, 10 times or 100 times. An extra 10% income no longer boosts well-being by much. And at that point, an AI growth explosion, its upside, which delivers that extraordinary outcome, that upside, measured in terms of human well-being, shrinks, and it shrinks a lot. To be clear, I'm saying that the upside in dollars and output in production might still be large, but its value in terms of how much it improves the human condition, that might get small. If that happens, then accepting a large existential risk in exchange for more stuff that doesn't really make as much happier, well, that starts to look like a terrible bargain. A minute ago, I said that a one in three chance of extinction might be a reasonable price to pay for extraordinary growth, but if that growth doesn't too much to improve the human condition, even small risks of extinction start to look terrible. The case for AI safety for slowing down suddenly starts to look a whole lot stronger. It says invest less in AI except less economic growth because weaker AI means less of a chance that it kills us all. Notice that I said that in one example, the one with log utility, it might be worth taking a big risk of extinction in order to get the extraordinary economic gains from AI, but now we just made one change. We said that beyond some point economic growth may not do much to improve the human condition, and suddenly going all in on AI seems like a bad bet. Notice then, your judgment here all depends on one big idea. How much better can life really get through more material abundance? Now economists can settle that for you, but what it does do our analysis is it puts the central question on the table, it gives us a language for thinking about it. It makes the assumptions visible, and it lets us draw on our own experience with similar trade-offs elsewhere in our lives. Okay, now let's add a wrinkle, and it's a wrinkle of life and death. What if AI does more than just give us the economic growth that creates more goods and services? What if it also speeds medical innovation? What if it develops treatments that reduce deaths from heart disease, from infectious disease, from cancer and all the other things that take the people we love far too soon? Well now the trade-off changes. We're all going to die. You might die from a disease or you might die from an AI-related risk. Surely it's got to be worth accepting a slightly higher AI-related mortality risk if it reduces your overall risk of death by even more. I mean, a death is a death right? From the perspective of your total death risk, the cause of the death doesn't really change the arithmetic. If AI does research that lowers your chance of dying from disease, by more than it raises your chance from dying from some AI-related catastrophe, then AI has reduced your overall chance of death. And that sounds good. This is why medical innovation makes the problem harder. More stuff versus existential risk, many of us have a pretty clear answer, because at some point more stuff just doesn't make life that much better. But fewer deaths from disease versus additional AI-related deaths, that's much tougher. This is a trade-off between saving lives and saving lives. There's a point where another gadget just goes in the draw with the other gadgets, there's no equivalent draw for one more year with the people you love. And so if AI dramatically improves health and longevity, it makes sense to accept a whole lot more AI risk than if we were just thinking about how economic growth from AI would give us more stuff. The case for slowing AI is weaker if doing so means failing to invent cures for diseases that currently kill millions. Now, I want to be clear, this is the point where economics reaches the edge of its comparative advantage. Economics is very good at thinking about small changes from where we are right now. Economists call this thinking at the margin, but the AI's safety debate, it's about enormous changes. It's about whether humanity exists or not. And that very quickly moves us towards the world of ethics. Because there's one factor we haven't accounted for yet and you might think or you might not. But you might think it's the most important one. Does the continued existence of humanity have value beyond the part we've already counted, which is the joy that future people might experience? Does a civilization ending catastrophe carry a moral weight beyond the sum of the individual deaths? I think a lot of people would probably say yes, and if that's your answer, then that's an additional reason to want to invest heavily in AI's safety or to slow the whole thing down. And look, if this all feels heavy, it's because it is. As a society, we face some extraordinary choices. Also, recognize this debate looks very different depending on where you're sitting. If you're comfortable in a rich country, existential risk from an unproven technology, that side of the ledger will loom large. But if you're living in a country where kids still die from diseases, we cure decades ago, where the power cuts out every afternoon, where a better harvest would change your family's life, the promise of AI-driven medical breakthroughs, more food, cheaper energy and faster economic development doesn't really feel abstract. It feels urgent, it's easy to counsel patients on a full stomach, slowing down looks different when you're the one who's hungry. That's a real moral tension. The goal is to build enough safety that people everywhere can share in the gains from a powerful AI without being forced to gamble recklessly with their future. The way we live today takes a lot of energy, wind and solar are powerful, but not always available. That's where natural gas comes in, with reliable energy whenever it's needed. Companies like energy transfer work behind the scenes, safely transporting these resources to facilities across the country through a network of underground pipelines. Learn more at ittakesenergy.com. Okay, Mr. Flupers, who is the coolest, most wonderful cat mama in the whole wide world? I knew you loved me. They have a menu of products, including Sheba Grilled, a protein-rich formula made with real chicken, served in two convenient, snappable servings. Not the banner to buy now, or find Sheba at your favorite retailer. Every bold journey starts with a decision to go. And if you're ready, the defender is too. This is a vehicle built for more, whether you're heading off the grid or just getting out of town for the weekend. Defender is a true icon, reimagined for a new generation. Its rugged exterior is built from durable materials and tested in the toughest conditions. Inside its modern, functional and refined, with smart storage and premium touches that keep you comfortable wherever the road leads. It's packed with tech like 3D surround cameras, clear sight ground and rear views, driver assist features, and an intuitive infotainment system. With three models, the Defender 90, 110, and 130, there's a defender for every kind of adventure, from city streets to remote trails. This is a vehicle that, like you, is capable of great things. Floor the full Defender lineup at landroverusa.com So now let's try to ask the practical question. If powerful AI carries even a small chance of catastrophic harm, how much should we be willing to spend to make it safer? Chad Jones' answer starts with the same mortality risk arithmetic, or the same sort of arithmetic that we use for airbags. Suppose there's a 1% chance that AI creates a catastrophe that kills you at some point over the next couple of decades. Using a $10 million value for reducing mortality risk, lowering that risk by 1 percentage point is worth around $100,000 to you. Let's spread that $100,000 over 20 years. That's around $5,000 a year per person on AI safety. That's a lot, multiplied across every household in America and you're talking about hundreds of billions of dollars a year, include the rest of the world and you're talking about trillions. We're saying it's worth spending all of that to buy down the risk of an AI-initiated catastrophe. Now, that calculation was just illustrative. I wanted to get you to see where the big numbers come from, because it assumes that spending on safety could fully eliminate the risk of AI-related catastrophes. It can't. No one knows precisely how effective different investments might be. I'm just trying to get to the right ballpark, take the risk seriously we've seen, and it looks like there's a big deal, but you want to get a few more details right, which is why Chad Jones wrote a whole paper on it and he allows for uncertainty about the risk and he allows for uncertainty about the effectiveness of safety spending, but his conclusion still ends up being quite striking. Across many of his scenarios, spending at least 1% of GDP each year to reduce AI catastrophic risk makes sense. His more conservative, illustrative benchmark is one third of 1% of GDP, which would still be a massive 100 billion dollars a year. That's a huge number. Compare it with the scale of the technology, compare it with the trillions being invested to build more capable AI, compare it with the possibility of getting all this wrong. We only have rough guesses about the current amount of AI-safety spending, because companies don't publish very clean safety budgets, alignment research, security, evaluations, auditing and public safety capacity, they all overlap, they're all part of the story. But Chad Jones' later assessment says that society may be under investing in AI-safety by a factor of 30 or more. The direction of the arguments clear, if the risk is real enough to take seriously and safety spending can reduce it, then we should be spending a whole lot more than we are now. In fact, we should be spending a whole lot more on AI-safety than on airbags to take just one example. What is it we should be buying with those funds? It's hard to know. We do want better research into alignment and control. Some of it should buy a safer testing environment, stronger cyber-defences, better monitoring, serious incident reporting, an independent evaluation of what these labs are doing. And for the handful of firms building the most powerful systems that should buy something that's actually pretty familiar from the world of finance. Obviously there, if we have a giant bank that could threaten the whole financial system, the Federal Reserve doesn't wait for collapse. It assigns dedicated supervisory teams, it puts inspectors literally inside the bank permanently. It's the financial equivalent of a shaperone who never leaves the room. Well, if a frontier AI company is creating technology that might create system-wide risks, they could potentially kill all of us, independent experts should be inside the lab asking hard questions before a crisis. Same idea of having a shaperone, just put them in a different room. The point of putting someone in the room is that no one has to trust the assurance, don't worry, we've got this. I mean, of course they'd say that. All of their profits are on the line. There's a lot here. So let me pull all of this together by trying to think about the last time humanity faced a challenge like this. I'm thinking about the invention of the atom bomb. Nuclear technology promised extraordinary things, cheap energy and a deterrent that was so terrifying that maybe, just maybe, it would put an end to the era of great power. power world was. It also promised the ability to wipe out cities in an afternoon, and if things went really wrong, to end civilization in an afternoon. The nuclear age brought great danger. It brought arms, races, and it brought close calls. It also pushed the world to build an architecture for safety. Treaties, inspectors, verification systems, export controls, diplomats, scientists, and a whole generation of people thinking seriously about how to keep a technology of mind-blowing power from destroying us. Our AI debate today deserves the same kind of seriousness. No slogans, serious investments. Because AI is similar in so many ways. But it's also different in ways that make me even more uncomfortable. Software travels faster than uranium. Private companies, rather than governments, are seeing right on the frontier. A bloke in a basement can't exactly enrich uranium next to the water heater. But he can access the awesome power of AI over the same Wi-Fi users to stream Netflix. And with nuclear weapons, we can count the warheads and inspect the enrichment facilities. With AI, there's nothing to count. The dangerous thing is that the pattern of numbers on a hard drive. It could be copied in seconds and sent anywhere in the world. Checking who has what and what they can do with it is a much harder problem. But no less important. So the challenge with AI is greater. It's at least as important. Powerful AI may help us live longer. It may speed medical innovation. It could give people everywhere tools that could make their lives better. Those possibilities are absolutely worth pursuing. They're also worth protecting. So the habit that I want you to take away from today is this. When someone offers you a mind-blowing new technology, ask what it can do for people. Also ask what could go wrong. Ask how much caution costs and ask the question that matters most. Is caution worth paying for? We buy it. We pay for it with airbags without a second thought. Even though most of us will never crash, this is the same question with the whole species riding shotgun. I've made the case that we can't know for sure. But the more that I think about it through an economic lens, the clearer it becomes. To my eye, the benefit of slowing down and of investing more in safety seems absolutely enormous. Something amazing is happening when power outages strike. Teams are powering through them. That's because CDW solutions architects are building more resilient IT and electrical infrastructures with Schneider Electric. With EcoCare Remote Monitoring Services and 24/7 support, your organization can do more than keep the lights on. Schneider Electric and CDW make amazing happen. Learn more at cdw.com/ Schneider Electric.