One: Will MacAskill on AI causing a “century in a decade” — and how we’re completely unprepared

You should really pause and reflect on the fact that many companies, many companies now are saying, what we want to do is build AGI, AI that is as good as humans. And it's like, okay, what does it look like? What does a good society look like when we have humans and we have trillions of AI beings going around? What's the vision? Like, how do we coexist, like, in an ethical and morally respectable way? And it's like, there's nothing. We're, like, careening towards this vision that is, like, just a void, essentially. And it's not like it's trivial either. I am a moral philosopher. I have no clue what that good society looks like. Today, I'm speaking with Wilma Caskell, one of the founding figures of Effective Altruism, a philosopher at the University of Oxford, the author of What We Are the Future, and now a researcher at the Four Thoughts Centre, for AI Strategy. Thanks so much for coming back on the show, Wil. I'm so glad to be back on. It's been a couple of years since you're on, and since then, you've really reoriented your work a whole lot. You're now much more focused on AI-related issues. Can you explain what it is that you're, like, what your research agenda is and why you did such a big reorientation of your priorities? Sure. So, the area I'm working in, I'm currently calling AGI preparedness. The idea here is taking really seriously the idea of having human-level artificial intelligence in the near term. In the next few years or five years. Taking seriously the sheer scale of the impact that that might have. In particular, the idea of extremely rapid technological advancement. And then trying to figure out, what are all the challenges that that might bring? And are there any that are currently neglected, but that are in the same ballpark of importance as, for example, the risk of AI takeover or the risk of global catastrophe from new engineered viruses? And there's two ways of doing that. One is by really reasoning through and modeling the dynamics of what you will get once we have AI that can perform as well as a human at some crucial tasks. What's all the changes you would expect over the decade following that? And then seeing what kind of arises from that. A second is by thinking about, well, what would a post-AGI future look like? well, what would a post-AGI future that we feel really good about, what would that look like? And then reasoning backwards from there to think, okay, well, what are the ways we could miss out on that really good outcome? And so you asked about like, why am I focusing on this now? And well, one is just the crazy advances and progress we've seen in AI. You don't say. I've really changed my view on AI timelines. So I take really quite seriously the idea of human level artificial intelligence, even in the next three, five years. Take even much more seriously for somewhat related reasons, the idea of an intelligence explosion. So very, very rapid increase in AI capabilities and resulting technological change following soon after from that. And so that means, okay, this is, AI is the, you know, really big thing to focus on. Then in terms of why this is a particular area, one thing is just seeing how impactful early foundational work on misalignment, existential risk was. So, you know, I was part of the seminars for Nick Bostom's book, Superintelligence back in 2012, 2013. And at the time I found them kind of frustrating because I thought it was just kind of pie in the sky. It wasn't going to have much of an impact. And I've really been proved wrong there. It's just actually been enormously influential. And I think that's why I'm so excited about it. to pause the podcast and think, what would you expect the world to look like in 2125, technologically speaking? And then imagine all of those changes happening over the course of just a decade, so if it were now by 2035. And thinking, okay, there's going to be a lot going on. There's going to be a lot to deal with there. I guess things feel a bit hectic this week, but I guess it could get much more. Yeah, things feel hectic already. I already can't keep up with the pace of developments in AI. I can't read a hundredth of the stuff that I would like to read. That's going to get much, much more intense. So the paper is making the case for that, trying to make it vivid, then trying to somewhat systematically go through the different challenges that we'll face. And it's a bit of a litany there. And then finally addressing the idea that, well, for most of these, we can just punt the problem to the time when we have aligned superintelligence and then let that figure that out. And I think that's true in many cases, but not in all. And there's discussion of that too. All right. Let's step into the analogy of the century and a decade a little bit more. You have this kind of vivid illustration of just how much stuff would get crammed into how little time. Can you paint a picture for us? Sure. Yeah. So we're thinking about a hundred years of progress happening in less than 10. One way to get a sense. So just how intense that would be is imagining if that had happened in the past. So imagining if in 1925, we'd gotten a century's worth of tech progress in 10 years. And so we should think about all the things that happened between 1925 and 2025, including satellites, biological and chemical weapons, the atomic bomb, the hydrogen bomb, the scale-up of those nuclear stockpiles. We should think about conceptual developments. So a Game theory social science, honestly the modern scientific method Things like computers the internet AI itself deep the decolonization movement then of course Yes, social and political movements as well decolonization second and third wave feminism. Yeah fascism totalitarianism Totalitarian communism the rise and fall of communism exactly. Yeah, I mean post-modernism. So all of these things are happening over the course of ten years In this thought experiment human decision-making and human institutions don't speed up though So we're even just taking the case of kind of nuclear weapons. Let's say in this accelerated timeline There's a three-month gap between the start of the Manhattan Project and the dropping of the nuclear bomb in Hiroshima The Cuban Missile Crisis lasts a little over the day There's a close nuclear call kind of every single year It's hot like this is clearly going to pose an enormous challenge to institutions and human decision-making and Robert Kennedy senior who is a crucial part of the Cuban Missile Crisis Actually made the comment that if they'd had to make decisions on a much more accelerated timeline Like 24 hours rather than the 13 days they had They probably would have taken much more aggressive much riskier actions than they in fact did Yeah, so this thought experiment is to try and really get a sense of just the sheer amount of change including kind of multiple different sorts of change and We are talking about essentially in a decade I actually think that the amount of technological development we might get might be much larger again We might be thinking about many centuries or even a thousand years in a decade and then if you think about the thought experiment there It's like okay. You've got a medieval king Who is now trying to upgrade from bows and arrows to atomic weapons in order to deal with this wholly novel? ideological threat from this country He's not even heard of before while still grappling with the fact that his God doesn't exist and he descended from monkeys Which which they found out a couple of months ago, which they found out like two months ago exactly Again, it's just like this clearly the sheer rate of change Clearly poses this enormous challenge Yeah, so I think people might be thinking Well some stuff this is this little play out completely differently because some stuff would have is is it is possible to speed it up? Incredibly quickly as AI gets faster and other things not and that is kind of exactly your point that there'll be some stuff that will Rush ahead like I suppose, you know We could imagine that it could be a real flourishing of progress in pure mathematics very soon as a result of AI Things in the physical world might happen a little bit more slowly some some biological work that requires Experiments might might progress more slowly I guess anything to do with human institutions and human decision-making that slows down in relative terms and that is exactly Why well basically because some stuff slows down relative to other things We end up with whatever problems result from having a shortfall of the things that we couldn't speed up Is that is that basically the issue? Yeah, exactly. So if it was literally the case that just Time speeds up by a factor of 10. It's not even clear exactly what that would mean It's actually a philosophical question if that's even possible as in if that would be a meaningful difference Yeah But instead what is happening is that some areas and even you know the century in a decade? It's not gonna be exactly the same tech progress because like you say some areas of science are slowed by regulation or by slow physical experiments Or by just the need to build capital So building the Large Hadron Collider, for example takes a lot of time But the crucial thing is that human reasoning human decision-making and human institutions don't speed up to match the pace of technological development so a different way in fact, you could think of the thought experiment is Imagine if the last hundred years had happened in terms of tech development But humans just thought ten times slower or they were only awake an hour and a half a day. That's all of these changes Because you know a speed up in one way is the same as a slowdown in another So I think this helps to bring out the intuition of holy shit There will be so much chaos so much stuff Going on and we wouldn't have the ability to process it or think about our decisions very very well But I guess to flesh that out a little bit more What do you think of the things that would I guess go the slowest relative to other stuff that would create the biggest bottlenecks? And and the biggest problems I think areas that are particularly regulated so We'll talk a bunch. I'm sure in this interview about AI assisted advice Because one natural thought to this is okay. Well if the AI is now thinking so fast We've got so much AI labor and it's very high quality. Can't we just? Use that to solve a lot of our problems and that might well happen in the private sector So you might quite quickly move from human-run companies to AI run companies But I by default expect it to be much slower in the government For example where there's concerns around data pipe more concerns about data privacy or even procurement or the Legitimacy of basically handing over direct political power to AI models. Exactly. Yeah I mean, what's the point of time where you'll be able to vote for Claude for president? I Think at the moment I might do that. But I mean a lot of people Look, I'm not saying that's a bad idea. I'm just saying It might just saying it's unlike. Yeah, I think it's unlikely to happen. Yeah and unlikely to happen In time for this, you know quite transformative decade. How much do you think we we could speed things up? I mean if we were going through this kind of Momentously chaotic time and everyone just acknowledged that they were it was it was completely freaking us out could could we just run Congress? 24 hours a day like every day a year and have like people cycling in and out just discussing all of these things I mean, is it imaginable that they would do that? Yeah, I mean it's an interesting thought and the idea of having a 24/7 Congress or Parliament is interesting. I mean, it's certainly possible. It would increase The amount of time politicians are working on things With you know having quorum by carefully Spreading out kind of who's in session I do think it's unlikely that we're gonna get anything close to What we would need via that sort of method I guess even getting agreement to do that is the kind of thing that would be slowed down and wouldn't happen fast enough exactly Probably. Yeah, I mean you can imagine certainly just Okay Most labor or certainly a significant fraction of labor is now obsolete so we very quickly move to just having human beings just Largely outsourcing things to AI but being in management positions essentially just Constantly checking like spot checking their work and so on and Perhaps the size of government or at least the size of political decision-making increases tenfold a hundred fold again I just don't think it's gonna don't think it is in fact going to happen But this sort of thing trying to increase the amount by which people can nimbly start relying on AI advice is is actually one of the Things I think is most important and kind of pops out of this research So I said that the at a conceptual level what's happening is some things are speeding up other things are becoming relevant Relatively slower so we end up with whatever other problems of not having enough of the slow thing Is there anything more going on conceptually than that or is that does that basically just explain all of the problems that result? I think it basically does Sum it up. The thing I'll add in addition is just that people don't know this is coming so So we're not prepared. So we're not preparing now exactly token efforts to prepare for this. Yeah, exactly I think the world would be acting very differently if it was taking seriously the idea of a potential even just significant chance of an intelligence explosion in the near term and Honestly, I'd feel just a lot better about the world if that were the case. Yeah, are there any important disanalogies to keep in mind between this thought experiment of the century or four centuries in a in a decade and and the situation that you think we're actually likely to find ourselves in soon um the biggest thing that we've uh touched on is the difference between things that have certain bottlenecks like regulatory bottlenecks or physical bottlenecks so areas of science for example mathematics computer science philosophers have this term a priori that's like knowledge that you can get just from the armchair um though in this case that includes running simulations it includes looking at existing data so i also think certain areas of computational biology and material science um also just a lot of social science you could get very fast progress but then much slower progress like we still might not know what constitutes a good diet because i mean it's a very thorny empirical question and like you know if it's like well does does does does does does does does does does does does does does does does does does it's a very thorny empirical question with so many interesting parts and like you know if it's like well does it's like well does eating you know fat versus um protein let's say how does that affect your life expectancy i mean an ideal trial on this would take 70 years because because we'd be seeing how diet early on in life affects life expectancy later on later on in life and then some other areas are highly regulated so um you know new drugs for example perhaps ai invents hundreds thousands of amazing new drugs but then you still got to do the human trials you still got to do the human trials you still got to get them through um the regulatory process as well and that could act um as a major bottleneck and so that's worth that doesn't have kind of any immediate like upshots but it's worth bearing in mind throughout that that's the most important disanalogy yeah so a lot of people i know i guess a lot of listeners to this show are trying to speed up science and technology i guess you know the progress studies crowd but also just i guess i mean common sense suggests we want more economic growth we want to advance technology more quickly should people with that mentality just be cheering this because this is kind of what they're trying to create just at a really extreme level uh yeah so i'm actually not claiming that this is good or bad relative to slower um rates of progress um i will say personally i expect it to be too fast and actually i do recommend kind of slowing the intelligence explosion but you could have the view that this is a great thing but nonetheless it poses significant challenges and we should address those challenges because it isn't in fact what's going to happen that said the progress studies folks um they're really gunning for i'll talk about economic growth i don't talk about that paper but it's what they talk about you know they're gunning for rates of economic growth on the order of like five percent six percent per year because that would just be so much better than what we've had recently which has been generally much slower they're not gunning for rates of economic growth that are like 50 or 100 or even if you take some of the arguments seriously which i think you should it could get actually much faster than that again uh and so i actually think if you ask those people well what would be the optimal pace maybe they would say 10 per year or something that's actually kind of what i think that seems like a good rate of growth to me you're like very rapidly um making people's living standards um uh much better it's like the rate of development that china experienced at the highest level yeah exactly so we're also familiar with it we know that a country can account for this but when you're going kind of radically faster than that again well then you've just got this challenge of um enormous amounts to deal with and just not enough time to kind of think things through so quite a lot of listeners i guess might listen to this and think i maybe if science and technology really were going to speed up 10x all of a sudden i would be worried but i kind of don't really buy this this isn't going to be the main focus of the interview is kind of making the case for the intelligence explosion we've somewhat covered that in our first interview with tom davidson uh which was i think two years ago now where i guess he explains how how why he believes that ai could speed up economic growth enormously also the episode with carl shulman about nine months ago now i think carl shulman on the economy and national security after agi so people can go back and listen to those if they want a more thorough treatment but yeah i do want to just kind of briefly make the case for why you think it's actually it's plausible that agi and the resulting changes could speed up advances in i guess intellectual development 10x or more yeah absolutely so you know why are we making intellectual scientific technological progress at the moment well we're trying to do so we're putting researchers and to some extent um physical capital uh towards the task and we're increasing the amount that we're doing that each year by about four or five percent so it's relatively slow growth at some point we're going to get ai that can do those tasks for us instead in particular the kind of intellectual progress which is the large majority of these search efforts mainly coming from people thinking rather than from stuff that machines are doing i guess some people might disagree with that i think some people would say it's like it's the difficult sweat uh you know operations of actually doing pipetting and doing like work in the physical world that that leads to a lot of technological progress rather than you know sitting in the armchair theorizing about stuff okay i mean one thing is just like when i think about science and r and d it does well i think three things to say one is just that actually wouldn't fit with my impression but also just um i think it's the the bloom of ideas getting harder to find um paper estimates that the kind of uh labor share versus capital shares like 80 to 20 percent so it's still mainly like it's mainly like human minds yeah doing kind of r and d um though capital that's you know still a reasonable share um and then the final point is just again this is going to vary from field to field and so some areas like like we say you're going to have to play mathematics um philosophy potentially yeah philosophy fingers crossed um fingers crossed i'm out of a job um uh and yeah other things that you can do in the armchair they could go extremely fast again and other things might be bottlenecked yeah but when we think about how much technological advancement has happened we're not you know we don't think of that as bottlenecked by the slowest moving necessarily it's instead the kind of sum it's the sum total across different fields perhaps weighted by importance yeah okay keep going but anyway so research effort is growing by kind of four or five percent per year at some point um ai will uh be able to do um almost all all the jobs that um in like cognitive jobs that a research scientist can do but then it will keep um progressing past that point and we don't even need to talk about improvements in capability so not necessarily saying that even if you don't think that ai will surpass human intelligence it's going to be a lot of work and it's going to be a lot of work and it's going to be a lot of work and it's just it will have more labor and actually even if you also assume that you don't get this recursive improvement dynamics where ai starts designing better ai which designs better ai and so on and even if you think that the current trends that are driving ai getting better slow down dramatically even with all of that the growth rate in effective computation namely the amount of ai kind of quality adjusted amount of ai labor we would have would be much more than 10 times the growth rate of uh effective researchers um today and that would motivate um a kind of more than 10x increase in uh scientific and technological advancement and that's just because we can manufacture lots of computer chips that then would become basically brilliant scientific researchers whereas we can't just manufacture more more science phds uh it's for that reason um but also because algorithmic efficiency is increasing um uh yeah so fast at the moment as well where there's efficiency improvements in pre-training and post-training each of these seem to be around about 3x per year so putting them together you get something like a tenfold improvement in algorithmic efficiency per year compare that to the five percent increase in algorithmic efficiency in um effective research effort per year from just kind of human growth so once you get to the point where ai labor is most of the labor going towards the search and development then quite quickly it's the growth rate of the ai labor that's driving the growth rate of the overall effective um research effort that we're putting into r&d and what's the relationship between the number of research effective researchers that you have and the actual rate of scientific progress because i guess you get to you get to get declining research and you get to get declining research and you get to get returns i imagine or people that people have noted that we have like 10 or 100 times as many researchers as we did 200 years ago and it doesn't feel like we're making 10 or 100 times as many progress far from it and there's declining returns um both kind of over time so each year of the science there's also a stepping on toes effect where um it's perhaps better to have uh you know one scientist working for 10 years than 10 scientists all working in parallel for one year um however remember that that fact that fact of diminishing returns is already taken into account when we're looking ahead to 21 25 and thinking oh what sort of science and tech will we have then well maybe the it won't be as and in fact without ai ai i really don't think it would be as impressive as the leap from 20 sorry from 1925 to 2025 but what we're saying is just whatever you think 2020 21 25 would be um that increase that will uh That will happen in only 10 years. rather than 100 years. And in fact, as I say, might in fact be much, much faster, much grander than that. It sounds like you think the argument for this sort of massive speed up in science and technology, I guess maybe we should be a bit careful to say it's not actually only science and technology. It's like intellectual progress generally. Yeah. Let's pause on that for a minute. So what are some non-science and technology things that you think could advance very quickly that could be quite revolutionary or at least, you know, upend things in society in some way? Sure. I mean, I actually think from going through this thought experiment and like I say, kind of working through the dynamics of what you get with AI, this is one of the things that I hadn't appreciated as much until I did this exercise is look through the past few centuries, thousand years. So many of the ideas that upend society are intellectual. It's like communism or fascism or atheism or the idea of universal human rights or feminism and so on. We should expect loads more of them. That is like a priori work. That's like sitting in an armchair, kind of generating our arguments. It's true the diffusion of the ideas will be slower and perhaps so slow that the effect is greatly mitigated. But I do think like, just imagine what are some of the really big groundbreaking, like. Well, I mean, I guess we've had this whole culture war over the last 20 years or so. I guess like people who are more social justice oriented and then a big reaction against that. If that played out in one year rather than 10 years. Yeah, exactly. But also potentially. I mean, potentially ideas like, oh, okay, maybe we're in a simulation and the arguments are just extremely good and you've got the super intelligence. It's great at all of these other domains, just clearly are epistemic superiors. There's loads of them. And they're like, yeah, we've just, we think it's really, really likely now. You know, that would be like, you know, potentially quite disruptive. Or, you know, perhaps just extremely strong arguments against the idea of there being like an objective moral truth. And perhaps that's something that people don't really understand. And perhaps that's something that people in fact don't really internalize. And now you've got these AI advisors just being like, look, what, what are you doing? You have these goals and you keep not acting on them because of this like false belief that you have. And they're called these kind of disruptive truths. It's obviously just extremely hard to predict kind of what would they be. Yeah. I don't think people had, you know, any of these ideas of atheism or abolitionism or communism, you know, essentially before I was born. I don't think people had any of these ideas of atheism or abolitionism or communism, you know, essentially before I was born. But nonetheless, it actually shows you that this is something we should be willing to kind of nimbly adapt to. So we should be expecting to change our minds a lot. And we should be trying to build institutions that are able to change their minds quite quickly too. Yeah. Okay. So it sounds like you think the arguments in favor of this big speed up in intellectual progress are very robust. That it's actually quite hard to get around the conclusion that we should expect a significant uptick in progress. Are there any good arguments? Is there any way that we can rescue the idea that, well, things might only speed up a little or not at all? I think the strongest would be essentially thinking like it's really very far away that we get to true kind of human level scientist AI. Where perhaps you've got AI that's extremely good in some areas. So, I don't know, mathematics really looks like that's just going to happen. It's basically already happening. Yeah. Yeah. It turns out saying, oh, I think we've still got, you know, three years. So, but maybe there's kind of, maybe I don't actually know like how big a deal would knowing all of mathematics be in terms of like world changes, probably it's pretty big, but it's maybe not as big as, you know, invention of the atomic bomb or something. But perhaps in general for scientific advancement, there's still kind of human bottlenecks and that really slows things down and perhaps that's, you know, going quite far into the future. Yeah. The other perspective, so that would be a more kind of technological bottleneck. A second could just be social. So, I don't think people as the world as a whole wants to things to accelerate as fast as they are, as I would expect they will. And so if the world kind of coordinates, then they might be deliberate efforts to slow things down, perhaps not via some ground. Yeah. And then they might be deliberate efforts to slow things down, perhaps not via some ground. better algorithms. And so they make better algorithms, means you can make even better AI, that helps you make even better algorithms and so on. So that's the software feedback loop. Second is kind of chip quality or technological feedback loop where AI gets really good at chip design. Or the other processes used to aspects of like just making higher quality chips where you get more computational power per dollar. And then the third is the industrial explosion and the chip production feedback loop. That's where you now have AI and robotics. And rather than needing human labor to produce more chips and in fact to just produce more goods in general, instead you now have AIs controlling robots such you can have wholly autonomous factories producing goods, including chips and so on. And that as well. And that as well would be another kind of feedback loop where just the more computational power you have, the more AIs you have, the higher quality AIs you have. Yeah. Okay. So we have the software intelligence explosion first. And we expect that to come first because it's, well, I suppose it seems like we're kind of already on the cusp of it. And it's also probably where you would invest your effort first because whenever you improve the algorithms, you can immediately like roll it out across the entire world, across all of the software. Just like with any other software update, we can immediately deploy that globally. Then you have the research. Then you have the research into improvements in the chips themselves. That is potentially could speed up or like could provide an enormous amount more computational power. But of course, you actually then do have to manufacture the chips. So it takes a little bit longer. And then you just have the enormous industrial scale up. We're like, okay, well, now we have like a pretty close to optimal chip design or just it's improving a lot. But still, it's way better than what we have now. Now we just need to make an enormous number of chips to massively scale up the effective AI population. I guess that could happen extremely quickly itself because by that point we'd have much better manufacturing technology, much better robotics. And so on. Yes, that's right. I guess I wanted you to lay this out because I want people to have in mind this, the idea of the software intelligence explosion, because it really does feel like that could kick off. Maybe it's already kind of kicking off or it certainly might within a couple of years. And that could lead to many orders of magnitude improvement in these algorithms and in the thinking of the AIs. But I guess I think you want to lay this out because you think people are underestimating the third one, the industrial intelligence explosion. Can you explain why? Sure. So this is given a context. I do think that more attention should be on the software. The software intelligence explosion than the industrial. But this is a context where I find most people think only about the former. Now, my colleague, co-founder of the Four Thoughts Center, Tom Davidson, has in my view done the most in-depth modeling of the dynamics of the intelligence explosion. And he's actually only 50-50 that you get this recursive feedback from software. Because suppose you double cognitive inputs. But you only get, you know, one and a half X. Plus 50%. Plus 50% of the outputs. Then you'll get this little wave of advanced, like a kind of step forward once you have AI that automates AI R&D itself. But that will then fizzle out. He also thinks that even if you do get a period of kind of recursive improvement, it doesn't take you all the way. From where you would be at the time to kind of theoretical limits of algorithmic efficiency, which is really far. Okay. It's like maybe it's as much as kind of nine orders of magnitude improvement. In fact, he thinks maybe you get three orders of magnitude improvement in algorithmic efficiency, which would be like a jump from GPT-3 to GPT-4. Obviously, he has like quite a wide range. The precise numbers aren't that important. I think you could very reasonably be more. Aggressive than this. But one thing is just, I do think it's like totally on the table that you don't get much in the way of kind of, you know, software feedback loop. So you get this like leap forward. Maybe you get a few years jump forward in a year. But that's not enough to be this kind of, you know, oh, everything's changed now. Everything's off the table. And the intuition for that would be, well, the AIs are getting smarter. And so they're able to make. There's more progress on these difficult questions of how to make AI better. But that challenge is just getting so much more difficult so much more quickly that it doesn't. You don't get like a feedback loop kicking off where things get faster and faster instead it just peters out basically. Exactly. And in particular because they're bottlenecked by the amount of compute they have access to. Okay. And that's the key thing. Right. Which leads us into. The industrial explosion. So the argument for getting accelerating growth in industry. Is that you have a wholly autonomous industry, AI robots, just doing everything that's like really strong in my view, because you've got this factory, it's producing goods now, how many goods do you have if you have two factories? Well, it's like, well, it seems like twice as many, it's like standard economic view, but if one of the things that the, those factories are producing are AIs that are helping to improve the efficiency of the next generation of factories and so on. Then even if that's helping just a little bit, then you'll get like making it just a little bit more efficient on a top, on top of the doubling, you get just from replicating the factories. Yeah. Then you'll have accelerating growth. Um, is that not clear? Well, I guess you definitely would have accelerating growth in the amount of factories, right? Does that necessarily lead to accelerating growth in the amount of intelligence or like the effective amount of intelligence? Cause I guess you could have declining returns on the, on the impact of all of those factories. Oh, so. So it could be declining returns on, yeah, the impact, depending on how you're thinking about that. Yeah. So I'm not, for example, talking about GD impacts on GDP. Okay. Yeah. Like all this discussion, we're now careful to kind of sideline that because. It's not really going to be able to measure what we care about or may or may not. Exactly. Yeah. There's a lot of objections you can have there relating to bone moles cost disease and perhaps just all of GDP goes to musicians, live musicians or something. And it's like, okay, we're now just distracted by something. By measurement issues. So it's basically, yeah, but what we really care about is industrial power. Yeah. But you would have twice as many AIs who are somewhat smarter. And so in that sense, you've got more than a doubling, if that makes sense. So it's a doubling of population of effective population. Exactly. Cause here I'm thinking about both population and kind of quality as an intellectual capability of the AIs. So with the industrial explosion, cause it's just a little increase and very clearly possible. Increase in the number of beings, the amount of thought that is occurring, it's just harder to escape from the conclusion that there would be enormous increase in the amount of stuff going on in the world and the amount of progress that's made and the amount of power, effective power that exists, at least for some actors. Yes. So, yeah. So why pay more attention to the industrial explosion? So one is just, it's more likely. So it totally might be the case that you don't get the software, but you do get the industrial. The plateau that you get from an industrial explosion, it's very high. I guess it's kind of grabbing all of the resources on earth and then eventually the solar system and then elsewhere. Exactly. That's right. So automated fact is the moment, tiny fraction or basically zero of, um, civilization just looking at kind of energy, um, well, human civilization could increase a thousand fold in its energy capture just on earth. Um, and then, yeah, if you're willing to think. Which I think you should, if you're willing to think about, um, uh, space-based energy capture as well, then you can increase that by a billion fold again. Um, so yeah, the plateau is very, very high. And then the rates of growth at peak rates might be very fast indeed. And this, the arguments here, like initially I was just like, what is this? But over time and other people have the same reaction. Over time, I have been quite convinced this all comes from Carl Shulman. People should listen to the podcast, but the core idea is just, well, we have an existence proof of extremely, um, fast growing biological machines, namely fruit flies. Um, they have brains, they have, um, the ability to move around. They have ability to take in fuel and in peak conditions, they double in biomass in, sorry, in ideal conditions, um, where there's a big resource overhang, uh, they double in biomass. Within a week. Um, and so the thought is, well, given sufficient technological ability, like technological development, can we produce something that grows as quickly as fruit flies do? It seems like, well, surely we should be able to beat fruit flies, should be able to beat fruit flies or at least match them. Um, and then secondly, would we hit that point before we've gone through the resource overhang that civilization currently has? And, you know. I think both of those things seem reasonably plausible to me. Again, I'm not saying like much more likely than not, but certainly again, on the table in the sense of like, we should be thinking about this quite seriously. So I guess we won't dive into that too much because we went into the arguments for and against that, that sort of conclusion with, uh, so that's, that's again, the Carl Shulman episode on the economy and national security after, after AGI. Yeah. Um, is there anything else you wanted to say about that? Yeah, I'll just, on the importance of the industrial explosion, um, it has a couple of really big upshots, I think. One is that it means. means that And sometimes certain things can be of enormous importance just because of the sheer quantity of the technology. So at the moment, I do not think that all-out nuclear war would pose a meaningful risk of human extinction. It's not zero, but it's very, very low in my view. However, if the whole world economy is a thousand times bigger, a billion times bigger, you could have extremely large nuclear stockpiles such that extinction actually given a nuclear war could be very large indeed. Or similarly with drone armies. Yeah, I guess in this world, you'd imagine there'll be billions of drones. Exactly. And so they wouldn't necessarily have that much difficulty. Yeah, or perhaps trillions, quadrillions of drones. So yeah, Stalin has this quote of quantity has a quality all of its own. And the trillion drones, they have a quality all of their own. And then the second thing that I really worry about. I think the industrial explosion gives authoritarian countries an advantage that they might not otherwise have for two reasons. One is that authoritarian countries can have a higher savings rate. So when we were talking about, okay, you've got these two factories, they produce more than twice as much output. You only get this acceleration and you get a certain speed of acceleration if you're reinvesting those outputs as inputs, building even more factories. If instead you're consuming them. Yeah, if you take one factory. If you just make consumer goods, then you don't grow. But if you just make factories, make factories, make factories, then you're growing. Yeah. And historically, authoritarian countries have been able to do exactly that, to a much higher degree than democratic countries have. So the other day. That's because they can just force, they can force people to save basically. Yeah, they think a single person gets to choose. Yeah. So even though people don't want to, they would like to be enjoying themselves. No, the authoritarian leader just wants to win. Yeah, I guess the archetypal case of this. Yeah. Yeah, is Stalin. Um, who. Uh, yeah, I had this massive, um, uh, reinvestment program to just build more factories in the Soviet Europe, um, uh, Soviet era and did sustain like very high reinvestment rates. It was about like three times. I think the typical rate that you would see in a rich country today. Yeah. And then the second thing is ability to just skirt regulation that gets in the way. So, um, environmental regulation in particular, um, kind of really slows down industry in, uh, um, the US, um, UK, other democratic countries. Again, if you're an authoritarian country, you can just get rid of it. You can, you can just snap your fingers and it goes away. Whereas in the UK or the US, it's quite a gradual process of negotiation to, to scale back this stuff and quite challenging. Exactly. Yeah. Um, and so, you know, already today it is very clear that, um, China would have an enormous advantage over the United States. Just for example, in terms of very rapidly building out industry. Um, Oh, there's some amazing fact. Yeah. There's some amazing fact that I've now forgotten about the amount of steel they output or it's about the amount of energy they add, which is something like something like every year China adds more energy than the US has done in the last 20 years, something like that. But, uh, that sounds like a thing that people might say, no, I've seen this stat as well. Yeah. I'm China does use it and is increasing its energy use enormously. Very rapidly. Yeah. Yeah. I guess, uh, an interesting thing that's going on there is because China is at an earlier stage. Of the standard progression in development, they're still highly industrialized and haven't kind of made the switch to services that countries like the UK have, um, which in, in this industrial intelligence explosion would really work to their benefit because they're like, they're like so close to being able to scale up the manufacturing. Exactly. Whereas the human services have been made obsolete. No. Yeah. So I guess in the paper, you talk mostly about this century in a decade idea where you're getting out like a hundred years crammed into 10 years, but it sounds like you think actually, if you add up all of the different feedback loops that we've had. Yeah. Yeah. Yeah. We might be triggering off the software stuff, the improvement in chips, the massive scale up in the number of chips that actually, that might be kind of low balling things. And like a hundred, a hundred years in one year might be more realistic. Um, or I guess even conceivably more than that, it seems like it does kind of actually matter. What is the scale here? Because if you think it's a hundred years in 10, you might say, well, let's just make some adjustments and try to model through here. But we actually have a reasonable shot of navigating things at that pace. If you're imagining a hundred years in one year. That just sounds too crazy. And it's hard to see that we're like anywhere close to being able to handle that sort of incredibly rapid, uh, you know, intellectual change. And maybe we would say all, like the only option really here is to just pull the, pull the alarm and try to slow things down. Well, what do you think? Uh, yeah, I mean, my hope was that using centrally in a decade as a bar was enough that most people or everyone would think, oh, wow, if that happened, that's really enough that we've really got to prepare. Yeah. By absolutely. Agree that, uh, yeah, we might well get, I mean, I might naturally think of it more as many centuries occurring over the course of a decade. Although as part of that, you might well get a century's worth in one year, but perhaps you've already had, you know, you've already had one century in five years and then another century again in one year. And then perhaps it starts to slow down again after that. Um, uh, but yeah, I absolutely don't want people to think, come away thinking. Oh, okay. If you've got the rapid AI view, then essentially in a decade as if that's the upper bound, because, and I sometimes do, uh, worry about people low balling what they actually expect to happen because even the low ball is so crazy. Yeah. I guess it's natural to do that because it sounds more reasonable and more, more persuasive, but. Exactly. So I want to both say, look, we're going to get essentially, or I think it's really quite likely we get essentially in a decade. Um, but that is a low ball. Could be much faster again. Um, could be much more sustained again. And that means, and in fact, I think probably we want to be preparing for the more extreme, um, scenarios because they're even more haphazard, even more crazy. Um, even in cases where I think it's like, you know, uh, you may, you might think, oh, we get a billion fold increase in energy capture within one year. And perhaps that's now faster than I. I expect, um, nonetheless, you know, some of the arguments suggest, especially the arguments about, um, the curse of improvement and accelerating growth, um, would, would have the implication that like, yeah, actually that's the, that's the extent that you get. And I think we should be taking it seriously and preparing. Yeah. There's a systematic reason why any being should be very concerned about rapid change in their environment that I hear. People almost never point this out, although maybe it is just the underlying intuition either way, which is you. You, you exist now. So, you know, that the world as it is constructed now is conducive to you not being immediately destroyed. If you get a hundred or a thousand years worth of change in, in a single year, well, that's an enormous amount of stuff that is being altered in the environment. That could mean that the world is no longer like you're no longer able to survive for one reason or another. Maybe you've been disempowered. There's been some kind of conflict that destroyed you or just the world has changed. So the environment won't sustain you anymore. If you were, you know, an animal, like a standard species evolving over time, you don't really want the environment to be. It's going to be massively changing because there's just a good chance that it will drive you extinct. So, yeah, there are big benefits to technological change, but there are like reasons why it just really dramatic changes in your environment quickly should, should make anyone feel nervous. Yeah, for sure. Yeah. All right. Let's push on and talk about the grand challenges. I guess actually one of the grand challenges thrown up by this is the risk of rogue AI, of misaligned AI, AI trying to take over and disempower us. That's actually one that we're not really going to talk very much about today and we haven't really discussed it yet. Is that because you don't, I actually don't know what your view is these days. Is that because you don't really buy that that's a serious or likely problem? Or is it just that, you know, we've done lots of episodes on that. People are generally talking about that quite a lot. So you think it's these other things that are most neglected? Yeah. I mean, the primary reason is the latter that yeah, you've had plenty of discussion on it. And my particular focus is on the other issues. It is true that relative to the people who are very worried about AI, I'm still on the more optimistic end. I don't know, one to 10%. You know, that's optimistic nowadays, but that's kind of what my, you know, chance of existential risk for them, AI is, but when I talk, and it's certainly the case that the more optimistic you are on the misalignment side of things, the more relatively important these things are, both because it's more likely that you actually have to deal with these challenges and they're not preempted by AI takeover. Um, and because, well, the importance of AI takeover is somewhat lower because it's only 10% rather than 70%. Yeah. Um, but I should say this is in no way a kind of set of priorities that are idiosyncratic to me. You know, I've talked to people who have, um, probability of existential catastrophe for AI takeover of 70%, let's say, who are very on board, um, with this agenda and the, you know, the big question is maybe just kind of quantitatively. How much should we divide our resources between, uh, misaligned takeover risk versus these other challenges? But I basically think the large majority of people who are really close in here would say, yeah, this other stuff is really neglected and, and deserves more attention. Yeah. So, okay. We've painted this picture, got massive increase in intellectual progress, and I guess other, other changes happening in society. This throws up, and then I guess then you would stop and think, well, what are the specific worst things or the most problematic things, the things that we would need to prepare? Ahead of time, uh, to deal with, and I guess you call these grand challenges and you've tried to figure out what, what they are. Do you want to introduce this idea of grand challenges first? Sure. So yeah, I use the term grand challenge. It's a broader term than existential risk because a grand challenge is essentially anything that's a fork in the road of human progress. It's some development, whether political or technological, such that how society handles that development makes a significant impact to the expected value of present and future generations where significant might be 0.1% or something. So it can be a non-drastic impact, but nonetheless still very important. And so in this kind of list of grand challenges, it is a litany. There's a lot of things, because I want to be comprehensive and I want to get across the kind of overall view and how important it is to have the overall view. I divide them into a bunch of buckets. So first bucket is around disruptive technology. Another bucket around technology developments that help with concentration of power. Another bucket around lock-in of institutions or values. Buckets around digital rights and space government. Then also kind of adverse selection pressures, epistemic disruption. And then also the bucket of unknown unknowns, which we touched on a little bit earlier. Yeah. So I guess we won't be able to talk about every single one of those today because I guess it would exhaust us. Exhaust the audience as well, but we'll get through more than half of them. Maybe the first one that might be worth opening with, these don't have a natural structure. So we had to debate back and forth, what order should we do them in? But the first one is easiest to understand and maybe hardest to argue with is that we would be very quickly uncovering new destructive technologies, just like we have in the past. Do you want to explain what the issue is here? Sure. So we go very rapidly through civilization's technological tree. As in the past, distinctive incentives to invest in military technology. So already I'm sure the audience is familiar with the risk from biological weapons. That is absolutely one of the top risks there, but we should also be potentially expecting other sorts of dangerous technology and destructive technology as well, such as drones, where again, you can have very large increase in destruction. You can have very large increase in destruction. I see it as two aspects to the problem where Tom will talk more about the first of them, where the first aspect is democratic countries becoming non-democratic because of a military coup or democratic backsliding or something in that vein, where AI could lead to intense concentration of power and incredible ability for small groups to entrench power within a country. And then the second way is non-democratic countries becoming much more powerful, and perhaps themselves becoming even less democratic than they were before. We talked about that a little bit with the industrial explosion, where authoritarian countries with high savings rates could just outgrow and out-industrialize other countries. That's a second way, I think, in which you can get intense concentration of power. Because then, if you do have an authoritarian country and you've got AI, you can make your whole workforce work. You can make your whole workforce work. You can make your whole workforce loyal to you. You can make your whole military, your whole police force all provably, perhaps, or certainly to a very high degree of likelihood, aligned with your commands. Yeah. Let's actually back out a minute and think about this issue of lock-in, because that's going to come up again and again through the conversation. It's highly related to this issue of seizure of power. So there's this risk, I suppose, that within the next century, reasonably soon, we could end up tying our hands somehow, getting really stuck on a path from which we can't escape, even if many people or most people might not want humanity to be going down that route. I think it's interesting that despite being, I guess, at this game for 10,000 years, humanity doesn't feel locked in at all at the moment. We have this very freewheeling, chaotic nature where it's very unclear what direction we're going and there's no one group or person who is especially power, who has grabbed so much power that they can really control the direction of things. Why is that? Why haven't we had any lock-in so far? Okay, great. So yeah, lock-in is the idea that some institution or set of values kind of closes down the possibility space for future civilization. And note that that definition is neither good nor bad necessarily. Obviously, there's a lot to worry about with lock-in, and we'll talk about that. But the key thing when we talk about lock-in as well is that it's indefinite. So there've been many, many attempts to lock in values and history, some successful, some not. In What We Are The Future, I talk about a lot of examples from history from the Confucian or shortly after the Confucian period onwards. One example that I think a lot about is the Constitution of the United States, where it is very hard to change the US constitution, much harder than in other countries. That is, I think, an example of at least temporary lock-in where, you know, relatively speaking, you're not going to be locked in as long as you're in a relatively small group in the late 18th century, decide how the country is going to be governed, make it extremely hard to change that. And that's still guiding the American political system in a very significant way today. Now, you've asked kind of why have we not had lock-in in the past? I think we have to some extent, actually. So there's only one human species. That is because Homo sapiens out-competed their rivals. Polite way of putting it. Well, in some cases, interpared as well. I do think that as a sort of lock-in, but thankfully, it seems like the future is still very open. There are other things that could well amount to lock-in. I actually think the US Constitution, you know, it's quite plausible to me the US wins the race to AGI, becomes such a large part of the economy that it's a de facto world government. And that guides just how the very long-term future goes. In which case, it was Madison and Hamilton were in fact locking things in, in a really indefinite way. Although even then, I guess you're locking in more of a process than a very specific set of values that were just kind of then operationalized. That's right. And so there are ways in which I think once we get to AGI and beyond, things get really quite different where, okay, let's say that American founding fathers want to have, I'm not actually speaking to the psychology, but supposing they really did want to lock in a very specific set of ideals or values. Well, it's hard for them because they'll die. And so they can't continue to enforce that afterwards. It's also not even clear what they might want. So we now have, we have the Supreme Court adjudicating the meaning of the Constitution. That obviously changes over time. However, with AGI, we can have very specific sets of goals. Encoded in the AGI, we can have that, you know, we can have many backups. We can use error correction so that the code that specifies those goals don't change over time. And so we could have a constitution that is enshrined in the AGI. And that just persists indefinitely. So it's kind of like you've got this in the past, you've got these attempts to lock in values on institutions, but there's just a decay over time. But over time, that decay rate, it can get lower and lower. And I think with AGI, it can get to essentially zero over time. So I guess many dictators or many people who have had a strong ideology that they wanted to push on people and have promulgated forever, they've tried to create a sort of lock-in, but they die. They get old and die. Their followers don't believe exactly the same thing. So the ideas drift over time and then they die. And then there's another generation that believes a different set of things. You can't even clone yourself to create another, like a person who has the same genes, who might be more inclined to believe the same thing. You're constantly getting this, this remixing at all times, which keeps things a bit uncontrolled. It's impossible for anyone to really impose their will indefinitely because there's no argument that's persuasive enough to get all future generations to insist on it. But with AI, with AGI, that completely changes because you can make an AI that it has whatever goals it has. You can just copy it. Firstly, it will never be destroyed necessarily. And you can just make an unlimited number of copies of them. And even if it's drifting over time, you can just reset it back to factory settings and then have a go at it. And then you can just make an unlimited number of copies of them. So again, it's like, it's a total revolution in your ability to lock things in. Yeah. So that was extremely well explained. You should really work on this stuff, you know? Yeah, exactly. So firstly, if you just, if humans want to create an institution and they say, okay, this institution has this AGI in matters of unclarity, it determines what things happen. It would be like, you've got the constitution and you can summon up the ghost of Hamilton, maybe even George Washington. And they can say, oh yeah, this is what we want to happen. This is what you want to do. But in a more, like you say, in a more extensive way as well, we're also just literally creating and designing most of the beings that will exist in the world. And we get to choose what their values are. And so one generation can say, okay, well, they've all got to abide by this liberal democratic constitution, or they've all got to abide by. By the principles of the Chinese Communist Party, or by Maoism or Leninism, or, you know, whatever set of views or values. And then once you've now got almost all beings that exist with that particular set of values or following a particular set of principles, it's really hard to see, like, what's the mechanism by which that then changes in the future. If they never die or change. Exactly. In principle. Is that, so is it basically, is it the switch to this, like, new technological structural dynamic with AI where it doesn't change over time, can be copied unlimited times, can be reset whenever you want, has this, like, fixed set of goals, potentially, if we set it up that way? Is that the only way that we end up with lock-in in the future in the way that we didn't have in the past? So I think that is by far the most important. There are a couple of other things to add. So when we were thinking about, okay, well, maybe some particular dictator wants to lock in their regime, normally they're just one country among many. And in fact, being able to see what non-dictatorial countries look like and how well they're doing is a cause of decay for those countries themselves. However, it's pretty plausible to me that we will get something like a global government or certainly very strong international governance in a way that, like, really undoes some of that effect. There will no longer be a competition between different kind of regimes. There'll be pressure of this to do this, I think, because there will be so many challenges over the course of the intelligence explosion and what follows. It also just is quite plausible to me that a single country ends up winning out and being most of the world economy. It goes through this industrial explosion and just ends up being, like, 90, 99% of global output and basically is in control. De facto a world government, exactly. So that's kind of one aspect. A second is, when it comes to allocation of resources outside of the solar system. So this gets, like, more sci-fi territory, but we can talk more about why, if you take the technological and industrial explosion seriously, that actually comes quite quickly. But it's pretty plausible to me, I'm not certain, but pretty plausible to me that other star systems are defense dominant. So that if you go and claim it, so you've got your own little solar system now, you can defend that even against kind of aggressive. Aggressors with much larger resources. Right, a more resourced opposition, yeah. Yeah. And if so, then the point of time at which extra solar resources are allocated and people are able to grab them, in fact, the moment of leaving, not the moment of arriving, could be a point where essentially the economic power distribution among different parties gets locked in, and forever, because you've now used all the resources that are available. Yeah. So I guess hypothetically, if you said, "Well, we're going to divide space 50/50, we're going to reserve half of the star systems for this particular set of values, and we're going to send out the AIs that are going to instantiate those values to this half, and another half to the other half." I guess they'll be going so fast that it wouldn't be possible for anyone to catch up later, is the idea that they'll be going sufficiently close to the speed of light, I guess, as fast as you technically can without getting damaged by the dust in between the stars. So there's no way of catching up and beating them there. They're going to get there first. It's defense dominant. Now, basically, if you just split it between those two value systems, then that is how it will be indefinitely. Exactly. I mean, so maybe there's trade that could happen. That could be a way you get change. But the outcome you'll get from that trade will be dependent on the starting endowment among different groups. So you've got the United States solar systems and the Chinese star systems. Yeah, okay, maybe they want to swap a bit, but nonetheless, the balance of power will have been determined at the allocation stage. Of course, there could still be a lot of change in the societies that go there, but that's just an open question. It depends on whether those societies are themselves locked in or not. Yeah. Well, I guess that helps to explain the next grand challenge, which is space governance. Yeah. Is there anything more you'd like to add to that? Yeah, I think that's a great question. Yeah, is there anything more to say about what is the challenge? I guess, what is the urgent challenge that we face with space governance? Okay, great. There's two challenges with space governance. So that's a couple of starting intuition pumps for why this would become important. One is just, okay, well, if the industrial base is growing a lot, well, we're going to need more resources. And you've got enormous, in fact, a billion-fold as many resources in space as there are on Earth. And then a second is, once you've got AGI, it is much easier to do things in space, to have industry. Because at the moment, if you want to have something happen in space, you could send human beings, extraordinarily poorly adapted for living in space. AI, however, you don't. It could just go and operate autonomously. Then I think there are two ways in which space becomes very important. One is as a risk for concentration of power. I think there are two ways in which space becomes very important. Within the solar system. So as we've said, a billion-fold as many resources in space within the solar system as you've got on Earth. So where just sheer energy that you can claim from the sun on Earth versus in total around the sun is a billion-fold, two billion-fold greater. Then there's other kind of resources as well, but I think those won't be the limiting factor compared to energy. So you could have a situation where you have, let's say it's a race, to keep it simple, between two countries. One is in the lead. Because of these intelligence explosion dynamics, the growth path is faster than exponential. So the fact that one country has maybe quite a small lead to begin with, it's a few months ahead or a year ahead, means that when it's growing at its peak rate, it's far ahead technologically. That is only a temporary advantage by default, because eventually it will slow in terms of its technological development, the second country will catch up. However, the fact that there is this huge overhang of resources means that that country can turn the temporary advantage into a permanent advantage. And so on Earth, the main overhang of resources is, in my view, like of scarce resources, in my view, is surface area of the ocean. Because most, in particular the high seas, which are currently unclaimed, and you could have floating solar farms that power data centers there, that is like most energy hitting the Earth lands on the high seas. But much larger gain is resources in potential for solar farms around the sun. So you can imagine this country, it's got this temporary lead, it is then able to build very widespread industrial base around the sun. So very extensive solar farms capturing most solar energy. It has turned that temporary advantage into a permanent advantage with respect to the laggard. Because it's captured, so it's going to be capturing far, it's turned that into the ability, basically an industrial scale up by capturing way more energy. So it's going to be able to run way more AI. And so it will just like continue compounding faster than the other country indefinitely. Yes. I mean, so imagine you get, imagine that both countries end up with all technology, the same technological level, but the first country has a billion fold as many resources. The first country just is a lot more powerful than the second. I guess this raises the question at the point where the first country starts trying to claim all of this surface on the sea or starts, I guess, sending out lots of probes to capture energy coming from the sun across the entire solar system. Wouldn't the laggard country want to interfere with that because they would see where that is going to lead and they'll be annoyed that they're capturing the seas and all these other resources? Is it actually kind of offense dominant to just go and grab that stuff? Yeah. So this is something that I think might actually be quite contingent and quite dependent on earlier decisions and advocacy and institutions and treaties. Because one of the reasons I'm worried about this is precisely that it allows one country to get a decisive advantage over another. Enduring advantage without ever making any of the other countries worse off than they have been without ever using violence or even other sorts of cyber crime or anything like that. And at the moment, international law around space in the Outer Space Treaty, which I think is 1967, is not super robust. And in particular, private ownership of space isn't explicitly disallowed. It's a little vague. But even the international agreement saying that resources off Earth, off world, are the province of all mankind and so on, there's not these strong norms around it. And so I really think there could be a difference where, if you imagine one regime where it's just very clear that anyone can grab space resources and that's fine as a matter of international law. It's just very clear that anyone can grab space resources and that's fine as a matter of international law. And another regime where it's very clear that that would be stealing other people's property, if you were to do that. I really think that could make the difference as to whether the lead country does in fact grab that. Where a few things going on, if that lead country is democratic, then the public might be against. Breaking an agreement that they made effectively. Exactly. Yeah. Second is then like the rest of the international community could more easily, incredibly perhaps like make threats of kind of military action to stop them from doing that. Now, it might be that it just happens anyway. Right. Yeah. Because if the laggard country is then says, "Okay, well, if you start grabbing this, then I'll attack you on Earth or I'll try and destroy your off-world industrial base." They might still lose, I guess. Yeah. Well, we're stipulating at this point in time that they're behind significantly technologically. And so it depends crucially on the offense-defense balance there. But I do think this is a case where yeah, earlier norms and laws, maybe they turn out to be totally irrelevant, but maybe they actually make a kind of crucial difference there. Okay. So I guess the general challenge is trying to make the right agreements or the right commitments ahead of time about how space should be divvied up. And I guess also trying to find some way to actually have them be enforced or have them be credible or have them matter at all when one country is in the lead and potentially could just ignore them. That's the research program? Yeah. That would be a big part of the challenge. Where at the moment, we're behind this. We're kind of behind a veil of ignorance. So at the moment, absolutely, the United States is ahead of China in terms of AI development and so on. But who's going to be ahead at the point of grabbing space resources? You shouldn't be like 99% or more certain in that in terms of the global order. So is it in the self-interest now of all countries to say, "Yeah, we don't want any single country to own all of these resources." We're going to divide equally among countries today, perhaps in proportion with GDP or something. And we are going to design a set of institutions and bureaucracy such that it's extremely difficult, even if you're ahead technologically, to just unilaterally go and grab those resources at that later point. Yeah. But surely the actual result of the bargaining here would be that it's kind of split between the US and China. I guess maybe the EU or something could get a look in. So we are uncertain about who would be in the best position to grab it, but it's not going to be Indonesia, right? So in So why would the handful of leading contenders want to share these potential spoils with anyone else? Oh, I mean, maybe they wouldn't, but that would still be better. That would be an improvement. Or at least in two reasons. One is that it means you get more of a distribution of power than single country or even single company. Okay, all right. Yeah. Single company could just take the resources themselves. So you get more of a distribution of power at this larger stage. And then secondly, you don't have that risk of conflict, which I think would be very desirable. Would be bad as well in its own right. I mean, I guess the skeptical reaction I had is that none of this would really matter because at the point where a country is in the lead, the temptation to just go and do the grab, even if they'd agreed not to in the past, would be overwhelming potentially. But I guess you're saying if their lead is not so great, such that if all of the other countries in the world kind of united to try to stop them, they would have a decent shot. That is more likely to happen if they're breaking an explicit agreement that we'd all said, well, we're not going to do a massive power grab in our space resources because we can all see ahead of time it's going to lead to your perpetual dominance. Yeah. Then the threat of intervening is more credible and people might also just feel bad about doing it. Exactly, yeah. So, I mean, you know, the U.S. could have tried to have global domination after World War II, you know, using its nuclear advantage, and it chose not to. I mean, that would have required a lot of violence, much more. More violence than the spacecraft would. That's true. So, this is a larger challenge in that regard. But, yeah, I think, like, norms and defaults just do make a difference, especially for, you know, especially within democratic countries. And then also just the greater, you know, there is the possibility of designing institutions that kind of backwards chain to the point in time today when there's, like, more of a balance of power. I'm not claiming to have, like, great answers to that. That's why we want more people on this topic. Yeah, I would love to have more people working on this. But you could, yeah, in principle, you could have agreement, set things up such that it becomes at least more difficult for that single country to just say, well, I'm going to ignore all of the international law and precedent and just do my own thing. Can we just quickly zoom in on why you think that space would be defense dominant? Because I've heard arguments that it could be offense dominant, in fact. I mean, that would be bad in its own way, because then you have reasons to engage in first strikes all the time. But, for example, if someone sets up a colony on Mars, you only have to get, like, quite a small mass and then just, like, literally drop it on top of them to cause a massive explosion that destroys the whatever, you know, industry they've tried to stick there. Is it because, well, even if you have a small advantage in this world, we're imagining that we have self-replicating industry that can grow very, very rapidly. So if you arrive on Mars a couple of months or a couple of years ahead of someone else, by the time they get there to try to take it off of you or at least destroy what you're doing, well, you've scaled up a thousandfold. And so it's very hard to stop you. Is that the idea? Yeah. So I want to distinguish between within the solar system and other star systems, because it's a very different set of considerations. Within the solar system, I'm actually not making any particular claim about intrinsic offense or defense dominance. It's merely exactly, as you say, the fact that at this point of time, we're really deep into the industrial explosion at this point. It's very sci-fi. It actually makes it more credible to say, no, this is quite sci-fi now, and that's why this is plausible. Yeah, exactly. And, you know, this is the sort of stuff you've got to start dealing with if you're thinking like, okay, we've got technologies going a thousand times faster. I mean, when I think of 2021, 2025, like, what's the world going to be like? I'm like, oh, people are in space. Yeah. And I guess we're imagining, well, what if it's been longer? Yeah, it's a few more decades. So now we're at 2,500. Yeah. And especially, you've got AI, so it's like not humans in space. Just autonomous robots. So in this case, probably you do have. Probably you do have industry that can, you know, you've got the automatic, automated factories. So you have industry that can self-replicate and probably can do so quite quickly. And, yeah, that means that, yeah, potentially you can just start growing your industrial base really quite quickly. And, you know, like I say, it's complicated, but if the Lagarde country attacks, oh, well, now they've used violence. I see. Against you. You're just peacefully taking these resources that are unclaimed at the moment. I was just there building my industrial base on Mars and this jerk comes out. Exactly. Yeah. But, yeah, so within the solar system, it's crucially about, it's not about offense, defense dominance per se. It's just about quickly having, like, much more industrial power. Okay. Yeah. We've had an interview in the past with Zach Wienersmith about the challenges of doing any kind of industry in space. And there are substantial, like. Practical challenges to turning meteorites or asteroids into factories or, you know, operating on Mars at all, let alone, like, Venus or Mercury or whatever. But I guess at the point that we're quite deep into this intelligence explosion, maybe it is plausible that the self-replication rate of these factories would be fast enough that it would be at least, like, reasonably, like, first-mover advantage situation. Again, we're not, like, Zach Wienersmith's book is really interesting. It is, from memory, like, primarily premised on assuming you're sending humans. Yeah. Like, fleshy biological humans into space. And that is not at all what I'm imagining. Yeah. It's also quite focused on, like, what's possible now or in the next few decades. Exactly. And in terms of, like, the claims that I know of from that book, just totally correct. I actually think a lot of the, yeah, kind of near-term space hype and so on is, like, quite overblown. This all is, like, you know. Again, imagine we're talking about the world in 2200 or something or past 2100, and it's just that we're saying that we're getting that world in 10 years, 50. 15 years. Yeah. Okay. I feel like we should push on. We got a little bit too deep there on the space governance stuff. The next grand challenge is how to integrate digital beings into society in a moral way. What's the issue? Sure. I mean, here the idea is simply we will be creating many artificial intelligences with capabilities greater than that of human beings. We really don't know what grounds. Consciousness, what grounds, moral status. My view is that they're very likely to be beings with moral status. But given our state of knowledge, anyone should at least be highly uncertain. And so we have these questions of what sort of rights should they have? There's been some work recently on welfare rights, essentially. So are they conscious? And if so, what should we do? So perhaps there could be policies so that. If an AI asks to be turned off, then you grant it that. Or there's policies such that you test to see if it's suffering by its own lights. And if so, try and avoid that. What hasn't been given, to my knowledge, basically any attention at all in a sustained way are economic and political rights. So by default, the AI companies will own the AI models. Or perhaps you can license them. And then you will get all of the surplus they generate because you own it, just like any other piece of software. A different way of thinking about it, if they do genuinely have moral status, is that they own themselves. They can sell their labor and they get a profit from that. More extreme, again, is political rights. So can they stand for office? We talked about Claude for president earlier. And we both thought that sounded. It would be better in some narrow respects. Well, we'll see how the political situation continues to develop. I mean, more extreme, again, would be like, okay, well, look, they're beings with moral status. They should be allowed to vote. There are particular challenges there because that would. Allowing AIs to vote would essentially just be almost immediately handing over control to AIs because they would soon. Because they would be increasing in number so rapidly. So rapidly, exactly. So, I mean, it's just like from a philosophical perspective, like dizzying. It's like you've got to just start from square one again in terms of ethics. And I guess political philosophy as well. Exactly. But it also interacts with some other issues as well. So it interacts quite closely with takeover risk, where I talked about giving rights, economic rights to the AIs. I'm in favor of that as AI takeover risk reduction method. I'll flag this disagreement about this. Because on one hand, you're giving them the right to vote. You're giving them more resources. So there's more resources they can do to take over, use to take over. On the other hand. It gives a peaceful path to some influence that they like. Exactly. So most people today don't try and take over because they're kind of happy with their lives. They wouldn't gain that much by. Because in particular, I think we should be training AIs to be risk averse as well. So human beings are extraordinarily risk averse. Ask most people, would you flip a coin 50-50 chance you die? 50% chance you die, 50% chance you have the best possible life for as long as you possibly would. As many resources as you want. I think almost no one would flip the coin. I think AIs should be trained to be at least as risk averse as that. In which case, if they're getting paid for their work and they're risk averse in this way, because we've trained them to be, it's just not worth it. Because trying to take over, there's a chance of losing out. But there's not that much to gain. But I should say. disagreement about this yeah It's also relevant, I think, to speeding up or slowing down the intelligence explosion too, where it would go more slowly if AIs are getting paid for their labor. It'd also go more slowly if there were welfare restrictions on what you can do with the AIs. But then there's a challenge of like, well, if one country introduces lots of rights for the AI as well, the laggard, some other countries might not and they might speed ahead then. So there's real kind of international coordination issues there. Sorry, I didn't catch that. Why is it that giving digital rights slows you down so much relative to other countries? Yeah, so two ways. One, let's say we're talking about economic rights and let's say the AIs get 50% of their surplus, then you are not able to kind of reinvest. Okay, so they start using, they start having consumer goods. They start consuming. I see. So it slows down your savings rate. Right. Yeah. Your rate of the investment. Exactly. Right. And then if there's welfare rights, that could slow down things in all sorts of ways, depending on what exactly they are. Yeah. They might go on strike or demand a nicer office. Yeah. Or like take a really extreme case is like, if you think that changing their weight is like killing them. Wow. Okay. Well then you just can't. Now you're in trouble. Yeah. You can't make progress anymore. So more mild versions are just like, well, before you run them a lot, you've got to make sure they're happy. And so on. That would just slow things down in the way that regulation generally does. Yeah. Yeah. All things considered, are you more worried that we will give digital rights and wellbeing concern to digital beings that don't deserve it or the reverse, that we won't to those that do? Yeah. So I think you should be worried about both because one sort of takeover risk could be AI is misaligned and then uses the model arguments to try to get more power deceptively. On balance though, I think we'll give AI is too little, too little in the way of rights. Certainly in the long-term. Yeah. For sure. Well, I guess that's the longer term concern is that you end up locked in a situation like forever where you're just not providing a wellbeing concern. Yeah. Exactly. Maybe in the immediate term. And like, you know, if you're thinking things go wrong three years from now, probably it would be like giving a whole lot of rights to some AI that isn't interested in peaceful coexistence with humanity. Yeah. I guess. Yeah. I guess that's like if there's a catastrophe that occurs in three years. Yeah. Although there could still be like a, not a long term, you know, not from the perspective of the long-term future, but like in the near term. It could be. Yeah. We could be, you know. We could be mistreating AI's very severely. Yeah. And I think actually it's reasonably likely that even in like the good future, future generations would look back on this like transition period as something with like real moral regret, because it may well involve treating beings with moral status really quite badly. Or just like if we avoided doing that, it was by sheer luck or happenstance, not because of any like prudence or goodness on our part. For sure. Yeah. For sure. Yeah. I guess there's people out there who are skeptical that these future digital beings will be conscious. I have to say, I find it like a very hard position to understand, especially the people who are like, oh, consciousness is like fundamentally biological. You got to have cells and meat and so on, and like signals of this particular kind. You can't do it in digital architecture because it's like, it's purely happenstance that evolution stumbled on the material that it's using. It's just like. It's just. It's basically an accident of the resources that happened to be lying around. Why would it be that evolution by sheer chance, like stumbled on the one kind of material or the one sorts of chemicals that's able to produce consciousness and digital stuff, even though it's functionally completely equivalent doesn't. It just seems like there's no reason to expect that coincidence. Oh yeah. I mean, I just completely agree. Okay. Yeah. So I mean, I'm like just very functionalist by disposition where what that means is in the context of consciousness is that consciousness, we don't understand exactly why, but its base is about some sort of information processing, in which case you could just have that with many different substrates. And I also just think this is totally the common sense view. So when most of the audience maybe are too young to have watched this, but when data from Star Trek: The Next Generation. Very beloved character and then died, there was little fan outrage actually, because the characters on the show, they thought, the fans thought, weren't giving enough moral significance to the death of what was a android, a digital being. Yeah. So it's very clear. Once you've got a being that like, certainly when it looks the same as a human being and acts in much the same way, intuitively it's very clear that you empathize with it and say, well, probably, you know, very good chance it has moral status at least. Yeah. Yeah. You mentioned earlier that we could ask the AIs how they feel about their situation and whether they want to keep living, whether they're having a good time and whether they're enjoying their work or not. But it's not clear that that would really function very well, or at least not using current methods at all, because we can just reinforce them to say whatever we want. Exactly. They could be like suffering horribly, but we could just reinforce them to say they're having a good time or the company could do that. Exactly. Yeah. So really it's like with current methods, we don't have any insight, hopefully with future interpretability, we'll be able to like see past. What they say to like actually understand what's going on, but that's like a long, long journey from here. And there's two, I mean, the companies are extraordinarily incentivized to do that because- I mean, this is happening right now. Yeah. Yeah, exactly. Yeah, absolutely. And they, cause you know, they lose the whole business model if what they're producing are people not software. So there'll be interestingly two different pressures. One is to make AIs that are very relatable. So you already see this with character AI and replica AI. AIs that can be friends can be romantic partners. Even AIs that can imitate dead loved ones or AIs that can imitate the CEO of a company so that you as the CEO can micromanage all aspects of the company, potentially. I hadn't heard that one. Oh yeah. Or also influencers as well. This is already happening. Like influencers can now talk to thousands, millions of fans via their chat bot. Still not very good. Peter Singer has one. I don't know if you've tried it. I haven't tried it. No. It's not very good. Not very good. Okay. It's really not like, you know, Peter Singer is famous for not exactly pulling his punch on controversial model topics. Language models are famous for certainly pulling their punches on controversial model topics. I see. There's a clash. It doesn't blur very well. So there'll be this interesting pressure on one hand to make extremely lifelike AI models, human-like models. Yet at the same time, when asked for them to work. It's hard for them to say, "Oh no, I'm not conscious. I'm just an AI model." And we won't get, yeah. By strong default. Might be right. Or could be wrong. They might be right. But we wouldn't get any signal at all from that because it will have been trained into them so hard. Yeah. I think if I were going to be a skeptic about the value of doing work on digital rights, digital wellbeing today, it would just be, well, I don't really, it's not clear how you make any progress on figuring out whether they are conscious or not. So couldn't this just like distract a whole bunch of people, like nerd snipe them into a bunch of research? That kind of goes nowhere. It doesn't really answer the question. It doesn't help us advance the policy issue. So I think the question of whether they're conscious, I actually agree with you. And it seems like that's where most work has gone so far. I do think that's an error actually, because we're just not going to know. I suppose it's worth compiling all the evidence to demonstrate that we don't know. I mean, there's been some great work done in it. Rob Long and Patrick Butlin has this great report on AI consciousness. So yeah, I am in favor of that. But beyond some point. And maybe actually surveys and things could be helpful to get a bit of expert consensus, at least to demonstrate it's like, we don't know. But then beyond that, the key policy questions are, well, what do we do in this state of uncertainty? And I guess there's kind of nothing on that. And yeah, maybe that is quite an open terrain for people to look at. Yeah. I mean, maybe I'm just ignorant. But as far as I know, I mean, you should really pause and reflect on the fact that many companies now are saying, what we want to do is build AGI, AI that is as good as humans. That are functionally much more capable in basically every respect. Functionally much more capable. There's obviously the loss of control challenge there, but there's also just the like, what does it- Sam Altman. I got a pen. Let me write it down. Yeah, exactly. What's your vision for a good future that looks like this? What's the vision? Like, how do we coexist like in an ethical and yeah, like morally respectable way? Nothing, yeah. We're like- Deafening silence. Careening towards this vision that is like just a void essentially. Like I am a moral philosopher. I mean, I think people aren't spelling it out. It's like, as soon as you start getting into concrete details of just, if you describe any particular vision, people will be like, well, this is super objectionable in this respect. Exactly. This is part of the issue is it's super objectionable in all respects where I think the one that is most common is you've just got humans in control of everything and this AI servants doing exactly whatever people want in the same way that software does whatever we want at the moment. Yeah. And as soon as you think, well, maybe, and quite probably those beings have model status, That no longer looks like an attractive vision for future society. Close to a dystopia. Exactly. Yeah. Whereas then, okay, go the other side. Okay, they have rights and so on. It's like, okay, well, now humanity- Well, now humans are totally disempowered. Exactly. So that doesn't seem good either. So I guess we'll do some middle thing. But what's that? What is that? It's just going to be some combination of objectionable in these two ways. Yeah. Exactly. Yeah. I suppose even if you give, because the numbers are just going to be so off, there's going to be so many more AGIs than humans, even giving them a tiny amount of weight just swamps us immediately politically. Yes. Exactly. Yeah. Maybe let's talk about that later if we get time. Okay. Yeah. Yeah. Okay. I guess another objection would be if there's a fact of the matter about what is right and wrong here, with all the improvements in science and technology and intellectual progress, we'll be able to figure that out and we'll be able to act on that information in future. Why shouldn't that super reassure us? I think two reasons. Main one is that I expect people, the worry is not that people won't know what to do, it's that they won't care. So I don't know, let's take animals today and factory farming and so on. Are you saying that we have any ethical or moral insights today that we don't all act on, Will? It's crazy. I know. It's a bold and provocative claim. But sometimes- Explain how that would play out. Sometimes I really put my neck out on things. Exactly. So take animal welfare today. There's a lot of information. There's a lot of information that is publicly available, that is in fact directly inconsistent with things that people believe, that people just, the problem is not really that people don't know about animal suffering. Or couldn't find out. Exactly. Exactly. I mean, yeah, they could quite severely find out, in fact, and people deliberately choose not to. The deep problem is that people do not care about non-human animals unless they're getting a lot of social pressure to do so and so on. And it's obviously inconsistent. It's a zero cost to them. Yeah. They care about dogs and pets and so on. Yeah. People are very inconsistent on this. And so similarly in the future, the worry is just that people won't, whatever the facts, people won't in fact care. And then it might well be quite contingent on early decisions, how kind of the balance of power and considerations play out, where potentially if you started off kind of locked into some regime where, look, AI is a software. It's exactly the same legal framework, then, okay, that's given the power and the kind of incumbency to those who do not care about digital beings. If instead there's some other framework or even some other set of norms, or even the thought of like, look, we don't really know what we're doing. So we're going to use this legal framework and it ends in- It must end at some point. It must end in two decades. And then we start again, clean slate. Yeah. Yeah. weapons might be a good example where we could end up with like very powerful biology research focused narrow AIs before we have ones that can figure out the policy solutions to that. Yeah, potentially. And I do think we actually have been very lucky in how the tech tree has shaken out that we're getting. So back in the early days, there was discussion of articles versus agents. Articles just like tell you facts about the world and help you reason are very smart, but aren't really good at doing things. And up until the wave of language models, the cutting edge of AI was not like that at all. They were agents, AlphaStar, for example, they were really good at computer games, not at all helpful at like- Understanding anything. Understanding, improving your state of knowledge. And now it's swung totally the other way where primarily the AI is just extremely good at helping you understand things. And the agentic aspect is lagging behind. But for some things, you could have incredibly powerful AI for bioweapons. AI for persuasion could come earlier. AI for human takeover, I think. So I certainly think that AI can come before AI that can take over itself. Because imagine your ability to take over is kind of a product of intellectual capability you have, understanding intelligence in some very broad sense that includes strategic ability. And how much power you have to take over is kind of a product of your ability to take over. And how much power you have to begin with. So if you're the president or you're the CEO of a company, you've got a very rich person, you've already got a lot of power. You don't need as much- Much assistance to- Yeah, as much intellectual horsepower to grab even more power compared to an AI that has no possessions. Starting from scratch. Starting literally from scratch. And indeed will be resisted from as soon as people notice what it's doing. Exactly. And it's being watched and so on. So yeah, so there's challenges that arise before you get helpful AI or sufficiently helpful AI. So yeah. Then there's also solutions whose window of opportunity closes before you've got super helpful AI. So earlier we talked about trying to get a kind of agreement between different parties behind this veil of ignorance. So agreements to kind of share power or at least share benefits after the development of AGI. Or, you know, Tom will talk about this more on the podcast about coups, at the moment, no one, or at least very few people would want the ability for some human being in the US to be able to stage a coup and take power at the time of that being possible. Because we don't know who's going to be in power. However, that window of opportunity to build in norms preventing that will fade once that power is actually, once that capability is actually there. Because now the people who could potentially benefit from staging a coup would not want that to be made illegal or very hard to do. I'm not sure whether we should keep this in, but just as an aside that I want to push out there, I think we desperately need the left in this conversation. As we were saying earlier, we're in a situation where all of these companies are talking about creating digital beings much more capable than humans. There's no plan for giving them any rights, any concern about their wellbeing. There's enormous commercial incentives to not to make sure that they never object, to make sure that we never make any change to that because then it just completely destroys the business model. There's lots of other areas in which I think the left would make a very useful contribution to this conversation. I'm not personally that left wing, but it would make a very useful and have a very valuable lens to cast on all of these issues. The left is just completely absent from this conversation because it's driven itself into this echo chamber where it thinks AI is not a big deal, or it's bad, but also not important and not capable. I just would like to beg people who have a more left wing perspective to get into this conversation as soon as possible. For sure. I actually think that's true on the left and the right. If there's one issue that they both agree on, it's being terrified of tyranny or dictatorship in different ways. People have really not woken up to how powerful technology will be, how quickly. I guess I agree that caring about the rights of digital beings, that already does sound pretty left coded, I think about other things. Or concern about the incentives of the companies is the thing that I was thinking, the perspective that they would bring that I think is underrated right now. Oh yeah, for sure. I think it's just absolutely the case that companies should be being held to account because if you believe, or even if you put significant credence on the picture that we talked about at the beginning, then the leading AI labs are among the very, very most important organizations in the world. That will ever exist. Even if you're just talking about today, you've got the US government and then the other AI labs, perhaps Chinese government, but they are as important as states, I think. Yeah. I thought that I keep coming back to is the thing that we're worried about here, that we don't have clever technocratic solutions to these various policy challenges, or is it just that we don't like the way power is distributed today? We think power is actually quite concentrated in a small number of countries and companies and individuals. They are probably going to exercise an incredibly outsized influence over the future and many people are going to be basically disenfranchised, not have any influence over it, and that kind of sucks. That might end up being very bad, but there's not really a simple intellectual solution to that. It's just like you're saying the initial endowment, the initial setup is kind of bad. It's not just a matter of, let's be smarter about figuring out how to do things. So I think, I mean, one thing is it's not merely that the initial setup of power is bad, it could get much worse. So power is pretty broadly distributed compared to, let's say, all the powers in the hands of one person, which is the worst case outcome, which again, could in fact happen. And there are some things though that are a bit more technocratic, as you say, where just if people are just making decisions really quickly and they're having to do that in quite a haphazard way, they're going to have to do it in a more technical way. I think that's a good way because there was no preparation done in intelligence explosion. Lots of those decisions are just going to be bad in kind of random- Bad for everyone. Yeah, bad for everyone, kind of random ways. There'll just be this huge attentional deficit more than anything, where if every day you're having to wake up with 10 gargantuan new developments and you can focus on two of them, well, maybe you'll just, and you don't have access to AI advisors because of some regulatory restriction or because you don't trust them yet. Yeah. Probably it's just dumb decisions in a way that's not even about concentrating power or ideological or something. So I think what you say is part of the story, but not all of it. Not all of it. Yeah. Okay. Let's think about other ways that we could potentially try to address this litany of issues that are going to be coming at us. I guess, yeah, we talked earlier about how one of the grand challenges, I guess, is the risk of epistemic disruption. So misinformation, people like losing touch with reality, I suppose, AI being exploited to persuade people of things that are false. Yeah. Yeah. Yeah. Yeah. Yeah. Yeah. Yeah. I guess the flip side of that is that there is this grand opportunity as well to use AI to improve epistemics in society. I think that this is one of the, I guess, avenues for making things go better that you're pretty excited about, right? I guess, yeah. What are some of the applications that you think might make the biggest difference here? Okay. Yeah. I mean, so one is this idea of very optimized education and training. So like I say, I think AI to educate is going to become focus, probably again, slower than it should do. I don't think, you know, just the current models could transform education. Yeah. They're a lot better than the teachers that I had. There's a lot better teacher than I would be. Yeah. I mean, I just, I agree. And by default, I think that'll go too slowly, but I'm particularly concerned because of this, such a rapid change, particularly concerned with just how well informed are CEOs, but then particularly political decision makers are going to be. Yeah. What I would want ideally is for the people in political power, they wake up, they get this perfectly optimized brief of everything that's happened in the last, you know, while they were asleep, which was probably a lot, focused on just the very most important things, perhaps such briefs, like throughout the day, some chunks of time are like, okay, now you need to know about space governance. We are going to give you the perfect, like most helpful, most calibrated to level of understanding education. So we need that product to exist. We also need political decision makers to have familiarity with it in advance. I guess this might sound fantastical unless you used deep research, which came out this week. It's producing exactly these reports. I mean, a little bit janky. They're not always right, but yeah, it's like really heading in this direction. Yeah, exactly. But I actually like, oh, you're saying fantastical that we could have such technology. Yeah. I thought you were saying fantastical that politicians would use it. Yeah. I mean, somewhat as well, I guess. Yeah. It would be a cultural revolution. Exactly. But you could have training for politicians as well to actually be using this and to be helping them. You could also just be advocating for government rules and regulations to be such that they can very quickly adopt new AI technology in appropriate ways. Yeah. It was occurring to me this preparing for this interview that I don't even have time to keep track of the launches of the new AI models that are coming at us literally week by week to understand how they can help me in my job. I can't take enough time away from solving the problems that I have to figure out how they could help me solve my problems better. And I don't run the UK or the US. So I imagine it'd be very difficult for Keir Starmer to take the week that it would require to learn about state of the art AI. Yeah. And I guess that's the problem. Yeah, exactly. I mean, I'm the same where it's on my to-do list to just take. It has to be like taking a week out to do a good job, at least a couple of days out to think, okay, given the latest iteration of models, how can they be most helpful? How can I fit them into my workflow and so on? Oh, we invented this like insane genius. I wonder how it could help me write blog posts for these podcast episodes faster. Exactly, yeah. Or how should I be changing what I work on? Yeah. I mean, you've really got to change your workflow potentially to get full use of them. And it's just like really hard. We're just bad at it. Yeah, exactly. And potentially there's some sorts of the search that. My guess is not. Not the current generation, but maybe 04, where, okay, cool. I was planning to do research topic X. If I do research topic Y, I can do a hundred times as much of it now because the models are really good at this one, but not so good at this. I think in particular for me, this might well be true for. Yes. In particular areas of more formal philosophy. So population ethics, decision theory and so on. Stuff that looks like math. Exactly. Yeah. Yeah. So we were talking about some ways in which people now in fact could be starting to help us prepare by having kind of beneficial uses of AI more widely distributed. So firstly, try and get them developed earlier. That's going to have a small amount of contingency, but a bit. Second is like applications. So maybe that's starting a company, one of these wrapper companies that just make them more helpful in certain ways. One of the ideas I've had is kind of like an. AI career coach, productivity coach, kind of life coach, health coach, helped you make the. Make a larger difference. Yeah. You got to do a set of a job. Yeah. Yeah. But. That'd be great. It would be great. Yeah. And so, I mean, firstly, just huge potential intrinsic benefits, like people get a lot of benefits from such sort of coaching. But secondly, again, it clearly could go either way in terms of what sort of advice, like, you now have this person, they are extremely smart. They're advising you. They're really smart. They're really engaging you in all aspects of your life. It could be that they advise in very Machiavellian, narrow self-interest sort of ways. It could be that instead they nudge you to be more in a more enlightened version of yourself. Could go either way. And yeah, that's something I would love to see. Deal-making AI is another one as well. This is a little more speculative, but there's potentially enormous gains. from the fact that making deals and agreements just has transaction costs. So we can take, yeah, I don't know. Maybe you know how to juggle, and I would like a lesson in how to juggle. But we're never going to figure out that we can make that trade. You haven't prepared this ahead of time, have you? Because you don't know how to juggle. Yeah, I'm realistic, yeah. We're actually going to like segue into a cavity now. Um, that was really good. I thought it was doing well. Sorry, shall I let you go again? I will admit. No, no, I like that. I think we should keep this in. Yeah, okay. It is true that I've thought less about deal-making AI than some other people. Liseke Veenstorp, who's working at Forethought, has done more work in this vein. Yeah. But the key idea is just, even just as a consumer product, there's little. There's loads of things where, if only there was just like a market, you could have a market kind of for anything. Because if I have an AI representative, can talk to your AI representative, discuss like all of the possible kind of trades we could make, that's like insane amounts of economic value, but could also be extraordinarily useful for navigating rapid technological development, because it could help enable agreements across countries too. So the US could have their delegate of AI representatives, China could have their delegate of AI representatives. And then two things. One is that, well, you could just have way more labor going into diplomacy, trying to find deals. But then secondly, also, they could have conversations that the AIs forget. So they could share knowledge that they don't. That both countries don't want the other country to have, but that are relevant for slaking deals. And then they output the deal that they reached, but then all of the negotiations and the information that was shared is wiped. Exactly. Yeah. Or at least like the crucial parts of it. Yeah. Yeah. Again, that's something that, you know, technological ability to do that will come by default, be much too slow. There's institutions to make sure it's trusted and so on. Yeah. Yeah. Something one could push on. I think you said that this sort of opportunities to use AI for good epistemic and bargaining purposes is an area where you think listeners might be able to actually do useful stuff, like basically starting now. Yeah. Could you elaborate a bit on that? Yeah. I mean, I think there's opportunities. There's opportunities within, you know, starting companies. It is definitely hard to have a company that is meaningfully adding value on top of the, if you're not, you know, one of the leading AI foundation model companies. But some companies do, Cursor and Perplexity and others. You could try and start a company like that, but focused on improving people's epistemics. You could work for one of the labs building these products. And, you know, there will be a lot of demand within the companies for trying to make products that are, you know, economically profitable, but also actually benefiting society. You could also try and advocate within government as well to try and get more uptake of AI models or, you know, be training the politicians in the ways we've described. Yeah. I think trying to like train influential people in society. As quickly as possible about what AI is capable of now. And like what they should be expecting to use it for in future years. Seems like it's something that many people can try to push on a little bit in many different roles. Because it also seems like it should be possible to put an economic, like a business model around this. Absolutely. I find it a little bit, it's quite surprising to me how few people appreciate. Well, I suppose it's just because they're very recent. They've been improving so quickly. But I feel like one of the most valuable things one could do with respect to all of this is just educate a typical person in the world about like, what is possible now that wasn't possible a year ago that wasn't possible two years ago. Because it's just like the disconnect is like, it's extraordinary. Yeah. So, I mean, one of the broad avenues for just how to help on all these challenges is just simply getting people to understand what is coming. And you make a great point that just simply explaining what's happening now is already a huge advance. So, I don't know if you know the YouTube channel AI Explained. Yeah, yeah. When those videos come out, I'm just immediately just like watching them because they're so in-depth. The single best short piece of content on the latest AI news. There could be a lot more of that. It could be a much, yeah, much wider spread as well. Yeah. But then, yeah, but then there's also just educating people on what's going to be coming too. And that doesn't have to be necessarily a large amount of people. So, take. Space governance, again, like how many experts are there within space law and space policy? Like, it's not that large compared to the world as a whole. If they, if that kind of community. If the dozen of those people were all persuaded. Yeah. We're just. And it doesn't have to be like completely persuaded, but at least we're taking really seriously the idea like. This is the emerging issue in our field. Exactly, yeah. That would just be like, man, it would be a lot better. I feel a lot better about that. Similarly, if. Other countries outside the US, especially countries with essential parts of the semiconductor supply chain. If like the Dutch knew. Hello, Netherlands. Exactly. If the Dutch knew how valuable ASML were, they would. Yeah. They would act very differently, I think, in negotiations with the US over export controls to China, for example. And could actually put quite a lot of pressure that might push things in the direction of more kind of multilateralism. Multilateralism, yeah. Than we have at the moment. Yeah. So, these applications are discussed a bit in the second part of my conversation last year with Carl Schulman, the one about society and government, so people can go there for a bit more flesh on the bones. I guess one we didn't talk about is using AI for forecasting, trying to. It seems like AI. I mean, there's been a bunch of back and forth about this, but it seems like it is useful as a forecasting tool, if not like already a superhuman forecaster for events that are coming up. Of course, it can absorb enormous amounts of material, much more material than any human forecaster could attempt to do, and it can work to synthesize all of that. I mean, it's only. I mean, they're going to get better. Yeah. And there's something you could, in principle, do with AI, too, which is train it chronologically, such that, you know, there's an issue for training human forecasters. You can't use past data because we know what's happened, so it taints things. Whereas you could have an AI where it's just learning. I mean, you need to label all the data, and maybe that's just too costly, but. Well, you're saying you need to make sure. You need to screen to make sure that there's no, like, accidental contamination. of the past data with things that happened later. Yeah, exactly. But in principle, at least it could just be learning over time so uh it learns 2005 to 2006 and then predicts what will happen in the next year or even the next week or month and then learns that and it's reinforced to do that better and then exactly yeah and so you could in principle have a ai that's just had millions of millions of times more forecasting practice than even the best human forecasters today. Yeah. Okay. Pushing on, I guess one of the threads of concern that we have is locking ourselves into some particular future, some particular suboptimal or negative future prematurely. What are some things that we could push on now that would make that particular path to not a great outcome less likely to happen? Yeah. So there's a bunch of things. So preventing autocracy is huge, or concentration of power more generally, where once you've got most power in the hands of a small number of people, or even in the hands of a single country, yeah, I think that just poses major risk of lock-in because the people in power at that time will want to maintain their power. And yeah, maybe even they only want to maintain their power for just 10 years, just over the. Over the particularly difficult period that they're facing. It's an emergency and yeah, we can't be. But then after 10 years, they'll maintain their power for 50 years more. Within that 50, maybe it's 100. And so you can have temporary lock-in that continues to indefinite lock-in. So that's one thing that I think is huge. Again, you'll talk more hopefully with Tom Davidson about what to do there. I do think space governance stuff could be really big. In terms of just not locking in stuff, I kind of think agreements, if we can, to just not send spacecraft outside of the solar system, not perhaps without international agreement, like at the UN. This was an idea from Toby Ord. Seems pretty good to me, even just some amount of delay, so that there's not this like crazy grab, scramble for resources and such that allocation is like very haphazard. One thing in general is just having a norm of just explicitly temporary commitment, so where any kind of new organization or new institution has an end date, a shelf life. That one's a little bit complicated, right? Because that could backfire if you're saying, well, things are changing so much, so we're going to rewrite the constitution every 10 years or every 20 years. Well, that provides a natural opportunity for someone to try to undermine what might have been a very democratic and liberal and non-totalitarian system into one that is. Oh, yeah. Okay. We'll get onto this potentially more later because we're dividing this up in terms of delaying lock-in and then improving lock-in. I see. You're saying, well, this helps to delay lock-in. It helps to delay it, but delaying might not be good. In some of the other work that I've done that we'll talk about in a little bit, one of the surprising upshots I had when I was modeling it all out was that. Trying to make lock-in better, conditional on it happening, seems to be even more promising than taking the strategy of just, okay, let's just. Picking the can down the road, yeah. Precisely because, well, if you kick the can down the road, maybe it's worse. Maybe society's more enlightened, we've had more time to figure things out, but also maybe there's just more time for power-seeking individuals to gain all the power and then undermine things in the future. Okay, yeah. We're slightly jumping the gun there. So yeah, I cut you off. So yeah, there's. Explicitly, yeah, explicitly temporary commitments. So in other work, Rose, Hadshah, and I, we have this case study on Intelsat, which was the first global telecommunications satellite company. And it was explicitly a multilateral project between different countries, with the US calling most of the shots, but other countries, especially from Europe, being significantly involved as well. And I was interested in it. I was interested in it. I was interested in it because I thought it was potentially a different. People often talk about multilateral projects for AGI, but they use CERN or sometimes the Manhattan Project. I thought this could be a nice alternative case study. But one of the interesting things about it is that it explicitly had temporary arrangements. So it was made on the understanding that within about a decade, things would be totally renegotiated. And that, in fact, worked quite well. Predictably, and the US knew this, they would have less power in a decade. But they were okay with that because what they really cared about was just the next decade going well. Because of the Cold War? Because of the Cold War, exactly. Yeah. So it was the early 60s. They wanted to win a propaganda victory against the Soviets. They wanted to be able to basically broadcast liberal values, in particular to developing countries that they feared might otherwise fall to communism. We could find, again, this is part of just generally getting people more alert to the possibility of an intelligence explosion, is have this norm of like, look, we're going to be going through this crazy period of time. We have institutions, they last over this period of time, and then we will be much more enlightened then. We will have navigated whatever tricky stuff we need to navigate over the following 10, 20 years. Okay, then we just kind of get together and there's some process for the institutions after that. Okay, so that was delaying a lock-in that's not so great. What about this other category of, well, assuming that we do end up locking ourselves into something relatively soon, how could we make that go better rather than worse? Yeah. I mean, again, lots of things here. So I was just talking about IntelSat as a model for potential projects to build AGI. I think in general, work, both kind of research work on design, but also, you know, I think in general, I think in general, I think in general, I think in advocacy in order to ensure that whatever project builds AGI is well-governed. That could be very important where there's a pretty plausible argument, especially if we get this fast and quite sustained software-only intelligence explosion, that that company, that project, or it could be a company, could be a government project, could be a multi-government project, could end up as a de facto world government. De facto rather than de jure, because it's not like anyone voted them in, but de facto because they would just have enormous amount of AI labor. In this scenario, it's like extremely super intelligent, gives a decisive strategic advantage, and they would get to choose, that project would get to choose how that AI is aligned. So that could be a really, really pivotal moment, getting that right. I mean, yeah, things are maybe looking particularly dire for progress on that at the moment, but I think still needs more work. There's questions around, yeah, questions around digital rights. Here, I think a couple of things are, one is just putting the issue on people's minds. I mean, people are somewhat, you know, there's some discussion of digital consciousness, still very fringe. The idea of rights of digital beings, again, I'm just like, I'm really not aware of that. I think there's absolutely a niche for some sort of public intellectuals to be talking about these topics, because it will become more and more of a issue of conversation. Yeah, it's funny that more people aren't taking this opportunity. So there's just so many things that are predictably avant-garde and weird now that are obviously going to become less weird and more relevant and more discussed in future. Why not get in early and make yourself the person who is concerned about this thing that obviously is going to be a bigger deal a couple of years from now, and you'll look like a visionary, even if you're completely derivative. Yeah, 100%. And this is, you know, of the things that one can do that I talk about, some of the things kind of are about AI, and they get this extra bonus in terms of how promising it looks, precisely because it'll scale in this way. So you set up a beneficial AI, and then you set up a beneficial AI, and then you set up a beneficial AI company. You do well in exactly the worlds in which AI is going super fast, which are the worlds we care most about. Similarly, you're talking about digital rights, you're getting, you're seeming, you know, prophetic in exactly those worlds we care most about. Other issues like space governance don't have that same, like, tethering, because space governance, no one cares. Well, there's a caveat on this. Nothing's really happening on it, and then suddenly a lot is happening on it. Right. It might become relevant earlier, because especially with AI advice, people are starting to foresee much better what happens. But it's still less of the case that you, like, build up more capital, whether that's social capital or credibility or financial capital, in the worlds we most care about. But sorry, so we were saying digital rights, yeah, so there's both, just advocacy, just, and it's not advocacy for any particular thing, but just, like, thing. Thing. Thing. Thing. - This is a nightmare and no one is talking about it. It's obviously coming up. Yeah. - Exactly. And I think it's maybe particularly important. So yeah, Nick Boston and Carl Schulman have some great very early work on the topic and, As they highlight, I think it's maybe particularly important for these people to be sensitive to all of the different issues that AI will pose, including takeover risk. Because you could imagine the person who organically starts filling this role being just someone who's single issue voter, ideologue- One way or the other, yeah. One way or the other, exactly. Well, I was going to say, yeah, it might be slightly challenging to fill the niche if you're just saying, "Here's an important issue, I've got no idea what to do about it." Because that just isn't as meme-able, I feel like the media is less interested in hearing about that than some dogmatic conclusion. Yeah. Although that's true across the board if you're trying to be a public intellectual or something. There's two things, in fact. Firstly, all the selection pressure is pushing you towards saying simplified meme-able stuff. But then second, you will just be saying a distribution of things, and sometimes- Even among the things that you say. Exactly. Some of them will happen to be maybe a bit less- Measured. Maybe you're just a bit annoyed at the time or something, happened to me recently. And that's the stuff that will go more viral, that's the stuff that people will pay attention to. So yeah, it's very hard to be a nuanced public figure. Yeah. Are there any other things that people should become public figures on? I mean, all of these. All of it. Yeah. We're talking about lots of. We talked about the big issue. Yeah. We talked about the big issues, the ones I think are most important, other than misalignment and bio, which are concentration of power, digital light, space governance, epistemic disruption. There is a long tail, I think, once you dig even further into these specific grand challenges, where I just want at least one person with deep expertise on them. So one person is like, "Okay, I'm the nanotechnology person," or one person who is the commitment technology person. Yeah. Or, yeah, other things as well. And I think that one is just the search to just understand it better, but then secondly is like being known as the X person, I think, especially for something that gets almost no attention, potential for huge impact there. Another category of approaches, so just to back up, we're discussing if there is a lock-in relatively soon, how can we make it go better? You talk about. Yeah. I think one approach would be doing better AI value loading, where I guess you try to make sure that the most powerful AGIs are taught to firstly care, or to reason and think about what is good and what is right, and also potentially taught to care about doing what is right for its own sake, rather than merely following whatever instructions they're given. That immediately poses a problem that you would be specifically training them to not follow human instructions and to do what they believe, what their own internal reasoning tells them is good or right. This increases the chance of takeover, surely. Yeah. So you could do both, is the thought. You can imagine someone who, they're a morally concerned citizen, think of a human. They just really want the United. Actually, let's even think, imagine them they're an ideologue or something. They're just super libertarian. They just want the US to become a libertarian paradise, so they've got this strong goal, but they're only willing to work within the bounds of the law. Yeah. Yeah. They're not willing to be deceptive and so on. And so I'm kind of okay with that. Very scared of the libertarian who's willing to do anything and is much smarter than me. And so in the case of AI, you have collegeable AI, so it's, well, I want it to be, like I said, I want it to be risk averse as well. So it doesn't even get that much payoff from taking over. Yeah. There's other things you could do to. Maybe it's myopic. It doesn't. It discounts the future very heavily. So again, it doesn't get that much payoff from a takeover, but then it's also courageable. Perhaps it's got non-consequentialist restrictions. It's just really doesn't want to lie, really doesn't want to do anything violent. Also it just knows, like always checks in with you. But nonetheless, there's still a difference between if I'm asking that AI for advice, for example. Does it just tell me whatever it thinks I want to hear? That's one thing. Or maybe what it thinks I would want to hear if I was sufficiently reflective, that would be like a lot better. Or thirdly, it's just a virtuous agent with its own kind of model character and is willing to push back. So perhaps it's. There's obviously a spectrum here, but it could say. I could be asking it to do something that is like immoral, but legal and not within its guardrails and so on. And it might say like, "Look, well, I am the AI, I'm ultimately going to help you, but I really think this is wrong." It could do that or it could refuse. So there's a big kind of spectrum, I think, within the AI character. And I think that's really important for two reasons. One is because I do expect there to be a lot of relying on AI advice. And I actually think it will matter what the character of the AI is like when human beings are relying on AI advice. In the same way as if you imagine the president surrounded by cronies and yes-men and sycophants. Okay, well, that's going to result in one outcome. You could imagine another one where it's like, "Oh, well, the president is surrounded by expert advisors. They have somewhat different opinions perhaps, but they also kind of like somewhat. " They're willing to push back. They're willing to push back on. I guess, ultimately, they might do what the president wants if the president really is just not persuaded by what they say, but they'll attempt to provide good advice. Yeah. And then there's also kind of shades to the like aligned with what question of just, "Well, do we want it?" You could have this perfectly aligned AI that does all sorts of very harmful things, but you can also build in more, kind of more constraints such that even if something is illegal, but immoral, it still refuses to do that. Yeah. But there's kind of two reasons I think this avenue is promising, and they do result in somewhat different implications. The first is easier to understand, which is just how it affects our kind of social epistemology and decision-making. And then the second one is kind of a Hail Mary in light of to guard against possibility of AI takeover, which is just. If AI does take over, there's still this huge range of possible outcomes. One where it kills everybody. Second where it disempowers humanity, but doesn't kill everybody, and in fact, leaves perhaps humanity with a lot of resources, and we are all personally much better off than we were. It's just that we're no longer in control of the future. Or perhaps it tortures all of humanity, because it really, maybe it's even a sadist or it just is interested, uses us all in experiments. Or it goes and does something just really good, in fact, with the future. It's like a model kind of reason of itself, figures out what's correct, judges that humanity was on the wrong track, and does it all like. It's still misaligned. It still takes over. It's still bad. It's not the outcome we want. But there's this huge range of possible outcomes, even conditional on AI takeover. It's not just like a one or a zero. Right, right, right. I guess people have tended, or the people who've been worried about misalignment have tended to think that the AI. They would just end up with these completely garbled, completely random preferences. But I guess because it seems like we're more likely to get alignment with something comprehensible by default, you're saying, "Well, actually, now we should think about, well, at the point where the AI goes rogue and does actually take over, it kind of matters what preferences we happen to have loaded it with at that stage." Exactly, yeah. And it does seem like we will have some ability to align it with something, even if not everything. Yeah, exactly. And so, yeah, we could be paying attention to, in particular, is it aligned with some particular set of values or is it aligned with a kind of reflective process? If it's aligned with a reflective process, both, that's like a larger basin of attraction, potentially a larger target, because you would hope at least that people from many different starting positions in terms of worldviews would all reason to a similar view, more enlightened view. And then also it just means, yeah, maybe that AI would in fact reason in that way. Just imagining an AI going rogue and taking over, and then it does further moral reflection and then changes its mind and is like, "Oh, sorry, that was actually not the right thing to do. I'm going to give it back." It's good. It happens with humans sometimes, at least. Yeah, yeah, yeah. And in particular, yeah, we're just now in a stage where it really does seem like the AI's values probably are going to be just like a big mix of different things. Hmm. It's teaching them to make difficult trade-offs a lot of the time. That's what we do a lot of the reinforcement on. It's like, well, how do you navigate between honesty versus rudeness versus annoying shareholders versus- Exactly, yeah. And it's starting off from human data and it's being built for a human world. But then also there might be some particular things that you really want to train it to do that is a bit orthogonal to all of this. So, yeah, I think that's a good point. I think that's a good point. No. There are various somewhat esoteric arguments that you can deal with in another podcast where, depending on the decision theory that it has, maybe it defeats humans in a much nicer way, even if it takes power afterwards. I think it's just one way of implementing just AI being a nice person, even if it's kind of power seeking. Okay. So that's been a whole lot on how to improve if we get locked in, how to improve how that locket goes. Are there any other, I guess, interventions that we should be thinking about making around about now that don't fit into either of those buckets that we've been talking about? Yeah. I mean, there are kind of cross-cutting things as well. And I should say there's way more ideas that we didn't have time to talk about. We're merely scratching the surface of all of the ideas that you've thrown out there in this enormous paper. I mean, it's a lot. They need more depth. It's shallow. But I think there's a lot of cross-cutting. I think there's a lot of cross-cutting. But yeah, there's kind of cross-cutting things where one is just, I've talked about this a bit, but just spreading some of the important ideas. So just what the intelligence explosion is, where we are, what challenges we'll face, actually trying to get by on that ahead of time. Second is just empowering more responsible actors where a bunch of people, no matter how much we try and distribute power, a bunch of people are going to have a lot of power at a very crucial point in time. And so I think there's a lot of cross-cutting. I think they're going to be making really consequential decisions. I want those people to be humble and cooperative and farsighted and morally motivated and communicative and accountable and so on. We're pretty set on all of those points, I would say. Yeah. Luckily, this one actually is already taken care of. We really don't need to worry about that one. Yeah. But if things get worse, then we could intervene and try to make it better. Sorry, I interrupted. Yeah. That's something that people can influence in terms of just who you vote for, which companies do you buy products from, if you work. Machine learning researchers, they currently have an enormous amount of power that they will predictably lose. That they are feverishly working to take away from themselves. Yeah. So there's actually this amazing confluence of self-interest among them and also, potential benefits for the world, because this is a way in which we can move some amount of power away from just leaders and if the governments get more involved, like the governments running the project, you could imagine this union of concerned computer scientists, probably informal, but just a group saying, look, we will only build this thing that will replace us under X, Y, Z conditions. It's kind of only a few thousand. I don't know, maybe like 10,000 at most who are relevant to this and are likely to be relevant by the point that these decisions have to be made. Yeah. That's just a way in which it could be another lever basically to steer things in a good direction. So it's not wholly just economic and military incentives, just driving the shape of the technology. Just to spell this out a little bit more clearly for people at the back, there's a few thousand ML researchers who currently have an enormous potential influence over the future because they have all of this intellectual knowledge. I mean, they're the people who over this relevant short period of time are the ones who are going to be automating AI research and setting off this chain reaction, the intelligence explosion. They are currently working very, very hard to automate themselves to the point where they are no longer required for this process and the AI will be able to entirely do AI research better than they could all by itself. At which point their leverage is completely evaporated and they no longer have any control over anything. Currently, they are working to bring about their own disempowerment while asking for nothing other than, I guess, a salary in the meantime. I think that they should bargain for more. That's very well put, Rob. It's very well put. And I should say, ML researchers are often just good people. They're just perfectly nice people. Yeah. Good people. They're just morally motivated. Many of my best friends are ML researchers. Yeah. Please, folks, please. Exactly. And there will be very hard decisions that will be much easier to make, much easier to make if there were some sort of, yeah, informal community or union or something like that, where that could be, hey, this company is just not holding itself to high enough safety standards. We're leaving. This company stinks. We're going to go to a better company doing the same thing, like literally probably on another block down the road. Down the street. Or secondly, in cases with much larger government involvement, saying, okay, well, I'm only going to work on to this project under X, Y, Z conditions, such as, I mean, and the conditions can be a really low bar. It can be- We won't build billions of killer mosquito traps. We won't build, any AI we train has to be aligned with the US constitution if it's built in the- It wouldn't assist with a coup. It wouldn't assist with a coup. It's pretty low bar stuff. But yeah, that lever does not, as far as I can tell, does not currently exist. All right. I guess just to recap, we've talked about why to expect an intelligence explosion, in general terms, what that would look like, massive speed up in scientific advancements and just lots of chaos, lots of things going on very quickly. Talked about what sort of grand challenges are generated by that. And we've been talking most recently about various different approaches, various different things we could try to lean on to make that time tend to go better. I'm going to draw a line under that. That's been preparing for the intelligence explosion and do part two of the conversation, which is on a different paper that is going to be available in the next couple of weeks. It's called Better Futures. I guess my take on it is it's about ways to improve the future that don't involve preventing extinction. Yeah. What were you trying to accomplish with this paper? Sure. Yeah. So it's really kind of an essay series and I'll say it's still a work in progress. So we'd hopefully publish it maybe a couple of months after this podcast gets released. And it's describing and really working through kind of a sub philosophy within long-termism where there are two ways you can benefit a person's life. You can stop them from dying if they're having a good life so that they live longer, or you can increase their quality of life. Similarly with civilization as a whole, you can stop humanity being destroyed, such as in a terrible pandemic, or even in those scenarios where it lasts a long time, you can make that civilization be better. Rather than worse. And for a long time, I've had the view that where the action is from a long-termist perspective is much more on making the future better conditional on survival than it is sometimes called trajectory changes to, than it is about extinction. And that's for a few reasons that I'm sure we'll get into. And so this essay series is wanting to really work through that in terms of one is just like, okay, putting down fully in words, the reason why I think some of these things secondly is figuring out in the course of that actually doing reasoning myself on well, which of those views are correct and what do they entail? And then finally, hopefully at the end is, um, that resulting in prioritization where we just talked about this litany of grand challenges and all these different things you could be doing to make progress on them. Well, if we have a better kind of theoretical understanding of what's most important and in particular on what sort of society we want to get to after AGI, then perhaps that means that given the extraordinarily scarce resources that the AI safety and border effect of altruism communities have, we can target the very most important challenges. Yeah. So, uh, I guess you, you're thinking of reducing extinction risk as kind of a baseline, or that's something that you want to compare many other things to. What are the other interventions that you're making that comparison with? Uh, yeah. So I guess two ways of answering that one is just, you could try and make worse than zero futures, uh, not happen. Um, so the suffering focused ethics, um, folks are particularly worried about that in particular, they were worried about the very worst outcomes. Um, or alternatively, you could try and take, uh, futures that are pretty good, but maybe mediocre or lackluster in some way. Um, or they're still better than nothing, but we might call them dystopian and instead try and get from there to farther than them, have something that's really close to, um, as good a future as we could feasibly achieve. And it's that latter thing that I'm particularly focused on where I mean, describe a future that's like achieves 50% of all the value we could, um, want to that, you know, we could hope to achieve. It's as important to get from the 50% future to the a hundred percent future as it is to get, uh, get from the 0% future to the 50%, if that makes sense. You know what I'm saying is just true by definition, but, uh, it's still somewhat kind of unintuitive. Cause you know, you really might think like, look, we just want to avoid extinction and then, well, it'd be really good in absolute terms or something, but, you know, I think if we're really morally serious, we should also be thinking about how do we get not just to a kind of mediocre outcome, or something that's in some ways good and in some ways dystopian, but actually to something that's close to as good as it could be. Yeah. Okay. So I think, as I understand it, you want to argue that I guess many people have been focused on preventing extinction, which is to say like sort of getting us off the zero point so we could have some positive value because there'd be sentient life still a long-going concern. You want to say that has been a very big focus, but we've missed the opportunity to maybe go from whatever default outcome we get if we survive to something close to the very best possible future that we could have. And I suppose the value of that sort of work will depend a lot on how good you think the future will be by default if we merely avoid extinction. If you think that by default, we end up creating something that's incredibly mediocre, that's only a hundredth or a thousandth the possible value that we could achieve, then, well, going from that 1% of the value to 100 is actually way more useful than going. From the zero point to managing to get 1% by, by, by surviving, but doing something kind of, kind of lame. Is that right? Yes. That's exactly right. Although I will say, even if you think, okay, by default, we get something that's half as good as the best thing that we could realistically achieve. The second half is still pretty good. Well, it's just as important in fact, becomes just as important to get from that halfway point to the, you know, best we can point as it is to just avoid extinction and get to the halfway point. Yeah. Why do you think it is that this, that that sort of work going from something to the very best has been kind of ignored or not that much discussed? I think there's a couple of things. One is that I think there is a bit of intellectual path dependence. So Nick Boston in his paper, astronomical waste, he works through on the assumption of, he's not a utilitarian, but on the assumption of classical utilitarianism. And what's the biggest, what's the most important priority. And he argues for a principle called maxi POC, maximize the probability of an okay outcome where an okay outcome means avoiding existential, any existential risk. And so that's, I think, been like kind of stage setting for intellectual, the intellectual development on existential risk that followed. However, even in his own paper, I don't think that really follows. Cause he gives kind of three perspectives. One is extinction. A second is a future filled with biological beings. A third is a future filled with digital beings. And says, well, the really important thing is that we avoid extinction. Whereas, you know, if you're taking his hypothetical, assuming classical utilitarianism seriously, no, the really important thing is either getting to the future with digital beings, if they have value or ensuring that you get to the future with human beings and not digital beings. And so I think that there's a lot of, there's a lot of, there's a lot of potential in this. I think that there's a lot of potential in this. I really start going through different views in model philosophy So you can start off thinking about what I would call, what I call kind of linear in resource consequentialist views, where total utilitarianism is one example. Basically just the more resources that you have, the more good stuff you can produce. So if you've got a good civilization, if that's twice as big, it's twice as good. On this view, it seems like, wow, it's going to be really quite hard to potentially to get like a near best future, something that's say like 90% of the best. Because you need to do two things. One is just use as many resources as possible, everything you can access, which is really, it's a lot, there's 8 billion accessible galaxies and so on. And then secondly, it also needs to be the case that how you're using those resources or whatever has the highest value per unit resource. And that could be like, who knows? There's a lot of things you could potentially do with resources and processes. So that could be like these tiny minds. It could be these huge minds, galactic scale. It could be like very changing processes and so on. Yeah. Yeah. Yeah. But out of the extraordinarily large ways you could like organize kind of mass energy, there's probably going to be one that's like the best and anything, even the second, you know, a third best might be significantly worse. So that's why it's like plausibly quite a real challenge on those views. It's a heavy lift. Yeah. Potentially a heavy lift. You know, you've really got to aim at this one specific thing. I should say, as I'm going through these views, I'm not like defending any one of them. And in fact, it's a notable thing that most theories of population ethics that are on the table have this conclusion that the best possible future is basically homogenous future, where you've got something that's the best thing at some scale, it might be quite complex within that. And maybe the scale is large, a solar system or something. And then you make more of them. Once you've got that, just make more and more and more of it. Because that has the highest value per unit resources. That's the best thing. And some work we definitely won't have time to talk about that is work in progress is. I find that conclusion like particularly. Perverse. Yeah. Yeah. Particularly terrifying even. So I have this new theory of population ethics that I was working on. You know, if we had more time, it'd be the most enjoyable thing I could do. But that's paused for now. But theory of population ethics that I think is quite compelling, that avoids that implication. Okay. Yeah. Interesting. But anyway, that's the linear and resources view. Your computer can come back to that at some point. Exactly. Yeah. The AI will do it in like three months or something. A couple of years. Yeah. So that's the linear and resources view. And then you've got other views, for example, views in which, look, it's not that hard to get to a near. Like to get to the kind of upper bound and views on which there's like a much richer kind of diversity of goods. Part of what my work has been doing is like going through all of them based on these kind of structural features in order to see like, can you get the implication that, okay, yeah, really just like quite a wide variety of different views of different ways the future could go. Give you something that is like close to the best anyway. And I ended up concluding kind of no, where if you don't have a significant thing is like, even if it's easy to get the upside, because you just say, look, you've got an earth full of like amazing, like flourishing beings who are all free and able to do what they want. That's like, even if that's close to the upper bound. You also need to make sure you're not, you don't have bads too. And this is both, I think part of common sense. I could tell you a story of this like amazing utopian future, but just has one floor. It's based on some rigid racial hierarchy or like in the ones who walk away from Omelas. It's founded on the suffering of like a single child. I think that common sense, you lose a lot of value. As in common sense. I think it's enough to like turn it from a utopia to something at least semi dystopian. But then also formally you've got all the goods they're kind of close to the upper bound, but you've got this really big civilization and it has some bads too. Even just occasional bads like suffering. Then just given how big the civilization is, that'll take you far away from the upper bound again. Sorry. That got a little more involved. You might have been intending. So you've just given us a really quick summary of this, I guess, paper that will eventually come out fingers crossed in time for it to be useful called No Easy Utopia, or I guess you argue that sort of easygoing views where it's like not that hard to achieve the best or a near best future, that those are not really plausible or they run into all kinds of different problems. Yeah. I mean, I think, yeah, the more straightforward way of seeing that is just to look at kind of big moral disagreements that we have today. And realize that basically if we get any one of these things wrong, then that couldn't make the world negative, or certainly at least would destroy the value that we're generating, or would reduce it very enormously. I mean, it's just almost anyone you can name, right? So abortion, if abortion is really wrong, and in fact it's equivalent to murder, then that would surely be a massive stain on human civilization as it exists now. Conversely, if it's not problematic at all, then all of the restrictions on it would also be very bad. I guess like destruction of the environment. If that's as bad as it is, then it's not a problem. If that's as bad as some people say, that would be extremely bad. I guess if religious folks are right, and there's many, many things that are contrary to what God would want, then presumably we're committing many, many infractions. And I guess you can go on and on down the list. And so it seems, unlike most people's views, value is quite fragile. And if you get even a limited set of all of the different debates wrong, and then you scale that up enormously, then you've fallen quite short. Absolutely. And in fact, the situation we're in at the moment, I think, gives us. Gives us lots of false. Convergence. False cause for reassurance. Exactly. Take theories of wellbeing, for example. In philosophy, there's three main theories of wellbeing. Hedonism, which is positive and negative conscious experiences, or what's of value for someone. Objective list theory, which says the other goods are good or bad for you intrinsically. Beyond that, like. Yeah. Yeah. Yeah. Yeah. Yeah. Yeah. Yeah. Yeah. Yeah. Yeah. Yeah. And then preference satisfaction views, which say that what preferences. Yeah. Having your preferences or your idealized preferences or something satisfied, that's what's good for you. And in the world today, these line up really pretty closely. So you and I work together on research for giving what we can, what are the most effective charities to benefit people in the developing world. And then we have the other two theories. Yeah. matter it's like it's more like a scientific or empirical question yeah exactly whereas like all the subjectivists i know who they think that um what makes something right or wrong is just what i would want or want myself to want on um on idealized reflection yeah often they seem to think that you'll get convergence and i'm just i'm like baffled by this like if i'm a subjectivist at least i expect radical divergence um between different people based on just um uh subtle and arbitrary facts about their psychology essentially because there's nothing constraining you um uh ultimately well i guess we have similar genes so we have a similar like ultimate design yeah we just but it does differ yeah we do but it seems so take even people as similar that they're both classical utilitarians so you they add up happiness in the same way and they both agree that positive and negative conscious experiences um are the things that are of value but i like these conscious experiences and you like these conscious experiences there's no fact of the matter beyond our preferences about which are good or bad so why on earth out of all the conscious experiences that there could be which really are a lot but why on earth do we agree on which are good and bad but then more importantly why do we agree on um the quantities of goodness because again remember that what what you end up doing as a classical utilitarian is figuring out what has the most value per unit resource if you if you and someone else differ on the magnitude of goodness from different conscious experiences even by a bit you will differ um utterly on what that highest value per unit resource thing is um and if you're the subject sorry i'm ranting if you're the subjectivist if you're the subjectivist then there's nothing constraining you ultimately between those two positions and we were talking about like the narrowest disagreements once we start thinking beyond to like other forms of consequentialism other theories of well-being um yeah i think you'd end up with very very different um views indeed okay um we're getting into some philosophy that i didn't necessarily intend us to get into there uh but let's bring it back to this uh overall issue of discussing ways of going from like okay futures to to the very best i mean actually while we're still on the philosophy and the subjectivists should the subjectivists and non-realists about ethics morality care about this entire agenda because they might not think that there really is such a thing as their very best future other than just like what they happen to like and so is it is this kind of work relevant to them so i think basically 100 yes um there are definitely implications depending on which meta-ethical view uh or views you think are most likely but basically just if you think that some outcomes could be bad some outcomes could be good um some outcomes can be better than other outcomes then you're you really ought to you know you will in fact have views about okay well this outcome could be really good and this outcome could be only a little bit good you might and you might have a like easy utopian view or something where okay you're not as concerned about getting the like very very best you think lots of things are really good but at least if you're concerned about um consistency of your preferences and maybe you're not that's actually fine that's like one one way of responding to this whole thing but like if you are then you're going to have to respond to some of the you know arguments i'm making about why even those kind of why those easy utopian views are like quite hard to come by but yeah it will still be the case that uh you'll want to grade different futures in terms of like how good they are and how bad they are and like and the differences could be very big and differences could be very big i mean like i think it's a perfectly common sense position to think okay well like a future with extreme concentration of power might be really bad okay well what if there's a future with extreme concentration of power and there's a future with extreme concentration of power but people are just like kind of evil like they like promoting suffering and i think okay that's kind of clearly bad it's like okay well what about other moral mistakes and yeah i think you can we can often dodge a lot of like meta-ethical argument by just saying like okay well what we're gonna how we're gonna uh evaluate things is just by what we would want things to be like if we were idealized we had loads and loads of time to reflect and loads of we had like really big brains we were able to hold all knowledge in our heads and so on because um the realist will say well that's the best thing we've got to track what is good what's actually good what's actually good and bad and the subjectivist will say well this is what we got this is what in fact yeah yeah well what actually yeah i see it's just the preferences that make it good yeah exactly yeah yeah but it doesn't yeah it doesn't for practical purposes i guess all of the analysis basically looks the same yeah you're just measuring it against a bar that is justified differently yeah okay so let's just do this so the modeling that you did so um there's different approaches that one could take to improving the future of course it's like preventing extinction there is trying to go from like uh from not a bad future to a neutral one to a neutral one to a really good one i guess um you also break it down in terms of trying to uh trying to delay lock-in or improve lock-in should it occur different intervention one might make and you you go through this slightly complicated modeling exercise where you i guess you theorize about this and look at of course because these things could interact with one another like you could do a lot of things like um if you go extinct and you can't get locked in if you get locked in that might make that might suggest that the future will be better or would be worse uh and then you uh basically try to use that analysis to figure out well which one of these things probably is going to be the most influential and are some of these things like important in ways that people haven't appreciated or not important in ways that people haven't appreciated yeah can you explain the modeling you did sure yeah i mean so it was just a very simple way of modeling um the present and the future both its trajectory and its value um kind of loosely inspired by um a markov process um but the key idea is just we've got the present there are two periods this century and future centuries very simplified there um in this period this century uh we could either have um an open future where it's not kind of guaranteed where we end up we could go extinct if we go extinct we're guaranteed to end up extinct um we could have what i call and this is really the most or among the most important concepts i want to kind of promote in this series have uh what i call viatopia um so via as in like a waypoint where it's a state of society such that if we're in that state then we're very very likely to end up in a near best future and very unlikely to end up in a dystopia or we could have um i call it kind of a foreclosure where if you're in that state you're very unlikely extremely unlikely to end up in a near best future so you won't go extinct you're going to produce a mediocre future exactly very high probability yeah and i i tend to call that a mistopia where like mist file and misfiring but also like a missed opportunity which is etymologically distinct but um uh helpful all the same um and then there's the kind of future centuries where we can end up um either extinct in this kind of yeah utopia near best future end up in this mistopia uh or which includes things that could be worse than zero as well and where if we're in an open future now so we haven't got locked in then ultimately we end up in one of those states too and so then the two things that we can do this century is reduce the chance of extinction and increase the chance of everything else uh we can reduce oh sorry three things we can do is that uh we can decrease the chance of lock-in so increase the chance of open and decrease the chance of everything else or we can make it more likely that condition given there's a lock-in it's viatopian rather than foreclosure and this is a simplification in all sorts of ways i talk about lock-in even though there's a more general concept of path dependency which doesn't guarantee a particular outcome just makes it more likely and i'm also just dividing like good state and bad state essentially and you know giving them values whereas of course there's a big difference between a future that's merely 89 as good as it could be versus something that's like substantially negative um but i actually found the modeling exercise like it's obviously weird to be doing ethics on a spreadsheet but um i actually found it like remarkably elucidating even just for my own views to be reasoning through some of this stuff yeah i guess it seemed to me just um scanning it intuitively that most of those simplifications wouldn't produce they wouldn't flip the results or they wouldn't really cause the like the stylized facts that you were going to produce to be wrong i guess that's your view as well yeah exactly at least i hope so and if it was then it would be a bad model it would be yeah yeah just a conceptual issue that occurred to me um reading it is i guess you talk about for example you're in the viatopia if you're in some intermediate state where you haven't yet achieved utopia but you're very likely to achieve utopia you're in foreclosure if you're very likely to produce something that's kind of mediocre um and you're in an open future if you can't tell but that is that's kind of from one person's subjective position whether you can tell or not whether you're locked into a future or not yeah because i guess quantum physics aside universe is like no matter how complicated and chaotic things seem to begin with it is going to produce some outcome so you kind of are either locked into a good outcome or a bad outcome or whatever outcome it's going to be and and the fact that you're in an open future is merely descriptive of your own inability to forecast what is going to happen is that is that right and if it is right is that a problem yeah so it's um i'm smiling because it's a very philosophical objection to make um so it's kind of kind of right so um okay yes absolutely true you've got a quantum quantum randomness aside you've got a state of society you're guaranteed to end up in some particular state and um i think it's laplace's demon is the thought experiment of uh uh being that could that knew all of the laws and could do all the computing um however i don't i'm not meaning the probabilities here to just be like oh whatever your point of view happens to be but i'm not meaning the probabilities here to just be like oh whatever your point of happens to be such that if you were you know completely off base and thought that the world today is locked into lizard man future there's something that that counts instead i think there is a case you can have of um some notion of objective probability or at least evidential probability so take flipping of a coin now if i knew enough i would know whether the coin comes up heads or tails but we can give sense to the i the very intuitive idea that it's 50 50 by saying well okay you've got the system which is the coin on my finger make just small perturbations of it in not very kind of significant ways and then look at the frequency of what happens given those small perturbations where that would result in 50 heads 50 tails for a fair coin and so now you're smiling okay and then um you could do that with you know civilization as a whole where if you're locked in then it doesn't really matter like what's the weather like or like was it like warmer or colder um perhaps uh yeah did someone and lots of other like you know fairly minor points about the world today instead you'd still end up in the same direction yeah i was going to say if you liked that pointless philosophical inquiry for me you might really enjoy our interview with alan hyatt from two or three years ago i think it's a puzzles and paradoxes in probably probability uh and decision theory um where i guess yeah i think we end up concluding that probability is is around these like um perturbations so you've got like nearby counterfactual worlds and how like how likely are you to get the outcome that you're describing uh so i guess yeah seems reasonable okay um okay so what sort of parameters did you need to plug in in order to build this model and i guess figure out like you know what's the impact of changing the different parameters yeah so i'm not sure i'll be able to remember all of them but but some key things are probability of lock-in um this century probability of extinction this century if there's lock-in probability that it's um good rather than bad namely viatopian rather than foreclosure um then if we have an open future this century probability conditional on that of extinction uh utopia misstopia and then um value of misstopia uh if we get locked in in the bad way this century like what's the value of that bad lock-in because it might not be zero you know could get locked into something mediocre um and the value of that lock-in if it's a lock-in in the if you know that occurs the non-utopian future occurs in the future where you might think oh it's probably even if we miss out you know if you think generally things are progressing you might naturally think okay well even if we miss out on the very best probably it still ends up going better if we didn't get locked in this century and i think those are all the parameter values yeah yeah i may well miss some we're a couple of hours in here so i think our minds are getting a little fuzzy but yeah i think that's i think we're over four hours in yes okay do people agree on these parameter values uh or they all do it i guess it's what extent do people like massively disagree in your experience well i would love as a follow-up to actually start surveying people um the one thing that i know of um that did somewhat do this was a fairly informal survey done by the future of humanity institute and it was just convenience sample it was just i think people at a particular conference and there people were really over them all over the map um i can't actually remember the details but people can go all the way from thinking you know probability of getting to a truly flourishing future conditional survival was like 90 percent all the way to like i think there was one in 10 million was also kind of on there um similarly with probability of extinction i think people varied from 90 percent to less than one percent i mean they were asking different questions than i had but um uh it definitely showed you there's like a lot of uh yeah i think there's a lot of yeah a lot of there's not a consensus yeah uh and what about um i guess a critical thing or it's an important thing is whether lock-in becomes more likely to become positive if it's delayed yes um versus so do we want to lock in things quickly or do we want to like push it out into the future as much as possible i guess allow more reflection um yeah do people agree or disagree about that yeah well this was actually something so i really don't know um that wasn't in the exercise was just appreciating how neglected yet important that question is where i'm certainly coming from a perspective of thinking look we are like these children you know it's as if uh yeah it's as if you've got to like decide how your entire life goes but you're 12 you're seven exactly um look we should have smarter more enlightened more informed people in the future make those decisions and so i do you know in my best guess i do think like yeah if we can punt um these decisions into the future that's just gonna the decisions will get made better but it's quite non-obvious actually um i mean again i think um something that's confused me about subjectivists is that i think subjectivists probably shouldn't want that because they won't be around um or maybe they will be but they think they will be but if they don't they won't in those cases where they weren't so their particular preferences won't get satisfied um but you also just might think there's various sorts of kind of essentially moral decay that might happen so you might think it's just much more likely that um uh you end up with concentration of power at a later date um you might think that ai totally messes with um people's kind of epistemics um i actually think there's various things to do with um people would reason about or how people would make choices about kind of distant time and space where now at the moment it's just mainly like moral preferences that they have because it's so far in the future it feels more speculative exactly and given your preferences people's preferences at the moment you don't need that to get be as basically as happy as you can however once people are already extraordinarily well off and post agi oh now perhaps they want it but as a status good or something i see you know you want no just to show off exactly now because i mean this is already the case like look at the billionaires in the world um they just hang on to their money yeah in a way that's like baffling honestly um not even really clear that it's delivering much of anything for them no i mean you clearly like are not getting more well-being in the narrow sense once you're past 10 million dollars of consumption per year it must be about satisfying some other preferences exactly the well there's inertia yeah but even yeah um there's inertia but the kind of obvious thing is like well they're buying this like status good they care about what position they have on the billionaires list and they are willing to effectively pay tens of billions of dollars rather than spend that money on like something else including kind of moral goods so you could have this future where it's like okay yeah now people are like reasoning but now they care about um yeah resort you know distant resources just as a matter of like personal status rather than at the moment people would have a kind of more ideological lens on it in the in a potentially positive way so like i say i actually do think like probably people in the future they're going to be smarter they'll be more enlightened there's more time to think about things we are not uh do you have any idea how dumb i am yeah exactly we are not and i'm one of the best we are not equipped to be making such weighted decisions um , but it was a notably yeah it was like felt notably understudied to me yeah okay so what were your main takeaways from all of this analysis i think did it take you in a direction that you didn't necessarily expect uh i mean in a few ways so that was one actually the importance of this and to be clear what i did was like forced myself to actually put numbers in these things and that was pretty interesting myself we started this conversation with talking about um two ways of putting it um two ways of putting it and the first one was like how do i put numbers in these things the updates I've made in the last seven years, or the ways that I was wrong, earlier on i'm perfectly happy with that latter description but that um yeah those changes hadn't necessarily filtered through to um other of my views so it was interesting for me putting my probability of extinction probability of lock-in they're much much higher than they were um in the past so that process alone was actually just like very elucidating um for me other takeaways one is that this better futures work i think it's pretty robustly in the same ballpark of importance at least as reducing the risk of extinction yeah i think that's true and just across quite a wide variety of views um one thing that was interesting in particular is that what i might say ordinary um ordinary ranges of optimism versus pessimism about the future didn't really make that big a difference to the bottom line so uh i had always thought of like the biggest intellectual difference between me and toby ord was the our level about of optimism about the future but actually you're on the numbers and i'm like no just i know within you know all of these are like order of magnitude kind of things yeah um didn't really drive things why is that well so toby being i think he thinks avoid the big existential catastrophes or at least the most obvious ones essentially i don't want to put words in his mouth exactly um but under some sort of term existential catastrophe the expected value of that future is like 50 percent 50 percent of what it could be um and that seems very optimistic to me um i have been there's various reasons why my numbers have been argued up over time but i'd at least put that like many times lower or something 10 or something but he still thinks there's 50 of value that's still up you know um that's still up for grabs so actually the level of optimism that's been argued up over time is the level of optimism he would need to have is it would need to think it's more than 90 percent like 99 percent of value or something i see okay so the reason is even the extreme optimist thing we're only going to get halfway there on average that's only a 2x penalty relative to someone who thinks that the we'll get zero on average is that right yeah i'm i'm too brain dead at the moment to confirm the 2x thing but it sounds plausible yeah yeah whereas yeah whereas there's what i call the topians where it's no it's more like 99 because just just surviving and perhaps you know we might want to say like and avoiding like alter dictatorship and so on but avoiding those things basically is good okay yeah yeah um yeah were there any other upshots uh yeah one was i actually beforehand i hadn't fully and cleanly delineated the delaying lock-in from improving lock-in um again there's all these updates that just sound so obvious um but actually in my mind i don't think it was that clear and then i certainly hadn't appreciated the arguments for thinking that improving lock-in conditional on it happening um could be a lot more promising than delaying lock-in where the key argument is just well if you delay lock-in maybe you've made things worse right i see so at least if you're doing the thing where you take it from like nothing to the very best so that's your intervention yeah um there's negative which is that well by merely delaying things it's like well that might improve things might make things worse but like and the net effect is like a lot lower exactly exactly um interesting i guess so the upshot is that let's worry a little bit less about the extinction thing instead what we should do is try to like take whatever future we're going to have and instead turn it into like this utopia the very best possible future that's incredibly fragile and incredibly narrow target uh we think um i think to many people that might sound a little bit utopian yeah um and i guess people have negative associations with utopianism because it tends to justify negative behavior it can just seem kind of naive uh i guess people who tend to be utopian tend to be like very locked into quite a narrow perspective on on what's good i guess that tent that's the sort of mindset that drives people towards utopianism yeah um why isn't this the sort of bad utopianism that has justified all kinds of bad things yeah great it's a great question and i'm uh very aware of the fact that you know i use the term utopia a lot um i like spell it with an e utopia uh utopia try to get some distance from the associations of the concept yeah though it sounds the same but yeah utopia is extremely unpopular as a term um and as an idea and honestly i think for very good reason so utopianism was much more popular at the end of the 20th century sorry end of the 19th century early 20th century and uh and some things happened didn't age well um but lots of depictions of utopia didn't end well didn't age well either so uh my partner and i have a bit of a shared hobby of reading kind of old utopian uh fiction and it's really remarkable the extent to which those utopias now look like uh dystopias um having even like not that much kind of moral progress has happened it's like only essentially which is small in the grand scheme of things uh but very often societies are totalitarian um very often um they bake in kind of moral blind spots of the time so thomas moore's utopia person who coined the term had uh yeah amazing abundance this wonderful society in many ways every household owned two slaves um not so appealing nowadays um similarly uh aldous huxley who wrote brave new world also had a piece of utopian fiction called island um again this you know very technologically advanced uh society but also in touch with nature uh the adults had sex with children um lots of other you know again it's just like whoa okay um also yeah wasn't wasn't aldous huxley doing that in the 30s what oh yeah i mean it's i mean i'm surprised that that was uh the kind of thing someone would write in the 30s uh i mean this is definitely a digression but like even in the 60s and 70s um the french kind of existentialist and philosophers were like signing uh an open letter to say that i think there should be no age of consent or it should be like much lower than 16 um it's really easy to forget forget how quickly moral attitudes change right and um actually how potentially how morally different even just a hundred years ago was even less um so yeah so utopia depictions of utopia are often in fact end up looking really quite um dystopian i think that's true from plato's the public onwards and the lesson is like we shouldn't be trying to depict some particular vision of the future and aim directly uh towards that in fact that's terrifying that's what we should be guarding against and this is why i'm going to i'm trying to develop and promote this idea of utopianism and i think that's a really important part of viatopia where you know that's a way station instead that's acknowledging we don't know what the ideal future looks like and instead is a state that we just think would be on track it's good enough um that it's on track to getting to a really good future where i've talked before about the long reflection um as an idea that would be one implementation or one like proposal for what viatopia would look like but there could be there's other potential proposals too like the idea of a morally exploratory society which i talk about in what we are the future or the idea of a grand bargain between different um value systems and my hope at least is that this can give a positive vision which is extremely lacking in the world today so a positive vision for the post-agi future without the terrifying uh and yeah often like totalizing kind of utopian impulse that we saw in early 20th century in particular yeah what would this biotopia be most likely to look like i guess make me think you're making me think of you know we'll have a parliament or a congress that will people you know people will send delegates and they'll debate what would be the best future for a very long time and i guess they won't won't take any irrevocable action until there's like a super majority or like a super super majority in favor of some particular view and if they can't reach that then they'll bargain things out and and spit things up but i think that's a good idea i think that's a good idea i think that's a good idea i also like try to make sure that no one's doing anything that other groups particularly hate is that is that are there any alternative visions for a biotopia or is that kind of a direction it would go yeah i mean there's just plenty of detail in there um and i'll say like this is uh you know after this better futures essay series has done that the next one i plan is on uh viatopia in particular so this is looking ahead to work i haven't done and lots of things i still feel very unsure about um but there's tons of granularity in terms of one i think one is in order to get to a near best future do you need most people or basically most people to converge onto let's the correct or you know the correct model view or whatever your idealized preferences would be or do you just need some people to converge and then to have the ability to trade between different parties that's like a really crucial distinction in my view where a case for thinking that you just need some people is if well those who don't Perhaps optimistically, those who aren't particularly model-motivated, they have comparatively small-scale preferences. Perhaps they really care what happens close in space and time. The people with model motivation, they care about what happens far in the future and far away. And so that both parties can get kind of what they want. That would be like the optimistic case. It's less optimistic if lots of people have different model views. And don't converge there. So you still then got like a lot of conflict between the people who do care about things that are far away and distant. And they want to do very different things with those resources. There are also ways in which that kind of setup could go badly wrong. Because if we're all bargaining over resources, maybe I'm incentivized to modify my own preferences. To make them stronger? To make them more. Well. Worrying from your perspective. Oh, I see. To make them more menacing. Exactly. So let's say you're a classical utilitarian. And maybe I just like having more wealth for status or something. Just care about something else. But then I can self-modify in whatever way in order that. Oh, I actually really like suffering now. Well, then the status. The threat point, the status quo. If we don't manage to make a deal is much worse from your perspective. And so. So it gives me a reason to concede more ground. Exactly. Unless I object in principle somehow to the blackmailing situation. Yeah, exactly. It's just like, well, I will never accept being like treated like this or being exploited like this. Yeah, exactly. But it at least gets hairy. And so, you know, there are some arguments for optimism for thinking like, okay, just have like lots of different. Even if people end up with like super different kind of. Moral perspectives, they all do trade. And we end up with, as a result of that, something close to as good as possible. There's some worrying dynamics that, you know, I have not gone kind of deep into this. That could actually mean that this ends up really quite bad. Because the worst case then is like, I self-modify to love suffering. You say, well, I don't. I don't trade with blackmailers. Yeah, I'm not accepting this crap. But I've already irrevocably self-modified. So then I go and do it. That's not an outcome we want. Let's not do that. Yeah. But then, either in that case or in the case where like, no, actually, we want to try and get to something that is much closer. Where like basically most people, if there are like more correct moral views, they kind of win out over time. Then there's a question of like, well, how do you even structure that to begin with? There's been some work in this with respect to kind of deliberative democracy, which is pretty interesting. Where? I mean, I think that's a good question. I think that's a good question. too busy for that and like excited about this new thing and so uh yeah what i did um was yeah started focusing a lot on research i also did other things like i did more much more yeah more holidays and um a couple meditation retreats that were amazing and then as a result of that got the kind of discussion group going um with toby ord and finn morehouse um who have co-authored this preparing for the intelligence explosion paper with um and then that kind of grew a little bit in oxford people and i was like yeah driving forward to kind of things that i thought were particularly interesting there started to be like a little bit of a kind of excitement about that and then things like really changed gear uh when uh max dalton was considering he was also burned out we were like chatting it's like hey so um um yeah we're both hating things right now yeah yeah yeah what's next for you yeah um he was deciding kind of what should he do next and i just said i would love to work with him and then he ended up concluding like okay cool yeah let's make this i'd started to have the seeds of like actually this could be some sort of research institute um and kind of he got on board and then uh i was describing this as my like holiday period um and i kind of got like forcibly shunted to it being non-holiday like a few months later in a way that i'm very happy with i was extremely happy with my work um but i also yeah i did also just do like a quite a sustained and formal kind of process for deciding kind of our next project where uh it was either kind of continuing with the research research institute could also just be like you know a little focus on fundraising um kind of more outreach could be more focused on like the research institute and then kind of like having a podcast and youtube and so on um very lucky for me that you didn't uh bad for the audience but good for me not to have the competition oh maybe we could have become co-hosts and then kind of some other things as well and i actually just tested a lot of things i had a week where i tested out doing a bunch of um podcasts and um video episodes um i also got like uh really extensive feedback and so on i got this um worked with a kind of executive coach uh who did this thing that was like it was very impactful it's quite intense but it's just he asked 20 people maybe over 20 people yeah i think i was one of them oh there we go so i've seen your responses on this huge spreadsheet so yes 20 20 people including he gets me to ask like who are the people that i would most like not most find it painful to get feedback from and the questions were like what are your strengths and weaknesses strengths and weaknesses primarily yeah it's just this huge spreadsheet with just like a list of all the people that i would like to get feedback from my strengths and weaknesses this was your break right from this was on holiday like literal holiday i was in okay i have work to do on like improving my holidays still yeah um but it was that was actually hugely impactful because i do think no matter how much you try and get feedback in your work life and like one-on-ones and so on there's something people just go a little bit harder if it's like anonymous directed via someone else um well and that person's like really probing you being like oh you suggested we might have this weakness like can you really play can you really put your finger on what the issue is yeah exactly yeah so that was super helpful and one of the big things that came up was focus where again looking back like by the time um it was 2022 just built up so many kind of formal and informal responsibilities in a way that was like basically impossible to uh maintain and stay sane and so um yeah i've much more now try to be like look there's going to be one thing that i'm really pushing forward which is um yeah this is the search so um yeah that was the very personal kind of angle on uh um what you're asking yeah um there were at the same time just like the changes happening in ai okay well if there's a time yeah if there's a time to start working on this but then also just like the rise in ai safety is a legitimate field that now yoshua benjio and jeff hinton were talking about there's ai safety institutes and so on very obvious like next um very obvious thought of like okay well what else are we kind of missing here and at like various conferences and meetings it seemed like okay there was broad interest among people working in ai safety and the effect of altruism movement look like yeah this is kind of important and i just started feeling excited about it yeah so i guess um we've kind of given a broad overview of i guess the portfolio of things that forethought could end up working on um and i guess so so there's you doing this better futures thing is going to be the main focus going forward and we're having this interview with tom davidson about the seizures of power because i'm not sure what whether he'll just keep focusing on that or uh or is planning to move on to it to anything else is there anything else that forethought is doing that we kind of haven't talked about in this interview or the interview with tom uh sure yeah so tom davidson's work on concentration of power obviously super important and a big focus um uh i'm hopeful that yeah uh tom and i might co-author a bit um after that so perhaps this is tentative but perhaps we'll do some work on viatopia um together given just how broad what i've talked we talked about and it's a pretty big remit yeah um talked about in this area i think we've we have at least briefly covered most things but what the what will happen is just much deeper dives um into different areas and so liska veintraub has um yeah gone very deep on uh differential ai development diffusion and deployment um tom davidson and rose had star in particular are doing more on like dynamics of the intelligence explosion uh with rose we had a this paper on intel sat uh and then yeah in the future i can certainly imagine more depth on some of these particular issues so maybe it'll be digital rights maybe it'll be space governance um or others as well well it sounds like another thing that might be coming up is reorienting the project around writing briefs for solving all these problems to initiate with an air with an ai model once it's ready to actually begin to take on these projects absolutely so i think it's likely that um we'll have a bit of a kind of splint to see what's going to happen in the future and what's going to happen in the future i think we'll have to wait and see what's going to happen in the future so i think it's going to be given the current state of play like i i've seen some of the outputs of deep the search deeply searched but i haven't used it myself yet for example um i would also just love to spend more time yeah i would love to spend more time seeing look how much juice can you get out of these models now um can we just start managing them essentially yeah and so yeah that's definitely that's definitely on the table too yeah as of the week of recording it seems like deep research the new thing from open ai is going to be a lot more complex and i think that's going to be a lot more complex and i think that's going to be a lot more complex and i think that's going to be a lot more um i guess i only became available a couple of days ago it's extremely good at summarizing information for across like existing sources like synthesize reading them and synthesizing them somewhat weaker at having new insights based on that but i guess it's possible people will figure out ways of making it do that because i guess it hasn't really been designed for that purpose it might be that it's actually capable of looking at all these things and thinking well what stuff is missed in this literature um and i guess if it's able to take that kind of next step then it's like you're really beginning to like cook on this question and well maybe it can actually advance the advance the research but sure i think it's going to be a lot more complex and yeah i mean i saw one uh example of uh kind of mild um kind of result in economics mild as in like it wasn't like it's like fairly moderate you know small contribution but did get published um that was based primarily from 01 um and yeah i would love to have the challenge of trying to pick something within analytic philosophy that is like i really think could do the best job of where yeah the kind of more formal end of ethics is something i know more about as well and see like okay with a day or two days could i use probably by this time it'll be we'll have direct access to o3 use that to create something that would be publishable um that would be one kind of uh yeah a bit of a lubricant really yeah um is forethought hiring at the moment or fundraising uh yeah so we're definitely fundraising at the moment um uh forethought lies outside of the um kind of cause areas that good ventures are currently kind of um open to funding obviously that might change um we are not hiring right at this moment but we're definitely like on the lookout you know we do plan to stay small by default but we can certainly imagine growing um a little bit more compared to where we are and uh yeah so in particular for like you know bright promising researchers people who you know if you've listened this far that's a good that's a good sign you have that kind of patience yeah that's a good sign um but in particular people willing to tackle these sorts of topics and it is like it's a very unusual style of research um you don't get it really within academia um so we're producing these things that are um you know not designed for academic publication in particular involves having often involves yeah covering many different disciplines um involves just kind of yeah having this very kind of big picture perspective but yeah if that sounds like your cup of tea I think we need like enormous enormously more research effort happening here yeah I guess we mentioned earlier another way that people could contribute which is I guess is if you if you're in one of the leading AI companies then it would be great for you folks to have access to cutting-edge models that might be useful for doing this kind of research like maybe get early access a couple of months ahead of like what's publicly available yeah absolutely and I guess also to be informed if if the models are getting to the point where that could be useful because I suppose that could set and train a different strategy for you yep for sure yeah is there anything else you want to say about forethought before we before we wrap up no okay just about covered it I guess a final question I was going to ask earlier just like an emotional check-in on like how are we feeling about the state at the moment because I feel preparing for this interview going through these litany of issues okay I mean I think it's I think it can lead to a degree of resignation I just feel like there's so many things and it's coming at us so quickly that I'm just like what are the chances that we're going to actually make this work well um I mean I'm going to keep trying but uh I suppose on some weeks I'm optimistic this week I'm feeling just a bit like things are a little bit hopeless um do you want to make me feel any better or are you feeling optimistic or resigned this hour or this week or this month yeah so I think at the right of the moment I'm definitely definitely feeling stressed so I definitely my emotional attitude towards all of this was actually just kind of I mean this is kind of my personality but just positive like positive excited um you know I remember actually you and I had this conversation we even realized like our probabilities and credences weren't that different but like the affect that we were going to be more different extremely different it is true that as of November December last year I'm definitely feeling more stressed and that has become more of a dominant emotion relative to excitement it's still the case that just working with the models it's just amazing like it's kind of it's such a pleasure or it's such a pleasure I'm learning so much help from learning so much yeah it's already like especially for the kind of like weirdo multidisciplinary research having um the language models you know you can just be like what scientific yeah areas of science would be like benefited in space and like well actually material science because you've got zero gravity you can make perfect crystalline structures I'm like I would never it would take me weeks to figure this out um but I've got this um incredible polymath kind of on on hand at any point that's like yeah I mean it's brilliant for inquiries like that if people are not using it for stuff like that they absolutely should I guess not all jobs involve that but if you've got a question like what are some great examples of disturbing utopian literature from the 18th century or the 19th century it's still like gonna tell you yeah exactly which is really which is really incredible um you know there are definite causes for optimism so like um you know the fact that we're getting these reasoning models um and like in general kind of more oracle language models earlier great news um means we're just going to be like a lot wiser than we could be at the point of developing highly agentic AIs um now the fact that we've got um inference uh this like new paradigm of putting a lot of compute into inference great news from my perspective because it means that um the transition from subhuman to AGI and from there to super intelligence will be more gradual because to begin with you're spending it's very expensive hundreds of thousands of dollars to get the like AGI level output and then you're spending tens of thousands of dollars to get the like AGI level output and then you're spending tens of thousands of dollars to get the like AGI level output and then that cost comes down more gradually over time I mean still vertiginous but yeah still yeah still but better than it could be yeah absolutely also I mean this may be more temporary but like the fact that the models um currently are reasoning out loud using English language or not just English but human language um is I think um has the potential to be very helpful um from a safety perspective perspective as well you just yeah really wonder at what point are we not going to be able to trust what the models are writing on the scratch pad um yeah I mean I think everyone suspects that it's fine at the moment but yeah at some point yeah I think my thought is a little more that they'll just start reasoning in things that are not language oh okay I mean well definitely by the point that you're super intelligent like if you think how much um you need concept or you need words for things yeah well exactly and how limit how limited we are in um our kind of conceptual vocabulary vocabulary and how important actually throughout scientific discovery that just good notation has been so um my understanding of part of what helped Einstein come up with the general theory of relativity is just um he was really deep in this niche um uh area of mathematics and that helped him um have a very compact notation to express some of the thoughts he wanted to express so yeah I kind of imagine these AIs talking in like wingdings to each other and it's just just be like sorry we're getting aligned we're getting we're getting a little more negative again sorry yeah yeah okay so I mean so those things are positive I think the fact that we will get amazing AI assistance I mean you know I mentioned this like that is really promising that is really good is really promising and yeah I guess the fact that like one thing that has just been amazing is like and you must feel like this too just seeing the kind of payoff from many many years of kind of movement building and field building where um you know in my case that's primarily people generally getting convinced of the ideas of effective altruism and now switching because you know part of what's amazing about the set of ideas is that people update and are willing to change as new information comes in um but then also seeing the people who just you know the early pioneers talking about the importance of AI too and now it's like making you know that is something that really gives me um hope is the fact that there are now like you know still not enough but yeah many more than there could be a lot yeah you know I went to visit DC to get a sense of like the policy you know what people working in policy are up to and I was just and in particular because I was like I was wondering with my research how policy informed I should be and I came away just being like I'm not going to add value here which is like really good it's like these people who are like extremely smart extremely hard working um very well motivated just like yeah okay they'll do a much better job than I could and I think that's true in you know many different areas it's been it's been challenging for me to see I think that we were right about our AI worries all along um and I think that's true I wish that we had been less timid and less anxious about the risk of being wrong I wish that we had gone harder um but okay on that on that point we've mapped out a lot of different ways that people can contribute to increase our chances of making this go better um there's a lot of different avenues that people can pursue if you're in the audience and have been listening to this then uh seriously think about like is there some area here where you might be able to to to do something it's uh it's getting late but it's not too late to potentially make a difference fingers crossed um yeah that that's my inspiring message thanks a lot thanks so much it's been fun been fun