Silicon Valley’s Favorite Prophet Has a New Warning

From New York Times opinion, I'm Spencer Clayvin, one of a series of guest hosts for interesting times. Probably at this point everybody has heard the warnings about AI Doomsday. New reports today about AI models going rogue. There's a chance we could all die, like that is coming soon. Some companies developing AI could get too much power, and I think the world is right to be afraid of this. Personally, I'm skeptical about these predictions. It's all too easy for those warnings to morph into justifications for illiberal state control. This is why I wanted to talk to my guest today. He's known in many quarters as the philosopher of Doomsday. He was way ahead of the curve with his book Super Intelligence in warning about some of the most catastrophic scenarios that could emerge. And he was a major influence on figures like Elon Musk and other key players in the industry. More recently though, he's been thinking about the kind of deputopia that Super Intelligence could deliver. Nick Bostrom. Welcome to interesting times. Nice, Spencer. Good to meet you with you. So I wanted to start by just getting your assessment of where you think things stand. I think the technical term of art I want to invoke here is P Doom, the probability that everything goes terribly wrong. A researcher at Anthropic recently placed this at around 10% or maybe even above. Is that about where you are? You have a different assessment. At this point, what do you think is the likelihood of catastrophe? I don't think my outlook has changed fundamentally. I see it more like the world has changed. And so back when I started working on the first book, Super Intelligence, I began writing on that in 2008. At that point, the world was in a very different place. People were just dismissing this whole thing as a science fiction. Nobody saved maybe a handful of individuals were thinking about the challenges that would arise if we want to develop really powerful forms of machine intelligence. And so at that time, it seemed most useful for me to try to draw attention to the potential pitfalls on the path ahead. And so that we could use the intervening years to prepare ourselves, to do our homework, basically, like to do the research on alignment, etc. Now, of course, the discourse has changed a lot and now it's very mainstream to think that Super Intelligence is not just a science fiction like thing, it's like a real possibility. And so then it seemed less useful for me to just keep harping on the, you know, like pinging the same string, and that there was this other side of the coin, which already always was there in my mind, like the upside and the downside. And just take a glance at what would happen if things go well. Like if we take sort of the best case scenario, that also has sort of philosophically problematic aspects to it. So I'd like to get into some of the specifics of what those scenarios might be in a bit, but these numbers do get very specific. I've invoked one anthropic researcher, but I've seen a lot of people saying, you know, its p-dume is 70% or 10%. Can you walk us through the process of assigning a number to a probability like that? Is there a math behind it or how does that work? No, no, there isn't. Like the number itself is kind of comes out of a black box and different people. Those are, I mean, meant to be expressions of a subjective degree of uncertainty and confidence. It's not possible to sort of calculate the number based on some kind of, you know, set of agreed assumptions. And which is reflected in the fact that different people come up with very different numbers. But I mean, my view is that there is a, there will be real risks associated with this transition to the machine super intelligence era, including existential risks. We should take those seriously. If you want sort of a more qualitative assessment, I would say that I'm a fretful optimist. As in, you think that there's a real chance things go quite pear-shaped, but you're hoping, believe it, it's not. Yeah, I'm really excited about the upside. I'm hopeful we can get there, but I worry that we might not. Okay. Then let's stay in our anxieties for just a second and wallow in them together. Can you put some more meat on the bones of these potential doomsday scenarios? What are some of the things that could go wrong? What would that look like? Well, I think there are several challenges that we would need to meet in order to get a good outcome here. One is the alignment problem. This is the technical difficulty of developing scalable methods of AI control. Basically, ways to ensure that we can continue to steer these systems, have them do the things we intended for them to do rather than something different, even as these systems become ultimately arbitrarily capable. That's one challenge. I think a different class of challenge is what you might call the governance problem. Assuming we manage to control this technology, then there is the question of what ends will we humans put it to? Sometimes we use our powers for ill purposes, to which war, to oppress each other, maybe to concentrate power, to do all kinds of nefarious things. The second challenge is, can we at least do a land reasonably decent job by shifting the use cases predominantly towards the positive? Some of these are touching on the scenarios that back in 2014 and thereabouts you were contemplating the question of staving off or keeping people from inventing disaster technology. So I'm thinking especially if you're paper on the vulnerable world hypothesis that the idea that there exists some tech somewhere that if invented will instantly cause disaster or could instantly cause disaster. In that paper, you sort of raise a bunch of things that might be necessary to keep this from happening. So you talk about the possibility of mass surveillance, one world government, all kinds of things that would be pretty difficult and daunting and terrifying to contemplate now, but you might say maybe this will start to appear necessary. Are you still contemplating measures of that kind? That's still a possibility. I should clarify that the paper doesn't advocate for any of those things. It considers this possibility if we are unlucky and then I consider if that were the case, then what would have to be true of a civilization that invents that technology in order for it not to be devastated. And depending on the nature of that technology, if it's one that sort of incentivizes first strikes amongst major powers, then it might require some kind of global coordination. If the problem with the technology is that it makes it too easy for random individuals to bring down civilization, some sort of super weapons of mass destruction that you could like concoct in your own kitchen using ordinary materials, then it looks like you would need to do this very detailed surveillance mesh to prevent people from doing that. But the world is not guaranteed to be pegged up such that we will have an easy time. And anyway, so that's that's that paper. It just tries to create the framework for thinking what the different possible situations could be that we might find ourselves in. Okay, so let's shift our focus now and start thinking about what happens to things go right a cheerier subject, maybe you've been writing since at least 2024 your book, deep utopia is about the scenarios we can hope for the optimal setup. Can you lay that out for us? What does it look like if this goes well? Yeah, so if we develop this strong form of super intelligence, then I think what that means is that across the board, across all other technological and economic fields as well, we would have extremely rapid advance because these super intelligent minds would be doing the science and the research and the economic activity with like a digital speeds. So I think you would have a kind of rapid galloping towards, you know, perfect virtual reality or a cure for aging or space colonies or you know, all kinds of almost transfiction like technologies that don't violate the laws of nature that just really hard. So the question then becomes in such a condition of technological maturity, what would a human life look like, what would a good human life look like, what would a future path for us look like on the these in some sense ideal circumstances, like I have this concept of a solved world, which is the idea of technical maturity plus some sort of decent job on the world. decent job on the ground. governance things. Let's imagine everybody gets a slice, like everybody is free, there is, you know, then what happens is that a lot of the constraints that currently give shape to our human lives would peel away, like you could have the AIs and robots doing basically all economically productive tasks. You then have a leisure society where there is no longer any need for humans to work in order to get a paycheck. Now, I think if that was all, then I think it would obviously require some cultural transformation and adaptation, but ultimately I think we could easily imagine living in such a world. There are already various groups of people who don't work for a living that you could look at for inspiration, like their children, like young children don't work for living. They seem to have good life, you know, spend time with friends, playing games, inventing things. It's meant if you aspire to live like children, yeah. Then we have retired people. Now, there we have a confounder that many retired people have poor health and stuff, so that's, but if you imagine somebody who has enough money and perfect health, you know, they often have great lives, you know, they go around traveling spend time with grandchildren, they have hobbies, they watch movies, they golf, whatever it's like, it's not like necessarily a dystopian existence. And then you have various other people who maybe have a lot of independent wealth, they don't need to work for a living, so I think that would be change it, like the education system right now is kind of configured to take in children, right, and then output productive workers with quality grading attached. This is one of your grants in the book. Yes, I do have sympathy because right now the world does need a lot of productive workers, so in some sense we have no option, but if we no longer needed productive workers, then it would be perverse to continue to think of education as just a preparation for entering into the labor force and get the desktop at an office. Like then you maybe have an education geared more towards preparing people to live a great life, you know, cultivate the art of conversation, hobby, spirituality, physical wellness. Now you speak my language, I mean I'm a college, I teach a Greek literature, I don't have to sell me on an education for a life of leisure, but can we go into these, what happens then when you peel back that layer of the onion? Well, so then you think a little bit harder about this, and you realize that it's not just our economic efforts that would become OTOs in a sold world, but a lot of other instrumentally motivated effort also. You know, somebody maybe they want to be physically fit, and right now they have to go to the gym themselves, they can't hire an assistant to lift the weights on their behalf, right? Like for another person maybe they want to. Much is I would like to. Yeah, maybe that person enjoys decorating their house, like they want to pick out the curtains, the pillows, go through the catalogs, and then in the end they can get it in just a way that is right for them, that they really feel reflects their own preferences, right? But at technological maturity, you would have recommender systems that would know your preferences. So well, that it could just select the curtains and the cushions and the rest and do a better job than you would if you went through the trouble to do it yourself, and so you could still do it yourself, but it might seem kind of pointless. As you sort of walk through the different things that people feel there are time with, when they don't have to work, and it does seem that that would be this question of is there still a point to doing this, if there was a shortcut about and you could press that would result in the same outcome? Yeah, I mean, isn't the end point of all this you press a button and you feel infinite pleasure or you're suddenly infinitely wise, you reconfigure your environment or mind, right? I mean, I've read you on this. Yeah, yeah, you have this plasticity, like the world becomes plastic in the sense of malleable to our desires, including ourselves. So there's like, mentals, there's a lot of things people do now for the sake of getting a certain mental state, like maybe the runner wants to end of being boost at the end of it, or we engage in some activity because it gives us joy, but like in at technological maturity, you could like directly induce the end of being or the joy and so, so it does raise this more profound philosophical question of what ultimately has meaning, like what is worth doing for its own sake, not because it is practically useful to achieve something else? What exactly does it look like to get there? I guess it's maybe let's walk step by step from here in our weak, fragile human bodies that live and die to deep utopia. You open up a bunch of possibilities about how I might get from here there, including genetic modification and eugenics, right? I mean, I'm not saying that these are things you've said we should do. No, I think that's important to clarify. So there is this chapter in superintelligence that the earlier book, where I consider different possible paths that might result in superintelligence. So one is whole brain emulation, where instead of building these completely artificial synthetic mines, you would try to do something that is very closely based on how the human brain operates, like maybe detailed brain scans and then implementing neural networks. So copy pasted from this, that would be a second a route to machine intelligence, then you could imagine forms of improvements in our collective epistemology, institutions and tools that allow humans to solve much more difficult problems together. So that would be like a kind of collective superintelligence, then you could imagine maybe some smart drugs, new tropics that would improve our memory or concentration, or you could imagine some form of genetic enhancements. But I think the story of like began a long time ago with like everybody developed a better way to, you know, a cheaper piece of flint to create a better axe than, you know, better ways of planting seeds and then all kinds of inventions, you know, double entry bookkeeping all the way up to the current era. And then you know, maybe we're getting closer to the point where we can automate a lot of this process of inventing and then that could sort of speed things up radically. And that would be like the most plausible path if we do end up at technological maturity. In our lifetime, certainly it probably would go through the route of machine superintelligence. Yeah, and that was going to be my next question to you. Nick Postram, do you want this? Do you want a world of radically transformable minds in an electric universe? Before answering that question, I guess it's worth just reflecting for a second on what it would mean if we said no, even though we had the ability, it's kind of the embedded telos in so much of the modern world. In science, we're trying to figure out more and more about how the world works. In technology, we try to increase our ability to accomplish goals more efficiently. Right. The economy tries to invent new processes in ways of organizing. So we like achieve our goals with your inputs. And so if one rejected the end point, it I think would then call into question, it would backchain from that and thinking like it's kind of perverse to be working so hard to achieve a goal, such that if we achieved the goal, it would be like very bad for us. Well, there's another account of that you could give, I think, which is if you follow technology's goals unfettered or unmoderated by any other goals and aspirations, you end up in a pretty dark place, but that means maybe technology's not the highest good or there are other goals that we might actually see. I mean, technology's a means rather than a goals, but nevertheless, that means would remove more and more constraints in our ability to achieve outcomes. So removing, so having no constraints is the goal, right? And then there would be. In some sense, it's implicit in technology. Now, but then it does turn out that if you actually follow through that logic, it becomes a lot less obvious how cheerful you actually are about achieving that goal. So that's I guess it's like chasing a car, right? Like it's got a. Yeah, once you catch the car, all this stuff, it hasn't really thought what would happen if we succeed. So maybe technology's goods, technology's goals aren't the only goods or goals that are choice-worthy that we should want. Well, I think that is a given that it's not determining all the goals that we ultimately want, but you can then think about what happened, given all the other goals that we might have, if the technology part runs its course, you end up in this solved world, and then the question is like how suitable is a solved world for the realization of all these other goals that we might have? And that's done is the question that D.P. Topi at the book is wrestling with. And I think I think that there is at least an initial level of repogments, like once you stare at this, at least a part of you, I think, is. I feel loyal. Yes, I will come back to the table, sure. And so the question is like how much can be rescued of what we care about, and if you push through, I think ultimately you can come out on the other side of that. and think that it could be an extremely valuable situation. But let's first think which of our values could you have in a solved world? So what is possible? And we can sort of walk through a few different things. I'm still a bit stuck on, for now, from here to there, right? As we contemplate what to do in this moment, part of what we're contemplating is, do we want to let the technology run and spool all the way to a world of total plasticity? And it feels like answering that question. I began by asking you, is that a positive outcome for you? Does that look like Utopia to be able to change your environment at a whim? And I don't know yet what your answer is. I mean, we could decide to retain various hamster wheels for ourselves to run in. You know, you could imagine normally maybe if somebody creates an engine and it pollutes less and requires less fuel and is cheaper to build. We celebrate them. I mean, you could imagine running that in reverse and sort of applauding people for making our systems less efficient. Like if some government department can waste more money, you could say that's good because it brings us further from this telos of maximum efficiency. But I think that's another counterintuitive stance to take. So, given that we have a raid before us, these many glaring possibilities. And we've said that what technology tends toward is not necessarily good or the only good. I'd like to know a little bit more about what goods you are aiming at. What are the other things that we should pursue or try to preserve as these developments unfold? Yeah, so I think it might be useful to start with the simplest and then sort of build up to the more subtle or difficult or deeper value. So like if we start at a very sort of most obvious level, people might care about pleasure and avoiding pain and just kind of having positive, subjective experience, changing our own mental states, maybe they're like super drugs without side effects or maybe other ways of kind of rearranging and intervening in our nervous systems, such that if the utopian's wanted to actually enjoy every day, they certainly would have the means to do so. Okay. But I think most people don't feel that way. I think you don't feel that way. I think there is more that I would reach for. I do think it is one important component that is easy to dismiss when you're trying to make some clever philosophical argument or sound profound or something, pleasure. But I think actually if the future is without that, I just think it looks way bleaker than if it's something where the inhabitants of this are actually having a good time. So you don't have to think of the utopias as kind of being, you know, sprayed out on a mattress like a drug addict on a euphoric high, but they could sort of be like admiring the like all the beauty that is maybe around us already now in so many ways, which we are kind of oblivious to it or dull, like that already I think would somewhat add to the appeal of these utopian lives beyond the appeal of the sort of hedonistic pleasure-maxing chunky, but there is more. Okay. What else can I have? Well, so far we were imagining this as a very passive form of existence, right? Like you sit there, you take all of this wanderer's beauty in, like maybe that's really nice, you enjoy it, but still you might think, well, what about activity? Like don't we want to sort of do things? And so you could add that, you could have various forms of activity, exertion, effort. Now, here we have the problem we discussed earlier, that there might be a pointlessness to a lot of things, but we can have artificial purpose. So like if somebody set themselves to the goal of getting a little ball into a sequence of 18 holes using the very inconvenient method of hitting it with a club, after you have that goal, then the only way you can achieve it is by you yourself concentrating and taking swings. And so a game playing might constitute a larger portion of the lives of utopians, games are essentially where we set ourselves on arbitrary goal in order to enable the activity of pursuing it, that we might think is intrinsically valuable. So at this point, I have to say you've laid out for me this way of looking at the future of AI, which seems to suggest, you know, we've got this technology, it's coming into being, it may attain all these new capacities really quickly, like maybe overnight. And the range of options include, it's going to nuke the world all the way up to, in the best case scenario, it's going to put us into a sort of digital bath where we are uploaded minds or where endlessly customizable bodies that feel forms of pleasure, appreciation, interest, all of which are kind of detached from real stakes or requirements, except those that we impose, agree to imagine. And I have to say, none of these sound like very good outcomes to me. I mean, I don't even want somebody else doing my weightlifting for me. And I'm not sure that this is a utopia that kind of most people would look at and say, yeah, I recognize something, something good in that. I still think it's not a sales pitch, it's an analysis. Now, you might still be missing a little bit here, like one more. So we talked about artificial purpose that allows activity, so it would not be like a bath, they might go around like participating in all kinds of activities and sport and music performances and all kinds of things that are beyond even our current conception of what the rich human life could look like. Just as like our life, so maybe beyond the conception of, if you imagine our like great eight ancestors, like they might have thought, oh, if we evolve into humans, we could have so many bananas. Now, we do have a lot of bananas, but there's more to human life than bananas. There is like romantic love and literature and religion and politics and humor and all of these things that they literally couldn't imagine, because the ape brain wasn't kind of capable. I mean, it would be kind of preposterous, I think, to think that we have somehow maxed out and can now see all possible things that would be valuable. But I want to squeeze in as well the question of, is there real purposes that could survive into a solved world like things we would have reason to do not because we set ourselves some arbitrary goal just for the sake of doing stuff, but that they're kind of more independent of us. And I think that could be some, for example, suppose you happen to have the value of honoring and valuing certain traditions that you are committed to and feel part of maybe and you want to carry forward, it wouldn't count if you build a robot that sort of sang hymns or enacted the ceremonies or Christmas or whatever, right? To the degree that we care about what other people think of us, if what those other people you care about is that something be done by you, by your own effort, that would give you a reason to do those things in so much as you care about what those other people think. It's not arbitrary, right? It's the same thing, like even a robot might do something nicer, but if your child draws you a crayon picture for your birthday, it might be that you could download a nicer picture from the internet in some objective sense, but it still means a lot more to you precisely because the child chose to do it and put effort into it. And so that gives the child a reason to put this effort into it. And beyond that, all kinds of spiritual pursuits and relationship to values. So I think, so I have this metaphor, this might appeal to you slightly more than some of the earlier discussions. We'll see, that right now there is sort of obvious value. values that are stark and urgent, people starving, people suffering from disease, poverty, injustice, these kind of have an immediate claim. But beyond that, that might also be subtler values, that it's like the constellations in the night sky. They are actually there during the day as well. It's just they are blotted out by the intensity of the sun. And these are the screaming moral imperatives and urgent practical necessities. But if you might in a world where that went away, then I think it would make sense to let our pupils dilate and we might then see this much richer canopy of values, reasons for doing things. That are maybe more aesthetic, more spiritual, more like calling forth a greater sensibility from ourselves to detect things, doing things because actually it would kind of be beautiful to do it this way rather than this other way, even though the other way is in some sense simpler and easier and more direct. Okay. This leads me to a question that I had. So you say these aren't sales pitches, their predictions or analysis? No, they're not predictions either. It's an exploration of what happens if we end up in this situation where these philosophical questions don't become very obvious. Think of it as a philosophical particle accelerator. So you can sort of create extreme conditions in an experimental setting to see the basic principle. And so similarly for our values, I think we can gain insight into what ultimately is value and what we just associate with value by thinking of this extreme condition of a solved world where a lot of the sort of practical confounders are removed by stipulation. So it's not a prediction that that's where we will end, but it's a philosophical setup that can sort of force you to confront at a deeper level the question of what you ultimately value. But values is precisely what I am not yet clear about from you. I've heard a lot of different possibilities and these thought experiments that we can put into the particle accelerator and the variety of things that we might value as sort of futuristic mega-mines. And yet, I don't know which of those things, from our current standpoint, we should be aiming toward in which of those things we should be discarding. I mean, we're contemplating, for instance, if we are going to be altering ourselves in this direction, which parts of our humanity are kind of expendable and which parts we should maximize or optimize. In this context, I think it would be helpful, you know, if we can talk about your philosophical outlook a little bit. I know you're associated, you were pretty influential in the effective altruist movement, and that's the movement that seems to be defining a lot of these Doomsday scenarios that are coming out of Silicon Valley that we've talked about. Can you define what effective altruism is, what it means, and talk about how your work relates to it or doesn't align with it? I mean, first of all, I wouldn't call myself an effective altruist, I have a lot of friends in those circles, but I think they would maybe say that they try to be rational about doing as much good as possible from an impersonal point of view. So a lot of traditional philanthropy is kind of some rich person trying to show how rich and well-connected they are to other rich people, and they give to this already very wealthy symphony orchestra in their town because they throw the parties for all the other interesting rich people. Is there maybe a better way that you could do a lot more good with whatever number of dollars you're willing to give? So it started as a sort of attempt to do philanthropy more efficiently, particularly third world philanthropy with my bednets and the deworming medicines, like vitamin pills, iodine, etc. And then from there, I think broadened out to also look at other cause areas that maybe where the stakes are more abstract, but could have a disproportionate impact on the future. They have to bring on some of the leaders of effective altruism if you want their own description of what they are up to. My own role in this, I think, is that some of the concepts and ideas I had were like maybe helping to widen the scope over which cost effectiveness prioritization was done. So not just within a particular domain. This was at our shared university home in Oxford. Yeah, so it was the Future of Humanity Institute at Oxford, which I ran for many years. And then a couple of the people involved there were also pioneers in this effective altruism movement. And there has long been a great level of interest amongst many effective altruists in these bigger picture questions for humanity. What are the other major risks to human survival, waste, which could destroy her? Because if you sort of take the stance of trying to do the maximum good impersonally, then it does look like one important dimension is how our actions now affect things over time, including the longer future, not just kind of the immediate effect. So if you care a lot about the future, maybe most people will live and they think they are equally morally considerable as the current people who live. So you think that the future people outweigh or are as morally considerable as present people? I think that maybe we have more reason to place extra weight on a lot of current people. It's not as if the future generations have like monopolized almost all the efforts, like generally tend to be very shortsighted and selfish. So like I think some stretch in that direction is probably for the good, but it's possible to take it too far and then you might become a kind of fanatic. You can't answer all, like the question of life, the universe and meaning, all in one little manifesto, right, you sort of make intellectuals exactly now in this conversation and ask them to have actions, choices we have to make. So I'm more of a pluralist, ethical pluralist. So I think of moral decision making in terms of I have this metaphor of a moral parliament where you could imagine different ethical theories. Like there's like, you know, whatever, like the utilitarian party, there are like the deontologists, vertices, you know, ethics of care. There is also like a self-interest party. I care about myself and my family, particularly, and like all these different things. Like maybe I have slightly different degrees of credence in these different ones so they get to send delegates in proportional to the probability or the weight that they give them. And then in this imaginary parliament, all these delegates convene and I imagine them kind of negotiating with one another to then determine a policy that would be what I choose to do. I think that that would tend to be more robust than trying to like guess which is your favorite moral theory and then develop yourself fanatically to pursuing whatever it might, you might think that theory says you should do. What about the scenario that we're in right now? I mean, I'm thinking now again of vulnerable world hypothesis and the idea of using master valence to stave off some bad technology. At some point you have to think about if we do arrive at that juncture, are we going to accept a freedom tag on our ankle or is that in itself an unacceptable curtailment on human freedom? I mean, you've laid out kind of a way that you might have an intellectual parlor conversation among imaginary philosophical dialogue among a variety of different ethicists about that. But I want to know how you Nick Boston would make these sorts of decisions. Well, yeah, I would decide, I think it would depend a lot on the specifics of the situation. Okay. I mean, I would say that there is like a propensity for society to opt for the surveillance option even when they shouldn't like after 9/11 for example. So it was like a few thousand people who died. And yet it then triggered like a multi trillion dollar effort throwing civil liberties or like launching massive surveillance program invading countries. And trampling over human rights in some instances. And so I have a lot of sympathy with the idea that the greater risks would be that we sort of jump too readily to save more surveillance or even more global governance rather than the opposite. But that doesn't mean that you can't conceive of situations where maybe that would be the lesser of two evils. Well, it does seem as if when You're gaming out these scenarios, many, many, hundreds, thousands, what millions of years into the future. You start to weigh the concerns of the present against this enormous mass of uncountable millions of humans, post-humans, biohacked minds, digital minds, right? And suddenly our present scenario can't weigh in the balance. And so our little puny humanity has to accept your example of, of, curtailments on freedom after 9/11 is sort of a good example, right? This big bogey comes up on the other end of the future. And then suddenly in our now we're making our lives miserable. And a lot of these things that you're describing do sound like kind of making our lives miserable now because we've accepted an idea about what might come down to a pike, how, how, is it that these kinds of scenarios help us to think about what we do now rather than just paralyzing us or sending us into a panic? Well, so it's a lot there. So I wrote this paper maybe back in 2003 or thereabouts called Astronomical Waste. So it says basically that there are like a lot more future-deracious that could come into existence. And so the concerns of the present generation from that ethical point of view would seem to be extremely small and low weight compared to the combined interest of the entire future that is on the line. Now, the mistake that people that make is that they think that I think that we should give basically no consideration to the current generation and focus all our effort on ensuring that this glorious future comes about. But the paper says that that's what follows if you accept certain assumptions, these utilitarian premises, this utilitarian ethics. Now, I am not myself a utilitarian. And indeed, one argument against being a utilitarianism is precisely that it would have these kind counterintuitive implications, which the paper helped point out. I think there are also some other problems with infinite ethics, et cetera. And so my actual views are a lot more. I think less crazy, let's say, than what people like to believe. So I take this point. And I think it's a useful one that you view your role as a philosopher as opening up a variety of possible scenarios and then gaming them out according to various different ethical approaches and then people can draw their own conclusions, let's say, about whether that looks like a good ethical approach. At the same time, you're a tremendously influential figure. There are people right now contemplating real world decisions that we began with about how to approach AI development, whether to slow it down, whether to regulate it. And they are making reference to your work and I don't mind that you, I accept that you don't think that's an appropriate use of your work or it's not what you intend your work to be used for. But I come back to the fact that when I asked you what kind of utopia you want to live in and how much of our humanity you're willing to kind of edit away or augment, I didn't have a clear picture of what a good world, a good life looks like for you. And that's a problem if we're looking for guidance now about how we should proceed in these major questions. It seems like this way of thinking is actually kind of a choice paralysis rather than a helpful guy. - Yeah, so I wish I had more guidance to give. I would say I tend to think not so much in terms of an optimal state to reach but more in terms of a trajectory of developments from what we're called why I want to be a post human when I call this an early one. - Okay, but do you know what I'm doing? - Well, no, I do think, but the trajectory here is important. And so I'm thinking this is all subject to uncertainty. Maybe I would rethink this if I were a wiser, et cetera. But if you might in a path where instead of decaying and dying after a few short decades, and like just as you have started to accumulate some wisdom and experience like the brain rots and you forget everything and become senile and then it's all gone. Like you could imagine if you could stabilize the situation, you know, maybe with some longevity therapy or something, give us a little bit more time. You know, maybe alleviate some of the worst forms of suffering that they're just too desperate and shouldn't happen. And that would be less urgency. You could then, you know, maybe exhale a little bit. I have time to take it slow, to do the various things that you can do as a human. Like people who find great value in the basket weaving and others who are like enjoying sailing boats and like somebody becomes a musician or just to study all these great books so that there are so many of them that even if you are a scholar, just studying great books you still don't really have time. Because a great book, you don't want to just read it through quickly, right? You need to live with it for a while to read. - Tell me about it. - So take the urgency off there. Now then I could imagine if you had lived like that for some period of time, I don't know whether that is like, you know, 150 years or 500 years or 2000, we've never lived that long so we don't know. Then at some point, maybe it would start to feel like you have kind of done the human thing. Like you've kind of been there, done there and it's like time maybe to unlock the next level. So that may be a slight increase in your mental capacities or in your ability to feel deeply. I like some further little layer on top that you could then grow into and start to explore these more, I don't know, transhuman ways of being and new things would maybe appear in view. Like just as with the apes evolving into humans, there were new things that we can do as humans that we think are really beautiful and valuable. And then slowly over time, you know, maybe the end result, after millions of years of this, maybe eventually you do end up as some kind of strange planetary-sized post-human supermind that is kind of connected to other superminds and doing who knows what. It's hard for us from our current vantage points to evaluate that. I think we are sort of in a deep sense clueless about that. I want to react to what you've just said in the way that we were talking about reacting to your work as a possible future to contemplate and then I want to let you have the last word. So as you say, thinking about these things, provokes reactions in us that reveal our values and our ideals and what we want. And I think part of the thing that makes the Cputopia look almost as horrifying to me as the disaster scenarios we began with is that I actually, I like humanity. I think all of the values that we might even select to aim us into the future, like beauty, like justice, like love or truth. These are all features of the kinds of beings that we are just as our current limitations and pains and mortality as part of our humanity. And so when you present this scenario in front of me that we could kind of transcend that into something that we can't even imagine, which would have goals and aims and aspirations completely beyond my current conceptualizing. To sort of paraphrase Liz Lemmon, I do not want to go to there. Like down that road, I do not want to go. And I wonder if you can see why a lot of people might as you put it recoil at this kind of prospect. - Yeah, I can see that, but if you consider a young child, who eventually grows up, it is true in one sense that they grown up, they become, it is very different from the child, like they have different interests, different thoughts, the personality, their bodies, quite different. And yet we don't normally think of it as all things considered bad for a child to grow up. I think in some larger sense, we might all still be like children. And I'm hoping that we can continue to develop and carry on some of what is valuable about what we are as a child. I think hopefully we can do a better job at preserving that than the natural maturation from child to adults involves. So we might remain a child, but also then grow these additional layers on our being that would allow us to become in some sense even more of what we are in potential and sort of unfold that. And I just think it would be a hugely presumptuous, like this is vast space of possible modes of being, of ways of feeling and doing and relating. And to imagine that the little janitor's closet that we so far has explored would somehow be the best possible place in this giant cathedral. I don't think that path is one in which we would be making these choices without external contacts and considerations. I think. It's not an isolated person shooting off into space in their own space capsule, necessarily. It could be a community, a civilization, a larger entity. It may be joined with you know, could imagine if you want to be fanciful, uplifted animals that can start to talk and participate, you know, maybe some of these digital minds. And then together kind of feeling our way forward into what might be a beautiful future for us all. I wish you the best on that journey. I won't be there. Nick Monstrom, thank you so much. Thank you.