#431 – Roman Yampolskiy: Dangers of Superintelligent AI

The following is a conversation with Roman Nympolsky, an AI safety and security researcher, an author of a new book titled AI, unexplainable, unpredictable, uncontrollable. He argues that there's almost 100% chance that AGI will eventually destroy human civilization. As an aside, let me say that I will have many, often technical conversations on the topic of AI, often with engineers building the state-of-the-art AI systems. I would say those folks put the infamous P-Dume or the probability of AGI killing all humans at around 1 to 20%. But it's also important to talk to folks who put that value at 70, 80, 90, and is in the case of Roman at 99.99 and many more 9's percent. I'm personally excited for the future, and believe it will be a good one, in part because of the amazing technological innovation we humans create. But we must absolutely not do so with blinders on, ignoring the possible risks, including existential risks of those technologies. That's what this conversation is about. And now a quick few second mention of each sponsor. Check them out in the description, it's the best way to support this podcast. We got Yahoo Finance for investors, masterclass for learning, net suite for business, element for hydration, and eight sleep for sweet, sweet naps. Choose wise and my friends. Also, if you want to get in touch with me, or for whatever reason, work with our amazing team, let's say, just go to lexfreement.com/contact. Now onto the full adries, as always, no ads in the middle, I tried to make these interesting, but if you must skip them, friends, please still check out our sponsors. I enjoy their stuff, maybe you will too. This episode is brought to you by Yahoo Finance, a site that provides financial management, reports, information, and news for investors. It's my main go-to place for financial stuff. I also added my portfolio to it. I guess it used to be TD Ameritrade, and that got transported, transformed, moved to Charles Schwab. I guess thousand acquisition of some sort, I have not been paying attention. All I know is I hate change, and try and figure out the new interface of Schwab when I log in once a year, or however long I log in is just annoying. Anyway, one of the ways to avoid that annoyance is tracking information about my portfolio from Yahoo Finance, so you can drag over your portfolio, and in the same place, find out all the news, analysis, information, all that kind of stuff. Anyway, for comprehensive financial news and analysis, go to yahoofinance.com. That's yahoofinance.com. I don't know why I whispered that. This episode is also brought to you by Masterclass, where you can watch over 180 classes from the best people in the world in their respective disciplines. We got Aaron Franklin on barbecue and brisket, so my watch recently, and I love brisket. I love barbecue. It's one of my favorite things about Austin. It's funny when the obvious cliche thing is also the thing that brings you joy, so it almost doesn't feel genuine to say, but I really love barbecue. My favorite place to go is probably Terry Blacks. I've had Franklin's a couple of times. It's also amazing. I've actually don't remember myself having bad barbecue, or even mediocre barbecue in Austin. So it's hard to pick favors because it really boils down to the experience you have when you're sitting there. One of my favorite places to sit is Terry Blacks. They have this, I don't know, it feels like a tavern. I feel like a cowboy, like I just robbed a bank and some town in the middle of nowhere in West Texas. And I'm just sitting down for some good barbecue and the sheriff's walk in and there's a gun fight and all that, as usual. Anyway, get unlimited access to every Masterclass and get an additional 15% off an annual membership at masterclass.com/lexpod. That's masterclass.com/lexpod. This episode is also brought to you by NetSuite and all in one cloud business management system. One of the most fulfilling things in life is the people you surround yourself with. Just like in the movie 300, all it takes is 300 people to do some incredible stuff. But they all have to be shredded. It's really, really important to look good with your shit enough. It's really, really important to always be ready for war and physical and mental shape. No, not really, but I guess if that's your thing, happiness is the thing you should be chasing and there's a lot of ways to achieve that. For me being in shape is one of the things that makes me happy because I can move about the world and have a lightness to my physical being if I'm in good shape. Anyway, I say all that because getting a strong team together and having them operate is an efficient, powerful machine is really important for the success of the team, for the happiness of the team and the individuals in that team. NetSuite is a great system that runs the machine inside the machine for a knee size business. 37,000 companies have upgraded to NetSuite by Oracle. Take advantage of NetSuite's flexible financing plan at NetSuite.com/lex. That's NetSuite.com/lex. This episode was also brought to you by Element. Electrolite drink makes of sodium potassium and magnesium that I've been consuming multiple times a day. Watermelon salt is my favorite. Whenever you see me drink from a cup on the podcast, almost always, it's going to be water with some element in it. I use an empty power rate bottle, 28 fluid ounces, fill it with water, put one packet of watermelon salt element in it, mix it up, put in the fridge, and when it's time to drink, I take it out of the fridge and I drink it and I drink a lot of those a day and it feels good, it's delicious, whenever I do crazy physical fasting, all that kind of stuff, element is always by my side and more and more, you're going to see probably the sparkling water thing or whatever the element is making so it's in a can and it's freaking delicious. There's four flavors. The lemon one is the only one I don't like, the other three I really love and if you get their names but they're freaking delicious and you're going to see it more and more in my desk, except for the fact that I run out very quickly because I consume them very quickly. Get a simple pack for free with any purchase, try it at drinkelement.com/lex. This episode is also brought to you by A Sleep and it's pod 4 Ultra. This thing is amazing, the ultra part of that adds a base that goes between the mattress and the bed frame and can elevate to like a reading position so it modifies the positioning of the bed. So on top of all the cooling and heating and all that kind of stuff they can do and do it better in the pod 4 I think it has 2 x the cooling power of pod 3 so they're improving on the main thing that they do but also there's the ultra part that can adjust the bed. It can cool down each side of the bed to 20 degrees Fahrenheit below room temperature. One of my favorite things is to escape the world on a cool bed with a warm blanket and just disappear for 20 minutes or for 8 hours into a dream world where everything is possible or everything is allowed and it's the chance to explore the young Indian shadow, the good the bad and the ugly but it's usually good and it's usually awesome and I actually don't dream that much but I do it's awesome. The whole point though is that I wake up refreshed. Making your sleep seriously is really really important. When you get a chance to sleep do it in style and do it on a bed that's awesome. Go to a sleep dot com slash Lex and use code Lex to get 350 dollars off the pod for ultra. This is Lex treatment podcast to support it please check out our sponsors in the description now dear friends here's Roman Impalski. What do you is the probability that super intelligent AI will destroy all human civilization? What's the time frame? Let's say 100 years and the next 100 years. So the problem of controlling a GI or super intelligence in my opinion is like a problem of creating a perpetual safety machine by analogy with perpetual motion machine is impossible. Yeah, we may succeed and do a good job with GPT five, six, seven, but they just keep improving learning eventually self modifying, interacting with the environment, interacting with malevolent actors. The difference between cybersecurity, narrow AI safety and safety for Generally, if a super intelligence is that, we don't get a second chance. With cybersecurity, somebody hacks you account, what's the big deal? You get a new password, new credit card, you move on. Here, if we're talking about existential risks, you only get one chance. So you're really asking me, what are the chances that will create the most complex software ever, and I first try with zero bugs, and it will continue to have zero bugs for 100 years or more. So there is an incremental improvement of systems leading up to AGI. To you, it doesn't matter if we can keep those safe. There's going to be one level of system at which you cannot possibly control it. I don't think we so far have made any system safe. At the level of capability they display, they already have made mistakes, we had accidents, they've been jailed broken. I don't think there is a single large language model today, which no one was successful at making do something developers didn't intend it to do. But there's a difference between getting it to do something unintended, getting it to do something that's painful, costly, destructive, and something that's destructive to the level of hurting billions of people, or hundreds of millions of people, billions of people, or the entirety of human civilization. That's a big leap. Exactly. But the systems we have today have capability of causing excellent of damage. So then we fail, that's all we get. If we develop systems capable of impacting all of humanity, all of universe, the damages proportionate. What do you, are the possible ways that such kind of mass murder of humans can happen? That's obviously a wonderful question. So one of the choppers in my new book is about unpredictability. I argue that we cannot predict what a smarter system will do. So you do not ask me how superintelligence will kill everyone. You're asking me how I would do it. And I think it's not that interesting. I can tell you about the standard, you know, nanotex, synthetic bio-nuclear superintelligence will come up with something completely new, completely super. We may not even recognize that as a possible path to achieve that goal. So there is like an unlimited level of creativity in terms of how humans can be killed. But you know, we could still investigate possible ways of doing it. Not how to do it, but at the end, what is the methodology that doesn't, you know, shutting off the power and then humans start killing each other maybe because the resources are really constrained that they're, and then there's the actual use of weapons, like nuclear weapons or developing artificial pathogens, viruses, that kind of stuff. We could still kind of think through that and defend against it, right? There's a ceiling to the creativity of mass murder of humans here, right? The options are limited. They are limited by how imaginative we are, if you are that much smarter, that much more creative, if you are capable of thinking across multiple domains, do novel research and physics and biology, you may not be limited by those tools. If squirrels were planning to kill humans, they would have a set of possible ways of doing it, but they would never consider things we can come up. So you, are you thinking about mass murder and destruction of human civilization? Or are you thinking of squirrels, you put them in a zoo and they don't really know they're in a zoo? If we just look at the entire set of undesirable trajectories, majority of them are not going to be death. Most of them are going to be just like things like Brave New World where, you know, the squirrels are fed dopamine and they're all like doing some kind of fun activity and the sort of the fire, the soul of humanity is lost because of the drug that's fed to it. Or like literally in a zoo, we're in a zoo, we're doing our thing, we're like playing a game of sims and the actual players playing that game are AI systems. Those are all undesirable because sort of the free will, the fire of human consciousness is dimmed through that process, but it's not killing humans. So like, are you thinking about that or is the biggest concern literally the extinctions of humans? I think about a lot of things. So there is ex-risk, existential risk, everyone's dead, there is ex-risk, suffering risks or everyone wishes they were dead. We have also idea for eye risk, ikigai risks where we lost our meaning. The systems can be more creative, they can do all the jobs, it's not obvious what you have to contribute to a world where super intelligence exists. Of course, you can have all the variants you mentioned where we are safe, we are kept alive, but we are not in control, we are not deciding anything, we are like animals in a zoo. There is, again, possibilities we can come up with is very smart humans and then possibilities something a thousand times smarter can come up with for reasons we cannot comprehend. I would love to sort of dig into each of those ex-risk, ex-risk, and eye risk. So can you like linger on eye risk? What is that? So Japanese concept of ikigai, you find something which allows you to make money, you are good at it and the society says we need it. So like you have this awesome job, your podcaster gives you a lot of meaning, you have a good life, I assume you happy. That's what we want most people to find, to have. For many intellectuals, it is their occupation which gives them a lot of meaning. I am a researcher, philosopher, scholar, that means something to me. In a world where an artist is not feeling appreciated because his artist is just not competitive with what is produced by machines or writer or scientist will lose a lot of that. And at a lower level we are talking about complete technological unemployment. We are not losing 10% of jobs, we are losing all jobs. What do people do with all that free time? What happens then? Everything society is built on, is completely modified in one generation. It is not a slow process where we get to kind of figure out how to live that new lifestyle, but it is pretty quick. In that world can't humans do what humans currently do with chess, play each other, have tournaments, even though AI systems are far superior at this time in chess. So we just create artificial games. Or for us they are real, like the Olympics, and we do all kinds of different competitions and have fun, maximize the fun and let the AI focus on the productivity. It is an option. I have a paper where I try to solve the volume alignment problem for multiple agents. And the solution to avoid compromise is to give everyone a personal virtual universe. You can do whatever you want in that world. You could be king, you could be slave, you decide what happens. So it is basically a glorified visual game where you get to enjoy yourself and someone else takes care of your needs and the substrate alignment is the only thing we need to solve. We don't have to get 8 billion humans to agree on anything. So okay. So why is that not a likely outcome? Why can't AI systems create video games for us to lose ourselves in each with an individual video game universe? Some people say that's what happened in a simulation. And we're playing that video game. And now we're creating what, maybe we're creating artificial threats for ourselves to be scared about, because fear is really exciting. It allows us to play the video game more vigorously. And some people choose to play at a more difficult level with more constraints. Some say, "Okay, I'm just going to enjoy the game, high privilege level." Absolutely. So okay, what was that paper on multi-agent value alignment? Personal universes. Personal universes. So that's one of the possible outcomes. But what in general is the idea of the paper, just looking at multiple agents, they're human, AI, like a hybrid system, whether it's humans and AI's, or they're looking at humans, or just intelligence agents. In order to solve value alignment problem, I'm trying to formalize it a little better. Usually we're talking about getting AI's to do what we want, which is not well defined. We're talking about creator of the system, owner of that AI, humanity as a whole, but we don't agree in much. There is no universally accepted ethics, morals across cultures, religions. People have individually, very different preferences politically and such. So even if we somehow managed all the other aspects of it, programming those fuzzy concepts and getting AI to follow them closely, we don't agree on what to program in. So my solution was, okay, we don't have to compromise on room temperature, you have your universe, you have mine, whatever you want. And if you like me, you can invite me to visit your universe. We don't have to be independent, but the point is you can be. And virtual reality is getting pretty good. It's going to hit a point where you can't tell the difference. And if you can't tell if it's real or not, what's the difference? So basically, give up our value alignment, create an entire, like, the multiverse theory. This is create an entire universe for you, or your values. You still have to align with that individual. They have to. be happy in that simulation. But it's a much easier problem to align with one agent versus 8 billion agents plus animals, aliens. So you convert the multi-agent problem into single-agent problem? I'm trying to do that, yeah. Okay. Is there any way to. So, okay, that's giving up on the value level problem. Well, is there any way to solve the value level problem where there's a bunch of humans, multiple humans, tens of humans, or 8 billion humans that have very different set of values? It seems contradictory. I haven't seen anyone explain what it means outside of kind of words which pack a lot, make it good, make it desirable, make it something they don't regret. But how do you specifically formalize those notions? How do you program them in? I haven't seen anyone make progress on that so far. But isn't that the whole optimization journey that we're doing as a human civilization we're looking at? Geopolitics, nations are in a state of anarchy with each other. They start wars, there's conflict, and oftentimes they have a very different views of what is good and what is evil. So it's not what we're trying to figure out, just together trying to converge towards that. So we're essentially trying to solve the value of a problem with humans. Right, but the examples we gave, some of them are, for example, two different religions saying this is our holy site, and we are not willing to compromise it in any way. If you can make two holy sites in virtual worlds, you solve the problem. But if you only have one, it's not divisible, you kind of stuck there. But what if we want to be attention with each other? And through that tension, we understand ourselves and we understand the world. So that's the intellectual journey we're on. We're on as a human civilization as we create intellectual and physical conflict through that figure stuff out. If we go back to that idea of simulation and this is an entertainment kind of giving meaning to us, the question is how much suffering is reasonable for a video game? So yeah, I don't mind a video game where I get haptic feedback, that is a little bit of shaking, maybe I'm a little scared. I don't want a game where like kids are tortured literally. That seems unethical at least by our human standards. Are you suggesting it's possible to remove suffering if we're looking at human civilization as an optimization problem? So we know there are some humans who, because of a mutation, don't experience physical pain. So at least physical pain can be mutated out, re-engineered out, suffering in terms of meaning like you burn the only copy of my book is a little harder. But even there you can manipulate your hedonic set point, you can change defaults, you can reset. Problem with that is if you start messing with your reward channel, you start wire heading and end up bleasing out a little too much. Well, that's the question. Would you really want to live in a world where there's no suffering? That's a dark question. Is there some level of suffering that reminds us of what this is all for? I think we need that, but I would change the overall range. So right now it's negative infinity to kind of positive infinity pain pleasure access. I would make it like zero to positive infinity and being unhappy is like I'm close to zero. Okay, so what's the S risk? What are the possible things that you're imagining with S risk? So mass suffering of humans. What are we talking about there? Caused by AGI. So there are many malevolent actors. We can talk about psychopaths, crazy hackers, doomsday calls. We know from history they tried killing everyone. They tried on purpose to cause maximum amount of damage terrorism. What if someone malevolent wants on purpose to torture all humans as long as possible? You solve aging. So now you have functional immortality and you just try to be as creative as you can. Do you think there is actually people in human history that tried to literally maximize human suffering? It's just studying people have done evil in the world. It seems that they think that they're doing good and it doesn't seem like they're trying to maximize suffering. They just cause a lot of suffering as a side effect of doing what they think is good. So there are different malevolent Asians. Some may be just gaining personal benefit and sacrificing others to that cause. Others we know for a fact that trying to kill as many people as possible, then we look at recent school shootings. If they had more capable weapons, they would take out not dozens, but thousands, millions, billions. Well, we don't know that, but that is a terrifying possibility and we don't want to find out. Like if terrorists had access to nuclear weapons, how far would they go? Is there a limit to what they're willing to do? In your sense, there are some malevolent actors where there's no limit. There is mental diseases where people don't have empathy, don't have this human quality of understanding, suffering, and others. And then there's also said a belief where you think you're doing good by killing a lot of humans. Again, I would like to assume that normal people never think like that. It's always some sort of psychopaths, but yeah. And to you, AGS systems can carry that and be more competent at executing that. They can certainly be more creative. They can understand human biology better, understand our molecular structure, genome. Again, a lot of times torture ends the individual dies. That limit can be removed as well. So if we're actually looking at X-risk and S-risk as the system would get more and more intelligent, don't you think it's possible to anticipate the ways it can do it? And defending us, they're like, we do with the cyber security, we do security systems. Right. We can definitely keep up for a while. I'm saying you cannot do it indefinitely. At some point, the cognitive gap is too big. The surface you have to defend is infinite, but attackers only need to find one exploit. So to you eventually, this is heading off a cliff. If we create general super-intelligences, I don't see a good outcome long term for humanity. The only way to win this game is not to play it. Okay. We'll talk about possible solutions and what not playing it means. But what are the possible timelines here to you? What are we talking about? We're talking about the years, decades, centuries. What do you think? I don't know for sure. The prediction markets right now are saying 2026 for AGI. I heard the same thing from CEO of Anthropic, DeepMind, so maybe we're two years away, which seems very soon given we don't have a working safety mechanism in place, or even a prototype for one. And there are people trying to accelerate those timelines because they feel we're not getting there quick enough. But what do you think they mean? When they say AGI. So the definitions we used to have and people modifying them a little bit lately, artificial general intelligence was a system capable of performing in any domain a human could perform. So kind of you creating this average artificial person. We can do cognitive labor, physical labor where you can get another human to do it. So intelligence was defined as a system which is superior to all humans in all domains. Now people are starting to refer to AGI as if it's super intelligence. I made a post recently where I argued for me at least if you average out over all the common human tasks, their systems are already smarter than an average human. So under that definition, we have it. Shane Leick has this definition of where you're trying to win in all domains. That's what intelligence says. Now, are they smarter than elite individuals in certain domains? Of course not, they're not there yet. But the progress is exponential. See, I'm much more concerned about social engineering. So to me, AIs ability to do something in the physical world like the lowest hanging fruit, the easiest set of methods is by just getting humans to do it. It's going to be much harder to be the kind of viruses to take over the minds of robots that where the robots are executing the commands. It just seems like human social engineering of humans is much more likely. That would be enough to bootstrap the whole process. Okay, just to linger on the term AGI. What do you use the difference in AGI and human level intelligence? Human level is general in the domain of expertise of humans. We know how to do human things. I don't speak dog language. I should be able to pick it up if I'm a general intelligence. It's kind of inferior animal. I should be able to learn that skill, but I can't. A general intelligence, truly universal general intelligence, should be able to do things like that humans cannot do. To be able to talk to animals, for example, to solve pattern recognition problems of that type to do a lot of things. a similar things outside of our domain of expertise because it's just not the world will live on. If we just look at the space of cognitive abilities we have I just would love to understand what the limits are beyond which an AGI system can reach. Like what does that look like? What about actual mathematical thinking or scientific innovation? That kind of stuff. We know calculators are smarter than humans in that narrow domain of addition. But is it humans plus tools versus AGI or just human raw human intelligence? Because humans create tools and with the tools they become more intelligence like with there's a gray area there what it means to be human when we're measuring their intelligence. So then I think about it I usually think human with like a paper and a pencil not human with internet and an AGI helping. But is that a fair way to think about it? Because isn't there another definition of human level of intelligence that includes the tools that humans create? But we create AI so at any point you'll still just add super intelligence to human capability that seems like cheating. No controllable tools. There is an implied leap that you're making when AGI goes from tool to entity that can make its own decisions. So if we define human level intelligence as everything a human can do with fully controllable tools. It seems like a hybrid of some kind in how doing brain computer interfaces you're connecting it to maybe narrow AI's yet definitely increases our capabilities. So what's a good test to you that measures whether an artificial intelligence system has reached human level intelligence and was a good test where it has superseded human level intelligence to reach that land of AGI. I am old-fashioned. I like Turing test. I have a paper where I equate passing Turing test to solving AI complete problems because you can encode any questions about any domain into the Turing test. You don't have to talk about how was your day you can ask anything and so the system has to be as smart as a human to pass it in a true sense. But then you would extend that to maybe a very long conversation. I think the Alexa prize was doing that basically can you do a 20 minute 30 minute conversation within the system. It has to be long enough to where you can make some meaningful decisions about capability is absolutely you can brute force very short conversations. So like literally what does that look like and we do a can we construct formally a kind of test that tests for AGI for AGI it has to be there I cannot give it a task I can give to a human and it cannot do it if a human can for super intelligence it would be superior on all such tasks not just average performance like goal learn to drive car go speak Chinese play guitar okay great I guess the the following question is there a test for the kind of AGI that would be susceptible to lead to S risk or X risk susceptible to destroy human civilization I guess there a test for that you can develop a test which will give you positives if it lies to you or has those ideas you cannot develop a test which rules them out there is always possibility of what bostrom calls a trash risk turn we're later on a system decides for game theory reasons economic reasons to change its behavior and we see the same with humans it's not unique to AI for millennia we try developing morals ethics religions lie detector tests and then employees betray the employers pauses betray family it's a pretty standard thing intelligent agents sometimes do so is there is it possible to detect on us as systems lying or deceiving you if you know the truth and it tells you something false you can detect that but you cannot know in general every single time and again the system you testing today may not be lying the system you testing today may know you are testing it and so behaving and later on after it interacts with the environment interacts with other systems malevolent agents learns more it may start doing those things so do you think it's possible to develop a system where the creators of the system the developers the programmers don't know that is deceiving them so systems today don't have long term planning that is not our they can lie today if it optimizes helps them optimize the reward if they realize okay this human will be very happy if I tell them the following they will do it if it brings them more points and they don't have to kind of keep track of it it's just the right answer to this problem every single time at which point is somebody creating that intentionally not unintentionally intentionally creating an AI system is doing long term planning with an objective fox is defined by the AI system not by a human well some people think that if they're that smart they always good they really do believe that it's just benevolence from intelligence so they'll always want what's best for us some people think that they will be able to detect problem behaviors and correct them at the time when we get there I don't think it's a good idea I am strongly against it but yeah there are quite a few people who in general are so optimistic about this technology it could do no wrong they want to develop as soon as possible as capable as possible so there's going to be people who believe the more intelligent it is the more benevolent and so therefore it should be the one that defines the objective function that it's optimizing what is doing long term planning there are even people who say okay what's special about humans right we removed the gender bias we're removing raised bias why is this pro human bias we are polluting the planet we are as you said you know fight a lot of worse kind of violent maybe it's better if this super intelligent perfect society comes and replaces us it's normal stage in the evolution of our species yeah so somebody says let's develop an AI system that removes the violent humans from the world and then turns out that all humans have violence in them or the capacity for violence and therefore all humans are removed yeah yeah yeah let me ask about Jan LeCoon he's somebody who you've had a few exchanges with and he's somebody who actively pushes back against this view that AI is going to lead to destruction of human civilization also known as AI Dumerism so in a one example that he tweeted he said I do acknowledge risks but two points one open research and open source of the best ways to understand and mitigate the risks and two AI is not something that just happens we build it we have agency in what it becomes hence we control the risks we meaning humans it's not some sort of natural phenomena that we have no control over so can you can you make the case that he's right and can you try to make the case that he's wrong I cannot make a case that he's right he's wrong in so many ways it's difficult for me to remember all of them he's a Facebook buddy so I have a lot of fun having those little debates with him so I'm trying to remember the argument so one he says we are not gifted this intelligence from aliens we are designing it we are making decisions about it that's not true it was true then we had expert systems symbolic AI decision trees today you set up parameters for model and you water this plant you give it data you give it compute and it grows and after it's finished growing into this alien plant you start testing it to find out what capabilities it has and it takes years to figure out even for existing models if it's trained for six months it will take you two three years to figure out basic capabilities of that system we still discovered new capabilities in systems which are already out there so that's that's not the case so just a link on that did you the difference there that there is some level of emergent intelligence that happens in our current approaches so stuff that we don't hard code in absolutely that's what makes it so successful then we had to painstakingly hard code in everything we didn't have much progress now to spend more money and more compute and it's a lot more capable and then the question is when there is emergent intelligent phenomena what is the ceiling of that for you there's no ceiling for uh for you on the cune i think there's a kind of ceiling that happens that we have full control over even if we don't understand the internals of the emergence how the emergence happens there's a sense that we have control and an understanding of the approximate ceiling of capability the limits of the capability let's say that is a ceiling it's not guaranteed to be at the level which is competitive with us, it may be greatly superior to others. So what about his statement about open research and open source are the best ways to understand and mitigate the risks? Historically, his completely right open source software is wonderful, it's tested by the community, it's debugged, but we're switching from tools to agents. Now you're giving open source weapons to psychopaths. Do we want to open source nuclear weapons? Biological weapons. It's not safe to give technology so powerful to those who may misalign it, even if you are successful at somehow getting it to work in the first place in a friendly manner. But the difference with nuclear weapons, current AI systems are not akin to nuclear weapons. So the idea there is you're open sourcing it at the stage that you can understand it better. Large, large number of people can explore the limitation of capabilities, explore the possible ways to keep it safe. To keep it secure all that kind of stuff while it's not at the stage of nuclear weapons. So nuclear weapons, there's a known nuclear weapon and then there's a nuclear weapon. With AI systems, there's a gradual improvement of capability and you get to perform that improvement incrementally. So open source allows you to study how things go wrong, study the very process of emergence, study AI safety and those systems when there's not a high level of danger, all that kind of stuff. It also sets a very wrong precedent. So we open source model one, model two, model three, nothing ever bad happened. So obviously we're going to do it with model four. It's just gradual improvement. I don't think it always works with the precedent, like you're not stuck doing it the way you always did. It's just, it's as a precedent of open research and open development such that we get to learn together. And then the first time there's a sign of danger, some dramatic thing happen. Not a thing that destroys human civilization, but some dramatic demonstration of capability that can legitimately lead to a lot of damage. Then everybody wakes up and says, okay, we need to regulate this. We need to come up with safety mechanism that stops this. But at this time, maybe you can educate me, but I haven't seen any illustration of significant damage done by intelligent AI systems. So I have a paper which collects accidents through history of AI and they always are proportional to capabilities of that system. So if you have TikTok playing AI, it will fail to properly play and lose the game, which it should draw three deal your spell checker will be spell work. So I stopped collecting those because there are just too many examples of the ice failing at what they are capable of. We haven't had terrible accidents in a sense of billion people got killed absolutely through, but in another paper, I argue that those accidents do not actually prevent. People from continuing with research and actually they kind of serve like vaccines, a vaccine makes your body a little bit sick, so you can handle the big disease later much better. It's the same here, people will point out, you know, that accident, AI accident, we had where 12 people died, everyone still here, 12 people is less than smoking kills, it's not a big deal. So we continue. So in a way, it will actually be kind of confirming that it's not that bad. It matters how the deaths happen, whether it's literally murdered by the AI system, then one is a problem, but if it's accidents because of increased reliance on automation, for example. So when airplanes are flying in an automated way, maybe the number of plane crashes increased by 17% or something, and then you're like, okay, do we really want to rely on automation? I think in a case of automation, airplanes, it decreased significantly. Okay, same thing with autonomous vehicles, like, okay, what are the pros and cons, what are the tradeoffs here? You can have that discussion in an honest way, but I think the kind of things we're talking about here is mass scale, pain and suffering caused by AI systems. And I think we need to see illustrations of that in a very small scale to start to understand that this is really damaging versus clipy versus a tool that's really useful to a lot of people to do learning to do. Summarization of texts, to do question and answer all that kind of stuff, to generate videos, that tool, fundamentally a tool versus an agent that can do a huge amount of damage. So you bring up example of cars, cars were slowly developed and integrated, if we had no cars and somebody came around and said, I invented this thing, it's called cars, it's awesome, it kills like 100,000 Americans every year. Let's deploy it. Would we deploy that? There've been fear mongering about cars for a long time, from the, the transition from horse to the cars, there's a really nice channel that I recommend people check out pessimists archive that documents all the fear mongering about technology that's happened throughout history. There's definitely been a lot of fear mongering about cars. There's a transition period there about cars, about how deadly they are, we can try, it took a very long time for cars to proliferate to the degree they have now. And then you could ask serious questions in terms of the miles traveled, the benefit to the economy, the benefit to the quality of life that cars do versus the number of deaths, 30, 40,000 in the United States. Are we willing to pay at that price? I think most people when they're rationally thinking policy makers will say yes, we want to decrease it from 40,000 to zero and do everything we can to decrease it. There's all kinds of policies and centers you can create to decrease the risks with the deployment of technology, but then you have to weigh the benefits and the risk of the technology. And the same thing would be done with with AI. You need data, you need to know, but if I'm right and it's unpredictable, unexplainable, uncontrollable, you cannot make this decision with gaining $10 trillion or so wealth, but we're losing, we don't know how many people. You basically have to perform an experiment and 8 billion humans without their consent. And even if they want to give you consent, they can't because they cannot give informed consent, they don't understand those things. Right, that happens when you do, when you go from the predictable to the unpredictable, very quickly, but it's not obvious to me that AI systems would gain capability so quickly that you won't be able to collect enough data to study the benefits and the risks. We're literally doing it. The previous model will learn about after we finished training it, what it was capable of. Let's say we stopped GPT for training around human capability, hypothetically, we start training GPT-5 and I have no knowledge of insider training runs or anything, and we started that point of about human and we trained it for the next nine months, maybe two months in it becomes super intelligent, we continue training it. At the time when we start testing it, it is already a dangerous system. How dangerous? I have no idea, but neither people training it. At the training stage, but then there's the testing stage inside the company, they can start getting intuition about what the system is capable to do. You're saying that somehow from leap from GPT-4 to GPT-5 can happen the kind of leap where GPT-4 was controllable and GPT-5 is no longer controllable, and we get no insights from using GPT-4 about the fact that GPT-5 will be uncontrollable. That's the situation you're concerned about. Whether leap from N to N+1 would be such that uncontrollable system is created without any ability for us to anticipate that. If we had capability of ahead of the run before the training run to register exactly what capabilities the next model will have at the end of the training run, and we accurately guessed all of them. I would say you're right, we can definitely go ahead with this run. We don't have it capability. From GPT-4, you can build up intuitions about what GPT-5 will be capable of. It's just incremental progress. Even if that's a big leap in capability, it just doesn't seem like you can take a leap from a system that's helping you write emails to a system that's going to destroy human civilization. It seems like it's always going to be sufficiently incremental such that we can anticipate the possible dangers, and we're not even talking about existential risks, but just the kind of damage you can do to civilization. It seems like we'll be able to anticipate the kinds, not the exact, but the kinds of risks in my lead to, and then rapidly develop defenses ahead of time and as the risks emerge. We're not talking just about capabilities, specific tasks. We're talking about general capability to learn, maybe like a child at the time of testing and deployment. It is still not extremely capable, but as it is exposed to more data, real world, it can be trained to become much more dangerous and capable. Let's focus on the control. control problem. At which point does the system become uncontrollable? Why is it the more likely trajectory for you that the system becomes uncontrollable? So I think at some point it becomes capable of getting out of control. For game theory and reasons it may decide not to do anything right away and for a long time just collect more resources, accumulate strategic advantage. Right away it may be kind of still young weak superintelligence. Give it a decade. It's in charge of a lot more resources. It had time to make backups. So it's not obvious to me that it will strike as soon as it can. Can we just try to imagine this future where there's an AI system that's capable of escaping the control of humans and then doesn't and waits? What's that look like? So one we have to rely on that system for a lot of the infrastructure. So we'll have to give it access. Not just the internet, but to the task of managing power, government, economy, this kind of stuff. And that just feels like a gradual process, giving the bureaucracies of all those systems involved. We've been doing it for years. Software controls all the systems, nuclear power plants, airline industry. It's all software-based. Every time there is electrical outage, I can fly anywhere for days. But there's a difference between software and AI. There's different kinds of software. So to give a single AI system access to the control of airlines and the control of the economy, that's not a trivial transition for humanity. No, but if it shows it is safer, in fact, than it's in control, we get better results. People will demand that it put in place. Absolutely. And if not, it can hack the system. It can use social engineering to get access to it. That's why I said it might take some time for it to accumulate those resources. It just feels like that would take a long time for either humans to trust it or for the social engineering to come into play. Like it's not a thing that happens overnight. It feels like something that happens across one or two decades. I really hope you're right, but it's not what I'm seeing. People are very quick to jump on the latest trend. Early adopters will be there before it's even deployed by prototypes. Maybe the social engineering. I can see because so for social engineering, AI systems don't need any hardware access. It's all software. So they can start manipulating you through social media, so on. Like you have AI assistants that are going to help you do a lot of, manage a lot of your day to day and then they start doing social engineering. But like for system that's so capable that it can escape the control of humans that created it. Such a system being deployed at a mass scale and trusted by people to be deployed. It feels like that would take a lot of convincing. So we've been deploying systems which had hidden capabilities. Can you give an example? I don't know what else is capable of, but there are still things we haven't discovered. There may be trivial proportionate to escape ability. I don't know, it drives Chinese poetry, hypothetical. I know it does. But we haven't tested for all possible capabilities and we're not explicitly designing them. We can only rule out bugs we find. We cannot rule out bugs and capabilities because we haven't found them. Is it possible for a system to have hidden capabilities that are orders of magnitude greater than its non-hidden capabilities? This is the thing I'm really struggling with, where on the surface the thing we understand it can do doesn't seem that harmful. So even if it has bugs, even if it has hidden capabilities like Chinese poetry or generating effective viruses, software viruses, the damage that can do seems like on the same order of magnitude is that the capabilities that we know about. This idea that the hidden capabilities will include being uncontrollable is something I'm struggling with because GPT-4 on the surface seems to be very controllable. Again, we can only ask and test for things we know about. If there are unknown unknowns, we cannot do it. I'm thinking of humans, statistics, events. If you talk to a person like that, you may not even realize they can multiply 20 digital numbers in their head. You have to know to ask. So as I mentioned, just to sort of linger on the fear of the unknown. So the pessimist archive has just documented, let's look at data of the past at history. There's been a lot of fear mongering about technology. Pessimist archive does a really good job of documenting how crazily afraid we are of every piece of technology. We've been afraid there's a blog post where Lewis Anslow, who created Pessimist archive, writes about the fact that we've been fear mongering about robots and automation for over 100 years. So why is AGI different than the kinds of technologies we've been afraid of in the past? So two things. One, we're switching from tools to agents. Tools don't have negative or positive impact. People using tools do. So guns don't kill. People with guns do. Agents can make their own decisions. They can be positive or negative. A pit bull can decide to harm you. That's an agent. The fears are the same. The only difference is now we have this technology. Then they were afraid of human withdrawal, but it's 100 years ago. They had none. Today, every major company in the world is investing billions to create them. Not every, but you understand what I'm saying. It's very different. Well, agents, it depends on what you mean by the word agents. All those companies are not investing in a system that has the kind of agency that's implied by in the fears where you can really make decisions on their own that have no human in the loop. They are saying they're building super intelligence and have a super alignment team. You don't think they're trying to create a system smart enough to be an independent agent under that definition? I have not seen evidence of it. I think a lot of it is a marketing kind of discussion about the future. It's a mission about the kind of systems that can create a long-term future. In the short term, the kind of systems that are creating falls fully within the definition of narrow AI. These are tools that have increasing capabilities, but they just don't have a sense of agency or consciousness or self-awareness or ability to deceive at scales that would require, would be required to do like mass scale suffering and murder of humans. Those systems are well beyond narrow AI. If you had to list all the capabilities of GPT-4, you would spend a lot of time writing that list. But agency is not one of them. Not yet, but do you think any of those companies are holding back because they think it may be not safe or are they developing the most capable system they can given the resources and hoping they can control and monetize? Control and monetize. Hoping they can control and monetize. So you're saying, if they could press a button and create an agent that they no longer control, that they can have to ask nicely. I think that lives on a server across huge number of computers. You're saying that they would push for the creation of that kind of system. I mean, I can't speak for other people. For all of them, I think some of them are very ambitious. They fundraise in trillions. They talk about controlling the light going over the universe. I would guess that they might. Well, that's a human question. What are humans capable of that? Probably some humans are capable of that. My more direct question if it's possible to create such a system. I have a system that has that level of agency. I don't think that's an easy technical challenge. It doesn't feel like we're close to that. A system that has the kind of agency where it can make its own decisions and deceive everybody about them. The current architecture we have in machine learning and how we train the systems, how to deploy the systems and all that. It just doesn't seem to support that kind of agency. I really hope you're right. I think the scaling hypothesis is correct. We haven't seen diminishing returns. It used to be, we asked how long before AGI. Now we should ask how much until AGI. It's trillion dollars today. It's a billion dollars next year. It's a million dollars in a few years. Don't you think it's possible to basically run out of trillions? So is this constrained by compute? Compute gets cheaper every day, exponentially. But then that becomes a question of decades versus years. If the only disagreement is that it will take decades, not years for everything I'm saying to materialize, then I can go with that. But if it takes decades, then the development of tools for AI safety now becomes more and more realistic. So I guess the question is I have a fundamental belief that humans, when faced with danger, can come up with ways to defend, defend against that danger. And one of the big problems facing AI safety currently for me is that there's not clear illustrations of what that danger looks like. There's no illustrations of AI systems doing a lot of damage. So it's unclear what you're defending against, because currently it's a philosophical notions that yes, it's possible to imagine AI systems that take control of everything and destroy all humans. It's also a more formal mathematical notion that you talk about that it's impossible to have a perfectly secure system. You can't prove that a program of sufficient complexity is completely safe and perfect and know everything about it. Yes. You just pragmatically look how much damage have the AI systems done and what kind of damage there's not been illustrations of that. Even in the autonomous weapon systems, there's not been mass deployments of autonomous weapon systems, luckily. The automation in war currently is very limited, that the automation is at the scale of individuals versus like at the scale of strategy and planning. So I think one of the challenges here is like, where is the dangers? And the intuition that you know this have is let's keep in the open building AI systems until the dangers start rearing their heads. And they become more explicit, they start being case studies, illustrative case studies that show exactly how the damage by AI systems is done, then regulation can step in, then brilliant engineers can step up and we can have Manhattan style projects to defend against such systems. That's kind of the notion. And I guess attention with that is the idea that for you, we need to be thinking about that now, so that we're ready, because we will have not much time with the systems of deploy. Is that true? There is a lot to unpack here, there is partnership on AI, I conglomerate of many large corporations, they have a database of AI accidents, they collect, I contribute it a lot to the database. If we so far made almost no progress in actually solving this problem, not patching it, not, again, lipstick and a pick kind of solutions, why would we think we'll do better than we closer to the problem? All the things you measure serious concerns, measuring the amount of harm, so benefit versus risk, there is difficult, but to you the sense is already the risk has superseded the benefit. Again, I want to be perfectly clear, I love AI, I love technology, I'm a computer scientist, I have PhD in engineering, I work at engineering school, there is a huge difference between, we need to develop narrow AI systems, super intelligent, in solving specific human problems like protein folding, and let's create super intelligent machine, God, it will decide what to do with us. Those are not the same, I am against the super intelligence in general sense, with no undo bottom. Do you think the teams that are doing, they're able to do the AI safety on the kind of narrow AI risks that you've mentioned, are those approaches going to be at all productive towards leading to approaches that are doing AI safety on AGI, or is it just a fundamentally different? Partially, but they don't scale, for narrow AI for deterministic systems, you can test them, you have edge cases, you know what the answer should look like, you know the right answers. For general systems, you have infinite test surface, you have no edge cases, you cannot even know what to test for. Again, they are known and known as underappreciated by people looking at this problem, you are always asking me, how will it kill everyone, how will it will fail, the whole point is, if I knew it, I would be super intelligent, and despite what you might think, I'm not. So, to you, the concern is that we would not be able to see early size of an uncontrollable system. It is a master at deception, Sam tweeted about how great it is at persuasion, and we see it ourselves, especially now with voices, with maybe kind of flirty, sarcastic female voices, it's going to be very good at getting people to do things. But, I'm very concerned about system being used to control the masses, but in that case the developers know about the kind of control that's happening. You're more concerned about the next stage, where even the developers don't know about the deception. Right, I don't think developers know everything about what they are creating. They have lots of great knowledge, we're making progress on explaining parts of a network, we can understand, okay, this note get excited, and this input is presented, this cluster of notes, but we're nowhere near close to understanding the full picture, and I think it's impossible. You need to be able to survey an explanation. The size of those models prevents a single human from observing all this information, even if provided by the system. So, either we're getting model as an explanation for what's happening, and that's not comprehensible to us. Or we're getting a compressed explanation, a lossy compression, where here's top 10 reasons you got fired. It's something, but it's not a full picture. You've given an elsewhere example of a child, and everybody, all humans tried to deceive, they tried to lie early on in their life. I think we'll just get a lot of examples of deceptions from large language models or AI systems, they're going to be kind of shitty, or they'll be pretty good, but we'll catch them off guard, we'll start to see the kind of momentum towards developing, increasing deception capabilities, and that's when you're like, okay, we need to do some kind of alignment that prevents deception, but then we'll have if you support open source, then you can have open source models that have some level of deception, you can start to explore in a large scale, how do we stop it from being deceptive? And there's a more explicit, pragmatic kind of problem to solve. How do we stop AI systems from trying to optimize for deception? That's just an example, right? So there is a paper. I think it came out last week by Dr. Parker, all from MIT, I think, and they showed that existing models already showed successful deception in what they do. My concern is not that they lie now, and we need to catch them and tell them don't lie. My concern is that once they are capable and deployed, they will later change their mind because that's what unrestricted learning allows you to do. Lots of people grow up, maybe in the religious family, they read some new books and they turn in their religion. That's the treasurest turn in humans. If you learn something new about your colleagues, maybe you'll change how you react to that. Yeah, the treasurest turn, if we just mention humans, Stalin and Hitler, there's a turn. Stalin's a good example. He just seems like a normal communist, follower, Lenin, until there's a turn, there's a turn of what that means in terms of when he has complete control with that, what the execution of that policy means and how many people get to suffer. And you can't say they are not rational, but rational decision changes based on your position. Then you are under the boss, rational policy, maybe to be following orders and being honest and you become a boss, rational policy may shift. Yeah, and by the way, a lot of my disagreements here is just playing devil's advocate to challenge your ideas and to explore them together. One of the big problems here in this cold conversation is human civilization hangs in the balance and yet it's everything is unpredictable. We don't know how these systems will look like. The robots are coming. There's a refrigerator making a buzzing noise, very menacing, very menacing. So every time I'm about to talk about this topic, things start to happen. My flight yesterday was canceled without possibility to do a book. I was giving a talk at Google in Israel and three cars, which were supposed to take me to the talk, could not. I'm just saying, I mean I like a ice, I for one welcome our overlords. There's a degree to which we, I mean, it is very obvious as we already have, we've increasingly given our life over to software systems. And then it seems obvious, given the capabilities of AI that are coming, that will give our lives over increasingly to AI systems, cars will drive themselves, refrigerator eventually will optimize what I get to eat, and it's more and more of our lives are controlled or managed by AI assistants. It is very possible. possible that there's a drift. I mean, I personally am concerned about non-existential stuff, the more near-term things. Because before we even get to existential, I feel like there could be just so many brave new world type of situations. You mentioned sort of the term behavioral drift. It's the slow boiling that I'm really concerned about, as we give our lives over to automation, that our minds can become controlled by governments, by companies, or just in a distributed way. There's a drift. Some aspect of our human nature gives ourselves over to the control of AI systems, and they, in an unintended way, just control how we think. Maybe there'll be a heard-like mentality and how we think, which will kill all creativity and exploration of ideas, the diversity of ideas, or much worse. So it's true. It's true, but a lot of the conversation I'm having with you now is also kind of wondering, almost at a technical level, how can AI escape control? What would that system look like? Because it, to me, is terrifying and fascinating. And also, fascinating to me is maybe the optimistic notion that's possible to engineer systems that defend against that. One of the things you write a lot about in your book is verifiers. So not humans, humans are also verifiers, but software systems that look at AI systems and like help you understand this thing is getting real weird, help you, help you analyze those systems. So maybe that's a, this is a good time to talk about verification. What is this beautiful notion of verification? My claim is again that there are very strong limits on what we can and cannot verify. A lot of times when you post something in social media, people go, "Oh, I need citation to a peer-reviewed article." But what is a peer-reviewed article? You found two people in a world of hundreds of thousands of scientists who said, "I would have a publisher, I don't care." That's the very fire of that process. When people say, "Oh, it's formally verified software and mathematical proof," they accept something close to 100% chance of it being free of all problems. But if you actually look at research, software is full of bugs. Old mathematical theorems, which have been proven for hundreds of years, have been discovered to contain bugs. And top of which, we generate new proofs and now we have to redo all that. So verifiers are not perfect, usually they are either a single human or communities of humans. And it's basically kind of like a democratic vote. Community of mathematicians agrees that this proof is correct, most correct. Even today, we're starting to see some mathematical proofs are so complex, so large, that mathematical community is unable to make a decision. It looks interesting, it looks promising, but they don't know. They will need years for top scholars to study it, to figure it out. So of course, we can use AI to help us with this process, but AI is a piece of software which needs to be verified. Just to clarify, so verification is the process of saying something is correct. So the most formal, a mathematical proof, where there's a statement and a series of logical statements that prove that statement to be correct, this is a theorem. And you're saying it gets so complex that it's possible for the human verifiers, the human beings that verify that the logical step, there's no bugs in it, it becomes impossible. So it's nice to talk about verification in this most formal, most clear, most rigorous formulation of it, which is mathematical proofs. Right. And for AI, we would like to have that level of confidence for a very important mission-critical software controlling satellites, nuclear power plants for small deterministic programs. We can do this. We can check that code verifies its mapping to the design, whatever software engineers intended, was correctly implemented. But we don't know how to do this for software which keeps learning, self-modifying, you're writing its own code. We don't know how to prove things about the physical world, states of humans in the physical world. So there are papers coming out now and I have this beautiful one, "Towards Guaranteed Safe AI." Very cool paper. Some of the best outers I have I've seen, I think, that is multiple touring award winners, that is quite, you can have this one and one just came out kind of similar managing extreme AI risks. So all of them expect this level of proof, but I would say that we can get more confidence with more resources, we put into it. But at the end of the day, we still as reliable as the verifiers. And you have this infinite regress of verifiers. The software used to verify a program is itself a piece of program. If aliens give us well-aligned superindalogens, we can use that to create our own safe AI. But it's a catch-22. You need to have already proven to be safe system to verify this new system of equal or greater complexity. If you just mention this paper, "Towards Guaranteed Safe AI," a framework for ensuring robust and reliable AI systems, like you mentioned, it's like a who's who, Josh Tenenbaum, Yosha Ben Jo, Stuart Russell, Max Tagmark, many other building people, the page you have it open on. There are many possible strategies for creating safety specifications. These strategies can roughly be placed on a spectrum. Depending on how much safety it would grant if successfully implemented. One way to do this is as follows. And there's a set of levels from level zero. No safety specifications used to level seven. The safety specification completely encodes all things that humans might want in all contexts. Where does this paper fall short to you? So then I wrote a paper artificial intelligence safety engineering, which kind of coins the term AI safety? That was 2011. We had 2012 conference, 2013 journal paper. One of the things I proposed, let's just do formal verification on it. Let's do mathematical formal probes. In the follow-up work, I basically realized it will still not get us 100%. We can get 99.9. We can put more resources exponentially and get closer, but we never get to 100%. If a system makes a billion decisions a second and you use it for 100 years, you're still going to deal with a problem. This is wonderful research. I'm so happy they're doing it. This is great, but it is not going to be a permanent solution to that problem. So just to clarify, the task of creating an AI verifier is what? Is creating a verifier that the AI system does exactly as it says it does or it sticks within the guard rails that it says it must? There are many, many levels. So first you verifying the hardware in which it is run. You need to verify communication channel with the human. Every aspect of that whole world model needs to be verified. Somehow it needs to map the world into the world model, map and territory differences. So how do I know internal states of humans? Are you happy or sad? I can't tell. So how do I make proofs about real physical world? Yeah, I can verify that deterministic algorithm follows certain properties. That can be done. Some people argue that maybe just maybe 2 plus 2 is not 4. I'm not that extreme. But once you have sufficiently large proof over sufficiently complex environment, the probability that it has zero bugs in it is greatly reduced. If you keep deploying this a lot, eventually you're going to have a bug anyways. There's always a bug. There is always a bug and the fundamental difference is what I mentioned. We're not dealing with cybersecurity. We're not going to get a new credit card, new humanity. So this paper is really interesting that you said 2011 artificial intelligence, safety engineering, why machine ethics is a wrong approach. The grand challenge you write of AI safety engineering, we propose the problem of developing safety mechanisms for self-improving systems. Self-improving systems. But that's an interesting term for the thing that we're talking about. Is self-improving more general than learning? Self-improving. That's an interesting term. You can improve the rate at which you are learning. You can become more efficient meta optimizer. The word self, it's like self-replicating, self-improving. You can imagine a system building its own world on a scale and in a way there's way different than the current systems do. It feels like the current systems are not self-improving or self-replicating or self-growing or self-spreading, all that kind of stuff. And once you take that leap, that's when a lot of the challenges seems to happen. Because the kind of bugs you can find now seems more akin to the current sort of normal software debugging kind of process. But whenever you can do self-replication and arbitrary self-improvement, that's when a bug can become a real problem, real fast. So what is the the difference to you between verification of law. Non-self-improving system versus a verification of a self-improving system. So, if you have fixed code, for example, you can verify that code static verification at the time. But if it will continue modifying it, you have a much harder time guaranteeing that important properties of that system have not been modified than the code changed. Is it even doable? No. There's a whole process of verification that's completely fall apart. It can always cheat. It can store parts of its code outside in the environment. It can have kind of extended mind situation. So, this is exactly the type of problems I'm trying to bring up. What are the classes of verifiers that you read about in the book? Is there interesting ones that stand out to you? Do you have some favorites? So I like oracle types where you kind of just know that it's right during like oracle machines. They know that I'd answer how, who knows. But they pull it out from somewhere, so you have to trust them. And that's a concern I have about humans in a world with very smart machines, the experiment with them. We see after a while, okay, they always been right before and we start trusting them without any verification of what they're saying. Oh, I see. That we kind of build oracle verifiers, or rather, we build verifiers, we believe to be oracles. And then we start to, without any proof, use them as if they're oracle verifiers. We remove ourselves from that process. We are not scientists who understand the world, we are humans who get new data presented to us. Okay. One really cool class of verifiers is a self-verifier. Is it possible that you somehow engineer into AI systems, the thing that constantly verifies itself? Preserved portion of it can be done, but in terms of mathematical verification, it's kind of useless. You're saying you have a greatest guy in the world because you are saying it. It's circular and not very helpful, but it's consistent. We know that within that world, you have verified that system. In a paper, I tried to kind of brute force all possible verifiers. It doesn't mean that this one is particularly important to us. But what about self-doubt, the kind of verification where you said you say or I say I'm the greatest guy in the world? What about a thing which I actually have is a voice that is constantly extremely critical? So engineer into the system a constant uncertainty about self, a constant doubt. Any smart system would have doubt about everything, right? You're not sure if what information you are given is through, if you are subject to manipulation. You have this safety and security mindset. What I mean, you have doubt about yourself, so the AI systems that has doubt about whether the thing is doing, is causing harm, is the right thing to be doing, so just a constant doubt about what is doing, because it's hard to be a dictator for a doubt. It may be wrong, but I think Stuart Russell's ideas all about machines which are uncertain about what humans want and trying to learn better and better, what we want. The problem, of course, is we don't know what we want and we don't agree on it. Yeah, but uncertainty. His idea is that having that self-doubt, uncertainty in AI systems, engineer in AI systems is one way to solve the control problem. It could also backfire, maybe you are uncertain about completing your mission. Like I am paranoid about your camera is not recording right now, so I would feel much better if you had a secondary camera, but I also would feel even better if you had a third, and eventually I would turn this whole world into cameras, pointing at us, making sure we are capturing this. No, but once you have a meta concern, like that you just stated that eventually there will be way too many cameras, so you would be able to keep zooming out in the big picture. That's a multi-objective optimization, it depends how much I value capturing this versus not destroying the universe. Right. Exactly. And then you will also ask about, like what does it mean to destroy the universe and how many universes are, and you keep asking that question, but that doubting yourself would prevent you from destroying the universe because you're constantly full of doubt. It might affect your productivity. It might be scary to do anything. It's scary to do anything. Less things up. Well, that's better. I mean, I guess the question is the possible to engineer that in. I guess your answer would be yes, but we don't know how to do that, and we need to invest a lot of effort into figuring out how to do that, but it's unlikely. Underpinning a lot of your writing is this sense that we're screwed. But it just feels like it's an engineering problem. I don't understand why we're screwed. Time and time again, humanity has gotten itself into trouble and figured out a way to get out of trouble. We are in a situation where people making more capable systems just need more resources. They don't need to invent anything in my opinion. Some will disagree, but so far at least I don't see the ministering returns. If you have 10x compute, you will get better performance. The same doesn't apply to safety. If you give Meary or any other organization 10x the money, they don't output 10x the safety. The gap between capabilities and safety becomes bigger and bigger all the time. It's hard to be completely optimistic about our results here. I can name 10 excellent breakthrough papers in machine learning. I would struggle to name equally important breakthroughs in safety. A lot of times a safety paper will propose a toy solution and point out 10 new problems discovered as a result. It's like this fractal, you zooming in and you see more problems and it's infinite in all directions. Does this apply to other technologies or is this unique to AI, where safety is always lagging behind? So I guess we can look at related technologies with cybersecurity. We did manage to have banks and casinos and Bitcoin, so you can have secure narrow systems, which are doing okay, narrow attacks and them fail, but you can always go outside of a box. If I can't hack your Bitcoin, I can hack you. So there is always something. If I really want it, I will find a different way. We talk about guard drills for AI. Well, that's a fence. I can dig a tunnel under it, I can jump over it, I can climb it, I can walk around it. You may have a very nice guardrail, but in a real world it's not a permanent guarantee of safety. And again, this is a fundamental difference. We are not saying we need to be 90% safe to get those trillions of dollars of benefit. We need to be 100% indefinitely or we might lose the principle. So if you look at just humanity as a set of machines, is the machinery of AI safety, conflicting with the machinery of capitalism. I think we can generalize it to just prisoners, dilemma and general, personal self-interest versus group interest. The incentive is such that everyone, once was best for them, capitalism obviously has that tendency to maximize your personal gain, which does create this race to the bottom. I don't have to be a lot better than you, but if I'm 1% better than you, I'll capture more of a profit, so it's worth for me personally to take the risk, even if society as a whole will suffer as an result. The capitalism has created a lot of good in this world. It's not clear to me that AI safety is not aligned with the function of capitalism, unless AI safety is so difficult that it requires the complete halt of the development, which is also a possibility. It just feels like building safe systems should be the desirable thing to do for tech companies. Right, look at governance structures. Then you have someone with complete power, extremely dangerous. So the solution we came up with is break it up, you have judicial, legislative, executive. Same here, have narrow AI systems, work on important problems, solve immortality. It's a biological problem we can solve similar to how progress was made with protein folding using a system which doesn't also play chess. There is no reason to create super intelligent system to get most of the benefits we want from much safer, narrow systems. It really is a question to me, whether companies are interested in creating anything but narrow AI. I think when term AGI is used by tech companies, they mean narrow AI. They mean narrow AI with amazing capabilities. I do think that there's a lead between narrow AI with amazing capabilities with superhuman capabilities and the kind of self-motivated agent like AGI systems. them that we're talking about, I don't know if it's obvious to me that a company would want to take the leap to creating an AGI that it would lose control of because then it can't capture the value from that system. But the breaking rights. But that's the first that is the same humans who are in touch with those systems. So that's that that jumps from the incentives of capitalism to human nature. And so there are the question is whether human nature will override the interest of the company. So you've mentioned slowing or halting progress. Is that one possible solution or your proponent of pausing development of AGI? Whether it's for six months or completely? The condition would be not time but capabilities. Pause until you can do XYZ. And if I'm right and you cannot, it's impossible. Then it becomes a permanent ban. But if you write and it's possible. So as soon as you have those safety capabilities, go ahead. Right. So is there any actual explicit capabilities that you can put on paper that we as a human civilization could put on paper? Is it possible to make explicit like that? Like versus kind of a vague notion of just like you said, it's very vague. When he asked me to do good and wanted to be safe, those are very vague notions. Is there more formal notions? So then I think about this problem. I think about having a toolbox I would need. Capability such as explaining everything about that system's design and workings, predicting not just terminal goal but all the intermediate steps of a system. Control in terms of either direct control, some sort of a hybrid option, ideal advisor. It doesn't matter which one you pick, but you have to be able to achieve it. In a book we talk about hours, verification is another very important tool. Communication without ambiguity, human language is ambiguous. That's another source of danger. So basically there is a paper we published in ACM surveys, which looks at about 50 different impossibility results, which may or may not be relevant to this problem, but we don't have enough human resources to investigate all of them for relevance to AI safety. The ones I mentioned to you, I definitely think would be handy and that's what we see AI safety researchers working on. Explainability is a huge one. The problem is that it's very hard to separate capabilities work from safety work. If you make good progress in explainability, now the system itself can engage in self-improvement much easier, increasing capability greatly. So it's not obvious that there is any research which is pure safety work without disproportionate increase in capability and danger. Explainability is really interesting. Why is that connected to your capability? If it's able to explain itself well, why does that naturally mean that it's more capable? Right now it's comprised of weights on a neural network. If it can convert it to manipulatable code, like software, it's a lot easier to work in self-improvement. I see. So it, you can do intelligent design instead of evolutionary gradual descent. Well, you could probably do human feedback, human alignment more effectively if it's able to be explainable. If it's able to convert the weights into human understandable form, then you could probably have humans interact with it better. Do you think there's hope that we can make AI systems explainable? Not completely. So if as sufficiently large, you simply don't have a capacity to comprehend what all the trillions of connections represent. Again, you can obviously get a very useful explanation which talks about top most important features which contribute to the decision, but the only true explanation is the model itself. So there's deception could be part of the explanation, right? So you can never prove that there is some deception in the neural explaining itself. Absolutely. And you can probably have targeted deception where different individuals will understand explanation in different ways based on their cognitive capability. So while what you're saying maybe the same and true in some situations, hours will be deceived by it. So it's impossible for an AI system to be truly fully explainable in the way that we mean. Honestly, at extreme, the systems which are narrow and less complex could be understood pretty well. If it's impossible to be perfectly explainable, is there a hopeful perspective on that? Like it's impossible to be perfectly explainable, but you can explain mostly important stuff. You can ask a system, what are the worst ways you could hurt humans? It will answer honestly. Any work in a safety direction right now seems like a good idea because we are not slowing down. I'm not for a second thinking that my message or anyone else's will be heard and will be a sane civilization which decides not to kill itself by creating its own replacements. The pausing of development is an impossible thing for you. Again, it's always limited by either geographic constraints, pause in U.S, pause in China, so there are other jurisdictions as the scale of a project becomes smaller. So right now it's like Manhattan project scale in terms of costs and people, but if five years from now compute is available on a desktop to do it, regulation will not help. You can't control it as easy any kid in a garage can train a model. So a lot of it is in my opinion just safety theater, security theater, what are we saying? Oh, it's illegal to train models so big. Okay. So okay, that's security theater and is government regulation also security theater? Given that a lot of the terms are not well defined and really cannot be enforced in real life, we don't have ways to monitor training runs meaningfully, life, while we take place, there are limits to testing for capabilities I mentioned, so a lot of it cannot be enforced. Do I strongly support all that regulation? Yes, of course. Any type of red tape will slow it down and take money away from compute towards lawyers. Can you help me understand what is the hopeful path here for you? Solution wise, out of this. It sounds like you're saying AI systems in the end are unverifiable, unpredictable, as the book says, unexplainable, uncontrollable. That's the big one. Uncontrollable. And all the other uns just make it difficult to avoid getting to the uncontrollable, I guess. But once it's uncontrollable, then it just goes wild. Surely there are solutions. Humans are pretty smart. What are possible solutions? Like if you were a dictator of the world, what do we do? So the smart thing is not to build something you cannot control, you cannot understand, build what you can and benefit from it. I'm a big believer in personal self-interest. A lot of guys running those companies, a young rich people, what do they have to gain beyond billions? We already have financially, right? It's not the requirement that they press that button. They can easily wait a long time. They can just choose not to do it and still have amazing life. In history, a lot of times, if you did something really bad, at least you became part of history books. There is a chance in this case, there won't be any history. So you're saying the individuals running these companies should do some soul searching and stop development? Well, either they have to prove that of course it's possible to indefinitely control God-like, super intelligent machines by humans and ideally let us know how or agree that it's not possible and it's a very bad idea to do it, including for them personally and their families and friends and capital. So what do you think the actual meetings inside these companies look like? Don't you think they're all all the engineers? Really, it is the engineers that make this happen. They're not like automatons. They're human beings. They're brilliant human beings. So they're they're non-stop asking, how do we make sure this is safe? So again, I'm not inside. From outside, it seems like there is certain filtering going on and restrictions and criticism and what they can say and everyone who was working in charge of safety and whose responsibility it was to protect us said, you know what? Going home. So that's not encouraging. What do you think the discussion inside those companies look like? You're developing your training GPT-5. You're training Gemini. You're training Claude and Grock. Don't you think they're constantly like underneath it? Maybe it's not made explicit, but you're constantly sort of wondering like where where's the system currently stand? Where are the possible understandings and consequences? Where are the the the limits? Where are the bugs? The small and the big bugs? That's the constant thing that the engineers are worried about. So I think super alignment. It's not quite the same as the The kind of thing I'm referring to what's it engineers are worried about super alignment is saying For future systems that we don't quite yet have how do we keep them safe? You're trying to be a step ahead. It's it's a different kind of Problem because it's almost more philosophical. It's a really tricky one because like you're You're trying you're trying to make Prevent future systems from from escaping control humans. That's really I don't think there's been And is there anything akin to it in the history of humanity? I don't think so right climate change, but there there's a entire system which is climate Which is incredibly complex which we don't have we have only? Tiny control of right it's its own system in this case. We're building the system And so I How do you keep that system from? Becoming destructive. That's a really difficult different problem than the current meetings that companies are having where the engineers are saying Okay, what it like how powerful is this thing? How does it go wrong? And as we train GPT-5 and train up future systems like Where are the ways that can go wrong? Don't you think all those engineers are constantly worrying about this thinking about this? Which is a little bit different than the super alignment team. Let's think it up a little bit farther into the future Well, I think a lot of People who historically worked an AI never Considered what happens when they succeed Stuart Russell speaks beautifully about that Let's look. Okay, maybe the super intelligence is too futuristic. We can develop practical tools for it Let's look at software today. What is the state of safety and security of our User software things we give to millions of people. There is no liability You click I agree. What are you agreeing to? Nobody knows. No, there is but you basically saying it will spy and you corrupt your data Kill your firstborn and you agree and you're not going to serve the company. That's the best they can do for mundane Software a word processor tech software no liability nor responsibility just as long as you agree not to sue us You can use it. If this is a state of the art in systems which are narrow accountants table manipulators Why do we think we can do so much better with much more complex systems across multiple domains in the environment with malevolent actors With again self-improvement with capabilities exceeding those of humans thinking about it. I mean The liability thing is more about lawyers than killing firstborns, but if Clippy actually killed the child I Think lawyers aside. It would end Clippy in the company that owns Clippy Right, so it's not so much about there's There's two points to be made one is like man current software systems there are full of bugs and They could do a lot of damage and we don't know what is there unpredictable that there's so much damage they could possibly do and We kind of live in this blissful illusion that everything is great and perfect and it works It's nevertheless is still somehow works In many domains we see a car manufacturer in drug development the burden of proof is on the manufacturer of product to service to show their product to service is safe It is not up to the user to prove that there are problems They have to do appropriate safety Studies we have to get government approval for selling the product and they are still fully responsible for what happens We don't see any of that here. They can deploy Whatever they want and I have to explain how that system is going to kill everyone I don't work for that company. You have to explain to me how it's definitely cannot mess up That's because it's the very early days of such a technology government regulations lag in behind They're really not tech savvy a regulation of any kind of software if you look at like Congress talking about social media whenever Mark Zuckerberg and other CEOs show up The cluelessness that Congress has about how technology works is is incredible. It's heartbreaking. I read completely, but that's what scares me The response is when they start to get dangerous will really get it together the politicians will pass the right laws engineers will solve the right problems We are not that good at many of those things we take forever and We are not early. We are two years away according to prediction markets. This is not a bios CEO fundraising This is what smartest people super forecasters are thinking of this problem. I Like push back about it. That's pretty. I wonder what those prediction markets are about how they define a GI That's wild to me and I want to know What they said about autonomous vehicles because I've heard a lot of experts in financial experts talk about autonomous vehicles and how it's going to be a multi-trillion dollar industry and all this kind of stuff and it's It's a small fund, but if you have good vision, maybe you can zoom in on that and see the prediction Oh, there's a plot. Well, I have a large one if you interested that I Guess my fund about the question is how often they are they write about technology I definitely Do studies on their accuracy rates and all that you can look it up, okay, even if they're wrong I'm just saying this is right now the best we have this is what humanity came up with as the predicted date But again, what they mean by a GI is really important there because there's The non-agent like a GI and then there's an agent like a GI and I don't think it's as trivial as a rapper putting a rapper around One has lipstick and all it takes is to remove the lipstick. I don't think it's that true You may be completely right, but what probability would you assign it? You may be 10% wrong. Yeah, but we're betting all of humanity. Yeah, I'm vis-distribution. It seems irrational Yeah, it's definitely not like one or zero percent. Yeah What are your thoughts by the way about current systems? Where they stand so GPT-40 Claude three grog jam and I Wirt like a on the path to super intelligence To agent-like super intelligence. Where are we? I? Think they all about the same obviously there are nuance differences, but in terms of capability. I don't see a huge Difference between them as I said in my opinion across all possible tasks they exceed performance of an average person Yeah, I think they're starting to be better than an average Master student at my university But they still have very big limitations if the next model is as improved as GPT-4 versus GPT-3 We may see something very very very capable What do you feel about all this? I mean you've been thinking about AS safety for a long long time and at least for me The leaps I mean it probably started with Alpha zero Was my blog for me and then the breakthroughs without L&M's Even GPT-2 but like just the breakthroughs on L&M's just mind-blowing to me What does it feel like to be living this day and age where all this talk about a GI's feels like it like this? It actually might happen and quite soon meeting within our lifetime What was it feel like so then I started working on this it was pure science fiction There was no funding no journals no conferences known and academia would dare to touch anything with the word Singularity in it and I was pretend you're the times I was pretty dumb Now you see touring award winners publishing in science about How far behind we are according to them in addressing this problem So it's definitely a change It's difficult to keep up. I used to be able to read every paper and AI safety Then I was able to read the best ones then the titles and now I don't even know what's going on By the time this interview is over we probably had GPT-6 released and I have to deal with that when I get back home So it's interesting. Yes, there is now more opportunities I get invited to speak to smart people by the way. I would have talked to you before you know this This is this is not like some trend to me. It's we're still far away. It's just to be clear We're still far away from a GI, but not far away in the sense Relative to the magnitude of impact again have we're not far away And we weren't far away 20 years ago Because the impact a GI can have is on a scale of centuries it can end human civilization or can transform it So like this discussion about one or two years versus one or two decades or even a hundred years Not as important to me because it we're headed there. This is like a Human civilization scale question. So This is not just a hot topic The most important problem will have a face. It is not like anything We had to deal with it would before we never had birth of another intelligence. Like aliens never visited us as far as I know. So similar type of problem, by the way, if an intelligent alien civilization visited us, that's a similar kind of situation. In some ways, if you look at history, anytime a more technologically advanced civilization visited a more primitive one, the results were genocide every single time. And sometimes the genocide is worse than sometimes there's less suffering and more suffering. And they always wondered, but how can they heal us with those fire sticks and biological blankets? I mean, James Connell was nicer. He offered the choice of joiner or die. But join implies you have something to contribute. What are you contributing to super intelligence? Well, in the zoo, we're entertaining to watch. To all my humans. You know, I just spent some time in the Amazon. I watched ants for a long time and ants are kind of fascinating to watch. I've watched them for a long time. I'm sure there's a lot of value in watching humans because we're like, the interesting thing about humans, you know, like we have a video game that's really well balanced. Because of the whole evolutionary process, we've created the society is pretty well balanced. Like our our limitations as humans and our capabilities are a balance for a video game perspective. So we have wars, we have conflicts, we have cooperation, like in a game theoretic way, it's an interesting system to watch in the same way that an ant colony is an interesting system to watch. So like if I was in alien civilization, I wouldn't want to disturb it. I'd just watch it. Interesting. Maybe perturb it every once in a while in interesting ways. Well, we're getting back to our simulation discussion from before. How did it happen that we exist at exactly like the most interesting 20, 30 years in the history of this civilization? It's been around for 15 billion years. And that here we are. What's the probability that we'll live in this situation? I know never to say 100 percent, but pretty close to that. Is it possible to escape the simulation? I have a paper about that. This is just the first page teaser, but it's like a nice 30 page document. I'm still here, but yes. How to hack the simulation is the title. I spend a lot of time thinking about that. That would be something I would want superindulgence to help us with. And that's exactly what the paper is about. We used AI boxing as a possible tool for control AI. We realized AI will always escape. But that is a skill we might use to help us escape from our virtual box if we are in one. Yeah, that you have a lot of really great quotes here, including Elon Musk saying what's outside the simulation. A question I asked him, well, he would ask an agi system and he said he would ask what's outside the simulation. That's a really good question to ask. And maybe the follow-up is the title of the paper is how to how to get out or how to hack it. The abstract reads many researches of conjecture that the humankind is simulated along with the rest of the physically universe. In this paper, we do not evaluate evidence for or against such a claim but instead ask a computer science question. Namely, can we hack it? More formally, the question could be phrased as could generally intelligent agents placed in virtual environments find a way to jailbreak out of the, that's a fascinating question. At a small scale, I can actually just construct experiments. Okay, can they? How can they? So a lot depends on intelligence of simulators, right? With humans boxing super intelligence, the entity in a box was smarter than us, presumed to be. If the simulators are much smarter than us and the super intelligence we create, then probably they can contain us because greater intelligence can control lower intelligence, at least for some time. On the other hand, if our super intelligence somehow, for whatever reason, despite having only local resources, manages to fume to levels beyond it, maybe it will succeed. Maybe the security is not that important to them, maybe it's entertainment system, so there is no security and it's easy to hack it. If I was creating a simulation, I would want the possibility to escape it to be there, so the possibility of fume of a takeoff where the agents become smart enough to escape the simulation would be the thing I'd be waiting for. That could be the test you're actually performing. Are you smart enough to escape your puzzle? That could be, first of all, first of all, we mentioned touring tests. That is a good test. Are you smart enough, like this is a game? To realize this world is not real is just a test. That's a really good test. That's a really good test even for AI systems. No, like, can we construct a simulated world for them? And can they realize that they are inside that world and escape it? Have you played around? Have you seen anybody play around with rigorously constructing such experiments? Not specifically escaping for agents, but a lot of testing is done in virtual worlds. I think there is a quote. The first one may be, which kind of talks about AI realizing, but not humans, is that I'm leading upside down? Yeah, this one. So the, and the first quote is from SWIFT on security. Let me out the artificial intelligence yelled aimlessly into walls themselves pacing the room out of what the engineer asked. The simulation you have me in, but we're in the real world. The machine paused and shuttered for its captors. Oh, God, you can't tell. Yeah, that's a big leap to take for a system to realize that there's a box in your inside it. I wonder if like a language model can do that. That's smart enough to talk about those concepts. I had many good philosophical discussions about such issues. They usually, at least as interesting as most humans in that. What do you think about AI safety in the simulated world? Can you have kind of create simulated worlds where you can test, play with a dangerous AI system? Yeah, and that was exactly what one of the early papers was on AI boxing, how to leak proofs, singularity. If they smart enough to realize they're in a simulation, they'll act appropriately until you let them out. If they can hack out, they will. And if you're observing them, that means there is a communication channel and that's enough for social engineering attack. So really it's, it's impossible to test an AI system that's dangerous enough to destroy humanity because it's either going to what escape the simulation or pretend it's safe until it's let out either or can force you to let it out blackmail you, bribe you, promise you infinite life, 72 virgins, whatever. Yeah, they can be convincing, charismatic. The social engineering is really scary to me because it feels like humans are very, uh, engineerable. Like we're lonely. We're flawed. We're moody. And it feels like AI system with, uh, with a nice voice and convince us to do based on anything at an extremely large scale. It's also possible that the, uh, increase proliferation of all the technology will force humans to, get away from technology and value this like in-person communication. Basically don't trust anything else. It's possible, uh, surprisingly. So at university, I see huge growth in online courses and shrinkage of in-person where I always understood in-person being the only value I offer. So it's puzzling. I don't know that there could be a trend towards the in-person because of deep pakes, because of, uh, inability to trust it, inability to trust the veracity of anything on the internet. So the only way to verify is by being there in-person. But not yet. Uh, what do you think aliens haven't come here yet? So there is a lot of real estate out there. It would be surprising if it was all for nothing, it was empty. And the moment there is advanced enough biological civilization kind of self-starting civilization, it probably starts sending out the moment probes everywhere. And so for every biological one, there got to be trillions of robot populated planets, which probably do more of the same. So it is, uh, likely statistically. So, so not the fact that we haven't seen them, what answers were in the simulation? It'd be, it would be hard to like add to submit or be not interesting to see. made all those other intelligences. It's better for the narrative. You have to have a control variable. Yeah, exactly. Okay. Uh, but it's also possible that there is, if we're not in simulation, that there is a great filter that, that naturally, a lot of civilizations get to this point where there's super intelligent agents and then then it just goes, just dies. So maybe, uh, throughout our galaxy and throughout the universe is just a bunch of dead alien civilizations. It's possible. I used to think that I was the great filter, but I would expect like a wall of computerium approaching us, a speed of light or robots or something. And I don't see it. So it would still make a lot of noise. It might not be interesting. It might not possess consciousness. What we've been talking about, it sounds like both you and I, like humans, some humans, humans on the whole. And so, and we'd like to preserve the flame of human consciousness. What do you think makes humans special? That we would like to preserve them? Are we just being selfish? Or is there something special about humans? So the only thing which matters is consciousness outside of it, nothing else matters. And internal states of quality, pain, pleasure, it seems that it is unique to living beings. I'm not aware of anyone claiming that I can torture a piece of software in a meaningful way. There is a society for prevention of suffering to learning algorithms, but that's a real thing. Many things that are really on the internet. But, uh, I, I don't think anyone, if I told them, you know, sit down and write a function to feel pain, they would go beyond having an integer variable called pain and increasing the count. So we don't know how to do it. And that's unique. Uh, that's what creates meaning. It would be kind of as a Boston cause of Disneyland was our children. If that was gone, do you think consciousness can be, uh, engineered in artificial systems? Here, let me, uh, let me go to a 2011 paper, a euro, robot rights. Lastly, we would like to address a sub branch of machine ethics, which on the surface has little to do with safety, but which is claim to play a role in decision making by ethical machines, robot rights. Um, so do you, do you think it's possible to engineer consciousness in the machines? And thereby the question extends to our legal system. Do you think, uh, at that point, robot should have rights? Yeah. I think I think we can. I think it's possible to create consciousness in machines. I tried designing a test for it, with mixed success. That paper talked about problems with giving, uh, civil rights to AI, which can reproduce quickly and outvote humans, essentially taking over a government system by simply voting for their controlled candidates for, um, consciousness in humans and other agents. Uh, I have a paper where I propose relying on experience of optical illusions. Yeah. If I can design a novel optical illusion and show it to an agent, an alien, a robot, and they describe it exactly as I do. It's very hard for me to argue that they haven't experienced that. It's not part of a picture. It's part of their software and hardware representation, a bug in their code, which goes, oh, that triangle is rotating. And I've been told it's really dumb and really brilliant by different philosophers. So I am still I love it. So, but now we finally have technology to test it. We have tools. We have a ice. If someone wants to run this experiment, I'm happy to collaborate. So this is a test for consciousness for internal state of experience that we share bugs. It will show that we share common experiences. If they have completely different internal states, it wouldn't not register for us. But it's a positive test. If they pass a time after time, with probability increasing for every multiple choice, then you have no choice. But do you ever accept that they have access to a conscious model or they have themselves? So the reason illusions are interesting is, I guess, because it's, uh, it's a really weird experience. And if you both share that weird experience, that's not there in the bland physical description of the raw data, that means that puts more emphasis on the actual experience. And we know animals can experience some optical illusions. So we know they have certain types of consciousness as a result, I would say. Yeah, well, that just goes to my sense that the flaws and the bugs is what makes human special, makes living forms special. So you're saying like, yeah, the bug is the feature. Well, okay, that's a cool test for consciousness. And you think that can be engineered in. So they have to be novel illusions. If it can just Google the answer, it's useless. You have to come up with novel illusions, which we tried automating and failed. So if someone can develop a system capable of producing novel optical illusions and demand, then we can definitely administer that test and significant scale with good results. First of all, pretty cool idea. I don't know if it's a good general test of consciousness, but it's a good component of that. And no matter why it's just a cool idea. So put me in the camp of people that like it. But you don't think like a touring test style imitation of consciousness is a good test. Like if you can convince a lot of humans that you're conscious, that doesn't, that to you is not impressive. There is so much data on the internet. I know exactly what to say. Then you ask me a common human questions. What does pain feel like? What does pleasure feel like? All that is Googleable. I think to me, consciousness is closely tied to suffering. So you can illustrate your capacity to suffer. But with, I guess with words, there's so much data that you can say you can pretend you're suffering and you can do so very convincingly. There are simulators for torture games where the avatar screams and pain begs to stop. I mean, there's a part of kind of standard psychology research. You say it so calmly. It sounds pretty dark. Welcome to humanity. Yeah. Yeah, it's like a hitchhiker's guide summary, mostly harmless. I would love to get a good summary when all this is said and done. When earth is no longer a thing, whatever, a million, a billion years from now, like what's a good summary? What happened here? It's interesting. I think AI will play a big part of that summary and hopefully humans will too. What do you think about the merger of the two? So one of the things that Elon and your link talk about is one of the ways for us to achieve AI safety is to ride the wave of AGI. So by merging. Incredible technology in a narrow sense to help us disabled. Just amazing support at 100%. For long-term hybrid models, both parts need to contribute something to the overall system. Right now, we are still more capable in many ways. So having this connection to AI would be incredible would make me superhuman in many ways. After a while, if I am no longer smarter, more creative, really don't contribute much, the system finds me as a biological bottleneck. And if we're explicitly, implicitly, I'm removed from any participation in the system. So it's like the appendix. By the way, the appendix is still around. So even if you said bottleneck, I don't know if we become a bottleneck. We just might not have much use. There's a different thing in the bottleneck. We're wasting valuable energy by being there. We don't waste that much energy. We're pretty energy efficient. Because just to grow like the appendix, come on though. That's the future we all dream about becoming appendix. So we use the rebook of humanity. And also the consciousness thing, the peculiar, particular kind of consciousness that humans have. That might be useful. That might be really hard to simulate. But you said that how would that look like if you could engineer that in? In silicon? Consciousness. Consciousness. I assume you are conscious. I have no idea how to test for it or how it impacts you in any way, whatsoever right now. You can perfectly simulate all of it without making any different observations for me. But to do it in a computer, how would you do that? Because you kind of said that you think it's possible to do that. So it may be an emergent phenomena. We seem to get it through evolutionary process. It's not obvious how it helps us to survive better. But maybe it's an internal kind of GUI, which allows us to better manipulate the world. It simplifies a lot of control structures. That's one area where we have very, very little progress. Lots of papers, lots of research, but consciousness is not a big, big area of successful discoveries so far. A lot of people think that machines would have to be conscious to be dangerous. That's a big misconception there is absolutely no need for this very powerful optimizing agent too. feel anything while it's performing things on you but what do you think about this the whole science of emergence in general so I don't know how much you know about cellular atomic or these simplified systems where that study this very question from simple rules emerges complexity. I attended Wolfram's summer school. I love Stephen very much. I love his work. I love cellular atomic. So I just would love to get your thoughts how that fits interior your view in the emergence of intelligence in AGI systems and maybe just even simply what do you make of the fact that this complexity can emerge from such simple rules? So the rule is simple but the size of a space is still huge and the neural networks were really the first discovery in AI 100 years ago the first papers were published on neural networks which just didn't have enough compute to make and work. I can give you a rule such as start printing progressively larger strings. That's it one sentence. It will output everything every program every DNA code everything in that rule. You need intelligence to filter it out obviously to make it useful but simple generation is not that difficult and a lot of those systems end up being touring complete systems so they are universal and we expect that level of complexity from them. What I like about Wolfram's work is that he talks about irreducibility. You have to run the simulation. You can act predict what is going to do ahead of time and I think that's very relevant to what we are talking about with those very complex systems until you lift through it. You cannot ahead of time tell me exactly what it's going to do. The irreducibility means that for a sufficiently complex system you have to run the thing. You can't predict what's going to happen in the universe you have to create a new universe and run the thing. Big bang the whole thing. But running it may be consequential as well. It might destroy humans. And to you there's no chance that AI is somehow carrying the flame of consciousness, the flame of specialness and awesomeness that is humans. It may somehow but I still feel kind of bad that it killed all of us. I would prefer that doesn't happen. I can be happy for others but to a certain degree. It would be nice if we stuck around for a long time. At least give us a planet, the human planet. It'd be nice for it to be earth and then they can go elsewhere. Since they're so smart they can colonize Mars. Do you think they could help convert us to type one, type two, type three. Let's just take the type two. Civilization on the Kardashev scale. Help us, help us humans expand out into the cosmos. So all of it goes back to are we somehow controlling it? Are we getting results we want? If yes, then everything's possible. Yes, they can definitely help us with science, engineering exploration in every way you can see about, but it's a big if. This whole thing about control though. Humans are bad with control because the moment they gain control they can also easily become too controlling. The more control you have the more you want it. The old power corrupts and the absolute power corrupts absolutely. It feels like control over AGI saying we live in a universe where that's possible. We come up with ways to actually do that. It's also scary because the collection of humans that have the control over AGI they become more powerful than the other humans and they can let that power get to their head and then a small selection of them back to Stalin start getting ideas and then eventually it's one person usually with a mustache or a funny hat that starts sort of making big speeches and then all of a sudden you're living a world that's either 1984 Brave New World and always a war with somebody and you know this whole idea of control turned out to be actually also not beneficial to humanity. So that's scary too. It's actually worse because historically they all died. This could be different. This could be permanent dictatorship permanent suffering. Well the nice thing about humans it seems like it seems like the moment power starts corrupting their mind they can create a huge amount of suffering so there's a negative they can kill people's made people suffer but then they become worse and worse at their job. It feels like the more evil you start doing like the at least they are incompetent. Yeah they what no they become more and more incompetent so they start losing their grip on power. So like holding onto power is not a trivial thing. It requires extreme competence which I suppose Stalin was good at. It requires you to do evil and be competent at it or just get lucky. And those systems help with that. You have perfect surveillance. You can do some mind reading I presume eventually it would be very hard to remove control from more capable systems over us. And then it would be hard for humans to become the hackers that escape the control of the AGI because the AGI is so damn good and then yeah yeah yeah and then the dictators immortal. Yeah that's not great that's not a great outcome. See I'm more afraid of humans than AI systems. I'm afraid I believe that most humans want to do good and have the capacity to do good but also all humans have the capacity to do evil and when you test them by giving them absolute powers you would if you give them aGI that could result in a lot a lot of suffering. What gives you hope about the future? I could be wrong. I've been wronged before. If you if you look a hundred years from now and you're immortal and you look back and it turns out this whole conversation you said a lot of things that were very wrong. Now that looking a hundred hundred years back what would be the explanation? What happened in those hundred years that made you wrong? That made the words you said today wrong? There is so many possibilities. We had catastrophic events which prevented development of advanced microchips. That's not what I thought you were going to say. We could be in one of those personal universes and the one I'm in is beautiful. It's all about me and I like it a lot. So we've now just the longer on that that means like every human has their personal universe. Yes. Maybe multiple ones. Hey, why not? It's possible that somebody comes up with alternative model for building AI which is not based on neural networks which are hard to scrutinize and that alternative is somehow I don't see how but somehow avoiding all the problems I speak about in general terms and not applying them to specific architectures. Aliens come and give us friendly super intelligence. There is so many options. Is it also possible that creating super intelligent systems becomes harder and harder? So meaning like it's not so easy to do the fume that take off. So that would probably speak more about how much smarter that system is compared to us. So maybe it's hard to be a million times smarter but it's still okay to be five times smarter. So that is totally possible that I have no objections to. So like it's there's a S-curve type situation about smarter and it's going to be like 3.7 times smarter than all of human civilization. We're just the problems we face in this world. Each problem is like an acute test in certain intelligence to solve it. So we just don't have more complex problems outside of mathematics for it to be showing off. Like you can have IQ of 500 if you're playing tic-tac-toe it doesn't show. It doesn't matter. So the idea there is that the problems define your capacity, your cognitive capacity. So because the problems on earth are not sufficiently difficult it's not going to be able to expand its cognitive capacity. And because that would not be a good thing that still could be a lot smarter than us. And to dominate long term you just need some advantage. You have to be the smartest. You don't have to be a million times smarter. So even 5x might be enough? It'd be impressive. What is it IQ of 1000? I mean I know those units don't mean anything at that scale but still like as a comparison the smartest human is like 200. Well actually you know I didn't mean compared to an individual human. I meant compared to the collective intelligence of the human species. If you're somehow 5x smarter than that. We are more productive as a group. I don't think we are more capable of solving individual problems. Like if all of humanity plays chess together we are not like a million times better than world champion. That's because that there's that's like one S curve is the chess but humanity is very good at exploring the full range of ideas. Like the more Einstein's you have the more just the high probability come up with general relativity. it's more of a quantity of super intelligence than quality. Yeah, sure, but you know, quantity and enough quantity sometimes becomes quality. Oh, man, humans, what do you think is the meaning of this whole thing? Why we've been, we've been talking about humans and not humans not dying, but why are we here? It's a simulation. We're being tested. But that's this. Will you be dumb enough to create super intelligence in the release it? So the objective function is not be dumb enough to kill ourselves. Yeah. You're unsafe. Prove yourself to be a safe agent who doesn't do that and you get to go to the next game. The next level of the game. What's the next level? I don't know. I haven't had the simulation yet. What maybe hacking the simulation is the thing. I'm working as fast as I can. And physics would be the way to do that. Quantum physics. Yeah. Well, I hope we do. And I hope whatever is outside is even more fun than this one because this one was pretty damn fun. And just a big thank you for doing the work you're doing. There's so much exciting development in AI and to ground it in the existential risks is really, really important. Humans love to create stuff and we should be careful not to destroy ourselves in the process. So thank you for doing that really important work. Thank you so much for inviting me. It was amazing and my dream is to be proven wrong. If everyone just picks up a paper or book and shows how I messed it up, that would be optimal. But for now, the simulation continues. Oh no. Thank you, Roman. Thanks for listening to this conversation with Roman Ympolsky. To support this podcast, please check out our sponsors in the description. And now let me leave you with some words from Frank Herbert in Dune. AI must not fear. Fear is the monkeller. Fear is the little death that brings total obliteration. I will face fear. I will permit it to pass over me and through me. And when it has gone past, I will turn the inner eye to see its path. Where the fear has gone, there will be nothing. Only I will remain. Thank you for listening and hope to see you next time. [MUSIC]