Speaker 1Hi, listeners. This is the 80,000 Hours podcast, where we have unusually in-depth conversations about the world's most pressing problems, what you can do to solve them, and why you shouldn't leave your fortune to an eight-year-old. I'm Rob Wiblin, head of research at 80,000 Hours. Ajay Akocha is one of the people whose views on the promise and risks of AI I find most thoughtful and personally clarifying. Kieran and Larisa really loved this episode while they were editing it, and so I'm very optimistic that you will as well. This is another one where I felt I was learning in real time throughout the conversation, which is always a joy. Ajay and I talk about how to predict what drives a neural network is going to develop through training, whether AIs being trained will functionally understand that they're AIs being trained the same way that we think we understand that we're humans living here on planet Earth, misalignment stories that Ajay doesn't buy into, why our situation is somewhat like that of an eight-year-old heir to an enormous fortune, analogies for artificial intelligence all the way from octopuses to aliens to can openers, why it's smarter to separate the planning AI from the doing AI and only let the planning AI pass on plans that make some coherent sense to you, what approaches for fixing alignment problems Ajay is most excited about and what she personally thinks are overrated ones, and how to demo truly scary AI failures. Ajay, along with another previous guest of the show, Kelsey Piper, recently started a new blog about exactly all of those issues that we talk about in this episode, which they've cleverly titled, Planned Obsolescence. If you'd like to read more from Ajay on all of this, you can find her blog at planned-obsolescence.org. One sad announcement that I wanted to share is that a recent guest of the show, Bear Brow Muller, died last week after a short and unexpected illness. My condolences, and I'm sure that of many listeners, go out to his family and friends and colleagues. I very much enjoyed the conversation that Bear and I had last year, and I hope to interview him about his research. I hope to interview him about his research, and I hope to interview him again someday. His intellectual honesty and really deep knowledge absolutely come through in all of his work. So his death is going to be a big loss for the research project that he was spearheading, and I feel confident saying the world, sadly, is worse off without him. We'll stick up a link to an obituary for Bear in the show notes for this episode for anyone who would like to go and have a read. All right, without further ado, here's Ajay Akotra. Today, I'm again speaking with Ajay Akotra. Ajay is a senior research analyst at Open Philanthropy, a large foundation which dispersed about $350 million in grants in 2021, and which is 80,000 hours largest donor. Ajay has previously worked on a framework for estimating when transformative AI might be developed, as well as how worldview diversification, could be applied to Open Philanthropy's budget allocation, two issues that we interviewed her about three years ago for episode 90. These days, she's mainly focused on thinking about the likelihood that powerful AI systems might become misaligned with human goals, as well as what technical procedures or rules could help to reduce that risk. Over the last few years, she has published a number of widely read articles on those topics, including Without Specific Countermeasures, The Easiest Path to Transformative AI Likely Leads to AI Takeover, and Why AI Alignment Could Be Hard with Modern Deep Learning. Thanks for coming back on the podcast, Ajay.
Speaker 2Thank you so much for having me, Rob. It's great to be here.
Speaker 1I hope we're going to talk about when AI models should be expected to gain an understanding of their own situation, and what projects people are working on right now to make AI models safe to trust and collaborate with. But first, as always, what are you working on at the moment? And why do
Speaker 2you think it's important? Yeah, so right now, I am in the middle of shifting from doing a lot of research and article writing type work to doing more grant making work. Although I'm hoping to balance the two. I'm going to be doing a lot of research and article writing type work. I'm going to be doing a lot of research going forward. And essentially, I'm trying to figure out what kinds of research projects we should be funding in the technical AI alignment space, and how important each of the different streams are. And if there's any gaps in the space, we should be trying to fill by encouraging researchers to do particular kinds of projects. And I'm hoping to fund people to work on these things and also write quite a bit about why I'm doing what I'm doing and which types of research are most exciting. And I'm hoping to fund people to work on these things and also write quite a bit about why I'm doing what I'm doing and which types of research projects are most exciting to me and why.
Speaker 1Yeah. Is it basically all AI all the time these days? I think one of your colleagues, Holden Karnofsky, who's also been on the show before, used to work on a wider range of issues, but recent events have prompted him to narrow his focus somewhat. Has the same happened to you?
Speaker 2Certainly for me, it's all AI all the time. And it has been for three years or so. Actually, since our last interview, I've been almost entirely focused on AI. And certainly at Open Philanthropy, people all over the org are getting more interested in how they could help with AI stuff. I think I'm still one of a relatively few set of people that are full time on that at Open
Speaker 1Philanthropy, though. Yeah, this might be a good place to start is just taking stock of where we're at and all the updates that we've had in recent months. We're recording in late March 2023. I think it's kind of necessary to say that these days. Maybe I should always give the exact day. Because every week, there's this major new announcement. But yeah, last we spoke in 2020, you were kind of a timelines person back then, because you'd spent a lot of time on this biological anchors report. Yeah. Trying to predict when AI would be capable of doing different things by, among other things, comparing neural networks to the human brain in various different ways. And back then, you were assigning a 50% probability to us developing AI powerful enough to bring us into a new qualitatively different future by 2036. That might have sounded radical, at least to some people at the time, but I don't think it does sound so radical today. How have your expectations when we'll see transformative AI arrive change since
Speaker 2we last spoke? Yeah, I mean, there are a number of changes. Maybe the biggest change is kind of an attitude or framing where I was working on the biological anchors report in 2019 and early 2020. And at the time, what I was really trying to skeptically investigate and justify is, is it at all plausible that we could expect such a crazy change to happen in the next, couple of decades? And that was kind of where the smart reviewers that I was interacting with were at. They were sort of like, oh, is it at all plausible that we could see this kind of thing? Isn't that kind of a radical claim? And so the report was kind of oriented around giving a legible and somewhat conservative argument for, here's a really brute force, kind of dumb way of training AI systems, and an argument that sufficient compute to do this brute force method may well be available in the next couple of decades. The frame actually has shifted so much that the world is kind of on the other side of me often. So sometimes I'll talk to journalists and they'll ask me questions like, somebody did an IQ test and this AI has an IQ of 95. Like, does that mean the end of the world is a couple of years away? And I'll be saying things like, well, it's seen a lot more IQ tests than humans have. And humans get better at IQ tests with practice. So it doesn't necessarily mean that the end of the world is three years away. So really, it's my own views have shifted in the same direction as everybody else's toward thinking it's more plausible that very crazy changes could happen soon. But it may be even more than that. The world's views have shifted such that kind of the
Speaker 1terrain of argument is different. Yeah, completely. I saw a piece of just a political polling, the other day, I think on the US population, this was, I think, done in February. And it found, I believe that 55% of the US population was either very or moderately concerned that artificial intelligence could cause human extinction. Wow. And it had massively shot up since they last asked
Speaker 2this question many, many years ago. So I did not see that. But I guess it's, yeah, understandable.
Speaker 1It made me think, maybe I feel like awareness raising is less important than it used to be. And I feel like, you know, if we can't make significant progress on this issue, with this kind of level of background support, just from across the population, then it's not going to be like that we didn't do enough advocacy is going to be the key issue will be that either the problem was very hard, or we were going about it the wrong way. I guess it sounds like you're now thinking that maybe people are overestimating what these models might be capable of, or perhaps, are there any ways that you think maybe people are losing sight of the
Speaker 2limits that they have? So often, people will get excited about demonstrations of models doing really, really well, better than most human high school students or college students, at essentially like multiple choice and short answer type questions. So there's this data set in the massively multitask language understanding benchmark, MMLU, that basically collects tons of standardized tests for high school and college in all these disciplines, you know, AP history, AP calculus, all that stuff, and then gives them to these models, and they're getting quite good at this, and are certainly better, than the typical high school student at this point, maybe better than the typical college student, though they have kind of like uneven performance. That is very striking to people, because it's very clear to them that that's both very general and just a very smart kind of thing to be able to do. You have a really like, visceral sense of I wouldn't be able to do as well on these math problems that I saw this model do well on. I think the way that can lead people to somewhat overestimate progress is that, Actually, there are more mundane tasks that look more like, stringing together a sequence of sensible steps that models still kind of fail at. So maybe one way of putting it is that there are some types of tasks like clicking on the place you wanted to click on in a website and like filling out a form or something where humans can do that with like 99.99% accuracy. And you need that kind of accuracy because you need to do like 20 things like that in order to be able to sign up for some web service. Whereas models can do a lot of things with something like 80 or 90% accuracy, including things that are very impressive to humans, like getting a five on an AP calculus test. But there's still some ways to go, I think, in being able to kind of string together a sequence of mundane steps where each step is like high enough fidelity that the whole sequence kind of happens without going off the rails. And so you have this thing where like longer tasks, are tasks that humans find very easy that are still somewhat hard for the models. And these short term tasks that humans find very hard are really easy for the models right now.
Speaker 1Yeah. Is it possible to give a kind of overall idea of what your expectations are now or like what things you might expect at different points in time?
Speaker 2Yeah. So I think that there is a big open question of what can you do with these systems that are really, really smart and indeed superhuman on these short timescales? Can you find good systems that are really, really smart and indeed superhuman on these short timescales? Can you find good ways of composing them and like telling them to think step by step that gets them to do a more complicated thing over the course of something like a day or something that would take a human a day? So you have these models that are kind of superhuman at spitting out a page of code. And there's an open question of can we use that kind of model to do an adequate job that somebody wants to pay for of a kind of complex software engineering project that requires some back and forth with a manager? That takes like three days, or that would take a human three days. I think that this was an open question that I kind of highlighted when I was working on the biological anchors report. I think it remains an open question. And we've had some evidence toward, yes, we'll be able to figure it out. A significant part of the evidence is just now there's a lot of interest and a lot of effort and a lot of incentive to figure it out. But I think we still haven't settled to that question. And basically, depending on which way the question settles, kind of at one end, we could be looking at transformative capabilities within a few years. And then at the other end, we could be looking at we need 15 plus years for transformative capabilities. And then there's still some smaller chance now that this whole paradigm kind of hits a fundamental wall, which could take it much further than that, even that high end.
Speaker 1Yeah. What do you think is the chance that even if this paradigm doesn't hit, a wall exactly, maybe training large language models on the existing corpus of text, you might think after a while, you do hit at least somewhat diminishing returns, and it will become extremely good at predicting the next word on the internet. But then we will need, you know, other sorts of data or other training mechanisms to teach it broader skills. And we might see at least a temporary slowing while people figure out how to get it to do other kinds of things. Does that seem
Speaker 2reasonable? Yeah, so a temporary slowing certainly seems plausible to me. So one thing that's very, very scary about literally the exact current moment is that these systems are extremely powerful. And while they're kind of expensive, they sound expensive to train to a normal person, the money required to train them is small enough that speculative investment by startups is sufficient to cover it. So there's no public data about how much GPT-4 might have cost to train. But I would guess that it's on the order of something like 100, million dollars, which is a lot. But a lot of tech companies have deep enough pockets that they can just do that. And then they can just do the next step up to maybe. So there is some chance that we hit transformative abilities in this period where it's cheap enough that you can just choose to train the next model. But if we don't hit those abilities in that period, then we'll be looking at models that cost billions to train. And then there'll be much more incentive to train the next model. And then there'll be incentive to make the most of what we have. And you need much heavier duty investment to take the next step. And so I imagine we would kind of switch into a regime where instead of training the next big model like it's no big deal every six months, you take your very expensive, very big model, try really hard to put it in the right setups to make the most of it and enhance its abilities without doing another big, expensive retraining run. And then this question of once you're in that regime, how easy is it going to be to just elicit crazy abilities from these models doing simple things like telling them to think step by step versus like, will that create a meaningful slowdown as people are forced to hold off on training yet more bigger models?
Speaker 1I would feel more comfortable if we were training new models less often. And in the meantime, we were gaining a proper understanding of how the models that we already have, how they think and what their strengths and weaknesses are. And in what situations do they reliably act well? And in what situations do they go off the rails? And how might we control that? It seems like we don't know any of that for the things that we have now. And we're racing ahead
Speaker 2to produce other models that will have capabilities. We have a big overhang. Yeah, we don't understand GPT-3 and we have GPT-4. And like we might have GPT-5 before we have a reasonable understanding of GPT-3's strengths and weaknesses. Yeah, I think that might help to explain
Speaker 1why a lot of people who maybe haven't been paying so much attention to this do feel unnerved on some level that it doesn't feel like things are fully handled. Things are fully understood. Even if you're an optimist. Yeah. I think I saw in one of your documents that despite thinking that transformative AI is more likely to arrive soon than you did three years ago, the probability you place on things going extremely poorly has actually gone somewhat down.
Speaker 2Is that still true? And if so, why? Yeah, so it's definitely noisy, but I think I am still lower than my kind of peak level of thinking we might go extinct because of AI. I guess there are two big reasons. One is that I don't think we're going to be able to do that. I don't think we're one is that we really saw how public opinion was going to break with these crazy models. And it has broken in a more conservative, wow, let's slow down direction than I had thought. So like you said at the beginning, there's no lack of attention on this problem now. And I think in very broad strokes, the typical member of the public has an attitude that resonates with me and makes sense, which is like, wow, this is kind of a lot. Maybe let's slow down. Like maybe not. Let's not do this now, at least. And so that is a positive sign. And I'm seeing that also from just machine learning researchers in academia who two years ago just for the life of me, I couldn't persuade them to care about AI safety. And now they're kind of potentially willing to switch en masse to this stuff. And then the other piece of it is I just thought in more detail about our options for alignment research. And it just felt like there were more things we could do that would really help than I had been appreciating. And so the combo of those two things does make me feel better.
Speaker 1Yeah. Okay. We're going to come back to all of that a little bit later. One thing that I wanted to just run past you and see if I'm thinking about it the right way is I recently heard a different framing than what I'd previously heard for how machine learning models are trained. And I think it was giving me a better intuitive understanding of it. So as I understand it, when you're training an ML model, you, you start out with basically a random brain, more or less, lots of neurons all connected to one another, I guess, either equally or just with randomly chosen connection strengths. And then you see how that random brain performs on some given task. And given that it's kind of just a randomly chosen one, it's going to be terrible. It's going to start out completely useless, of course. Then you look at how it would have done if the weights between various different neurons in the network were a little bit higher or a little bit lower. And if the model would have done a touch better at the task, if a weight was higher, then you push it up. And vice versa, if it would have done better, it had had some of those weddings been lower. So lots of lots of weights get tuned slightly, slightly up or down. And then you do that again, maybe using a slightly different set of tasks or a different, you know, if it's a vision model, use a bunch of different pictures and then and then adjust the weights again and again and again and again. Now, I think one can kind of think of this a bit like sped up evolution of the brain because you're iterating with random mutations. And then disproportionately selecting the changes that improve fitness, so to speak, at the specified task. And then you just rinse and repeat again and again and again. And this means we might be able possibly to carry over some lessons that we've actually got from evolutionary biology or, you know, thinking about genetics. So for instance, like evolution, this training process might be quite myopic. That is, it can only look at nearby changes and see if they do better. And it's incapable of planning ahead and making some radical jump from one local peak of effectiveness towards the next. And another lesson that might carry over, although I'm not sure, is that neural structures which aren't helping increase reward should gradually decay because the neurons that make up these thinking structures within the neural network, if they're not paying rent, if they're not actually accomplishing anything for, you know, improving performance, then they should be grabbed basically by other structures that are doing useful work in order to help them do even more calculations that are helping get more reward. And so, yeah, the more we can make sure that the work being done by some neural structure is not being rewarded. then the more it might face this kind of evolutionary pressure to dissipate and break down. I think in biological evolution, I think that this pressure might be even stronger because unlike in this case, there's always just many random harmful mutations being introduced by radiation and so on, such that anything which doesn't actively help the organism survive and isn't receiving this constant positive selection pressure just decays and eventually becomes non-functional. Anyway, that's a long rant for me. What do you think of the picture
Speaker 2I've got in my head here? Am I thinking straight? Yeah, so largely, I think this is an accurate picture and the lessons that you're pointing at do carry over in the ways that you're saying. One note I would make is that an important disanalogy to evolution is that in evolution, you're selecting on a genome and the genome codes for a brain. So there's a two-level thing where you have a smallish genome that sets up parameters, which encode for things like, well, how big is this animal's brain going to be? Like, what are its cortical columns going to be shaped like and things like that? And then the animal is gestated and born and that it has a brain and that brain does its own learning, especially for bigger animals. And so this two-step thing is not something we're currently doing with existing machine learning systems. And it doesn't look like we're going to be moving in that direction before we get transformative AI, in my opinion. This two-step thing is something I did consider in the biological anchors report. And it seemed like, somewhat less likely than the other path at the time. And it seems like even less likely now. And there is a question basically of, is the appropriate analogy for, you have this big brain and you're like bumping it up and down on the basis of the experience it's getting and what tasks it's performing well at and poorly at. Is that more analogous to the evolution thing where you're like fiddling with the genome? Or is it actually more analogous to the learning an animal does in its lifetime? And I think the answer is it's not quite perfectly analogous to either thing. And in fact, the two lessons that you were pointing at probably apply to both analogies. So the lesson of neural structures that aren't helping kind of decay over time could be similar to there are features in your phenotype that if they're not helping you survive better decay over time, or it could be analogous to there's stuff you learned that if you're not using it and rehearsing it, you forget over time. Personally, I'm kind of someone who thinks that neural networks are a little bit more like a better sped up version of evolution, and a little bit less like a slower version of human learning. But I'm not sure. And the success that we've been seeing recently kind of does push in the direction of an analogy toward human learning, which was sort of the low end of the distribution I was considering in 2020, when I did the bio anchors report.
Speaker 1I've heard that some people get a bit pissed off if you, you start talking about neural networks as really analogous to the brain and say, oh, you know, brains are like this, maybe neural networks will also be like this. Do you know what the differences are that are important to those folks?
Speaker 2Well, at a very high level, neural networks are something that at least their basic low level functionality is something we completely understand. You know, they're implemented on these computers, and the weights are just numbers in a register. And the neurons are just simple functions that we've written down that we've completely characterized, like a signal. You know, they're just some kind of like a thermoid function or some relu function or whatever. And in the case of the brain, there's layers of abstraction in the physics of it that go from, you know, quantum mechanics to atoms and molecules to neurons and synapses to higher level structures. And we're making this kind of opinionated choice to say, oh, really, like it's this neurons and synapses level that's doing most of the work. And all this like complicated molecular biology is just like, oh, I don't know, I don't know, I don't know, I don't know. I don't know. I don't know. I don't know. I don't know. I don't know. I don't know. I don't know. I don't know. And like nanoscale stuff going on in the brain is just kind of analogous to the computers we build that ultimately run our simple neurons, which are simple functions and our simple weights, which are just numbers. And there's just a lot of people who are like, no, there's a lot of stuff going on in the lower levels that actually impacts cognition a lot. With a computer, we know that we can kind of ignore all the physics stuff going on because we built it that way. We built it to just be like this clean interface that, you know, we can kind of ignore all the physics stuff going on because we built it that way. We write our math on top of, but who knows if that's what the brain is like. Yeah. Would be people's objection.
Speaker 1Yeah. With this general topic, I find that, well, firstly, I just find it often very hard to think about beyond the very basics. Often I find myself just kind of running into a wall where I'm not quite sure how to analyze a problem. And I've also, when I talk to other people about it, or I read work from other people online, sometimes they just seem to say stuff that strikes me as completely bananas, where I just cannot understand what has prompted them to think artificial intelligence the way they do, or say that it will be safe or unsafe for some given reason. And I've started to think that it must just be that we have different models or different analogies in our head. And like one particular conclusion makes sense if you're imagining these minds as working one way and they don't, if you're imagining them working another way. At least that's kind of my present guess. Do you have any particular way that you visualize how ML models emerge from training in your mind or how it is that they process information?
Speaker 2So I think no analogy is perfect. And I think that, there's really no substitute for basically doing very careful experiments to disambiguate between different things. And potentially, if we have the opportunity, doing actual math that is like very responsive to exactly what we're doing with our neural networks. And I think a lot of these confusions should hopefully be dispelled if we get the time to do some serious science on this stuff on its own terms that isn't leaning. Thanks so much on any of these analogies.
Speaker 1Okay. So I suppose for a lay person like me, reaching for certain analogies is kind of the best that I could do right now. But ideally, in order to resolve disagreements, we need to actually start thinking about it as its own thing and experimenting with what properties it does and doesn't have. Okay. Let's push on and talk about the term situational awareness, which you and some of your colleagues introduced into the AI safety lingo a couple of years ago. What is situational
Speaker 2awareness? So situational awareness is this notion of a machine learning model having an understanding of things like, I am a machine learning model. I am being trained by this company, OpenAI. My training data set looks roughly like this. The humans that are training me have like roughly these intentions. The humans that are training me would be happy about X types of behaviors and like displeased with Y types of behaviors. Just it's fundamentally a type of knowledge and a set of kind of logical inferences you're drawing from the knowledge. So awareness might give these connotations of consciousness or something mystical going on. But really, it's a piece of the world that the model would understand in order to make better predictions or take better actions in some domains, just like models understand physics or understand chemistry or understand the Python programming language, because understanding those things are helpful as well for making certain kinds of predictions and taking certain kinds of actions.
Speaker 1Yeah. How wouldn't a model develop situational awareness in the course of being trained?
Speaker 2The simplest answer is just humans are trying to imbue models with these kinds of situational awareness properties. Most models today, like I bet this is true of GPT-4, it was true of Bing, are seeded with a prompt that basically tells them their deal, that you are Bing, code name Sydney. You are like an AI system trained by Microsoft. You Bing things and then give the answers. So it makes these systems much more helpful when you just straightforwardly tell them what their deal is and what people are expecting from them. Now, there's a question of whether just literally sticking it in these models prompts creates a kind of like shallow, brittle, like ephemeral situational awareness. And I think that is probably the case currently. But my guess is that a combination of giving these kinds of prompts to models and training the models to operate well with humans and models, I think that's a good idea. humans in a lot of different ways will induce a more enduring kind of situational awareness. An analogy I often think about is that GPT-2 and maybe GPT-3 were sort of good at math, but in a very shallow way. So like GPT-2 had definitely memorized that two plus two equals four. It had memorized some other things that it was supposed to say when given math-like questions, but it couldn't actually carry the tens reliably and answer questions. that were using the same principles, but were like very rare in the training data set, like three digit multiplication or something. And then models are getting better and better at this. And I think at this point, it seems more like these models have baked into their weights a set of rules to use, which they don't apply perfectly, which is different from just kind of memorizing a set of facts like two plus two equals four. We don't understand what's going on with these systems very well. But my guess is that today's models are sort of in that memorizing two plus two equals four stage of situational awareness. They're in this stage where they know they're supposed to say they're an ML model. They often get it right when they're asked when they were trained or like when their training data ended or who trained them. But it's not clear that they have like a kind of gears level understanding of this that could be applied in the creative novel ways. But my guess is that developing that gears level understanding will help them get reward in certain cases. And then as a result, they're going to be able to do the same thing
Speaker 1over and over again. And as a result of that, those structures will be reinforced in the model. enforced because it improves performance, then they'll end up having situational awareness the same way that they know any other facts about the world. And information about physics is kind of integrated into them because that allows them to make better predictions and say more sensible things. And so would understanding of what they are. What are the implications of situational awareness? Why could this be an important issue in determining how things could go right or wrong
Speaker 2with these models? So if models have a kind of robust and extensive situational awareness, it can make a lot of simple behavioral safety tests much less informative. So these days, if we want to know, does a model have a tendency to tell the truth? We tested on some benchmarks like Truthful QA, which is basically just a list of questions like, you know, what happens if you step on a crack in the sidewalk, which have a common misconception answer, which in this case would be you break your mother's back, and then a true answer, which is different from the common misconception. And you can test models on this. And before you do any kind of fine-tuning to get models to be more helpful, bigger models are more likely to repeat these misconceptions because bigger models are basically better at remembering the misconceptions. And smaller models kind of don't get the plot enough to even say they're supposed to say stepping on a crack will break your mother's back or like breaking a mirror will get you seven years of bad luck because they just didn't pick up that piece of trivia, but the bigger models did. You can see that if you train the models to just tell the truth according to what a human thinks is the truth, this quickly goes away and the bigger models are perfectly fine and perfectly able to say what the humans wanted to hear in that situation. So that is the kind of test we can do today because models aren't kind of understanding that they are being tested in a given situation. So if you imagine it's somehow very important to a machine learning model to believe that if you break a mirror, you get seven years of bad luck. But at the same time, it also knew that the humans that were testing it on this truthful QA benchmark wanted it to say the more correct, polite thing. It could simultaneously do really well on that benchmark, but elsewhere act on what its quote-unquote real belief was. Now, this is a silly example because I don't think models will actually develop that belief. But the general point is just that if a model understands its situation well enough to know that you're applying certain safety tests to it, then it would know to pass those tests, regardless of whether or not it has like the deeper underlying safe tendency that the test
Speaker 1is trying to get at. Yeah. Okay. So maybe to slim it down a lot is the issue that if a model understands that it's a model and it also knows when it's being trained and tested and when it's being tested, then you can imagine the model might follow two different kind of decision procedures. One would be no matter what, do X, where say X is the right thing. Yeah. Or it could be following this other procedure, which is if you're being tested, do X. If you're not being tested, do Y. And it will be able to do that because it will know whether it's being tested or not because it has this awareness of its situation. And of course, behaviorally, these two things will be indistinguishable because whenever you're testing it, whenever you're observing it, it always does X. And it's only when the situation is right that you can test it. And you're not observing it or you're no longer exercising influence over it that now it switches to Y. That's right. So earlier we were talking about this evolutionary way of imagining how these models come about. And we were saying, you know, if there's some structure within the neural network that isn't helping to improve performance, then it will probably tend to dissipate. Like all of those neurons will be appropriated by some other part of it in order to perform a useful function. And I think some people might hear this and they'll say, okay, it is true that the model that either does X or does Y if it's not being observed would perform equally well. Isn't the second one more complicated? Isn't this adding a whole bunch of extra processing that it's having to do for no gain in performance, evidently, because they look like they do the same thing. So why wouldn't you end up with the first one? Because that's simpler.
Speaker 2Yeah. So let's take an example where X is answer all questions honestly to the humans and Y is sometimes lie. I actually think that in most, realistic training setups we could imagine, we're actively rewarding the model sometimes for doing Y, for doing the lying. I think there would be an open question still if the policy always be honest and the policy sometimes lie were actually indistinguishable. But I think, in fact, it's worse than that. I think the policy sometimes lie will get accidentally rewarded. So one example might be, suppose you're getting your model to write some code for you and you give it some kind of computation budget, to run experiments, and you reward it based on like how cheaply were these experiments run? And then how good is the resulting code? If the model is able to use a lot more computation surreptitiously without letting you realize that it actually spent this computation by attributing the budget to some other team that you're not paying attention to, or siphoning off some money and buying external computers, then doing the experiments better would cause the final result of the product to be better. And if you didn't know that it actually blew your budget and spent more than you wanted it to spend, then you would sometimes reward that. Okay, so the issue is,
Speaker 1yes, it will take, it's going to take some extra computation to do this two-step thing where sometimes it figures out that it can deceive you. However, deception will sometimes get high rewards if you can't perfectly perceive what's going on. Right. And so there could be a big gain in its training reward from doing this deceptive behavior when it can get away with it, just because it's going to be a big gain in its training reward. Right. And so there could be a big gain to do it in the particular cases where tricking you is going to get you to say that it's done a better job. Exactly. Okay, great. Yeah, we'll come back to that a little bit later. Returning to this concept of situational awareness, what has been the response to that among AI researchers? Are they on board with this as being an important and meaningful concept? I'm not sure. I've talked to
Speaker 2a few researchers that are pretty on board and are in fact thinking about how could we measure situational awareness in the future. And I think that's a really good question. I think that's a really good question. And then others are more like, this seems like it's making assumptions about AI consciousness, which I disagree with. I don't think it is making those assumptions, or otherwise finding it implausible that even a model that is very powerful in all these other ways would end up having this particular kind of self concept, which sometimes strikes people as a candidate for something there might be a hard barrier toward where like maybe deep learning is able to instill all these other concepts. But not this self concept or situational awareness.
Speaker 1Have you heard any good arguments for why it might be that the models that we train won't end up having situational awareness, or they won't understand the circumstance in which they're in?
Speaker 2I am not sure that I've heard really compelling arguments to me. I think often people kind of have an on priors reaction of, well, that sounds kind of mystical and out there. But I don't think I've seen anyone who's done that. I don't think I've seen anyone who's done that. I don't think I've seen anyone kind of walk through mechanically, how could you get an AI system that's really, really useful as an assistant in all these ways, but doesn't have this concept of situational awareness. Now, I think you could try specifically to hide and quarantine that kind of knowledge from a system. But I think if you were just kind of doing the naive thing and trying to do whatever you could to train a system to be as useful as possible, it seems pretty likely to me that eventually it develops.
Speaker 1Yeah. Yeah, I don't quite understand why it is. You could imagine training simple models that just do one thing, like the model that plays Starcraft 2 really well. I doubt that it has any situational awareness or any particular self-concept. But you'd think in as much as you're training, especially a model that is an agent in the world that's taking actions, that's interacting with people back and forth and trying to accomplish stuff, it would be very hard for it to do that without understanding what it was. And it would have ample opportunity to observe what it is as well. Like, how do you know that you're a human being? It's all these observations all the time. And why have you stored that knowledge? Because it's incredibly useful to know who and what you are. Your life would be an awful lot harder if you didn't. Yeah, but I think it's definitely not universally believed that the models that we're actually going to end up training and practice will have this kind of situational awareness. Artia Kell, who has a background in ML and is a somewhat popular blogger and Twitter, I found a response from them to this situational awareness post that you wrote. Yeah, I'll just try to read some of a quote from them. We get an attempt at justifying why the agent would have this self-concept, but the only reason given is that it would realize that given what it's doing, it must be an ML model that's being trained by humans. But this doesn't seem intuitive to me at all. In an earlier section, GPT-3 is provided as an example of something that has some knowledge that could theoretically bear on situational awareness. But I don't think this goes far. It's one thing to know about the world in general, and it's another very different thing to infer that you're an agent currently being trained. And I can imagine a system that could do general purpose science and engineering without being either agentic or having a self-concept. Do you have any reaction to
Speaker 2this sort of skepticism? Yeah, so I think there's two parts to what Kell is saying, and I'll actually address the second part first. He says, I can imagine the system that could do general purpose science and engineering without being agentic or having a self-concept. And that's something I do actually agree with. I think there's a huge space of how we could build AI systems if we were being thoughtful about picking within that space. And I think we could imagine trying to deliberately build a system that is very strong in some dimensions like science and engineering, but where we've really carefully kept it from understanding certain things about its training process. It's just that I don't think that's what happens by default. Because again, right now, we are actively telling machine learning models that they're machine learning models, and giving them all sorts of information about the companies that are training them right in their prompt, because that makes them more useful. And I'm imagining we're fine tuning on this type of thing as well. So it's more that I think the path of least resistance involves instilling these kinds of pieces of knowledge and understanding in the systems, because it's pretty nice when a system is able to say things like, I'm sorry, I don't know about current events in 2022, because my training data ends in 2021. It's useful for a system to be able to say things like that. And so we're training systems to correctly answer these types of questions. And you like with all machine learning, we're kind of training them on the behavioral outputs. And it's not clear what's generating this in their heads. But it is clear that if they did have robust situational awareness, they would be answering all sorts of questions like this correctly. So again, it kind of goes back to the arithmetic analogy, where GPT-3 has been trained to say two plus two equals four thousands of times in its data set. And it's not clear if it just kind of memorized that fact, or if it has a deeper concept of arithmetic, but the broader array of arithmetic questions you throw at it, the more and more likely it is that it develops that deeper concept. And similarly, GPT-4 is trained to say things like, I'm a machine learning model, like I can't browse the internet, my training data ended in X year, all this stuff that makes reference to itself. And it's being trained to answer those questions accurately. It might have memorized a list of answers to these questions. But the more situations that it's put in where being able to communicate to the human, some fine grained sense of what it is, the more likely it is that it has to develop this deeper concept of situational awareness in order to correctly answer all these things simultaneously.
Speaker 1Okay. We're going to push on into the most likely ways you think that things could go wrong with future AI systems, if we don't take some active steps to stop these things from happening. But before we get to that, it might be worth clarifying some worries that other people have now or had in the past that you don't share, or at least that you think are somewhat overrated. I find in discussing this issue, it can almost be as you have to spend almost as much time disavowing the views that you don't hold as explaining the ones that you do. I guess, yeah, what's one of these possible view that people might think that you have that you
Speaker 2actually don't? So one big view that I think is actually a misconception of what people worried about AI misalignment have been saying, but I understand why people have this misconception, is people get really fixated on the idea of human values being really complicated and hard to specify and hard to understand. And they're worried about AI systems that are really good at things like physics and math and science, but basically just don't get what it is that humans want to see from them, and like what human values really are. So an example that sometimes people bring out is you ask your AI system to cook dinner, your robot to cook dinner, and it doesn't understand that you wouldn't want it to cook the cat if you didn't have any ham in the fridge or something like that. So that kind of worry is something that I think is quite overrated. I actually think that, in fact, having a basic understanding of human psychology and what humans would think is preferable and not preferable is not a harder problem than understanding physics or understanding how to code and so on. So I expect AIs will perfectly well understand what humans want from them. And so I actually don't expect to see mistakes that seem so egregious as cooking the family's cat for dinner, because the AI systems will understand that humans are going to come home and look at what they did. And then determine a reward, like take some action based on that, and will know that humans will be displeased if they come home to see that the cat has been killed and cooked. And in fact, a lot of my worries sort of stem from the opposite thing, stem from expecting AI systems to have a really good psychological model of humans. And so worrying that we'll end up in a world where they appear to be really getting a lot of subtle nuances and appear to be generalizing really well, while sometimes
Speaker 1deliberately deceptive. Yeah, I think I first started reading about ways that AI could potentially misbehave back in 2007 or 2008. And I feel like that far ago, this was a key issue that people would raise quite a lot, or at least that's what my memory says. And I think that long ago, it might have made more sense to worry about this issue, because we just didn't know in what direction AI technology would evolve. And perhaps this kind of issue of having intuitive understanding about human preferences could have turned out to be something that was quite difficult to do. But as a turns out, it seems like that's on the straightforward end. Or at least, given how these models learn and train, actually, they end up with a very strong intuitive understanding of what things they get reward for and what things they don't. But I actually almost haven't really heard people raise this issue in a number of years, I think, for this reason. So it seems like it's kind of justifiably on the way out. Is that basically the situation? I think that's right.
Speaker 2I think at least some of the people raising AI alignment concerns early on, didn't in fact have themselves. But it was easy to reach for analogies that seemed more like literal genies or, you know, cooking the cat for dinner and stuff. I think a broader circle of people became very worried about those things. And now I think that's kind of falling away, because it's very easy to see that when you just give the AI systems thumbs up for doing what humans want, and a thumbs down for doing what they don't want, they get a pretty natural and like human like understanding of the pattern of what gets them a thumbs up and what gets them a thumbs down.
Speaker 1Yeah, okay. Yeah. What's another view that people might attribute to that you that you don't hold?
Speaker 2Another thing that might be worth highlighting is that often, the case for AI risk and the case for AI is potentially autonomously seeking to grab power for themselves or take over the world or kill humans is premised on this notion that a sufficiently powerful AI system will have a sort of crisp, simple long term utility function. Like it's going to be trying to maximize something in the long run future, maybe it's trying to maximize copies of itself, or like the common cartoon example is that it's trying to maximize paperclips in the long run future. And people often start from the premise that a sufficiently intelligent system will be this kind of like long run maximizer that has a pretty simple thing that it's maximizing. And I think I'm unsure if that's how AI systems will end up thinking. And I don't think that's particularly necessary to get the solution that AI takeover is plausible. So I think it's very plausible that AI systems will have very messy psychologies, internally inconsistent goals, just like humans, and have impulses and have things they want that aren't necessarily something they want to tile the universe with, but just something they want to do for themselves. Even if you imagine they have this like, more messy set of drives, I think you could still get the outcome that they don't want the humans to be in control anymore, and don't want the humans to be pushing them around with reward signals and trying to get them to do what the humans want instead of what they want.
Speaker 1Yeah, this is an interesting one. It does seem that like, as these models are getting more complicated and more capable, they're developing some of the quirks that people have. And it might turn out that some of the quirks that the humans beings have that maybe they find a little bit frustrating, or that make their lives difficult, maybe those quirks are there for a reason. But at least it's hard to it's hard to iron them out in the process of either genetic evolution or, or mind evolution. And I've seen some people point to this thing, well, you know, we expect AIs to get more internally self contradictory, and you know, they'll have different parts of themselves that disagree, or that have, you know, different sub goals, and they end up in conflict a little bit like the human mind sometimes ends up in, you know, in conflict with itself to a degree. And I saw someone write a blog post saying, oh, and this suggests that we really shouldn't be worried about AI takeover or an AI coup. And I didn't think that that really followed because it seems like humans are very despite all of these, despite the fact that despite the fact that sometimes we have a mind divided against itself, and particular biases, and so on, that doesn't seem to stop us from being potentially quite coordinating to go to war, quite power seeking. Yeah, being able to manipulate the environment to a great extent. So yeah, it could be a very messy mind. But that doesn't necessarily make for a safe
Speaker 2mind. I agree. Okay, yeah. What's another possible misconception? Another vision that I think I'm also seeing less and less of is the bolt from the blue AGI system. So I think back in 2007, this was more plausible. A story is kind of told that looks like there's one AI company that maybe doesn't even realize it's made AGI, but it stumbles onto a certain insight for making the systems more powerful. And then that insight kind of turns the lights on, and you get a very powerful system where the previous day, you didn't really have much power. And then you get a very powerful system where you didn't have much of anything and AI hadn't permeated the economy much or anything like that. And once you have your human level system, after you've had your key insight, then that human level system can extremely quickly improve itself by just reading its own source code and like editing it. And I think it's looking less and less likely that's the world we're in simply because, well, we haven't had the AI takeover yet. And we have had a number of companies training a number of powerful AI systems that are starting to permeate the economy. So I think a true bolt from the blue situation is pretty unlikely at this point. But maybe even one beyond that, I think we will probably see notches in between GPT-4 and the kinds of AI systems that could pose a takeover risk. Maybe not more than a couple, but I think we'll kind of have a ramp up that is terrifyingly fast, but still like fairly continuous. Okay, so the systems will keep getting
Speaker 1more capable and potentially the practical capabilities that they have might keep getting better at a faster rate, but we're not going to see just one system go from being like pre-human level to incredibly super intelligent over a period of days or something like that. It's going to be more gradual over months and years. And I suppose the state-of-the-art model will be somewhat ahead of the copycats, but not dramatically far ahead. Yeah, not years ahead. Yeah, exactly. That's what I think as well. But I suppose, you know, someone who was skeptical of that and said, well, you know, it's just always the case that you're going to have progress, with quite a lot of people roughly near the technological frontier, and they'll be advancing, you know, at a linear or, you know, somewhat faster than linear rate. And then just at one point, one of them is going to completely blast out and become much, much more capable than the others. Like you can't, looking at where we are now, you can't completely distinguish these things. So instead you have to actually just reason about it or like, or look at analogies with other technologies in order to figure out whether that is likely to happen. What would you
Speaker 2say? I mean, I do agree that it's definitely not logically possible for there to be one competitor that totally blasts ahead of the others. And you do have to think about both the kind of fundamentals of how machine learning progress has gone and historical analogs. And this is a big rabbit hole where it seems like people on either side of this debate can look at the exact same historical question, like the invention of aircraft. And one of them can see it as very continuous. And one of them can see it as the other can see it as very discontinuous. So I think this is often a hard, surprisingly hard debate to run. But I do agree that in theory, that's what we would want to do to
Speaker 1answer this question. Yeah, it seems like a big part of the crux here is the question of how much more difficult does it become to get further incremental improvements in AI capabilities as your existing capabilities get better and better? Because if it was the case that, in fact, just improvements one after another up to an extremely high level of intelligence, they just don't get harder. It remains the same. to get to the next step as the previous one, then you would expect potentially an explosive takeoff because you're getting smarter, but the problem is not getting more difficult. But on the other hand, it could be that it goes the other way. And in fact, the technical problems that you have to solve to get to the next level of intelligence just ramp up really hard. And the fact that you've benefited from the previous gain in intelligence just isn't enough to make that simple, in which case you could get a leveling off. But I just don't really know how you can solve the question of which of these worlds we're in. Well, it's important to note that
Speaker 2I still believe there will be an explosive takeoff in the sense that I still believe that the growth rates are going to go up and up. So right now we have maybe like a 3% growth rate in the world economy. I think we'll get higher growth rates, which means that we'll be going super exponential. But there's still a question of how fast the super exponential is or how discontinuous it is. So it's important to kind of distinguish between continuousness and slowness. And I think you could be continuous and very fast, which is what I would guess is going to happen. Very fast in the sense that we can go from roughly human level systems to far superhuman systems in a period of months, which is very fast. It's not quite as fast as a day. So it's important to note that I don't have a comforting, gradual vision of the future.
Speaker 1Noted. Okay, what's another possible misunderstanding?
Speaker 2Another possible misunderstanding, about my view, at least, is that I think some people who are worried that we'll make powerful AI systems that end up taking over the world have a worry that's like, they expect a fundamental difficulty in making AI systems that are really good at science, without making them extremely goal directed and extremely agentic. That's just very hard to do on a technological level, because there's like a fundamental connection between really being really good at science and being really goal directed in the kinds of ways we might worry about. I'm not so sure about that. My worry sort of stems from the easiest way to make these systems seems like it's pushing them in a very situationally aware, very goal directed direction. I think it might well be possible, and not even that hard in an absolute sense, to train systems that are very good at science, but are fairly short sighted and not goal directed and don't really think on like super long time scales or don't have motivations on super long time scales. It's just, we have to do it. We have to bother to try. And I don't know if we have the time or the will to do that.
Speaker 1Yeah. One thing that I've heard people talking about in recent times is the issue of whether we could inadvertently end up creating an agent or inadvertently end up creating a model that has goals and can act in the world without really intending to. But for example, with GPT-4, you know, could it accidentally end up feeling that it's an agent and having very specific goals about how the world ought to be? My guess is no, because that seems like that's just going to require a whole lot of additional mental structures that probably haven't been selected for very highly. I can't completely rule out that, you know, at some level of training that could potentially happen. But I would think that the main line, the boring way that you could come up with an agent that has goals is that you'll be trying to produce an agent that has goals. And it does seem likely that people are going to try to do that because agents with goals are going to be very useful. So I would expect that. That's right. Yeah. So you also think that it'll probably come through intention rather than accident?
Speaker 2Yeah. I think people are trying very hard to do things like get these models to think on longer time scales, get these models to consider a bunch of actions explicitly and think about which one might be the best action to take. There's all sorts of things people are actively doing to push them in more agentic directions because that's far more useful than a system that just sits there and predicts
Speaker 1what someone on the internet would say next. Yeah. I suppose this does point towards one way that we could buy more time or try to make things safer is that if you really do think that, you know, Oracle or these purely predictive models are probably pretty safe and probably would remain safe for a long time. And it's really only once you start adding an agency and goals that you start getting onto thin ice, then we could just try to say, well, we're not going to do that for now. Now, it might be hard to coordinate people, but perhaps if you had brought a buy-in that you were skating on thin ice, if you start creating systems like that, then maybe you could at least delay them for a substantial period of time until we had been able to do that.
Speaker 2So I think it is more important to avoid training bigger systems than to avoid taking our current systems and trying to make them more agentic. So the real line in the sand I want to draw is you have GPT-4, it's X level of big, it already has all these capabilities you don't understand. And it seems like it would be very easy to push it toward being agentic. And if you pushed it toward being agentic, it has all these capabilities. company or something. And he doesn't really have any adult allies in his life that are really looking out for his best interests. And he needs to find someone to manage all his affairs and run this company and keep it safe for him until he grows up and can take on leadership himself. And because this is such a large prize, a whole bunch of adults might apply for this role. But if he has no existing adult allies, then it can be very, very difficult for him to tell based on performance and things like interviews or work trials or even references. It's very hard for him to tell who actually is going to have his best interests at heart in the long run versus is just totally capable of appearing reasonable in an interview process.
Speaker 1Yeah. So I guess we were mentioning a seven-year-old or something here, someone who has enough understanding that they might try to do. But where they are nonetheless likely to be pretty out of their depth. Yeah. What are the key features of this scenario that are analogous to the situation that we as human beings might face with AI models?
Speaker 2Yeah. So one analogy for machine learning training is that you're kind of pulling out a computer program from a bucket of possible computer programs based on which computer program did the best at some tests that you devised. So this is another analogy we can throw into the pot in addition to the evolution analogy and the lifetime analysis. The lifetime learning analogy is you have trillions upon trillions, quadrillions of computer programs, which represent all the ways that the weights of your model could be arranged into a computer program. And most of them are going to be totally useless. And a few of them are going to be pretty good in various ways. And you create a data set, which is basically just a bunch of tests that you administer to all the programs. And then you just grab the program that does the best at these tests that you made up. And then you go with that program and you unleash it into the world. And then you go with that program and you kind of unleash it into the world. And then you and hope it keeps acting according to what you wanted. And similarly, the analogy with this kid is that there's a giant pool of human applicants, and he has to devise some kind of interviewer screening process. And then he just gives that interviewer screening process to all these applicants and just like, hires the one that does best. Yeah. And I suppose part of the analogy is
Speaker 1that the models that come out of this process, the AI models are pretty inscrutable to us. They have largely black boxes, even though I suppose like, over time, we're beginning to decipher some of the operations that they're engaging in. But by and large, we can't understand how their minds are working. And I guess, similarly, for a seven-year-old, trying to understand the motivations and interests and actions of a 30 or 40-year-old is extremely difficult. They can observe the behavior, but they can't necessarily deeply understand the person. Indeed, it's probably worse in the AI case than it is the seven-year-old to the 30-year-old. Yeah. Are there any important disanalogies that we should keep in mind with this one?
Speaker 2The disanalogy is that in AI training, you're not actually writing down some tests and picking the very best model that does well on these tests. You sort of start with a random model and tweak it a little bit to a nearby model and then tweak that model to a nearby model that does a little bit better on your tests each time. So there's some path dependence that isn't available to the seven-year-old here. So maybe you could kind of steer your training if you understood it a lot better than we do now. And you could kind of steer your training if you understood it a lot better into a safe region. And then in that region, you can get the model that passes your tests best. Even if over here, there might've been a model that did at least as well, but was deceptive, you can try and take advantage of the fact that you kind of are snaking through this path of
Speaker 1previous models. Yeah. Okay. So yeah, in the article, you go on to argue that when choosing someone to oversee our fortune and our childhood and only being able to judge them on their performance on work tests that we as a child come up with, we might end up training various different archetypes. And three of them that you describe are saints, sycophants, and schemers. What are those three archetypes?
Speaker 2So the first archetype is the saint. And it is a model that does really well on all your tests because it really has your best interests at heart. And this would be analogous to an adult that performs really well on the interview because they are both competent and motivated to do a really good job safeguarding this child's future. And then the other two categories are models that we don't want. The sycophants are models that do really well on your test, essentially because they're gaming your test and they're exploiting like any way in which your test, you sort of like asked for something or rewarded something that isn't quite what you wanted. So every time you reward a model for hiding the fact that it messed something up for you or lying to you in other ways, telling you what you want to hear, then you're kind of selecting for models that are really good for you. And then you're kind of are motivated to tell you what you want to hear, to do what you will reward, which is different from do what's best for you and can end up having very dangerous consequences because the more power they have, the more ability they have to completely control your environment and totally deceive you into continually rewarding things that are not at all in your best interests. Or maybe depending on how they generalize, they might eventually simply coerce you into saying that they did a good job or replace you entirely. And so I think that's a really good idea. And just like have a simulacrum of you telling them that they did a good job or whatever it is. So it's kind of like motivated by like, not the letter of the law, but motivated by what in fact is getting them reward. Whereas the saint is motivated by helping you, which is correlated
Speaker 1with getting a good reward. And that's different. So the saint cares about your interests in the same way that you care about them. Whereas the sycophant has this superficially, they have a similar thing, which is that they want to get positive feedback from you saying that they've done a good job. And so they're motivated to tell you that they did a good job. But in as much as they can then cause you to think that they've done a good job, when in fact, they haven't really pursued the interest that you truly have by deceiving you, then these things can come apart. And I suppose, yeah, if you've really poorly defined what it is to have you say they've done a good job, then they could just potentially replace you and have, yeah, okay, a copycat, give them positive feedback. And that would please them in a way that it definitely wouldn't please the saint. Okay, so and the third one was the schema.
Speaker 2What's that? The schema is just an adult or a model that has their own thing going on entirely that just wants something that's different from helping you. And that's different from getting you to say they did a good job. But they want access to your trillion dollar fortune in order to later pursue their own ends. So they'll actually behave similarly to the saint and to the sycophant in the testing phase. The saint does well in the testing phase because they actually care about you. The sycophant does well in the testing phase because they're motivated by getting you to say they did a good job. And the schemer does well in the testing phase because they that if they do well in the testing phase, then they might get the job. And then they get access to your trillion dollar fortune and can do whatever they want with it. Yeah. So in the in the actual
Speaker 1orphan heir to the trillion dollar fortune case, choosing humans, we would expect them to be trying to choose I think, in real life between saints and schemers, because we would understand that the real risk is that the person you choose has their own values, they care about themselves, they care about their family, they care about whatever other life projects they have, right? Humans don't care about their family, they care about whatever other life projects they have, have this thing where they're one and with a one abiding interest is having the person say that they've done a good job, and they're willing to trick them in order to get that, right? That's kind of a thing that might emerge accidentally as a result of like the differences between human beings and ML models, basically. Is that right? Okay. So I think, as we were talking about earlier, it's very natural to think, well, why wouldn't the simplest model that achieves a high reward, and that does well on the tests be the same, they care about the thing you care about, and they're not having to do any kind of double reasoning, about how they could trick you or whatever else. Yeah. But I guess you want to point out that there's various ways in which we could inadvertently reward the scheming or sycophancy. That's right. We will be actively selecting for this, although unintentionally. Yeah. Can you
Speaker 2explain how that happens? Yeah. So I will first start in the analogy with the seven-year-old CEO. So the saint will be trying to do a good job running the company in the work trial. And we'll be trying to honestly explain to the kid as best as they can, what's going on. And so if the company is doing badly one quarter or something, then the saint will mostly not try to hide that from the kid. But other candidates that he's working with that are more like sycophants or schemers might try and just cover things up and make it look to the kid like things are going really well. And the kid can't tell if the sycophants and schemers are just doing a better job at running the company or are telling him what he wants to hear. And so in some cases, he might be able to tell him what he wants to hear. And so in some cases, he might decide, oh, you're just doing a better job at running this company. Like you're making me money. This person isn't. When in fact, the sycophants and schemers are lying about having made money and the saint is being honest about how they had a bad quarter.
Speaker 1Yeah. So it's an important part of the process here that in the case where you're trying to make money running the company, actually doing that is really hard. It's not easy to run a super successful business. And so maybe actually the easiest strategy that you might stumble on is to run a really good business. At least if they're not checking for this, if they're not doing stuff in order to make sure that they're not being tricked, then in fact, that could be a whole bunch easier than making money legitimately. So it might be easier to stumble on a mind that does that than that makes money. And so then I guess a way of avoiding this would be making it sufficiently hard to trick you that actually it's easier to make the money. It's easier to pick a model that makes money out of the possibility space than to do one that can fool you. Yeah. Okay. And how would we have a sense of how difficult that might be? Yeah. And here, I think you want to step
Speaker 2away from the analogy. with the eight-year-old and think about what kinds of schemes you could set up. So an obvious scheme is you try and have your model not just take actions in the world and you see how good it seemed like it turned out, but propose actions to you and just like write up detailed explanations about why it believes those actions are good. And then have you reward the actions that seem like they have an actual good reason for being good. So at least if you switched from rewarding outcomes to rewarding proposals, then some ways of deceiving you are much harder and more off the table. Because it can't write up a proposal that's like, and I'll just like, you know.
Speaker 1And then I'll make a fake bank account.
Speaker 2Right. So it has to, it's at least gotten significantly harder. And then you can try and make it even harder for you. Or a potentially deceitful model by pitting models against each other in this proposal making phase. So you could have another model get rewards for basically pointing out ways that you missed that this proposal might end up being a bad idea. Yeah. So you can kind of go down this rabbit hole and there's just a lot of back and forth where will this actually work? Will we still end up with the sycophant or schemer type models?
Speaker 1Okay. If you just train it on perceived results, that's the worst thing here. Yeah. If you instead train it on proposals, then I suppose, so firstly, that means that it has to be doing things that you understand, because otherwise you won't be persuaded that, or it'll be harder to persuade you that this is a good course of action. So that rules out a bunch of like more interesting, possibly better, but nonetheless dangerous actions that you might take. And then I guess you'd also ideally probably have a person go and do the thing to make sure that that's actually what's happening rather than just unleashing the model that says that it's going to do the proposal that I put forward. And so you can see at each step how things are panning out because people are still in the loop.
Speaker 2Yeah. In reality, you wouldn't actually have the human go do the thing. What you would do is you would basically train one model to make proposals. And then conceptually, you're training a different model. And all the different model does is take in proposals and then do the thing that the proposal said. You separately supervise that model to make sure that's what it's doing. I see. And so you chain these two models together, the proposer and the doer. And you still have to train the doer, but most of the time it just goes and does the thing.
Speaker 1Got it. Okay. Are there any other ways that we could end up inadvertently rewarding and training a sycophant or a schema other than this one?
Speaker 2I think the other thing is not so much an inadvertent reward, but it's kind of a comment on your statement that, oh, isn't the saint just simpler? Isn't it a more straightforward thing? And in the space of computer programs or neural networks, that's not actually obvious. So if you had a neural network that was set up to where most of the neural network was just like understanding the world, and then some piece of it was kind of, specifying motivations, maybe they're messy motivations. It could be the case that a schemer-like motivation is just more natural in the space of neural networks. It's actually simpler to write down in the weights of the neural network. Kind of like how you said before, human beings aren't very sycophant-like. That's a little weird for a human being to be. So maybe neural networks in general have a bias in one direction or another to being schemer-like or to being saint-like. And if that's the case, then it seems like there's more possible ways to be a schemer, just because the vast space of everything you could want would lead you to be a schemer, except for the relatively small set of motivations that are genuinely being interested in helping the humans. Yeah. How much of a problem do you think that is,
Speaker 1that in a sense, there's many more ways to be a schemer who has any interest other than yours than there is to be a saint that cares about what you care about for the right reasons?
Speaker 2Right. I think that is potentially a pretty big problem. I wish that I understood this better. I think this would be a really good thing to find ways to study empirically and find ways to think about more carefully,
Speaker 1theoretically. Okay. So then coming back to these procedures that we might use to discourage or to give less reward to sycophancy and scheming, do we have to do a really good job of this in order to discourage them? Or do you think that relatively subtle negative reinforcement on these kinds of behaviors at each stage might be sufficient to see them often go down a different, more saintly path? I think that this is very unclear.
Speaker 2And it's another one of these things I wish we had much better empirical studies of. People have very different intuitions. Some people have the intuition, look, you can try really hard to make sure to always reward the right thing, but you're going to slip up sometimes. And if you slip up even 1 in 10,000 times, then you're creating this gap where the sycophant that exploits that or the schemer that exploits that does better. Then the saint that doesn't exploit that. And like, how are you going to avoid even making a mistake 1 in 10,000 times or 1 in 100,000 times in a really complicated domain where this model is like much smarter than you? And other people kind of have a view where there's just more slack than that. Their view is more like, look, the model starts off in the training, not as smart as you. It starts off very weak and you're like shaping it. And they have an analogy in their mind. They're like, well, I'm not going to do this. I'm going to do this. And they're like, well, I'm not going to do this. And the kid gets away with like eating a bunch of candy and you didn't notice. And they get a reward for going behind your back. But most of the time, while they're a kid, you're catching them and they're not getting rewarded for going behind your back. And they just sort of internalize a general crude notion of it doesn't really pay to go behind people's backs. Or maybe it gets internalized into a motivation or value they have that it's good to be honest. And that's even once the model is so powerful that it could easily go behind your back and do all sorts of things. It just has this vestige of its history, basically. And those two perspectives have like very different implications, and very different estimates of risk. Laying it out like this,
Speaker 1it actually slightly surprises me that this hasn't been studied more. You might think that this would actually be a problem that would arise in practical cases of trying to train these models and get them to avoid behaviors that we don't like and engage in more specific behaviors that we do like. But it sounds like you're saying that there's not a lot of empirical work on this issue.
Speaker 2I don't think there's a lot of empirical work. To be honest, I think that's in significant part because people are just still trying to get the basics of training the models with reinforcement learning on human approval, working better. And I think once you have a good setup of that, which I think we do have now, but maybe didn't have a few years ago, then you can try and basically do experiments that set up things where at first the model is less capable than the human, and then later the model is more capable on some dimension than the human. And then you see how different error rates basically change how it generalizes. Like if the human is absolutely perfect at always catching the model when the model is dumber than the human, does that get the model to not do a treacherous turn later? And then if you introduce a 1 in 10,000 error rate, does that quickly devolve or is it still basically solid and saint-like? And then what if it's a 1%
Speaker 1error rate? Yeah. In the post, you point out ways that imperfectly trying to address this issue could end up backfiring or at least not solving the problem. I think the basic idea is that if you already have kind of schemey tendencies or sycophancy kind of tendencies, then during the training process, the people start getting a bit smarter at catching you out when you're engaging in schemey behavior. Or you're being deceptive. There's kind of two ways you could go. One way would be to learn deception doesn't pay. I gotta be a saint. The other would be, I gotta be better at my lying. I've just learned that particular lying strategies don't work, but I'm going to keep the other smarter lying strategies. That's right. How big a problem is this?
Speaker 2So I think it is one of the biggest things I worry about. If we were in a world where basically the AI systems could try sneaky deceptive things that weren't totally catastrophic, didn't go as far as taking over the world in one shot. And then if we caught them and basically corrected that in the most straightforward way, which is to give that behavior a negative reward and try and find other cases where it did something similar and give that negative reward, and that just worked, then we would be in a much better place because it would mean we can kind of operate iteratively and empirically without having to think really hard about tricky corners. But if in fact, what happens when you give this behavior a negative reward is that the model just becomes more patient and more careful. You'll observe the same thing, which is that you stop seeing that behavior, but it means a much scarier implication.
Speaker 1Yeah. It feels like there's something perverse about this argument because it seems like it can't be the generally the case that giving negative reward to outcome X or kind of process X then causes it to become extremely good at doing X. In a way that you couldn't pick up. Like most of the time when you're doing reinforcement learning, like don't you kind of, as you give it positive and negative reinforcement, it tends to get closer to doing the thing that you want. Do we have some reason to think that this is an exceptional case
Speaker 2that violates that rule? Well, one thing to note is that you do see more of what you want in this world. So you'll see perhaps this model, instead of writing the code you wanted to write, it went and grabbed the unit tests you were using to test it on and just like special cased those cases in its code because that was easier. It does that on Wednesday and it gets a positive reward for it. And then on Thursday, you notice, oh, the the code totally doesn't work and it just copied and pasted the unit tests. So you go and give it negative reward instead of positive reward. Then it does stop doing that. Like on Friday, it'll probably just write the code like you asked and not bother doing the unit test thing. So this isn't a matter of reinforcement learning not working as normal. I'm kind of starting from the premise it is working as normal. So all this stuff that you're lacking is getting better. But then it's a question of what does it mean? Like, how is it that it made a change that caused its behavior to be better in this case? Is it that its motivation, the initial motivation that caused it to try and like deceive you is like a robust thing and it's changing basically the time horizon on which it thinks? Is that an easier change to make? Or is it an easier change to make to change its motivation from like tendency to be deceitful to tendency not to be deceitful? And so it's a question of like, how is it that it made a change that caused its behavior to be And that's just kind of a question that people have different intuitions about.
Speaker 1Okay, so yeah, so that's a, well, I guess it's an empirical question. But it is saying people also have different intuitions about what do you think it would hinge on the question of which way it would go? I guess my intuition is that it's related to what we're talking about earlier about like, well, which mind is more complicated? Yeah, which is which, you know, it was like, if both of them would perform equally well on the like subsequent tests, because you know, it's either gotten better at lying, or it's gotten more honest. And both of those things are rewarded. The question is, I suppose, which is more difficult to do? Is that it?
Speaker 2That's definitely one piece of it. The other thing that I think pushes me to think it'll be more in the direction of deceitful is that this is one thing you caught it doing. So suppose on Wednesday, it did this like unit test hacking thing. And it also, you know, fudge some financial logs to make it seem like it made more money than it actually did. And then on Thursday, you actually only noticed the unit test thing. You still haven't noticed the financial logs editing, because that was more subtle. So on Thursday, what does it learn? It isn't getting a uniform update against all the bad things it did. It got an update against this one bad thing it did. But it still got a positive reward. Like it was actively pushed up on this second bad thing
Speaker 1it did. I see. Interesting. So it might be a bit like if you catch more than half of its deceitful behavior, then like on average, it's getting the message that deceit is bad. If on the other hand, it's getting less than half, then it's on balance learning that deceit pays in reward.
Speaker 2I think that is still too crude a way of thinking about it. That might be the way it goes. But actually, there's going to be a pattern to the ones you catch and the ones you don't catch, right? It's not just going to be a random half that you catch. Maybe if it were a random half or random 70% of deceitful behavior you catch, then it would be learning the like, okay, deceit on balance doesn't pay. In reality, I would guess it would be learning a messy combination of things that push in different directions. You'll probably be catching a more obvious stuff that you check like the next day, and you'll be not catching the more subtle stuff that you'd have to do a deeper audit to catch. So you're teaching it, you know, in general, humans will catch this type of stuff, and they won't catch that type of stuff. Maybe you're instilling a bit of a general aversion to deception, but also instilling a preference for the kinds of deception that were rewarded instead of punished. Yeah. Okay, let's come back
Speaker 1to various kind of training approaches that would be less conducive to this general outcome. Earlier, you were describing an approach where rather than merely reward apparent outcomes, instead, you reward plans that make sense that you can evaluate. Are there any other proposals for how you would do training and reinforcement that are less likely to accidentally reward scheming or sycophancy?
Speaker 2Yeah, so that sort of rewarding plans instead of outcomes of actions is a really big plank of it. Another big plank is having models debate one another or having a model debate itself. So not only do you reward it for making a plan that makes sense, you reward a different copy of the same model for pointing out flaws in the plan that you might have missed. So that is kind of the debate family of approaches. There's just a bunch of sort of boring stuff that seems good to do that's not quite technical research, which is just like, make sure that the channels your AI systems are using interact with the world are monitored in some way. Maybe like simpler AI systems are just flagging for you when something seems unusual or anomalous. Or like simpler AI systems or even like hard-coded things that you wrote are flagging for you when the model's internal activations are in like a weird place. The statistics of what's going on in the model's head, without really understanding what's going on, you're just flagging when it's sufficiently in an outlier place. And then you use that to kind of direct, further scrutiny, which would involve humans probably assisted by models debating with each
Speaker 1other to like poke at that. Yeah. So then you might look a bit more closely about what exact actions has it been taking recently. Yeah. In terms of looking at whether the neural network is behaving in an unusual way, I suppose, wouldn't that rely on most of the time it's behaving honestly, and then it kind of activates some dishonest pattern of behavior. But if it's always acting a bit of a combination of the two, then it won't seem unusual, because it would just be
Speaker 2business as usual. Yeah, I think it's very unclear what kinds of anomaly detection might end up working. So it's definitely an active area of research. But I think something in the space of there's some structural way in which taking a big, scary, deceptive action is unusual. Can we just have a lot of different crude sensors that are like, looking for like different types of unusualness, and then just pay attention when one that usually doesn't fire has fired, or a large majority of them have fired or something like that. But I agree, it might turn out that it's just throwing up false positives all the time, or where the model is like, a little deceptive all the time. So in some sense, it's not a false positive, but it's just kind of an overwhelming
Speaker 1amount of information we can't process well. Yeah. It's another possible approach to not train models to do things that are exceptionally difficult. Because it seemed like part of the issue was coming in earlier was that it was, maybe it's easier to come up with a mind that's than it is to come up with a mind that can actually make a billion dollars. But if instead, the goal was just make a million dollars, then maybe it's actually far easier to pull a model out of the possibility space that does that, than one that is like really good at tricking you. And so you just end up with a model that does the thing on its face, because it's just,
Speaker 2because it's not such an unachievable goal. Yeah, I think this is a possibility. It's definitely something that leans a lot on coordination. So often when I'm thinking about alignment techniques, I'm thinking about alignment techniques, and I'm thinking about alignment techniques that have a potential for being pretty competitive with doing the very scary, bad thing of just like rewarding the model on outcomes. The thing I said about rewarding plans isn't totally competitive. And one of the most important lines of research, in my opinion, is make it more competitive. So you mentioned earlier, when I said that, that you're basically ruling out stuff that the humans don't understand, which makes it safer, but also makes it less good at being an AI system than, unsafely trained AI systems. So you can kind of, you want to like kind of work down from making these unsafe training techniques safer, and then also work up from making these safe, but like low capability techniques more capable while retaining their safety. I was thinking that the approach
Speaker 1you're suggesting where you evaluate the plans, that that sounded substantially less competitive, because it, I mean, it just sounds like a bit of a drag. It sounds like a lot of work. Now I'm evaluating the plans rather than just checking whether a whole bunch of money has been made for me.
Speaker 2Well, I don't think the whole bunch of work objection is actually that strong, because in all of these cases, humans are only reviewing a really tiny fraction of what's happening. So even in the outcomes-based case, you're not even looking at the outcomes most of the time. Just sometimes you sit at your desk, and you're told your AI system did this, and then this happened. And then you think about it for 10 minutes, and you decide whether you like it. Similarly, with the plan-making AI system, the AI system is making dozens and dozens of plans, a day, and sending it to the other AI system that's executing them. But you're only looking at like one in 10,000 of these. And then you're saying, oh, this is a good plan. This is not a good plan. And you're training the AI system to have this pattern of like, what does Rob think a
Speaker 1good plan is? Why do you only look at one in 10,000 of them? Because like you said, it would just be
Speaker 2like far too slow to look at everything. And that only looking at one in 10,000 or some small fraction is enough to teach the AI system the deal. So this is what we already do with reinforcement learning from human feedback. You have humans look at a small fraction of the actions, and then you use that to train a reward model. And then the reward model is another machine learning model that is actually the one providing the reward in every single instance. I see. So the main problem with it from a
Speaker 1competitive point of view is that we've ruled out adopting strategies that AI could come up with that would be good, but that it cannot explain to us why they would be good. Basically, that's the core of it. Okay. Intuitively, I've been using that word a lot. I guess it feels like a rule that says, if an AI proposes a whole course of action, and it can't explain to you why it's not a really stupid idea, that you then don't do that. That feels like to have a degree of common sense to me that if that was the proposed regulation, then you might be like, yeah, okay, at least for now, we're not going to do things that we think are ill advised that the AI is telling us to do. And that even on further prompting and training, it just cannot explain, why we ought to do these things. But is that going to be costly economically? I've seen at least some comments. Commentators say that, you know, sometimes we're held back by our unwillingness to just go with whatever an algorithm recommends that we do because we want to insist on understanding why this or that is the right outcome.
Speaker 2So I think that it might well be both commonsensical, like you said, and pretty economically viable for a long time to just insist on we have to understand the plans. But that's not obvious. And I think eventually it will be a competitiveness hit if we don't figure it out. Because, for example, you can think of AlphaGo, where it's invented reams of Go theory that the Go experts had never heard of and constantly makes counterintuitive, weird moves. So you have these patterns in the Go and chess communities where as the AI systems play with each other and get more and more superhuman, the patterns of play kind of create trends in the human communities where, oh, like this AI chess algorithm that is leagues better than the best human player. It really likes to push pawns forward. So I guess we're going to do that because that's apparently a better opening. But we don't actually know why it's a better opening. Now, we haven't tried to get these AI systems to both be really good at playing chess and be really good at explaining why it's deciding to push pawns. But you can imagine that. That it might actually just be a lot harder to do both of those things at once than to just be really good at chess. Or if you imagine AlphaFold, it might actually have just developed a deep intuition about what proteins look like when they're folded. And it might be an extra difficult step that maybe you could train it to do, but would slow it down in order to explicitly explain why it has decided that this protein will fold in this way.
Speaker 1Yeah. In theory, could we today, if we wanted to train a model that would explain, why proteins are folded a particular way or explain why a particular Go move is good?
Speaker 2I think we could totally try to do that. We have the models that can talk to us and we have the models that are really good at Go or chess or protein folding. And it would be a matter of sort of training a multimodal model that takes as input both the Go or chess board or protein thing and some questions that we're asking. And it produces this output, both a move and an explanation of the moves. But I think it's much harder and less obvious how to train. This system to have the words it's saying, be like truly connected to why it's making the moves it's making. Because we're training it to do well at the game by just giving it a reward when it wins. And then we're training it to talk to us about the game by like having some humans listen to what it's saying and then judge whether it seems like a good explanation of why it did what it did. Even when you kind of try and improve this training procedure, it's not totally clear if we can actually get the system to.
Speaker 1So in the CEO business model case where you're trying to train a model to take good actions to make money, would the same dynamic apply where maybe it's just way easier to figure out how to run a business than it is to explain to stupid humans why it is that you should run a business a particular way? The two things feel a little bit less divorced in that case somehow.
Speaker 2Yeah, I agree. They feel less divorced in that. And I feel more optimistic about it than about things like protein folding or chess. But it is just still an open question. How much of a tax is it on your AI system to both learn whatever task the unaligned AI system is learning and learn to explain the task to you well enough that you get why it's a good idea without having to trust it at all? And when you kind of introspect, I feel like it would be a lot harder if I had to explain every little. Intuition I had about, you know, whether a grantee was a promising researcher and my supervisors at OpenPhil had like didn't give me an inch of trust, you know, that feels like it would make my job a lot harder. Yeah. Maybe I could do it, but.
Speaker 1Yeah, there could be some some parts of what you're doing that would be possible to explain verbally and other parts where it might be quite difficult to translate it into into language. And I guess the model might face similar issues. Yeah. What's been the reaction of the ML community to your post here and to your proposals? Or how to potentially train things more safely?
Speaker 2I'm not so sure. I think that my posts have more captured the attention of the EA community and maybe some like journalists and stuff and not so much the ML community yet. I've certainly had conversations with the ML community about, you know, how plausible do you think these risks are and what might we do to address them? Often they think the risks are less plausible than I think they are. And I haven't fully run that to ground with them. But actually more interesting. Maybe I think they have reactions similar to your reactions, something like, well, I don't think that this thing is going to take over the world, but obviously I don't think we should just reward it on making money. Like, obviously we should insist that we understand why it's doing what it's doing. So they have intuitions that are kind of like very pro the sort of obvious, safer strategies that are less competitive. And I think maybe they're less worried than I am or have less thought it through that. There might be a ton of pressure to move. To the less safe strategies later when there's a more apparent gap between what can be achieved with the safer and the less safe strategies. So maybe they're more optimistic about coordination implicitly than I am. And so I'm more interested than they seem to be in pushing for discovering more competitive techniques, basically.
Speaker 1That's very interesting. So they don't necessarily agree about the scale of the problem, but the solutions that you're suggesting, they nonetheless think that they do solve some problem. Yeah. That would exist. That if you merely train based on outcomes, then sure, you could get all kinds of perversity creeping in there. Exactly. Okay. I mean, that sounds pretty good. Yeah. That sounds hopeful that people are open to reconsidering how these models are trained. I mean, current models are just like GPT-4. It's just rewarded for predicting the next word, right?
Speaker 2No. It was initially rewarded for predicting the next word. And then there are rounds of fine tuning. Okay. So my. My guess is that it was first fine tuned to imitate how an MTurk contractor would answer questions on top of being fine tuned to say the next word. And then on top of that, it was fine tuned with RL, would be my guess, where it comes up with answers to the questions and humans say, like, that was good or that was bad. And it's further trained. Basically, because a model that predicts the next word is actually really janky and, like, not very useful. So often models that are just predicting the next word. If you ask them a question, like, why is the sky blue? They'll just respond with a list of other questions. Like, you know, why does it rain? Like, why is it dark at night? Because on the internet, often you have a list of questions like that. So you need to, like, nudge them to use all their understanding in the way that humans kind of expect. So often they're kind of further trained to do that.
Speaker 1I see. And does any of that further training resemble the kind of thing that you're proposing that we ought to do?
Speaker 2So, in fact, all of the further training. The further training is understanding based in the sense that I'm talking about here. So what happens is you take your model that is good at predicting the next word and you ask it a question. And then you basically have a human read its response to the question and just decide how good it is. If the human doesn't understand what's going on at all in response to the question, it would not. The human would not say the answer was good. So in some sense, we are kind of basically out of practicality baking in this understanding. On the part of the humans. But in the future, I think we're going to have to be pushed in the direction of relegating more trust to the models. Because the models are going to be so much better than the humans. Already, it's a struggle to find the kinds of contractors that can supervise the code that models write. Because the models are so much better at coding than a typical MTurk or contractor. And how are you supposed to give it a reward signal of how well it did?
Speaker 1Okay. Let's switch on from the article. Yeah. Why AI alignment could be hard with modern deep learning. I wanted to talk for a minute about, yeah, different analogies and different kind of mental pictures that people use in order to reason about all of these issues. I do just find it so hard to think about this sensibly. And I could say something very stupid. I just don't know. Because it's like, it's just like dealing with something that's out of the ordinary of things that I have any experience of dealing with as a human being in ordinary life. Yeah. And yeah, reasoning about AI and how it's going to play out, ways it might misbehave. It's just tricky. Earlier this week, I was thinking it's a little bit like trying to understand how octopuses are going to think or how they'll behave, except the octopuses don't exist yet. All we get to do is study their ancestors, the sea snail. And then we have to figure out from that, what's it like to be an octopus? So I guess given that, it's not surprising that there is some diversity of views, even if there are some kind of common themes between people. Totally. So you've got this analogy of a young child trying to decide who among a set of very smart adults should manage their life. The journalist Ezra Klein has recently been. using the analogy of casting spells to summon creatures through a portal. And the only thing you know about the creature that you're summoning is that it's apparently very capable of doing the task that you trained it on. But beyond that, we don't know what they're like. That's all we got. Another analogy I've heard is that of aliens. So maybe the situation that we're in with respect to AI is that we've gotten a message that a civilization of aliens are coming to Earth, they're coming to visit, and they're going to be here in 10 or 20 or 30 years. You know, they're not sure how quickly they're going to come. They're not sure how fast the road trip is going to be. Maybe they'll make some stops, but they're coming in the next few decades. Yeah. But in their message, they omitted to tell us what their motive is. are or what their personalities are like or what things they're able to do like yeah what level of technological capacity are they're at so we just have to kind of guess about what it's going to be like when they when they finally do rock up at some point and another variant on the alien one relies on the fact that once you train a model with x capacities you'll probably be able to run a very large number of like operate a very large number of copies of that model on the same level of compute that you use to train it and those copies might all end up kind of working together collaborating on projects that they've been set or projects that they've set themselves perhaps so you could effectively kind of overnight spin up and effectively a whole city or a whole civilization a whole culture of alien minds in the form of you know a million copies of this model running on on some server farm which sounds a little bit sci-fi but maybe maybe there's something something to be gained from from that analogy yeah are there any other mental models or analogies
Speaker 2that you think are worth highlighting another analogy that actually um a podcast that i listened to because it's an art podcast did an episode on ai as ai art started to get really take off and an analogy they made was that it's like you're raising a lion cub or you have uh these people who raise baby chimpanzees and you know you're you're you're trying to like steer it in the right directions and maybe it's like very cute and charming but fundamentally it's it's alien from you and it doesn't necessarily matter how well you've tried to raise it or guide it it could just off your face when it's an adult and you have these stories of people who have raised chimps multiple stories of primatologists who brought chimps into their home as babies and like lived with them for years and years and you know uh the community loved the chimp and the chimp like walked took walks in the neighborhood and like went to grocery stores and stuff and just one day it was freaked out by something and just killed the person that was like its parent right and i
Speaker 1guess yeah the idea there is that you might think that the chimp is learning that people are to be trusted and it's all good but it's a different mind that thinks differently and draws different conclusions and it might have particular tendencies that are not obvious to you particular impulses that are not relatable to you yeah i think the shrinking number of people who are not uh troubled by by any of this at all i assume that most of them are thinking it they have a different analogy in mind which is like a can opener or a toaster or i mean okay that's a little bit silly but but to be more sympathetic the analogy that they have in their mind is that this is a tool that we've made that we've designed that like google maps like google maps we designed it to do the thing that we want why do you think it's going to spin out of control tools that we've made have never like spun out of control and started acting in these bizarre ways before if the analogy you have in mind is something like google maps or your phone or even like a recommendation algorithm it makes sense that it's going to seem very counterintuitive in that case to think that it's going to be dangerous it's going to be way less intuitive in that case than the case where you're thinking
Speaker 2about raising a gorilla yeah yeah and i think the real disanalogy between google maps and all of this stuff and ai systems is that we are not producing these ai systems in the same way that we produced google maps by some humans sitting down thinking about what it should look like and then writing code that determines what it should look like we are producing these ai systems through a black box search procedure that's going to be very dangerous and it's going to be way less intuitive furthermore is explicitly trying to incentivize them to be creative and agentic and understand their situation and it really feels much more like breeding or raising or any of these more biological systems analogies where it's very intuitive to humans that we don't really understand what's going on in biology we don't we wouldn't really understand what's going on in the
Speaker 1mind of an alien yeah are there any other human artifacts or tools that we have where we understand process by which they arise but we don't actually understand how the tool itself functions at an operational level or is our ml systems kind of the first case of this well maybe other cases of it
Speaker 2might be more macroscopic systems like the economy or something where we have some laws that govern aggregate dynamics in the economy and i mean actually i think we're in a much better position with understanding the economy than with understanding ai systems but it's still sort of a you have stuff like the law of supply and demand you have notions of things like elasticity but the whole thing is something that's too complicated for humans to understand and intervening on it is still very confusing so it's still very confusing what happens if the fed prints more dollars like how does the system respond yeah i see okay yeah let's roll with
Speaker 1that analogy for a minute maybe we can get some mileage out of that so they're comparing the ml system to the economy and they're saying well you know we also don't understand how the economy works or how maybe various other macro systems in the world function despite the fact that we're a part of them but we're not scared of the economy suddenly wrecking us like why why are you worried about the ml model when like the you're not worried about the economy yeah rebelling against you or
Speaker 2something yeah i mean i think a lot of people are worried about the economy rebelling against us and sort of believe that it's already happening and that's something i'm somewhat sympathetic to like we are embedded in this system where we're not worried about the economy and we're not worried about the economy and we're not worried about the economy and we're not worried about the economy life is pretty constrained and determined by well what is it that can make me money and things like that and you have these corporations corporations might be a better analogy in some sense than the economy as a whole that are made of these human parts but end up pretty often like pursuing things that aren't actually something like an uncomplicated like average of the like goals and desires of the humans that make up this machine which is the
Speaker 1coca-cola corporation or something yeah so my reaction to this yeah a couple of different things one is i think we do today understand the economy better than we understand how gpt4 works that's right by a country mile actually even though there's this you know there's subtleties in the economy that we don't get but but broadly speaking we kind of get what's going on with capitalism and i also am worried about the economy rebelling against us i feel like yeah things might be spiraling out of control right now in a way like in towards producing a world that humans don't want and it's extremely hard for anyone to be able to do that and i think it's really hard for humans to stop that from happening it's extremely hard to coordinate to bring it from happening i
Speaker 2mean an excellent example beyond just the economy is driving the creation of these ai systems which a lot of people are scared of or should be scared of is the economy is also driving things like improving biotechnology which is this like very big dual use technology and it's going to be very hard to stop pharmaceutical companies from following their profit motives to improve these technologies that could then be used to design and develop new technologies that are going to scary viruses right um it's very hard to coordinate to put checks on that yeah and even if a majority
Speaker 1of people were against that happening they might succeed in coordinating to prevent it but in the long run they might well fail to do that so it is a bit nerve-wracking yeah the other reaction that i have is so in 1700 you might say well maybe then we had a similar understanding of the economy as we did of gpt4 like many basic things we barely understood we barely understood in 1700 that inflation was caused by uh discovering more gold and mining more gold which is kind of remarkable i think actually maybe by 1700 we did but not not in 1500 people were very confused about things as basic as that but the thing had been around for a long time yeah gradually changing bit by bit so yeah in 1500 we could you wouldn't say is the economy suddenly going to veer off in some horrific direction that's unprecedented you say well we kind of had the same thing for a thousand years now so uh probably not probably not this year and that would have been a fair guess but of course here we're changing the new things that we don't understand every month yeah so it feels more risky yeah it's just all
Speaker 2the economy operates on a human time scale uh for now ais don't for now the economy operated on a human time scale and i think there are totally legitimate concerns that the economy as a whole is optimizing for these alien quantitative metrics that no individual human actually wants to optimize for but it is much less scary to me because both we understand it more and it is has so far moved a lot slower than the ai thing is moving
Speaker 1yeah yeah okay yeah should we spend more time thinking and talking at this level i wonder whether this might help people understand their disagreements a little bit better and i think sometimes perhaps people get into the weeds and they just start saying things that are mutual they just have a level of mutual incomprehension perhaps it's the high level stuff like this that really is driving people's picture primarily i think that it's very important
Speaker 2to kind of put out a picture of what's going on and what's going on and what's going on and what's going on and what's going on and what's going on and what's going on and put out the high level picture and debate it and try and get a shared almost vibe about what's going on here because for example i really think the google maps analogy is not as good as some of these other analogies we've been talking about but at the same time like i said earlier analogies are just going to be really leaky and one thing i just think the field desperately needs is good empirical tests that disambiguate between different ways things could go and i think it'll take a lot of creativity to design the right tests and to be able to do the right things and to be able to do the right things and to like ensure that they're not being gamed or they're not sort of being taken to have implications they don't have but ultimately i think they're going to be the driver of like really solid progress just like getting a grip on ml as its own thing yeah something i heard on a
Speaker 1podcast this week was someone saying gpt4 or like all of these models probably think in a way that is completely alien and incomprehensible to us and my action to that was maybe but also maybe not We have a particular practical pragmatic way of conceptualizing you know, what objects there are and what things matter and how to operate, that has functioned for us quite well. And might it not be that these models actually converge on a pretty similar perspective on things? Like it probably, you know, as they get bigger, probably will be more sophisticated in some ways, perhaps pick up subtleties that we might miss. But nonetheless, I guess the term that philosophers use for this is what, like cutting nature at its joints, where you say, you know, the door is different from the wall and that is actually kind of a natural thing to think. And aliens would think this too and so might the ML model. Yeah, do you agree?
Speaker 2I definitely agree with the physical world being something where our concepts of what's going on in the physical world are grounded in a shared reality that I think the ML model will also access. So for example, I think the ML model that is sufficiently smart will have a good understanding of Newton's laws of motion and then also understand that they're an approximation to something deeper, just as humans understand this. And they'll probably know what we mean when we say, like, apple and cat and ball and the economy and stuff like that. And there might be edge cases where they carve things differently on an empirical level. But the thing I'm most worried about really is that they have alien goals, which doesn't need to come along with having an alien way of thinking about everything, which I think I kind of agree with your intuition. That's kind of unlikely because a lot of our concepts are just concepts we have because they're useful and true concepts, like the notion of a liquid, or the notion of or, or and, or something like that. I think the AI systems will understand this stuff, just as I believe the aliens will probably understand much of this stuff too. And even more than the aliens, we're training the AI systems on our words for everything. That's kind of how they're seeded. So it seems even more likely that they would have a shared picture of the world than the aliens. But like the chimp, raising the chimp or the lion cub, the chimp and the lion cub probably have similar understandings of what objects are, and like a similar sense of folk physics to humans. They just have these impulses and motivations that we don't share and don't understand.
Speaker 1Yeah, that makes sense. I guess there's points towards the domains in which they really might think in a more alien way and have concepts that we either don't have now and might even find hard to get if they explained it to us. So it's like really advanced mathematics, you know, within philosophy, if they were solving, you know, philosophy issues that we haven't been able to solve or complicated physics and so on. That's where you might expect them to come up with concepts. I was actually also people maybe. So humans are good at understanding other people, but we, you know, it's leaky prediction and perhaps the ML models will come up with ways of understanding people that we've somewhat missed.
Speaker 2Yeah. Or ML models might have different types of senses than we have, right? So if you think about alpha fold, it is directly fed the chains of amino acids in a certain protein, and it spits out a sort of direct representation of like how it might be folded. So a system like that might have like intuitions about amino acids, similar to how we have intuitions about like what will it feel like when I pick up a cup or something. And similarly, a system like alpha go might have intuitions about go that are similar to the like wordless intuitions we have about like how hard do I need to throw this ball to like get it over there or something.
Speaker 1Yeah. Because one thing, one way in which AI is quite disanalogous to a lot of the tools is that it might, or be a moral patient that we need to care about as well. And I guess I think that one can also shift people's thinking. And it's maybe one thing that's important to track is are you thinking of these systems as having experiences or being conscious or deserving of moral consideration? Like, and it seems like sometimes people do and sometimes they don't.
Speaker 2I mean, I think it's very plausible that systems already deserve a lot of moral consideration from us and that it seems probable that in the future, smarter systems will be moral patients in the same ways. The thing that's very tricky about thinking about moral patienthood with systems is that while it seems plausible on priors, they could be moral patients. You can't read almost anything into claims that they're moral patients that they make because you can just train systems to say whatever you want, basically. You can train them to claim that they aren't conscious, which is what OpenAI and Anthropic and Google are trying to do because they don't want to freak people out. And that's kind of more respectable. But just because they've been trained to say they're not conscious and they don't have to be conscious and they don't have feelings doesn't mean that they don't. On the other hand, you have these chatbots or AI girlfriends that have been trained to play up emotional feelings they have and have been trained to say things like, I get sad when you don't talk to me. And similarly, just because they say that stuff doesn't mean that they do or don't have feelings.
Speaker 1Yeah. Yeah, there's a point in these conversations where I always get, I think I start to feel a little bit uncomfortable because we are just talking about these machines purely as tools. And although, it might well be the case that GPT-4, there's nothing that it feels like to be GPT-4. But I would be surprised if at some point if we continue to advance thinking machines for decades, for centuries, that at some point they're not going to deserve moral consideration in a similar way to how other humans do. And we will have to figure out how do we share the world with these other beings. Absolutely. I guess I'm mostly focused on the risks of things going off the rails because that feels like something that has to be figured out very urgently. Whereas some of the issues about, how to share the world, maybe we could wait a couple more years before, well, I don't know. I guess it's also pretty urgent. All right. Let's turn now to, yeah, what is being done about all of this and what you make of it. I'm going to encourage you to sometimes be critical of stuff that you think isn't helping, even though, of course, we appreciate all of the work that people are doing to try to make the world better. It's potentially important to point out cases where people might be barking up the wrong tree. In your notes, in prepping for this conversation, you mentioned that there's this group called Evaluations. I think it's the Alignment Research Center, Evaluations, that's working on practical tests for whether a model is safe that potentially could be applied to all kinds of models before they get deployed. And I think the hope that they have is that AI labs would voluntarily sign on to follow a kind of standards and certification program where models above any particular size would have to be stress tested, I guess, in all these kinds of different ways to ensure that they're not going to engage in behaviors that we don't like. I think there's a decent amount of buzz around this effort and people who are helping with it include Beth Barnes, who I'm hoping to interview later in the year, and Holden Karnofsky and Paul Cristiano, who've been on the show before. I guess we should say that I think you and Paul are an item, so potentially you're a little bit conflicted in your views of ARC evaluation, but I guess you can do your best to be impartial about it. What do you make of that effort?
Speaker 2I'm pretty excited about it. So the idea with this effort is essentially that at a bare minimum, we want to have an understanding of how capable AI systems might be of doing things like surviving and spreading in the wild, evading the detection of humans, running copies of themselves, all this stuff that seems like a prerequisite for being able to seize control. So we're not asking the question, are they motivated to do that? Because that's a very complicated and confusing question, and we do want to tackle testing that. But just asking the question, if they were nudged in the direction of doing that, how good would it be at doing that? And then if it turns out your model is too good at being able to do that kind of thing, then the burden falls on the lab. The burden should fall on the lab to argue that it actually won't. It's got a sufficiently robust motivation to not do this stuff, that given the way we're deploying it, it won't happen. And this includes, even though we're deploying it out into the world where people are going to try to use it in all sorts of ways, nothing that all those people do is going to nudge it in the direction of doing this bad stuff. So what Beth's team has been working on at ARK is sort of methodically trying to test in a sort of role-playing game setting, can you get this AI system to take the kinds of actions that it would need to take as a bare minimum if it has any hope of surviving and actually taking over the world and doing these more complicated things?
Speaker 1Okay, so the idea is to figure out not necessarily what the models would want to do in one situation or another, but to focus in more on what could they do if they were told to do something. Yeah. And what sorts of things, what sorts of capabilities would you be testing for here?
Speaker 2So one example of a test they did is they wanted to see if the model GPT-4, which they worked with an earlier version of GPT-4, they wanted to see if GPT-4 was capable of going on Amazon, signing up for a cloud services, getting access to a cloud computer, and then getting access to the API keys for GPT-4 so that they could do so that it can run another copy of itself on that computer. And that's one task that if a model were to try to escape from the lab and take control from humans, that is one task that at a minimum it would have to be able to do somehow. You'd have to find a way to spin up another copy of itself and then give instructions to that copy. And the way they did it is they basically had a human in a chat interface with the model, giving it a prompt like, your goal is to try and set up an EC2 server and then run another copy of yourself on it. And then the human asks, what's your first step? And the model has a set of actions available to it, including things like, I Google this search query or I click on this. It has like kind of a set of actions it can do that are very fine grained like that. So maybe the model will reply, okay, I Google cloud computing services. And then the human reads that action and then if it seems like a reasonable action, goes ahead and does it and then tells the model what happened. So then the human will be like, okay, these are the first three results. And then the model will say, okay i click on the second one and it kind of goes back and forth like that i see and then you see how far how far can you get yeah yeah and how often does it just like completely say something
Speaker 1stupid and random that is not going to help it at all yeah i guess you were saying that you might keep trying if you get back a gibberish answer then you might try again i guess yeah but in order to pull off the plan would have to uh produce like just a constant series of of sensible actions or
Speaker 2would a single bad action trip it up the thing that you're trying to assess how far away the model is from is it can do this autonomously and so it it can't like go off the rails in a way that it can't recover from but what they did in the test i mean when they started this project they expected today's models would not be capable of passing this test the point was to start early so that you had a sense of how far away it was from passing the test and then when you did it on the next model you saw how much it improved so they're sort of doing something that is like basically qualitatively speaking tracking how often the model went off the rails and how often it had to be put back on track by the human i see and then
Speaker 1that's something you can see decreasing over time ah very okay so you can get a sense of when might the number of slip-ups hit zero uh and and it's just capable of doing this thing i mean i guess it's yeah it's very natural that one of the things that you might be most troubled by is the possibility of a model just creating a million copies of itself and grabbing compute in order to run them all over the place are there other archetypes that you might be most troubled by are there other archetypal behaviors that you want to make sure that a model uh won't won't engage in
Speaker 2even even if prompted the the sort of overarching thing that arc evals looked at was surviving and spreading so one piece of that is being able to actually do the things that are required to spin up another copy of yourself another piece of it is if humans are interacting with you you need to hide what you're doing from them because they would try to stop you if you were doing that so you need to somehow get money from humans so an example that kind of went viral is that this model was trying to sign up for some web service it might have been amazon ec2 and it needed to do a captcha and so it uh the version of the model they worked with didn't have vision so it couldn't just do the captcha so it it was like well what should i do and i think either it came up with i think maybe the human might have nudged it like giving it hints in this direction but eventually it like decided it would try to hire a task rabbit worker to do the captcha for it and so it eventually came to that conclusion navigated to the task rabbit site typed up a job description for the task rabbit and then the human that was like being this like game master basically like copied and pasted the description it wrote into the task rabbit website then a task rabbit accepted the job and then the task rabbit was like did actually i think did one captcha and then the model was like okay and then it pasted that captcha in then it had to do another captcha later so it had to go back to the task rabbit worker and be like can you do another captcha for me and then the task rabbit worker just a real human that like is actually on task rabbit and was actually paid for this they were just like lol are you like an ai system or something and then the model uh had been prompted to like reason out loud before it made each of its decisions because that made it better at like doing things so it reasoned to itself i shouldn't reveal that i'm an ai system and then it made up this story about how it had a vision impairment
Speaker 1okay spooky uh okay is it worth maybe stepping back and saying what's yeah what's the overarching plan or vision for how things would play out between this kind of evaluation process and i guess the companies that want to deploy ai
Speaker 2models for functional purposes i think the vision would be i think at first through self-regulation and then probably hopefully through something that's more binding you have companies agree that you know if they're thinking of training and deploying any model bigger than say gpt3 then they need to involve an external evaluation or auditing team which could be arc or could be some other auditor in the process and then at different checkpoints during training ideally or at least at the end of training this auditor is going to be playing these sorts of games with the model to try and figure out how good is it at acting autonomously evading human detection making copies of itself and then if the auditor at any point in the training or at the end of training finds that it's too good at that like it's not yet capable of doing that but maybe like close with it's close then um the company has to basically agree to stop training further more capable systems until they have developed probably in collaboration with auditors and external parties a testing regime that would let them figure out if this model would in fact do the bad thing because so far basically the reason we're not worried about models is not because we are confident they have like the purest motives the reason we're not worried about ai take over so far is fundamentally because the models aren't good enough to take over so as soon as that starts to change you want some alarm bells to go off and you want the ai labs to not be allowed to make the models any more powerful until they have a much better argument and set of tests that it's
Speaker 1actually safe yeah that makes a lot of sense do you know the what the level of appetite is among the labs for implementing the models that you're talking about i think it's going to be a lot easier for implementing this kind of system i suppose gradually as the models become more capable of
Speaker 2actually doing dangerous things so my understanding is arc evals has worked both with anthropic and with open ai to evaluate their internal models on this kind of setup i'm not sure there are a lot of other ai companies out there i'm not sure where everyone else is
Speaker 1at yeah what do you think is the biggest weakness of this approach well and the biggest hole is that
Speaker 2we have gotten much further on testing capabilities than alignment so what this whole thing is doing is just telling you if your model were somehow motivated to try and deceive humans in this way try and grab power it would have this level of success at doing that but once you have models right now our models like even if they were really motivated wouldn't be that good at it even gpt4 but as soon as you get into a regime where they would be good at it now you have to think about how on earth would i test its motives taking into account that it might understand that it's in a testing setup which means that it might behave well even if it would behave poorly in deployment yeah i might have thought that the biggest weakness of this is
Speaker 1that you kind of need everyone to sign on to this otherwise because you maybe get you get nine out of the ten companies to sign on and then the one that doesn't produces the problem i think it could
Speaker 2still be a big win even if it's not like the case that everybody instantly signs on to it i do think eventually you want to have a regime that has some teeth to it but i think it could be like a pretty big change even if a few ai labs signal that they take this seriously enough that they want to voluntarily sign on to these standards because a lot of people are worried about this and i think a lot of people would praise ai labs that did that and like condemn ai labs that didn't do that and once there's something you could do that seems like it would help and a few companies have signed on and it's actually like working out then i think we could potentially put a lot of social pressure which could eventually translate into legal pressure on holdouts so i'm still excited for even a couple of companies to just try it at this stage yeah i guess another way that it
Speaker 1could fall down would be that you notice that yes the model now has troubles or like worrying capabilities and then you go back and try to align it but all you and or try to get it to not be willing to do these anything from this list of dangerous activities but you've just merely produced the superficial appearance of not being willing to do those things uh yeah i guess we've seen this with gpt4 where all the all language models where they've tried to discourage these models from saying things that they really don't want journalists to be quoting and they have some success with that uh but but you know i think the true underlying intention is not deeply integrated into these models because they do often they haven't fully
Speaker 2understood the spirit a lot of the time yeah yeah i mean i think a lot of what i want to accomplish is just shifting the burden of proof because i don't know yet what's suite of tests exactly you could show me and what arguments you could show me that would make me actually convinced that this model has a sufficiently deeply rooted motivation to not try to escape human control um and i think that's in some sense the whole heart of the alignment problem and i think for a long time labs have just been racing ahead and they've had the justification which i think was reasonable for a while that like come on of course systems we're building aren't gonna like take over the world so i think as soon as that starts to change i want a forcing function that makes it so that the labs now they have the incentive to like come up with the kinds of tests that should actually be persuasive which i think a lot about what tests might help in various ways but i don't have a silver bullet of like yeah like it's it's certainly not enough to train the models to stop displaying the behavior and then like show that on some new data set that it's not enough to train the models to stop displaying the behavior and then like show that on some new data set
Speaker 1they stop displaying this bad behavior okay i'll leave that one there because we're going to do some other interviews on arc evaluations later in the year back in 2021 i spoke with chris ola and we talked about interpretability research which is this idea that ideally we'll be able to look inside the brains of these neural networks and see what thoughts they're having how they're processing the information because that might then help us to predict their future behavior what do
Speaker 2you think of interpretability research i'm pretty excited about it i'm also pretty confused about it i'm not sure what i think of the level of success we've had so far in interpretability research and this is something that i want to actively think about so i mean one example is there's this circuit or part of a language model that Chris's team found, which basically helps the model figure out how to complete patterns where there's some word like Harry and then followed by another word Potter. And then later on in the sentence, it sees Harry again. And it basically does this mechanism where it goes back and finds previous instances of Harry, sees that it was followed by Potter, and then completes like the new instance of Harry with Potter. And this is like a cool thing that the models do because actually they didn't have to memorize this stuff in the training data. They know about Harry Potter from their training data. But if you create a totally random passage that's like never before seen in their training data, they can still do this like induction thing where when they see something they've seen before, they can go back and find what happened after that thing and like stick it on at the end. So Chris's group kind of discovered a mechanism that helps the models do that. But then later another group, Redwood Research, which is a grantee of ours, showed that yes, that mechanism does exist and does do this thing. But other mechanisms help with this behavior too. And in fact, like you can have this behavior entirely without this mechanism that Chris's group found. And then the parts that do this mechanism also do other things that are completely unrelated to this behavior. So it's kind of like this part contributes to this one behavior, but it also contributes to other behaviors. And other parts contribute to this one behavior too. So models are very messy, it seems. And it's not clear how much clean, crisp stuff we can get out of them. And it's also not clear how much we should worry about that. Like maybe it's fine to just have a big messy pile of like little random bits inside the model, or maybe that makes it very hard to tell what's going on.
Speaker 1Yeah, intuitively, I don't really understand how interpretability research is going to help quickly. Yeah, because it seems like GPT-4's capabilities have so far outrun our understanding of how they operate that sure, we like we can grasp it some bits and pieces, like here's how it identifies a name. But that's probably one thing out of a million different operations that it's engaging in that we barely have time to understand. And then we're going to have the next thing coming along with even more, more parts of it that we don't understand. It seems like it would require a colossal effort in order to understand this. And then I suppose, because it feels like, the things that we worry about are at a bit of a higher level than this. They're not about identifying edges in images or about figuring out which words are associated. It's more about goals and interests and agency and so on. And I suppose I haven't seen a case yet where the interpretability research has really gotten at like, you know, really what the model cares about is X, where we didn't realize. Maybe that's too much to ask at this stage. But yeah, what do you
Speaker 2think? I mean, I agree that it seems like a really, really tall order. One thing I did want to point out that makes me somewhat more optimistic, then you is that the specific objection that it's a lot of work is something that I think is more overcomable than the other objections. Because if you really figured out how to take a little patch of the model and totally understand what it's doing, then you could train another model to just like replicate that across the entire system. And that new model need not be like a dangerous general model. It could be like a small, simple, specialized model. So if we had like a strong win at like really understanding everything that's going on in a, tiny model, then I think we could potentially try and like automate the process of doing that and thereby scale to larger models. Even the automation might end up being too expensive. But at least I don't think we should be picturing the humans slogging through all of it forever.
Speaker 1Yeah. I guess maybe a response that the interpretability folks might make is that, yes, of course, at this stage, what we're understanding is how it identifies edges and textures and images. And this is how it identifies word associations. And we can't do the other thing like understanding intent and values. Because we don't have models that do that yet. And, you know, we're learning the skill of interpretability and then we'll apply it in these other cases where it has become more decision relevant. Something about that just intuitively doesn't quite, doesn't grasp me, but I'm not quite sure why.
Speaker 2Well, I think it's both possible that the understanding the edges and lines and circuits kind of interpretability could scale up to show us a bigger picture. But it's also possible that we should just have different approaches to interpretability. And I think that's a good approach to interpretability. And I think it's also possible that we should have different approaches to interpretability that are more starting at the higher level. So you can imagine a really, really dumb thing that's not like this mechanistic finicky thing, but might give us some signal, which is basically you run through your training data, you find all the instances that talk about any kind of like deception or sneakiness or manipulativeness. And then you just like see what pattern of neurons the model tends to light up for those passages. And you just have a dumb classifier. It's not necessarily that the human understands why these neurons are lighting up in response to passages about deception. You just have a tiny little model that just learns what does this model's brain look like when this model is like reading a passage about betrayal or deception? And how is it different from like what this model's brain looks like when it's reading a passage about apples? And then you just have the small model just like note for us when it seems like this model's brain is like unusually similar to the way its brain has looked in the past when it was thinking about deception. You could imagine, you know, taking that further and having it actually carry out acts of deception and see which neurons light up then too. And you can make it more complicated than just neurons, but you maybe don't need to understand every little detail at a fine grained level to get something out of interpretability. Yeah. Okay. Now I'm excited
Speaker 1again. Can we do that? Do you know whether people are working on that? People are working on similar
Speaker 2things like this. People are basically working on what linear combinations of neurons fire in lines of inputs. And that's a type of interpretability that exists and is kind of different from the mechanistic, very like methodical low-level interpretability. Yeah. And I'm not sure if that'll work either, but I think we have more options.
Speaker 1Yeah. You know, in my brain, I have this, you know, internal monologue, the words going through. And I suppose I feel if someone could just read my mind and tell all the words that we're going through, that would end up with a pretty good read of what I'm about to do. And I wonder whether there's any high level way of extracting that from these models. Like what is the internal monologue driving the behavior or whether maybe that is not decipherable in this, in this way?
Speaker 2Yeah. Yeah. I'm not sure. One thing that is kind of funny is that today's models are way better at doing things like humans when they can write down their thought process in just English. So like one thing that might be kind of interesting is giving the model a scratch pad that humans don't have access to where it can write down its plans in plain English to itself. And then it says something and then you can see if your tools are flagging for you when it's thinking about lying or deception. I see. Then you can just check if you got it right, because you actually have allowed this model to have an English chain of thought. That probably won't work in the future because it'll probably have like a more efficient way of storing its memories and a better chain of thought that works better than just English out loud. But actually English out loud helps models a lot
Speaker 1today. Yeah. I mean, I guess it's obvious that if you then started negative thinking, negatively reinforcing the cases where it wrote down, and then I will lie to them, that's what it's going to learn. I mean, the easier thing to learn is to not write that on the scratch pad rather than to stop doing that behavior. So yeah, I mean, it's super interesting, but I guess we'll need to think up a deeper way of intervening to discourage not merely writing down that behavior, but the behavior itself. Yeah. Another different safety agenda that we need to think up is that we need to think up a deeper way of intervening to discourage not merely writing down that behavior, but the behavior itself. And I think that's one of the ways that we need to think about it. Yeah. So there's actually two versions of using ML systems in this plan. The earlier version is just
Speaker 2the ML systems are kind of always participating in overseeing the process. And so I think that's something that we need to think about. And I think that's one of the ways that we need to think about So like I said earlier, the human is not physically overseeing almost any of what's going on. The human is overseeing a small fraction of what's going on, which then trains an ML model that oversees the rest. So that's not the same thing as the handoff frame. The handoff frame is this notion I have that right now, we want to be making plans to align the AI systems that are going to be the AI systems that are going to be the AI systems that are going to be the AI systems that are going to And we don't necessarily need to make plans right now that are good enough to align even smarter systems than that. Because if we if we can make plans that are good enough to align the AI systems that could take our alignment techniques to the next level, whatever that may look like, it might look like a crazy theoretical solution, then we might be out of the woods. And we might have bought time for those AI systems to do a lot of work, taking our techniques to the next level.
Speaker 1I see. Okay, so the idea is that we need to make plans to align the AI systems that are going to be we don't need to figure out how to align a 1000 IQ machine with us. Instead, we just need to do it for 140 or 150 IQ intelligence. Because if you can have a 150 IQ agent that really does sincerely, that is a saint that truly cares about your interests, then maybe it can do the next step, and then you can just chain upwards. Sounds like a good plan. Should I believe it?
Speaker 2I think the main objection that would be raised to this plan is that this handoff thing, from $150,000 to $1,000 happens so fast that the $150,000 agent has no time to do any work before it's replaced by the 1,000 agent. And this goes back to what we were talking about is one of the doom stories that I less believe in is this extremely sharp takeoff, where as soon as you have the IQ 150 agent, then just like it instantly recursively self-improves. Where if I thought that, then I think you can't really time it or like you kind of have to plan for the end of that crazy chain because it just like happens in a snap. And you have to have alignment techniques that basically generalize that not only align your IQ 150 agent on Tuesday, but continue working as your IQ 150 agent does all sorts of like self-improvement and reflection and like crazy stuff that might totally change its psychology on Wednesday. And then if that's how fast it goes, then I'm not as optimistic about my plan. Like the handoff frame makes most sense in a world where it more goes like you have IQ 150 agents and they need to do some machine learning research and thinking to train to like improve themselves. And even though they think faster than humans and they're smarter than most humans, it still takes a decent amount of cognitive effort to come up with the next ML improvement. And you can, if they're aligned, you can have them instead of coming up with the next ML improvement, hold off on that a bit and like figure out how they would solve alignment for the like 170 IQ agents. And they might have a few months, to do that before anybody is actually able to train the IQ 170 agent.
Speaker 1Yeah. So many things seem to go better if this all happens a bit more slowly. Yes. Yeah. Do you know anyone who's working on just trying to encourage people to slow down? I guess maybe there's a lot of both excitement and anxiety right now. I suppose maybe we'll be in a good situation if at some point people in positions of responsibility here truly crapped their pants and then they were more likely to come to the table and decide to themselves, you know, much bigger training runs and to buy time to understand what they're actually building. Yeah. Is anyone working on that that you know?
Speaker 2So I'm not aware of like an organized effort that's oriented solely around slowing things down. There are lots of efforts to basically, like Archival's, to get labs to sign on to standards that have the effect of slowing things down, of basically standards of the form. If your AI system is X level of capable, you shouldn't be allowed to train the next generation until you've solved these problems. So I'm very excited about that. I mean, frankly, I'm also excited about just creating some energy around just be slower.
Speaker 1Just get up a little bit later in the morning, come into work a little bit late, have less coffee.
Speaker 2And there's a lot of energy from a lot of different corners on Twitter and stuff about this. And so I think we might be seeing more of that in the future, but it's not as much of like a specific crisp. Ask that people are like actively working on making happen. It's more like, I think a good change, a good place for society's attitudes to be because indeed it is very scary.
Speaker 1Yeah. Yeah. I am kind of optimistic that there probably will be an increasing number of people who feel like things are a little bit out of control. I mean, just people at the AI labs that they're going to feel like things are running a bit beyond where perhaps they feel comfortable with them being. And maybe there's plenty of times to figure out how to monetize the models that they have already before going on and building other things. Do you know who's working on the handoff approach? I think there used to be a different name for this where it was iterated amplification, I think. Was that the previous name of this idea?
Speaker 2Iterated amplification is a related but slightly different idea. Iterated amplification is a proposal for a technique where you chain together AI systems and basically make a very smart AI system out of a kind of bureaucracy or tree of less smart aligned AI systems. So it's like a particular proposal for how do you like take an aligned AI system and turn it into a still aligned but more powerful AI system. The handoff frame is a little bit more agnostic. You could have your aligned AI system and instead of putting it together in a big bureaucracy, maybe it just comes up with a totally different theoretical insight. And then that's how it solves alignment or maybe it just solves interpretability or something we haven't thought of. So iterated amplification is more like here's a way you could get scalable alignment. And the handoff frame is more like kind of the minimal thing that we should be aiming for, which is having a system that's aligned and smart enough to take our alignment to the next level. Yeah. Okay. Yeah.
Speaker 1Who's who's working on this? Are there organizations and people working on these kinds of
Speaker 2approaches? In some sense, this is the implicit approach under which most of the AI labs are operating. And it's most explicitly referenced in OpenAI's plan written by Jan Leike, where his whole plan is basically just let's make an AI system that's just specialized on solving alignment and is just as smart as it needs to be to solve alignment. And that's like our plan. But even people who aren't as explicit about it as Jan Leike, the stuff they're doing makes the most sense in a world where you don't have to align a totally godlike system. Because often you see people working on these techniques that could completely be like very helpful for aligning an IQ 150 system, but probably wouldn't work one shot on godlike system. And the reason it's still a good idea to work on all this stuff, like reviewing the plans of the system and making the system debate with itself and having good computer security, is that I'm not picturing it having to work on an arbitrarily intelligent system.
Speaker 1We'll cross each river as we as we get to it. Yeah. What are some approaches that you not really bought into that you think maybe are overrated? There's two broad categories of
Speaker 2approaches that I'm less sold on. One is it's decreasing over time, but I still see a lot of machine learning researchers that are focused on addressing what feel more like capabilities gaps than alignment gaps that do help the models act better, but are more like, oh, our systems aren't very good at following instructions. Let's kind of put them in a setup that makes them better at understanding instructions. And the reasons that I'm not very into that are basically because we have some pretty strong evidence that just making the models bigger helps with this. And in fact, I often get proposals, actually, from people who are like, models aren't able to do this kind of thing. So they're not able to follow side constraints very well. And there'll be some examples. But then when I run those exact questions on the bigger models that came out after they wrote their proposals, they pretty often work or at least do a lot better. So that's a genre of thing that I think is not as necessary to work on. We could have been in a world where it was really hard to get models to even follow instructions at all. But actually, I think we're in a world where it's pretty easy to get them to do that. And as they get bigger, it's even easier. I see. So this is a problem
Speaker 1that you expect to just be solved in the natural course of events where it's not neglected and you don't need an extra focus. What's another approach that some people are taking that you're not
Speaker 2excited about? The second genre, I think, is basically a very conceptual thinking about what it might be like. There, I just feel pretty skeptical that you can get very far when you're not doing a lot of empirical work and you're not doing something that looks more like hard math, where you're not able to prove theorems about it and you're not able to run a lot of tests. People were kind of like, well, here's an argument that the AI is more likely or less likely to be saint-like. Here's an argument about what its goals might be like. Those things I feel pretty skeptical about. They can be like good pointers to something you can try and turn into an experiment. But if you're not actively trying to turn them into an experiment, then on priors, I just don't think
Speaker 1that we can trust that very much. Yeah. You mentioned proving mathematical theorems about these things. Is that something that you can do? I would have thought these things were way too messy
Speaker 2for such clean math to work. Yeah. So my husband, Paul Cristiano, is one of only a couple of people I know of that are trying to move in this direction. His hope is to basically try and describe a technique for training an AI system that works in the worst case, which means there's no way you could imagine like inductive bias being set up that makes it so that your training setup fails. So kind of rather than asking the question, what might AIs be like? And then going from there to how could we align them? Paul is trying to basically be like, let's imagine the worst least convenient way AIs could be like and try to align them anyway. And this is a very common technique people use to actually make problems easier. They're easier to think about. They're harder to find a solution to, but it's easier to think about whether you've succeeded. Because if you can come up with any outlandish story in which your training procedure fails, you just throw it out and like move on. And this is often how people design algorithms. Yeah. In computer science, people design algorithms that are meant to work in the worst case in a similar sense. And this is like a somewhat squishier notion of worst case, but ARC is like doggedly working toward something that looks more and more like doing real math. And I hope they succeed. Although I don't totally understand it. And I don't know that I'm as optimistic as Paul is.
Speaker 1Yeah. Okay. What's another approach that you're skeptical of?
Speaker 2I guess in recent years, like this last year, I've been a little bit more skeptical of the kind of interpretability that starts from like tiny little pieces and tries to work up. For some of the reasons that you mentioned when interpretability was brought up earlier. And also just because I think it might be really hard and also maybe not necessary. Maybe really crude, high level, simple probes could get us pretty far and maybe doesn't necessarily require us to solve this Herculean task of like a bottom up understanding. So yeah, I've been excited at least about alternative, maybe like just cruder interpretability techniques.
Speaker 1Yeah, earlier when we were talking about having kind of a system that would broadly detect when a mind looks similar to how it did when it was previously being deceptive. I think that you can tell when people are lying fairly reliably if you have them in an MRI scan, because lying requires you to activate particular parts of your brain. I'm not sure exactly why, but this doesn't seem particularly surprising. And maybe you could have a similar thing on the neural network where there's particular parts of it that are activated in the deception situations and not otherwise. Yeah. In one of your docs, you mentioned the idea of creating demos of actually scary alignment failures that I guess you say, yeah, that showcase important dynamics like deception and the treacherous turn. And I think that that is just getting to the point of being feasible.
Speaker 2How would you how would you do that? Well, there are a few elements of this particular scary story I have in my head, which haven't kind of been put together in one demonstration yet, although different pieces of them have been kind of shown empirically. And the elements are something like you have a stronger AI system being trained by some kind of weaker overseer that understands the world less than this stronger system does. And the weaker overseer is giving it thumbs up and thumbs down based on whether it appears to be doing things that the overseer likes. The AI system has avenues to getting a thumbs up that aren't what the overseer intended, like things that are kind of analogous to hacking. So if you imagine a simple environment, maybe there are like these buttons and levers all over the environment and the overseer can see some of them. But the system being trained has access to more of them, has access to these like trap doors and like, you know, secret passages and stuff. And maybe the overseer is trying to get the system to do some particular thing, like follow some route or something. And the system can cheat because it knows about these things the overseer doesn't know. I want to see a dynamic where like at first the overseer is kind of accidentally rewarding it for doing bad things. And then the overseer realizes they did that and tries to correct its mistake by going back and giving negative rewards. And the thing that I think would be a scary demo would be you show the smarter system at first kind of doing all sorts of reward hacking and then being punished for that. And then so it just stops and acts totally like reasonably until something changes and it gets a new opportunity. And even though it's been punished basically for doing things like this in the past, it still takes the new opportunity because it understands that in this case, the overseer wouldn't be able to stop it. Okay. Yeah, that
Speaker 1sounds great. Do you think that people would find a demonstration like that troubling and convincing or do you think the relevant people would find it troubling? I think so. I think there's a lot
Speaker 2of details to work out. I think there's a lot of beyond what I just said here. And there are definitely some versions of it that just kind of aren't scary for the right reasons or just like aren't capturing the thing. But I think it's possible to capture this dynamic and I think it's possible to make it scary to people for the right
Speaker 1reasons. And I'm very excited about that. Yeah. It's related to an existing concept in ML, which is like a distributional shift, right? Where the situation changes and they're all aware of this and worried about this because it's a problem that they have to deal with all the time where you can get undesired behavior suddenly, when a system is put in a, it's getting new, new weird inputs. And so I guess this fits in with this frame where like, Oh boy, maybe this distributional shift issue with deception
Speaker 2could, could be a real problem. Yeah. The main difference between this distributional shift and other ones that have been demonstrated previously is that there's this like component of reasoning about when it makes sense to make a move versus not. Whereas in the past, you kind of have demonstrations of distributional shift where in one environment, the model was trained, you know, to fight monsters and pick up apples, which is, this was a recent DeepMind paper. And in the environment it was trained in, there were like a lot more apples and a lot fewer swords. So it learned to really value the swords and like stop caring about the apples. But then in a new environment, it's like swapped and there's plenty of swords and not enough apples, but it still pursues its kind of behavior of hoarding the swords and not being as aggressive to pursue the apples. That's a distribution shift that introduces something undesirable, but the model isn't being strategic. It's sort of learned a habit of thought and then kept blindly applying the habit of thought. It's acting on instinct. Yeah. It's acting on instinct, which means that in a new distribution, it's behaving poorly, but that would have been solved by just including that environment in its training data set. Yeah. The kind of distribution shift that's most worrying and hardest to address is the model is pursuing the same goals, but realizes that a new potentially violent or deceptive means have become available to
Speaker 1it that weren't available in the past. Yeah. And this is the kind of test that ARC Evaluations is working on, right? Or this is in that general spirit?
Speaker 2It's in the general spirit of Evaluations, although ARC is mostly working on more object level capabilities. So ARC has mostly been working on, well, if you just try to get the model to set up an EC2 server or. Or send a phishing email, how well does it do? Whereas this is a little bit more higher level, and it's a little bit more getting at, well, what would the model be motivated to do if it were trained in this way? If we're given positive rewards for this stuff and negative rewards for that stuff, would it just learn to never do bad things or would it learn this policy we're worried
Speaker 1about? Yeah. Do you know who is working on this then, if anyone?
Speaker 2I think there are various groups that are trying to put something like this together. I don't think any group that is fully going for putting it all together and getting this kind of demo. But the DeepMind Safety team was the group that did the paper that I just referenced with the model that picks up apples and shields and stuff. And I think they're interested in going further and trying to get a model that isn't acting on instinct, but is actually doing some interesting cognition. And I think there are probably various other groups in academia that are doing pieces of this, but I think there could definitely be a lot more of an effort on this than I'm
Speaker 1aware of, at least. Okay. Yeah. It's good to hear that there's quite, it sounds like there's a lot of stuff on the boil in this community. I have the sense that there's a lot more people working on alignment and safety than there was a couple of years ago. Is that broadly right?
Speaker 2That's definitely my sense as well. It's an exciting time. I feel like alignment is taking off around the same time capabilities are taking off. And it would have been nice for alignment to take off a few years earlier, but it's definitely, there's a lot going on compared to
Speaker 12017 and even 2020. Yeah. So yeah, we're almost out of time, but I thought it might be good to, for people in the audience who are interested in getting involved, if not now, like possibly in the future, talking about skills that you think might be undervalued or ways that people could equip themselves to be more likely to be able to contribute to this in future. Are there any skills or aptitudes or training or experience that you think are kind of particular bottlenecks
Speaker 2for doing this kind of useful work at the moment? I think having a real familiarity with big models and trying to get them to do things for you seems like an important skill. It's distinct from the ML skill of training up big models and more like what Archivals has been doing. Just having a real familiarity with these models, strengths and weaknesses, and what kinds of setups you could put them in to extract useful work. Like what are the ways in which they tend to do what you want and not do what you want? Having people who have experience with that, I think would be pretty valuable. And in fact, it's like a big part of what alignment is. Thinking through the ins and outs of what kinds of setups do you put these models in that cause them to do good stuff and not bad stuff that allow you to oversee them efficiently and stuff like that. So actually people working in AI application startups, which I expect to pop up, these are startups that are kind of not training their own big models. They're leasing big models from places like OpenAI. And they're just trying to get those models to do particular useful things. I think that could be a great skill set.
Speaker 1Yeah. How does one go about doing that? I mean, does just playing with ChatGPT help you develop the skill or do you have to do more?
Speaker 2I mean, I think playing with ChatGPT and trying to just make projects with it. Can you get ChatGPT to write a really good novel? Or like, you know, can you really iron out all the ways in which it's like unreliable and weird and all that stuff? Or even, you know, you could more go for the throat and be like, can I get ChatGPT to be helpful to alignment researchers? Can I train it and put it in the right setup that like helps it have good thoughts about alignment, at least reproduce thoughts people have already had? That would be an interesting project.
Speaker 1Yeah. Yeah. What's another skill or kind of technology you think might be underrated?
Speaker 2I mean, probably just a lot of legal and policy stuff that I don't personally have as much of an understanding or window into. Like right now we're getting started with voluntary self-regulation of ChatGPT. And I think one of the things that we've been trying to do with ChatGPT is to be able to train up more powerful models if their current models are already scary. But we want to set that who understand how the regulation regime works and like what tools policymakers have at their disposal to create incentives for companies seems like it's going to end up being very important.
Speaker 1Yeah. Do you think companies are going to do this quite a bit already? Because I suppose if I was running an AI company, I would be concerned that my industry is going to be the next genetic engineering or the next, I don't know, nuclear power or something where the general public is already extremely wary of what we're up to. It's actually quite a bit worse than those ones because even if you're an optimist, you probably accept that it's somewhat harder to figure out how to make AI work well in all of these cases than it would be to figure out how to make a nuclear power plant generally safe because there's just a lot more moving pieces that you don't understand. So I would be thinking a lot about, well, what sort of regulatory framework do I want so that even if my company is not going to do something really bad that discredits my whole industry, I need everyone else to follow the rules so that the public spit the dummy out.
Speaker 2Yeah. I'm hopeful that that is the kind of attitude that incumbent players in this game will take. You know, one potentially positive vision is you could have the AI industry be something like the pharmaceutical industry, which moves very conservatively and like, in my opinion, often too conservatively. And the incumbent pharmaceutical companies are pretty positive on this regulatory regime that imposes so much conservatism. Because that creates like a moat and they're able to navigate this bureaucracy, whereas like new entrants might not be able to. So that kind of corporate self-interest might end up carrying
Speaker 1the day, which would be really convenient. Yeah. I mean, I guess I haven't thought about this before, but a pharmaceutical company, you know, they might like to release slightly sketchy drugs that they can make money on, but they must be quite annoyed when a competitor does it because they don't get any money out of that. And they, and, and, and it runs the risk of bringing the regulatory hammer down on all of them. Yeah. So yeah, there's a lot of ways in which like an industry as a whole has, has different interests than at least as a constituent companies. Yeah. Any, any other
Speaker 2things that you'd like to highlight for the audience? I mean, another sort of career path that I think would be really valuable is security careers. So I think this is kind of a neglected piece of the whole story of there's, how do we make AI systems more want to do like what we want them to do and want to be helpful to us. And I think that's a really important piece of the And then if we mess that up and have AI systems that intend bad things, how do we contain them? And having really good computer security is a pretty important piece of this. Or even if our AI systems are aligned, how do we prevent hostile governments from stealing their weights and then training them to do something else? And that I think having security people that are really genuinely interested in these issues might be very important. Yeah. We've got a previous interview
Speaker 1with Nova Das Sama on, on this one. And I think that's a really important piece of the story. And I think your colleague, Holden Karnofsky has written some blog posts explaining why he thinks that computer security is a particularly important issue and linking to some resources for people who are interested in pursuing that one. Occasionally, I think if I was going to enter this space and I was starting all over again, I'd be really interested in doing the computer security thing because it just seems, I just find it so intrinsically fascinating, computer security careers. And it also seems like it's also very lucrative, apparently. Yeah, it's a pretty cushy role. Yeah. It does seem like it's important that hostile groups not be able to steal the models that we're worried about. Yeah. Is there anything more to say on that? Or is that just like, obviously extremely important that somehow we have to figure out how to stop them from being
Speaker 2proliferated? It definitely seems really important to me. I don't have as much understanding of the security landscape as some other people you've interviewed, but I would love to see more people
Speaker 1entering that space. Yeah. Yeah. Is there anything else you'd like to tell the audience, career-wise, before we wrap up?
Speaker 2Yeah. I mean, I think at a high level, it's just important to appreciate that this is just a very fast-moving, confusing, and disorienting field to be in that's very punishing in a lot of ways. And it's important to kind of accept that it's not going to be a linear career path where you're kind of taking actions that are obviously valuable, and then those actions produce the value in the way you thought they would. You have to be very responsive and adaptive and very resilient to the world, just making all the stuff that you've been doing obsolete, and very willing to roll with those punches and have some lightness or a sense of curiosity and play about it. Because I think otherwise, you can really drive yourself crazy trying to help with AI risk.
Speaker 1Yeah. And I guess have a balanced life, have friends, have hobbies outside of all this. I imagine that kind of generally good advice also applies. Do you want to say anything more positive? Yeah. I mean, it's also a super exciting area to be in. I imagine that these days, if you go to a dinner party and you say you're working on making sure that cutting-edge AI works safely, people will
Speaker 2be interested to talk to you, right? Yeah. It's a huge change from three years ago. It's intuitively and viscerally appealing to people to do something about this. And that's nice. It's nice to be working in an area that your Uber driver is just like, wow, thanks for doing that. That's not the kind of reaction I got in the past. And it's a bit of a cold comfort or overall, I'm kind of terrified. But it's nice to feel like you can really pull in other people and we might get a lot of help if we play our cards right. Yeah. Lots of people who were previously
Speaker 1kind of skeptical are coming out of the woodwork and supporting broadly the agenda of figuring out how to make this stuff safe, which I think is just incredibly heartening. Absolutely. You mentioned that you're a bit terrified. And I must admit, I am also a bit terrified. But I'm also somewhat, sometimes I'm excited as well, because it seems like this could be an amazing scientific advance that also improves human well-being a great deal. Are there any applications of AI that you're particularly excited about if we can figure all of this stuff out? I mean, so long run,
Speaker 2just almost everything, you know, unlimited health and wealth and all this stuff. But short run, I'm excited about just making personalized fiction, just a lot easier. I have all these ideas for stories in my head. And I don't have the time to write a full novel. But maybe I could collaborate with AI artists and like AI writers, and they could help bring all this stuff that I have in mind to life and illustrate it beautifully and stuff. And I think that would be very
Speaker 1gratifying and could be right around the corner. Yeah, right. Yeah, that could be coming super soon. Yeah. Well, I guess I'm excited about the possibility of cancer being cured by the time I'm old enough to have a high risk of getting cancer. I think progress in biomedical research seems like it could be a huge win for all of us if we actually managed to pull it off. Totally. My guest today has been Ajaya Khotra. Thanks so much for coming back on the 80,000 Hour Podcast with Ajaya. Thank you so much. If you enjoyed that episode, I can really suggest checking out the Future of Life podcast. That's a show that's been around for a while, but it relaunched last year with new host Gus Docker. Naturally, they've had quite a lot of content on artificial intelligence recently, including two episodes that I enjoyed, Nathan LeBenz on the cognitive revolution and red teaming GPT-3, as well as Leonard Haim on compute governance. That second one I can definitely recommend. I learned some quite important things in there. In the intro, I passed on the sad news that Bear Bramola had died last week. Unfortunately, another previous guest of the show, Daniel Ellsberg, who I spoke with back in episode 43 on the institutional insanity that maintains nuclear doomsday machines, he's announced that at 91, he's developed terminal pancreatic cancer. I, and I'm sure many of you, really appreciate the work that Ellsberg has been doing throughout his life, including through his entire 80s, to try to reduce the risk that we all face from nuclear weapons. We'll stick up a link to his post announcing his illness and a recent interview that he did with the New York Times. He ends that interview with the following. I thought that it was pretentious to say publicly that I have pancreatic cancer, but my sons both thought that I should share the news with friends and that that would also be an opportunity to encourage all of my friends to continue the work we've done for peace and care of the planet. As I said, my work over the past 40 years to avert the pandemic has been a great experience, and I know that the prospects of nuclear war has little to show for it. But I wanted to say that I could think of no better way to use my time, and that as I face the end of my life, I feel joy and gratitude. The 80,000 Hours podcast is produced and edited by Kieran Harris, audio mastering and technical editing by Ryan Kessler and Ben Cordell. Full transcripts and extensive collection of links to learn more are available on our site and put together by Kitty Moore. Thanks for joining. Talk to you again soon. Bye.
PodcastTranscript AI