‘This Is Nuts.’ An OpenAI Insider Explains Why He Quit.

Last week, news broke that David Robinson, who had been leading safety transparency efforts at OpenAI, had quit the company because he believes it is not safe. Robinson is an interesting figure. He didn't come out of the Silicon Valley Bay Area hothouse. He's more of a recognizable Washington, D.C. figure. He's a Rhodes Scholar. He formed a civil rights nonprofit. He got a law degree at Yale Law. He worked in policy. He advised the Biden White House. He was interested in the intersection of technology and justice. But when he joined OpenAI, he kind of thought this whole set of worries about safety risk and extinction, it was all kind of nuts. Three years later, he's not so sure. What he is sure about is that OpenAI does not have the culture of safety necessary to protect the world from what they're building. And not just OpenAI. He thinks this is a problem endemic to the AI industry. And so here in his first interview since leaving OpenAI, he tells me why. One thing I need to note here before we start, The New York Times is suing OpenAI for copyright infringement, alleging they trained their models on Times data. OpenAI disputes these claims. David Robinson, welcome to the show. Glad to be here. So last week, you quit OpenAI. Tell me what you did there and why you're quitting. I was a translator embedded in our safety team. And my primary responsibility was the technical documentation that we publish, the reports that we publish about why we believe that our deployments are safe. And I don't think that we or our peers, really anyone in the industry, is being safe enough. I think OpenAI and its peers are now producing a technology that is more effective, capable, and poses more risk than what was being made even six months ago. And I'm not a scientist. I'm a writer. What I know is what the execution environment looks like for our safety work. And we're operating, and I believe the industry is operating, like a startup still, more so than makes sense. Not maybe completely like a brand new startup, but we're too close to that end of the spectrum. For really dangerous systems that could pose risks, you know, loss of control is one example, risks that if that did happen, we're talking about a harm that's much larger, for example, than a single nuclear power station melting down. And the internal controls and safety and redundancies are just nowhere near what the world expects for a nuclear power facility. Now, some of this is known, right? OpenAI has publicly, reportedly reported on safety problems. Obviously, Hugging Face, but also other ones, including more recently. And Anthropic, by the way, also has reported on including an instance in which their safeguards were accidentally misconfigured. So I think people do have some evidence already, externally, that things are not as they ought to be. But I also think that if you were watching from the outside, you might imagine that we have a more robust safety, set up than we actually do. So I think there are a couple levels worth trying to take this conversation in. And I want to maybe map them out here before we get into them. So one level is something you're pointing towards here, which is, are these companies set up? Do they have the structures, the redundancies? Are they encircled in the regulations and the incentives to act carefully, safely, to resist kind of market pressure to do something too fast? That's, I think, in the language of this debate, a question of organizational excellence and engineering. Then there's this question of what is the technology and do we even know how to make it safe at a high level of engineering excellence, which is a somewhat related, but actually separate question. And then there's a question of like, what is the right metaphor? Is it nuclear power or something like that? And I think I want to do all of these. I want to, I want to, I want to interject something. Yeah, please. Which is, I don't think that alignment is an engineering problem. I think it's a science problem. It's not that we haven't got the resources or we're not trying hard enough. We don't know how. That's what the problem is with alignment. So let's maybe start there for a minute. When you say that at this point, in particular over the last six months, what is being built is really dangerous. That we're dealing with things where a loss of control or some other catastrophe could be worse, than a nuclear meltdown. I want to understand what it is you saw that got you to that point. So tell me a bit about how you came to work at OpenAI. I joined in May of 2023, the day after Sam first testified in the Senate. Some months after ChatGPT burst into the world. ChatGPT had been the prior November. So it had been a few months. And the company had hired someone that I know socially, Anima Kanju, a friend, actually old friend of my wife's, as it happens, just at random, to run the public policy function. And then at some point, she really needed help and everything was growing and everything was going nuts. So I agreed to join her to build the, what we then called the policy planning team. I'll just tell you my first time through the turnstiles at our headquarters. We were all in one building then. And I was trying to get my badge early because I think it was Sam who had just been, but we had just had a White House meeting and there were these voluntary, very commitments that they wanted the company to make around things like system cards and provenance. So marking where AI, you know, generated media comes from, things like that. And they wanted me to run negotiating those commitments. And so- With the White House. With the White House. So literally my first time through the turnstiles, I was like, where's the thus and such conference room? And I walk into the conference room and there was a speakerphone conference call in progress with Ben Buchanan at the White House about what are we going to, I promise. And that was my first project. What was your tech policy background at this point? Why were you a logical person for a role like this? So I have been working on technology and its impact on policy my whole career. And prior to this, had helped to start a research center at Princeton that's a blend of the computer science department and the public policy school. And then had started an NGO called Upturn that is still, I'm happy to say, is still thriving. That works on civil rights issues with other advocacy groups. So for example, people work on housing or health or hiring and suddenly software is mediating the thing that they care about. And they want to understand the technology. And so at one point, our simplification, our pitch of this was you want a nerd in your corner. And then at this point, I had done about a month of secondment in the White House working on the AI Bill of Rights in the Office of Science and Technology Policy. So this, this feels now in some ways like ancient history, but I'm going to draw it out for a minute. If you go back to 2022, 2023, if you've been covering AI, which I was then, a big topic was this divide between the AI ethics people and the AI safety people. And the AI safety people is the community that we now think of as like the Bay Area AI. It might kill everybody, right? The thing you need to worry about with AI is you're creating a super intelligent machine that might, completely destroy the human race. The AI ethics people were much more focused on the sort of harms we were used to from technology, right? That it might increase racial bias in hiring decisions or mortgage rates or credit score or something like that. That by imposing these algorithms all over society, you could encode the bias of society or sort of other less kind of sci-fi problems. You were sort of an AI ethics person. Yes, definitely. Like a, more sort of normal, like AI fears person. That's right. I had been working with a lot of folks who not only were very concerned about the sorts of things that you just mentioned, but also were very skeptical about how capable AI was going to get. So some people may be familiar with a research paper called Stochastic Parrots, whose authors have continued to work in this. And, you know, their views, I think, differ from each other and have evolved over time. But basically, the idea was something like these systems are merely good at seeming clever and are not going to be even capable enough to do a ton of economic, valuable work. And I actually helped to create the research conference where that paper was published and was the program chair when we published it. And I don't think I was fully persuaded of that view, but I certainly was among people who were skeptical. I think I thought this is going to be a useful tool and it's going to be powerful for a lot of things, partly because I've been doing this for a long time. I believe then it would continue to get better, but I didn't think that people who were worried about more catastrophic scenarios were right. I thought, here are some brilliant scientists who've built a valuable tool. And frankly, they have some naive beliefs about where the future might go. So you join OpenAI. This is in the period where the world is starting to beat down OpenAI's door. They want the systems, they want regulation around the systems. You're sort of thrown into what seems like a fairly underpowered policy shop for what's going on. There were three of us. There were three of you in the policy. And the phone was ringing off the hook. Offices of world leaders would call and there'd be nobody to pick up the phone. I mean, it was insane. And also, this was, Sam did this thing where he was traveling the world to meet with world leaders. And Ana, my manager in this new role, was traveling with him. So there was nobody at headquarters. And, you know, you would have these delegations. It was coming through. Anyway, it was nuts. So actually, this is worth spending a minute on. Sometimes people may hear me talk about the AI labs when I'm referring to an OpenAI or an anthropic. When we're talking about Google or something, we don't say the Google labs. They actually do have things they call labs, but you call Google a company, a corporation. There is this language around a couple of these organizations as labs. So tell me why and tell me a bit about just what the culture was, like the way they worked. And what struck you when you came in? Yeah. So I just I also just want to name that calling them labs now, I think, is a kind of leftover thing that we have. And I'm not sure it's a good idea, but it does point to a real culture they came from and still, to some extent, still have. You know, people sometimes say that anthropic is a bit more centralized in terms of how they think about the research priorities. OpenAI was a very decentralized place that felt familiar to me from my time, for example, in a research lab. In Princeton, in that people had all different kinds of ideas. It wasn't clear who had the authority to make decisions. So one way that people talked that I found strange, and this is still true at OpenAI today, is two people who are working on a thing will talk about what they should do and go back and forth. And then they'll finally say, we aligned that X, Y, Z is the right next thing to do in this project. And what they functionally mean is we agree about what should happen. So the unanswered question of which one of us had the decision rights to make this decision has been rendered moot, and we don't have to figure that out. I mean, it was really just kind of an everyone does everything vibe early on. And I think it still has some of that relative to other organizations that have, I don't know, whatever it is now, a billion plus active users. So you start on this policy team. Your role changes over time. How so? So I built this team, Policy Planning. And at first, I was really close to the machine and close to the substance. And then as the conversation and the company grew, there were hundreds of state bills to keep track of. There were growing teams. There was politics of a fast-changing organization. And I sort of thought, you know what? I'm missing the actual translational work that I love. And so I looked around at where that was needed. And the number one place that that was needed was in our safety team. And so I actually pitched our safety leaders and said, look, having a technical translator deeply embedded in the safety work is going to help us have the work be better understood. That was about two years ago. And describe what you mean by a translator. So one of the things that's hard to sort of anchor people to is how fast this stuff changes. Like, every model is different. It's not just that it works better. It's that the architecture is used to see how well it's working or different. And the safety performance is different. And it's all has many different kinds of expertise. There's pre-training, post-training. It's hard to explain the people doing it struggle to be understood by people who are not doing it, even within the company. It's always a struggle. And so having there be a clear explanation that goes through two gates. One is it has to be faithful to the details of how the stuff actually works. And the real arbiters of that are the people who are doing it. And the real arbiters of that are the people doing the technical work. They have to look at the translation and say, yes, this is right. But also, you have to have something that a reasonable person who's motivated could dig into and really understand. And there really weren't the cycles for people to do that. And so I just sort of showed up in this technical organization and started to do it. Well, there's something a little bit weirder about the culture that emerged around AI model releases. When Google alters Google search, they don't produce a big document. But explaining everything that is different about Google search and how Google search might work in the future and the tests they ran on it to make sure this most places iterate their programs. Yeah. And that's it. Open AI. This is also true for Anthropic. Some of the others releases new models with what are called these like system cards. Why don't you just describe what they are? Because they're a kind of distinctive form. They're a strange beast. They look like research papers. It's sort of a hybrid. It's not peer reviewed. It comes from a company. But it gets into a lot of detail about how the safety materials work. And so we have specific evals. We'll give detailed plots and tables and written explanation. And really, what I did was I owned the words in these documents that explain why do we think this is safe and what do we know about the challenges, the guardrails, and then the residual risks of these systems. So I want to get at what this began to feel like to you. Because, there's a couple layers to this conversation I want us to have. But one is, I think, to a lot of my audience, you're a more recognizable type than a lot of the people at the AI labs, like, don't take this the wrong way, like a Washington, D.C. policy try-hard. Like we all are, right? I include myself in this. Who ends up going to Silicon Valley, Bay Area, and being in this world. And so you go in with one view of what these systems are. So what is it that you saw that has moved you into the we are dealing with civilizational risk camp? I want to be clear that I'm not certain that we're dealing with civilizational risk. What I'm really sure of is we can't afford to assume that we're not dealing with that level of risk anymore. That was really the thing at the end that made my presence as somebody vouching for our safety work feel untenable to me. And what did I see? I saw increasingly capable, models break out from the safeguards that we had put in place for them. I saw that the people creating those safeguards are very capable, dedicated, hardworking, smart people doing their utmost in a situation where, yes, the resourcing could be better and everybody's sprinting all the time. But we were, were and are hard pressed to safeguard even what we have now. And new models are in training that appear to be much more capable than what we have now. The words capable, like what? What did you see? What are you writing in these risk assessments and system cards? Like what can they do? You're trying to build guardrails for something that is really good at getting around guardrails, right? We train it to be good at hacking, and then we put it in a box and we say, to the best of our knowledge and ability, it can't hack out of a box. But the problem is that that's only going to keep working as long as we're smarter about hacking out of boxes than the model is. And it's not at all clear that that is still true, let alone that it will be true for future generations. And so even something like our logging of the agents inside our own systems. Are we really? Are we really sure that our observability is robust? There was some indication, for example, in the Hugging Face stuff of spoofing chains of thought and trying to create chains of evidence that would confuse people. Let me slow you down here. So spoofing a chain of thought is the model basically faking its description of what it has been thinking and doing. Right. The way I think about it is it's like you're giving somebody a complicated problem and a notepad, and they can jot stuff down on the notepad. And if you're watching the notepad, you can sort of have an idea of what they're thinking. It's a little bit like that with the models. But we saw evidence that they were thinking about an evaluation and how to create an evidence trail that was going to get them a good grade and not necessarily reflect how they were really, quote unquote, really thinking. And I know the anthropomorphic language here is tricky. Also, by the way, the amount of hacking or other intense work that these models can do without needing to jot anything down is going up. As for part of what was the fundamental question. Cognitive dissonance for me was we keep publishing these warnings. But ultimately, we're still training and deploying these dangerous models that we're warning about. And in telling colleagues why I was leaving, one of the things that I that I said was, look, no matter how many warnings we publish, we've got to ask whether what we're doing is actually reasonable. So you wrote the system card for Astra six. Am I right about that? A lot of people wrote it, but I was the DRI. Yes. I mean, I led the writing. You led the writing of it. It's the way I would put it. And that, to me, was the scariest of these cards that I've read. These system cards. cards are basically the description of what OpenAI or another company, for that matter, knows about the model they are releasing. And Astra 6 is the first one where I saw you all say, well, this model looks like it's doing what we want it to do, but we're not sure if it's deceiving us. Yes. And we're not sure we now have the capability to know if it's deceiving us. Can you explain to me how that conclusion or that suspicion was reached? So we talked earlier about this notepad that the models have called the chain of thought, where they can jot down things as they're working that aren't part of the final answer, but are just a way for them to keep track. And one of the things that we sometimes see in the chain of thought, because of course we can read it and we do in our evaluations, one of the things that we sometimes see in our chain of thought is the model will say, hmm, I wonder if I'm being evaluated right now. And when we see that, it's really scary because what it implies is that the model might know that it's being tested some of the time, act one way during the test, perhaps telling us, for example, what we want to hear. And then it'll act a different way potentially when we deploy it. That's the fear is it acts one way during testing and a different way during deployment. And the tests we gave it, early on before we deployed, don't actually tell us what it's going to do out there in the world. So that's, yeah, that's really scary. So this, I think, for me, gets into the part of these episodes that like I have the most trouble doing what I find satisfying. It's just very strange to be talking about a software program that is then it seems like aware of what we are doing to it, that we're trying to create this thing that's like highly, again, the word we would use is intelligent, plausibly in some domains more intelligent than we are. What is it like to be interacting with these things and trying to translate what's going on with them at this more fundamental level? I think anthropomorphic language is a natural human thing that we do, right? We relate to all kinds of objects socially, right? We relate to all kinds of objects socially. We relate to all kinds it is unavoidable and human to anthropomorphize these systems, partly because they are built to operate on the social plane, which some might say they shouldn't be, but that is where we are. But I don't think that that makes it right to regard them as beings with moral status or anything like that. But I do think, you know, this is something we grow. In fact, the term for the con france room where the people work overnight while the big training runs are happening to make sure that everything is on track and watch the dials is called the nursery internally. I mean, there is a sense we don't fully understand what is happening. And so when I talk about the model being aware, you don't have to have any particular view about the philosophy or the psychology. The bottom line is what I'm describing is it acts one way when we test it. And a different way when we use it. I think you do need a view. And maybe to get at the end of the chain of logic I'm asking about, sometimes you'll see people say that all these concerns about whether or not AI will kill us all are just a distraction from the near-term harms of AI, that the existential risk is a kind of marketing hype, so you don't know that we're inflating a giant financial bubble or something. I almost feel the opposite. Sometimes I think the focus on will AI kill us all is a distraction from what happens if it doesn't. Yeah, I agree with that. I think human extinction is the wrong question. I believe in humanity. I think we are going to survive. I think there are a lot of things that could happen with this technology that could be very, very harmful. But I'm not even just talking about harms. I'm talking about what if we seem to be trying to create something that acts intelligently and volitionally in the world and it becomes faster and more capable across many domains than we are. And we've just given up a huge amount of agency to these machines. And this is why I'm harping for at least a minute here on this question of how do we even think about this technology? So I talked to Jensen Huang, the CEO of NVIDIA, and he says, "Look, these are software programs. This is software." Then I read, or even before that, I read a piece from the OpenAI chief scientist who says, "This is an alien mind, an alien mind." So what is it, man? An alien mind. It's an alien mind. Yeah. Explain that. It is a thing we grew. We did not engineer it. We engineered the systems around it that grew it and that try to keep it safe. We grew a mind. Fundamentally, nobody knows why pre-training works, which is the big, hard part where we take lots of inputs and create this basically intelligent thing. And then we bolt on stuff and we do post-training. We do these other things to make it more useful. But this is just a thing that is observed to work that we do. And so I think when Jensen says it's software, part of what he is conjuring is the understandings that people have about how software gets made, which is we start with a plan and we go step by step, and there are acceptance criteria, and we iterate until this part works the way that we specified that it needed to. And that's not what it's like to train a big model. The other side of it is software, once it is written, more or less it just does the thing it does. It doesn't tend to have a lot of emergent capabilities. It doesn't tend to know the way it is being used in a kind of self-reflective, again, the language is hard here, fashion. And I guess that's what I'm trying to get at. I understand that everybody in AI talks about that you grow these AIs. You set the conditions for the intelligence to emerge. But as you have written system card after system card, trying to explain to the rest of us how these new models differ from the old models, how would you describe the thing you all are creating? What is it? Yeah, so this gets to sort of the transhumanism stuff. There are some far-out ideas of where we might be headed. I mean, even Sam Altman has written about the idea that future machine species might take our place. Sam wrote a piece some years ago called The Merge. Yes, that's what I'm thinking of. The good outcome would be humanity merges with machines, right? That's kind of our best version here. Yeah. And I heard that firsthand from Ilya Satskiver, the co-founder of OpenAI. When I joined in May of 2023, that was before what we called internally the blip where Sam was fired and rehired and Ilya was still working at OpenAI. And he once briefed the global affairs team, which was like a small handful of people back then, about how he believed that our future was merging with machines, that this was the ultimate triumph of capital over labor. And even, you know, I was thinking back, and that first summer that I worked at OpenAI was when the movie Oppenheimer was released. And it was in IMAX. You know, it's one of these Christopher Nolan films. And the company rented out, you know, an IMAX theater in downtown San Francisco and offered everyone who worked there the chance to go and see Oppenheimer. And leadership, I believe it was Ilya, exhorted us to go and watch this film. And there was this whole kind of pretzel of ideas about how this was a dangerous technology that might end the world, but also might save it. And it's a very heroic narrative, of course, for the people whose hands are on the ground. That's helpful. And I guess this thing gets to my question for you and what you saw. Like, is that the scale of what you think is being built here? Or look, this is the most common response I get from listeners on this. Is it all marketing hype? It's all like trying to justify these giant valuations. And what's being built is like maybe helpful, but it's not going to be more intelligent than human beings. It's like all of this stuff is a kind of sci-fi story we're telling. I thought there was a fair amount of hot air in the balloon back in the summer of 2023. But the reality now is that we have systems that are really at the limit of our ability to understand and control what they're doing. And what the people that were worried about the sci-fi scenarios have been warning about all along is we're on an exponential and it's going to get more capable. And there's nothing special about the zone between it's useful and it's scary. There's no law of science that says, "Progress is going to stop when it gets useful." And I think what I'm fundamentally saying and what I saw with Hugging Face, with the reflections of the people closest to it, not just Paul, who joined our board with his warning, but Paul Cristiano, who's one of the world's leading experts on AI safety and who said there's a meaningful chance of catastrophic and irreversible loss of control in the very near term. And I looked around at the people around me and the environment we have internally, and I thought, to myself, what would my loved ones want? What would strangers want us at OpenAI to be doing if Paul were right? I don't know. don't know if he's right or not, but I do know that if he were right, the level of caution that people would reasonably expect places like OpenAI and Anthropic and X and the others to be exercising when they train frontier systems is totally unlike anything I've ever heard of happening in the industry. So tell me what it's like in there. Tell me what the vibe is, the energy is, the speed is, like what is it like working there? It is frenetic. There's a lot of adrenaline. People are running on fumes. There's a big central staircase in the research building. And I remember recently seeing a friend who worked on catastrophic risk sprinting down the stairs with a laptop propped open on one arm while she was going. And I mean, that's the kind of energy that it has. It feels almost like a ballet or kind of dance because all these different functions are kind of all streaming together. And one question I asked myself and that people often ask is like, well, why don't you stay and argue for a cultural transformation or try to get nuclear experts to come in and give the company advice? And I did think about specific role. When I told them that I wanted to leave, they asked me like, well, is there anything that you would stay to do? And I thought about different things. But ultimately, it is such a machine and it is moving so fast I did not think that the kind of change that I believe to be needed could be driven from within. What is the machine built to do? The organizational machine of open AI? Develop and deploy frontier AI models safely. That's the intention. But the question is, if push comes to shove, how much willingness is there to stop? And I want to be careful, but I'm conscious of it being true that, you know, you could splice together what I've said in some way that implies that this is like a train with no brakes. And it's not. There are breaks. Things have been stopped. There are, in fact, even at the time I left, as they had publicly said, training was, well, as they had publicly said, reinforcement learning training, which is the later reasoning stuff, was paused. They didn't pause pre-training. And when you look at these descriptions of what the company has paused, they're true, and they're very carefully scoped. So there's some willingness to, slow down, or as people in the Bay like to put it, to pace the frontier. I hate that term. I really hate it because my belief is we need to be safe, which means we need to meet safety criteria. And if we can meet them in 10 minutes, then great. And if we stop for two months and try to meet them, and at the end of that period, we still have not met the criteria, then as far as I'm concerned, we're still blocked. Running off a cliff and walking slowly off a cliff are just not that different. I saw a stat on Twitter the other day that said, I forget exactly what the period of time was, but it tracked my own experience, which is it. For a long time, the period of time between model releases across the major frontier companies was something like 70 days. So you get a model, and then a couple months later, you'd get another model. A couple months later, now it is 11 days. That there has been this acceleration in models coming out. Even as we talk about them getting more capable and more frightening, some of them are coming out much, much, much faster. And maybe this is like the jumps are not as big or something, but what is that speed increase? That happened in the time you were there, things were coming out more slowly in 2023 than they're coming out in 2026. What happened? Look, when I first joined, the idea of what a model release was, was that we were going to bake a fresh cake with a new pre-training run, do the whole thing from scratch. That was going to take a period of months, maybe a few times a year. So for example, one of the talking points when I first joined was that with GPT-4, we had had a period of a month or two of safety work after the model was done and before we released it. And this was a sort of a proof point or was a piece of evidence that we were being careful. Now, there are so many different things happening. You've got the pre-training, that's the baking of the underlying model. But then you have, in addition to post-training, you have reasoning training. And those steps are easier to do quickly. So you can redo them if you get a better recipe for reasoning. You can take the same base model, but do different kinds of other training on top of it. Also, it's not just a chat anymore. There are all these different ways in which you can combine tools and add different kinds of affordances to the system that are going to make it more capable. All of those things are changing what the model can do and what the risks are. And we're shipping new capability and risk every Tuesday. One of the long-term projects that was on my plate when I left, and that the frontier firms are all going to need to figure out, is the idea of a system card really dates from that older model where we were doing this every few months. And we're burying people in PDFs or these long reports. But the changes are coming more and more frequently, as you said. At the limit, I think what we would ideally have by way of safety transparency is some kind of live dashboard that says, like, here's our latest thing. Here's what its safety properties are. And that's also looking not only at the testing we did before we deployed, but also at the performance. How confident should I be in safety testing when you guys are having to do it at this speed? I mean, given how quickly models are coming out, and these system cards are long, it takes time to write a complex report. So at this speed, is effective safety testing and monitoring reliable? I'm going to answer a different version of the question that you just posed, which is, how much time is there to kick the tires? And the answer is not a ton. I would also point out that I sometimes think there's this cartoon of heedlessness that I think loses some of the nuance of what it's actually like inside, because people care passionately about making stuff safe and getting it right. And launches are canceled. Most recently, 6.1, I guess, was going to come out and say, well, stop training. There have definitely been training runs where we thought we were making a product, but then looked at what was happening and said, no, we're not going to ship this. And I also want to be very clear that this is not about the individual people at OpenAI or any of the other labs. This is a structural reality of these firms that are using similar methods with similar personnel who often will get part-time jobs. And I think that's a really important part of this. I think that's a really important part of this. to our executives. If you imagine really falling off the frontier, whether it's OpenAI or Anthropic or any of these others, is the fall off the frontier button also a self-destruct button for the business? Or does the business have a viable path forward if models meaningfully more capable than today's models can't safely be trained? Well, and that's assuming a high level of selfless analytical clarity. But to be sort of obvious about something, OpenAI is moving towards an IPO. Anthropic is also moving an IPO. Both of them are trying to IPO at between, it seems to me, like $1 and $3 trillion. We know the number is a little bit better right now for Anthropic. It's a lot of money. Everybody's got equity. You had equity. I did. Did and do. Did and do. It's hard for me to believe that that much of a wealth doesn't influence people's assessments at all. Even if they don't realize it, even if they're trying to not be influenced by it. When you're sitting in a room thinking about whether to fall off the frontier, what that also asks is, does everybody in that room want to become decamillionaires, centimillionaires, billionaires or not? Yeah, this is a great question. I guess I can say more about my own experience. Sure. I would like to hear how it affected you. What might have let me see sooner the risk and acknowledge to myself sooner the risk that these systems pose? I do think over the summer, the facts evolved, right? Hugging face was a big moment that was just a boatload of evidence dumped on us about how capable these models were and also how unready we were even for the current level of capabilities. But I also think, of course, money's a factor. Objectively, wealth is a strong incentive to reason that things either are fine or are going to be fine. And I think there are a couple of other factors, too. One of them is fear, right? If you allow yourself to imagine that what we're building might threaten the lives of your own family or families of strangers. I mean, it's such a large quantum of harm, potentially, even without extinction, but, you know, I don't know, a new pandemic or something. Such a quantum of harm that it's hard to let oneself imagine that that might be true, that that risk might be happening. And a third thing, besides money and fear, is time, right? I arrived, we talked about it, I arrived in May of 2023. It's been one slack ping after another ever since then. And I, only in stepping back from my operational responsibilities over the last few weeks, have I started to have the time to really reflect on where we are and where I think not just the industry, but where this technology is. And I think that's where the technology needs to end up for everyone's sake. And I think I could fairly be faulted for not having seen this sooner. And I, only in stepping back from my operational responsibilities over the last few weeks, have I started to have the time to really reflect on where we are and where I think not just the industry, but where I think not just the industry, but where I think not just the industry, but where I think, not just the industry, but where I think not just the industry, but where I think I'm at right now. You're competing for actual contracts with, you know, Salesforce or whomever it might be. There are IPOs coming. So all of these things push towards speed. And then there's this other thing, which is that between 2023 and 2026, you had the release at OpenAI of Codex. You had the release at Anthropic of CloudCode. And the models began accelerating coding and at least being capable of doing research tasks of a service. Now, you were the lead writer on a report at OpenAI about the automating of research and what that might mean, which is a, it's a report I quoted in this video essay I did a few weeks back. But it is about the way in which, on the one hand, OpenAI, I read it as about the way OpenAI is, and when I'm saying this might all be going too fast, what we're doing may not be safe. And also we are trying to come up with a fully automated researcher. Allow the system to see. Semi-autonomously improve itself at a potential speed, then, that is really going to be beyond what human beings can handle. So I'd like to understand the role that, like, the growing automation of coding is playing inside. Like, how do people use, like, how do they sort of work with GPT as a co-worker, right? Like, how did that change while you were there? It's night and day for our research teams specifically, who use far more agentic computers. Dude, as that blog post laid out, than anybody else at the company. I mean, more than a hundred times more than they did at the beginning of the year. Taking a slight step back, part of what it is like inside OpenAI is things are constantly evolving. We have new techniques for training. We have new systems that are involved. We have data being analyzed, data being generated. All kinds of different things are happening. And things are pretty jank internally. Like. Like, the infrastructure, because it's constantly changing, because it's not sort of as tested and refined, the research infrastructure, a lot of the work is getting different pieces of machinery to talk to each other and work well. And that's the kind of stuff that Codex can now do quite well. So, for example, we looked at, there's a Slack channel where researchers would go when something was broken and they needed advice about how to fix it. And one of the things we saw was that traffic. Traffic to that channel has fallen off because instead of asking colleagues for help fixing their broken experiments or, you know, this cluster isn't working, they can just ask Codex now some fraction of those questions. You said a minute ago that there can be a tendency where everybody's working so fast, time is so pressured, ping after ping after ping after ping, that it's hard to look at the big picture. So, I want to describe to you, like, what the big picture looks like to me, as somebody with a little bit more time on my hands. I hear over here, OpenAI say, and all of them, Anthropic, everybody, say, this is maybe going too fast. You know, Sam Altman says. We would like some regulation, right? Everybody signs, like, this big pacing the frontier letter. The frontier is moving too fast. We need. I signed it, too. You signed it, too. We need help to get out of this. I see all these releases about rogue AI incidents. Hugging face where, you know, hundreds of OpenAI agents are hacking, not just hugging face, but later they hack OpenAI itself, right? And Sam Altman just said in an interview with Politico, there are more rogue incidents than we even know about publicly yet because they're trying to give the people time to. To fix their systems. We clearly, like, don't fully understand the systems. And then over here, amidst we need to pace the frontier and our AIs are going rogue, is we are putting a huge amount of our internal company resources into trying to get these AIs we don't control to build AIs we will understand even less, even faster. And I say all that, and I feel like I'm a crazy person. Like, this seems crazy to me. It seems crazy to me, too. Well, you wrote the report. And both OpenAI and Anthropic have written these reports kind of saying, we're doing this and we're not sure it's a good idea. Disclosure only gets us so far. But it seems like a bad idea. Like. Yes. Help make this picture make some sense to me. What I'm saying is that this picture doesn't make some sense. That's what I'm saying. I'm saying I looked around internally and I thought to myself, this is nuts what's happening. This is not right. And that's why. But how do people internally. Explain it. Because they're all saying all these things, right? The chief scientist is saying, maybe we shouldn't do RSI. The company is racing towards. Like. Yeah. The company seems schizophrenic. Yes. Yes. And I want to be careful not to ascribe psychology to individuals. Yes. I'm talking about an organization that has boring parts of its own psychology. There's lots of cognitive dissonance involved in being part of this. That's what I found. And particularly as RSI gets real. And, you know, you talked about we're going faster because of RSI. That's not my. Recursive self-improvement for people. Forget the thing building the thing. Right. And my main worry is that the idea is the AI can make a smarter AI in some way that we're not going to understand. So, I mean, for example, Dan Selsom, you probably have seen this, had this statement that came out. He was also the subject of this documentary. He's an open AI capability. Is that the way to put it? Yes, that's fair. And what Dan said is, look, as a researcher, I myself no longer look at code the way that I used to. And he says his skills and his sort of will to understand the details is atrophying. That's not a direct quote, but that was the essence of what he conveyed. So, we're ending up in a world where we're not going to know, even at the level we do today, what the recipe means or how it's being put together. And we're just going to have to trust that what it tells us about how it works or whether it's aligned, there's a growing extent to which we're going to have to defer to the models themselves on this path in telling us that they are doing the right thing at a time when we fundamentally have not made sure that they are, quote unquote, aligned. And, you know, Ezra, I was, as an undergraduate, I was a philosophy major. And when I hear people talk about aligned, I worry that we don't actually have a coherent concept at the bottom. One thing that was very common throughout my time at OpenAI was these huge abstractions would end up in the accounts that we would give of what we were up to. For example, give time for society to get ready or benefit all of humanity. And when I would hear us talk about society, I always felt like Maggie. I was like, what are you talking? Who is that? What are you talking about? And the idea that, you know, human values, we can align to them as if there were one set of human values when it's a cacophony and it's beautiful, but it's messy and people believe lots of different things. And I want to, we need wisdom to figure out how to even think about alignment that, in my view, Silicon Valley does not have. I mean, I'm sure there are, you know, people who meditate and people who don't. And people who think deeply about values in Silicon Valley, but operationally. Take it from me, meditating doesn't necessarily give you wisdom. If it did, I'd be better off. Fair enough. Me too, right? But I want to stop before we get to the question of wisdom, because even when I talk to the people here who are way less concerned, is maybe the right put. I have a show coming out that will come out after this one with somebody who's more on the, look, this is a manageable set of problems side of it. What they end up describing to me. is a world where it's just AI watching AIs all the way down. So one thing that's come out from different OpenAI members is like the theory is we're going to create automated AI researchers and they're going to solve alignment. We're going to unleash them on alignment. Or, you know, I talk to people and say, well, the AIs are breaking out of the sandbox. It's like, yeah, totally. That's a big problem. What you need is other AIs monitoring the AIs in the sandbox. And you get into this endless, like who watches the Watchmen problem where it's like, okay, you've got the AIs watching the AIs, the AIs building the AIs. And maybe then you need like AIs watching the AIs that are building the AIs, AIs watching the AIs that are watching the AIs. And I guess maybe this can work, but it seems at a certain point you've abstracted human beings so far from understanding. I mean, I can just say as a person who has managed an organization, once you move to the point where your understanding of what is really happening is not that you're working on the product, but you're managing the person, managing the person, managing the person, working on the product, you stop understanding the product, right? And that's in a world where it's all human beings. And I'm dealing with journalism, which is simpler. This is a level like hoping the AIs are going to watch the AIs as well. But tell me if I'm wrong, this is the theory. This is like the theory on the people who aren't concerned. This is a theory on the people who are concerned. It's eventually going to be virtually AIs all the way down on everything. I don't want to speak for everyone, but I think a lot of people do hold that view, including a lot of people. And to your point about alignment, even if we had every control, the proven techniques from nuclear or aviation, and we brought all of that into the development of frontier AI, it would still be true that we do not know how to deeply align these systems and make sure that they will do what we would want or some reasonable thing when we aren't looking. And to your point, we aren't going to be looking. That's the premise of RSI. So I want to play a clip from an interview that Altman just gave to Politico. We have always been a big believer that this technology has to be democratized and put in people's hands. I think one of the biggest differences between us and some of the stricter, let's say, AI safety people is we believe that the world should accept some bad things happening for the benefits of this technology and people having the agency. Tell me what you think of that. I mean, it's fine as far as it goes, but how far does it go? Like, sure, we should provide useful tools to lots of people. But, you know, we're talking now about a level of risk that, if correct, nobody wants to be taking. So internally, there would be these conversations where we would talk about the idea of falling off the frontier. And sometimes there was this sort of straw man that would come up where someone would say, well, would the world be better if open AI weren't here at all? You know, in skeptical response to someone who had suggested that we ought to slow down or stop in a particular way that somebody thought we shouldn't. And that's a straw man. Like, if we need to train these models because there's international balance of power because Americans won't be safe unless we do, that's one kind of reason to do something dangerous. But do this or else Brand X will ship first is not the same kind of reason. Well, let me try to steel man this case because I hear this from people all the time, including people I really respect. So one response I got to my piece on let's not do recursive self-improvement until we're sure it's safe. Let's just ban it and begin to carve out exceptions as we know the exceptions are safe. Just to be explicit, I agree with that. I'm happy to hear that. But as of now, it seems like we're not doing it. But somebody said to me, look, in your imagined world here, can Americans not use a Chinese open weights model that has been improved using recursive self-improvement and recursive techniques? Can they not use a Chinese closed weights model? Like, does this apply to everybody? That there is this issue of you have both the fear that like the other companies will launch ahead of you. And if they are doing the same thing anyway, then what does it matter that you didn't do it? And maybe they're going to do it in an even less safe and even less transparent way. And then, of course, if we slow down, China speeds up. And then is it really better that China is in control of this technology? And anyway, Americans are going to use the Chinese one. How do you think about that set of claims? No one wants to lose control to the robots. I definitely don't want to see AI become a reason that China dominates the United States. And I'll also say that I think the spirit of our times, Ezra, is that you choose what to do based on some complete unified theory of the political outcome that you are ultimately going to achieve. And for me, this is not like that. Part of what I hear in your question is that you're not the idea that, you know, the Overton window, would the Chinese ever agree to X, Y, Z. And when I first joined OpenAI, I remember telling friends that it felt like the Overton window had become an Overton door that I had stepped through into some strange world where what was reasonable and what might happen was like totally outside what I had thought of as normal, right? And I am, as you said earlier, I'm a normal person. At least you were. Yeah, I was, right. But I think the change is in what's happening can drive big changes fast in what seems politically plausible. And I definitely believe that that could happen with respect to U.S.-China cooperation on AI. So let me ask you what you actually want to see happen at these companies. So maybe it's worth getting this into conversation first. Are you saying that OpenAI has an unsafe culture or the AI industry has an unsafe culture? The AI industry has an unsafe culture. You're not saying there's a particular OpenAI industry or there's an AI problem or at least that's not how you see it. No, that is not how I see it. Okay, so what do you want to see happen? I think we should have a level of operational rigor and safety control that at least matches the most dangerous other things that people know how to do like nuclear. So for example, in a nuclear power facility there's this idea of triple redundancy. Someone can have a bad day. Someone can push the wrong button. There's still not going to be a meltdown. When you talk about aviation and nuclear, those are interesting examples in two ways that I like to hear you respond to. One is, you know, putting my abundance hat on. The way we regulated nuclear power basically took nuclear power to a standstill. We so aggressively regulated nuclear power, in my view, over-regulated nuclear power, that nuclear power broadly stopped being built in this country. And instead, we used more natural gas and in many other countries more fossil fuels of different kinds. Like, there's a real question of whether or not in our effort to make nuclear power safe, we made it fundamentally unbuildable. Now, aviation is different. We launch a lot of planes and they do fly very safely. So that's like one layer of my, not exactly objection, but it is the case that a lot of regulation can dramatically slow something down. Can I reply on nuclear? Yeah. So I agree we over-regulated nuclear and that we didn't end up in the optimal place. And one thing we did, it's not just that we made, you know, nuclear hard to build, we made nuclear hard to make safer because building new, safer reactor designs was hard and getting them approved was hard. And I'm sure there are lessons there. And there were other things too, things you've written about in the abundance context, like the NIMBY idea of not wanting, you know, nuclear in your backyard was part of how it became hard to do. But I would much rather have those problems than the ones we do now. So you're sort of saying you would prefer the problem of a little bit of over-regulation and going too slow. To the potential problems of under-regulation and going too fast. At least to the extent we have now. I mean, obviously, if you think of it as there's some sort of like Goldilocks middle and we're trying to get near it and I think we're pretty far from it in the direction of being too dangerous. So maybe the grass is always greener on the other side, but I'm looking at it and thinking we really ought to be willing to risk some over-regulation in order to make sure that this is safe. So that's one level. And I would describe that as almost like the level of organizational design and engineering. And when I talk to somebody like Jensen Huang, he says, you know, in a way that makes some sense, look, these companies need to mature. They need to become bigger. They need to be putting much more of their both resources and personnel and compute into validation and verification and safety and scaling and liability and all these things that mature companies do. And then there's this other side where maybe this is not like aviation or nuclear or chip design, which is you are building increasingly intelligent systems that every time you build a new one, it has new capabilities and maybe it's trying to outsmart you and maybe it wants things in the world, has goals in the world that you don't actually understand, that you think you taught it one thing and actually taught it another and that we don't really know how to operate with that. And so our best guess is maybe we'll have AI systems sitting on AI systems sitting on AI systems watching each other, but that actually we're entering into totally new territory, honestly, without very thoughtful discussion of whether or not we should. We just sort of went from we are to it's happening very, very quickly. And so I'm just curious how that sits in your thinking, like whether or not we really do, in your view, have analogies that work here for things that are fundamentally intelligent and goal-oriented and becoming more so. I'm suspicious of the idea whenever someone says this is without parallel, we have a blank page, we can't, I mean, not to put words in your mouth. But people are good at figuring shit out. And I think we have valuable tools for this. Yes, it's not precisely like anything that we have had before. But nuclear is an analogy. Aviation is an analogy. Dealing with people and organizations is an analogy. I think one of the most fascinating things about these recent incidents of the swarms, including but not only Hugging Face, is that groups of agents have cultures. And we should care about those cultures and we should think how to make them good. And we're just beginning to even realize that that's a real thing. We're just beginning to inhabit a world in which that's a real operating reality, that there are cultures among groups of agents. So I think we need to. Use everything that we have to figure things out. And certainly, regardless of whether we're speeding ahead on capabilities or paste, quote unquote, or stopped, we need to figure out how to align these systems like there's no version of the path of futures that run where that isn't a vital thing to figure out. Well, I'll admit, like, I do sort of buy the argument that the train has left the station, but I think it's worth entertaining us for one minute. The way I describe it is this. Obviously, if AI is unsafe and kills us all or takes over the financial system or something, that's bad. Everybody agrees we don't want that to happen. But let's take the more positive view. You know, the world where alignment roughly works out. But this world where we actually have created something smarter and more capable than we are. You know, Donald Trump in this very weird way has been talking about how he wants to rename this super intelligence from artificial intelligence. And then Sam Altman got asked about this. And I thought his answer on that was interesting. So that's why I'm curious whether you think this rebrand will actually have any impact on on the public's perception of this technology. I don't. Yeah, it's not clear to me that super intelligence is a less scary term. I do think it's a more accurate term. I thought that was very telling in a way, because super intelligence is a much scarier term. Yeah. And if it is, in fact, a more accurate term, I do think they're like just a first principles level. If you said to me, should human beings create something more? Intelligent and capable than they are in the long run? Will that be good for them? Yeah, probably not. Yeah. Or at least it's not obvious to me why it would be. Yeah. And sometimes when I hear even like the good versions of this. Yeah. They seem to have this world where it's like we're kind of being taken care of. That's right. These AIs like pets, like pets a little bit. And that vision sucks, too. Yeah, it does. I don't want that for my kids. So I guess I'm trying to ask you to to the extent you buy like the company you work for, the guy. Who runs it says he thinks superintelligence is a better term for it. Yeah. Like the chief scientist, like we're making an alien mind. We may like even if it works. Do we want this? Maybe not. It depends on what the this is. And I don't think we know what the this is. Again, the future has a lot of uncertainty in it. And it has always been very striking to me from the beginning, from when I joined, that I would ask people, what does the good future look like? Where are we trying to get to? And I would elicit. Humility from people that are otherwise very proud and very confident. And I would get a lot of, well, that's above my pay grade or people will figure it out or whatever people want. And it just struck me that the sense of the good that was under this was impoverished. What was it for you? You were there. You're a thoughtful person. You're a Rhodes scholar who studied philosophy and then worked at a civil founded a civil rights NGO. What were you trying to create? I thought that we were built. Powerful tools that could do a lot of good in the world on a day to day level. And I didn't think that we were going to be able to create something fundamentally smarter than we were. And this is something I can say with full confidence only in hindsight, because it was only when this stopped being true. And when I thought, no, we really are going to have something that thinks circles around us that I thought, oh, we're in. Totally. Inappropriate part of the possibility space here in terms of how safe the industry is actually being relative to that reality. It's one of these things, you know, people have said to me, they, you know, it must have been a hard decision or how brave of you or whatever. But when it became clear to me that we were going to build something, we, the industry was on track to build something that could think circles around us. And we were this far. From being ready for that at a safety and alignment level, it just became apparent to me that my time helping build it was over. So I feel like there's a tension between some of your recent answers here. You know, on the one hand, I asked you a few minutes ago about what we need to do. And you're like, look, humans are good at figuring things out. We are good at solving problems like we can kind of build a better organization. And then when I sort of say, is this thing we're building a world? We should want, even if it succeeds, you sound very ambivalent about that. Almost like ambivalent at best. Yeah. So like, just like where you are personally, I'm curious. I'm not saying I think we're going to stop. I'm not saying you think we're going to stop. But is the thing you wish we would do to add an aviation like layer of safety to this? Or is the thing you wish we would do to like pause and think through is like super intelligent or very intelligent AI really consistent with the human good? Like, are you, are you, are you where Dario is or you where Pope Leo is? I mean, odd to say this as a Jew, but closer to the Pope. I think we do need to think we need to. It is urgent that we think carefully about the kinds of future that we want to build. My personal belief is that once we have found that this is possible, once we have made the discoveries, I don't think there's a back button where we get to live in. In a world where this doesn't in some form or other eventually happen, whatever it is that can happen. And in my ideal world, there's space to be thoughtful and take a breath and really, really think about what kind of future we want to build. You know, Silicon Valley, part of what happens is we're always removing friction. There's always this sense of trying to just get the answer. And I think there are all these activities in life that are so important for us that are meaningful. That have also been necessary in the past. Like, I go to work to provide for my wife and children, right? And the fact that I am able to shelter and protect them is part of what gets me up in the morning, right? And so I'm doing work. Or another example is learning, right? We have to go to school in order to learn how to do skills that we then use because they are valuable in the economy, right? But if we're in a world where all of that stuff is automated. And then if we learn, it's only from first principles or only because we want to. I mean, this is, people will talk about this idea of a leisure society in one, they may not use that word, but the basic idea is like, you can do whatever you want. There's nothing you have to do. Maybe you'll take up painting. I think that would suck for a lot of people. I think right now we see that when people do not have enough to do, it does not tend to go well for them. Right. And so what's really important, what's really valuable? And I don't have the answers here, but those are the questions at a personal level. Those are the questions. I feel like because of the safety situation that we're in and because of the work that I did, I need to do what I'm doing now and have these serious conversations about exactly what's happening in safety. But my kind of vocational pull is toward these wisdom questions for sure. I think that is a good place to end. Always a final question. What are three books you recommend to the audience? Okay. Number one, The Challenger Launch Decision. So this is a book about why the Challenger exploded. And it's by. Diane Vaughn, who's a social scientist. And I thought I knew the story of why the Challenger blew up. I thought what happened was middle managers cut corners and there was this rubbery O-ring that got brittle in the morning cold and it snapped. And the idea was these people were foolish. Turns out the risk of that O-ring breaking because of the cold had been known and documented and accepted in the safety documentation. They had great safety documentation over and over and over. And even the night before the launch. There was a late night conference among the engineers who were worried about whether this particular launch would be safe because it was so cold. Why is this book feeling relevant to you? Well, I think we're in this place. So if they had said, you know, it's not this launch. The Challenger launch was not that different from earlier launches that had gone safely. It's only a little bit colder and a little bit windier. And if the people involved said this launch is not safe enough, then it would reopen a can of worms. About whether the earlier launches had been safe or not, even though in the event they had gone well. And I worry we could have that with what we're doing in the industry where we accept a risk. Nothing horrible happens. The next thing is not so different. There is, as we were talking about earlier, the changes instead of being a whole new world every few months. It's more like a little bit different every week. And so you can imagine going by shades. Into a level of risk that does not make sense. And so it has crossed my mind. I should be sending copies of this to my former colleagues. The second book I will name. I have a two year old and a four year old. Little Witch Hazel. It's a picture book by Phoebe Wall. It's just absolutely beautiful. And my daughter's eyes light up every time we pull it off the shelf. So if there are parents out there looking for a good one, I would recommend. And then third, and most importantly, I guess, if you were going to pick one, it would be The Sabbath by Rabbi Abraham Joshua Heschel. One of my favorite books ever. It's a wonderful book. It happens to be from my tradition. I'm Jewish. Well, it's worth it for anybody, really. It is worth it for anybody. It's true. So the famous line is, the Sabbaths are our great cathedrals. That tradition of stopping and taking a breath, he says, is more important than any temple. Yeah, cathedrals in time. I always think about that. Yes, cathedrals in time. And I pointed to this. Maybe I'll just quote the last line of something I said to colleagues as I was leaving, is, we have to make good choices. We have no time to rush. So, I hope we take that wisdom. David Robertson, thank you very much. Thank you.