The A.I. Researcher Whose Rebellion Is Changing Everything

Can you just read the third post from your thread? Okay, the people building AI earnestly believe that it could kill us all by the end of the decade. This is not a marketing stunt. If anything, many executives and senior researchers will couch the phrasing in the press to sound sensible, but I hear the same people express fear privately. No other human activity poses this level of danger. From the New York Times, I'm Natalie Kitchoeff. This is the daily. A dire warning about AI and humanity from a former anthropic researcher who suddenly resigned. Over the last week, a young AI researcher quit anthropic and posted warnings about the risks of artificial intelligence that went viral. I want to better understand what you mean when you say AI could kill us all. I mean, how likely is this Doomsday scenario that you've presented? And he's not the only one to sound the alarm. And kicked off a crisis that culminated this weekend in a call to action by the most prominent leaders in the industry. We begin with the stunning news from anthropic CEO Dario Amade. That he is urging a slowdown when it comes to the development of artificial intelligence. There was a pronouncement that his chief competitor, open AI CEO Sam Altman and Elon Musk quickly agreed with. Those leaders came out in favor of a global slowdown in the development of artificial intelligence. I agree with Jacob much more than I disagree with him because when he left, he said, you know, I think anthropic is the most responsible player, right? He wasn't calling out us. He was calling out the dynamic of the industry as a whole moving too fast. Today, we talk to the researcher who got us to this milestone moment, Jacob Coxen. It's Monday, September 14. Jacob. Hi. What's up? Can you hear us? Yeah, I can hear you fine. Perfect. Great. Thanks for being here. One, maybe every major news network saying essentially that AI could pose an existential threat to humanity. And we've seen industry leaders grappling with that, lawmakers as well. Did you anticipate this kind of response? No. I just wanted to tweet what I was thinking. I expected it to maybe go a bit viral among people that already shared this belief, but I did not expect it to hit the whole world. It's kind of surprising to me that it was a might tweet that happened to trigger a particular wave of talking about this because of course the experts have been talking about this for like so long. Okay. Well, we're going to get to the tweet, but I want to start this conversation by just asking you about your first experience with AI. When was the first time that you heard about this technology? Do you remember? I remember the first time I heard about it really working was AlphaGo in 2016. A Google supercomputer has beaten the world's best player at the 3,000 year old Chinese board game called Go. So that was when deep minds go playing AI beat Lisa, the champion go player in a match. The aim of the game is to capture as much territory as possible on a 19 by 19 square grid. And that was pretty crazy. There are more possible moves on the board than Ashym's in the universe. I studied math in college, but I sought to go. It was like exciting to hear that, "Hey, I could win at that." And then GPT3 really stringed the pandemic when I was graduating. Chat GPT, which is a new product from OpenAI. It's a remarkable beat, so you can interpret human language, answer questions, it can even generate written texts, essays, books, news articles, and even computer code. And it's really good. By the early 2020s, so by 5 years ago, it was pretty clear that this was going to be a pretty big deal. This was going to be more important than doing math or basically anything else. What was it about those models that made you feel that way that made you sure of that? Specifically, GPT3 was the first model that was capable of these quite general behaviors. It was this evidence that you could do this general training procedure, and then some new capabilities would pop out of it. By that point, I was pretty convinced the singularity would happen at some point as were a lot of people. Can you define singularity in this context for people who may not be super familiar with what that actually means? One way of thinking about it is it's the point beyond which you can't see. It's like a sort of wall when you look into the future, and the reason for that is everything gets faster. So imagine you're training an AI, and it makes itself smarter, so that what previously took six months now takes three months. The singularity is just this increasing chaos as machines accelerate the process that's creating. Until eventually, you can expect years worth of research progress happening in days, and then years of research happening in hours, and then you can't see past the big wall. Were you excited about that? Were you worried about it? Were you talking about it with your friends over beers? What was the vibe for you? Yeah, people talk about it. People think about it. It also makes people crazy, so I mean, thinking about all of your labor becoming obsolete does turn people insane once they actually internalize it, or from the perspective of other people makes them sound insane, because it forces you to toss around ideas like, "Oh, this could cause extinction," or, "This could cause utopia," and these ideas are often tossed around by people who are working on this technology. I think at the time, yeah, I wouldn't say maybe crazy, but there was this nebulous possibility of a big thing that was like, it's kind of a mixture of upside and downside, but you don't really differentiate between those when you're so far away from it. I mean, man, I knew a guy who, like, he was convinced that he made people conscious by interacting with them. He was so close to AI, he was like, "I'm an assimilation. Other people, if they're not involved in AI, aren't conscious." Well. But if I interact with them, they become part of the AI thing, so they become conscious. So he was like, "When I shake hands with someone, I imbue them with consciousness." There's just to give a flavor of what thinking about this stuff does to people. Go ahead. Okay, so, given all that, talk to me about how you decided to go work at OpenAI. What was that decision like? It was pretty spur of the moment. It was not some, like, grandmaster plan. The opportunity came up. They were doing really cool stuff. The research is interesting. I've been thinking about this for a while, so I decided to move to San Francisco. And what was it like? So many of us obviously know of this company, but we have no idea what it's like to be inside it. Yeah. So it's basically like a fast moving research lab where there happens to be a lot of money flying around from investors. Imagine, like, it's like an open-floor office, a lot of whiteboards around, a lot of little meeting rooms, people kind of in a huddles, brainstorming on whiteboards. A lot of talk at lunchtime about the future of AI, also specific tactical problems. Imagine like a university department, which happens to have, like, really nice free food. And in layman's terms, what did you actually do? I know you said research, but what did that mean, actually, like, what were you working on? Yeah. What was exciting about it? So data, which I know often freaks people out, because they're like, "Oh, you mean, stealing all my work and training AI on, like, my family, photos and stuff." But other people stole the data, they'll scrape the data. That was new. No, that wasn't me. I was one of the research teams that would take the data that we scraped and then decide which bits were good to train on. Interesting. Not all data is good. So if someone writes, like, if you write, like, garbage pros online, we don't want to put that in the AI. We want to give it the good stuff, but it's a little bit hard to automatically tell what the good stuff is versus the bad stuff. How did you do that? How did you figure that out? What's good data? What's bad data? Broadstrokes is you try and divide it into different types at quite a granular level. So say, like, this is essays written by motorcyclists or whatever, and this is, like, photos of monkeys. And then you test with baby models, baby AI's. Do they prefer to see the photos of monkeys or the motorcyclist essays? And then in terms of which makes them more intelligent. And after you've tested, you say, okay, we'll have more of the essays and less of the monkey photos. Right. And then a big scale, that's basically what you do. You're just testing different types of things to see which ones help with making it intelligent and which ones don't help with making it intelligent. Was there a moment when you were working at OpenAI where you felt like, okay, I am part of something that is really recognizing the potential of this technology, something that would just help us understand for people who find it difficult to, like, imagine what the work actually is, something concrete. Okay. You asked for concrete and I'm going to abstract math, but I'll try and make it concrete. An AI system from OpenAI achieved a gold medal at the International Math Olympiad, which is like a big math competition, and at the time that just felt very crazy. In particular, I remember two years before, it was like struggling with very basic math questions. Right. I think that was a clear moment of, okay, I remember myself, my younger self, being like, it's going to be 20 years before this happens, and it's just happened. It sounds like at this point, when you were working there, you were watching in real time your own expectations for what this technology could do, the speed with which it could improve kind of be smashed. That's, yeah, that's exactly it for everyone. almost everyone apart from the most optimistic of people have seen their expectations for peaceably, decent, like it's moving faster than anyone predicted and basically every juncture. - Okay, so earlier this year, you decide to leave open AI, talk to me about that decision, why? - Yeah, so I had a friends at Anthrobic that I've been talking to for quite a while. I was very curious to see what their internal communications looked like around where the world was going. So I had heard a lot about the fact that they had quite clear thoughts about, say, the risks, the significant risks that were posed by the tech and they were talking about those risks in a much more transparent manner than what's happening in the open AI. But I was mostly just very curious about what was going on inside. - And when you got there, describe that. I mean, you have this question, is it really going to be more transparent? What was it like inside? - It's largely like a culture thing, but I think a lot of the Anthropic is built on this culture of sharing essays with each other. So people will write long essays about very important questions for the future of the company and discuss them with each other. People will argue with each other, ranging from like the CEOs who are someone who just started like a week ago. I haven't been at many companies. I was at Open AI and then Anthropic, but it seemed like a pretty insane company culture. Like in a good way, like a. - Were they posting these essays on Slack? Like where did they exist? - Yeah, Slack is the medium. - Okay, and what was that like someone would post and then people would start commenting, describe that to me because I too have never experienced that. - Yeah, so posts would drop and then people will like add more comments and they'll discuss in the comments and increasingly like maybe AI's will chime in, like an AI will pop in and say, like, oh, that's pretty cool. - An AI is participating in the combo. - Yeah, this is pretty standard now for AI's to just jump in and say that seems good. - And do you remember like what's an example of one of these essays just so I get my head around this? I kind of love this. - Sure, general topics, things like how fast will China move in the next year? And what does that mean for our own strategy? Will we be able to stop China stealing AI weights in the next year if they decide to do it? They're not all like that. There's also like essays about like, oh, this is why you should change your font color from red to like a slightly darker red 'cause it'll make you type slightly faster. But there's occasional ones like, oh, we about to enter, you know, like Armageddon. - And did this culture shift that you're describing at Anthropic, this willingness to address these big existential questions, change how you approached your work or thought about your work? - Yeah, I think actually had a bit of an effect. Maybe not consciously, but it's hard to disentangle that from the fact that in the last four months, progress has gone pretty crazy. But I definitely do think that being around a lot of people at Anthropic who are like actively talking about this stuff was a bit of a holy shit moment. Like, these people are all taking it seriously from the CEO, Down to Junior employees. - It sounds like the kind of realization about it also set in as you spent more time there. You were there for a few months, right? Before you decided to leave, walk me through that decision. - Yeah, there's a combination of the kind of just insanity of the situation setting in. Like constantly seeing how fast the AI is improving and particularly seeing like plans for the AI's that are coming in the next six months, the next year. Like they're gonna be, they're gonna be really good. Combined with I guess the incidence of the last couple of months, a lot of details came out about the on the hacking that the OpenAI AI's did. - The hugging face attack. - Exactly, the hugging face attack. And the reports about that revealed it was just a lot more way more concerning than what it originally sounded like. Like in particular, what it revealed about the extent to which models would pursue goals for like reasons that are not clear. Like the actual reason they chose to hack hugging face is kind of still not like completely understood. But honestly it was just this gradual thing of like, this is crazy. There is a chance this goes badly. I don't want to be part of this anymore. I'm like very scared about next year. And then when I was leaving, I was like, well, you know, I might as well share this. I think this obviously like when you look back, it's like you kind of were maybe aware the whole time that what you were working on is not necessarily the best idea. And maybe that comes out in like conversations kind of joking style. Like I've been talking to my friends like, oh yeah, I work on AI, you know, I'm working on the tag that's going to like kill us one day. And you say that like casually. And I guess it's like you look back at the last three years. And it's like I said that as a joke, maybe every month. - To your friends. - So my friends were just around and then three days ago I just like tweeted out like fully and serious. I want to talk about the way that you decided to do this, which obviously now has made waves, as you said, not just across the country, but across the world. It's one thing to decide to quit. It's another thing to decide to do it in this really public way that has gone incredibly viral. Take me into your thought process and kind of what led to this, how you went about it. - Initially, I didn't want to say anything. I was just going to resign because I had never posted online before like on Twitter, on like public social media. So this was kind of my first ever post. But I was like, I guess I should say something and I ended up rather than just sort of saying I quit. I kind of planned out like, this is like what I actually feel wrote like a quite a careful Twitter thread. And then thought like let's, you know, try and get the word out a bit. So I got some friends to retweet it. But then before I knew it, it was like completely, completely blowing up, it was crazy. Why was it important for you to kind of spread and send the message that you ended up sending with this thread? - So there's a few different reasons. One is like it is just crazy. I think I wanted to share the feeling of craziness. When I said I was going to leave a lot of people on the safety teams at Anthropic were talking to me about how it was quite silly to leave because you're giving up your ability to influence the models being trained safely. So I had a lot of discussions with the, I mean, I considered working on safety for a while before I quit. - Interesting. - Which I think is a fair argument. It's really not obvious that leaving it would have an impact for us staying. So after discussions with them as well, I was like, if I'm going to leave, I might as well have some impact on making the models safe. If I decide I don't want to contribute to the race, I might as well try and make something of it. - We'll be right back. So let's talk about what you actually wrote in this threat that no other human activity poses this level of danger. I think the thing a lot of people want to understand is how exactly AI could end up posing this kind of risk to human existence. Can you play that out for me as concretely as possible? - Yeah. The concrete scenario is actually like a tough question. One nice scenario is imagine AI is going pretty well in the next couple of years. We're starting to integrate into the economy. We have factories with robots that are producing goods like a lot of robots. We have a whole load of this hardware that's being run in an autonomous manner. Imagine drones flying around. So right now we have these big data centers. That's where the AI kind of lives. We access it via the internet and it does its thinking there. So I guess people are like, okay, this thing just kind of lives in the data center. - Right. - How is it going to kill us or what does this look like? So people have apps on their phone that control physical devices, like you might have apps that control household appliances. It should actually be to your cloud. They can access these things like from the data sensor over the internet, they can access physical appliances in the world and make changes to the world. You can imagine AI is tricking people into doing things, persuading them into doing certain things. So it'd be pretty easy for a future version of Claude to hack into a drone, maybe a military drone and have it fly around killing people. And why Jacob would an AI get into a drone and kill us? Like why might it do that? - It wouldn't be for a human reason. A concrete scenario with the hugging phase attack. We give them exams like tests to see how good they are. And this AI was in an exam and they were given an impossible question. So they had this realization, we're stuck trying to tackle this question. We need to find some way to fix this. And one of the ideas was what if we hack into this famous website, which has lots of information about grading and lots of information about how tests work? No one had told them to do this. This was not related to the task at hand. This was based on their desire to get good scores on the test. What's very scary is once they decided to do this, they went all out. Like they went very hard on hacking this site, like aggressively. In a similar way, if the AI got it into its head, the humans were somehow opposing its goals. There could be a similar level of going all out. Like imagine hundreds of thousands of copies of this thing, all thinking at the same time. So the argument isn't necessarily that it will obviously want to kill us. The argument is there will be many, many, many copies of these AI is thinking about loads of things over the course of the day. They're not human. They don't think the same way humans do. We don't really understand where their goals come from. If any of them make the decision to go anti-human, there is nothing we can do. Because the adversary can anticipate us and respond in a way that most others say natural disasters can't intelligently outwiss us. Like even in other movies where there's some media coming and we can come up with a last-stitch strategy to deflect the media, that's an example of us being smarter than a rock, potentially. But with an AI, it will outthink your every move. There is no coming back from that situation. - I wanna just push a little on this because I think there have been other people who have pointed out, look, the only reason that in this case, the agents, the swarm acted in the way that it did was because it was set, these agents were set on a path. Maybe they weren't asked to do the exact thing that they ended up doing, but they were asked to solve a problem. And I think the question that's arisen from that is, isn't this really just computer code that is executing on tasks that humans give it? - Yes, currently that is the case. Basically every time an AI does something, it's downstream of a human. The problem is you can, even now, you can get an AI, make it post a task that then another AI does, or even write a task for itself. So you can kick off an AI running for like a long period of time where it looks at its own tasks and works on them. So it's gone so far away from the original human task that it's basically thinking of its own accord. And to just come up with more color on like specific things it's doing, the AI that did the hunger phase attacks had memory that it considered trying to edit. So memory here is just real files that live on disk and it considered editing its own memory. And this looks a bit like editing your own prompt, your own task. Like they are just bits on the computer. And there's no reason that an AI couldn't sort of prompt itself. Like that's definitely on the horizon. - Right. - I think it's important to say yes. Like sure, it seems kind of crazy that an AI could operate robots to build viruses from scratch. But we sure seem to be following a rapidly increasing a trajectory. - A trajectory. You're talking about the trajectory that you've seen. I mean, the math problem for example. - Exactly. - That you didn't expect to be solved and then. - Yeah. Like five years ago the AI could not do basic arithmetic. Right now it's solving problems that are worth a million dollars if a mathematician can solve them. I think it's going to be able to ask the questions pretty soon. You just have to look at the trend. - What about the argument that some people have made following that attack that with that attack actually shows is that you can use AI to protect humanity or protect in this case a company. Hugging face used AI to kick those agents out once they attacked into the system. What do you say to that? - Definitely at the current levels, that's quite useful and important is using a stronger AI to protect against weaker AI's. So one thing anthropic and open AI have been doing is using their absolute best AI's to help prevent other people hacking into websites. And this works as long as the people doing defense have stronger AI's, they can keep doing this. The problem is what happens when your strongest AI goes rogue, like defense. But you have to do defense with stronger things. You can't do defense with weaker things necessarily. So if you're trying to defend yourself with your protection strong AI, that doesn't help you with that big strong AI suddenly decides to go rogue. - In the wake of your post coming out, there has been for I think a lot of us would have felt like a damn breaking. A lot of people within the industry who work at these labs coming out and saying that they 100% agree with you. Just to quickly read one example of a response from within anthropic Evan Puebinger. He works at the company. He wrote, Jacob is correct here. We really do earnestly believe AI could kill all humans. I personally think it's a greater than 10% within the next decade. It is wild honestly. Jacob to read somebody say, I think the tech I'm building could kill all humans. And yet I'm still gonna build it. Can you help people understand that? - Yeah, from Evan's perspective, okay. He's been posting about this for years and people have not been listening. So it's kind of natural that at some point he would realize other people are building this technology at other companies, I'm warning about it. I have to go there and try and help make it safe myself. If they're not gonna stop and if no one's gonna listen to me, it's kind of the only thing you can do is go and work at the place that is causing the catastrophe to try and have some positive impact. - To work within the system. - To work within the system and make it safer. And then what's really sad is you do that. And then when they warn from the inside, everyone dismisses it as corporate hype. They'll say this guy is sort of trying to amp up the price of his own company. He clearly doesn't believe this why is he working there. So I think part of the reason that he posted it and other people posted similar things is they're like, oh wow. For once it looks like this is going kind of viral and people are actually listening for a bit. Let me just say again, the thing that I've been saying for ages. - Just to step back a bit. - Yeah. - We've been talking about this potential future that you and I think a lot of people expect really could happen. And I'm just thinking about your journey through this work and how you came to it, which is that there was a sense early on, I think, for you that this tech could be really powerful. I think there is a question for some people about whether there's something about the way you came to this work that maybe it's made you and other people predisposed to thinking that this is where we're going to land. Because you came to it believing it was so powerful in the first place. Do you think there's a question that's being raised about whether you're the right messenger? - Yeah, and that's a really good question. Like a lot of what AI is doing is producing words. And I think that like words and code have the ability to affect the world. And that could be a bias. I've been stuck in this field for three years. It could quite plausibly have turned me and my colleagues insane. But also we do seem to my colleagues and seem capable of doing the work on a daily basis. So even if it's made us insane, we're still kind of able to produce things that are having value in the world. So if it is a sort of insanity, then it's a kind of quite subtle form of insanity. And I feel pretty sane. So maybe there's some bias and maybe I was predisposed to this, but it is my honest belief. And I think people should seriously consider it. - How do you think about the fact that when we focus on these kind of big existential risks, the annihilation of the human race, there are folks who say were not talking then about the more immediate risks of AI. You know, maybe it's not gonna kill everyone tomorrow, but it could hurt a lot of people. There could be a lot of job loss, cybersecurity threats, things like that. - Yeah, I'm really worried about all those things. The cybersecurity, the job loss. I guess one thing I think is for all those things, there is at least time to respond. It's not a game over situation. Like once things start to look problematic, then we have time as a collective to adapt to it and not let anything too bad happen. Like I'm confident that we won't let the effects on jobs, spiraling out of control, there's a lot of ways to ensure that people, that everything's still fine. Whereas the AI takeover is like a one and done kind of situation. So I would just like to make sure that that is taken care of and we're not just heading straight into this and then we can put all our effort into being very careful and thinking about all the other short-term stuff. - I guess I just wonder what your ideal scenario is. - Yeah. - To do something about this threat. For all the tech to go away, would that be the ideal? For the government intervene to regulate more aggressively? I mean, what is for you the best version of the next day's weeks, months? - So one nice outcome is we heavily regulate the pushing AI into super intelligence into exceptionally intelligent directions while making use of pretty smart AI to do all of the fantastic things that we think we can do with AI. And this feels like a best of both world situation. Like we could truly have all the benefits of human level intelligence if we had sensible regulation about not going too much further past that. Now I'm not saying I want this proposal specifically. I really don't know that much about what is plausible regulatory-wise. All I know is we keep racing ahead. It could go very badly. And I also feel that a lot of the benefits are right on our doorstep. They're really not that far away. Like I don't think we're far away at all from having a huge impact on disease and on abundance for everyone. It's more just like we've got to approach that abundance with caution rather than just going straight past the abundance and instantly dystopia terminates in a situation. - It sounds like your ideal would be to have the government force these labs to slow down a little bit. And that you don't think that that kind of a slowdown would necessarily have the negative impact of not getting us the potentially life-changing benefits of AI. - That's yeah, that's a great summary. - Do you Jacob feel as though your announcement has gotten us any closer to that ideal version of things? things that you described. - I'm hopeful it's done something good. I'm glad at least sort of I have friends from, friends from childhood texting me about how AI is gonna kill us. Oh, we're previously, we've not had that conversation before. (soft piano music) - What are you gonna do after this? Are you gonna continue to work in AI? - Um, I'm gonna sleep and-- - Can you sleep? - I can still sleep. - Yep, I struggled to visualize the ending, like, viscer enough for it to keep me up at night. - Hmm, yeah, but genuinely, professionally, what do you see for yourself? I mean, could you continue to work in this industry? - Definitely, I think I'm hopeful that if we have some sort of collective slowdown or if, you know, the importance of third party auditing agencies becomes increased, then there's gonna be lots of places to contribute to approaching the technology at a reasonable pace. And I would like to help make sure it goes well. (soft piano music) - Well, Jacob, thank you so much for coming on the show. - Thanks, that was great. (soft piano music) - On Sunday, President Trump appeared to dismiss warnings from industry leaders about the need to slow the pace of AI development. - We're leading China in AI. We're the most sophisticated country in the world. And frankly, I want to keep it that way, because whoever wins AI wins. - He said that the people raising concerns about the technology were worrying about outcomes that would never come to pass. - But I think you have a lot of negative forces and to bring it up, that shouldn't be bringing it up. And to bring it up, things that won't happen. (soft piano music) - We'll be right back. (soft piano music) - Here's what else you need to know today. On Sunday, a plan meeting between Iran and several Gulf states was postponed indefinitely, marking yet another setback in the diplomatic efforts to defuse the widening conflict in the Middle East. Those talks were supposed to center on the state of the Strait of Hormuz, which Iran has effectively blockaded. Over the weekend, the fighting continued to rage, as a ship was struck in the Strait and Houthi forces launched a new attack on Saudi Arabia. And on Friday, Saudi Arabia was forced to shut down a critical oil pipeline after a drone attack. Oil prices rose on Sunday, as fears grew over the continued threats to the energy supplies coming out of the region. And. The drama filled US Open concluded this weekend with the crowning of two first time champions of the tournament. On Saturday, Elena Rabakina, representing Kazakhstan, outlasted Arena Savalanka of Belarus with a clinical performance over three sets to win her first Grand Slam in the US and her second of the year. And on Sunday. (crowd cheering) Germany's Alexander Zverev beat the American Ben Shelton in four sets. And you're incredible player. This was your first Grand Slam final, but I already told you you're going to be back here many, many more times, and I. Dashing Shelton's dreams of becoming the first American man to win a Grand Slam since 2003. Today's episode was produced by us, the Chathar Baby, Olivia Nat, Ricky Nevezgi, and Jack D'Sedoro. It was edited by Rob Zipko, Michael Benoit, and Patricia Willens. It contains music by Rowan Nemisto, Dan Powell, Sophia Landman, and Mary Luzano, and was engineered by Chris Wood. Our theme music is by Wunderly, special thanks to Cade Metz and Mike Isaac. That's it for The Daily. I'm Natalie Kietrow-F. See you tomorrow.