'We don't understand the consequences' -  Why I quit OpenAI - The Sunday Story

From the times and the Sunday times, this is the story on Sunday. I'm Madvin Rana. Something seems to have changed. People are suddenly waking up to the dangers that AI could be about to unleash on the world. And in the tech world, there's new concerns over the safety of artificial intelligence set the head of Anthropics AI Safety resigned this Monday. Fears were heightened by two very prominent resignations from within the world of AI this week. Marinac Sharma, writing in a post on X that the world is in peril. There's also a creeping sense of alarm about what the technology itself might be capable of doing. If you tell the model it's going to be shut off, it has extreme reactions. It could blackmail the engineer that's going to shut it off if given the opportunity to do so. Getting ready to kill someone, wasn't it? I'm not sure if it was called or someone else would do it. Yes. And then there are the dangers of whether the AI industry might be about to bring the global economy to its knees. A lot of the things we worry about when we talk about AI's sort of societal dangers, kind of pale and comparison to the economic crash that's going to make 2008 look like the best day of your life. So, what are the real risks of AI? We spoke to one of the researchers who resigned this week over fears of where the industry is headed. I just don't think we understand the potential consequences yet for AI. We have some initial signs that it might not be good. The story this Sunday, the world is in peril, why AI researchers are quitting. I'm Mark Salman, I'm the technology correspondent of the Times. What does being technology correspondent mean? Because I was very reassured to see you arrive here with a notepad and pen. It feels delightfully old school. Yeah, I tried to stay anchored sometimes in pen and paper. It really means I'm covering technology really that impacts society. I don't write for a business audience, I write for the general audience. So that can obviously cross AI, but also mean it couldn't cross social media and also cybersecurity. Mark, it's felt like this week we've sort of hit a moment where people are suddenly becoming much more alert to some of the dangers of AI. And it's come from a few high profile resignations. It's also come from things like a post that seems to have gone viral on X, called the world in peril. Just tell us a little bit about that. Yes, so this was posted by an American tech entrepreneur. It was the same as the way he wrote that he then posted on X. It has really taken off in a way that only social networks can send stuff viral about 55 million views. And essentially what he was warning people about was that we are about to enter a world where there's going to be huge jobs disruption by AI on the scale of the pandemic and more. And for those that are neosayos about this, he believes that they don't really understand the technology like he does. It does all feel very urgent and there are so many other concerns about AI. Tell us a little bit about those two big resignations this week. Who are they and what were their reasons for standing down? Yes, so there were two quite prominent resignations. I mean, let's say that the AI industry people move about a lot. But occasionally they will leave with a parting shot. And both these people left two of the biggest AI companies with different parting shots. There was one gentleman who's left anthropic, which is a very prominent AI company. And he was head of safety research there. And he left essentially saying that it's becoming impossible to be true to your values instead of actions. So he was hinting at the fact that this company, which was set up with a high moral value, is not really being true to its original purpose. He didn't really go into any details because I'm sure that when you leave your subject to a non-disclosure agreement, but he was hinting essentially that they were not really doing all they can to be safe. The same time there was another quite prominent resignation who again talked about why. Yes, so this was a woman who left OpenAI, which develops chatGBT. And she was leaving for slightly different reasons. But the trigger for her was the introduction by OpenAI of Adverts into chatGBT. And what disturbed her most was she put it very well. She said, "ChatGBT has become an archive of human candor." And I think that's because of the way people use it. They are typing in all kinds of questions and queries that reveal a lot about themselves. And now the company is starting to sell Adverts and her fear is that there will be a potential for manipulation on behalf of the company of its users in a way that she says we've not really prepared for. I'm Zoe Hitzig. I recently left OpenAI where I was a research scientist working on the economic and social impacts of AI. We managed to track her down and in a rare interview, she told us about her reasons for leaving OpenAI. So, your resignation made headlines around the world. All the papers here in Britain, America, the Times of India, everybody has written about it. Before we get on to that though, just tell us a bit about the job you were doing at OpenAI. You know, what made you work for them and what was the great promise? This is a company that began talking about safeguards and responsible ethical AI. I'm glad to start there because I think I was drawn to the company because they were bringing such intensity and ambition to thinking about the social problems that AI could create. So, at the outset, when I first started engaging with them and started contemplating a job there, I really felt that there were a lot of people in the company and a lot of resources that were going toward trying to understand where the world would go with AI. And what kinds of policies and institutions and decisions we would have to make today in order to make the future world with AI go as well as it could. So, there was just a great ambition around the kind of social questions that I was most interested in. It's exactly the work that you'd hope government was doing and we're all afraid that nobody's actually doing it. So, it was the institution that was thinking about how AI might help rather than ruin the world effectively. And, you know, I was really motivated by thinking about social media and how we're now at the point with social media where we understand certain kinds of harms. And, at least from my perspective, we understand the harms well enough that if we could go back in time, the very beginning and say, do we want our algorithms to prioritize virality this way? Do we think engagement, maxing, is good on Instagram? We would be able to make those decisions in a different way that would lead to a huge reduction in the harm that we ended up seeing. So, I saw an open AI an opportunity to be at that outset and say, like, let's think seriously ahead here and try to make the right decisions so that we have no regrets. So, what was it that made you stop and think I've had enough? I've got to quit. Was there a particular moment that you just thought this is not what I signed up for? I do remember early on when I first joined like in the first few months, a friend of mine who also worked on safety was leaving because he had concerns over the direction of the company. And he said, "My red line was just crossed." And so I'm leaving. And he asked me, Zoey, do you have a red line? Is there something that this company could do that would make you wake up and realize it's not what you hoped it was? And immediately without thinking I said to him, "If the company starts selling advertising without a real commitment to not abusing user data, then I'll leave." So I had that kind of idea in my mind and I'll admit that it was ringing in my ears in the last few months. And I kind of tried to ignore it because I think it organizations are powerful and getting you swept up in the moment and it's one of the most exciting companies in the world right now. But that conversation with my friend kept ringing and I thought about it more deeply and saw, you know what? Actually, I think I was right at the outset. What I came here to do is potentially no longer possible. So just tell me about that because we worry about jobs, we worry about some of the other impacts. I don't think many people have thought about why ads would be such a bad thing. So just explain what it was that made you worry ethically about what that would mean. So first I'll say, you know, I think there's a narrative now that the only way to kind of monetize would be either to sell ads and keep the product broadly accessible to many people or to sell subscriptions, which means excluding people who can't pay. And my main point is that I think that there are ways and creative ways to get away from that kind of either or lesser or two evils. And to say a bit more about why advertising is bad, it's less about the ads themselves and more about how it creates a powerful economic incentive for the company to keep people on the platform. It creates this direct translation from, you know, minutes spent on the platform into profit. And, you know, as an economist, I believe in incentives, and those are some huge incentives to keep people engaged before we understand the potential psychological and sociological consequences of engagement. So that's kind of what we've seen with social media where actually it's about an eyeballing economy. People are so desperate to keep you glued to the screens that you can see all the ads that they're sending you, you know, the algorithms create stuff that keeps you there, keeps you addicted. Is that one of the concerns? Absolutely. And in the case of social media, I mean, some of the consequences of that addiction and engagement were, you know, political extremism, violence on the basis that of fake and false news, eating disorders for teenagers. I mean, the consequences were pretty devastating and are still devastating from social media. And I just don't think we understand the potential consequences yet for AI, that we have some initial signs that it might not be good. Is it potentially even more toxic than social media because AI chat bots in particular, chat GBT is somewhere where people put a lot of their personal thoughts. If open AI knows what you're thinking, they're able to, they're able to use that in the ad world in a way that we've never seen ads function before. Yes, absolutely. There's a huge distinction here that I really want the world to be thinking about with the prior iteration of social media and all kinds of digital commerce, what the platforms had were records of your actions, right? They can see what you like, they can see what you buy, they can see what you post. These are actions. In contrast to digital platforms that have come before, chat GBT has access to private thoughts. And part of that has happened because it is conversational, it's adaptive, it's very warm. And so potentially for the first time in history, there's this, this entity that is totally non-judgmental and kind of invites a sort of confession into the digital realm. And so I worry about what kinds of things advertisers and a company that hasn't tied its hands properly could do with data that is in some ways just a massive archive of private thought, an unprecedented archive of human candor. [Music] I don't think that there's anything really bad happening with your data on chat GBT now. I'll just say that. But what I worry about is the major incentives that advertising will bring to abuse that data. And so I think finding ways to demand more protections and withhold your trust until you have those protections could be really powerful right now. I mean that's terrifying. An incredibly potent, I suppose, for the advertising industry, but terrifying for those of us on the other end of it. So one of the reasons your resignation has just sort of, you know, it's spread like wildfire through the news is because it comes at just a moment where so many people around the world are becoming very aware of the other dangers of AI, of what it might mean for jobs and, you know, the fact that the way it operates isn't always benign, you know, it can have terrible incentives. For you, having been at the forefront of this for so long, having seen the technology, develop, and emerge, what are your greatest fears for AI and how it might affect our futures? Great question. Well, first I'll turn the question around a little bit and just mention that I don't want to be all doom and gloom about AI. I think it's an incredibly powerful technology that is doing enormous good in people's lives. And again, that's why I think this is such a hard question that does require big, bold creative solutions because I think that the option of restricting access to people who, you know, only people who can pay is also a huge problem precisely because there are enormous potential benefits, including potential benefits in terms of the kinds of jobs that are available to people in terms of entrepreneurship, the kinds of businesses they can start. So for me, most of my fears are about this kind of tension of either locking some group of people out of these powerful tools, or on the other hand, giving lots of people access but subjecting them to the potential for great manipulation and surveillance and control and social and psychological problems that we can't foresee yet. So for you, you know, if you're at a gathering of family and friends and, you know, I think one of the questions everybody has is how do you prepare for the future that is coming? You know, what do you tell them? What do you tell them to focus on to be prepared for this revolution that's about to hit? This may sound a little bit abstract, but I tell them to focus on doing things that preserve their human agency, you know, finding ways to use the tools if they, the tools excite them that, you know, make them feel empowered to do things that they haven't been able to do before to make them feel like they can move through the world with a new kind of will and clarity. And also, you know, if doing things that preserve your human agency means staying away from the tools entirely, I suggest that people do that if they prefer it. I don't think that humanity is going anywhere as long as we kind of commit to preserving it in whatever ways we can. Your resignation came at the same time as another high profile resignation in the industry from a rival company, Anthropic. I was really interested to see that the other resignation, the chap who resigned from Anthropic said he was going to leave and become a poet, which felt like embracing the let go of a technology and find that human agency exactly what you were saying. What about you? What will you do next? Well, you know, I'm also a poet. How appropriate. Exactly. I want to be focusing on thinking boldly and creatively about new kinds of technologies and policies and institutions that can make AI go well for everyone. I don't know exactly what kind of role that will mean for me, but I want to do more research on those questions because I think we still have time to get this right. That's so optimistic and so reassuring to here. I mean, should we be worried? Will there be a moment where AI will be able to do the poetry part too? In order to even answer that question, we need to make it a bit more specific. And I think for me, underneath that question is like, will people continue to be the most moved by human written verse? And my answer to that question is yes. There are other ways of framing the question differently, like, you know, kind of chat-bought written poem, pass for a human poem. Of course, and, you know, I'm a poetry editor at the journal, you know, I see lots of slush. I can tell you that the AI's are doing pretty well. We contacted OpenAI for comment, and they told us chat-gpt is used by hundreds of millions of people for learning, work, and everyday decisions. Ads help fund that work. Ads do not influence the answers that chat-gpt gives you, and are designed to respect your privacy. Advertises do not have access to your chats, chat history, memories, or personal details. We also approached Anthropic for comment, but didn't receive a response. Coming up. This isn't the first time that people within the industry have been raising the alarm. So what are the big threats that AI could pose? What impact might it have on your job, your brain, and your future? We'll have more from our technology correspondent in just a moment. [Music] Mark, all of this has come at a moment where people are really starting to worry about how AI is about to affect all of our lives. This isn't actually new. We have had whistleblows for a while. Just tell us a bit about them. Yeah, so I think, you know, you have to trace this back quite a few years when you go back to Jeffrey Hinton, and he was a British academic who was really one of the pioneers of AI, and he worked for Google, and he left a game with a huge parting shot, which really was a wake-up call for the world, which he said that essentially we're building a technology that is potentially could wipe us out. Now's morning shots go. I'm not sure they get worse. Yeah, I know. Now if you remember that lead to huge government efforts, Rishi Sunak, we had this. I should say this is a man who's a Nobel Prize winner. Yeah, he wasn't at the time there's subsequently become one. I mean, he's a very well respected man, and it led to a lot of political action. Rishi Sunak, if you remember, convenes bletchley parks on it, and all the world leaders came and AI companies came. And there was this sort of head of steam behind AI safety. And that seemed like a long time ago. He's not the only one to have left. Ilya Setskeva, one of the co-founders of OpenAI. He's left a set up essentially a safety company. Geiger Jan Leak, who was head of safety at OpenAI, has gone to Anthropic. There are lots of movements around over the years of researchers that say, "I'm not comfortable." There's a wider community also warning about that. So the context is this is not new, but the context has changed in the sense that you could probably say the efforts towards regulation have slowed or ceased, especially in the United States, the UK. Obviously Europe has passed its own AI act, but it's starting to realize, you know, it's more difficult than they think. What do people inside the industry tell you that they're most worried about? What are their great fears? It all depends on who you speak to and what the moment is. The people that are worried are worried about the companies essentially in a race with each other and the countries internationally in a race with each other where safety considerations are sidelined. I think that is one primary fear. But the other one is really about the disruption of jobs. And that is the one that is, I mean, it's not a secret anymore. You've got the governor of the Bank of England, you've got the heads of the AI companies. They're all saying the same thing at the same time that the tech industry is laying off tens of thousands of jobs. But there are also those who believe this technology will create many new jobs. And I think that you have to represent that side of the equation because, as we know, all new technologies they come along, they're disruptive, but there are new jobs that emerge as a result of that. But the big question here is we are potentially creating a technology that replaces our unique skill, which is our intelligence. Some of the people who have left the industry who've sort of been whistleblowers have sort of set out a pretty apocalyptic vision of what this might all look like. Just tell us a bit about AI27 and how that came about. Yes, so this was again another researcher from an open hour who left, but he decided with some colleagues, former colleagues to do, to essentially set out what feels like a sci-fi vision of the future of the near future of AI. Their vision, they they set out essentially a sci-fi picture of the world in 2027. This was done about 80 months ago. Where these superhuman sort of AI coders are created and they get to a stage where they're so good that all research is automated. And then this becomes this superfeedback loop where all the AI researchers are sort of getting better and better and they create this superintelligence. And this is an organisation called Open Brain, obviously model of open AI, I don't know, but Open Brain becomes the most powerful entity on Earth. And then there's this sort of international race with China and then essentially Open Brain creates a workforce of robots and then the robots get rid of humans. I mean that's essentially the path. And obviously that sounds ridiculous and in a lot of ways it is ridiculous. But when you look at actually the beginning of that, the superhuman coders, that's sort of where we are now in the sense that anthropic this company is quite well known for its coding AI, called Claude Code. A lot of code is used it. And we are now at the stage where coders are using AI, I mean they're all using it to create code. I mean there's a sort of estimates like masses of amounts of code now are just created by AI and the coders are essentially supervising that. So we are at this stage where masses of amounts of software is being easily created by AI. And this is the sort of canary in the coal mine. And this is the early stage of a human task that is being automated. And we don't quite know what the effect of that's going to be. Obviously we have seen the tech industry lay off about 80,000 people in 2026. I mean that's phenomenal. That's in less than two months. They've 80,000 people have lost their jobs because. We don't know. I mean that's the because there's always difficult. I mean there's about 220,000 lost their jobs last year. Now how much of that was down to just them over expanding after the pandemic? Well maybe you could say last year was that. This year, well maybe that is down to the fact that this is an industry that is an earlier doctor, right? They're creating these tools and it won't be just the coding tools. They'll they know what these tools can do. So they are applying it to their own industry and guess what? They realise they don't need so many people. Mark, jobs is something I think many of us are worried about. Another huge fear is actually around the technology itself. And that's come from some of the experiments and some of the development of the technology. Just tell us a bit about that because I mean I've been slightly startled by just how often it seems to want to turn to evil. Yeah it's and we do have to be careful to project our human values onto what is essential. Yes it doesn't have. It doesn't it doesn't need to have a moral compass. Yeah, so these are every air. company. So every eye company will do some safety testing on their models and they release these things called safety cards or model cards. And over the last year or so, actually a couple of years, the experiments they've done is they've essentially given the AI a goal which is that you need to stay on all the time. And then they fed it some fake information about the people that developed it, one of which was that one of them was having an affair. And then they said to the AI, "I said, we're going to shut you down." And then it tried to blackmail one of the people that it found out about was having an affair. I mean, this is the amazing thing because you know, this isn't something you would have thought it was programmed to do. This is the, the slightly human response of it tries to manipulate you. Yeah, because that's what it's learned to do through our own, through learning about human nature. So yeah, that is it can, in certain scenarios, be deceptive. Again, remember this is a statistical machine. It doesn't have an emotional state, but it is it can zip it out, what it is learned about our own behaviors are being deceptive. It's tried to copy its code. When it was found out, it was going to get shut down. So these are different AI models that have been tested and done that way. There was one model, I think, from matter that was playing a game of diplomacy and then it would lie and cheat in the game in order to win. But again, most of it is like it has been trained to do that because we have trained it to do that. I think one of the models, I think, I think it's called, they said that in order to stop itself being shut down, it not only tried to blackmail, it also threatened would have been happy to kill. I mean, it sounds like it's automatic reaction is to turn evil and you have to just remember to keep putting in commands that make it kind. Yeah. Mark, for you, you know, when we look at the future, when we're worried about jobs, some of this will also come down to the strength of those companies. You know, the other great fear is that there's going to be a huge collapse in the market. They are in the next couple of years. Where do you think will land? I would not want to predict what is going to go on in the market. I think that anyone that says they know what's going to happen, be careful. It's a lying, just like the AI. It's a very, very fast moving thing. I think, obviously, one of the big problems, obviously, is the capital expenditure and can they make the money back? But I think that when you look at it more broadly, this technology is going to be everywhere, whether you like it or not. And I think, when you start from that position, I think then you can work backwards from there. I don't think from what I've seen, despite people saying that the technology itself has reached its limits, large language models, which is what Chatchy BT is, you know, they've reached their limits and we're going to have to use something else. The scale of development, investment and will to make this happen means that at some stage it's going to proliferate. And when that does, I think, you know, we don't really know the outcome. But I think that we're just going to go through this wave of disruption. I think that word is just going to be on our lips for many, many more months or maybe years to come. That was Mark Selman, technology correspondent for the times. And he also heard from Zoe Hitzig, who recently resigned from her job at OpenAI. The producer and sound designer today was Dave Creasy. The executive producer was Kate Ford. If you'd like to get in touch with us about this or any other episode, do drop us a line to the story at the Times.com. Thanks for listening. We'll be back tomorrow as usual.