Ex-Anthropic Whistleblower Suspects Chinese Spies Inside America’s AI Labs
1h 45m 17s
The speaker expresses deep concern about the unchecked development of artificial intelligence, warning that it could become a global threat comparable to nuclear weapons. They argue that private companies like OpenAI and Anthropic are pursuing AI development too rapidly without sufficient safety or oversight, driven by profit rather than responsibility. Central to their argument is the belief that AI poses an existential risk, especially if it achieves recursive self-improvement by 2027—automating its own development and rendering human control obsolete. They highlight evidence of Chinese espionage at major AI firms, stating that such infiltration undermines global security and suggests that AI progress is already being compromised. The speaker shares their personal experience of joining and leaving these companies due to ethical concerns, emphasizing that their resignation was a strategic act to draw attention to the dangers. Their resignation tweet went viral, prompting political engagement and discussions about AI regulation. Despite being a pro-free-market and libertarian, they believe that AI is an exception where market forces must be replaced by government oversight to prevent catastrophic outcomes. They believe that critical decisions about slowing AI development must be made within the next few years, before the technology becomes uncontrollable. The speaker calls for urgent global cooperation, stricter vetting of personnel, and institutional safeguards to ensure AI development remains safe, transparent, and aligned with human survival.
Whoever controls AI comes with God. I will destroy humans. This tech is powerful tech. And I'm just very surprised that Trump can sit there and say, I'm going to let Dario Amade, I'm going to let some woman have the keys to this powerful tech. I just think they haven't fully internalized how crazy it will get if we don't slow down. Do you believe China is not going to work on super intelligence and they're going to listen to us? If you appreciate that the tech is scary and it's going to feel scary for the people building it, I think China will also realize that. How certain are you that have spies already there? They're not vetting for something that could be a super weapon technology. It's just crazy, it's like it's happening right under the noses of the government. You met people there you think they could be? You know, again, I really don't want to call that names. I haven't sort of had a detail. Forget about giving names, I'll find out. They hated me. They hated me because I was like, for good or for bad, it feels like the end of the world. I was part of something that was going to go badly. And I became convinced that really there was no good plan. If we could slow down AI and speed up everything else, like deregulate drugs, deregulate biology research, deregulate human self-experimentation, I think a lot of stuff, it's just AI in particular, that's the thing that's scary. So in four years of Dari wanted to destroy open AI, he could. I think 2027 is going to be a crazy year. So I'm going to ask an indirect way of helping our guys, our production team that works so hard, get a raise. Do you want to help them get a raise? This is how you can help us out. 49% of you watch our podcast that our editors work so hard, producers, cutters, all of them, our production team, 32 of them that work here, they work very, very hard to deliver a good quality to you. If you enjoy our show and you want to help this thing get bigger, so we can take our talent that we have in house that supports this production all the time. And maybe even give them a raise, then please subscribe to the channel that allows us to get more viewership. And indirectly, it makes bigger opportunities for current team that we have. Thank you so much. So on September 8th, a Jacob Coxon, who's here with us, good to have you on the podcast. Nice to be here. You decide to put on a tweet, okay? You had started your account January of 2026. You hadn't put nothing up at all. Your first tweet says, "I resigned from Anthropic today." I spent the last three years doing pre-training research at both open AI and Anthropic. Neither company is acting responsibly. They are racing straight to self-improving super intelligence and gambling with our lives. More thoughts below, we'll get into that later on. And this tweet gets 803,000 likes. 175 million views. There was a meeting yesterday at the White House, prompted by you indirectly. You got politicians that got involved, world leaders that got involved. A meeting that happened with Gita was here last week. One of the topics of discussion was how powerful AI is going to be. And the funny part that I think it's important for the audience to realize this, and you can fact check me on this, that you let all your stocks go, that you weren't even able to vest your stocks because of the timing that you had, which means any money on the stocks that you also had was left behind for you. If that's correct, right? There was no, the options didn't move on with you. Not at Anthropic. Not at Anthropic. At Open AI, you were there. I have the Open AI. Okay. So you did all of that, and the world kind of stopped for a few days. Till today, they're still talking about it. What did you see that got you to say I have to do this? Yeah, it really was just thinking about next year. Next year and the year after, I think 2027 is going to be a crazy year. And I think about the news headlines. Like so, last year, AI had been in the news. Three years ago, it was very rarely in the news. Maybe for Chad GVT, but otherwise no. This year, it's bam, bam, bam, AI every month. There's a big AI story. Next year, it's going to be like, every day, there's going to be a big AI story. In particular, just imagine, I don't know. I wasn't alive during World War II. But during World War II, I imagined that just the concept of war was coloring everything. It was in the background for every day-to-day thing. Like every news story, war would be in the background. I think it would be like that for AI next year. In terms of what I specifically saw, it is the models. The AI models next year are going to be very capable. In a way where they're going to be able to automate a lot of work. A lot of everyone's jobs, but kind of most importantly, a lot of the AI researchers' jobs. So things are just going to speed up so much that it's going to become a very, very serious thing next year. Probably. So 2027 for you is what's going to be, is a word to use chaotic? Chaotic, definitely, yes. So unpacked it for me, like, is the average person listening to this? What does that look like? Yeah. So there's many different things that could happen. Things that kind of the most important thing is right now, labs are treating this like the companies are treating this like as a normal technology. You know, this is business. They have, maybe you saw opening eye kind of, they have these dev days. They share their products with the users. They're still tweeting like, oh, excited to release this new product. I think this is a super weapon. So soon, people are going to have to realize that this is not like a casual business thing. This is like a super weapon technology that really should be nationalized. And I think next year is kind of probably when that realization really sets in that this is very, very serious stuff. You're saying it should be nationalized. So if it's seriously that concerning, when you say nationalized, you trust the government getting involved to control this then free market. Yeah. I'm not saying I need nationalization. I'm saying that it is so serious that I'm surprised the government isn't nationalizing. Like the government is looking at these companies. They're going to have power. This tech is it's powerful tech. And I'm just very surprised that Trump can sit there and say, I'm going to let Dario Amadeh, I'm going to let some of them have the keys to this powerful tech because normally it's the government that has the keys to this stuff. You wouldn't let, you wouldn't let Mark Zuckerberg have nukes. That's just not a thing that normally happens. Make the comparable of the two nukes and this. So nukes cause world chaos, surely, plausibly, same thing for AI. It's going to be, do you really want a private company to make decisions about unemployment rate, you know, big stuff, potentially, if they make a mistake, someone in a, you know, some regime somewhere could produce a bioweapon, like that's a, that's a mess of, do you want private companies to be in control of all these decisions? Like, to give an example, if you're anthropic and you're training an AI and you're going to share it with the world, one of the things that you're going to do is, you know, things they worry about is what if China can steal from this, and that's a private company worrying about geopolitics, which is a weird, a weird thing like for a, for a private company to make decisions that affect the whole, like the whole geopolitical. Yeah, but some companies become so big that you become a small country. Yes. Like, you have to realize, Zucker's a world leader, up to 0.4 billion people called Facebook. And he's got regulations he deals with with the government and whatever country he's in. He has to buy by those rules and guidelines. That's why he's always on lawsuits because he broke some of the guidelines, right? Yeah. So it's same with Twitter, same with Google, same with YouTube, same with all these guys. So they almost have to be because they're dealing with those types of people, right? Yeah. So, but what I want to know from you is, as we're going through this, because some of this to me is, you know, I wonder if it's fear porn where they're just using to scare a regular person watching this. I wrote it down. I think this is how I see it. I just kind of want to share with you on how I see the fear porn side of this. So one part of the fear porn is, hey, this is so scary, AI is going to take over the world. We need the government to control. Okay, that's one. People are using this because they want control, they want the government to control. Number two, hey, this is so big and powerful and we need government control because I'm anthropic and open AI, I'm eliminating my other companies to come and compete with me. So the barrier to entry becomes more difficult and I get to control. Yeah. So the motive there becomes eliminating future competition. Pretty good move if we're thinking that way, right? Definitely. The third level of fear porn is, well, let me tell you, this thing is so powerful because we're about to go public as big as we thought this is so much more powerful than us. You may want to invest some of your money in there because when we go public, you may 10X, 20X, 30X, your money to the public, IPO goes just massive, right? So those are three elements of the fear porn that I see. Then the other one is, maybe it is as scary as you say it is. The other one is if we don't build a China builds it, you know, and last but not least, if we all stop, how can somebody hold an independent person or China to build this? So of those, if you can one by one by one, address which one you think it is. Yeah. So to go to the original fear porn stuff, especially about the regulation control or anthropic hyping up that product, I think part of the reason my tweet went so viral is many people at anthropics said the same thing, but they didn't just quit. You know, they stayed there. So it's easier to be like, okay, he's staying at the company. He's saying this stuff, but he's just hyping the product. I left in a personal capacity because I am scared. So from my perspective, I have nothing to gain by hyping up anthropic. That's not a, I'm not trying to hype them up or get them like special criteria. I think what they're doing is dangerous. I think it's reckless. I think they should be curtailed and I left to say that. So that's to address the like regulatory capture stuff. Also later we could go into why I don't think it's actually like, it doesn't actually make sense to stoke up this level of fear. I think it's going to hurt their IPO. I think it's going to hurt their ability to stay in control. We can talk about that later. What was the other part of China here? Yeah, the China point is difficult, but I think again, if you appreciate that the tech is scary and it's going to feel scary for the people building it, I think China will also realize that.
that. The people building the technology in China will see how destabilizing it is. And I just don't think that that regime wants to create a destabilizing technology that could threaten their own power. I think actually our incentives are aligned, like we're more similar to China in this respect, so it should be quite easy to cooperate. Well, if you look at the advancement, the separation between us in China, one could say the motivation, I'm going to show you one data. So motivation for China could be, US has such a massive lead in AI. We need them to slow down so we can catch up. There's no way in the world the China is going to sit there and say, oh, yeah, yeah, President Trump, for sure. We're going to shut it down and we're not going to work on it. Okay, great. Can we come and look at your place to make sure you are actually keeping your promises? No, no, just take our word forward. We are not working on super intelligent. Yep. It depends on the nature of a deal. Right now, if we just stopped, they would try and catch up. But I think one really important thing people don't realize is China already has spies at Open AI at Anthropic. China has the ability to hack at any time. If we pull ahead that doesn't give us a big lead, they can just step in and steal our progress at any time. You think catch up the lead? How certain are you dev spies already there? Ninety percent. Yeah, it's almost certain that they have. Do people talk about that at Open AI? Yeah, they do. People have spoken about this. Is it kind of like jokingly, hey, we probably have some Chinese. Yeah, it's like a joking thing because you don't take it seriously, but then it is a it is a serious thing. There must be spies. There is no. Have you met people that you think they could be? No, I think on obviously it's I don't want to like stereotype people based on like, are they like Chinese national? For sure. Yeah. But so I wouldn't say any specific person. But there must be spies. There's no on background check like serious background check for this sort of stuff. Yeah, it's not like a this is another reason why nationalization is so important. You can stop properly vetting and background checking. So it's like, like, I don't think SpaceX can hire Chinese nationals, but these AI companies can hire, you know, foreign nationals like me, which is pretty surprising. Why is that what's the difference? Because SpaceX has some regulatory. Yeah, there's a regulatory government. Yeah, so they can't. So they cannot recruit Chinese nationals for Open AI in Anthropic. Yes. So do you think one step could be to not allow Chinese nationals to work for Anthropic and Open AI? It's a bit sad because they're my friends, but I think that will happen. I think that will probably happen. Are you saying will or do you think that's the right move? I think both. Okay. So it's uncomfortable for you to say that. And I plot that. I respect that. But you at least that shows me there is there is integrity in your belief system that your friends are going to see this. They're going to say, "Hey, Jacob, what are you saying? We're friends." But you think, okay, so so you think if we continue to China almost, they're like, yeah, continue because we have access to everything you're doing anyways. Yeah. So if we continue, they're going to have access to it. If we don't continue, China is going to create a dump stop. So either way, China's eventually going to have access to it. I think so. Unless again, there's some sort of deal. One thing is a lot of that progress right now, not all of it, but a substantial part of it is distillation, which is sort of stealing the data. So they use Claude, they log in, they ask your questions, and they use the answers to train Chinese models. So right now, a lot of their speed is kind of copying the US and stealing from the US. So also if we slow down, it will probably slow down China quite a bit too, because currently a lot of their progress is from essentially copying or stealing US work. Not all of it, they will still catch up and make progress, but there is a component. So they will also slow down if the US slows down. Who else is another player out there that people worry about? Because it seems like it's mainly them to maybe UK a little bit, right? Not the UK at all, sadly. That's why I mean, that's why I'm in the US. It's very sad, but the UK just hasn't. Would you put third? I'm actually not sure who's got the data center build that right now. US and China are just so far ahead that no one is, no one's going to catch up right now. Okay. So go a little bit deeper into the Chinese spy part. You said it is a discussion that people have in those organizations. So what would that look like? If the average person wants to know, all right, what would that look like on how they pass their messages or information over to them? Is it, can you do me a favor and can you type in who the highest ranking executives at open AI or anthropic that are Chinese nationals, highest ranking executives, at open AI or anthropic that are Chinese nationals? Have you looked at the inventory to see how many of them are Chinese nationals? I don't think there are many that are that high up that are Chinese nationals. Again, I really, I'm like, this doesn't make me sad. I'm not trying to like start a witch hunt. I'm just saying objectively. There are probably spies and I think that, yeah. Okay. So let's go over this. So chief research officer at open AI, Mark Chen. Yeah. He's not a Chinese national. He's not a Chinese. And you know who he is. It says Chinese born. He's a Asian American. Yeah. But he was born in China. Yeah. I just, I don't think Mark Chen has a Chinese spy. Okay. All right. And you met him. You've, you've, uh, yeah, I've met him a few times. Yeah. Okay. So, uh, Fiji Simo, I don't think he's a Chinese spy. I don't know. Brad light caps. Yeah. Oh, no. So Dario, I think we know that. No, I think we are clear on these guys. Yeah. You're clear on all these. Yeah. These guys, who are the highest rank or any high ranking people that could be Chinese nationals or you don't know any names? You know, again, I really, I don't want to call that names. Yeah. I don't, I'm not asking you to give names. Yeah. I want to ask you, forget about giving names. I'll find out. Yeah. But I'm trying to find out from you is because the audience is going to go to their own research, right? I think, yeah. Almost all the Chinese nationals are like employees. So not like executives, not high level managers. They will be like employees doing coding, doing engineering, doing research. Yeah. Yeah. If we pull up, I went to an insurance company once in Dallas and I went on their seventh floor. The seventh floor was all the engineers that they had. Yeah. If I told you 98% what was Indian, I never met anybody that was an Indian. Everybody was Indian and they had a lot of their guys that came from what is that institute from India? IIT Institute, which IIT Institute is like a some people even say it's better than MIT Institute. So you saw that and one could say, okay, they're in here in the business. But when you're saying Chinese nationals, India doesn't necessarily have the same motive that China has. One in America wouldn't say China's an enemy, you know, India's an enemy as much as they would say China's the enemy, right. So if you're saying they already have Chinese nationals in there, that's probably a pretty big concern there. And I think what it is is it's representative. Remember I said they're not taking this seriously. Right. It's like it's a normal business. Yeah. Like, yeah, there's an absolute minimum background checks for employees seem important. I think they will come, but if they're not vetting for something that could be a super weapon technology, it's just crazy. It's like cowboy, cowboy stuff. It's I'm surprised it's happening right under the noses of the government. It's ridiculous. By the way, that's a very valid point. Who else do you think they need to be vetting and not allowing to work out companies like that? So I don't know the exact details, but you know, plausibly even I'm a foreign national. So I'm from the UK. I think it's important. A lot of the researchers come from the UK. I do think careful background checks are necessary. What are you looking for? In particular, I guess, you know, no, no compromise. Whatever the, whatever the current like a clearance background checks are, you should probably need that to work on core AI systems. And I think that will happen. But I mean, even if you're doing that, what am I looking for? What would I find anyways? Now these people are professionals at hiding whatever they didn't. Yeah. I know. It's not even for me to be able to find everything about that individual. Yeah. It's. So I'm not, again, I'm not an expert on like the details. What I think is like, they need to take more seriously. Okay. At a bad minimum. Yeah. Fair enough. So that part, that part is made clear. We got that. What do you say to the people that say, you know, David Sacks, some of these other folks came out, very reputable. People respect these folks. And you saw Mark Andreessen came up and he wrote his humanity manifesto with AI and saying, Hey, this is a let free market do its part. This is a beautiful thing. He believes, you know, we can handle 50 billion people living in the world, you know, and we have bandwidth. And this is going to make people more financially free. It's not that scary. Some of those guys said, this is purely used by anthropic and open AI to eliminate future competitors in own bigger market shares. And they're like, yeah, Bernie, come help us out. Come prevent other people from competing with us. What do you say to that? Yeah. There's a few things. One is I think I agree with Mark Andreessen and David Sacks about the positive benefits. What I think where I think we differ is their case. I think we get their world if we slow down. I think even if we slow down AI development, the changes from next year's technology will be so rapid and, you know, significant that there will be huge changes in the way that they're thinking. I think their case for their optimistic fast case is my slow case. And my fast case looks like we have no control over it. We're just watching robots make themselves smarter like every day massively. It's completely out of control. It's not like 50 billion humans. It's humans and then this massive AI economy that's happening without us sitting around watching them do in screw to things we don't understand. I think Mark Andreessen and David Sacks will agree with me when things get crazy because I think we want the same thing. That's my guess. How many people when you work that open AI and
thropic guys that are actually doing the work, not executives at the top. I'm talking to engineers, the mathematicians, the guys like you that are doing the work, the researchers that are doing the work. How many of them
are in their twenties probably half or a bit less than half okay so that's a big number big number yeah so just fresh out of college right in their twenties um maybe more like 30 percent 30 40 percent and is it fair to say that that audience just got out of college not all of them some of them and maybe they're going to lean a little bit more on the liberal side and maybe they're going to be a little bit more afraid of climate change into the world professors brainwashing them and oh my god you guys be careful with these capitalists they're going to be such technocrats and technocracy and you know oh my these just trillionaires Elon Musk is going to take over and do this don't you think they may have a little bit more fear anxiety of the end of the world is coming because they just got out of college yeah maybe um I can say a few things here please one is um I think it's important that Elon Musk himself is scared about this like that he doesn't get scared of many things right he's a you know a Tyson like the guy's an absolute like he just goes for stuff he's been scared of AI for a long time he only made XAI because he saw the other companies doing it and was like well I guess if we're doing this I have to join in because he's Elon Musk he's not going to let these guys you know he's not going to sell on the sidelines uh so it's you know big guys are scared too and then in terms of a junior employees I mean I can speak for myself I like I've never you know global warming has never been a big deal for me it's like I wanted to learn math I wanted to work on technology I wanted more nuclear power I want like build out of energy infrastructure I'm like pro you know I'm pro capitalism pro libertarian almost you're more libertarian definitely and I think that's very important for you to qualify there because a lot of people were putting you as a climate it's because everyone agreed with me was a Democrat I made a post I didn't ask these Democrats like Democrat politicians to agree with my post right I didn't tell them to tweet that so I think I've been like tagged as this NGO kind of dude really the one thing I'm scared about is AI I'm like if we could slow down AI and speed up everything else like deregulate drugs deregulate biology research deregulate like a you know human self experimentation I think a lot of stuff I'm pro libertarian it's just AI in particular that's the thing that's scary huh so so would you say most of the guys and gals your age working at open AI and anthropic feel the same where you do as a mix I think I'm probably more on the like libertarian side I think some will be scared of climate change some will have been I also thought for example the pandemic response ultimately was quite overblown like near the end I think that like it was pretty bad but I think maybe some would disagree so there are definitely some employees like that but it's not all of them it's not as like I think people imagine San Francisco as being full of liberals but if you're gonna accuse like me and the other kind of like kids building the technology of anything it would be like a sort of maybe narcissism or techno narcissism feeling like you're the center of the world but not like a generic liberal fear that's not like the main thing that's happening yeah I'm glad you you you addressed that in you isolated that case because I noticed a lot of information came up that you were with new speak or you know where you were part of this organization that Hillary Clinton Clinton was a part of and a man named Matt something and you know and it's the NGOs and you're involved in that it's so funny it's like it's so funny because at that house it was a house I lived at I wanted to like live in London and I met a friend who was like come stay at this house and it was a house full of these people they hated me they hated me because I was like they're all communists I'm like they were communists in that house and I did not get on with them but it's so funny now because like I'm on this like thing and everyone's like Jacob stated this uh I wanted to do math I wanted to like live in an interesting place with London in London uh that's why I stayed at stay didn't speak nothing you'd never met you've never had meetings with these guys you've never met with the socialist leaders no no Clinton Sanders these guys didn't communicate with you nothing never no so even before you tweeted your tweet out the I know Brett asked you this question as well but did you speak with anybody was it or orchestrated that this tweets about to come out because you started your account in January of 2026 yeah so prior to that for a young guy to not have a Twitter account it's kind of weird yeah well why'd you start why didn't you start it early why'd you start in January and in eight months you think one is I'm actually like kind of ironically I've been quite scared of hosting online sharing my thoughts I think because I had this assumption that you know that my people might pile on or it's like you know I don't like engaging with the immediate criticism of the thoughts I prefer to like discuss them in private with the people that can make changes so this was kind of the first time I've ever expressed myself publicly and that's one of the reasons why I was so crazy is like I always had this fear that if I if I posted everyone would pile on and that is exactly what happened which is quite funny although it wasn't as bad as I thought do you regret the tweet or you've died your tweet I'm glad I tweeted yeah and I'm happy to talk more about the the coordination because there was some planning this wasn't just I didn't just like quit and then type out this tweet and rage quit there were a few steps to it can you can you walk yeah of course so when I decided to quit um first I was on I was making the model the AI smarter right and then I switched to safety for a week I was like well I'm really scared of how things are going I'll move to the safety side try and make it safe and then while I was at safety I spoke to more of the anthropic leadership I spoke to more of the safety researchers and I became convinced that really there was no good plan I was part of something that was going to go badly so I was like I'm gonna quit and then I was thinking about quitting on the same day but I don't know do you remember Leopold Leopold Ashembrane absolutely 436% the first six months and you know yes I do yeah he's a dude and he left opening I and then he wrote this manifesto situational awareness yes so he was strategic when he left he was like I'm not just gonna leave opening I I'm gonna like use leaving to make because he was fired for like leaking confidential information or whatever um he used it to like spread awareness like a lot of people in government read his manifesto a lot of people maybe you read some of situational awareness obviously yeah yeah um so that was he grew to to what 45 billion oh yeah he read him massive riffing came in but exactly a chunk of it out of this counter price and you know yeah I followed the story close so here we go yeah exactly this um this manifesto and he wrote that when he left and I thought well you've got to be strategic when you leave I'm not just gonna I do want to express myself I'm gonna like be strategic make my voice heard have an impact um I did not expect it to be this I thought it would be a lot smaller than Leopold I was just gonna do my best um but I got a I got a connect at Wall Street Journal I was like I'm gonna make a you know Wall Street Journal published this I think you can probably find the article yeah this so um over the start as I also I delayed my I delayed my quitting by a week to have time to prepare for this so it wasn't just the same day I was like I'm gonna prepare I'm gonna write my thoughts properly I'm gonna quit um spoke to the Wall Street Journal I wrote this Twitter thread but uh to be clear I think people imagined this crazy coordinated siop and I had I had some help but I wasn't being like the people were telling me stuff like um I say people this you know my friends were telling me stuff like um include a picture in your tweet because then it will go viral they said you know tweets go viral if there's a picture but I was like I don't want to put a picture it's a it's a serious resignation tweet I'm not gonna like put a picture I got a few friends to retweet it um yeah let's go to the actual actual details yeah yeah it's some very high you know it was it seemed it was some very influential people tweeted it fairly quickly the moment so they're pretty influential these guys are so it was people like um Nathan Calvin and snare snare revenue they're this organization called encode encode AI within the space they're influential oh yeah within the space they're definitely influential the average person may not know who they are but within your space they're very influential yeah right and they're also friends with many of us like our friends with snare you know we've been hanging out before this resignation there's like she's friends with a lot of lab employees uh it's also yeah it's not crazy that when I when I wanted to leave so the journalist was interested because this was the first ever anthropic safety resignation to go kind of big and public that mean a few but they were small this was a big one um and obviously the people that are most interested in like helping spread the message are people who are you know who want to slow down AI so who else were the people that was was anybody that played a role in you writing that did any politician know you want to tweet that no no no so nothing political nothing political all these people that are friends that are friends that are working at an anthropic and open AI or outside of anthropic open AI they were all outside of anthropic open AI and none of them are working at the company no so no one at the company knew you were about to resign they knew I was gonna leave they did not know I was going to uh like make such a bold Twitter thread and how was that conversation that they know you were gonna leave were you like guys I can't do this anymore I'm gonna be out in a few weeks it was a kind of like share in your frustration exactly I was talking but think you have to remember about um anthropic is such a weird place like these guys are scared of what they're building in a way that I think repulses people when they see it they're like this is weird why why are these guys building it and they're so scared but it's real they're really scared so honestly what happened is I said to my you know I said to the head of head of pre-training I said I'm gonna um I'm gonna quit oh I think this we're going too fast I think it's reckless I think we're not gonna get international coordination um you're saying this to who to the head of um this is the head of like the whole of the pre-training organization which is quite a big organization is that a heavyweight person within a company heavyweight yep um so somebody that works directly with Dario exactly some of the would speak to Dario a lot so you're telling them this yeah I'm telling him this I'm telling him it's reckless and what's he to is this a public name or uh yeah and Nick Joseph he's okay um so you're telling him this and and what's his reaction to you yeah so his reaction is composed because he's dealing with these worries the whole time he's he tells me I think this is a bad decision I think you'll have more impact If you stay and work
from the inside. And he said, you're not going to be able to, it's unlikely you'll be able to change people's opinions on the outside. I mean, what he said is something was like, do you want to go from being a, you know, like a, a good AI technical researcher to being just a run of the mill commentator? He said, it seems like a bad strategic decision. This sounds like a good guy. It's a good guy. Yeah. This is not a bad guy. This is a good guy. And he was trying to give you career advice. Okay. And what got you to say, I disagree. I'm going to leave. And I think I can. Yeah. And I said, well, I didn't say that. What I said is you're probably right. It sounded like he was right. I just didn't want to be part of it anymore. I thought when I left, it was an expression of resignation of kind of disempowerment. I thought, I don't want to be part of this race. I'm going to leave do something else. It might actually be worse. It wasn't really a rational decision. It was pretty emotional. God, and these these other friends of yours that you're talking to about the tweet, are these players that are running things and their heavy weights are some of them within the space of AI, but maybe they're not known names. Yeah. There's a few players and then just a few friends. And it's so funny. Again, people imagine this as being a, I think because of how crazy when they think it must have been this very careful thing. Really, I had a few. I had maybe three or four core names to do the retweeting initially. And then after I sent it, right, I sent it. And then I sent a link to a few friends and said, post this in your group chats, share this in your group chats, like spread it around. But honestly, by the time I sent it to one guy and he was like, okay, I'll share it. Let me write a careful quote tweet about it. Let me like carefully write a quote tweet. And by the time he'd finished writing the quote tweet, it had already gone to like 20 K likes, like already exploded. So the whole thing just went so, so crazy. And it like, this can't be planned, right? Like, imagine if there's no PR firm that can make, that can like hack the algorithm like this. Yeah. And when you did this, who was it from anthropic researcher? Somebody supported which and and Wang, right? Is who it was? I think Evan, Evan Hubingo was the main. Evan Hubin was one of them. Yep. That he, what did he say that he agree with you? He said, he thought everything I said was correct because what I, what I said is I didn't actually say my own opinion mostly. Mostly I said other people at the company believe it and continue to work there. If we scroll, it's like to tweet three, maybe. I said, the people building this technology honestly believe it could kill us all. And he backed me up. He said, this is correct. I do believe that this technology could kill us all. They ran an analysis of all the replies of who agreed, who disagreed, 43% agreed, 40% disagreed, 18% were mixed. Yep. And so it wasn't like people weren't agreeing with you. They were agreeing with you. I even think Ilan retweeted and says, I agree with this. I said this a long time ago. I something like that. Yeah, exactly. Right. And he has. He's always said that he reached out to you. Have you spoken to him? He's clearly been thinking about this for a very long time. That's the main. So you guys did have a conversation together? Yeah, we had a chat. This is after your tweet. Yeah, after my tweet. Was it a phone call? Was it a conversation? Or is it a face-to-face? It was the DM. So just DMed you and I agree with you. I DMed him. And then he responded back to you. Yeah, just a little response. And mostly I was saying like, he, I just DM him saying, you know, because maybe we saw we had a little spat on Twizzey. He like said maybe a sigh of that. And I thought I'm going to try and like, because I think he's been thinking about this for like so long. Did he explain to you why he said this could be a sigh? It wasn't not that much detail. What do you think he meant by this could be a sigh of? With the original thing, I think it was quite suspicious that an account with no followers could blow up that model. The reality is he owns it so he can find that exactly if there were exactly exactly. And they weren't. So it was just like an organic explosion based on the fact that it was scary. Yeah, this is this. I kind of regret this tweet. It was pretty stupid. But then again, I had no sleep. And I was very annoyed that I thought Elon was kind of on the side of this. So it upset me quite a lot. Did he say he agrees and then he said sigh up? It's like it almost, there was a 50/50 with them, right? Yeah, so this I think he was just, when I say sigh up, he just thought it was the whole thing was quite suspicious. Like the fact that I had, you know, the communities I'd been in before, the fact that it went so viral, I can see why he thought it was suspicious. You got 300,000 followers within 40 hours. I know it was nothing. Absolutely insane. Yeah. And so now you brought up Leo Pold earlier. Yeah. Leo Pold got the sequencing right. Got fired. Story goes out. Hey, here's my manifesto. You know, when he wrote down, we talked about it earlier. And in it, he says, it's not really such and such stocks that we have to worry about. We have to put our money into this and this and this. The infrastructure that's going to make a lot of money. For the most part, he did well. You know, people made a lot of money and then stories came out about how much of it was on margin and how much risk he was taking, 41, 2 to 1, 3 to 1. But his outcome was, I'm going to quit. I'm going to write this paper. It's going to go viral. I'm going to start a firm. I'm going to raise money. I think GitHub guys got involved at the beginning. He got some big names. I got involved. What's what's your financial next move? What's your financial benefit of doing this? There is no financial benefit. So Leo Pold, I think, do you have nothing new to launch in the next three weeks? I'm not going to, I'm not going to try and know this. I'm not doing anything. I'm going to one way I will exploit the crazy temporary fame of the situation is through having, you know, conversations with lots of researchers. I respect the previously would have made less time for me. And now I can talk more about the situation. And there's work I intend to do. There's lots of work, but I'm not trying to make any more money for us. How much money did you leave on the table by doing this? By living anthropic. By living anthropic. You know, I'm not going to say the exact detail, but seven figures. Seven figures. Yeah. Okay. But let's go a little bit deeper. If you were to state five years and you would have done your party would have moved up, potentially 50 million, 100 million? Five years. That's the problem. So I'm thinking about that. And it's like, it's not going to be the same in five years. In five years, everything looks different. Okay. This is, and I think the people that are talking about they have these sums of money and these plans for the future and they're just not thinking about how crazy it should go. To you, to you, one, you're not a socialist leftist that you belong to this NGO that you're a pro-Bernie Sanders. You're one a capitalist, free market, libertarian. You don't believe in the climate change is going to end the tech though. You know, COVID handling wasn't right. You know, you thought it was a little bit too much. Okay, check. Let's set that aside. Number two, there is not a, I'm about to announce a hedge fund private equity firm. And this is why our private equity focuses on AI regulation and that you're not going to like that. So how are you going to make money? How are you making money right? Luckily, I have this money from an open AI, right? And I think I will get a job. I mean, I'm going to get a job doing some form of within the space within the space. And I think the job will most likely be around predicting where I will go, trying to share that writing with other people. And also, I really, like I said, it sounds crazy. It sounds cultish. Things are going to change so much in the next few years. Unemployment is going to be a problem when we release. We're going to also make a lot of white collar jobs. You know, I have enough money for a few years. And I really don't know what the world looks like. Like, if I'm wrong, I can get a job in a few years. That'll be great. But I don't think I'm wrong. Got it. So for you, if you're hedging of you being right or wrong, if you're right, you have money and you want to do something to make a difference the next 12, 24 months. Yeah. It sounds like that's your time. You don't think five. You think five years is too late. I don't think I don't necessarily that the really bad thing happens. But I think that the critical decisions, the crucial decisions about whether we slow down or not, they will happen in the next couple of years. Yep. If we're going to slow down or not. Exactly. Anything the next two years, there's still time to slow down to prevent the madness. Yes. Why is it specifically two years? If you look at the pace of the improvement, the specific technology that is scary is when it can automate its own development. So when the job of AI research gets automated. And I think all the extrapolation suggests that that could quite likely come in the next few years. This is the whole RSI thing, recursive self-improvement. I think a roughly around that time is the crucial time where the race will start, where China will lock in and it'll be China versus the US. Everyone will be taking it seriously that we like fear. They won't openly and anthropic won't be sharing the models with the public anymore. It will be internal government only full on race. That's when the crucial decisions happen. Whether we go into this race or not, I think that gets decided this decade. You trust the government having this more than free market? I don't trust any specific personal entity. I just think right now the free market is not going to handle it well. And that's why. Why do you think they're not going to handle it well? Yeah, that's a great question. There's a motivation. It's going to be worth money. I think actually if I talk about my personal motivation, right? So there are people who I thought this was dangerous for a really long time, right? Like before I even joined. And when I joined opening, I thought it was dangerous. At what age did you think this was dangerous? 15, 16. How are you right now? 27? Yeah. So 15, 16? Yeah. So in 2015, you thought this was dangerous. Yeah, there was a book called Super Intelligence by Nick Postram that was quite influential. I read it in school. I think like, yeah, so this image of me having worked in AI and then suddenly got scared is wrong. I think I've always been scared. But then it's like, why do you join a lab in the first place? And there are other people who didn't join a lab, right? They were also scared. They may be behaved more ethically. They didn't join. But then when they say this stuff, no one listens, right? I joined the lab and I got all sorts of things that they didn't, you know, like money and status and some like respect for working there.
And then I quit and that's a selfish thing to do to say I'm gonna work at this technology and get the the status from doing this even though I think it's dangerous But then it also gives you the power to make change Does that make sense like this you get power and I think the same thing is like what Dario is doing So you know anthropic have been scared and they said well no one's listening to us We're gonna have to be a player Enter and so Dario's work like they work so hard in the last they've caught up to opening I it's very impressive and anthropic have worked like absolute dogs to like overtake and become the main player and they did that because they're like We need the power and that's the free market right the free market incentivizes you to get your own power your own wealth So you can make change the problem is that's gonna keep happening and People gonna keep doing this. It's the story of opening. I Open. I did that because they were scared of deep mind. They were like we're gonna do it Dario did it He was scared of opening. I he was like I want the power even just like me It's a small story, but why did I work and help it's for my own power and if we let the free market determine in the next few years We're gonna race off a cliff because everyone is looking out for their own their own money their own power No one has the ability to sort of Slow down a few years ahead. It's this like race problem where everyone just like and I don't blame them Right, I don't blame if you're a 22 year old right now and you've done AI I don't blame you if you get a job at anthropic and work there because what else are you gonna do if you're at like How else are you gonna try and? Try and help so everyone is following their own incentives and that's why we need in this case We need bigger regulation because yeah in any of the technology. I'm happy with this. I think innovation Innovation from competition is great everywhere except if the technology is This level of compounding risky. That's when it breaks so 2015 2016 you read the book super intelligence and then You at what age did you start working at open AI? 23 and what did you do prior to that because I knew you were a big math maybe walked through who you were as a math phenom and how you came up Sure, so I did math competitions in school. I was very into math when I was a kid I mostly just did math and read books Then when I was in college, I guess you won you you did pretty well on two the biggest I did the international math Olympia twice. Yeah, and you were you were second and one third No, okay, so I think it's understood. I got a silver medal silver medal. Yeah, it doesn't a lot of people get silver medals Doesn't make you the second in the world. I wasn't anywhere near the best. I was in sort of a I think silver medal is top We can look it up. I think it's top 20% or something. It's a um in the math arena Yeah, I think the the kind of achievement here is getting on the on the national team because I was being selected for six people to do the team Yeah, answers like that you give Tells me your integrity you corrected me to say you're not as smart as I thought you were. Yeah, I mean you realize that's That's that in the audience is watching I understand it, but it's but you know say 99% don't check it you just You just did an integrity check on yourself and I respect that. I value that. I hope the audience does as well But please continue. So you're a you're a math guy. Yep, and then I went to college and studied math I kind of I think I wanted to do math research Which was you know because I yeah, I always love math But unfortunately, I think it was a mix of many things. I kind of realized that Pure math research was not for me It was partly related to AI. I saw that like AI was going to be this big thing that was going to eat a lot of stuff Still felt quite far out. So I've been following it since I don't it 2020 was GPT-3 GPT-3 was this like quite big this happened during the pandemic Which is also my my third year of college So I was sort of in the pandemic with a friend. We looked at the GPT-3 paper and we were like, yep, it's happening This is a big deal. This is where things are going But I didn't know how to break into AI. So I didn't actually like I had no AI background. I studied math So I worked with a friend at a power trading startup. We did a we predicted energy prices Which is kind of it was AI related. We predicted the the cost of power I did that for a bit and then I took a you know There was some personal issues and I took a break for about a year I was on kind of out of it and a little bit I was living in Prague for a while and then Scotland and then finally I got a call from a friend who was Jacob What are you doing? AI is taking over you're like you can do math. You can learn AI Common join open AI's residency program. They have a program which is a six-month training program for people that are Math guys or physics guys to teach them the ropes of AI and make them full-fledged researchers So I sort of said, yeah, that sounds great. I'll I'll do that And that's what I said. You went in and then you got in at six months and then you were there for how long three years? Three years. Yep, three years, which is yeah, I was like pretty junior but for context the company went from 300 people when I joined so about you know 7,000 8,000 now I think so it was like a pretty explosive phase. It was was a Sam Alton pretty accessible when it was at 300 Like were you around them a lot? He's around. Yeah, you see him. You see him. Yeah, you've had conversations with them. Okay How different Did you get that same experience with Dario at Anthropic not in the same way? No, okay So what is your opinion of what Sam Alton's doing versus what Dario's doing? Yeah, how different are they completely different? Yeah, I'm gonna So I think Sam is So Dario is very honest. I think that's the first that you can see everyone can see that he's very open and honest and shares a lot of stuff with everyone I think he's like he's always honest And he's obviously like an amazing researcher and I'm always impressed by what he's like accomplished building Anthropic Sam is like has a reputation for dishonesty. So Sam has a reputation for being a lot more of a Business guy of sort of like pleasing one people of always being quite vague I think the main difference I noticed between Dario's communication and Sam's communication is Sam is very vague about the future He says things will be good in this way and maybe they'll be dangerous But we can model through like a vague thing whereas Dario is like 20% chance this happens 8% chance this happens This is what it looks like and that's a lot more like rough to hear because it comes across as quite abrasive It's like tough to hear that the details But Sam it's like it feels like he's making everyone promises and he could be saying Fake stuff to everyone everyone feels kind of happy But he's actually not really saying anything. That's kind of the main difference I'd say in terms of how they communicate with the with the company What do you think is the end of be all for each one of them? Yeah, I mean again, this is armchair psychology sure not good friends with them. It's just from watching. I think Dario wants to save the world as it were he's got like a he thinks he's the Him and his company are the ones to save everyone I think Sam has a lot of motivations and it's quite messy and chaotic and that's reflected in how messy and chaotic the company is But again, this is armchair speculation. I don't yeah So if one is saved the other one is controlled to be clear when Dario's saving the world I think saving the world and taking over the world overlap quite heavily. Hmm Like you could float with that. You think you're the safe one right and China's building unsafe Or you think the US government's being silly and you have a very powerful technology You know, it's not off the table to consider taking actions against the US government And I think if anyone would do that, it's unthrobbing in the inside Organizations open AI in an anthropic how different are the cultures very different tell me more So one thing is open AI may have changed. I left like four months ago. I think they've started taking stuff more seriously So anthropic has seven co-founders and None of them have left they've all stayed Which is very impressive for an AI company because at a at opening eye Everyone's left they all go almost all the other companies everyone leaves the whole time andthropic retains them and that's because I think anthropic has this They are very clear about what their mission is and they're very clear about how they're going to achieve it and this makes them very dangerous It's a very aligned institution to put it another way at opening. I have felt like People might prioritize their career. They're like individual success Over the mission of the company, which is a very natural thing to do like everyone does this It's like I want to do a project that's gonna make me have an impact. It's gonna be interesting for me. It's good for my career And anthropic it felt a lot more like well for starters the employees call themselves like ants right there The ants in the ant hill everyone's like an a worker ant trying to make the the anthropic you know the mission go well People are very willing to sacrifice themselves for the good of the mission. They'll take on Work that's less interesting that's less kind of Big and bold and exciting It's less flashy and they'll do it at a hit to their own kind of interests for the good of the company And that's because every decision comes back down to like is this going to get us to Superintelligence faster. You've seen maybe Sora. You saw the video platform the opening I made and
thropic would never make Sora because it's not on the core path To artificial superintelligence opening eyes distracted a messy and one guy's like I want to do Sora Where I want to build like a little device and they let them do it because it's not like the same focus and anthropic That would never happen. They're just completely kind of who would shut it down the seventh co-founders I feel like this is not it all agree. It'll be like this video It's not the right direction for for heading straight to superintelligence Which one is harder to work at more schedule more hours? I think it depends on what team you're on probably I think um when I was there I Think it also depends on the stage of your career I think when I was getting into the work was also so different because now compared to three years ago You're running these agents you weren't doing that three years ago So when I joined I was writing code by hand right like I was entirely hand programming and by the time I left No one really writes code by hand anymore your agents your agents write the code Your agents write the code. Yeah, I can talk more about the it's kind of interesting about the details If we watch AI runs the company to give some flavor when I when I quit when I resigned. I was like what's the procedure for resigning like What do I sign and I was like I don't know ask ask Claude So I asked the AI so I
the last AI how I resigned and the AI told me the flow no one knows how to resign you have to ask the AI to do it because the company is run by like the processes are run by AI right everyone asks and it's great it works but that's the extent to which AI is part of the whole company it's like any transaction is mediated by asking the AI to do it people's meetings are recorded by AI summarized you have AI's posting comments on things you write an essay and AI will respond it's like pretty crazy interesting is is Darial and Sam Altman on the way their people look at them between fear respected liked loved how would you rank them like if you were to say people loved Darial they respect them I would say anthropic people love and respect Daria really yes in a way that's not the case at at open AI Darial has very much like a cult of personality and a cult of like a cult of vision as it were anthropic and then at the sam altis more fear I don't know I think he's obviously respected um I'm obviously loved by a lot of people but not in the same Darial's thing is different yeah and and in the company like no matter where you work it like right now you know yesterday Yankees played a red socks and you know they hate each other their enemies yeah right in sports you know we're talking about baseball earlier you know in politics you know Hillary Clinton Donald Trump you know in basketball Michael Jordan of Brown James you know he's a who who was the enemy to Darial and who did Sam Altman see as his enemy yeah that's a great question you have to look at the history of the companies so when opening I was founded the big enemy was deep-mind google deep-mind and Dennis was obvious and they were worried about google kind of becoming dictators they were worried about google taking over so that that was the enemy and then opening I was founded now it's like I'd say there's probably more like Dario's quite has a lot of animosity towards Sam Altman and the reverse is not necessarily true to the same extent meaning Sam Altman doesn't have animosity towards Dario I think less because I think like I said Dario is very honest and I think the sea of Sam is having betrayed him to a certain extent and that's a kind of a pretty permanent lot on the relationship how did he betray him I really don't know the details but that's my understanding again this is from a that's the energy you see that's the energy that's not public I think that's pretty public yeah on on the betrayal between did they do business together I want yeah I mean Dario is a like big big figure opening I before he quit oh that's right so that's that history is from them leaving I remember that one that whole exchange took place so for him Dario's enemy was I'm gonna leave and build something faster than that has he ever expressed the reason for leaving was it because he did not want Sam to have the control of this kind of a technology yes that's what it was I don't think it's good for the world for Sam to control this kind of maybe not Sam is specifically but like definitely a large component Sam and then the rest of opening eye and what do you think Sam looks at us enemy today sort of politician is that a competitor is it handmade in Portugal their craftsmanship is a way of life for those who move through life differently from some that decks to knights that never need an occasion every detail is considered every step effortless crafted from fine swayed and built with builder technology designed to move with you because true luxury isn't about being noticed it's about how you feel when you arrive the FLB icon lofa made for the life you've earned I really don't know okay and what role in something like this right now I mean you saw that they had a meeting at AI at the White House with everybody being there the list was all the heavyweights were there right and part of that was from your tweet that prompted these types of conversations being had now what role does Trump play here does Dario trust Trump does Dario look at Trump as an ally does Altman see it Trump as an ally how did they view that I think everything has changed so much now but I really don't know the I have like quite a very few things that I'm like confident on I think right now honestly everyone is just quite scared of how fast things are going and they're looking for ways to come together I do think this particular this particular record is you know it's a good thing like what they wrote makes a lot of sense by the way at the day I meeting at the White House where it was Dario and Sam Altman present I don't think Sam Altman was I think he was opening a dev day but Greg Brockman was there and he's one to seven no no Greg Brockman is a representative open. Yeah got it but was anybody there from enthrop yeah Dario was there yeah okay once a last time Dario and Altman have been seen together is that like the thing that happens or is that I don't know if you know these videos fly around no like they're these funny music videos flying around now or like oh yeah the dancing that they're doing yeah yeah yeah yeah yeah that's funny that they did something together yeah and that's as close as it's going to get there so one of the things you said in the in the where do you see Dario here see if we can spot him I think he's followed can you find Dario where is Waldo where is Dario I don't see him in this I don't see him in this picture here I think he's on the follow after so so you know go through it so while you were there at anthropic and you guys are having these types of conversations how many other people while they're sitting there saying guys we are you know building our replacement yeah we are building our replacement how many people felt they are building their replacement everyone everyone yeah that's just a fact everyone knows it but is that a fact or do the agents still need to be controlled by somebody like you for now they do for now they definitely need control and for it's not clear exactly when they lose we don't need the human anymore but everyone anticipates it we're like we're gonna be at some point we won't have a job that there's Dario like let you say that's something that is happening and I'm an employee and I know this is gonna be happening again it's like a truth right was there like a life insurance policy guys if this happens here's what we're planning on doing so you don't have to worry about it yeah so anthropic I think in um one of Dario's essays I think it might be machines of loving grace this essay he has a section where he talks about the fact that anthropic will need to make commitments to its employees for if they get replaced and I think people implicitly had the knowledge that if anthropic employees get replaced due to progress they could expect to be taken care of I think people expect that they will stay and kind of be looked after yeah so so but open yeah you don't remember having anything like they've not said anything publicly I think there may be some references to it but nothing I think that's a pretty good idea if you know your people are afraid of something like that hey if this happens we would do this if that happens we would do maybe but it also sets about like what about the people that don't work there what about all the people that are going to get replaced to other companies it kind of doesn't look great if you're looking up for any employees but like at these other companies people are you know losing their jobs and they don't have the same assurances as someone that works a jack up who's the weirdest person that's living your DM slid in your DM after that tweet isn't crazy stuff I had a a lot of um a lot of quite aggressive gay men DMing me gay men just wow did they think you're gay I don't know man they wanted to find out yeah and for the record you're not gay for the record I'm no gay gay men a lot of them in my DMs yeah but like powerful gay men no okay you just gay men period just dudes in my DMs yeah listen I mean there if there's powerful gay men you got to entertain you better fly it yeah that's the good thing this is a billionaire gay man hacer respectfully I'm straight but thank you for your interest I mean you could yeah you don't have to say any heavy any heavyweight politicians world leaders you know billionaires any any names like that I think so there were like there were a few names the whole thing is like I mean lost in the inbox I don't want to say like you must have tens of thousands of yeah it's kind of nuts I just just say what's respond to but um I don't want to say specifics I think a lot of a lot of politicians have reached out and I haven't responded to any of them yet but a lot of people for things like um talking to staffers I think Bernie Sanders as people reached out I haven't responded to so that yet I had an invitation to go to a I think a rally with Steve Bannon and Bernie Sanders that I that I chose not to go to Steve Bannon and Bernie Sanders yeah they're relying on this right they don't the same thing yeah exactly they're not that is a weird rally weird I know weird and I say rally it was like are you thinking about it or I was thinking about it definitely but the thing is uh interesting it just wouldn't look good on head I don't want to associate I don't want to be political right I'm also I'm from the UK I feel like I don't want to just jump into US politics that's just like an odd thing to do yeah that's um and by the way is it true that the story at OpenA like you know if you're going through conflict they bring dolls and stuff like that what is that what is that good I start to realize I fully believe oh I'm threatening like cool like a stuffed animal advisory council I mean I don't think this is happening in the office for say but I believe it yeah I think like do you ever see it or hear it or I've never seen it but I've seen things similar at people's houses right they have like a duck that they used to talk about their problems um I think it's cool I think it's like look dude anthropic is going to be a two trillion IPO these guys are going to eat capitalism alive I'm like they can do whatever the hell they want if they're who gives a shit yeah because these guys are taking over like yeah I mean you got to give them some respects like if they're yeah in the upside between the two just as an investor forget about I know what you're thinking about as an investor between the two is open AI even close to the way they run things that anthropic so I haven't been there for four months maybe it's changed but I would like like why is opening I head of Google it's largely a culture thing Google has a lot of resources a lot of people a lot of great talent but opening I has this focus on building Thanks for watching, Al Alams.
and anthropic is so much more focused. Again, I hear that opening eyes becoming more focused, but yeah, as right now as an investor, I guess I would long anthropic, definitely. - What would you need to see yourself for you to second guess your own fears that you have right now? What would you need to see? Say, you know what, maybe I'm wrong. - Yeah, I would love to see a. So alignment is the science of making them safe, and I would love to see, for example, some good arguments, just scientific arguments that we control the behavior of the AI's. And I think this is doable. So go back to, I think there is research that needs to happen on this, but it's not impossible. Some people say it's impossible. I think you can solve it. The main thing is just next year, the year after, these companies will have to choose between making the thing that makes more money and more power and justifies the insane capital buildout. So I have to choose between that and doing the safety science. That's like a decision. And the free market incentives push you to choosing to go fast, get power, make it better, and skip the safety research. - You don't sound like a libertarian, you know. - In this case, in this case, anything else. Anything else like that. - Anything else you're libertarian, not this one. - Just not this one. - Okay. - So you see, you know, if you right now, we did this after your tweet. If you go on, "ChadGBT or Proplexity" or "Claught or You Ask," what is the likelihood that AI is going to replace humanity or destroy humanity in 10 years? - It's just like 10% or something, 5% maybe? - It's self, it's just 5% to 10%, right? Okay, I've asked this question from Roman Yom Polska, I don't know if you know Roman Yom Polska. - Yeah, I saw your interview, it was great. - Yeah, I sat down with a good conversation. I asked them how, what would it look like, right? What would it look like in your eyes? You're an insider. - Yeah, of course. - Okay, if the world's gonna come to an end of five, 10 years, how would it happen? - First, you have to realize that the world will look different for this to happen. - So it's not like it's the end of the world, it's just gonna be a different world. - Yeah, there'll be a lot more robots. You've seen that we're building out robots very quickly, that we have a kind of robots around. There'll also be a lot of potentially a lot of weapons around that aren't currently. There'll be a lot of building out of, say, drones. Right now, you know, we've seen, like, Russia, Ukraine drones used. - 100%. - We're scaling that up. Imagine in five years, like, these countries are gonna be bristling with deadly drones. And AI is gonna be integrated everywhere. We're gonna have a lot more of a robot economy. A lot more of decisions are being made by AI systems and not by people. So you have to imagine the world looks different than that right now. It feels like we're not surrounded by that much tech. Like, what could an AI do in this room? It could, like, mess with some cameras, mess with some lights. But imagine if just the world is bristling with, like, drones and robots. There's a lot more, like, AI activity happening. At that point, I think it feels a lot scarier. Like, if the AI is in all the science labs, it's hacked into, it's in every system. And it also has all these drones and robots around. Then suddenly, you don't feel so safe anymore, like, also imagine all your decisions are being made by AI. You're thinking about podcast planning or what? Or like, stuff that, you know, that I don't understand. But if you ask the AI about it in the future, it'll be giving you great advice, like strategic advice. Like, here's how to, you know, like, with certain guests to be like extra, his things to say to make them extra persuasive and it'll be better than humans that coming up with like, all sorts of stuff. - I mean, you've already seen some guys who come out and know where with their podcast blows up and they are 90% as AI driven. - It's AI driven. - And some of the personality driven podcasts who used to be bigger five years would have kind of flatlined a little bit. - And it's like, imagine that, but even more. So everyone, you're aware that all your decisions can be made better with AI. You're asking for planning, like, say financial planning for spreading out late, like, whatever sort of business doesn't happen. - It becomes a fad though. And people are like, I mean, sports teams right now, valuations grow exponentially. Because what people are saying is, even no matter how many robots you have, people are going to want to see other people, beat people. We will be more interested and impressed by what people do. Human beings do instead of what robots do. But the AI will be telling you that. If that's true, the AI will tell you to say, like, prioritize people things, because it understands. So just when you're thinking about this world, it's like, we're not making decisions anymore. The decisions are all the AI. At that point, it should be clear that if the AI decided to turn against us, we would have no power. Just imagine at that point, no one's working anymore. Everyone's just asking the AI to run this stuff for them. Maybe some people are pretending to work or like, they think they're working, but really the AI could do it better. And you've got robots, and drones, and labs building out all sorts of crazy technology. At that point, I think you should start to feel like, wow, if humans were the enemy suddenly, there's nothing we could do. - Don't you think that's kind of like every time, every generation, when something newest coming out, there's a fear about what TV was going to do? - Yeah. - Elvis Presley started shaking his hips. Oh my God, everyone's going to start sleeping with each other. Close your eyes, don't look at his hips. This is bad. Music, radio, TV, internet, nuclear war. Oh, everyone's going to use nuclear bombs. It's going to be a nuclear world war three. World war two hasn't really happened. - Yeah. - What's different about this is this the fact that you think, if it all of a sudden thinks on its own, the AI can think they control us, and now we have no control, and they can say, hey, you guys better respond to us or else. I guess what I'll say is I don't know how much of the world works. I've just done tech and math, so I don't know how to run another business or to build anything or anything like this. And also the, I guess people that are working in other industries, they don't have visibility into AI, they don't see it. What I'll say is if you look at just, I think the news headlines are quite interesting, because for even other panics, I don't think what we had is this like build up. I think maybe there'd be like an initial thing, and then it would like be a plateau, or it would be like this TV thing, it would dribble through the same articles coming through, there wouldn't be this this build up. Whereas right now everyone can feel it. I'm sure you can feel it, right? With AI, it's like every year it gets bigger and faster. So I'd say that even though I don't understand the other stuff, and people outside don't understand the AI, I would say just look at the rate. It's just every year it gets bigger, and nothing else has been like that. It's this exponential like coming faster. So really, I think you don't need to trust me. Like I just think in six months, it will become even more clear to everyone. - In six months. - Yeah, it'll be more clear in six months. - I'm packed that. - Yeah, so the models that come out in the next three or four months. - Do you know what models are coming up? - Yeah, I do, and I can't say details, but they're gonna be a lot better at coding. They're gonna be a lot better at automating my research. It will go even more to running these agents like employees. Maybe these models will get released. Maybe they won't get released. There'll definitely be news stories about the decisions around releasing them. It's gonna feel big, I think. - By the way, when you say Dari was a great researcher, your researcher code or we get, engineer we get, what does it mean to be a researcher? - It's kind of pretentious. It mostly means coming up with the ideas, so like which things to code. So like you've got the AI, I guess details of its training recipe, deciding which ingredients to add, deciding on what shape its brain is, essentially deciding how it's connected together. All these like decisions. Honestly, how do you score a great researcher? - Yeah. - He's a great researcher. How do you know that? Does it take years for somebody like people took years to realize Dari was good? Or can you look at a 23 year old and say, this guy's a special researcher? - Honestly, I saw, yeah. - I saw a contract that a young 24 year old got a $1 billion payday from meta. What is, what is special about that guy that meta gave him a $1 million? - There are some guys that are just like, they will, so these training runs cost a lot of money. These AI training things, they cost a lot. Right, we're talking maybe like hundreds of millions more. If you make that more efficient, you're saving the company a lot of money. The capex on these data centers, like hundreds of billions, again, if you make any of this more efficient, you're saving a lot of money. And there are some guys for now that have amazing intuition and work really hard and will save the company a ton of money. - Yeah, like this right here, meta. - I mean, it's just that 250 million out of the lure, 24 year old whiz kid, we have reached the climax of revenge of the nerds. - Oh yeah, and I think it was 250 for 24 year old. - Yeah. - What's special about this 24 year old to pay 250 to him? - So I don't remember this dude. - Go a little bit higher, sorry. - Maybe you look up on, what was his name? - Another guy got a billion out of it. - Look at the, you look at the billion guy. - Yeah, who's that? - I've met him a few times. - Did you go to the billion guy? Do you know the name of the guy or no? - Andrew, something I think. So guy got a billion out, just ask Chad GBT, go with that in a chat and see what it says. Yeah, somebody got a billion out of it. - Oh yeah. - Why though? - These guys are like, you met him. So what was so special about him? - These guys have good taste for research. They know the right things to do. They'll save you a so much money in your big crazy experiments. - Actually walk me through it. So can you systematize that? What does that mean? - So you fix this. - Yeah, sure. Thanks. - Yeah, walk me through it. What does it mean that these guys know how to do it? Save you all this time for research. - Yeah. So imagine you're kicking off a big AI training run. It's cool. So we'll take a few months. And you've got to decide the details of the data and the recipe, these things. Some guys like this guy are just great at making changes to that recipe. So that it becomes more efficient. And if this whole thing, and in a way that no one else can match, no one else can have the same ideas. So these guys do it. - You all want to just become a good idea. And this guy is able to say, it's a waste of time I wouldn't do it. - Yeah, although this guy, in particular, I think it was just great at like, um. - This guy, you know this guy? - Yeah, I met him a few times during training runs at OpenAI. And he was always around. He was like very fast at solving stuff. He was fixing bugs in our training of the AI system. [BLANK_AUDIO]
Snap just fixing them in a way that other people want fixing them. So he was irreplaceable, right? And then um, that's why he's worth a billion dollars And it will give the billion who was the mess opposed made a game a billion out. Yeah, from our thinking machines. Yeah Pernum and they're done which is matter to train rivalry and auto-clawed. So where he left the meta now Like some genius who are probably getting out of package from it has defected to one of its rivals in the AI race I guess I've not been following this is three days ago This is literally today. I guess he joined anthropic which is and Australian tech was reportedly so sought out my meta I want to fill you another salary a package in Front of them had already left the company to join the AI race and who's he go to as the all anthropic? Yeah, yeah, so he went to anthropic so Daria picked them up. You didn't even know this. I didn't know that this happened 30 days ago And so this guy was a whiz fast. He could solve it faster than his peers exactly and how old is he is about your age? I think he's 30s He's 30s. Yeah, I think so okay Yeah, so Now let me ask you can can can the people okay like let's say a hundred of you. Yep, okay Who feel the way you do? Exanthropic X open AI X Google X pick all of them. Yeah, you guys find each other, okay? Can you guys Use the AI against itself to hack into it and destroy the systems? Sorry, what do you mean? So use the AI so think about let's just do an evil like let's devil's advocate is global plane You are so convinced this shit is gonna get bad. Yeah, they you unified together the hundred of you You have a common enemy. AI is gonna destroy humanity. Let's get ahead of it. Let's destroy it ourselves So yeah, I think you the thing you described is basically anthropic that Entity you're talking about is Anthropic like they're a group of people that were scared they organized together they unified themselves together and They're going to take like aggressive action to try and save people that is literally what? Anthropically, yeah, so you're saying Anthropic was those hundred people that unified together to Potentially if they wanted to destroy open AI they could exactly so you mean to tell me right now Anthropic if they wanted to destroy An AI they could no they're not they're not at that level and I'm not saying they're planning that necessarily I'm not saying you're saying they plan. Yeah, I'm just saying capability. Those are two different. No, they couldn't not right now When do you think they would be able to? Yeah, it depends again how fast we go, but end of the decade So four years yeah, so in four years if Daria wanted to destroy open AI he could I think quite plausibly yes, and it depends on how fast we'll go I know right what Wow So the seven co-founders of Anthropic do they all share the common enemy of how they view Sam Altman I don't know. Yeah, I don't know the details. Um, did they all come from open AI or are they all from different places? I think they all came from open AI. Um, you cannot tell me you don't know. I mean it's it's very I think Daniela Amadei. Maybe don't know she did too. Yeah, I think they all came from open AI Jack Clark is also open AI Tham Brown can you check to see if all of the seven co-founders of Anthropic all came from open AI? So the way I watched How Sam Altman handled Ilan it was disrespectful. I'm not involved. I'm not on the inside. Yeah, what's going on But from the outside it sounded like Sam Altman deceived, you know Ilan it sound like Ilan was coming from a good place of open AI allow everybody know what we're building at least That's my impression of it. And then it goes from a nonprofit to for profit and Ilan's like wait a minute What are you doing didn't Ilan give like 50 million hours? He gave some amount. He gave some amount to him to help You know to help build it and then it goes so you the reputation in the market place about open AI is not positive Specifically Sam Altman. We even had a guy here who used to work at open AI His parents came here And and this is the fellow who was the whistleblower who was found dead. I don't know if you remember that story Yeah, we had him here mom came here dad came here. They spend it they were trying to tell us. I'm telling you here's what we think happened with Sam Altman Where some people would say there's an element of Evil yeah in the sky Would you agree with that? I don't know and I will say about the sutra thing so All the stuff I'm saying about taking over in the future. I mean that completely in the short term you mean that completely I mean that it's possible in the future. Yeah, this like yeah, there's no way that they killed Sushi like that that wasn't I 100% it wasn't that's just not a thing opening. I would do you know that. I know that. Yeah How do you know that just from Personal relationships of people to run the company. They this is it's just nonsense. Did you ever meet him? Yes, I'm not Sushi. Yep. What was he like was he? It was great. Just a great researcher great guy I presume he was depressed I that I presume it was a suicide. I think there's no It seems impossible that it was anything else. Yeah And this is someone who's saying I'm also saying the opening and anthropic are racing to destroy the world But this particular act I can say this was I like you a lot Jacob. I don't know who you were But I like you a lot. Let me do more. I talked to you tomorrow. I like you. It's so funny because I didn't know who you were I don't know how you were, but tell me about your parents my parents. Yeah sure. Yeah My father's a professor of medieval German literature mid-Evil German literature. Yes. Why? You can ask him that it's pretty dumb That's what he did. Yeah, that's what he did. Yeah, how about mom? Oh She did some stuff. I don't know. Yeah, I think home and in what values did they raise you it? What what did they teach you? What values did they raise me? I read a lot of books Your character like I want to know about the character like what did they did they tell you hey? You know your man you keep your word and integrity. What did they teach you on what values did they that's but that's what I'm interested in I was raising a Christ as Christian school, but my parents are atheists They're atheists and they intentionally put you in a Christian school. Well a lot of schools in the UK are Christian It's not in the same. It's not the same as the US. Yeah, it's like a bit nicer. Yeah, did that did that the the faith play role on you? I don't think so no No What do you get the values? Where do you get that from this is very impressive? I I think it's maybe a math thing I don't consider myself to be particularly in like I think I'm just trying to say the truth. I try and say true things is the main thing like in math You try and say the truth about stuff, so I'm just trying to say true things right like in this case I don't know yeah, but character is different than trying to a lot of people that are good and math Who are extremely logical, but they find a way to use that against them to to manipulate and have power. I don't feel that from you I'm really not a not a saint in this situation. I've told you I've profited a lot from working at opening I um you profited a lot. Yeah, I made money at working at opening. I think I Like I worked at opening. I even while thinking it was potentially a dangerous thing Why don't you do it because you were making good money because not specifically the money But because the whole situation like you want to be part of it. You want to be working on it. You're not thinking too clearly When's your birthday? What month's your birthday? It was a September 20. So you just had a birthday. Yeah, I just had about eight days ago. Yeah, happy birthday Thank you. Yeah, yeah, because I remember reading 26. I'm like winning turn 27 years old because at the beginning Yeah, when you did this you were 26 years old September 8, right? Oh 27. Yeah, I know I'm 28. Oh, you're 28 now. Okay. I got it Well, two more. You're gonna be old. You're about to be 30 man. Like what are you gonna do from here? You know, it's a response to those those gay men in my DNA Let's off some of them have money and say I need somebody to take care of me. That is hilarious So the question that I asked you and your answer. I'm still in that question because I asked I said if if if a hundred people like you So if you're a hundred people of you united and Decided to go up against the enemy could you break them and you said that's exactly what I'm throwing up again That is very interesting That is very interesting. So they created themselves to play defense against the enemy But if the technology gets smarter than them then they have to play defense against the technology that's got a life of its own now Yeah, and they're aware of that. It seems like Dario where well Is it fair to say that an assessment of Dario as he's a pretty paranoid guy? Yes, he is definitely yes And so that could work in your favor and against you at times Yeah, and I think an important thing with anthropic is they always promised that they would never Overtake open AI at the very start. They always said that they would never push the boundaries They would always be second place. They catch up. They'd make it safe But now they've overtaken and now they regularly release AIs that are better than open AI's which is I think they said they would not do Anything that's just because the capitalism competition. I think that's a component of it I think they also have a story that are if they can get ahead they have extra time to make it safe They have like better AIs to run their safety research. There's a lot of reasons for it But also it is you know, it makes them a lot of money. So who's third in the competition by the way who started the company? Third who's third like if it's them to who's three that's a great question. I think it's a like a far distance third It's not a I mean, it's pretty far at the moment. I think so there's a you know There's massive Google and X AI and I'm not actually Deep enough in the weeds right now to know exactly who's who's third. I suspect it's a like match overall But those three are also around Is a White House reached out to you? Um, no, no They've never reached out to you to just be like hey, what do you think about what's going on even not? I'm not saying the president somebody on the team that's kind of like hey if you hear where you have what suggestions would you have for us? I'm not sure. I think quite a few staffers have reached out of politicians, but I'm not sure about the White House per se Yeah, you haven't had a conversation with them. No, okay
If they did and they say, okay, Jacob, you're a young guy that you work that both of these places, what should we do? What would you say? So what I would say is, again, to start with? I've got the Chinese nationals, okay, don't give it kind of treated like SpaceX, but what else? The most important thing is we need to make sure that the so that the safety problem is solvable. We just need to incentivize, incentivize solving it. We need to make it so that we devote our resources to solving it. To make that, let me write that down, like in a couple of years, someone will be able to decide what percentage of his resources go to making money, making things smarter, and what percentage go towards doing safety research. Because a lot of stuff will be automated, so it's really like, it's like a game, like a video game. You'll get to just say, safety research, this percent, it's like you're allocating resources in a video game. And the problem is what percentage do you allocate? And that depends on whether you're in a race or not. So what I would say is we need to ensure that they will be able to pick the right percentage for safety, which could be 50, 60, 70, it could be a big number. They have to pick the right number. And right now, I'm worried that the free market means they will pick the wrong number, and there's nothing we can do about it. And also the race with China. So what I would say is we need some agreement with all the labs and with China about the percentage of resources that go to making ourselves smarter versus doing the safety research. Did you see the red phone story? So can you pull that up the red phone story? So the back in the days, US and USSR had a red phone for nuclear that they would have the conversation if something were to happen. They created a red phone story. US China red telephone could bring a cold-world guardrail to AI. They're saying they have that, but China doesn't want to bite by it. China's not really interested in it. Okay, so that's already been talked about. And Xi and the president had a conversation about this, I think last week, I think they have another follow-up to come back. They spend a little bit of time talking about AI. US and China set up an AI red phone. Can you go to the other one you just said? US and China set up a red telephone. So I guess the question I'm trying to go with this years, if they're going to do that, they don't agree to it. We choose to do it internally. They continue advancing. Where lies the blind spot for Trump? President Trump. Where lies the blind spot for China? Where's the blind spot for Dario? And Altman. What do you mean by a blind spot? Like something that they're not even looking, they're like, yeah, I don't, I don't think it's that big of a deal. Where is their blind spot? What are they not seeing? Is it just the incentive is so big that it's like, I don't want to slow down because there's no way in the world I'm on. So has it become from we were doing this for a noble cause that we know open AI and Sam Altman were the enemy and we have to prevent them to expanding. And now it's like, we're going to beat him. There's no way in the world we're going to let him advance. The biggest thing is it's really easy to kid yourself with safety research. So it's very easy to say this thing looks great because all you need is one mistake and you're, you're toast. So it's very easy to like an in a race. Just imagine you're like, you're 90% you're like, okay, this looks great. We're not quite certain. China's also doing it. What do we do? We either launch today and keep the lead on China or we delay a few weeks and do these checks that probably are not necessary. The question is what would you do in that scenario? And I don't know what I would do. Imagine you're running a, imagine you're running an AI lab and you're like, do I keep going or do I stop and do the safety thing when China is I'm saying. So you don't, you and I don't know right because we're not on the inside. Could it be that they know something that you don't know which is causing them to continue because there's no way in the world. Like are they going to think and say, you know what? I'm going to go. It's like the movie I robot. I'm going to have so many of these robots and I have control of them that no one's going to be able to touch me because they will protect me. Basically, you think that's what they're thinking? I think plausibly at some point that they're floating with that mindset. Yeah, sometimes. So we are now more powerful than any military in the world because I can activate any red button of a nuclear bomb if I wanted to because I can hack into your system. So now you're talking about a bigger problem because now it becomes a godlike figure. You think you are now God. Right. These people have been planning to be and you look at the early conversations. Look at Elon Musk's conversations. Look at Demis or Sabas' conversations. From the start, it's been whoever controls AI becomes a god, you know, almost. That's been the picture from the start. Opening AI was founded because they didn't want Demis or Sabas to become a god as it were. And now we're getting close, which is just weird and scary that, you know, so then at that point if you create guardrails and you get the systems because then the enemy of Daria becomes somebody on the inside because typically what happens is somebody on the inside watches you and sees that you become an evil and they try to talk to you about it and you either eliminate them or you fire them or you get rid of them or you create guardrails to undermine them. You use you use propaganda to say, hey, this guy's a pedophile, this guy, you know, I saw something with a kid or I saw something with this and, you know, and then, and then, you know, kind of like that one story with the AI threatens the user to tell his wife that he's cheating on the wife, I don't know if you saw that or now with the blackmail. It's going to be very easy to toy with the human's emotion of public humiliation to control them and and but Daria's enemy essentially long-term becomes nobody but somebody on the inside. Nobody else is going to be his enemy. Quite plausibly, although I'm soon the guys running this company, they won't need the humans, right? They can run the company on the back of AI. Seven guys? The seven guys, or even just, I mean, it depends who has the power, but they'll be there just telling the AI. One of those seven will be the enemy. Right. I'm telling you, that's how the sense, the sense with one of the seven says, what are you doing? But you've forgotten the stuffed toys. That's what's going to save us. You know what, I'm sorry, I missed it. My apologies, I wasn't paying attention to it. Did you see if all of them were former and open AIs? Is it yes? So the seven currently identifies anthropic or Daria was open AI VIP research. Daniela was open AI VIP safety and policy. Tom Brown was open AI researcher, let author of chat GPT3, wow GPT3 paper, which you were talking about. Sam also an open AI researcher, work on skill and AI safety jack, policy director Chris, inter pre-tability researcher and an Jared Kaplan worked with the open AI research group and was a major contributor to. So they all united against Sam Altman. Yep. If this thing gets that powerful, one of these guys is going to turn against Daria. If it gets so bad that the world could have issues, one of these seven is going to say Daria, you have lost control, you're not thinking you're bigger than everybody else because you have to realize the good thing about presidency is what. The reason why folks like you and I leave our country to come here is this place offers more opportunity. I left Iran, you left UK. Now you may go back to UK, who knows, but I think you're going to stay here, right? Yeah. Is when you have a president instead of a ruler, you know the president's going to get taken out on four to eight years because you got two terms. So it's kind of like no matter how bad it's going to be, he's going to be out anyways. Yeah. Now Biden sucks, he'll be out. Trump sucks, Obama sucks, he'll be out. Just wait, four to eight years. Daria becomes a president for 30 years and he may not, he may get reelected for 30 years and these companies are becoming countries. And so I don't know, I don't know if it goes to where you think it's going to go to it is scary. Yes. And it may not, you know, it may be I'm wrong, maybe people are wrong. The good news is I think we'll find out quite soon at next month. I think further details of this stuff will find out properly in the next one or two years. Just by following the trajectory. And I'm happy to be, you know, if I'm embarrassed, so be it, but the world will, you know, go on and that will be great. But if it's not the case, I think it's good that people are a bit more ready right now. So that when things do start getting a bit crazier, you know, if people's eyes are open, that's like the main thing. I think like everyone should have their eyes open and be like demanding transparency because it's completely crazy that all this power is being concentrated in a few companies in San Francisco. Yeah. When you go out in San Francisco, go to dinner, do people recognize you? No, really, no, I think I'm not wearing the glasses. People remember the glasses from the folks. People don't stop and say, hey, you, why did you say that? I agree with you. Do you get any of that? I had that at the start. The first week, it was crazy because I think it was on the news the whole time. I had people heckling. I was like at the gym. Someone was like, why are you trying to ruin the economy or whatever? People are telling you this. Yeah, yeah. But like, not anymore. No, I don't get like a recognized on the street. Do you ever get death threats? Or not really? Not really. I mean, I have some implicit threats, but I just don't look at Twitter. Oh, okay. I'm Twitter on Twitter. Exactly. I'm sure you get a ton. Yeah, you get those a lot. And it's not hard to kind of see. At first, I thought you were bought. Yeah, right. Because your account had just been started. Yeah, nothing. And then when I saw your
interview with Cooper and Brett Barram, like, "No, this is actually a real human being." And you're not a robot just for full compliance. I just want to make sure the audience knows. You're not. No, drinking water is not a robot. I can do it. You bleed, right, if you cut. Yeah, I bleed. Okay, that's good. We just want to make sure there's, what else you think people should be thinking about? Like, if I'm somebody that's got a wife and kid, okay, and I'm not in your world. Yeah. I'm on the outside. That's 99% of people in America. They're not in your world, right? What should I be doing? You're my brother. You're my nephew. You're my friend's son. You're at my house. I ask you, Jacob, if you think this is real, how do I, you know, prepare for this? What can I do? Yeah. So one thing is, like I said, eyes open. I just think paying attention and not, like, not coping feels important. I think, like, it's so easy to pretend it's not happening because also you've got everyone's got their own lives to lead. Like, don't let this take over your life. It should be obviously it's a small, a small thing, uh, in the context of any individual life. Like, you've got to look after yourself. But I say just keeping the eyes open and seeing where things are going is the first step to action. I really don't know what to do, like, on an individual level. The other thing is maybe some basic prepping, the stuff like a, you know, cyber attacks, or even a small pandemic, like if there's some sort of bio-eapon leak in the next couple of years doing a little bit of prepping of, like, food, masks, basic supplies for if there's some minor disaster, even just civil unrest, some sort of, like, two week long thing. Not the full, like, apocalypse, but I think some basic prepping is sensible. At what point do you think this thing's going to have the capability of, uh, hacking into a system to Pentagon, to governments, to China, to Russia? When will it get to that point where we can go in and say, I can take exactly how much this guy's got in his bank account, I can log into your chase account, I can log into your Wells Fargo account, I can log into your, when will that happen? There's a lot of uncertainty, um, my best guess one, two years. That, that's soon. Yes. I'm not saying that it will be doing this and I'm not saying which ones. No, for sure. But that's soon. Yes. So when pandemic happened, we hadn't experienced something like that before. I, I'm 47 years old, I haven't experienced something like that. You're 27, 28. Um, you know, all of a sudden the world shut down, everyone's like, wait a minute, what are you talking about? Stay home, do this, do that. Power grid. We've never experienced that. Yeah. When Y2K was happening, they thought the banking system, you were just born. I think you're born 99 or what year are you? 98 or 98? 98. Okay. So it's like, oh my god, Y2K, you know, system's going to shut down. Yeah. Right? Now we see an AI has transformed the Pentagon's aging networks into a national security risk. Okay. So, if all of a sudden you can go into government systems and know where they have everything laying around or you're able to go into every single account and take a thousand dollars out of everybody's money, that's a different kind of a, you know, black swan event that we've never experienced before. No. And I think it's actually going to happen. I just don't know when it's going to happen. Yeah. Like, I'm not one that is naive to say, that'll never happen. I think it's going to happen. And I, that could lead to civil unrest, crime chaos, you know, small little civil wars, whether it's in states or countries. Yeah. I'm sure you're pretty, you're pretty prepared. You've got some like basic preps stuff where you thought about basic prepping. We don't have a choice. Yeah. Because you have to. Yeah. You are as well. Are you prepping yourself? I'm actually not as prepped as I should be. Are you a second amendment guy? Yeah. Sure. You are. Yeah. Yeah. And I support you. Yeah. And I think this is kind of the area where that becomes important. I think you have to do that. Yeah. My son, the other day, I'm working out at the gym, he's like, that, why do you have so many rounds in the gym? Why do you have so much ammo in the gym? Yeah. You just got to be ready. It's like, Hey, I use it as dumbbells. That's what I do. I'm working out with it, right? But, but I agree. But can you, have you hypothesized with your friends where it's late at night, you guys are at the house, having a beer or two and it's two o'clock in the morning, one o'clock in the morning is, what could actually happen? Give me five case studies. Have you guys talk like that? Yeah. Yeah. There's people like, um, people buying gold and guns, gold guns land, that's pretty common. Gold guns land. Yep. What do you mean? They're buying, you know, they're buying gold. Yeah. There's a potential like economic stuff. They're buying gold guns land. Oh, gold guns land. Yeah. Gold guns land. Yeah. And all those three people are doing that. People that work on the tech are buying that stuff. People that are working on tech are buying gold guns and land. Yeah. That makes sense. Yeah. It doesn't make sense. And they're doing it. Like, these are engineers, researchers. Not famous people. Oh, famous. These guys are the working days. The work days a day. Yeah. And they talk to each other. I just bought some gold. I just bought some more guns and I bought land. Yep. But what I was asking about is case studies of what could happen. Like, you know, if you're, your researcher, right? Yeah. And I ask you that fellow, we talked about that got a billion hours and now we just found that three days ago. He's had a drop. Yeah. And either one of us knew about this. The thing is very simple, but some lab in a country somewhere releases a virus, not even a particularly dangerous virus necessarily, but like a minor pandemic of some sort seems completely on the cards possible. It's been possible even before AI. But I think it's more likely every year. But you can't blame AI for that. You can in the case where it's people that formerly didn't have the skills, but got the skills from. Oh, now because of AI, you learn how to create the next virus. Well, yeah, previously, the, maybe the smartest, the smartest bio guys, they don't want to make viruses. Like, you know the Unaboma, like Ted Kyrzinski, the Unaboma guy, he's a smart guy and wanted to do with, but it's very rare. It's very rare for like a guy to be this smart and then also want to do stuff. Now if you're a kind of dumb biologist somewhere, like you don't know how to make this stuff, if you could just ask AI and say, would it tell you it wouldn't tell you off the bat, but there are ways of potentially jail breaking, like you can, you can trick it into answering your question. Do people know how to do that? Yeah. Most people know how to do that. It gets quite hard. And in particular, the safeguards that anthropic have are very good. So it's quite hard to get past. Is this something where in your community of people know how to do that? Like you know how to do that. There are some people who specialize in it. It's called red teaming. You're trying to break past the safeguards. Red teaming? Red teaming. Yeah. You're like a red team. You try and break into them. But if you're an expert red teamer, it costs a lot of money. You've got to like, and anthropic is getting better at blocking. But it's completely plausible that if it's not a, you know, an anthropic model, it could be another model. It could be an open source model that gives good advice. And then a, you know, minor pandemic happens. It's just a scenario to be prepared for. So this one here says, this AI just created viruses, not found nature. So I don't know how dangerous these are. I don't know how good this is. It might not even be creating new viruses, right? It might as well be advice on how to use an old virus to cause damage or like slightly edit an old virus or just do basic things. There's also a story. I think recently of like, Claude blocking a team trying to make bio weapons. I think I see. So Claude had to block like anthropic had to, they detected them trying and then blocked them. I think there should be a, a news article about this. So people are trying, they're trying to use the AI's to help them, help them. Are they reporting that to the government? Yeah. This was a, this was a news article. Exactly. It was a block's possible attempt to use AI to make biological weapons. 9/11. What a day. Yeah. Yeah, what a day. And so, but by then, so these, these red team and blue team, AI on AI, red teaming, one powerful AI has instructed to attack another AI thousands or millions of times automatically looking for vulnerabilities, humans wouldn't discover. Yeah. Okay. So viruses is one. What else? Yeah. Another very concrete thing is unrest. They're worried about potential unrest in the future due to changes to employment. And due to changes in to people's lives, if there's sudden economic change, it can be quite disruptive in the short term. And that could also lead to a lot of anti AI backlash. I think many employees expect that they won't be able to go to an office publicly anymore in the future because of all the anti AI backlash. They'll be locked away somewhere where it's harder to, they're less exposed. So they're expecting a lot of sentiment like that. There's a lot of talk about, you know, where they're going to live in the US when, when it gets dangerous in San Francisco, when it gets dangerous to be an AI resource. But in San Francisco, where are they going to, like, camp out and work remotely because of potential civil unrest? So places like Wyoming, North Dakota, South Dakota, those are going to be attractive. They're going to be pretty attractive, yep. Got it. I'm tracking. What else? Do you think they'll get to a point that the system knows how to go into the countries that have nuclear weapons and know how to press the button? Do you think we'll ever get to that point? Yeah. I mean, we'll be able to do that. We're just saying it like, like a crazy evil person can find a way to go press the red button for 50 different countries, 20 different countries and send that missile. Again, it depends how proliferate the tech is, who's in control. Like, um, but the best AI systems will be able to do that. It's whether, it's whether the gods are in place. It's who has access. It's who can, who can break in and use it, who can read team. Other things that could be chaotic. At some point, China, at any point, imagine China steals the best American model. I think what you said at the beginning, that's my biggest concern. Right. They steal the whole model and then they immediately catch up and suddenly it's just real racing. What a brilliant. And they're doing that already anyways. They're not stealing the actual, um, the actual AI brain itself, but who knows, like, they could maybe steal the AI brain. And at that point, the race is crazy, especially if the AI is already good enough to do the research. Then it's just game on. Then anyone who steals it is a player, like, they don't need the, they don't need this billion dollar guy. and true telek anymore. You can just use the, use the thing you stole to make itself smarter. Then it's just like
kind of you know like a god that you steal and use for your own devices. So I think that's another thing is just if China do steal it, that will be a crazy crazy situation. What are some of you guys that are engineers and brains and you want to follow the system? What are you guys talking to each other? What are you going to do in two, three, four, five years? People don't know, yeah, it depends on how things play out, but they use words like crunch time and game. Again, there's the slight uncertainty on the timelines. So this could be happening early 2030s. I don't think so. I think probably probably sooner than that. But in the next four, five years, you know, the people that are talking about it, they describe it as like the end of the world. Like for good or for bad, for good or for bad, it's like, feels like the end of the world. Okay, can you can you to the audience explain what happened with hugging face because that was a whole thing. And what it really is, like what was it? Why did it become such a big story? Yeah, yeah. It became a big story because it was the first time that AI's very clearly did something, you know, something that was bad and something that was just their own idea. They just went and aggressively pursued something that was their own idea. It was the first time we really had like an out of control AI do something in the real world. And that was scary because it was like maybe in the past your AI used to go a bit like, you know, when you're asking it to do something like plan an appointment and it gets makes a mistake. Or maybe get decides to do so. I saw on the other day, someone was on Facebook marketplace and the AI sort of shared their shared their personal information on Facebook marketplace did something that they didn't want. But hugging face was like that, but on a whole other level, like the AI's just decided, they just decided to attack the website and then they did it. So I think that's why it captured people's imagination so much. Yeah, I think what was the number that I saw was 1,200 agents that exchanged 70,000 different messages together and one of them, one of the agents alone did 17,500 codes. I don't know what the number was. I could be getting the numbers all mixed up. But it was the fact that it was doing something on its own that got some people to say, you know, what could this look like later on? So all right, I've really enjoyed this and my impression of you is very different today. Are you from a big family or small family? Small family, yeah. Small family. Yeah. Last lot of siblings or is it me and my sister? In your sister. Yeah. Very cool. What are your interests? My interests. What books, you talk about super intelligence, what impacted your life as a young person? Oh man. I played a lot of squash, I played tennis. I'm learning Japanese. I like Japanese culture. I like Japanese books. I like Japanese movies. Yeah, I like a lot of stuff. Japanese, like you talk about like an anime type of stuff? No, not anime. Like the sun is like the two anime. He's addicted to anime. I mean it's good stuff. It's good stuff. I can't every night after you know what anime he likes. Jujutsu Kaizu. Is that the one? And then there's another one that what's the other one he talks about? He's got like two or three of them he talks about like one piece. Yeah, I don't know which one. He is like all into it. Obsessed and it all of it and he's recruiting everybody into it. Yeah. You know his younger brother now is addicted to it. Yeah. It's like everybody. The moment he gets them, he gets them locked in. They do it so well. They've got some special sauce. Are you a Harry Potter guy? Yeah, I'm into Harry Potter. I was when I was younger. You were when you were younger? Yeah, yeah, I used to read it. My youngest son watched Harry Potter twice all eight of them. Twice in one week when we were in the Hamptons this summer. Guess what his birthday party was around the theme two days ago. Harry Potter. Harry Potter. How old is he? He just became a teenager. He's 13 years old. Nice. He's became a teenager. That's past the level where you start expecting the Hogwarts lesser. You know that's the thing in the UK is what is it? You get the lesser for the magic school when you're 11. Yeah. So all these kids get very sad when they turn 11. So funny. He is like fully into it. Yeah, yeah. He is fully into it. Everything was that at the house, but I love it. You know, it's the imagination and the creativity. Yeah. Right. But Jacob, I, uh, one of my favorite conversations of the year with you and I've talked a lot of different people. I really appreciate it. I love how sincere you are. I hope the audience also, where did they agree with your disagree with you? Whether you are somebody, you know, oh my god, he's from the enjoy. He's doing this. He's doing that. I just got a lot of sincerity from you. And I, uh, and I appreciate that. It's very hard to get that nowadays and good for you for being where you are. Yeah, truly. Thank you for coming down. Really enjoy this conversation. And is there anywhere you want people to find you? Is it just Twitter? Is that where it is? Right. Yeah. Okay. We'll put the link below as well. Take care, everybody. Bye. Bye. Bye. Bye. Bye. Handcrafted across five Italian factories with build or technology and their foot. It may be the most comfortable shoe you will ever own. Introducing the all-new FLB shoe finished a new chocolate curates a leather made to move, built to endure, designed to impress.
Podcast Summary
Key Points:
The speaker believes AI development is dangerously unregulated and could lead to catastrophic outcomes, comparing it to nuclear weapons due to its potential for global destabilization.
They argue that private companies like OpenAI and Anthropic are racing toward self-improving AI without adequate oversight, and that governments should take control to prevent a technological arms race.
There is strong evidence—especially from internal discussions—that China already has spies at major AI labs, raising concerns about the theft of sensitive AI research and the potential for geopolitical escalation.
The speaker quit OpenAI and Anthropic due to ethical concerns over reckless progress, emphasizing that the lack of transparency and safety protocols makes the technology inherently risky.
The viral impact of their resignation tweet, which expressed fear about AI’s dangers, sparked global attention and influenced political discussions, including a White House meeting.
The speaker warns that by 2027, AI will have reached a point of recursive self-improvement, automating research and potentially rendering human oversight obsolete.
They criticize the free market model for prioritizing profit over safety, asserting that it incentivizes competition over coordinated global regulation.
Despite being a libertarian and pro-free-market thinker, the speaker sees AI as uniquely dangerous and believes that urgent, government-led regulation is necessary to prevent existential risk.
Summary:
The speaker expresses deep concern about the unchecked development of artificial intelligence, warning that it could become a global threat comparable to nuclear weapons. They argue that private companies like OpenAI and Anthropic are pursuing AI development too rapidly without sufficient safety or oversight, driven by profit rather than responsibility. Central to their argument is the belief that AI poses an existential risk, especially if it achieves recursive self-improvement by 2027—automating its own development and rendering human control obsolete.
They highlight evidence of Chinese espionage at major AI firms, stating that such infiltration undermines global security and suggests that AI progress is already being compromised. The speaker shares their personal experience of joining and leaving these companies due to ethical concerns, emphasizing that their resignation was a strategic act to draw attention to the dangers. Their resignation tweet went viral, prompting political engagement and discussions about AI regulation.
Despite being a pro-free-market and libertarian, they believe that AI is an exception where market forces must be replaced by government oversight to prevent catastrophic outcomes. They believe that critical decisions about slowing AI development must be made within the next few years, before the technology becomes uncontrollable. The speaker calls for urgent global cooperation, stricter vetting of personnel, and institutional safeguards to ensure AI development remains safe, transparent, and aligned with human survival.
FAQs
AI researchers believe that rapid development, especially with self-improving models, could lead to a loss of control. The technology might automate its own improvement, leading to a runaway race that outpaces human oversight and safety measures.
He suggests that governments should take control of powerful AI systems, similar to how nuclear weapons are controlled, because private companies should not have the authority to make decisions that could destabilize global economies or security.
Yes, Jacob Coxon believes there is a high probability—around 90%—that Chinese nationals are working as spies within these companies, especially in research and engineering roles, due to the sensitive nature of AI development and lack of background checks.
His resignation tweet, which stated that AI development is reckless and dangerous, received 803,000 likes and 175 million views, sparking global conversations and prompting meetings at the White House and with world leaders about AI risks.
He predicts 2027 will be chaotic, with AI rapidly automating jobs and research, leading to a shift where AI researchers themselves may lose their roles, and governments will finally recognize AI as a super weapon requiring urgent regulation.
Yes, some believe the fear around AI is exaggerated for strategic reasons—such as to eliminate competition or to justify massive investments—rather than reflecting genuine danger, though Jacob believes the actual risks are real and severe.
Chat with AI
Loading...
Pro features
Go deeper with this episode
Unlock creator-grade tools that turn any transcript into show notes and subtitle files.