The Bitter Rivalries Behind the Race to Build Superintelligence

(knocking on door) Today, understanding the people who are building super intelligence. I want you to think back to 2017. In some ways, it probably just feels like yesterday. Nine years ago, your neighborhood probably looked exactly the same. There were cars, coffee shops, phone in your pocket, but in the history of artificial intelligence, 2017 is the distant past. ChatGPT doesn't exist andthropic doesn't exist. The mega project of AI construction that's currently driving American economic growth, it was on nobody's horizon. But the following things all happened in 2017. Number one, open AI, the nonprofit co-founded by Sam Altman and Elon Musk started work on GPT-1. Five years later, this technology would serve as the guts of ChatGPT. Also in 2017, a physicist at OpenAI, named Dario Amade, wrote a secret 26 page internal memo, entitled Big Blob of Compute, in which he made the argument that AI research should resemble a massive infrastructure project, where bigger models with more data using more chips would be the path to super intelligence. Nine years later, this thesis has proved eerily accurate as the entire AI buildout has been motivated by the idea that AI's capabilities scale with compute spending. And number three, also in 2017, AlphaGo and AI model went undefeated against the world's top ranked Go player. The model came from the company Google DeepMind, run by Demis Hassabis. Three years later, that company would use similar technology to solve the famous protein folding problem, achievement that many scientists considered impossible. For this, Hassabis would win the Nobel Prize. Nine years later, it is not a stretch to argue that the entire frontier of AI has been shaped by these three men. Sam Altman, the CEO of OpenAI, Dario Amade, now the CEO of Anthropic, and Demis Hassabis, now Chief Scientist at Alphabet. But who are these men? Who are Sam, Dario, and Demis? I'll sometimes read or hear coverage of this space that makes it sound like the men building advanced AI are all the same. They all want the same things. They are powered by the same greedy motivation. But these men are not the same. And today's episode is about the hidden motivations and rivalries among these three men that have arguably brought us to the precipice of superintelligence. What books and ideas inspired them when they were young? What do they fear? What do they really want? My guest is Kevin Rus, the author of a new book, the AGI Chronicles, the inside story of the race to create an artificial superintelligence. This is a story of bitter rivalries and secret documents. And as technology moving faster than the people building it can understand, let alone control. I'm Derek Thompson. This is Planning. (upbeat music) (upbeat music) This episode of Plain English is presented by Comcast Business. Did you know Comcast Business detected 79.3 billion cyber threats last year? That's why they're introducing security edge preferred. It's big business cyber security designed for small business. Get started with Comcast Business Internet for $50 a month for one year on a qualifying internet package and find out how to get one month of security edge preferred on them. Plus, ask about a $500 prepaid card. Limited time offer, restriction supply, requires 300 megabits per second internet, security edge, security edge preferred and wireless connect. Call today. Kevin Rus, welcome back to the show. Thank you for having me back. Always a pleasure. Kevin, what is super intelligence and why do you think we're on the precipice of building it? Super intelligence, you could think of as an AI system that is smarter than everyone at everything. There are sort of these gradations of AI capabilities. People talk about AGI, artificial general intelligence, as a computer program that is human level or better at most tasks or most intellectual tasks. Super intelligence would be the level beyond that where we have something that is vastly smarter than us at everything that we could conceivably do. Your new book is called The AGI Chronicles, the Inside Story of the Race to Create an Artificial Super Intelligence. It seems from your reporting that you think this is a race that is on the precipice of achieving something quite dramatic in the near future. What have you seen? Let's just say in the last 12 months that have made you think we were really making progress toward this goal as you defined it. Well, let's back up more than 12 months to 10 years ago when the book starts, which is we had these AI systems. They could do things like identifying objects in images or translating between languages or helping you with auto correct on your email. But those were narrow systems. They could only do one thing. They couldn't be repurposed to do 20 different tasks. Now we have these general models, cloud, chat, GPT, Gemini that can do a wide range of things near or above human level. We have systems that are capable of solving millennium prize problems in math, of discovering new protein structures, of helping people, you know, billions of people around the world every day in various ways. So I think that's what I've seen is just the continued march of this technology. And then the sort of, the non-public thing that I've seen is that the people at the companies who are building this stuff, I guess this is not a secret anymore, are terrified of it. They have been for years. And they're not getting any calmer as we approach what they think could be some kind of intelligence explosion, which we could talk about. That's sort of another concept that they think is not far off, and I'm inclined to believe them. One thing I think is so strange about this entire project is how small this world is. When we talk about the frontier labs, we were talking about three companies leading the race to build super intelligence. They are open AI and Thropic and Google DeepMind. Those companies are run by or have until very recently been run by three particular and peculiar dudes, respectively, Sam Altman, Dario Amade, and Demis Hisabis. I want to spend the bulk of our time trying to understand these men, who they are, what motivates them, their strengths, their weaknesses, and let's just jump right into it. Sam Altman, Paul Graham, the former head of Y Combinator, once famously wrote this about Sam Altman, quote, "You could parachute him into an island full of cannibals "and come back in five years and he'd be the king," end quote. What makes Sam Altman so good at becoming powerful? - He has this superpower, which everyone he's ever worked with, all of his mentors, they have recognized this in him. And he had it from quite a young age, of figuring out exactly what someone wants and giving it to them, or at least appearing to give it to them. So back in his days, running Y Combinator, people would sort of talk about this power of his and they didn't know where it came from, but people like Paul Graham recognized it very early. He was very good at reading the room, at figuring out, this person expects me to do this or wants me to do this, so I will do that. And it led to him becoming one of the most successful young entrepreneurs and investors in the valley. The flip side of it is that he would often get into trouble because he would tell one person one thing that they wanted to hear and then tell a different person something completely different that they wanted to hear, and he sort of got this reputation as a sort of slippery people pleaser, which is, I think, a criticism that many people who have worked with him over the years have identified and have made in pretty consistent ways. He just seems to be in the business of both being quite ambitious, but also of being quite eager to please. - One way to say this is that he is a brilliant deal maker in the mold of a JP Morgan. Another way to say it is that he's a bullshitter, and I wonder, after all of your reporting, whether you feel like underneath all of that deal-making, there is a core philosophy. Like, just to put my thumb on one distinction that might help you answer that question, there's a 2015 blog post that Sam Altman wrote, where he called machine intelligence, quote, "Something we should be afraid of." And when he found open AI with Elon Musk, he founds it as a nonprofit, because he thinks that this is a project that should not be a typical commercial business. But now he's running a business that is by some accounts valued at $800,900 billion, a trillion dollars, and later colleagues who work with him are very skeptical of the degree to which he's serious about AI safety at all. So one way to answer this question is to say, does he believe in AI safety? But what I'm really asking here Does he believe in? anything outside of just making the deal that increases his power? I believe he has a core. I don't know what it is. I'm not in his inner circle of people that he confides. And I've spent time with him, I've talked to him, I've had encounters with him as a journalist. But I've always sort of understood that he was playing the part in those interactions and wasn't necessarily showing me his true self and for what it's worth. I think most executives do this, they have kind of their persona that they put on. I think the thing that kept coming up in reporting on Sam and the people around him is that he is a person who believes first and foremost in agency. He has this famous phrase, you can just do things. And this is something that he believed at Y Combinator. He would tell these young startup founders that he was investing in, like you don't have to wait to change the world, you don't have to like, nobody has to give you permission. You can just go do things. And that is a core part of his belief about himself and about those around him. You get another way of saying it is like there's no grown-ups, like you are the grown-ups. And you can just do things that change the world without permission. He's also a fervent believer in scale, the power of scale. People would tell me these stories about he would walk into a meeting and he would just tell the people in the meeting to add a zero or two zeroes to whatever budget they had for their project going forward or increase the scale of a training run just because he thought that Bigger was better. So I think that love of scale is probably part of his experience running Y Combinator and being an investor before OpenAI. But certainly OpenAI's success has confirmed that just having this kind of delusional self-belief and belief in the power of scaling is actually what you need to win an AI because this is something that didn't really occur to me until midway through the book. Belief is an active ingredient in AI success. You have to believe that the scaling laws will continue, that putting billions or trillions of dollars into building data centers and acquiring compute will pay off. You have to, like the belief is the enabling force that allows you to win and if you don't have it, no amount of technical expertise can save you. Your book has several examples of this delusional self-belief that Sam Altman demonstrates and one example is as OpenAI is scaling their GPT technology and they've just come out with GPT 2. But they realize that they're going to lose the money spigot from Elon Musk and so they need to get a billion dollars from someone. And a billion dollars at that point in the AI super cycle is a lot of money. And so they decide to put on a little demo for Microsoft CEO, Satya Nidela with GPT 2 to rest a billion dollars from Microsoft. Can you tell this story and what this story represents about Sam and the company? Yeah, it's an incredible story and it hadn't been reported before and I was sort of delighted to find it out because it does really illustrate who these people are. So they go up to Redmond for the day from San Francisco to Sam and a few other OpenAI executives. They've got this very high pressure meeting with Satya Nidela and his leadership team at Microsoft and they've got this new model, GPT 2, that is still, it's not good by today's standards. You would never use it for anything that you needed done. But it can sort of complete a sentence, it can sort of generate coherent texts and it can do all of this more flexibly than any of the chat bots that had come before it. So they go up, they run this demo, they bring a Microsoft laptop borrowed from one of the OpenAI executives girlfriends because they're like Microsoft will think more favorably of us if we're using Windows. So they borrow Windows laptop, they go up, they run the demo and they're showing some sort of canned prompts that they've prepared that they kind of know GPT 2 will be able to do. And then Satya Nidela asks, can we see this thing? And another Microsoft executive sort of wants to try their own prompt that wasn't part of the canned demos. And so they're like, they're sort of looking around each other, they're like, can we do this? Can we do this? Yeah, I guess we have to do this. So they put in a prompt that's like, Satya Nidela was hired as a CEO of Microsoft and the first thing he did was blank. And then they hit enter and they're sort of like sweating as they wait for this thing to come back. And then it spits out the completion, which is that on his first day, Satya Nidela fired all the employees and pivoted the company to Azure, their cloud platform. And this draws a big laugh and it's sort of an amazing moment in time. Like an AI model has never been able to do this before. And it was very high stakes. Like had this sentence completion not worked, they might not have gotten the billion dollars from Microsoft, but as it turned out, that was, you know, exactly what they wanted to see and they invested a billion dollars in the rest of history. It's an amazing story. And one other piece that I think is really important is that Bill Gates at the time was quite biased against AI chatbots because Microsoft had a terrible experience with a chatbot named Tay or something. Do you remember Tay? It was like a glory. I just said no. You're the chatbot nerd here. You've had all of the deep intimate experiences with chatbots. I just, I have a thing for the range chatbots, and Tay was one of the more deranged ones we've had. It was this like experimental chatbot that Microsoft built pre LLM and they put it on the on like Twitter and people just started like messing with it and like turned it into like essentially a white supremacist within like 12 hours. It was like spewing like Nazi propaganda. And so the lesson that Bill Gates and other Microsoft executives learned from that is like don't make chatbots. That's like, that's a bad idea. Yeah. I think that's a really important ingredient in what makes a story so dramatic because Microsoft is not beginning from like a neutral place of maybe chatbots are good, maybe chatbots are bad. We have no experience. Like they're like, we've done chatbots. They're terrible. Yes. Who knows? Maybe if we, you know, invest $1 billion into this company, our chatbot might, you know, engage with the New York Times reporter and, you know, proposition him for marriage. You know, thank God something. That would have happened there. Right. That would have never happened. Right. Like, understanding the internal experience of like hinge points in business history and whether you're a listener who absolutely despises AI or loves AI, like the release of chatbots absolutely changed business technological and economic history. It was so interesting to read your reporting on just what an effing nothing burger the folks at open AI thought chat should be T would be at one point they're taking bets on how many people are going to sign up for this chatbot. Most people think 10,000, the highest bet at the company is Sam Altman saying no more than one million. I mean, we're talking about a technology now that has a billion, a billion users around the world or more. Can you flesh out the details of why open AI launched chatb tea? Why ironically it started with a rumor about a rival company launching their own chatbot and just what happened behind the scenes that gave birth to chat to BT itself in November of 2022? Yeah. It's an incredible story because it truly was an accidental hit. The people at open AI, as you said, many of them didn't think this thing would be very popular. And I think to explain why it's like you like we had to sort of step back into time machine to like the sort of summer and fall of 2022. At the time, GPT-4 was just sort of finishing being trained. This was the most capable model. It would not be released until the spring of 2023, but open AI employees already had seen versions of it and were playing around with it. And they were all racing to get this thing ready for a launch in early 2023. And then in November 2022, so a couple of months before the planned launch of GPT-4, this open AI co-founder John Schulman hears this rumor at a party where he hears that Anthropic, which is a splinter company that's sort of formed out of a group of dissatisfied open AI employees, is going to release cloud their chatbot. Now the rumor is not true, like they were, they had no plans to release cloud as a commercial chatbot. They were sort of testing it in a private Slack channel with like friends and family. But this gets sort of passed back to Sam Altman and the folks at open AI. And they're like, shit, like we have, we have this plan to release GPT-4 in a few months, but we think that this chatbot cloud might be really good and might kind of upstage us. And we don't want that to happen. So John Schulman proposes this sort of rushed launch of a thing that was originally called chat with GPT-3.5, which was basically to take their old model that had already been out for like six months for developers and put kind of a free, easy to use chatbot interface on it and just throw up in the doors as what they called a low-key research preview. Basically, it was not their best model internally. And I think that's why a lot of people at open AI had low expectations because they're like, this thing sucks. It's like six months old. We already have this better model that has finished training that we've been using ourselves. That's a lot better. Why would we like pump the brakes, throw a new skin on this old model? like put that out as if it's supposed to be impressive, but they did it anyway. They decided to make it this low-key research preview. They had low expectations and then it just caught fire. They had badly misjudged how impressive the things that they were using internally would look to the outside world. Really, really quick question. Do you think open AI would have inevitably released a chatbot with GBT4.0, GBT5.0 eventually, or was there something about that dynamic of believing the claw would become public that forced them to take their GBT technology and turn it into a chatbot when they had been thinking about an entirely new or different product or business model? No, they were fully planning to release GBT4 as a chatbot, and actually, they were planning with Microsoft, which had this exclusive partnership with open AI, had plans to build it into their products and make chatbots as well. It was going to happen in the spring of 2023, but the launch of chat GBT brought that up in the timeline to the fall of 2022. Good. It's good to know that. Okay. The last question about Sam Altman, tell us why he was fired and how his return of the company changed the culture of the company, and because we don't have time for you to explain all of the reasons why he was fired, why don't you just shade your answer toward the most salacious of those reasons? You want the dirt? Well, he was fired because the nonprofit board that governed open AI had been configured in a way that gave the more sort of safety conscious members of the board a temporary majority. And these four members of the board were also getting worried about how honest Sam was. This is, again, a pattern that has been observed by people close to Sam for many years. In some ways, it was exactly the reason that Daniel Amade and the anthropic co-founders had quit in late 2020. They just sort of saw Sam lying and observed that he would say things to different people that were contradictory. And so Ilya Satskiver, who was an open AI co-founder, chief former chief scientist also on the board, started sort of assembling this rap sheet on Sam Altman and Greg Brockman, the president of open AI, who was also on the board at the time. And he and Miramarati, who was then open AI's chief technology officer, start kind of secretly compiling this dossier about all of the times that Sam and Greg would say something to one person and a different thing to another person, or would undermine executives or pit them against each other. Not a bad management practices, but displaying that this was a consistent pattern. They send it to a member of the nonprofit board and they basically plot this coup that they're expecting will be greeted with a hero's welcome. They're like, oh, if we're seeing this stuff about Sam, then everyone else must be seeing it too. And we will be greeted as liberators, essentially. And they carry out this thing on a random Friday in November, and they just are sort of amazed and pulled over by how much that does not come to pass, right? The employees rally behind Sam and Greg, they demand their reinstatement, they threaten to quit and go to Microsoft, and there's a lot of like good gory detail in the book about how that all went down. But essentially, Sam engineers his own comeback with the help of open AI's employee base. And five days after he's fired, he's reinstated. People have heard this story a zillion times. But what's new and what I was interested in are the dynamics of this employee led resistance. And how and why they decided to band together to restore the position of this guy that I think a lot of them had misgivings about, even at the rank and file level. So it really, one of the most fascinating five day periods in business history, I think. And so I devoted two chapters five days because I was just like, it's not going to get any juicier than this. Yeah, a lot of chicken nuggets ordered, a lot of boba teas consumed, and you even have stuff about how like the image of the chicken nugget and boba tea became like the hammer and sickle emblem of like open AI's rather like the war medals of people who survived this period. It's really, really colorful and strange story. That closes this sort of mini chapter on on Sam Altman, who I think it's important to say emerges from this deal making tradition at Y Combinator is a deal maker and that is both his strength and his weakness. It's his strength in terms of being able to secure money, secure compute, which they are lapping the field in. That is the amount of chips that they have to train and run inference on their AI. But that's also his weakness in a way. It was almost his downfall because he's so much of a deal maker without a clear moral or philosophical center that people feel sometimes that it doesn't exist in their form is trusting for it. And in a way that's very, very different from our second character, Dario Amade. Dario now the CEO of Anthropic, but up until, I mean, in the first few years of open AI, he was there, their head of research. Before we get into his own personal history, I want to talk about his motivations. You report that one of Dario Amade's favorite books is Richard Rhodes, pulled surprise winning history, the making of the atomic bomb. And if you've read that book and I highly encourage people to read that book or really just read the prologue. The hero of both the prologue and the entire book is Leo Seilard. Leo Seilard. I don't know how to exactly pronounce the Hungarian name. >> It could be Salard, but it's fine. >> Salard. Okay. It's, it's S Z I L A R D. Let's call him Leo. Who was Leo and what does it tell us about Amade? And he sees himself in Seilard's mode. >> So Leo Seilard was a Hungarian physicist and he has this discovery, this breakthrough. He's the person who came up with the nuclear chain reaction, the thing that paved the way for the splitting of the atom and eventually to the atomic bomb. And he has this realization. There's this very dramatic scene where he sort of has this realization when he's waiting to cross the street in Richard Rhodes book. And then once he has this realization and proves it out, he tries to keep it a secret. He goes around to all of, he sends telegrams to all the scientific labs, all the nuclear, you know, the sort of ish labs in Europe, begging them not to look into this or not to publish their findings because he is terrified that the Nazis will discover this secret and use it to build bombs and use it to win the war. And he tries to classify it as a secret with the British Admiralty. He tries to, you know, patent it to sort of keep it out of the public view and he fails at that. The secret gets out. And after the secret gets out, he has this moment where he realizes that the only moral thing to do now that the secret is out is to build it before the Nazis do. So he, there's this sort of amazing sequence in Richard Rhodes book where he basically petitions the president with the help of his friend Albert Einstein to start the American nuclear effort that becomes the Manhattan Project, he then signs up for the Manhattan Project. And basically his theory is if we don't build it before the Nazis do, the world is going to end. Or we're all going to be living under Nazi rule. And Dariel Amade, not only read this book, it's his favorite book, he bought copies for early anthropic employees like on your first day at Anthropic for years, you would get handed your onboarding packet and a copy of the making of the atomic bomb, which is like 1,300 pages by the way. This is not like reading, not like reading. And he would, he gave out a Cillard Award at Open AI to like people who would make sacrifices on behalf of safety. It really idolizes this guy and has patterned his career after Cillards. And you know, the obvious parallel here is that Dariel Amade discovered a secret. He was the essentially discoverer of what became known as the scaling laws, this method of improving AI performance by just throwing more compute and data and parameters at a model. And he tries to keep it a secret. He like, you know, really, really tries hard not to let anyone else know about this discovery. And then once it gets out, he decides that he has to build a GI or super intelligence first before, you know, not the Nazis, but, you know, China or Sam Altman or someone else builds it before him. So this is the sort of the meta narrative of his career is that he sees himself as essentially a modern day, Leo Cillard, working to build a powerful technology in a safe way so that others don't build it in an unsafe way first. Yeah. I don't think you have to necessarily accept the entire argument for anthropics existence to at least appreciate the sort of internal logic of the sentiment. I'm building something that could be the most dangerous thing in the world, but I have to build it as fast as possible because it's important for us to have the thing first. And like, that internal logic does not make sense for most products, but it made sense for Einstein and Cillard with the atomic bomb. It made sense to Oppenheimer to a certain extent with the atomic bomb and everyone at Los Alamos. And it makes sense to people working at anthropic into a certain extent also open AI. That's just a part of the weird, um. Self-justification that motivates a lot of activity in the Bay Area that I think in many ways seems completely alien and sometimes downright immoral to people outside the Bay Area. But I think that Salard is an important skeleton key in terms of understanding at least how these people think of their own project. You can think they're wrong, but it's still an interesting skeleton key to understand their motivations. Totally. Amade love to write essays. He's written a bunch of essays andthropic circulates all the essays for all the journalists. And I say that I had not read in a while, is when you write about in your book his 2017 26-page internal document entitled "Big Blob of Compute." You were referencing this in your previous answer. And he argues, essentially, that AI researchers overthink things. They try to sort of finely tune all their models, but at the end of the day progress comes from this simple recipe of make the models bigger, get more data, get more compute, and scale, scale, scale. Can you explain just how important this paper has been in terms of pointing to the future of, and almost incepting, the future of the AI super cycle? Because reminding myself of the paper in your book, I was like, it's a little bit Babe Ruth pointing to the left field stands and then hitting the whole run. And he kind of just like says, we're going to spend trillions of dollars on this thing. And then that's basically exactly what's happened. So tell me, "Big blob of compute." What was Amade's thesis in 2017, nine years ago, and how has it come to pass? Well, I hope you have not read "Big blob of compute" yet, because I think this is the first time it's been published anywhere. This is not a document that Amade has published or even really talked much about publicly. So this was something, an internal research memo that I obtained that I'm going to publish the whole thing on my substack this week, I think, because I just think it's an important historical document. For exactly the reason that you just said was like, he points to left field. He says, this is how it's going to go. And then it essentially goes exactly like he predicted. So this was a paper that predates the LLM, GPT-1 had not been trained yet. But it is essentially the scaling hypothesis in miniature. He says, essentially, you're all doing this wrong, by you all means like other AI researchers. He says, you're building these small, narrow, single-purpose model to do discrete tasks. You're building language classifiers. You're building image recognition systems. You're building speech recognition systems. And you are training them at a much smaller scale than would be optimal. You're not putting enough compute or enough data into them. And you are engineering these kind of narrow solutions to your problems. When the real way to create better AI models is to just make them bigger. Like this is a principle from nature. Animals with bigger brains, all things else being equal are smarter than animals with small brains. So if you want to improve the generality and the performance of your systems, just create this mass of unstructured computational capacity, this big blob of compute. And he talks about these seven sort of variables, these knobs that you can turn, like how much data you put in and how much, you know, how much, you know, what goal you give the system. And then you essentially let the intelligence fall out of the model rather than trying to like code it in by hand. This was a, I think this is the most important document, maybe ever, in AI research, because it came at a time where open AI was really struggling. They were trying to do these sort of narrow things. They had a robotics team that was trying to build a robot hand that could solve a Rubik's cube. They were trying to teach AI's to play video games. But none of it was generalizing very well. Like they would have one success in one little area. And then they would like, it would just wouldn't matter. And big blob of compute really steered them toward language models. Because language models were a thing that you could put a big blob of compute behind that would get better, the bigger you made the blob. And that was the key insight that led them to sort of bet to go all in on scaling up language models. And it's why we have the trillion dollar, you know, build out that we have today because they realized and they proved that the bigger your blob of compute was, the better your model was. And there's another paper published in the exact same year that it essentially hooks up with, right? Attention is all you need because you get both big blob of compute saying we need to secure as much possible computing powers possible. Combined with attention is all you need a paper that comes out of Google that essentially says the transformer could be an incredibly useful, generalizable model to do this thing that all these smaller models can't do. And essentially, when those streams collide, you get this explosion of this modern super spectacle of AI building. Is that fair to say that these two 2017 papers sort of collided in hyperspace and created like the modern AI moment? Absolutely. I mean, it wasn't enough to just have the big blob of compute. You also needed an architecture that could accept a big blob of compute. You know, there were all kinds of language models before 2017, but they didn't work that well. In part because they just like you couldn't scale them in the same way that you could scale this new kind of language model that Google had developed, the transformer, which was basically a way to radically increase the amount of compute and training that you could do on a large set of text. So you needed both. You needed the piece from open AI was the belief in scaling and the big blob of compute and the piece from Google, which they didn't even recognize at the time because they didn't have the belief in scaling. They just thought, Oh, this is a cool thing to a cool way to make our language models better at like translating and writing emails. It's not going to like, they didn't think this would be a foundational technology, which is why they released this paper publicly. So they sort of dropped this thing in open AI's lap, the transformer, Dario sort of sees this and runs with it along with some other researchers at open AI at the time and outcomes of the AI revolution. This episode is brought to you by Lincoln. What if your vehicle could help give you a reset? Lincoln Rejuvenate is designed to do just that. This multi sensory feature combines expansive visuals, digital sense and a seat massage for the driver and is available in the 2027 Navigator SUV. Visit Lincoln dot com to learn more. Lincoln Rejuvenate does not operate unless the vehicle is in park when the vehicle is off or is any door is open. Do not use the feature in a closed garage or in other enclosed areas. See owner's manual for important operating instructions. In 2019, open AI releases GPT2 and at this point, Dario Amadea is still the VP of research at open AI. The release of GPT2 does a few things. One, it proves Amadea right. They use something like a thousand times more compute than with GPT1 and they can see that the model is generalizing. It's getting smarter across a bunch of different domains, including those it's not specifically trained on. I guess that is generalizability. But the release of the model also really scares Dario Amadea. Why? He is worried that a text completion engine that is good at spitting out plausible generated text could be used for bad ends. It could be seized by propagandists or foreign disinformation campaigns. It could be used to pump out misinformation and fake news on an industrial scale. He knows that GPT2 is not going to take over the world. It's just a text completion engine. He is worried about how bad actors might use it to do unscrupulous things. There's a big debate inside open AI about what to do with this. It helps to know, I think, that Dario Amadea is a sort of neurotic by temperament. He is also part of this community of AI safety warriors in the Bay Area. At one point in 2016, he's living in a group house with Paul Cristiano and Chris Ola and Jared Kaplan and all these leading lights of AI and AI safety. They are all sitting around at night, reading posts from less wrong and other AI safety-focused publications and talking about them and debating what the existential threats from AI might be. It's considered insane behavior in 2016 because nothing actually worked yet, but they were sort of looking peering into the future and imagining when systems did work and could be improved through scaling what that would mean for safety. Dario Amadea, obviously, no longer works for open AI, he's the CEO of Anthropic. You talk about all of these different rivalries that have been created in his final months of the company. It's basically like it's a three front war, it's Dario versus Ilya Satskyver, it's Dario versus Greg Brockman, the president, it's Dario versus Sam Altman. What can you tell us again, shading towards salaciousness here in your brief answer to help us understand why Dario Amadea felt like he had to leave open AI and start a new company. So he had been thinking about this for years. I think the- was not a rash decision and he had actually sort of thought very seriously about quitting before, mostly because he thought that Sam Altman was dishonest. He had this sort of running list of times where he thought he had caught Sam lying to him or saying something that wasn't exactly true or just doing things that seemed like bad judgment. Like this example that I just love that I was so happy to be able to like reveal in the book because it was very salacious and very funny was that apparently Sam Altman met Kanye West at a party and kind of offhandedly mentioned that they were doing this private beta test for gpc3 and offered to let him in. And you know Sam likes talking with celebrities and having celebrity friends and you can sort of see why he would want to like show off this new model that he built to this you know very famous you know person and he brings this back to open AI and they're like what the fuck man like Kanye is not well. He had just posted a video of himself like peeing on a Grammy award. He was not yet like you know talking about Hitler but like things were not going well for him mentally and here Sam is trying to offering to let him into this product that they think could be quite dangerous in the wrong hands and so they sort of put a stop to this. There are like you know dozens of incidents like this and the way it was explained to me is that none of them on their own would have been sort of worth quitting over but together the sort of snowball effect meant that one day in 2019 he essentially just sort of decided he couldn't do it anymore. He was not going to be successful at steering open AI towards safety from the inside. He wanted to leave and bring his co-founders with him and start something new. I think one question that I've had about Anthropic is how did they go from being a company that splintered off from open AI for the purpose of protecting this technology from the forces of commercialization only to become in just a few years even more commercially successful than open AI. How can you not even defend just describe why Dario and the other leader of Anthropic went from essentially thinking Anthropic should never release Claude as a consumer product to releasing Claude as a consumer product that makes tens of billions of dollars for Anthropic a year. Yeah, it's the central question of Anthropic. When I first visited Anthropic in 2023, it was a couple years after its founding but they had not yet released like a good chatbot that people were actually using. It was like 150 people and they were all cooped up in this office and they had sort of no business strategy. It was just kind of like a research lab but ChatGPT had come out and they had sort of been convinced and part by Dario that they needed to compete. Like if they wanted to be relevant on safety, they had to have a model that was as good as any on the market. And I think part of that was seen internally as kind of like Dario just kind of justifying something that he wanted to do anyway because he hated Sam Altman so much and open AI so much and couldn't bear the thought of losing to them that he sort of concocted this rationale of he calls it the race to the top. Basically, if you are in the lead or close to the lead, you get to sort of lead by example and everyone else has to follow you when it comes to safety and making your models more aligned. So this becomes his sort of his bet. But you're right. At the start, they had no attention of making money. I talked to lots of early Anthropic employees who were like, yeah, it was not even a subject that came up. We thought we were going to build safe AI systems and do safety research on other people's models. They had this list of crazy ideas for how they could make money including having Claude start a hedge fund and give them trading tips. Didn't end up working out. But they eventually decide, well, two things happen. One is, AGI turns out to be more expensive than they had bargain for. So in this founding document that Dario writes in 2021, it's called vision for a new co, he predicts that Anthropic would need about $10 billion to build a full-fledged AGI. And as we know now, as we now know, $10 billion is a vast undercount. You need like hundreds of billions of dollars, possibly trillions of dollars to build something that will fulfill the original vision of AGI. So they need more money. And they have raised a bunch of money, but they are going to need more, and so they have to have something to sell to raise that money. So that's part of it. There's a part of the story that you absolutely have to include, which is that their top benefactor is a little man named Sam Beckman-Freed. Yes. He is the initial investor who buys that who I think owns 13% of the company's equity at one point. Yes. So they Sam Beckman-Freed. And so once Sam Beckman-Freed is in jail, then not only do you need a lot of money, you also no longer have access to the person who owns 13% of your equity. So you need to go further and you can't rely on EA billionaires. And so you put that together and you say, okay, big blob of compute. We need hundreds of billions or tens of billions of dollars. We can't rely on philanthropy. We have to go through commercial ends. That means we have to build a commercial product. Like I can see step-by-step. It just doesn't necessarily make it like the most, the most moral step-by-step logic. No, and I think they've done a lot of moral, I think there have been some stances that have been both the right thing for Anthropics mission and the right thing for their business. And some decisions that have been more the right thing for their business than their mission. I think these deals that they've done with Gulf States, their willingness to raise money from dictatorships, I think has been their own employees have flagged that as sort of an instance of hypocrisy. And I think they will continue to face these pressures because they are now a huge company. And they are the fastest growing company by some measures in the history of capitalism. And yet it is still not big enough. They still have not reached where they are going. And so they will continue to have to raise money and make commercial sacrifices. And this is what I think is maybe the most interesting part of of this framework because and I'm quoting from your or summarizing your book here rather than coming up with this idea on my own. But I think it is both very important to point out that Sam Altman and Dario Amade are extremely different people motivated by not only different philosophies of the world, but even literally different books like Sam Altman is Ray Kurzweil and Nick Bostrom guy superintelligence and then you've got Dario Amade reading about the making of the atomic bomb by Richard Rhodes. But despite their extraordinary differences and mutual self and mutual loathing, they both end up in the same bucket of accelerate. They both have to release chatbots that make tens of billions of dollars a year. They both have to raise money from benefactors that aren't necessarily liberal capitalists. And so no matter where you start out in this race, you end up in the bucket of speed, speed, speed. I mean, that does seem like an important takeaway from this story. It's a funny thing, isn't it? It's like whatever your motives are when you start, you end up racing right alongside everyone else who's racing with very different motives, but you're kind of checking your sides to make sure that no one's pulling ahead of you. I think that's a really important point. And I think it's part of what is freaking out the insiders at the labs, the people like Jacob Coxon, the people who are writing essays and open letters about how we need to slow down. They see their bosses engaging in this kind of ego-driven brinksmanship, where it's like it is clear that part of the reason they are racing so hard is because they can't stand the thought of these particular rivals winning. And I think it makes them very uncomfortable. And it makes me very uncomfortable too. I wanted to introduce our last character here, Dennis Asabis. He might be the least famous of these three men, but he is the only man on this list who's won a Nobel Prize for their work in AI. And there's absolutely a version, I think, of the future, where his contributions might, in 20 or 30 years, be seen as the most important because of how significant the science contributions of Deep Mind are. Let's just start in the biggest picture. Who is Dennis Asabis? And what is he want from Super Intelligence? Dennis is a fascinating character. There's a great biography of him by Sebastian Malibi that just came out. Yes, it's one of his way more material on him than I was able to fit in my book. But basically, the story of Dennis Asabis is he is a child chess prodigy. He is a competitive chess player, you know, very highly ranked in the world for his age group. Sort of a brilliant and competitive mind. People have described him to me as the most competitive person in the world. There's this sort of famous anecdote about him, which is that this This outfit known as the Mine Sports Olympiad starts this competition called Penta Mind, where it's basically like an all-around Olympics of games, essentially. And they are trying to find sort of the best gameplayer in the world. And Demis Isabis learns about this contest, decides that's going to be him. He masters all five of the games included in the Penta Mind and wins the contest five out of the first six years. So he is the most competitive person in the world. And he's also very interested. He's a cognitive neuroscientist by training as a PhD in that. And so he's very interested in this question of what is intelligence? And how can you build it into a machine? And this is the thing that obsesses him. He becomes sort of convinced that it AGI is possible long before this is common wisdom. People sort of look at him as like a fringe exotic, like who's this weird British man talking about the coming machine god. And they set up this little company deep mind and they start making progress toward their path that they choose is teaching AIs to play games. So they start with Atari games, then they move on to Go and Shogi and all these other game Starcraft. But their thesis is basically that you can train an AI to play a game, get really good at the game, and then sort of expand the domain to include things that are less and less like games and more and more like tasks in the real world. And that that's higher going to build AGI. And this is a very different approach from Open AI and Anthropic. And they are not going the language, the large language model approach. In fact, you've got lots of quotes in your book of Demis and people working at deep mind explicitly rejecting the language approach, saying language is like a simulation of intelligence, but it's not intelligence itself. Can you help us understand how game based approaches to growing AI differ from Open AI and Anthropics emphasis on Transformers and large language models? Yeah, they're essentially competing philosophies and competing approaches to creating powerful AI systems. So the approach that deep mind uses and specializes in, the approach that helps them build this system AlphaGo that beats the best human Go players. And that's a very famous moment in the history of AI. That was known as deep reinforcement learning. This is basically trial and error. You give an AI system a task, make a go move that is likely to increase your probability of a win. And you give it a reward if it does that, and you give it a penalty if it doesn't. And then you just run it in simulation millions and millions of times. It gets a little better every time. And eventually it surpasses the best human Go players and becomes superhuman. Their view is that a perfect, elegant AI system should be able to learn without any human data whatsoever. You should just be able to stick it into a novel environment like a Go board or a chess board or something. And it should be able to kind of bootstrap its way to superhuman performance because it is learning from its experience. Open AI and transformers and LLMs are based on essentially the opposite view. They are built out of human data. Take the entire internet, jam it through a transformer, come up with a system that is extremely good at predicting the next words in a sentence, and use that to increase the overall intelligence of your model. There's this very famous debate inside DeepMind in these years known as the language is enough versus reward is enough schools of thought, which is basically the sort of microcosmic version of DeepMind versus OpenAI and Anthropic. Do AI systems develop better intelligence as a result of experiencing things and learning through trial and error, or do they develop better intelligence because they learn more of language? And language is what we use to encode intelligence in humans. I love that. Like, philosophically, I was just thinking as you were saying that. I remember that part of the book, but I didn't have this particular thought as I was reading it. The difference between those approaches philosophically is maybe a little bit like this. Tell me this metaphor is wrong. Language is enough. It's essentially like saying, if someone could memorize everything that had ever been written in science and great literature and human psychology, they would be super intelligent. They would be able to, if they could hold it all in their head, make connections and map new recombinations of ideas that would make them essentially super intelligent. That's language is enough. And then there's, reward is enough is kind of like, what if someone could live a million lives and experience in those million lives, how some decisions lead to happiness and some decisions lead to unhappiness and some decisions lead to making a company that makes ten billion dollars and some decisions lead to making a company that sucks. You wouldn't. Those million lives learn all of the positive and negative reinforcements that would lead you to becoming super intelligent. And so to kind of interesting, I think of like, this is like sort of a borhesian way of sort of summarizing these ideas. Like, would you rather, if you wanted to be super intelligent, memorize every word produced by all of the geniuses in human history or just live a million times? Like a funny way of putting it. And that is, that is almost entirely true. And as it turned out, in the end, you needed both. Today's, today's models are a combination of LLM's and reinforcement learning. So they all sort of converged on this hybrid solution. But there were years where like, open AI and anthropic or like, you know, open AI and parts of Google were like very interested in scaling up language models. And the people at DeepMind just said like, that's crazy. You cannot build super intelligence out of Reddit posts. Like, it's just not going to happen. And they were, they were wrong. And they, they, they have since, you know, realized the error of their ways that are now building LLM's too. So Sam Altman and Daria Amadeh hate each other. Altman and Hassabas met at the Vatican in 2015, 2016. As you reported, they also don't like each other very much. I have a question about why they don't like each other. But I'm going to put that on ice, read the book if you want to know why they hate each other. Last question about Hassabas specifically. You know, Alpha Fold, which is the protein folding technology that won the Nobel, which emerged from this RL theory of AI is a triumph. But Google doesn't have an AI consumer business that rivals open AI or anthropic. And Hassabas initially dismissed our language models as you've just said in a way that I think has cost Google significantly on the commercial front. So it's a mixed legacy. And I wonder if you, if you stop the clock today, how do you see his legacy? I see it as as both an incredible scientific legacy. I mean, as you mentioned, he's the only one out of any of these guys who has a Nobel prize, who has produced something that has actually revolutionized a scientific field today. I think his theory of change was correct in the broadest sense of like he believed in scaling, he believed in reinforcement learning, he was early on a number of these things, including just the belief in AI itself. But I think he, the dye may have been cast when deep minds was sold to Google. I don't think they necessarily had a choice. They needed to build bigger blobs of compute to run their experiments. They needed deep pockets. They needed to partner with a hyperscaler or someone with, you know, the willingness to spend millions or billions of dollars on what they were doing. But I think that ever since he sold deep mind to Google, he has been, in some cases, fighting against the commercial incentives and the pressure, the short-term pressures of being part of a large search-based internet giant. And, you know, he explicitly tried to get away from Google. There's some reporting in the book about this thing, Project Mario, which was sort of his attempt to kind of carve deep mind out of the larger Google apparatus. But that didn't work. And then he ended up running a division of Google for, you know, years with, you know, thousands of employees, which is not what he wanted to spend his time doing. Most more recently, he's been sort of kicked upstairs to this, like, you know, this sort of executive chairman position. And Google, you know, deep mind is firmly part of Google now in a way that it hasn't, you know, always been. So I think there's a tragic element, too, of like, here's a guy who really wanted to use AGI to unlock the mysteries of science and to discover, you know, new things in physics and math and biology. And who, you know, sort of by no fault of his own, was stuck running, you know, a large division of a commercial search giant that made most of its money through ads on search results. So look, I think it's too early to count Google out. They have lots of money. They have lots of smart people. And I think as long as, as Dennis is there, they will have, you know, his sort of inspiration, his sort of drive. And he's very inspirational to the people there. But I also think there is this reality that they are just, they are just a more traditional company than either of these other labs. And they have lots of competing incentives and feuding internal teams and bureaucracy to navigate. And it's just going to be a lot harder for them to move quickly. So summing up, we've got Sam Altman, who I think of as kind of the JP Morgan of AI, like the mogul, the deal maker. Someone whose company, Bet on Transformers, SHIP Chat UBT, started this whole infrastructure cycle and has continued, has been very successful at using other people's ideas to build his power and the power of his company. Big blob of compute was Dario's idea, but who leads the labs in compute secured by a mile, its open AI? And so there you've got Sam. Dario I see as you described as something more like a moral crusader in the Oppenheimer mode. Someone who's both an inventor, again like Oppenheimer, but also someone who's who's safety crusading both inspires some people and also rub some people the wrong way. You've got a lot of that in the book about how it opened AI, some people were just so tired of Dario using the moral argument to try to win every single corporate fight. And there's Sasabe, whose historical analog is harder to come up with because in some ways, he's kind of like the Nikola Tesla, like a great inventor who was a little bit in the shadow of his contemporary, but it doesn't work because he's like a Nikola Tesla who worked at GM or something because he's, as you said, one blocker for his success is that he's been inside of a bureaucracy rather than leading his own thing outside of that conglomerate. Final question to you, now that we've sort of talked about their different motivations, different inspirations, who do you think got the most right? I think scientifically, Demis and Dario got the most right both by betting early on various forms of scaling. Here's Sam Altman, not a scientist. I think he got the politics more right than they did. I think he understood what it would take to make all of the deals, to secure all of the compute. There's this incredible scene in the book where he's basically scheming to get around Elon Musk, one of his Nemesis, to get into the good graces of the newly elected Donald Trump. And I think he is a savvier political operator than either of his rivals and maybe than anyone on Earth. I mean, he might be one of the savvier steel makers and operators we've ever seen. And I think that has been his downfall at times too. So yeah, I think they've both, they've all gotten pieces of it right, but no one has gotten everything right. I think Dario's gotten the science more right than anyone. You can go back and look at these papers he was writing in 2016 and 2017. It's like he's an oracle who sees the future. But he is too honest. His fatal flaw as someone put it to me who's close to him is that he cannot tell a lie to save his life. I mean, you could see him at that gaggle on the White House lawn the other day, just sort of fidgeting and being visibly agitated and nervous. Well, all of the other AI CEOs are there just kind of, you know, smoothly kissing up to Trump and kissing the ring. And so I think that is that is a quality I admire in him, but it is also something that has caused him an anthropic a lot of trouble. Finally, I think it should be said. We've talked about three people in this book. There's a lot of characters in the AGI Chronicles and one character that's a little bit like a forest gump who just appears in every single story. Elon Musk not only co-founded OpenAI with Sam Altman, he also tried to hire Dario Amade to SpaceX and tried to acquire deep mind and Demise's work. I guess after he initially went to Google in addition to funding some of Demise's work. So it it is remarkable how like just cropping up through the book scene after scene is Elon Musk being like "Hey, by the way, I'm holding a string here." It's an amazing story, I'm really, really glad you wrote it. It's a wonderful, wonderful book. Thank you so much. Good luck with it. Kevin Russe, thank you so much. Thanks, sir.