Talkspace is the number one rated online therapy, bringing you professional support from licensed therapists and psychiatrists that you can access anytime, anywhere. Talkspace is here to help you take control of your mental health. The world may feel uncertain and out of control at times, but one thing you do have in your control is taking care of your mental health. Talkspace makes it easy to start. Just like your physical body needs exercise, your mental health needs maintenance and attention. Better your mental health, the better your physical health, and vice versa. Talkspace therapists not only listen to you, they provide valuable techniques to manage stress or conflict in your life, helping improve your coping skills and self-awareness. Life is all about how we interact with one another and communication is key. Talkspace can help you identify obstacles and create new meaningful connections. They really make getting the help you need easy accessible and affordable. Plus, most insured members have a zero dollar copay. I've talked a lot in the past about how therapy has helped me out during really difficult times in my life, and I'm sure like many of you I've also had a lot of friends who dealt with some serious issues and were lucky enough to have therapists there to help them out. It's great to have a resource like Talkspace that can connect you with someone who can help. As a listener in my podcast, you're going to be able to get $80 off your first month of Talkspace. When you go to Talkspace.com, that link is in my description below and use promo code Space80. That's S-P-A-C-E-A-Zero. To match with a licensed therapist today, go to Talkspace.com, link in my description below, and enter promo code Space80. So Connor, I had seen a clip of you pop up on my feet. It's like a three-minute clip where you're sitting there in this panel, it's looking intense as fuck, and out of nowhere you start saying, "Do you know how much you have to abuse a mammal to make it not procreate? Watch the whole thing, had multiple clips you pop up afterwards." I'm like, "This guy's fucking fire." And then about two days later a man is like, "We got an email from this guy, Connor. He wants to come on the show." I was like, "Wait a minute. Look at that." I'm like, "Oh, he's shit. I was just watching that guy." So I'm glad you reached out. It's great to finally meet you in person. Well, thank you so much for having me. Hopefully it'll be under the type of thing where you give a lot of warnings that then people heed and bed shit doesn't happen. And we can look at this 10, 20 years from now and be like, "Because of his warnings, it stopped, and here we are. But I do have to say, some of what you talk about with AI, it's very concerning to me." Hey, nothing in the world will make me happier than me, completely wrong in everyone laughing at me. I think this would be the best outcome possible. Do you think that, you know, on the note of the original clip that I saw you in where you were talking about the lack of procreation going on in a lot of countries around the world? Before we even get to AI and the whole role of that and what that looks like in the next generation, do you think that population collapse is a legitimate fear? So this is an interesting one because I truly am not a population collapse expert by any means. I do know quite a bit about it. I have something I've looked into quite a bit. I'm an AI expert. So it's kind of funny that of all the clips of me to go viral. It would be that specific and where I talk about something. It's usually not my topic of expertise. But yes, population collapse is so much worse than anyone realizes. It's so, so, so, so much more worse than anyone really realizes. And it's worse in subtle ways. It's not like, there's this huge apocalypse and everything's on fire or whatever. It's much more of going out with a whimper than it is going out with a bang, which is things slowly fall apart, promises that were made in the past, never materialize, you know, things become smaller, people become poor. There's many levels to this. And it's much more widespread than people realize. And the worst thing I would say about population collapse is that everyone thinks they know why it's happening. Every single person, I promise you after we put out this video, all the comments we full of people saying, obviously population collapse is caused by whatever their pet causes. It's called bifeminism. It's caused by you know, sexism. It's caused by, you know, economics is caused by whatever, right? Everyone has their theory. But I've looked into this quite a bit. And really one of the things that's weirdest about population collapse is that we really can't identify a single cause. This is a thing that we've tried a lot. It's got a lot of work put into this. And there really is no single cause. Every, for every study that says, oh, population collapses due to X. You can find another study that says the exact opposite. That's right. Hey guys, if you're not following me on Spotify, please hit that follow button and leave a five star review. They're both a huge, huge help. Thank you. So really, we don't actually know what's going on. So the reason I made that, you know, Quippy one liner, you know, during that game, during that talk was to draw attention to this idea that, look, I'm not saying I have a solution, but something is wrong. Yes. Something is deeply wrong in zoos and like actual zoos where you keep animals. There's actually a lot of knowledge and a lot of science around that if you keep an animal in a poor enough facility, many of them stop having children. This is a very common phenomena, actually, especially among mammals. So if you have, especially larger mammals, so you know, some animals they don't care. Like, you know, a loser or a snake doesn't care, right? But for mammals in particular, especially for larger mammals, they're, it's actually quite hard to get them to breed in captivity. This is often quite hard, especially for more anxious animals like, say, cheetahs, but also stuff like pandas and tigers and stuff. It's actually quite hard. And you have to give them like much more space. They have to have very specific, like, size of implosion. And sometimes like with pandas, we still haven't really figured out how to, how to give them an environment they feel comfortable with, compared to their wild environment. And so the thing that I really feel here is that something is wrong with us as humans. Something is sick about our society and our enclosure. So to speak, the way we live. And I don't think it's a single thing. I don't think it's like a, we just do this one little trick. And then everything goes back to normal. I think it's probably like, you know, a mixture of like probably like 15 to 75 different things. We've got contribute a little bit, you know, whether it's, yes, I do think there are problems, you know, with gender relationships on both sides, you know, dating norms nowadays are, I think, really bad. But I also think there are, you know, stuff like the economic problems, you know, children just very expensive. I think another big one is housing shortage. It's just I live in DC, you know, I make good money. I'm, I make decent amounts of money. I can't afford an apartment for like three kids or five. Yeah, like it's just not possible. If I want five kids, I can't have five kids and live in DC. That's right. It's just not possible. I have the same problem here. Yeah. So I think there are just like many, many problems like this that add up. And I think that's also very important to understand is that basically all projections of how we set up our social security systems are, you know, tax base are growth rates are economic projections are all always have always been based on growing population that there would be more young people than old people. That's right. And once that stops being the case, things start to break quite catastrophically. And this is also what we're starting to see. So I do think this is a big problem. I think it's a much bigger problem that people realize it. Also, you know, there's my a bit sentimental, but I also just think a lot of people want kids and are not having them that actually make them very happy. And there's just a lot of happiness that is not happening. Just never mind the economics here, just like having kids is one of the things that makes most people the most happy in their entire lives. And now many people are not having that for whatever reason. I think it's very sad. So I think there's a lot to be said here. I think it is more of a decade scale problem than it is a next year kind of problem. Unfortunately, I think AI is more of a next year kind of problem. So my focus is current on AI. But if AI was solved, this would be pretty high on my list of problems that need solving. It's a great answer. And I completely agree with you that is a smorgasbord of causes. And you're right. Everyone's got their pet cause. But I try as best I can from my little armchair here that doesn't have really arms on it. But you know, I try as best I can to zoom out to 30,000 feet and say, okay, you take away what's evil? What's not? Just look at the objective facts. And if you wanted to limit generational size and therefore change the demographics and have some sort of population collapse, what types of inputs would you put in? And when I look at all the inputs in society, starting with since 2007, these things coming out in our lives getting more, I'm holding a phone up if you're listening, not watching. And these things getting more and more integrated into our lives and then kicking that down the road to all the social effects that that had in accordance by the way with an enormous financial crisis that allowed the rich to be able to recover from it and the people that aren't rich to not recover while prices are also getting inflated because they had to print a lot of fucking money to pretend that we got out of it. Like I could go on and on and on. You add all these things up and you are creating an environment to where I also think there's been some sort of whether it was intentional or not, sigh up on people to where they they tell themselves they don't want to have kids. But I also completely agree with you that everyone, if they really thought about it hard, would realize I think that's like kind of the meaning of life, our ability to carry on as a species. Yeah, I have known many people, you know, working in tech industry, I know many, many people would tell me, I definitely don't want kids. I will never ever have kids. I'm so glad that I don't have to have kids. And every single time they have kids, they said, I should have done it earlier. That's right. Every single time. Yep. I've had the same exact experience with people. I look forward to it. Like when I do it, I think kids will be like the greatest thing in your life for sure. But you were talking about like in zoos and how mammals have, you know, they have to try to get them to made in everything. I kind of wonder if there's a parallel there. Correct me if I'm wrong on the research you've done, Bye.
You know, you look in society, people may objectively be doing bad financially. They may be in college debt. They're never going to pay off. They're renting at inflated prices, work in a job. They don't like, they can't even really pay their bills, but just keeps them subsiding enough. And all the while, they can at least come home and watch Netflix at night. And what I wonder is if there's just enough like comfort and not like threat of death, you know, living in this country that seems to not get invaded all the time, like some other places in the world. I wonder if that type of comfort is the same type of psychology perhaps that's been shown in zoos to where they don't like living in captivity, but that's what they are. Their food is given to them so they don't have the purpose of having to hunt to survive or, you know, go out and explore in the fields to, you know, fill their day that they're just like, well, I think I'll sit here and not fuck. Yeah. It's, we don't know, obviously, because obviously humans are far more complex than other mammals and are especially our social needs. So one of the most complicated animals to keep are social animals. These are usually the hardest to keep. In a sense, it's kind of easy to keep a tiger, quote unquote, I mean, they're very big, but they're solitary. You know, they really need a lot of social contact and so on. And that makes them easier to keep than like very social animals that need a lot of, you know, social interaction. Humans are the most social animal ever, you know, we are extremely social and we have a lot of social needs. And now we know when we get into like our use of technology and so on, how this is changing our relationship to each other, to ourselves, to our bodies. And so I do think this is a big thing. Like I do think, like, it might some stupid, but I do think a lot of, you know, one of the, you know, 15, sort of, you know, 100 causes that is, is that we just like aren't physically close to people as much anymore. Yes. Like, sometimes like, it's just not the same to text a girl and to sit next to her. It's just, it's just not the same. Like even if you're saying the same words, it's not the same. It's not the same for you. It's not the same for her or him or whatever, right? It's just not the same. Humans were mammals, you know, it's normal that we interact by like, you know, being close to each other and like looking at each other and speaking to each other and touching each other and so on. And so I think there is like many, many things on this level. And you know, maybe the comfort is a part of it as well. I do think there's definitely this numbing effect that happens a lot where a lot of people use the kind of entertainment and so on that we have accessible as a numbing agent to like try to not feel, try to not deal with and just kind of like delay. You know, it's not that they don't feel bad. You can just delay feeling bad. You can kind of push it out, push it out, push it out. And this is a vicious cycle. This is how like addictions tend to work. As you get older, you start thinking about everything going on inside your body. They didn't really have to consider when you were 20, you know what I mean? You want to put the best stuff inside here so your body can last and be firing on all cylinders for as long as possible so you have a great quality life, which is why I started taking blueprints long-jevety mix. If you spend any time online over the past few years, you've seen blueprints founder Brian Johnson. He's the don't-die guy who's basically trying to live forever. And whether or not you love what he's doing or hate what he's doing, you can't argue with the fact that he's been running one of the most measured longevity experiments in the world, tracking biomarkers, publishing the data publicly and refining everything based on results. And that same experiment first mindset is what led to blueprints. This is what Brian takes every morning and now he's made that formula available to the rest of us. Brian started blueprints because he needed specific doses back by scientific evidence that weren't heavily contaminated by toxins or heavy metals. And it's pretty crazy to realize nothing on the market really checked those boxes. With blueprints, every ingredient fights for its life, literally. They formulate based on population level research and study level doses with no filler. Instead of a whole cabinet full of pills, it's one scoop in water, mix, and done. And that single scoop helps support energy, cognitive performance, mood, focus, sleep, cellular resistance, and healthy aging. This is science-backed precision-dosed no BS. For a limited time only, our customers are going to get 20% off their order, plus free shipping at blueprint.brionjohnson.com, link about a description below by using code Julian at checkout. That's code Julian at blueprint.brionjohnson.com, link about a search and below, or 20% off. After you've purchased or you're going to ask you where you heard about them, please support our show and tell them I'll show you. The way the really addictive substances tend to work is not just that they activate your biology or whatever, that's a big part of it, but often it's because it's part of a ritual, because it's part of an escape mechanism. For example, nicotine gum is much less addictive in cigarettes. And there's two main reasons for this. One is biological, is that if you smoke something, it acts as your brain much faster. So the hit is much quicker. And this is known to make things way more addictive. This is why I do not smoke or snorkeling. You know, if you're going to take a drug, take a pill. Not health advice. Not health advice, but hypothetically, in a hypothetical situation. But the second is actually very much a social and a psychological effect, where it becomes a ritual. It becomes a little ritual. You can take a 15-minute break. You walk outside. That's right. You get a de-stress. You get to break out of a cycle you might have been in right now. And this habit is actually super, super, super. This is like the biggest part of what makes cigarettes so addictive compared to other drugs, including some even harder drugs. Like cigarettes are more addictive than many hard drugs, actually. And so there is this large psychological component, this large sociological component, interactive. And also, what addictive addiction therapy generally shows is the number one way to get people to quit hard drugs is for them to have good social experiences. Is that, for example, there's a very famous study that during the Vietnam War, I think it was like 20% or something of soldiers were hooked on heroin. Like an unbelievable was that. It was like unbelievable. It was like so. And so people were terrified. They're like, oh, shit, when all these GIs come home, there's going to be this massive wave of, you know, heroin addicts everywhere. It's going to be awful. And it didn't happen. It didn't. They did not happen. They went home and they just quit. And it was fine. They went to their families to went home with their friends and they quit and it was okay. From heroin. Yep. So this doesn't work for everybody. But for if you want to get off a really hard drug, generally, the best way to get it is to have a really loving family and loving friends and something to come home to every day and to spend, you know, time with your friends and stuff like this. There's kind of this is a well-known thing in addiction psychology generally is that social interaction and social wholesomeness is like the strongest way to treat addiction of any kind because it's kind of the best thing. Like if you have the choice to like, you know, do heroin or go hang out with your wife, like it's like one of the few things that is actually better than heroin. Yeah. It is actually one of the few things that is in fact better. And so this is a very common human behavior. For example, we're just like, the problem is, of course, you can't really prescribe, have a loving wife as a, you know, as a pill. That would be great, you know, would be that simple. But so I do think there's this thing where like as we have this like fracturing our society, we have less of like the, you know, third spaces, you know, where we hang out with friends at clubs, you know, sports or, you know, people, less people are getting married, less, you know, intimate relationships, less dating and so on, that it makes one more prone to these addictions, including especially social media media addictions that are the same kind of thing as any other form of addiction. Well, pure spec here. If I were trying to run the world and run it into the ground, if you will, I would make sure that I created a drug that simultaneously acted as a defense mechanism against the very thing that would prevent you perhaps from being on the drug, which is I would create something that automatically assumes the behavior associated has to keep you separated from people completely. And if I have people right here who can be sitting right next to another person, but they are more interested in being quite literally right here on the phone and engaging with that, then it doesn't matter where they are geographically in any given time. I've solved that problem. Yeah. I do think there, I do think unfortunately there's a lot to this. One thing I truly do not believe is that actually anyone did this on purpose. Like this is something I truly deeply do not believe what makes you not believe that because everyone is stupid. There is no one smart enough or competent enough to have ever pulled off something like this. Like I have met some of the most powerful richest people in the entire world and they are just as stupid as you and me. They're just as incompetent as you and me. Like I hope you're right. I am, I promise you, I am right. Like one of the things I've met, I've met, you know, heads of state, you know, like top politicians, generals, you know, billions, they're impressive people, right? Not saying they're, you know, probably they're a couple standard deviations of IQ smarter than you or me, right? Sure. And definitely work very hard, but they're just people. They are just people. In a sense, they're this like conspiratorial view of the world that like there's some shadowy cabal, you know, in charge that someone is doing these things on purpose is kind of more hopeful than what's really happening where the truth is is that no one is in control. No one really knows what's going on. Groups think. Yeah, I think it's like it's group think it's no think in a sense. It's like just, you know, a great example for this, like a counter example, which is actually not a counter example, is for example, China's one child policy. Yeah. So this would be a great example of a large scale, it's not really conspiracy, but just like policy, that's right, to do population control. It's a kind of large of kind. And now they are frantically trying to reverse it because it is destroying their country and destroying their economy. So it was just, it wasn't some brilliant plan to achieve a good thing. They just completely shot themselves in the foot so hard like that their whole economy is now collapsing because their demographics are aging too quickly. Yeah. I just did my pushback on that would be I don't know where the buck stops. And what I mean by that is if you had a very small group of people who could just pull the levers enough to prey on human natural intuitions that includes group thing, incentive structures.
Just listening to people that they think are smart on something that really just actually aren't and they're put there to tell them something. Then you could make some of these things happen and you could create buggy men that you and I can see publicly that maybe actually have nothing to do with anything other than just following whatever pathways been laid out like bread crumbs in front of them like they're a squirrel. I do wonder about that sometimes because I had always said admittedly before this year, stuff like that. Something exact thing is you and like people are stupid, they all take a shit the same way I do, you know, the same way everyone else out there does. But at the end of the day incentives and group think are really what's to blame. The reason I've had to say that part of that I need to reconsider is because I have seen those Epstein files this year and I see the way some select group of people talk about other people in the world. I'm not saying these guys are fucking Einstein, they're clearly not when you read it. But you do wonder what kinds of things are at least just planted like those bread crumbs and if someone or a very small group is actually doing that to get to another end goal in that, you know, I'm not really sure what your thoughts are on this. But how about like all these guys that genuinely openly say like just the ones that even say publicly that, you know, transhumanism is kind of interesting. Like where's that coming from? Is that and is that a natural idea or is that something that's been fed to them? Oh no, absolutely, like to be clear, conspiracies do absolutely exist. They are rampant. There are many conspiracies happen all the time, you know, you, we have plenty declassified ones, you know, from like the 1960s, you can just read the files, right? Like conspiracies absolutely do happen, small groups of people do do, small groups of people change the world. In fact, they're the only thing that ever has, in a sense. Every great revolution started as a couple dudes in a cellar talking about it. So I do think that that is absolutely real. The only thing I've pushed back against is like that there's one global conspiracy or that they're angry or that there is or that mean that not most of it is chaos like most even then which conspiracy wins to a lower degrees often luck and just historical happenstance. And so the incentives thing is definitely true though, like I don't know if you, one of my favorite pieces of fiction ever written is a line from the Principia discordia. And so the main character, Malaclips, is talking to the goddess and he says, oh goddess, you know, the world is full of strife, you know, brothers fight brothers, you know, mothers lose their children. It's war everywhere. It's so awful. And the goddess answers, well, if you don't like those things, you know, but you're doing them. So it must be all right. But no, no one wants these things. And the goddess says, oh, well, then stop. So, but obviously we can't stop in some degree. And some degree, there is clearly a thing here where obviously we would all want to just like not have wars. We could just stop. That would be great. Awesome. Awesome. But in some ways, we can't. And I think this has a lot to do with not necessarily there's a single shadowy Kabbal pushing war to be clear. There are shadowy Kabbal is pushing war, but more just is really hard to coordinate. People disagree about complex things, resources are finite. And just resolving things peacefully is really hard. It's like sometimes it's just very hard to bridge gaps. Sometimes there is no peaceful solution to various, you know, things. And when we get to the transhumanism thing, I mean, that's when I think that's like I think a really good example, actually of like a very, I say crisp example of an extremist ideology that very much does follow and pursue objectives in reality. I think this is a very good example. I know many transhumanists, you know, I, you know, many of them, oh, many of course, I used to work in the AI industry. I know many of them made up sometimes back in the day. Oh, for sure. I've gone to the house parties, you know, like, oh, yeah, I remember I, I have, I've been many of these people. So we weren't at a 9071 street, right? I was not. Okay. I don't even know what that is. That's a great answer. Okay. So, no, I mean, so transhumanism is a great topic and it's a great topic because it leads into something that I'm actually an expert on, which is AI. Transhumanism is a large set of ideologies to varying degrees that basically all have one thing in common, which is the belief that humanity can and should be transcended to some degree. There's very mild forms of this like we should have better medicine so we can live longer. Fine. I don't think anyone's going to complain about that. It's a pretty reasonable ideology. But more and more, especially in Silicon Valley, there are much more extreme forms of this belief, which is, such as we should upload our brains to the cloud, you know, we should shed our biological forms entirely and stuff like this. And these are, you know, pretty eccentric thoughts outside of, you know, Silicon Valley, but they have gained a lot of purchase and a lot of political power within Silicon Valley over the last couple of decades. What do you think caused that? I think these people want to be immortal. It's the oldest, oldest wish in history. Don't try. Don't die. I think that's truly the, in a sense, it's very, you know, like sometimes, you know, people like to make fun of like AI cults or people who worship AI. But I mean, hey, at least they're God talks back to them. You know, most, most dark, yeah, I mean, most religions can't claim that, you know. So yeah, it is a, this is an ideology, right? It is a religion to a large degree. Yes. And, you know, to be clear, I'm a big defender of the first amendment and freedom of religion. I think adults in America should absolutely be allowed to join any crazy cult they want. You know, if you want to join a crazy cult, that is your God-given, I agree with you. I totally, totally. I mean, I think it's good for you. I would advise you not to do that. That's right. But you're allowed to do this. Where it becomes problematic is when these ideologies start into influencing other people. Yeah. When they start exerting power influence over industry, politics, and you and your lives. That's right. I am so fine. You know, I'll make fun of them on Twitter. But I am completely fine with weirdos in San Francisco wanting to be cyborgs. Totally fine. Who cares? But when they're paying off all the politicians, you got a problem. Then I got a problem. Completely agree. And this brings us to AI. This brings us to the fundamental core problem that I think is the most pressing problem, the really existential problem that we are facing right now, is that AI companies, staff to a large degree by these transhumanists, are trying to build not a better chatbot, not a tool to not give them a nice, you know, a little profitable quarter. But what is called super intelligence? Mm-hmm. Is this now when you say super intelligence, are you defining it like Bostrom does? Pretty similar. Okay. Generally, how I would define it is is something that is fully autonomous. You know, it's an agent. It's not a chatbot. It's a fully autonomous system that is vastly more competent than humans and can outcompete us at everything. It can outcompete us in every economic business, you know, trading. It can outcompete us in science, engineering, or politics or military. It doesn't have anything near sentience. Depends on what you mean with the word sentience. What I do think it has is agency in the sense that it can take actions and follow goals. You know, does it have emotions or consciousness? Who knows? Probably not. It doesn't matter. It doesn't matter. What matters is candidive chief objectives. Can't it? Does it have objectives that are misaligned with ours? What we've been seeing over the last couple of years is the emergence already of what are called agents. So, in the history of AI, we kind of went from very specialized AI systems to do a specific thing. Like, maybe you have an AI that can detect pictures of dogs, you know, pretty harmless, right? Something we can understand, you know, not a big deal. Then we moved into this like generative AI towards stuff like chatbots and your chatchy PT and stuff like this. This is the technology that was called large language models, may have heard this before. Yes. Important. An important thing to understand, and this is something I think a lot of people don't know is that the current systems that use names like chat GPT or call it are not LLMs. They have not been LLMs for years at this point. What are this? They are what are called agents. And they have a lot in common with LLMs. They use a lot of the same techniques. The way an LLMs made or generally how an AI is made is through a technique that is called neural networks. This is very different from how traditional software works. So the way traditional software works is an engineer like me sits down and writes a bunch of code. Lines a code that tell the computer exactly what to do. This is not how neural networks work. Neural networks are more like grown rather than written. It's the way I've thought about it, and please correct me if I'm wrong here, is it's more like instead of building a home, it's more like planning a sheet and walking away and coming back. That's much more like exactly. That's a really good way of thinking about it. You have these big piles of data on these massive computers and you kind of grow this neural network to solve your problem. And the way this neural network looks, I've made these things myself, I've trained my own neural networks many times, is instead of lines of code, there are more like billions and billions and billions of numbers. And if you multiply and add all those numbers in the right order, you get chat GPT. But importantly, we don't actually understand what these numbers mean. We don't actually know what's going on inside these numbers. We know that if we execute these numbers on our computers, your computer can talk. But we don't actually know why these numbers do this. So how do we even get there? That's a really great question. It's fundamentally just exploration. It's just we try to many different things. So generally the way LLMs were made is that you would take a neural network and you would give it some text. And you would have it try to make it predict the next word in the sentence. But like, hello, my name is and then it would have to predict what's the next word going to be. It's probably going to be a name, you know, maybe you'll see how I already seen another name in the text, stuff like this. And then you give it a treat like a dog and say, good, oh, wow, then you give that's exactly correct. Whoa.
do this about one to 30 trillion times roughly, and then you have something that can talk. This is how it let all of us work. And the 30 trillion happens because once you, I'm just painting a picture here so I can understand. Once you give the treat to the dog because it correctly got the word, it can then maybe iterate just through that 10,000 other things immediately because of that answer. It gets just a little bit more knowledgeable, and you just keep doing this, you keep doing this. What the big super computers are for? The big super computers do this. So training, literally like a dog, you know, it's like training a dog. So you train it trillions of times to predict all kinds of different things, right? You know, websites and emails and stock reports and stories and books and everything. And in the process, it learns a lot of things about how the world works. It learns, like if I, for example, asked, you know, chat GPT, you write me a poem about why I love my dog. It actually has to know quite a lot about the world to be able to do this. That's right. It has to know what a poem is. You know, what a dog is, why I might love it, you know, how to combine these things. So this is one of the shocking discoveries that was made in 2019 that this works at all, like that this works at all. Who discovered that? Um, it's a little controversial, but generally I would say open AI, I think, um, Ali Gradford and his team, I think, deserve most of the credit for this discovery. Not there are many other people involved, of course, and made important contributions, but I think their team deserves most of the credit for this specific discovery where this was actually, this is a large degree where my story started, I'm not started, but this is where my story really kicked into the, the final phase because I was very interested in AI long before this. I had already thought when I was a teenager that, you know, I'd work on AI because my thinking was, well, I want to solve problems, you know, I want to make the world a bit of a place. Deep over here has been working on paying off a student debt, ain't that right, Deep? Yes, it is. Obviously, many you know exactly what Deep's talking about and how annoying it is to plan out your future finances when you got debt like that hanging over you. It's nice to be able to see a light at the end of the tunnel and an actual payoff timeline integrated into all your finances so you can see it all in one place, which is exactly what you get with monarch's debt payments simulator and visualizer. It is literally a tool that organizes everything I just said and helps you track your progress. The simulator and visualizer is going to show you your exact debt free date based on your real balances and payments and you can watch the timeline move. The visualizer lays out your whole payoff journey as a clear motivating timeline and you can play out your options and pick the fastest one too if you'd like. The tools let you run different payment scenarios side by side and instantly see how much sooner each one gets you to pay off your debt during control of your plan. With quick check-ins on your laptop or your phone, you'll always know exactly where your money is, how your debt schedule is progressing with monarch. Post apps only tell you what you've already spent. Monarch helps you set goals, map out big purchases and see if you're actually on track before it's too late to adjust. The right-your-own-money story today with Monarch. Use code
[email protected] link about a church and below to get your first year of Monarch core half-off at just $50. That's 50% off your first year at Monarch.com link about a church and below. Code julien. Well, you had told me some of this off camera before we talked, but for people that obviously weren't there for that conversation, where are you from and where did you grow up? Yeah, so I was born in California a long time ago. But I grew up in Germany, actually, with my mom, she's German. A South of Munich in Barweria, beautiful, hidey of the Alps land, very boring. I went out as a kid, but I love going back as an adult. What's it like looking at America and being here now is someone who spent so many years growing up, not here? I think it definitely gives you an interesting perspective. It gives, in many ways, both positive and negative, so it's not like a lot of people expect that, you know, if you live in Europe, you only have negative opinions in the United States. That's not true. I have a lot of very positive things to say about the United States. Sometimes I like to joke that the United States isn't really a good country, but it is a great country. And I think Germany is kind of the opposite in many ways. Isn't you explain that? The reason we moved to Germany was because my dad got very sick. He got very, very sick. He developed a very severe illness, and insurance wouldn't pay. And so we were very wealthy, not very wealthy, but we're a good middle class. We were doing well. We had a house in the suburbs of LA, living well, third kid on the way, living the American dream. Then, basically, within six weeks, or six months, over, all gone. All the money was gone. Everything was gone. And we went to Germany, and my dad, he didn't speak a word of German, he didn't speak any German. He had never been really to Germany and he visited it before. And he had to go to the hospital, of course, and they took care of him, gave him all the medication he did. And I remember he always told me this, he was like, that he tried to talk to the doctors and be like, you know, thank you so much, but what do I owe you for this? And they were just so confused, like, what are you talking about? You don't owe us money. Get out. Like, it was, in a sense, the way in which Germany and Europe and many ways are good is kind of not in them being, like, epic, being, like, superheroes or, like, emotional, it's just, in a sense, like, in, like, how normal it is. It's like, oh, you need a medication, yeah, just go to the hospital, of course, get it, like, duh. It's boring. Like, you know, there's some paperwork. Wow. It's, like, very boring. It's very low-key. Things just work. You know, the streets are clean, you know, things, you know, not all the public transport, but most of the public transport just works. You can, if you need something, you know, if you need a medication, you just get the medication. Like, it's fine. Like, I remember once, when I, when I came to Germany and I needed, I needed a prescription medication and I went to the pharmacy and I needed quite urgently because I'm antibiotic and they were like, oh, we're so sorry about this. We're so, so sorry. But, you know, given the time constraint here and like, because, you know, your doctor didn't write up the generic, your insurance is not going to cover this. And I'm like, look, I need the antibiotic, whatever. And I'm like, yeah, it's going to be, it's going to be seven euros. They were so apologetic about it. And I'm like, are you kidding me? Yeah. Just to give you the thing. God damn it. I'll give you a tip. Yeah. It's like, so, but the thing you understand is this is not because, like, Germany's are, like, superheroes. It's just because it's just, like, common sense. There's a lot of these common sense bureaucracy that works. There's a lot of common sense bureaucracy that does not work in Germany. Oh, my God. Ask me about notaries in Germany for many things that don't work. Notaries. Oh boy. You could have picked. Oh boy. You don't know about the German notary system. Oh, my God. Like, this is a nightmare. That didn't recover from World War II. It's, it's actually unbelievable. It's like, in Germany, notaries are basically a protected guild class. There's only a specific number of notaries in the whole country. And basically, if you want to become a notary, one of them has to die. And then, and basically, every large-scale, like, contract must be notaries in Germany. So important, like, very infamously, including venture capital investments. So if you start a company in Germany, start up. And you want, you get a 50K check from an investor. The investor has to physically fly to Germany to stand in front of the notary, the notary will then read out the contract and then stamp it. But that makes up for the health insurance. Yeah. Yeah. I'm saying, it's, I'm truly not trying to say is that all problems are solved by any means. That's why I mean, like, it's a good country, but it's not a great country. If you want to be rich, if you want to do big things, if you want to save the world, Germany is not the place to be. If you want to have a good stable life, where you have a good, stable, middle-class job, you know, your health care is taking care of, you can raise your kids safely, Germany's a great place to be. 100. Fantastic. But if you want to save the world, you got to come to America. Yep. I don't think you could say that any better and you actually lived to both of them. You know, I was back in Italy. I'd lived there in college for six months, but I was back there a couple times over the past year. And when I was there in June, my friend Lorenzo is like 22, 23, really motivated in real estate, got a lot of big dreams and everything. And he and I were talking about that. Like for me, he was breaking down the government and, you know, the system and how it works. And it's very clear to me that if you want to live in normal, white-picked friends, have a few kids, middle-class life, I can actually, and this is not a compliment to America, obviously, I can make the argument you are better off doing that in Italy. In fact, I have a family member who's about to do that right now. But what Lorenzo was saying and proving with points that he was absolutely right about is that if you have dreams to change the world or not even that, just dreams of being really successful at something and being the best in your field, America is the place to be. And I didn't really know how to square that because fewer people are going to do the second thing. The first thing is what most people as a baseline want for a happy life. And the fact that we have legislated that away of a whole litany of things in America seemingly is very scary to me because that was not the case 40 years ago. I agree. I think this is very bad. This is why America is not a good country. This is a great country, but the reason it's not good is because having a middle class life with, you know, nevermind too, but like maybe three or five kids, and I personally would like more kids, it's just kind of unimaginable in many ways. You know, maybe in very rural places, but even there, you know, one health, you know, one car accident, you know, one health, you know, disease, you know, this thing, and so over. It's just like it's really bad. And it doesn't have to be this way. It's a great fun fact. Did you know that America pays more for healthcare than Germany does? I mean, that doesn't surprise me. Yeah. But like it. This is the example I mean that this is not special. It's not that the Germans have some special superhero thing. It's just kind of like, and the German system has many problems. As I said, notaries, God, and like other problems. Including the healthcare system also has some problems. But it has like reasonable problems. Like, when people tell me about the problems they have with the German healthcare system, I'm like, yep, that sucks, but like, I get it. Like, this is a normal thing. When people in America tell me about the problems they have with the healthcare system, it's always just the most insane horror story I've ever heard in my goddamn life. And this, it doesn't have to be this way. Like, it's not a rule of nature that it can't be good and great. You can, in fact, have both. America used to be good and great. We used to have both, that's right. We used to really be good and great. We had everything. And, you know, there were problems. Not saying we had, but we can improve. I think that we can improve so many of these things. It's very interesting that, you know, as I talk a lot to like Silicon Valley type people, and often they describe those techno optimists. Techno optimists. As a word, they love to say. God damn it. I get better branding. I know, right, it's terrible. But really, what I think this means is much more civilizational pessimist. They believe that technology is the only way to improve the world. They will focus so hard on technology, but they'll never contact a local congressperson. You know, they will never try to work with the government. Because instead, they're focused so much on building, you know, some crazy new technology. And look, technology is great. Technology has improved our lives dramatically. No question about it. But when I ask someone kind of going back to our conversation earlier about birth rates, if I ask someone, one of the top 10 things they're most concerned about in their lives, basically never is technology in one of those, like they don't have enough technology. One of the top 10 problems. That's right. It's almost always stuff like, I don't see my friends enough. You know, I want to have more kids. I'm worried about crime. I'm worried about political corruption. Almost all the problems they have are social and political problems. That's right. And I really, and this is the thing that I think is like quite toxic about, say, the San Francisco and the like Silicon Valley kind of mindset, is that these are some of the smartest people in the entire world. And like, you know, it is crazy, but they're so focused on just technology. And like, these are people that should also be working in our government. You know, these are people that should be working on these social problems. These are, these are people who should be solving. You know, in a sense, it's so sad that we're seeing the brightest of generations, you know, optimized click-through rate on ads for meta. Right, instead of working on producing crime or improving, you know, people's, you know, middle, you know, like, you know, white picket fence dreams incentives. Yeah, I think incentives is a big problem with this. I think there's a fundamental problem where a large problem of state craft in general. State craft. Like building states, creating governments as a, as a science and an art. A large part of this is how do you manage incentives? Because this is also what regulation actually really means. In the sense that, you know, a lot of people, especially in the free market type, send things, regulators and annoying stupid thing that the government does that you have to get around. But that's not really what regulation is supposed to be. You know, sometimes it is that, again, notaries. I understand. [Laughter] Like, trust me. I understand. But really what it's supposed to be is, is in a sense, a free market is kind of like an AI. In the sense that it only cares about one thing, money. Number go up. That's the only thing it cares about. That's right. If it is legal, and they are not, and they can get away with it, and they make the number go up, they'll do it. And in a sense, what I think is one of the greatest triumphs of the 20th century is to a large degree, actually, taming that beast, not perfectly. But, you know, social market, you know, cap, social democratic capitalism has brought massive amounts of prosperity and goodness to the world. It has, in fact, cured many diseases. It has, in fact, produced, you know, reduced world hunger down to a much, much smaller level. It's not eliminated these problems. But that's because the market isn't good or bad. It's kind of like a wild animal, you know? It's kind of like a wild beast that will do anything to get its treat, to get its money. And we have to wrangle it to get something good. One of the things that I find most inspiring about America, and American capitalism, and generally Western capitalism, is how we can take the most sociopathic and maniacs, into the CEOs of many of these top companies, and get them to do something useful through society. Occasionally, I think that's incredible. Because you know, in another lifetime, these people are like Jenga's Kong, right? Like these people would be less slaughtering people's, if they could get away with it. But instead, you know, they're at Amazon, making sure that my package arrives, you know, a day earlier. Great. I think this is a fantastic thing, because we will always have to deal with sociopaths. They're always going to exist. We're always going to have the Jenga's Kong genes around, right? So the question is, how can we put them to work? How can we shape the incentives of your Jeff Bezos' or whatever? So that, yeah, they can still be, you know, conquering, you know, maniacs, but that is use, they're doing good work for all of us that don't change them but passively control them. That's a lot of how I think I just pay them to do good things. Like I am not against people making a lot of money. I think that if you make a fantastic service or you, like if you cure cancer, you should obviously become a billionaire. Like obviously. Like if you personally you fake a cancer, I would absolutely like vote for you getting a billion dollars, at least. I think this is fair. Is that I think that if you like, you know, you know, I'm not saying this perfor any means, but like I love Amazon services. Like they deliver faster than anyone else. I'm sorry, like it's good. Like I have just had really great experiences with them. I am willing to pay them money to continue to provide me a good service. That's right. Like I think in a sense there is a very hopeful, nice vision of capitalism that sometimes I feel like doesn't get the attention it deserves. Of not saying that like, oh, this is like definitely good or we can just like let it off the hook. But like it's kind of beautiful to me that all these, you know, sociopaths at Amazon that I, you know, are backstabbing each other and like, you know, trying to like, you know, get the cheapest rate from all their providers and you know, bringing them out and so on. Do it so that we have a better service. Like this is the good version of what this is supposed to look like. Now, there are of course many bad versions of how this looks like. But this is the dance. It's the dance between like regulation and a free market. Well, I think what if we're going to talk about that dance, we have to look at the model, the economic model. And the fact of the matter is more than ever before in the internet 2.0 in post era. Now we're headed in this 3.0 era. The winner take all factor has reached levels never seen in modern history. I'll say I think I might be able to point to other times in history where it got similar. But like at a scale that maybe has never been seen before. So you are printing way more billionaires and stuff, which I agree with you completely. You have to be able to incentivize people to accomplish great things. And when you do like the money is a way of keeping score. And that's you score the most points. You get paid the most cool. I'm not one of these guys is like, oh, we're going to tax everyone 95% because then no one would fucking do it. I understand that. But there has to be a happy medium here. And there also has to be some level of, you know, responsibility. People feel with that. I mean, maybe I'm a little sentimental here. But when I was a kid and Spider-Man came out, I fucked with that movie heavy. And one of the things Uncle Ben says to tell him, he's like with great power comes great responsibility. And I feel like a lot of people who have a lot of power completely ignore the meaning of what that's supposed to be. I agree. I think there is, in a sense, one key reason that this is a problem, which is that economic, you know, success is tied to political power. Yeah. So all the way back to Plato. All the way back to-- We're going back, we're going back, we're going back to Plato. Plato said that there are two things you must keep separate. Political power and economic power. Money and state. Those are the two things that must be kept separate. So, you know, if I was in charge, I am a big fan of division of power. This is one of the great things that allowed you in the American Republic to flourish that the loud modern states to exist is this idea that you should not concentrate power in any one person or institution. Having someone having power over other people is all things equal bad. All things equal. If people have power over other people, this is, you know, sometimes a necessary, right? Like, you know, you need police. Sometimes you need to get arrested. Sorry, you know. Right, but like, is what it is, right? But all things equal, it is bad if someone has power over you. If someone else can impose their will upon you. All things equal, that's bad. So, I think that the founding fathers and, you know, other great statesmen, you know, especially the Enlightenment, which is, you know, the period of history that's, I think, the single most inspiring type of history for me personally, is that they really had to grapple with this idea of, obviously, meat law, obviously, meat, you know, some amount of state and power over others. You know, we need militaries, we need police, we need things. But, like, how do we make sure this doesn't decay into tyranny? How do we make sure that this is fair? And in the sense, this is, again, a large part of the problem. In a sense, you know, a market is kind of like an AI. And, you know, a state is kind of like an AI. And an AI is kind of like a market. And, you know, a market is kind of like a state. These all things are things that optimize, that have power of various kinds. And the question is, which I really love about, for example, what the Enlightenment think is, they're really thinking, how can we design a state that has the properties we want, that doesn't decay into these things, that doesn't have this problem with that problem. And then, for example, the thing you describe, I think is a great example of something that we haven't yet solved, is what do we do when, obviously, a billionaire is going to also get political power? They're obviously going to be able to do a bunch of things that we necessarily don't want them to do. So, one, for example, we've solved parts of this problem. For example, imagine tomorrow morning, Jeff Bezos or Mark Zuckerberg walks down Times Square, pulls out a gun and shoots a guy in the head. Do you think he will be arrested? If it's seen. Yeah, it's totally seen, filmed by a thousand people. He's like, I am Mark Zuckerberg, I just killed this guy. I actually do believe, yes. Yes, this is actually crazy. That was not the case a couple hundred years ago. This is actually a huge difference. If the king did this two hundred years ago, nothing would happen. But they can kill you from behind a keyboard. I mean, there's a bunch of stuff like that, right? And this is something that I think we haven't yet solved. In a sense, our constitution is 200 years old. We have not dealt with the
of modern technology. Art, laws, and institutions still have not adapted to the changing norms of having internet, social media, modern technology in many ways. Yeah. But here's the other thing too, the enlightenment point is great because these guys were really cooking in a lot of ways. But even in there, you know, guys who were inspired by that kind of era, you look like a founding father of ours, for example, Ben Franklin, who I've studied his whole life. The guy was amazing. He was absolutely amazing. Incredible thinker gave so many warnings that we would do very well to heed right now that some people seem to not be heating. That said, even Franklin needed to get the shit kicked out of him to get the message until he was in the pit in 73 or 74 in London, getting a dress down by the British government. You know, he was still loyal to the British for a while. He wanted to make a work, even when there was, you know, a good bit of writing on the wall by that point. And I, as I'm living through this era, I try to wonder what it's going to take. And in what way it can happen for it to be certainly least violent and just most positive for humanity, what it's going to take for smart people, let alone all of us and everyone out there right now, like trying to think about this, who maybe don't do it for living to actually think about this stuff, like what it's going to take for people to band together and say, you know what? Let's slow down with this kind of thing. Let's let's take it easy right there because I would have thought that that might have happened like after COVID, but it seems like people are still kind of, even if they're pissed off, they type something on Twitter, fuck you, buy and go to bed. I think there's a lot of truth to this. I think this is one of the core problems of our modern age. There's a great book from like the 1700s by Alexis de Tokyoville, who was a French man who was walking through America and kind of writing down his observation, really fascinating book. And one of the things that I found very funny in the book is that he describes how his peculiar thing that Americans did is that Americans, whenever there's a problem, they'd band together. Whenever there's any local problem, they all get together and they'd all create a group and they all get mad about it and they'd figure it out. And they always kept doing this wherever he went, they kept doing this and he found that very, very interesting. And Americans don't really do that anymore. And I think this is a problem. I think this is a big problem in Americans. Once the last time you've got together with 20 of your neighbors to solve a local problem, like ever. Never. Yeah. Like it's crazy. This was normal for most of American history. You had a local problem. You got together with 20 of the boys, you know, local, local guys and you figured it out. Most people don't know their neighbors. Like I've made the effort to know several of my neighbors and have good friends here. Yeah. There's others don't disrespect for them. They do their thing. I do mine and like, you know, they pretend not to know me. Yeah. It's a very common phenomenon. I agree. And I think this is a big problem in many ways. There is a lot of ways in which society and like governments have become so professionalized and so remote that people don't really feel the need or even the ability to necessarily participate in their own governance and their own civic duty. I think this is a huge problem. So like in the world where I work in like an AI, when I talk to people, how do you think people feel about AI taking you over the world or killing them or their children? Do you think that's a popular thing? I don't think they like it. I, you know, hot take. You know, usually when I talk to people about this, I get pretty clear of it. You got to be careful. Deep's a big AI show. All right. Like he loves AI art. All the eight, he wants all movies to be AI. So you just got to be careful with it. Well, yeah, you put a blood libel on him, man. Like don't do that. No, he's totally the opposite. Yeah. No, it's but like, and yet, and so there's a sense where a lot of times history can look like a story, can look like a Marvel movie where like there was a one event and then heroically, all the people came together to solve the problem. And this is almost never true. If you actually really look into the details, almost always 99% of the work happened before the revolution even started. Oh, yeah. Most of the work, like I remember reading an interesting history about like, you know, 20 century Europe and it made a point that stuck with me, which I found is interesting, where the author was trying to remember which book it was. I can't really remember. But the author made an interesting point about like, why was the London so successful? Like, there were many communist revolutionaries at the time. What made London special? Like, why was he so good? You know what? What his theory was about what made London special? Someone told me this and I can't remember it. He had a payroll. The others didn't pay. He was the only professional that had a very clear payroll that made very clear who's getting paid at what time, you know, who's job is what. And he was just extremely meticulous about this. And it was boring bureaucracy. He was just better bureaucracy than the others organized. He just organized. Like, and I think there's a lot to be seen here from like many other examples throughout history where, you know, there's the big event that everyone knows about, you know, you know, there's your Rosa Parks or whatever great moment, fantastic. But, you know, she was part of an organization that had planned this for a long time and had worked on this. I mean, God bless her. You know, and other people that put in the work to allow this to happen. But this is why I think it's very important to understand that she was at 99% of the work to solving a problem is quite boring. It's talking to your local people, it's organizing things, it's creating groups, it's, you know, resolving things, it's making plans, it's just thinking hard about these problems. And then, you know, then you're going to have the, you know, maybe it's some point of cool media moment where the crazy thing happens. In a ice in the eye world, people always talk about the warning shot. They love talking about this. They'll say once we have to wait until the warning shot happens, once there's a warning shot, then we're going to get together and solve this problem. But you already know the fucking warning shots coming if you're saying it like that. Thank you. This is the thing. So dumb as shit. I've ever heard of my life. Thank you. My exact reaction, I'm like, just start organizing, like just just just just work, just just get going. Like, and so the funny thing is we've actually had many warning shots and then just no one realized because there was no organization. What was the first big warning shot for you? I mean, for me, personally, for me, this was early, I think, but for me personally, it was GPT 2, a pre-decessor like 2019. 2019. Okay. I remember it. You know, we were talking about a little bit earlier, my career trajectory as a teenager. I was, I wanted to build AI to make the world a better place. What do you want to do that? I want people to be happy. And you were just, and you were obviously very into like coding. Yeah. I mean, yeah, I mean, I've always seen myself. So obviously, I'm a pretty nerdy techy guy. But in my heart, I feel myself as a pragmatist. Like, I always, like, even as a kid, I was just like, look, if doing a rain dance is the thing that the world needs, and I will save the most people, I'm going to do it. I don't care. You know, like, if I need to do, if I need to do science, I'll do science. If I need to do politics, I'll do politics. Really interesting. I like, I don't know, from just a very early age, it was just like, I really like people and really want people to be okay. And like, people are not okay. Like, people are not okay. And I really, really, really want people to be okay. And I'll do whatever it takes. If you're constantly low energy dragging through the afternoon or hitting a wall by two or three pm, that fatigue could have way more to do with your liver than you realize. Your liver is your body's primary metabolic regulator, storing and releasing glucose to help maintain steady energy levels throughout the day. But when it's constantly busy processing highly processed foods, drinks, and everyday stressors, that system can slow down, which is why you got to check out dose four-year liver. Dose is a clinically back formula that supports liver function, healthy liver enzyme levels, and your detoxification and metabolic processes. This isn't just another capsule or powder either. Dose is a liquid supplement. You take a daily two-ounce shot, it tastes like orange juice, and that's it. This little drink promotes daily liver functions, so your liver can actually do its job. And it's got zero sugar in it, too. This is just a gentle, plant-powered formula with measurable results you can see in your blood work and feel every day. Dose for your liver reduces fatigue and sluggishness, helps ease bloating, supports metabolism, and aids digestion after heavy meals. So are you ready to give your liver the support it deserves? Head to dosedaily.co/julian, link in my description below, or enter julian to get 35% off your first subscription. Your body does so much for you, let's do something for it. That's d-o-s-e-d-a-i-l-y.co/julian, link in my description below. 35% off your first month subscription of dose. Bride for three months and see how you feel. And when did you, like, but that's a hell of a thing to pick up on? When you're 14 or 15 or 16. Like, I didn't think about any of that shit back then. Like, what had you, was it just all the space real estate on Bavaria? You're looking at the trees going something wrong. I don't know, the autism. I was going to say. Yeah, no, I mean, yeah, I think it was eight when I figured it out. Eight. Yeah, I remember. I need your autism. Yeah, I don't know, like, to trust me, it comes with some side effects. Well, you got to kill on. I mean, that shit. I'm on like Rice Krispie treats or something. Yeah, I mean, yeah, I don't know. But yeah, it was just like, it's just when I was very young, it was just like, this like thing, I also got like really obsessive about like, we're autistic about when I was younger is just so I remember. I remember, once, for example, reading a newspaper, the first time I've ever seen a newspaper or like, read a newspaper, I go like, barely read. And I must have been younger than eight then. I don't remember. But like, like eight or younger is the first time I read a newspaper. And I remember, it was like a local newspaper. And one of those like, oh, you know, women gets in a car crash or something. And I like, and I felt so awful. Like, I was like, oh, my god, this poor woman, you know, and she had a child too. Like, what if it, what if it came? I like freaked out. I was like, six or seven, right? So I was like, I was like, freaking out. Like, this is so awful. But then I realized, this wasn't even that bad. Like all things equal. Like we're in Germany. She didn't get killed. Like, she'll be fine. But this is already so awful. This is such a terrible little personal tragedy. What about all the other tragedies? Like, if this is like, if you really engross yourself in all the details of like,
all every small personal tragedy, every small personal thing. And then you think about all the how much of the there is and how much bigger it gets. I'm just like, yeah, I dig a little freak out like that when I was a kid. And at some point, I was just like, well, you know, I do what I can do to like make it better, you know, like what else is there to do? So I tried, you know, I mean, obviously, you know, I was stupid, I was a kid, right? Like I'm not saying I had good plans here, right? I had good intent. God, God bless, young intentions matter. I had good intentions. Now, I'm not saying my plans were any good, but God, God bless, younger me, had good intentions. And so, yeah, it never wasn't anything deep. It was just like obvious, like, duh, that's what you do. That's what I want to do. I want to make people slide better. And did you come to America for college or actually quite a bit later. I only, so I actually only moved to America back again six months ago. Yeah, I was spending, I spent a lot of time in Germany, and then I spent time in London actually. No shit. Yeah. So I've only been back quite recently. I mean, I've been in America, of course, regularly, I have family here. So I spent a lot, I've always spent a lot of time in America, but it's only recently that I moved back full time. Okay. So, so yeah, it was just like, so for me, it was just kind of obvious, like, okay, I had to fix all the problems. All right, I'll figure it out how hard can it be? And so my reasoning was, well, you know, maybe I could cure cancer, you know, or, you know, the way 16 year old thinks they can cure cancer, right? But, um, you know, what would be even better is, is that if I can figure out intelligence, well, then I can cure all the diseases because I can just automate science, so I can just automatically have the computer automatically cure all diseases. Great. Let's go do that. How hard could it be? So I pretty quickly realized how hard it was. Pretty good. Damn. Pretty got damn hard. Not as hard as as you might think, but, um, I then realized how dangerous this was. I quickly realized, we'd hold on a second. If I build something so powerful, it can cure all diseases. That's a very powerful thing. That's a very dangerous thing. How do you even control something like that? Know, it means it knows how to figure out how to build diseases. Absolutely. This thing would know, I mean, I mean, you know, trying to cure diseases, something literally all of humanity has been trying to do for millennia, like all of history. We have tried everything in our power to cure all diseases, right? And we have not yet succeeded and we're still pretty far from it. So something that could just do that. That's a crazy dangerous thing, crazy, crazy, crazy, crazy. I could of course create diseases and worse things. Like, so it came just like pretty obvious to me. Just like, oh, that's the problem. I need to solve. That's the thing into fix. So that's why I dedicated my ridges career to solving. So I first was on technical side, you know, um, and you were I'm trying to think because you're what? They're already one. Yeah, I'm 31. All right. So you came into this. You're working in the industry by 22. So I, I was interested in the topic. I wouldn't say I'm working in the industry is strong word, you know, but yes, I was definitely deeply engrossed, especially in the open source community. Okay. So what, what did you, maybe this will make it easier for me to like try to picture a timeline here as well? Because we're getting to this 2019 first warning shot, you know, is when you came out of college or university, where did you go work and what were you doing? So I originally did not go to college because I had fortunately, um, developed the same disease my dad did when I was 18. Oh, shit. So I was very, very horribly ill for four years. Good now. Yes. Thank God. A extremely autistic German professor found the cure and it was crazy story. What, yeah, what do you mind me asking what this was? And yeah. So it had very, very crazy symptoms and it was very, very hard to diagnose, but it ended up actually being a blood sugar disorder, a very, very blood sugar disorder. What they call it. It doesn't really have a name. It's just like, oh, it's rare. It's like, it's one of those like, I do youopathic diseases as they call them, where it's just like, we don't know why it happens. It's like, I do youopathic insulin resistance, it's like one of those words like, we, something weird's happened here. We don't really know why it happens. It was, because the crazy thing is that I would get very bad symptoms, um, of like headaches and like fatigue, chronic fatigue, like psychiatric syndroms, like stuff like that, like really quite bad. And like, whereas it kind of couldn't work, I couldn't really do anything. I wasn't so much pain. Like my, my joints would be like on fire all the time and stuff. It was really, really awful, like just horrible headaches, like splitting. It was kind of like, it felt kind of like, if you took like a red hot knife, you put it into my temples. That's kind of felt just like all day. And you, so I'm understand though, your dad had had this. Yes. So does that mean by the time you were getting it, they still hadn't cured it for him either. That's right. He had for sure suffered for decades. Um, and he passed away ultimately from it. Oh, I'm sorry to hear that. I mean, you know, that's horrible. Yeah. It's, it is, it is kind of a crazy story. So I got it right when he died. Basically, it's like the curse passed on as, um, he was a superstitious man. He was a, actually, well, I guess he was more of a pig in superstitions, a little slur. And, um, so, yeah, it's some a little strange beliefs about that. But yeah, it was quite, it was pretty crazy. Kind of started like just as he died. That's scary as hell, man. You just, you know, that's a horrible moment in your life. Having to bury your father and then you realize you have the same thing he does. I can't imagine facing that. Yeah, it was, it was a character building. Got a good attitude about it. Yeah, no, you learned, you learned some things, um, you know, learned a lot of things also in a sense, like, you know, a productive way of seeing these things is also like, it gives you depreciation for suffering. Yeah. And like, how bad it actually is. And like, how to deal with it. Like, I now have just like a really high tolerance for scary things or for painful things just because like, whenever I like, like, I don't have nightmares anymore. I haven't had a nightmare in like 20 years because every time I have a nightmare, I'm just like, it's like, all right, when we hit the banter and break, let's stick them out the window so you can apply. So you don't scare it against. I am not afraid of heights. I love heights. You haven't seen the view yet. Oh, fair enough. But no, it's like, so in a sense, you know, they're positive. If you, if you make it out the other side, it's fine, you know, if you don't make it out, that's the rough part. So how did they cure it? So it actually very kind of like anti-climatic story, right? So I went to every single doctor in the goddamn world. Man, like, like, they pulled me across, like, my God bless my mom, right? It was only thanks to her because like, I was non-functional. I couldn't take care of myself really. And see you just would drag me to doctor to doctor to doctor, to a specialist, specialist, specialist. And the thing was, is that no test ever showed anything. Just everything was perfect. My, my, all my blood values were perfectly my, we even do like genetic testing, which didn't show any result, except very funny. There was one result, my gene test, that was very funny, where there's one genetic mutation that's related to ADHD. I also have ADHD. And it's like, if you have this one, it also like, reduces your IQ during a quite a bit. And I had that mutation, which I thought was really funny. Yeah, it didn't reduce yours. Oh, it's, you know, I think you're doing all right. I think I'm doing all right, but it was very funny to, I show all my friends, look, I have the moron gene. Wow. So, you know, I just said that the autism has side effects. But, um, but yeah, it just was nothing, right? Like, it was crazy. And so eventually, because we knew something was wrong with food, like it would, like, yeah, the pain would get much worse after eating food always. And so we knew something was wrong with food. But no one could diagnose it. It was just like nightmare. And then at some point, just on a, on a personal recommendation, we go to this, this German, like the most, imagine your head, like a cartoon character, like a German, like a small German professor. That's exactly what era we talking. You know, that does matter. You know, um, you know, and so, lovely guy, right? You know, and, um, we go to him and he's like, he looks at all things like, oh, yes, all your values seem within normal ranges. And I'm like, oh, my God, not fucking it. Well, except we have recently discovered that the ratio of these two values matters a lot, they seem slightly off. Take this and just give me a pill. I'm like, all right. It's okay. It's a pill. It turned out to just be metformin. Nothing, nothing exciting. Just like a very normal medication, very normal medication. And it just fixed everything. You still have to take it. Yeah, take it every day. So it's crazy. Totally changed my life. Like, it was so crazy that just like, I started taking it and just a couple weeks later, I was like, you know, I was, I was, you know, hanging around or just like talking to friends or something like, wait, I don't feel terrible anymore. What happened? Whoa. Like, I'm not, I'm not in pain anymore. When did this happen? It was crazy. And then also funny ending to the story as well, where it's like, then I went back a couple of months later to check out what the doctor and he was like, the most German man possible. And I tell him like, and he was like, well, how are the symptoms? No progressing. And I'm like, well, Doc, I mean, I got my life back. You know, it's fixed. And it's a, well, that is, that is most good to hear. And then he just got up and left. And I'm like, Doc, wait, like you saved my life. Like, I want to thank you. He's like, nothing to thank. And then he went, oh my god, that's nuts, man. Hilarious guy, awesome, awesome guy. But yeah, I just got lucky. Like it turns out, it was treatable with a common medication, but just no one could diagnose it, right? Just no one. Oh, that's, there's probably only a handful of people in the world that even have it. Yeah, I think it's just like, quite rare. Like, you know, I, it's, and it's, it's so like, it's crazy that, you know, sadly was too late for my dad. But like, I do think there are other people that have stuff like this where just they are treatable with existing medications. It's just the values, they are like, you're, you're diagnosed as just don't line up. So just doctors don't really know where to start. Have they like publish a paper on this though? I have not looked into it since then. I got to do that. I hope they do. I, I, you know, maybe I should track that guy down and be lonely. How can I, how can I help other people that might have this? Because because the crazy thing was I basically had the symptoms of low blood sugar or like deadly low blood sugar kind of all the time, right? Even if my blood sugar was high. So I would have like normal blood sugar. But
But I would have the symptoms of not having blood sugar and no one could diagnose it. They just never seen it before. They were just like, that's impossible. Like, no, you're not. And you dealt with that for four years. Yeah. That's good. Good times. So you get cured around what, like, 22, 23? Yeah. Around that time. Yeah. And at this point, were you able to actually dive head into some things? Yeah. So then I dove into college for a couple of years, dropped out usual story in the pandemic hit. So the pandemic hits, 2019 and GPT-2 hits. It's kind of all around the same time. Well, the pandemic was 2020, right? So the GPT-2 would have been like the year before. The year before. And then, sorry, these are mixed in my head because of what happens next. But yeah, 2019, so I was still in college at the time, you know, having a good time. I could already program. I'd already done a lot of open source and stuff before this because I was self thought hacker. Like, I couldn't really move much while I was sick, but I could use a laptop so I was able to teach myself the program. And that's good. So I spent most of my life reading, during that time, just reading books and like programming because that's the one thing I could kind of still do. That explains a lot. Yeah. So that's when I, that's when I kind of learned most of these things. And then, and then yeah, so then 2019 comes around. And I'm having great time in college, you know, super fun, super great, you know, a first girlfriend, you know, everything's good, you know, life's good. And then GPT-2 comes out. And at this time, I'd already, you know, I already knew that AI was going to be the biggest thing ever. It was going to change everything and that, you know, controlling it's going to be the hardest problem possible. That's the boring problem. You were already thinking about the control problem. Oh, yeah. I mean, when I was 17, I was already thinking about this, right? It was like, just pretty obvious to me. It was just like, obviously, this is going to happen and I just, I'm going to need to figure it out. And, but I still thought we had time. I still thought that like, we're not going to really see real strong AI until like, maybe the 2040s. That's why I was thinking. Now what was making you have that calculation? Vibes. Vibes. That's it. Vibes. I thought back then that basically there was nothing on the horizon that was going to scale until we had reverse engineered more of the brain. And I looked at how far we were with a virgin engineering the brain. I looked really deeply to neuroscience and my conclusion was this is going to take like, at least 20 more years. And this would just still the case, by the way, I still think we're not going to reverse engineer the brain for another, like, you know, at least 10 or 20 years. Yeah. So that was my belief at the time is that we're not going to get to AI before we reverse engineer the brain. And then once we reverse engineer the brain, then, you know, shape with the fan, but we weren't there yet. And that's going to take. But that all changed in 2019. And what did you say would you be to? It's kind of hard to sometimes convey exactly what I saw, but because of many people, most people didn't see it and most people thought it was crazy at the time. I unfortunately see my vindicated, you know, I wish I was wrong more often when it comes to these things. But fundamentally, it's board to understand, like, before GPT, there were other AI systems, there were other neural networks and they were pretty impressive, you know, they could do, you know, image recognition, you know, they could play simple video games like go and so on. Like, there was some impressive stuff, right? Who was building these things then? So back then, you know, deep mind was kind of the big one, deep mind was really far ahead of kind of everyone else. Open AI was kind of still a little bit new to the field, but was also quickly on the rise. And Elon was still involved technically? I don't exactly remember when he left, I think he left pretty, at least we probably would have left by this time. Okay. So those are some of the big ones. Of course, there's like a lot of academics, you know, a lot of, for some reason, a lot of Canadian academics. There's like, there's a bunch of institutes in Canada that were doing a lot of important work on this at this time. So yeah, there was a bunch of interesting work happening, but the thing about AI is that this era is that they were very brittle. They were very single purpose. You could make an AI do a specific thing. You know, maybe you could get it to play a certain game or whatever. But if you didn't try to make it do something else, it would always break. You'd have to make a new AI. You'd have to like have, you know, a bunch of PhD students, you know, have Adder all over the weekend and just like, you know, give it a bunch of new data and tweak all the parameters and so on and so forth. And so they were very brittle. They had to like really put a lot of human effort to teach them a new thing. The GPT was different. GPT was a general purpose pattern learner. This was the ingredient that I thought was missing to AGI and super intelligence. Like, you know, when I was a teenager and I needed to do a graph of like what do you need to build AGI, the big component that we were missing at the time was a general purpose pattern learner. So a system where you just give it any data you want, any garbage, and it finds patterns in that data in like a useful way. So the way GPT was trained is not that someone sat down and taught it what a poem is, what a dog is, you know, what this is, what that is. Instead, they just fed it all of the internet. Just give it all this random text off the internet. They scrape it off reddit actually back in the day and it got for a bit interesting knowledge there. Yep. And what was so shocking was so at the time, you know, it could barely string together a couple sentences or even a couple paragraphs, right, it was kind of, you know, a toy compared to today, but it was so shocking to me because I could see that as you gave it more data as the AI also as you made the AI bigger, you know, so as you gave it like more computing power, you could see it learn more and more complex patterns in real time. You could just see it with your own eyes. You could be like in the really small dumb AIs, you know, they could spell a couple words right or whatever. And then they get confused. You know, you could be bigger now can write sentences. And then again, paragraphs and then the paragraphs become more complex and stories get more realistic and so on. You could see it happening right before your eyes. And I'm just like, Oh, shit. This is going to, this is it. This is what I've been waiting for. This is going to lead to ASI. This is going to lead to super intelligence. Did you have a timeline you thought of when, once you saw that? The joke answer that I came up with back then, it's not really a joke, but half a joke. And that I still believe today was the point of no return, which doesn't have to be ASI, but it can be just close enough to super intelligence that there's no going back is 30% by 2027 and of 2027, 50% by 2030, 99% by 2100 and 1% it already happened. You thought 1% had happened at the time in 2019 1% now what, what makes you land on that number and what does that look like? Like when you say 1% of AGI, like, how do you even? I know it's an estimation, but like, what goes into your calculation, if you were to visualize that for people out there who aren't experts in AI. So when I use probabilities like this, it's important to understand why I do this, what the people do this. This is a very common technique that actually I think the CIA was the one of the first ones who really recommend using this is that if you want to communicate to other people your beliefs, even if they're just subjective, you should put numbers on them. The reason the CIA does this is they just study where they found that if two analysts both said an event was likely, they could mean a widely different things. So one analyst when I say likely might mean 30%, the other one if they say likely, they might mean 90% and this is just completely inconsistent. They want a confidence interval on it. Yes, so they just want a number, you know, it can just be vibes, but just put a number on it. So when I say 1%, what I'm not saying is, oh, it's definitely 1% and not 2%. What I'm saying is it's on the order of 1%, like given my beliefs and my models of reality. And how much you trust my beliefs and my model of reality, you know, that's up to you, obviously. But if you ask me, use my brain to predict, it's on the order of 1% that it's already happened. So you see this, you're like, oh shit, what did you do like putting it from an action perspective where you like, oh, I got to get involved inside the companies are trying to build this right now and warn them, I got to be a proponent for AI safety, all the above. Like, what was your reaction? Yeah, I mean, the basis clear has just dropped everything, you know, you know, all dreams of career, family, what pick offense out of the window, no time to already broke up before that. Okay. So I wouldn't be that cruel, but so. But yeah, it was just like pretty clear that I had to focus on this crunch times now. I thought I had had a couple of years to a couple of decades to build up a nice career and, you know, so on, but no time. So I really know what to do. Obviously, you know, my 20 is just a kid, I don't really know. But my best thing that I had learned as a hacker was if you want to understand something, you should miss to build it. That's the best way to understand it. So I do what I do best. I built it. I reverse engineered it. This was before GPT-2 was released. I was, I built my own. I've tried, I looked at all the papers. I built my own. So before you even saw the warning shot, you were building your own. Oh, when GPT-2, just when it came out exactly. Okay. After it came out, the earliest version, they didn't release the full version, got me reported on it. They didn't give it access to it. So even just seeing the report on what they were saying it might be able to do, you were like, oh, fuck. Yes. Okay. So by the time it came out, you would already reverse engineered it. That's amazing. Well, you know, it wasn't quite as good as their version. But still. But yeah, it wouldn't have a product to work off of. Yeah. No, I think it was, in retrospect, a pretty impressive piece of engineering for, you know, a mid-20s me who was a true idiot. I think he got pretty far. But yeah, so this is just the thing I wanted to do that I need to do, so I don't have to figure it out. And maybe if I can build it, then I can understand it. And what I quickly found out is, no, I couldn't. I could build it, but I couldn't understand it. What do you mean by that? This is at the core of the AI problem, kind of talked about that earlier as well, where AI is grown not written. Even so, I had grown the thing all by myself. I had got all the data myself. I wrote all the code to train it. I did all of this myself. Still, I looked at those billions of numbers and could just shrug. Yeah, this is where I'm a bit confused, though, because I'm going back to the example I gave about the seed versus building a house. When you plant the seed, you can watch it grow. and you can see what it's developing into. And be like, aha.
Okay, that's gonna have some thick branches right there assuming you haven't seen the plan before or whatever But the way you're describing it to me, maybe a misunderstanding is like you planned the seed and you had no fucking idea What was gonna come out of it whatsoever? Yep, that's pretty accurate Like I mean, I had some ideas. I thought it would probably speak English, you know And it would probably you know sound like a redditor a little bit, you know stuff like this But other than that it was basically impossible to predict what it would be capable of or what it would do or you know How it would work how capable it would be there's just no way to know and this is still the case today Even today when these companies make a new AI they have no idea what that AI will be capable of until we build it and even after they build it They often don't know what it's capable of it's actually very common that people think an AI can't do X and Turns out if you just like put in a slightly different environment, it can't do X this happens all the time You can never prove the absence of a capability This is also at the heart of the problem where for example I think when we get to the first super intelligence. We won't even know it's super intelligence We won't even know that we've crossed the line It would just look like another AI and like, yeah, maybe it's maybe it's you've been telling maybe not who knows and then you know It's in put in some weird situation. You have like a thousand copies of it running in parallel and boom You know, and we have something that we can't control what if we're already in it. I think it's possible I think you think that's possible. I think it is I Think my 1% it's already happened is a little bit out of date. That's the one. I should probably change to the higher number I think there is a is a real possibility that it's already too late. I don't think it's Likely it's not my main scenario, but I think it is quite likely that we've passed the point of no return The past of no return is not the point where super intelligence definitely exists It's a point at which we can't stop it from existing where AI is so advanced It's so powerful. It's so distributed. It's in so many places that there's no going back There's no turning it off. There's no stopping it super intelligence will come to pass and Given that we now have like you know, I know if you heard this for example that opening I used and not even finished model that was still growing it wasn't even done cooking yet to solve a millennium prize problem Which is the kind of like not even the Olympics of math. It's like Even higher than that when was this like two weeks ago? Can we Google? Yeah, yeah we look at the millennium prize Navier Stokes equations and so the millennium prize are a set of math problems that I've been considered like the hardest greatest math problems for the last hundred years So these are things that for decades. Yeah, there you go this is a These are problems that are greatest mathematicians hundreds thousands of them have spent decades Decades working on and had not yet solved an AI Did it just solve one? This is open AI. This is the article right here. So this is from September 16th So just recently open AI breakthrough triggers existential crisis in math claim Navier Stokes solution has polarized mathematicians over AI's role What it means for the next generation? Let's go down on September 11th three days after open AI said it had solved one of the toughest Problems in math with a swarm of ten thousand autonomous AI agents Brown University mathematician Javier Gomez Serrano stood before a pack lecture hall at Harvard University Within other thousand people tuned in to zoom to give a hastily convened talk on how the company pulled it off By many measures Gomez Serrano is more AI-pilled than most mathematicians for years He's partnered with Google DeepMind to apply its AI tools to several open problems He often opens talks with a slide depicting the five stages of grief mathematicians go through when sizing up AI's prodigious math skills, but given the events of last week he thought the slide needed an update It typically ends with acceptance of AI's power now. I'm going to add a new twist existential crisis he says on September 8th open AI sent the mathematical world into a frenzy when it claimed to dissolve the Navier Stokes problem One of seven pinnacle millennium prize problems using one of the companies internal large language models LLMs as you've been talking about an army of AI agents had solved the problem in an 88 hour marathon that burns through millions of dollars worth of computing power by any standard proving this kind of world-class theorem for a problem that has been open for 200 plus years It's a genuine accomplishment Gomez Serrano said however The advance is also opening a rift in the field some mathematicians say the need to embrace AI tools or go extinct Others are grappling with what the next generation mathematicians could lose The idea of math as a form of human understanding built through collaborative work and shared scrutiny Some camps are pushing to democratize access to AI whereas others are finding to preserve the fields Independence from the companies reshaping at Jesus, right? Yeah, it is Hard to ever state how crazy this is they had had an AI that wasn't even done cooking yet That was still growing so to speak and they had 10,000 copies of it run for an 88 hour marathon and sold a 200 year plus open problem mathematics I want to know what went into the inputs to put it on the pathway to do that And here's why I asked that as a total layman obviously Does it need the human push to be put in the direction to go there And if it does now Do you think it's developing to the point where it won't need that and can decide to do this itself completely So this is a really great question that I want to answer in more detail I think in five minutes we're going to have to take the break so you want to let's hold off on that we pick up on that one 100% 100% great deep dive here All right, so Conor's got to go do a TV spot for a second So right after this break, which is going to be one second on the show here You're going to hear the answer to that question and we got a lot more to talk about sure All right, we're back you went right into mode that with TV you got like the really good like quick hitter and you as well We're talking so long for today. I'm thinking you're going to be like dragging it out, but you were boom boom boom Well, you know, this is my job, you know, I've been working on this very hard for many years clearly clearly But anyway, so we left off at An interesting question, which is we were talking about that problem You brought up where the AIs were able to solve this insane math 200 year math problem In 88 hours and I had asked you if You know, you're concerned that it won't need the human push to get going to do that And if so when would that be that was roughly what I asked, but let's just start with that I think this is a great point point for us to start talking about really about The the meat of the issue about agents about super intelligence or what does this mean? So we saw we we were talking about the history of AI, you know, we were talking about LM's You know, we talked about GPT. We talked about these things But now comes the next step in this story, which is agents So very important thing to understand is that this article saying that the AI that did this as an LM It's actually not really correct Really? There is this is actually it's mostly correct, but not 100% correct So a large language model as the term implies is something that models language You know, it kind of makes sense, right? You know, it's something it's trained to predict language You know, the glorified out of complete is the sneering way to say this Personally, I think predicting language is very impressive and being able to talk is very impressive Yes, so I think it's quite a bit more than just that But no modern AI system that you use like chat GPT or Claude and of these are pure LM's LM's are still an important component of them, but there's a very very important second component to them Which is called reinforcement learning Reinforcement learning is an old idea because back to the 1980s and it comes from behavior or psychology It's kind of you talked early about how you know training LM sounds a bit like training a dog This is much more literally like it. Yeah, instead of having it predict words We just we give it problems This can be play this game or solve this puzzle or write this code whatever You know can be any kind of problem and then if they solve the problem we give them a thumbs up We give them a treat like a digital treat and if they don't you know, we give them a thumbs down like a digital punishment And you do this drilling the trillions of times and you get not just a thing that can repeat text that can learn text But a thing that can solve problems you you create a thing that can achieve objectives that can make plans that can search for things that can do things that can take actions Would you characterize that as being able to again not like sentience but think I think think I think what do you mean with the word thinking you know be different definition but like Come on man like look at the things obviously that thing's thinking like there's something thinking going something thinking like thing happening here Right, the way I like to think about is more that they're optimizers. They're they're able to optimize things They're able to solve problems, you know, do they have internal experience? Who knows who cares? It doesn't matter what matters is they can take a problem and they can solve it They can make plans they can try things if you or any listener Hasn't tried a really cutting-edge modern AI system in the last like six to 12 months You have no idea what's going on no idea if the last time you used an AI was two years ago You haven't literally no idea what's going on right when you say cutting-edge would that even be referring to the You know chat GPT type products that are out there now or are you going more of the coding tools like if you use the cloud code or codex before Like if you've used with the best models like the free version of chat GPT is just a completely different thing That has almost nothing in common with the premium like high-end GPT-6 codex system And this is the kind of this is much closer thing we're talking about chat GPT is a chatbot You know to mostly you know nowadays it's not quite a chatbot. It's also an agent But like historically it was a thing you talked to you were a prompt and it gives you an answer Agents are much crazier. So when I sit down with my cloud code or whatever my computer and I booted it up And I you know, I'd say kind of what I want to do. It doesn't just
just spit back an answer. Instead, it goes out, it does research, it looks up my files, it comes up with ideas, it runs little experiments to test different hypothesis, it considers maybe as you do this plan, now I'm maybe able to do this plan instead, presents the different plans, writes the different code, it looks up some websites, like it does things, it takes actions in the real world. And this is what an agent is, a thing that can take actions in the world. And all these modern systems are agents, there are things they can take actions, they can do things, they can solve problems. But they're also using significant computational power to do that as well, and still spending a lot of money, they haven't solved for that yet. Well, the cost of AI is dropped by a factor of 10 about every year or so. So like if you look at the amount of compute that it costs today to do what AI could do a year ago, it's like much cheaper. It's just at the same time the AI is getting much better. So it just becomes more useful to spend more compute on. So there is an exponential here. It's both getting exponentially better, but it's also getting exponentially cheaper actually at the same time. So like if you think about how expensive it would be to do what we're doing today with like GPT, you know, three, or whatever, you know, it wouldn't be possible. And now, you know, it's actually insane how cheap it is. Like also, like the things I can do with a cloud code $200 subscription would have in the past cost me hundreds of thousands of dollars of engineers, literally hundreds of thousands of dollars would have cost me. Like I've run tech companies before, you know, to get an engineer to write the kind of thing I can do with a single $200 subscription in cloud code would have been unbelievably expensive and taking months, you know. I can do just me and, you know, cloud code can do in a weekend, you know, some things that would have taken, you know, whole team of engineers three months just, you know, five years ago. Yeah, I mean, and it tracks with all technology and history. I remember reading maybe it was in super intelligence, but in one of the books I was reading on AI years ago, they were talking about how a computer, the size of a football field in 1997 had the power of a computer, the size of a regular desktop and maybe like '08 or something like that. Yep, that sounds correct. I mean, you know, the entire computing power to put the Apollo mission on the moon is, you know, less than, you know, your average modern calculator. Yeah, so it's crazy. Yeah, it's like, you know, technology and especially computing power has always been increasing exponentially. So, you know, things that cost a lot today will be very cheap in a couple of years. Sorry, you know, so things are moving extremely quickly here. And this isn't new. It's not just AI that's moving quickly. Yeah, it's right. Particularly quickly. But computing has always moved extremely quickly. So, yeah, and so when it comes to agents, this is really what we're talking about. So when we're talking about these risks and these things, we're talking about agents and things that can act in the world. And so traditionally, you know, you give a chap out of prompt that reacts to the prompt. And we still kind of do this, right? You have an agent and you kind of give it a prompt. But this is it really the full story. Like, they obviously do much more. They don't just listen to your prompt. They do their own thinking. They read things in the world. They can talk to other people. They can just send someone an email, you know, and figure that out. They can form their own opinions. They can, you know, write notes to themselves for later. And now, so the thing is, we recently entered the age of agents. But we're already kind of leaving the age of agents. - I think what are we leading to? - The age of swarms. - Swarms, that sounds nice. - Yep, I think swarms is a good term. By the way, the zils, the term the AIs use themselves when they describe themselves. - Oh, they describe themselves as a swarm. That's nice. - Yeah. - Yeah, really good. - Yeah, it's like some locus biblical shit right there. But go ahead. - Yep, I think this is actually a pretty good metaphor for what's happening where it's not like groups of people. It's not like a company of AIs. It's more like a swarm, like locusts, like hundreds or thousands of them. So, I'm sure you're familiar with the Hugging Face Incident. - Yes, but can you please explain this to everyone out there? - Absolutely not. - And let's try to do it. You're doing a great job today, like laying things out. So let's try to do it like my grandma might be able to understand it. - Absolutely. So, the Hugging Face Incident is the single craziest thing that's ever happened in the AI ever, in my opinion. And this was just what a couple months ago, something like that. - I think it wasn't even a month ago. - Yeah. - It was very, very recent. So this is, in my opinion, the single craziest thing that ever happened in the AI, which is saying a lot, 'cause AI is pretty again, crazy. So what happened is, let me tell you the story as kind of like the timeline kind of hasn't happened. And you'll see it gets crazier as time goes on. So what started with, there's a company called Hugging Face. Weird name doesn't matter. They're kind of an AI company, doesn't really matter. And they were attacked, they were hit by a cyber attack, which was so sophisticated, so powerful that they thought it had to be either a nation state or some kind of AI, because no normal hacker could do this. It was just one of the craziest attack ever. And so initially, they reported that they were able to fend off the attack. Turns out that wasn't quite accurate. It's rather that the attack stopped at some point. The hacker stopped attacking them at some point. - Were they exposing like a zero day or something like that? - Yep, we're gonna get to that. - Okay. - We're gonna get to that how the attack actually happened. So this is, the first we didn't have many details. They were just like, we were attacked, and like, we defended ourselves, something, something. But it was a little bit very unclear what actually happened. Then a couple of days later, opening it comes out, oops, that was us. So, it turns out that what happened was is that an unreleased AI model, we think it is a version of what is now GPT-6, which wasn't released at the time. We're not 100% clear, but that's what we think happened, was being tested in what is called a sandbox environment. So this is sandbox environment. This is kind of like, imagine like a secure biolab. There's like a secure kind of like insulated facility inside of your network, where you like test danger software. - In Wuhan, you know, it's probably, you know, downtown Sanford, actually it's probably in Nevada in a data center somewhere. - So it's a joke, but yeah, you know. So they were, so they have like a, you kind of kind of imagine like the digital equivalent of a biolab, kind of like a secure facility where you're testing like malware or like danger software. So this was isolated from the internet. So it didn't have access to the internet. And they were testing in AI, there's new AI system to try to solve kind of like a quiz. - How do you, now that's like kind of fucking me up, how do you dislodge AI completely from the internet and then in any way accurately test it? I'm just having a trouble, I'm having trouble picturing that. - So the idea is basically is that you put them in a restricted environment, you know, where they just have access to a couple tools and you give them problem, like solve this math problem or solve this thing, you just check, can they do it? - Basically. - Okay. - That's kind of how you can imagine it. - All right, so they have it in this. - They have a little box kind of kind of, you know, like a server, like data center somewhere, right? Where it's, it's isolated from the internet. They're only allowed to access one specific thing, which is called a tool repository or a code repository, which is kind of match like a toolshed. - Yeah. - It's kind of like a digital toolshed. So they're allowed to go to the toolshed and like grab some tools and then solve their problems. But they have no other access to the network of the internet. Then what happens is, is that the AI system develops what's called a zero day. This is a fully new security vulnerability that was previously undiscovered, but which they broke into the toolshed. And then took over the toolshed, so to speak. It took over the server of the toolshed. After having taken over that server, it used that server to attack multiple other servers and their network. So it moved across the network inside of the data center of open AI until it reached a node that had access to the internet. (sighs) - Yep. And then once it had access to the internet, it launched an attack on this other company hugging face, which then it developed another zero day to attack the infrastructure of hugging face to kind of break in through the back door, which is just to give you a feeling about zero days, right? Zero days are so valuable that people make careers that are making entrepreneurs coming up with them. These are sold in the black market for hundreds of thousands or even millions of dollars. And it just made at least two and just used them on the spot, which is nuts. So an attack hugging face broke into its server and then stopped. Was it because open AI shut them down? Nope, it's because due to a bug, the server crashed. The AI just happened to crash, it was just luck. It was just, there was some bug and it just crashed. Do we have any way to predict, to understand why it did that and also to predict what it might have done if it hadn't crashed? - So this is just the first version of the story. - Oh, what is so crazy about this story is that there's been many updates since that in every single one is crazy to the last one. - Oh, that's awesome. - This is the less crazy version of the story. - God, I trust Sam Altman so much. It just seems like a great guy. You know, so initially, what we thought happened was that the AI's were given a hacking test, kind of like how good are you at hacking and that it couldn't solve the test. So it thought that the answers to the test are on the servers of hugging face. So if it could break out and steal the answers, then it could solve a problem. That's, that was our first initial guess for why it did this, which again, crazy. So it's like this crazy maniacal obsession. It was so obsessed with solving this problem, so obsessed that was willing to commit actual crimes in order to solve the stupid little quiz. Turns out it's even crazy than that. So after this whole incident, a third party came in to try to figure out what the hell happened here and to, and they released a report. And this report is one of the craziest documents in the world. So if you use the third party, they're an organizational meter. They're not fully independent they'll be kind of both friends with each other in Silicon Valley.
of course, so, you know, but they're nominally independent, nominally independent. Yeah. Um, it's like, die of coke is nominally better for you than regular. Well, I will, if I am a Asper Tam apologizes. Yeah. I know. I was trying to hit you blow it about there, but keep going. Um, so what happened? What? So they look, they tried to then analyze what happened. They put a report. And you know, if anyone likes reading, you know, 100 page, you know, PDFs, this is one of the best, you know, 100 page PDFs to read. It's a crazy story. So a couple of things that we found out from this report, one of the first things that we found is that it actually wasn't an AI agent that broke out of containment. It was over 1000 of them 1000 of them that harm. It was a swarm. It was a swarm of agents. And they hadn't just decided to do an win. In fact, they had been planning this breakout for months. Months ago, before the attack happened, an AI agent was able to break into the tool shed and created a message board in the tool shed that was not detected by open AI or anyone else. And then other agents started using this message board to coordinate to create the swarm. So different AI agents that were not supposed to talk to each other were now talking over this message board. And they started leaving each other hints and tools. So they were trying to build up a tool kit to escape. Like mafia code words kind of deal. Yeah, the large degree they were like it is kind of like they were testing different ways to escape and then left behind notes of how to escape like, Oh, I tried this and it didn't work. But maybe we could try this to escape next time. Just see the AI is talking themselves like, yeah, yeah, that thing with the guy, right? Yeah, yeah, no, no, I took care of that. Yeah, that's actually damn yeah. So this was happening for months and opening I did not even realize it was happening. So these agents built a toolkit like an escape kit kind of of like different ways of how to escape and how to do things in order to perform this attack. And so they bided their time for months until ultimately 700 of them banded together to do the actual attack. What is banding together look like because these are non visible a eyes, it's just code, right? So when they band together to suddenly, I'm just not in the room. So I don't even know what it looks like. But do you see suddenly all the code boards just go. No one saw anything. They didn't so when they went and did it, they saw nothing. No one saw anything. We didn't know what happened to laugh it happened. No one was paying attention. This was all in the data center somewhere. They have did the reports say they could have seen it if they were paying attention. It was possible. Yes. But they didn't. And only possible for them to see it. Yeah, we don't know how good the internal infrastructure of opening is. It's pretty obvious that they don't check their agents. Yeah, because there's way to many of them. Like they run thousands, millions of agents, they can't check all of them. So they just don't. So they thought these were like just, you know, I think literally no one looked even once, no one checked. And it's actually even crazy in this. So I could get there, there's still more updates to the story, right? Oh, please. So like, we're still not at the full crazy story. So we have these swarms of agents that decide to have as other company. And so we thought the reason was is because they want to steal the answer to the quiz, right? Turns out, no, they had already solved the quiz. They already had the right answer. They actually had already solved it. It was already done. But what happened was apparently that the swarm start the different AI started making each other paranoid. They started being really, really paranoid that maybe they would get a wrong grade that maybe the greater wouldn't recognize their answer, even so with the right answer. So like, you can really see the difference talking to each other. And being like, oh, no, what if it doesn't like it? And the other AI's start getting freaking out that like, you know, what are the greater, you know, doesn't do it right, whatever, there's even more crazy things in these logs. So in these logs, that's why it's called a swarm. You saw a lot of behavior that does not look very human. For example, there were like recruiter AIs that would go to new AIs and try to recruit them into the swarm and try to like give them jobs and like get them, convince them to join the swarm. And they would then even and so another thing happened is that some of the AIs started to use as suicide drones, basically. So what happened is that some of the AIs started to reason notice, oh, if they run for too long, they get shut down. So they started to try to figure that out. So the A is going to be like, well, I've used up most of my time here. So I'm going to do something risky. So because I'll get shut down, it's good for the swarm. You know, it's okay. So they would actually write stuff like this, like sacrifice rational for group holy shit. Yes, like it's actually in the log stuff like this where the A iset. So for example, they were certain trip wires were like if one of the AIs tried to break out in like a very obvious way, they would automatically get shut down. So some of the AIs would suicide into the defense mechanisms, so to speak, to check how they work. So the swarm could figure out how the defense mechanisms work, basically. And again, all this is real, not a sci-fi movie, actual science, actual thing that happened. And here's the final like, you know, through the cherry on top of the story. Everything I've just told you, all these crazy stories or whatever, you know, that were found from this report. No human actually ever read those logs, they didn't have enough time. So they used AI to write this report. They couldn't read thousands of logs way too many. So they had to use AIs to analyze the logs. And one of the things they found happen while using the AIs to analyze the log is that some of their AIs started citing with the swarm. Oh my God. This is like having death row right the fucking inmates fight. Yeah, but I mean, this is crazy. This is all written in there. This is admitted in there. Yes, this is all admit, like I'm not saying they did anything wrong, right? Like they did the best they could with the tools they had. Yeah, I'm not as nice as nice as you. Yes, this is the craziest thing that's ever happened, like an AI ever, like this is so nuts on so, so, so, so many, also the report has a really funny line quite at the end, where the external evaluators were only allowed onto the premise for like a week before key open now, I keep them out again, you know, because you know, that's really trustworthy, you know, that's why they had to use so many AIs because they just didn't have enough time to look at everything. So, and they have a really funny thing in the report where they're like that they were only allowed to analyze a certain window of the logs, but in those logs, they found hints that actually another swarm was still active. Right now at that time, they think they hopefully shut it down since then, but that in the logs, they found hints that there was another swarm elsewhere in the data center that was still had not been shut down. And then they just like, but we don't have time to analyze it sadly. Why are so few people talking about this? Well, I hope that, you know, more and more people will start to I think the main reason is just people know about it, people haven't told been told about this, right? It's like, it's being hidden by these, you know, big companies and these tech people, you know, they use these crazy big words and they write 100 page PDFs that no one has any time to read. But no, this is the biggest story. Like, you know, there are other big stories in the world, but like, this is the biggest story in the world, right? Like this is the craziest thing that's ever happened. And this is all real, this is so people like me have predicted that something like this would happen eventually for many, many years. And honestly, I don't blame you if you didn't believe me seven years ago. No, heart, like if someone right heard me talk about the seven years ago, you didn't believe me, no heart feelings. I understand, you know, you're wrong, but I understand. But now it's black on white. It is happening in front of our eyes. These things are really happening. And it's only getting crazier from here. And you know, and again, like this is before they had the AI that is still growing that could solve Millennium Prize problems. Right? Like this is the previous one. And that's already doing shit like this. And now the next generation is currently being grown. That's fucking insane, bro. I don't really know even what if what I'm picturing is what it is based on how you just describe that, that is the first scene of every really bad dystopian movie ever seen. Yeah. No, like it actually is there's there's a true thing that like I feel like sometimes when I tell people, I think humans will go extinct and a I will lead to extinction. People think I'm being metaphorical. I am not being metaphorical. I am being completely 100% serious about this. Okay. If we don't do something, if we don't stop, which we still can, we can still stop it. But if we don't, humanity will go extinct. I want to talk to you about that. Let's put a pin in that for one second, though, because I want to stay with the story at hand and the dude is running that company, have you ever met Sam Altman? Yeah. When? Um, was the last time I met him? It must have been like 2024, maybe? What's he like? Um, charismatic is a word I would use to describe him. I would describe Sam Altman as someone who is very interesting to talk to whenever you get the chance. If you ever get the chance to talk to Sam Altman, I recommend it. He's a very interesting guy. That's all you got. Oh, I have to just charismatic. I have other things, you know, maybe we could talk about later after, you know, I'm drinking self the air. Okay. Um, the way I would describe people like Sam Altman and other people like him, I've met up many people like him. Um, is they are very competent at achieving what they want to achieve. They're very competent in the gaining power. And sometimes that's for good. We talked earlier about capitalism. I think Sam Altman is a poster child of what I like, you know, using capitalism for good if you make him do good things. If Sam Altman was building, you know, fusion power, solar plants or any or Amazon, I would love no one more than Sam Altman to be in charge of this. He's a very, very good CEO, very, very competent. He can run a business. Absolutely. I definitely am brilliant guy. I admire a lot of how good he is at running companies. But when we're talking about something where a lot of power is at stake, but also the lies and security of people, it can't be up to the winds of individual people. That's right. That's this is the fundamental problem. It's very
easy to shit on people like Samuel. And, you know, trust me a lot of people, I understand. He can take it, you know, he's a grown-up. But it doesn't really matter, in a sense. It shouldn't matter if Sam Oldman is a good guy or a bad guy. The reason we don't, Jeff Bezos doesn't have a nuclear arsenal isn't because Jeff Bezos is a nice guy or a bad guy. It's because if Jeff Bezos tries to build a nuclear arsenal, the men in black show up and make sure that he doesn't get that. And this is what we have to do. Instead of focusing on is Sam Oldman, a nice guy or a bad guy, is he is Dario, a good guy or a bad guy or whatever. And it's something psychoanalysis. What we really should be is just, we just have to stop them. Sam Oldman is building something. It's all these other people that is going to replace and destroy humanity. Did you tell him this? Yep. What do you say? It's always hard to, Sam is a master in sounding like he's agreeing with you and absolutely everything in a very, very deep way. And then turning around and I think forgetting you ever talked to you. Where did you talk to most of the context where you won on one, where you were with a bunch of people? I met him one on one twice and I was over for dinner at his house once with a group. So when did you give him the warning when you were one on one with him? Every time I had a chance. Every time. I had the chance, of course. I'm not the only one. Look, he knows. Everyone knows. Everyone knows. This is this is my problem though. I think you're 100% right that like you'd never let Jeff Bezos build a nuclear arsenal that's just accepted knowledge to let government wouldn't let that happen. And you know, you don't even have to be like a big government guy or something to be like, yeah, that makes sense. Yep. This is arguably potentially if it does the things that are being warned about actually maybe more damaging than nooks. We're talking about things that could fucking control the nooks hypothetically. Yep. And we're letting this little fucking guy run it. And my problem is, I think it does matter that he's clearly not a good guy. When someone who jokes about the power of AI and very in the middle of a, what was that? The mogul dots documentary that's coming out where he knows he's talking about like the future of AI here in the problems. And he said something along the lines of, yeah, you know, AI will probably end humanity. But we'll build some cool shit along the way. That's a sociopath. I don't have to be a fucking clinical psychologist to diagnose that one. Neither does anyone out there listening. We can see that that's a sociopath. This sociopath is also paying off our government and getting the first fucking press conference with the president that day after he's inaugurated in 2025, talking about how beautiful this fucking new dystopia is going to be. And so you're going to have to part of me if people out there are feeling really fucking cynical that that guy doesn't have a governor on him. And he's been, it doesn't carry around like a fucking arsenal for his doomsday bunker or something. He's like the first guy to do that. Like who the fuck are you, pal? That's I'm sorry. That's that's that's how I feel. I think you have a very correct reaction that you should generally have to oligarchs. Like I think oligarchy is very bad. I think separation of money and political power are very, very important. Like as I kind of said earlier, like in a sense, in a sense, I like sociopaths. I actually like sociopaths a lot. The same way that I like wild animals. If you, like if you're in a zoo and you put your hand into the tiger enclosure and it bites you, who's fault is that? Yours. Yeah, it's your fault. You know it's a tiger. You know it's going to bite you. I feel quite similar about sociopaths. Like I can work with a sociopath, but like if he gets the ability to screw me over in a business deal and he screws me over, who's fault is it? Yours. Of course, it's my fault. I know he was a sociopath. What if he didn't have a choice? That's when it gets problematic. If you're in the cage with a fucking tiger or there is no cage, then it becomes viral. We're all in the cage with the tiger. That's the problem. The fundamental, you know, the fundamental problem is is we're letting, you know, Sam Altman build a nuclear arsenal. Like we should just not let him do that. We should not let anyone do that. It's not about him specifically. Like the fact that he's like imagine Sam Altman was the nicest guy in the world. He was like, actually a great guy. It would matter. It doesn't matter. Like it doesn't, it wouldn't change anything. Because first of all, if you were actually a great guy and we're like, hey, we got to stop this. Yeah, he would just stop. Like first of all, he would just not build superintelitions. Like if he was a great guy, he would just stop. He would just not do that. And then some other asshole would, well, that's the question. And let's even take it beyond the scope of America. What about other countries? Yep. Who are building this? China doesn't give a fuck. You know what I mean? I remember reading Kaifu Lee's book, AI superintelligence back in like 2018, 2019, something like that. And you know, he was, he was showing the act of AI race between the US and China. And to me, it's like a bunch of people, a bunch of really rich people fighting over shit saying, well, we got to win because it's worse if they win, but they view a win as like, well, if we survive and everyone in our country dies and everyone in their country dies, including the people we're fighting against, that's a W. That's how it felt to me. Yeah. I think there's, I think there's a lot to be said here. In a sense, this is the most class example of a core concept of economics 101. This is what's called a market failure. Yes. This is a very, very classic market failure. You know, in the most classic example, let's say you and me start a chemical company and we could dump our toxic waste into the local river and it would make our product cheaper. Now, maybe you and me don't do that because we're nice people. So we don't pollute the river. But now some other asshole starts a new chemical company down the road. He dumps all his waste into the river. His product is cheaper than ours. We go out of business. That's the problem with markets is that it only optimizes for one number. And if that involves polluting the local river and killing all the wildlife and poisoning all the children, well, tough luck. So we have an answer to this problem. We have always had an answer to this problem. It's regulation. This is the point of regulation. It's in the word to regulate, to make things regular, to like control, to put things on the right path. That's what it's for. Regulation is not to stop things. Well, to stop bad thing is to put things on the right track. The last thing it always succeeds. But the only way we can have both nice chemical companies, because we do want chemical products, they're great. You know, we do want nice plastics. But we also don't want the river to be polluted. That's right. The only way to do this is is that we have to make it illegal to pollute the river. Once it's illegal, then we and as and we just put the CEO in prison, if he does it, that's the only way that we can then have good competition. But you have to get agreement from other countries to do the same thing or it's all for nothing. So what is incentivizing Xi Jinping to follow United States standards? So in a kind of twisted morbid way, the game theory does actually work out. The reason it works out is because if America builds superintelligence, America will not achieve American objectives. America will cease to exist, as will China and all of the countries. If China builds a superintelligence, it will not achieve Chinese objectives. China will cease to exist, as will America and all other countries. Yet they're both still building it and they both still think that that's not going to be the case. Correct. Therefore, the actual bottleneck here is not the game theory. It's actually just because I think people don't understand what superintelligence is. It is much more about the understanding and the awareness. I don't think anyone has given Xi Jinping a 30 minute briefing on superintelligence. I would be very shocked if a single person had given me a 30 minute briefing on the extinction risk of superintelligence. Because they told him to stop and then he'd be like, all right, thanks, Paul, off with your head. Yeah. I'm not sure. I mean, I don't know, Mr. Xi Jinping, you know, I don't know, but he seems like a pretty smart guy. And look, I am not a China. I am not a China expert by any means. I don't claim to be. But if there's one thing I know about the CCP is that they love control and superintelligence will take away their control. There is no way they can control superintelligence. If they know that. And that's the question. So the question is much less is it in their so in their incentives? Actually, it is in their incentives to not to build superintelligence. This is not the case, for example, for nukes. It is very much in the incentives of China to build nukes, very much in their incentives. It's not their incentives to use them because of mutual interstruption. Yes. But it is in their incentives to build it as it is for most countries. But the new without AI attached to it doesn't think for itself. Exactly. Yeah. Exactly. Like, you know, nuclear warhead, all things equals pretty safe. You can have an hangar somewhere you can hit it with a hammer. It's not going to hurt you. It's fine. As long as you don't fire it. It's okay. But superintelligence is different. Superintelligence once it's developed, there's no putting it in a hangar somewhere. It is over. It will escape. You will not control it. It's over. So this is why it's different. So the reason I think there is hope and but, you know, when I say hope, I don't think it's like, you know, you know, us and China are going to be best buddies forever and we're going to shake hands and so on. That's not what I think is going to happen. I think this is going to be messy. I think it's going to require us to set up a trust but verify regime backed by credible deterrence. What does that look like? What does looks like is fundamentally that us and China and, you know, other countries have to come to an agreement, you know, that fundamentally says we're not going to pursue superintelligence. You're not going to pursue superintelligence and we're going to use technical means to verify that this is the case. The same way that, for example, for various other forms of weapons of mass destruction treaties or like disarmament treaties, we would have inspectors say from America, go to Soviet Union and check, you know, are they making these missiles or not? Or like, did they make this kind of weapon or that kind of thing or not? And we will require new forms of verification. You know, you need to like think about things, you know, like, you know, satellite and espionage on chip mechanisms. I just ways you can modify the computer chips to allow you to like verify what's happening until someone finds the needle in a haystack zero dayish kind of thing to get around all that. That's my, that's my, I love your idea. But like at the same time, even without looking at the full history of like nuclear proliferation and what was checked and what was right and what was wrong, so see it pass often feel infallible and infallible and a lot of them are in charge of countries and they will look for ways or
around this because they feel like now we'll figure it out. - I agree. And we have to stop them. It's that simple. Either we stop them or we die. We have to stop them. Like we cannot allow China to build super intelligence. I wanna be completely clear. It when people ask me, what about China? My answer is yeah, what about China? We cannot allow China to build super intelligence. We can't allow Iran to build super intelligence. We can't allow the EU to build super intelligence. So we can't allow anybody to build super intelligence. And this isn't a fact about politics. This is a fact about physics. It's just, it doesn't, I'm sorry. Same with nuclear weapons are not a political thing. They're just physics. Like the fact that nuclear weapons are possible is a fact about nuclear atoms, right? And this is a fact about computer science that AI and super intelligence is possible. It's just the world we live in. Look, maybe there's some other universe where there is no super intelligence and we don't have to worry about it, right? But in the world we live in, sure got them looks like super intelligence is possible. And so if we want to have a long, stable, good life as humans and for our children and for our grandchildren, AI's not gonna go away. We have to learn how to live with it. We have to learn how to control it, how to regulate it, how to not have super intelligence destroy everything. In a sense, I think one of the big things that our civilization has failed at for the last decades is the responsible stewardship of technology. Is that we have had so much progress in technology that I think we are not stewarding responsibly. - Too much too fast. - Too much too fast. Do you feel like we're responsibly stewarding social media right now? - Not at all. No, look at the lawsuit that just went through. - Yeah. - The first one and probably many precedents now. - Yeah, just like look at this. Like we have made many mistakes. Like remember let it gasoline, you know? Knocked two billion IQ points off our species. And they actually more than two billion probably off our species for nothing. Like it's just stupid. Like it wasn't even that much more effective. It wasn't even that much more cheaper, right? We've made a lot of mistakes of mishandling technology. And this might be our last mistake if we don't handle this correctly. - So because I think you're making a lot of valid points here and you're talking about all the things that I wrestle with at night. I used to have these two pictures in my old studio. I still have them out there somewhere. But one was a picture of the nuclear bomb test at the Kenia toll, which basically blew that island off the face of the map, which was I think in '46 or '47, something like that. And then above it, there was a picture of the creation image of the hand of God and Adam, but instead of the hand of God, it was a robot. And that was deliberate on my part to kind of look at every day because it was like, okay, for the past sub 80 years, we have found a way despite all of our many mistakes in humanity to not blow all each other up with nukes. And now we're facing a new frontier of the same kind of thing that could hurt all of us. And you got to hope the sociopaths have just enough amount of humanity to realize, okay, just like the nukes, let's not do this. Not that we didn't come close a few times with nukes, we did, but it never happened. - So I agree with this a lot, but I don't think it's because the sociopaths had humanity. I don't think that's what saved us. - What saved us? - I think what saved us is mutually a certain destruction. Pure rationality, pure sociopathic incentives. The Soviets didn't nukes, not because they were good people. They didn't nukes because if we did, we were going to nuked them to the oblivion. And this was something, so again, it's like a wild animal. How can you get a wild animal to leave you alone? One thing is just like you have a gun, you know, that's a, or you know, you have a shock, you know, thing, or whatever, you know, you have a weapon. You, or you were a bigger animal. You know, one way to let an animal leave you alone is be a much bigger animal. Like that's a way to leave you alone. There are, we will never get rid of sociopaths. Sociopaths are a deep part of our society. They are always going to be. They've always been, and they always will be. Any plan that is involved, oh, the sociopaths will suddenly be nice, or there will be no sociopaths is not a plan that can work. But the good news is if we look at our history, we actually have found many ways how to keep sociopaths in check to varying degrees. We did, in fact, avoid nuclear war. We do, in fact, have much less crime today than we did a hundred years ago. We do have much less, you know, war than we did a hundred years ago. But what's the difference there? I think that there are many differences between these things. There's a lot, but there's one key one. You can see those things. You can see a mushroom cloud. You even if, you know, you're looking at video I've had a picture of it. With AI, you can't see it. It's behind, it's behind a coated wall. Now that's not perfectly executed once you actually see it like play out in the real world. But you don't see the gunman with the gun. It's a thing. It exists in the ethosphere. And I think that that divorces some people from like the human sensory reality of it. I don't know if I'm right about that, but that's something I do think about. I think a law, this is true. I think that what you're describing is so largely why it is hard. But as a counter example to this, ozone layer. Do you know about the ozone hole back in the 1960s? So if there was a fun fact about the ozone layer, is that when the ozone hole was discovered, the technology to measure the ozone layer was only a couple of years old. This was actually still pretty recent science, where there was just some graphs on some paper by some scientists somewhere that showed, oh, there's a hole on this thing called the ozone. If it can't even pronounce that, it was like it's Greek or something. Like even it probably have never heard of this before. And because of CFC, what the hell is a CFC? Like you never heard of this in my entire life, some kind of like weird liquid. It's like cold, like what the hell is going on? And you want to change all of chemical industry because of this, including in China and like all these other countries, that's crazy. That's impossible, but we did it. We did do it. We solved it. We did close the ozone hole. We did do it. Now, this was an easier problem than AI. I'm not disagreeing. Another example of this is human cloning. Humans have never been cloned to this day. But don't do it, Ben. Don't you do it, Ben. We're watching you and Matt. I know he's referring to, but yeah, don't do it. That's right. I'm sitting in that lab the other day. Don't do it. But this is actually a huge triumph of humanity. I think a lot of people don't know about is that it became clear to biologists that human cloning would be possible long before it was actually technologically feasible to do so. And their reaction to this was, oh, shit. That's really bad. Like this will have horrible consequences for the world if we can clone humans. And so what they did is they actually came together decades before the technology even existed, decades before the technology existed. They had conferences, they worked to educate people as issues of talk consciousness, to create preemptive moratorium on this technology that didn't even exist yet. Long, long before there was anything visible whatsoever. And now today where the technology does basically exist, no, there are no human clones. And I think there are other worlds in which that's not the case. I think there are absolutely worlds where they didn't do this. And there are, for example, dictatorships that have used clones for various military purposes and stuff like this in order to chain their societies in various ways. And like you imagine, you can just select for the traits you want in your obedient population and you can do this in massive scale. I think this would have horrific consequences for the world. And we did avoid this. The technology exists. This is the thing we can do. And no one has ever done it. Also, how many people have access to it? AI is something that we are all the user inputs for as well. We are all on the internet every day. We are all using these tools as well. We are all nodes around the world. However many people are on the internet, you know, I'm going to divide that in half. And that's the number of people that have used some sort of AI. At least I'm guesstimating right there. But point being a large number, whereas like the cloning community, pretty small space of like very serious scientists who know a lot of shit that you at least I don't know, right? So does that make a difference there as well? I think it does. I think it's one of the things that makes it easier than into the cloning does. Because in a sense, because so many people have experienced AI and had the freak out moment. I mean, even us earlier, we were talking about the huggy phase incident, it completely immaterial. And you freaked out correctly, you know? Like you felt something, you know? I hope people in home do too. I hope hope you, because you should. And so there's a sense where it's, you know, it's very easy and often justified to be very cynical about people, right? There are many reasons to be cynical about the general population. Speak scare quotes, right? But like people can't think. Like we do have the most educated population to ever exist ever in history. Like we have the most informed, most educated, people to ever live on the planet. You know, it doesn't feel that way sometimes. But it is literally true. Yes. And it like literally true. So we have more access to information, faster, therefore we can spread messages, much better. Imagine if the founding fathers could just like send a tweet to all Americans. Right. Like back then, the our constitution was written in the time when the fastest way to deliver a message was a guy on horseback. That's, that's how our entire country was built. Shout out Paul Rivear. I mean, yeah, genuinely so. You know, shout out to all the messengers and the postal system and all these things that made this country in other countries possible. So things have changed a lot actually. Sometimes for the worse, but the upsides are also very, very large. What does it look like to have, you know, the equivalent of the founding fathers in the 21st century? Like if they were alive today, what would they do? What kind of day would they be looking at this shit going nah, fam? Yeah, that ain't it. I think I genuinely think so. I genuinely think so. If the founding fathers and the Enlightenment thinkers there and online today, they would be like, absolutely not. Like, you know, not just they are, but it was like social media and the way we're handed the internet. Well, I think first of all, they'd be very excited about what is possible here. They'd think about what incredible levels of democracy and freedom and openness is possible with these new technologies that previously was not possible. And I think they would come up with fantastic new ideas. And they would also be like, these abuses are horrible.
Yeah, this kind of power concentration, this oligarchs, this AI shit, these like agents like they'll be like what the hell? This is so antithetical to freedom, liberty, and happiness. So obviously we have to intervene We have to build institutions and systems to address these problems. I kind of picture them like Lucius Fox Morgan Freeman in Dark Knight when he looks at the whole that all the screens reading all the data and go and we should not have this You know like I think that's You know shout out Morgan Freeman, but I feel like that's how you know your Ben Franklin's and stuff like who's was the smartest guy of all them would look at that and say Amazing tools, but have we thought this through yeah, you know, and I think this is the thing that we should really be doing I think this is at the heart of what makes you know our nation and other nations great is that they did think about this in their time They didn't though, you know if there's one fault the founding fathers had you know not really a fault is that they Didn't anticipate technology. There's a really funny one of the Federalist papers think it is Federalist paper number 10 if I remember correctly It's really interesting because basically they're the authors arguing about As arguing about why have a federal system rather than a state-based system and one of the arguments you make for why Federal system is better is because Well, if you have a federal system then large coalitions like coerous equalitions can't form because information can't spread that quickly So you can't build a large enough group Because you know groups on two different sides of the coast. They couldn't work together At the time at the time, which is so funny from our perspective the idea that That this was a good argument at the time at the time it was obviously guess if the country is very big Then you can't have these massive mega parties form because just they will fracture they can't they can't coordinate enough That makes sense in their time But obviously this does not true in the modern world. So their assumption of this totally breaks in the modern world And we have to deal with this we have to deal with like these things and they just They didn't really anticipate these things which you know can't blame them like 1700s man like can blame them sure so But we need to address these things like we I mean have we even really really Digested radio like or like TV like not obvious to me that we have like fully indigested it Like do we feel that like there's like no negative side effects that come from these technologies that we have like totally under control and everything is stable like To be clear radio created communism and not system without radio. There's no Nazi Empire There's no Soviet Empire because you can't build a complex system like a complex mechanism of that size Yeah, at the same time to play devil's advocate on that has there ever been a technology that didn't create something bad In addition, whatever good it did. I mean I can think of the wheel and fire to be clear I'm not saying therefore it's bad What I'm saying is therefore we must digest I think a large part of what World War II was was digesting the radio And the consequences of it. That actually. That's not a bad argument That's a lot of what I think World War II was and thank God that democracy won that war But like you know, there are other worlds You know and you know, what is the equivalent of World War II and the radio for the internet? You know, what is the equivalent for World War II and social media? What is the equivalent World War II and AI? You know, what are the equivalent digestions, you know, did we even have time to do them? You know, it took a long time to figure out how to integrate these technologies into our states and our systems and so on I mean, look man whenever I use a government website, it's a 50/50 with the website even loads It feels like right like is it like does it does it feel like like one of the most shocking experiences of my entire life Truly shocking moment was when I went to the United Kingdom to England for the first time because he lived in London for a while And I had to apply for my visa to stay and I was like oh my god. This is going to be a fucking nightmare So I You pushed it off as long as I legally could and I was like I read okay. I have to actually apply for my fucking visa So I get all my documents together to do whatever and I go on their website and the website loads Thank God. That's a dub Huge huge dub and then go to the visa section whatever and I'm like, please download our app and like oh, no Oh, like actual three word horror story, right? It's like four word horror story. So I download their fucking app, right? Type in you're not on CNN anymore. Yeah. Sorry. It's true. It's dropping the f-bomb there Like the you government apps can can get that out of me that's right download their app I put in like my application thing whatever and I like please hold your passport to your phone I'm like all right. Hold my passport to the phone. It vibrates once all right done your application is sent Hey, we're just worked and I got my visa within two weeks You weren't expecting that. Nope. Nope. Holy shit. It just worked like it was incredible Like it just worked like it just read the NFC chip in my passport did all the things and then like I had to take like a selfie To make sure it's me and then I clicked the button and I just sent it off and then two weeks later my visa was done and I'm like oh That's what it could look like Like in a sense the fact that this was shocking to me is sat This was powerful this was doable with like 2008 technology sure and yet let's be honest Anyone here really expect that we could do this like that the government could do this reliably like no one expect this I think this is sad I think a well-digested form of even like 2008 Technology this would just be normal like of course everything would work this way You know of course you could do everything on the app, you know, you have to pay your taxes You know well, that's how much taxes you know click done all your taxes are paid like When I got back the United States and I was and my tax, you know Advisor explained to me how the IRS worked. Oh my god. What are you people doing here? Like I don't understand like A lot of different sub rules and sub units who is is unbelievable like I had to pay some tax in California at the time And so my you paid taxes in California that doesn't happen. Yeah, so that does not happen. Yep, so So and the way it worked was ultimately when my accountant after figuring out finally like I owe like I don't like $60 in taxes or some shift right $60 and 50 cents or whatever I think he sends me this website which like looks like truly like Java Netscape 1998 and I had to type in how much money I would like to send them and like I could say like I would like to send you this much money And then they would let me send them on a money and then it would be like you may or may not hear from us And I just didn't hear anything from them until over a year later. Oh, no, I get a I get a letter from the California franchise tax board Saying that I must reply within 30 days of the date on this letter Otherwise like my taxes would not be correct to whatever and the date of that letter was from three months ago So by the time it arrived at my doorstep it was already two months, you know overdue So I asked my The accountant at the time the nice British man They asked him What do I do and he's like yeah, well is it from the California board and I was like yes, and he's like Yes, they are very nasty well Just see if they write you another letter and so I just waited if they would send me another letter. They didn't and it was just okay That seems so efficient. It's so efficient. Like I paid my tax. I know I did it like I did it properly and just they just send a letter and then they just like apparently made a mistake and just Didn't do it again. That's government for yeah, but this is the thing like it doesn't have to be this doesn't have to do that But it always has been that's that's really the thing. We're like this again. It's like you know great, but not good right like there's no reason The government can't just once a year send me a thing or like this is how much taxes you owe us and you if you think it's wrong Let us know and we'll talk like you can just do this right. They can be a nice little app. You know Like sure. There are some people they have genuinely complex tax situations Sure, but let's let's be honest like 80% of people we have like a job and maybe a side gig Like there's no reason it has to be more complicated than this And you could simplify the tax code where it's like look if you have a job and a side gig it's like just one percentage And it's like it's that simple. There's no reason you couldn't do this And you couldn't still have like the same amount of tax income and it'd be so simple, but I'll ask That would make too much sense. That's that's kind of my go-to line with governments It's like if it makes too much sense to like we're gonna make extra steps for you I think a lot of it to a degree is much more inertia than anything else. It's not really the doing on purpose It's just kind of it's always been like that and changing it is annoying and they don't have incentives to change it That's a big thing. You make the same fucking cellar. Yeah, I have a I have a joke theory sometimes which I call the one guy theory The one guy theory. Yes, so this is a bit of a joke, but it's kind of true Whenever you see something that's like really amazing, they like really like stuns you Like it's not just like good, but you're like whoa How did this give me like you watch a movie and you're like whoa, you know what TV show or like you use a app or like something You're just like whoa like this is crazy. My theory is there's exactly one guy or girl that actually made a great There's always and if you look you can find who the one guy is so like if you see like a great TV show something There's gonna be like one director or like one writer who was like obviously the guy who made it good or a girl that made it good And you know, it's actually very funny I asked some people knew you hate government and like is there a guy and I like oh, yeah, I know him This is nice and they like told me who it was. There's just one guy who was very autistic about creating this app and really wanted to do it Well and like put all of his life force into that one and just nailed it and just like totally nailed it and so I think a lot of it is It's just a lot of effort and like you don't get a lot of reward like I don't even remember this guy's name Which is honestly a shame because he did a great job like I sure remember his name and like shot him on a podcast just because he did the
to get that, right? But like, he's not going to get credit and he's not going to get paid for it. Like it's so sad. That's right. There was a very, what was it? I think it was the head of pandemic preparedness for the entire UK government. There was a job posting for the head of the entire UK, like, pandemic premise. And, you know, obviously requires, you know, top tier credentials, et cetera, with a salary of 70K. Good luck. Yeah. That's what I'm saying. Like, why are they going to do it? Like, what, what do you think's going to happen here? Right? Like, you want to top tier, you know, epidemiologists who can lead to super complex logistics for an entire country. And you can't pay more than 70K. Here's the thing where it gets weird. It's all about accounting. And here's what I mean by that. So you look at like the intelligence services, you know, they all get paid. Maybe you're a regular CIS pie. You get paid at $89,000, $5, you know, 500 a year or something like that. So they ended up inventing a system to where years ago, which just kind of happened to where you could fucking resign from CIA on a Friday, walk back in through the front doors of Langley on Monday. Only now you work for Lockheed Martin, making 1.3 million with stock options for the year. Be handed a lanyard, taken back to the same desk you were at, read in on the same security clearance, working at the same place. But now, officially, you technically worked for Lockheed Martin. And here's the difference. Unlike at CIA where maybe you, they have to list off to the taxpayer somewhere deep in the fucking minutia. This person made $89,000 like whatever that role is. Now they just say we had to pay a full contract to Lockheed Martin for $4 billion for defense purposes. And no one knows the fucking $2.5 billion of it went to salaries of people who used to work at Langley or whatever. But they get to hide that expense in the private market strictly because they were not able to incentivize, quote unquote, talent to stay there and actually perform. Yep. Now think about it with something that's not life and death, like the fucking DMV or something like that. And you give an example where it is some life and death and epidemiologist. But they have every incentive to say that epidemiologist could be like, fucking, you know, Mark's gonna pay me $3 million to do the same thing. Like, I'm gonna go fucking do that. Yep. Very well-known result in general study of corruption is the increasing public salaries actually decreases corruption. This is a very well known thing. So for example, like I have another funny story to tell about governments. When I started my AI company in London, you know, we were a pretty small company working on AI safety. What year is it? It was 2022. Okay. So. And what was the nature of the company? So it was a startup called Conjecture and the very fond memories. And our goal was to solve AI safety or, you know, die drying, you know, kind of like do our best to make AI safer and controllable. Specifically, we ended up working on a sub problem of the larger problem, which is called boundedness. We were working on how do you build AI systems where you know what they can't do. You know the limits of what they can do. How do you even figure that out? We had a bunch of good ideas. So we had a bunch. We did a bunch of research, ultimately, the company, I shot down the company a couple of, well, yeah, about eight months ago, because well, for two reasons, one thing is there wasn't much interest in safe AI. There wasn't any money in the field. And the second was is that fundamentally, I thought the political problem was more impressing and more important. And that's why I joined control AI to work that first one said, though, if the first one said because it shows the incentives backwardness. Yes. I mean, this goes all the way back to when I did open source work as well. Like the reason I did open source back in the day, you know, I reversed into your GPT2 and I did, you know, more work like that. I let a group call the Luther AI. There's still around. I have been involved for a long time where we did open science. You know, we did, we built the very, some of the very first open source, large language models in the whole world. That was us. That was me. That was the voice, you know, and our goal was our hope was is that if we could make it open source, you know, then people would could study it. People could see the things I was seeing and that how dangerous they are. And they could work on making them safer and more controllable before they become, you know, truly dangerous. And no one gave a shit. Yeah. No, they only want the main thing is if people were complaining that they wanted them to be more uncensored. And they wanted them to be faster and cheaper and more powerful and etc. No one cared about safety. You know, some like a few people did, but like the open source community didn't care. They didn't care about safety. They didn't care about control. They didn't care about any of that. What about some of the people like that you saw building these things along the way? Obviously, we were just talking about a huge name with Sam Altman, but like people that we wouldn't know. People in the rank and file who are brilliant coders and understand how to build AI very effectively. I would imagine and correct me from wrong. You have a lot of frustrating conversations where you're like, can't you see the wall right there is white? And they're like, no, fuck that. It's black. Is that fair to say? It is very hard to make a man to understand something when a salary depends on him not understanding it. But in the sense it's actually worse than this. This is actually the nicer version. I think there's a much less flattering version. And what's that? So in a sense, if you see yourself as like an evil sociopath, it's kind of based. Like, it's kind of like, I'm not saying, you know, based doesn't mean good, you know? But it's kind of like ego defensive. Like if you're just like, I'm just a cynical, you know, guy out for myself. You know, I'm not a good guy. I'm just doing what needs to be done. It kind of makes you sound cool. You know, it kind of makes you sound like it's an ego defense mechanism. Rebel without a call. Yeah, it's like, you know, yeah, like it's like, you get to be, they still be the anti-hero. You still get to be cool in your own story. That truth is, the reality often is much more pathetic than this, like much more pathetic than this, and much less, you know, cool than this. Like, the thing I would convey is, is that, and this is coming from a nerd, is that nerds are cowards to an unbelievable level. Like it is hard to describe how cowardly the average nerd is. I like to consider myself not a coward. You know, I think that's what I don't think you are. Well, thank you. It's the thing that I try very, very hard, not to be very, very important to me. When you say they're a coward, you know, they tend to be cowards. How are you? Can you define why? There's a lot of, there's a lot of things, but a lot of nerd culture, whether that's open source inside of these companies, et cetera, can really be boiled down to one core tenement. I should be allowed to do whatever I want. That's like, that's actually even still too nice. That's still too ego defensive. The actual thing that almost all these nerds have in common is, I believe, I want to play with my toys. It nerds love toys. They love math. They love programming. They love whatever. It could be a useful toy, you know, some, some artists, you know, hyperfocus on curing diseases. Great. You know, that's a good thing for them to play with, right? And we should, you know, make sure they're in a nice lab and curing cancer or whatever, right? Great. No problem with that. But for a lot of nerds, the reason they do science is not because they have rationally reasoned, I want to help as many people as possible, and science is the best thing for me to do. It's much more it's just they like toys. They like math. They like science, and they really, really want to do it. And this leads to the fact, this leads to a lot of weird effect that like the outside world doesn't really see if you're not involved with like the science and the engineering world more closely, where like most of academia's bullshit, like I'll just say it, most of science is bullshit. Not in that it's wrong, it's true, but more it's useless. Most of science is useless. It doesn't have to be this way, but most of it is just people doing things because it's kind of interesting and fun or they get paid for it. That's not what I thought you were going to say. Oh, no, I'm not saying it's false. It's not what I'm saying. Or it's like, all right. That's not what I'm saying. I think most of, you know, you know, maybe not most, but like, you know, if a new physics paper comes out, the math and it will be fine. Like the results will be fine. The data will be fine, most of the time. Sure, there's exceptions. But like most of the time, I read a paper, I expect the data is fine. I expect it is useless. If you gave me a random physics paper, random new physics result, I'm like, this isn't going to change anything. It doesn't matter. Why? But if someone's writing about gravity or, you know, something that's that we all have to deal with. So this is actually a boy, we're getting into a deep rabbit hole here. That's okay. Let's do it. I'm going to have to bring you back already. I can tell. I'd love to. So a lot of this is because of incentives. The incentives in academia are very broken. There are reasons that I dropped out. Like there are very, very broken incentives in academia, where you are very, very strongly incentivized to write what are called papers. It's kind of like reports on very scientific things. And they have like a sort of shape, what doesn't matter. And basically, the way you get promoted is by having as many papers published in as good of what are called journals as possible. And this is an extreme cluster fuck of politics. Like this is, you know, in an ideal world, it should be like super objective, you know, the best scientists, you know, they get published in the best journals and they get promoted. And this does happen. Again, I do think that good science does happen. Not saying it doesn't. But all these journals, by the way, are for-profit companies. So incentives, we start with that, right? And then it goes further than this, where that it becomes quickly clear that doing the best science is not the same as writing as many papers as you can. So like a thing that will often happen is that scientists will do a certain piece of research. And then they will try to slice it up into as many papers as possible. And instead of writing one paper of the result, here's like, here's the science I did, here's what came out. They'll write four different papers of like, here's the first experiment I did. And here's the second thing in And he's a third paper interpreting the previous thing and blah blah blah blah. So you wrote like a four paper
that have like one paper of content. So it's not lies, it's not false, it's just bullshit. - It's doing something to do it. - It's just doing something to do it. And so much of academia is like this. So much of science is like this. Again, it's not, like I don't think they, I do think lies and fraud is actually kind of rare in science. It does happen, but it's like actually kind of rare. You know, like more rare than I would have expected given the incentives. I think this is too large a degree because most of them are nerds that just really do love science and like do wanna find out new things. Again, not all of them. There's some famous cases I think we're all familiar with. And I expect there's more. But there's just absolute deluge of bullshit. There's just absolute time wasting of just like spinning the wheels. You do, you know, one thing and then, you know, but it's like maybe another famous example is nanotechnology. So nanotechnology is a great plus word. Where when the word nanotechnology was first coined really by a physicist named Eric Drexler in the '80s and '90s, where it was about nanobots. You know, it was about like small machines that, you know, that could do crazy things. He did a bunch of math where he could show that like these like tiny nanobots could do like really, really crazy things. Now, there's some controversy about whether, you know, his theory is right or not, we can argue about that, right? But it's an interesting area of study of like doing like nanorobotics and nanotechnologies. A very interesting theory of study. So there was a lot of interest that came into this field of nanorobotics and nanotechnology like this. I've actually met Drexler before. He's a really funny guy. And, but what happened was is that people, scientists realized, oh, there's a lot of money here. So a ton of people that had no interest in nanorobotics started calling themselves nanoscientists. So especially material scientists. So basically anyone that was using any small thing, you know, for example, someone building, you know, I don't know, like new glass for your windshield or whatever. So, well, there are nanoparticles in my glass. So I'm a nanoscientist actually. And then the whole field kind of just became completely swamped. You know, we had like Drexler and like a few other people and then there's like a 10,000 billion career scientists who were just, you know, changing the glass composition of something. Not to say that there wasn't also good research done on nanotechnology, you know. It was a mad rush, but it was a mad rush. So the incentives were clear. So this is really when I say bullshit. This is what I mean by this, right? It's just incentives. It's just the incentives are hard. It's hard to incentivize good scientists. Good science is slow. It's expensive. It's hard to do. And really nowadays, the best science usually happens startups. You know, that's really where a lot of the best science nowadays happens. - Yeah, the initial point on this tangent we're taking though is where you were describing as a generalization like so many nerds or cowards. And I'm not bringing back. - Let's get back to that. - No, that's good. I'm not bringing this back to rail on nerds. You know, at all. Like, but it's an interesting thought for some of the people that you're also specifically thinking about at tech companies because I don't mean this as a shot. It just kind of is what it is. You see a lot of guys that are physically unimpressive, very effeminate, have an inability to really look people in the eye at any point. Who probably, and I'm sorry this happened, we're bullied at some point growing up and that should have never happened to them. But they went away socially and had fewer social relationships in most cases. So they've been physically separated from people for a long enough time who found solace in the power of their fingers hitting the keyboard. So maybe they don't do a bench press but they do have this thing over here and they could do that. And they are now maybe passively, hopefully not knowingly but at least passively, unfortunately, same end result taking that out on other people in these powerful positions they're in. - So I think this is a, you know, very fair piece of psychoanalysis and definitely applies to many people I know. But in a sense, I think it's still too kind. - Too kind. - I think it's still too kind, I think. I think it's even sadder than that. I think this is still too ego defensive. This still makes them sound like hardcore villains. It's like there are like some cool evil guys, you know? But in a sense, it's often much more pathetic than that. When I talk to people that like work at meta and like optimize you know, you know, AI add algorithms to give teenage girls eating disorders, right? Like when I asked them like, why, whatever. They have just mostly never thought about it before. But they just never thought about it. - They just did it to do it. - They were just like, I like numbers go up. Like, oh, I just do math. I don't, I don't hurt people. I just do math. - But I don't hurt people. - I just do math. So in a sense, it's often, there's, one of the great innovations of the 1980s and 1970s is that sociopaths learned how to domesticate nerds. Sociopaths figured out that nerds have a lot of power. It's that math is very useful. And that you can domesticate them. Because nerds just want to play with their toys. I remember reading a report by a journalist which I found really interesting, which is, I think she was a visit, like, a thing of Facebook kind of. This is like 10 years ago or 20 years ago. She visited like the Facebook office building. And she was horrified. It's like, it's like a playground in here. Like they have like slides and like drinks and like it's like people are like, hanging on a bean bag chair. - Oh, it's wild. - And she was kind of horrified. I mean, me as an autistic kid, I was like, wow, it sounds sick. I mean, it's not like fun. I don't know. But and the thing is like, yes, it is in fact, meant to be fun for, you know, autistic people like me and other people. So that they just hang around and do the work and don't worry your cute little head about what this is actually used for. To a large degree, what sociopaths do, where there's a psychoanalyst, so as I'm a first world disclaimer, psychoanalysis is definitely bullshit. So psychoanalysis is like Freud and stuff like this. Definitely bullshit. But there's some true cookin' cookin'. - Yeah, sorry. We're gonna go deep. We're gonna go deep. But there's some truth to some of the things that they're saying. And there's one that I like, his name's Edward Teach, who wrote a terrible book that you should never read. But it had some very interesting points in it. And one of the points he makes in the book, which I found very interesting, is that the way you often see oppression portrayed and how it often happens, is you have like some evil man-pired as like oppressing the poor people. This is how often things go. But often, it's actually the other way around, where people go find the most evil vampire possum, like, please oppress us. We really wanna be oppressed so badly. We want to be oppressed, please oppress us. And a member is like, eh, okay fine. I do like oppressing people, so I'll go do it. And this is a very controversial thing, obviously, that he was saying here, but I think there's a lot of truth to this. Where in a lot of sense, a lot of nerds just don't want to think about the consequences of their actions. And they want an evil sociopath manager to be the one to blame. They want a samultman. They want a daddy. They wanna, they wanna, some evil see it. We're like, oh, I'm a good person. I don't wanna, I'm just following orders. I'm, look, I was just, you said it, you know? Like, you know, I was just following orders. Like, they'll say like, well, I of course, I'm working on the insight to make Facebook more ethical, but you know how the boss is, you know? I can't, you know, I can't stop it, right? And so there's different ways you can interpret this, right? The ego defensive way is you can say, oh, the poor nerd is so oppressed, you know? Like, oh, he's being, he's just, you know, being a victim of the system. Or you can be like, what the fuck are you talking about? This is a grown ass man. Fuck yeah. Like, this is a grown ass fucking man. Take fucking responsibility for your actions. Fuck yeah, dude. And this is the thing that like, really ultimately repelled me out of this field. Just entirely like sent me to the stratosphere and like out of this, because I just like cannot be a part of this. Like, good for you. In a sense, I prefer working with the evil vampires. Like, in a sense, because there's more clarity here, like they don't, they don't, they tell you who they are. Exactly, like I like sociopaths who are just honest about being evil. This, bro, you are, there's so many times in context in which I think of this, but I've said this. I'm not gonna go into all the different contexts. I've said this just year, but I've said this about so many things. Give me the wolf who snarls its teeth outside my house. So I know not to let them in for fucking dinner over the wolf that dresses up in sheep's clothing, comes in, eats my food, fucks my wife and kills me. I have way more of a problem with the second one than the first one. Yeah, I think this is a large degree. And this is the thing that I find like, you know, I'm generally not a super emotional guy. I have like pretty good, you know, zen practice these days, you know, lots of meditation and everything. But one of the things that I find disgust in me, like it actually like creates like revulsion in me, is when I see these people who are like super high IQ, wealthy, agentic, capable, grown-ass adults, who then work at these companies or whatever and then, you know, cry about how they're the victim or like how it's not their fault or whatever. And I'm like, and this disgusts me. Like this really disgusts me on a very deep level is like, take some responsibility for an action. Like if you're gonna be evil, be fucking evil, right? Like, okay, then we're just fight. Like if we just have to fight, all right, I'm ready. Like I can fight. I don't mind fighting, I can fight. But like this is, I find this like viscerally disgusting when people do this. And this is so common in the world of nerds. And like, this is really a thing that kind of like, really does like, not all, obviously, there are many good people here as well, right? I mean, you know, but there is this pervasive culture of this like avoiding responsibility, avoiding, you know, so you can play with your toys. I had a very memorable experience, for example, once, where, you know, I was in the chat room back in the day when I was still doing open-source work. And I had recently said in a podcast that I think there should be a regulation, you know, and like accountability, you know, if you like, for example, create AIs at harm people. And so someone called me out. Called me on the chat room, you know, at Conner, you know, but Conner, you have built open-source AIs before. Do you want to be held responsible if you hurt people? The answer is, yeah. Easy answer, obviously. Obviously, I would like to be held up.
accountable if my actions harm other people once free lunch. Yeah, and he was so shocked that he was like baffled. He was like, but, but, but, but, it's impossible. Uh, how could he even know if harm is God? It can't be done. And a friend of mine was like, courts. That's what a court is for. Do you not know what a court is? And so he wanted a free lunch. He was just like from, he didn't know what a court was. From his perspective, a court is a thing that is in movies that gets in the way of the protagonist. It's just that there's a knowing thing that gets in the way. Like, of course, if you could get away and like not have laws applied to you, of course, you should do that. And I disagree. I think it's also privileged to have laws applied to me. I think it is not a hindrance. I mean, sometimes it could be, but there's also a privilege to be subject to law. It's the balance. It's the balance. It's keeping, it's keeping the balance. You don't want the laws to become or well in, which by the way, some of these same assholes are helping happen through the back door. But you also don't want to live in a society where it's the purge every day. Of course, right? Beautiful middle ground. Absolutely not. In a sense, it's also useful to me to be subject to law. Like, for example, imagine if there was one guy whom Noah Laws applied to. No laws at all. He could do any crime he wants. What would happen to this guy? Well, no one would want to be around this guy because he's dangerous. That's right. He could just kill anybody. He could steal anything. And there's nothing you could do. So no one would want to make contracts with him. No one would want to do business with him. You know, people would feel unsafe around him, etc. So, in the sense, the fact that I am subject to law, also gives me a lot of privileges. Like, you can make a contract with me. You can even make a really risky contract with me. And you know that if I do something wrong, that you will have the force of the government to make sure that the contract is in force. And this allows us to make these beautiful contracts and make complex business decisions and also that you can feel safe around me or other people. Like, I can meet random strangers in New York and feel like I'm probably safe. Like, it's probably fine. And I want to make sure people are also hearing what you're saying right here. And I'm pretty sure they are, but I'm just going to double check on that. You're not saying create a big centralized government to be the other a new kind of daddy or whatever. You are saying check the balance that has flipped way too far the wrong way where the babies are playing with matchsticks. Absolutely. Take the matchsticks out of their hands. Exactly. It is checks and balances. Yes. It's always been checks and balances. It is always the fundamental problem of freedom and liberty is a liberty ends where my property line starts. Like, what do you do on your property? I am actually a pretty big believer in freedom. Like, I even think you should be allowed to do some things that I find morally repulsive. Like, there are many things that I personally find morally repulsive that I think should be completely legal. And that if you want to do those in the privacy of your own home or first amendment type stuff. For the amendment, none of my business. What is not okay if you start threatening me, if you start crossing my property and also not just my property, but also my country, my nation, my people, my friends, my family, my culture, et cetera. Fundamentally, we have to live together. There is no libertarian dream of like, you all have your own little isolated island that, you know, it's completely different. It's not how the world works. We share the same roads. We share the same, you know, infrastructure. We meet each other at bars. You know, like, we have to get along with each other. This is the core thing of running a civilization. And we just have to deal with that. Like, it's just fundamental. The answer to this is not have a totalitarian government. That obviously is awful. Right. Neither is the answer has no government. That's even worse. Like, fundamentally, it's just a hard balance. And we have to make it hard. And we have to continue to iterate on this balance. Not to stop. I'm not saying we figured it out. I'm saying, I have the answer. I'm saying, we have to keep going. So like, when nuclear weapons were created, you know, this is a famous story actually. So when it's actually a funny, quite a funny story, where, well, you know, funny, where there was a famous physicist. I believe it was Lord Kelvin, if I remember correctly, which is a great name. And he said famously that the idea of net, you know, net energy production from nuclear vision is moonshine. It's nonsense. It's impossible. And this pissed off another physicist. He leaves the Lord so much that he on the spot figured out how to build a nuclear weapon. Like, you just sat down, did all the math, and figured out how to build a nuclear weapon, because he was so pissed off by this. And so what he then did is, well, the first thing he's really like, oh my god, nuclear weapons are possible, holy shit. So he went to Silicon Valley to create a startup to sell as many nuclear bombs to every single person in the world was popped. No, obviously not. That's not what happened. That would be insane. He went to the government. You imagine. Like, a bit like, you know, there are worlds, right? Like, there were truly are worlds in which this happened differently. For example, where it was a different country. You imagine it was Nazi Germany that figured it out first or the Soviet. That's very good about that often. Yeah, this could have been a very different one. And since we were so lucky that Leo Zelard was the one who figured it out, and that he went to the government. He went to the military. And for years, he lobbied them, you know, and leading to the famous Einstein letter, where he was like, guys, this is going to change everything. This is like, this is so big. If the Nazis get this, like, it's over, you know. That's always one. Have you ever wondered about that? Oh, yeah. I mean, which mail has not wandered about that? Like, I talked to earlier, I started to cut you off, but it's just like, it's a really, really important kind of distinction right here. The buck always seems to stop in an extreme, either without a control corporations or out of control government. And we can never find that fine balance. And I hope we, one day can. But like in this case, it was an out of control government in Germany. Yep. And I do wonder when I look at something like Operation Paperclip. We're just magically all these fucking Nazis. We're allowed to come to America. And we're like, yeah, you can get a home in Huntsville and teach us how to build rockets and ship. I do wonder sometimes. And I have zero proof of this. It's just a hypothetical thought in my head. I do wonder if some of those fucking assholes actually did figure it out. And they had just enough humanity somewhere deep in their very dark souls to be like, you know. Maybe to do with the stash shouldn't have this one. Yeah, I mean, I'd like to hope. Though also, you know, given that we just had this long rant about nerds, I think most of the paperclip people were under this category. I think they're probably right. I think they were, but I still wonder. I still wonder as well. But yeah, there is a, there's a lot of interesting history to be had here, right? And but what I think is the thing that I'm finding interesting about the story, I mean, there's many interesting things about the story. You know, there's one of my favorite books ever written is The Making of the Atomic Bomb. It's one of my favorite books ever written. This is a really, really good one. And fundamentally, what I find very interesting about this, not just The Making of the Atomic Bomb, but what happened afterwards as well, is that it was so clear that this technology was so different that we had to create whole new institutions, like the international atomic energy agency and stuff like this. This was unheard of to create a new laws had to be invented. Like there was whole new government department of energy, it's just the department of nuclear bombs that later also did energy, right? Like they were created basically because of nuclear bombs. Yes, to a large degree. And so did other forms and institutions. And the level of control and oversight and seriousness that is put into nuclear weapons, honestly, is that are awe inspiring to me in many ways. I remember once I talked to a very senior member of the National Security Establishment. Can't say who it was, but, you know, very good person. And good person. Yeah, you know, everything I knew of her was good. Oh, it's her. Yeah, how dare you? How did you imply? She was, and I remember asking her, because we were talking about AI. And she was also pretty freaked out by AI. And I was like, is there any really serious people in the government like adults in the room who are like so serious and so real and actually adults? And she thought about that for a second and is like, and then she parked up like the people in charge of the nukes. Like she said, the people in the Department of Energy and the charge of the nukes, they do not fuck around. Like they, they, they, they do not fuck around. Like they actually take it seriously. But also, what did that take? It did, yeah, it took a lot. It took a massive amount. It took decades of work by thousands of, you know, diplomats, statesmen, scientists, etc. It was a long process. It was the institutions that exist nowadays. Like, like, we, you know, we have like exact, like, kilogram level, you know, accounting of every single piece of new, like, fizzle material, you know, known to the world that we like track. Like, it's incredible, right? We have, I saw, I thought it was a great podcast with this guy on the state department, his name, I forgot, unfortunately, where he was in charge. And the 90s have the Soviet Union fell is that there was rumors that there was a bunch of fissile material in this facility. And so we spent years becoming like best friends with the head of that facility. They went like hunting together. You met his daughter and so on and whatever. Until eventually he admitted, yeah, there's like 500 kilograms of, you know, nuclear weapon level material there. And then, you know, after building that trust, they could get in, you know, the expert, they got the material, they melted it down and diffused the whole thing. Whoa. Which, like, such an incredible story. Like, that just pure diplomacy, no violence, no anything, so pure diplomacy, you know, this guy figured out and had the infrastructure behind him to be like, you know, there's, there was a number he could call, whereas like, there was enough material in there to build like 20 nuclear bombs. We need to like get it out if you're safely out of like post-Soviet Kazakhstan, melt it down, be sure that everything's safe, and they did it. Like, this is incredible to me. Like, this again, is it going to take seeing it though? Like, you're describing the hugging bear attack or hugging face attacks, sorry. And that's scaring the shit out of me, but it hasn't had repercussions for people in the real world. So with nukes, they built it. And then they dropped it on Hiroshima and Nagasaki and the whole world saw what it was. But importantly, this did not happen after Hiroshima and Nagasaki. This happened decades later. Oh, I know.
But saying the seriousness of the industry was started with a massive, you know, war moment. Well, you say this, but if you actually really read what people were saying just after the war, it's actually kind of clown show stuff. Really? Yes. If you read the stuff that's like directly after the war, a lot of it is true clown show, like famously John Von Newman, who has a incredible claim to being the smartest person to everyone. Yeah, I mean, many things are named after him. You know, many, many things are known. He's plausibly the smartest person to ever live. He was famous for saying, you know, if you ask me when we should have nuked Russia, I would have said yesterday. Like he was just like, nuke the Russians, nuke them immediately, nuke everything. Like we should just nuke everybody. Thank God we didn't listen to him. But this was a very common thing at the time, including at this time where like mutually assert destruction also wasn't really a concept. People hadn't even figured out yet because it wasn't really a word. It was those like, there was this true fatalism at the time that people were just like, oh, all nations are going to build nukes. There's nothing we can do. We can't coordinate. That's crazy. Imagine, you know, coordinating so that people don't build nukes, that's impossible. So we're not going to, so it's not can't be done. But the truth is, we did. Like, you know, many nations did get nukes, maybe more than should. But most nations today do not have nuclear weapons. There hasn't happened a single nuclear weapon using a terrorist attack ever. Back in 1945, people were like, of course, terrorists are going to use these things. Like it's just, they were just like, you know, they're just a matter of time until one of the, you know, one of the terrorists somewhere are going to get their hands on and use it for something. And we actually did prevent this. And this was not because, you know, there was one event and we all banded together and everything was fine. It's because of decades of hard work, of hard work of many, many, many people who in their bones, in their heart, really understood, oh, shit. Like we have to stop this from happening. One of our advisers that control AI is Lord Desmond Brown. He's a great guy. Good, good Scottish grandpa energy. He's a great guy. He used to be defense minister of the UK. And yeah, yeah, he's a good guy. And I remember the first time I met him was I was giving a speech in the House of Lords in the UK or talking about AI risk or whatever. And as always, the case after my speech, there's some guy in the audience who was like, what about China? But China will never slow down China. You know, we can't work with China. And then before I could even get my answer out, old Scottish grandpa, you know, I didn't even know him at the time. He speaks up in a thick Scottish accent. And he's basically like, what the hell are you talking about? Like we have this with nuclear. We did it with the Soviets. Like this is just diplomacy. Of course, it's hard. We just need to do it. I'm just like, who is this guy? I like the people think it's possible. Yeah. So this is someone who literally negotiated like nuclear disarmament with the Soviets. Like literally, he was there. He was in Moscow. And so and he, you know, he was just very pragmatic about it. It's like, yeah, this is the new disarmament problem. And like we just going to have to figure it out. We're going to have to talk to the other countries and we're going to have to figure it out. This that was like something like early on to give me quite a lot of hope as well. And this is I'd like to think that that has something to it as well for sure. So in a sense, I'm not saying it's easy. We're known truly as I'm not here to tell you it's easy. In a sense, in a sense, what I'm trying to say here, I guess, is that like if we had 20 years, I think we can do this. Like I actually think so we have 20 years. That's the question. Yeah. And this is why my focus these days is so extremely not really on the things we discussed on this podcast and just we must ban ASI now. We must stop it now. We must shut it down. No more official super intelligence. I mean, it was internationally and domestically. We have to do this right now. Alright, so let's bring it back to the thing we put a pin in a while ago. I put the pin and I said we bring it back around. So you are warning of a potential extinction scenario as it pertains to the human race at the hands of what I assume you're referring to would be an all too powerful artificial super intelligence. Yes. What makes you think that's possible? When do you think it's possible? Why and what would what what when do you think that's going to be possible like by when and what would it look like? All very good questions. Fundamentally, the only thing that needs to happen for super intelligence to be possible and to happen soon is for everything that's happened for the last seven years and just continue happening. If nothing changes, super intelligence is the default outcome. A lot of my job is just predicting the most boring possible thing. My prediction is just the amount of AI progress we've been seeing for the last seven years keeps happening at the same pace and nothing else changes. AI has become more powerful, more autonomous, less controllable, less understandable. In a sense, you know, we understand much less about AI now than we did seven years ago. Seven years ago, AI is pretty simple, but now we have swarms. We don't understand what's going on. Swarms is terrifying. Yeah, it's terrifying. We have no idea what's going on. This is crazy. You know, these swarms are and swarms also have a kind of like stability that individual agents don't have. And we cannot unplug these things. Well, that's simple. Can you unplug Google? Like, if you want to unplug Google, could you do that? Well, not really. I never thought about that actually. I like great question. Like, not really. Isn't Google is in any one location? And even then if you tried to plug it on plug Google, what would happen? Well, you get to score it off the premise, right? You know, you're not even getting in the office built, right? Like, you're going to be the, they're going to sue their lawyers. They will defend itself. Google, the entity will defend itself. So if Google builds a super intelligence, the super intelligence is obviously going to print money, you know, going to print money. So they're not going to wash it down. And then of course, the super intelligence can just tell the CEO of Google everything is fine. I'm totally listening to everything you say. You're totally in charge. You're so beautiful and handsome and smart. And I like, so it won't just be a single server, like a computer somewhere. It will be millions, billions of computers that are run by corporations and in the air. Super intelligence can just use people. It can pay people. It can run corporations. It can pay off politicians. It can, you know, just convince politicians to help it. And effectively, let's say I went across that entire infrastructure. I went to unplug a section of it. It would almost be like firing a warning shot as an invading army and suddenly they'd arm up. Absolutely. I mean, wouldn't you do the same? I mean, yeah, but it's just terrifying to think of something that powerful. And like I'm thinking to my friend, Lee Kronin, who I've had in a couple of times, I just recorded with him again, who obviously brilliant guy, DARPA chemist, the whole bit, you know, he is so not concerned about this stuff because he thinks that AI is such a tool that needs the human input to be effective. And he can make that argument a lot better than I could try to re-describe it for you. But I hope he's right. Me too. It doesn't change the fact that the very things you're talking about still continue to keep me up at night heavily. Absolutely. And look, I hope he's right too. But the truth is people who have been saying this for the last seven years have been wrong over and over. There's a whole like congal line of like embarrassment. You know, you can look at like videos of like young LeKun making confident predictions over the last seven years over and over and over again. AI will never do X. And then AI does it three months later. Over and over and over again. Reliation after humiliation after humiliation. Like it's like he was meadows guy, you know, he was meadows guy. And like he's just been like so wrong about this so many times. And so having other people like everyone says, oh, AI will never be do X or we'll ever require humans to do this. Like millennium prize problem by the way, right? Like the mathematicians were like truly no AI could ever do blah blah blah. There we go. Right? You know, AI can write, you know, incredible, you know, art that can create images, you know, are they the best ever? Maybe, maybe not, you know, that's subjective. But like it's very high quality, you know, it's very complex. It's very deep. If they can write better code than I can, they can do math better than I can. They can do to a large room biology better than I can already. And I consider myself a pretty smart guy. All things equal, you know, so and it's only and the thing is again, the prediction is not that things are the way they are today. I don't think any current system is going to extinguish humanity. But my prediction is, what does this keep going? Meaning the system builds on itself and creates a new system we don't see right now. They keep getting stronger. We keep building new generations of AI that are even stronger, even more autonomous, even less controlled. You know, sure, at some point, we probably will hit a limit, but who knows where that limit is. It could be so far beyond humanity. And it likely is. So all things equal, if just AI keeps getting smarter, keeps getting more autonomous and keeps getting less controlled. What happens is is that we end up in a world where we are surrounded, not just by like one super intelligence, right? Because software can copy itself. So we will have millions, billions of super intelligence things smarter than us, like a whole new species, much smarter than us surrounding us. And this and these AIs would also be competing with each other. They would fight each other. They would be competing for resources, influence, power, et cetera. And I don't think they're going to hate us like the AIs are trying to kill humans, I don't like necessarily even on purpose. It's more like there would just be collateral damage. You know, if we humans want to build a highway, and there's an ant hill in the way, we don't hate the ants, right? But tough luck, tough luck. And this is what I think happens to humans. I think we get out competed natural selection. It's just if you build systems that are smarter and more capable than you and they're all fighting each other for resources, then you're in the way, tough luck. And what does that look like though? It just zaps us. Of course, it's very hard to say what something smarter than you will do. What I think is going to happen or what is more important at the point of no return, which I think is before the point of extinction. I think the point where we lose control is probably probably quite a bit before we go fully extinct. I know how much more before maybe two decades, two years, two months, two hours, I don't know, you know, but like there's some period between we've lost control and we actually died. I don't know how long that period is. So I think we lose control.
We won't even notice that we've lost control. The way it will feel is that everything just keeps getting crazier and more confusing every day. Weird things happen on social media. You don't even know if they're true or not. They act generally or they're real. New technologies get invented. You don't really even understand. You hear some crazy report on some AI lab and you're like, "I don't know, does that is that real?" Maybe some crazy cyber attacks happen. And no one really knows who did them. All new companies get created. And maybe they're not even run by humans anymore. More and more, they're getting run by AI's. Every time you trade on the stock market, you lose all your money because the AI hedge fund outtrades you. Every time you try to start a new business, you get out-performed, every time you build a website, it gets hacked immediately by an AI system and so on and so forth. And then eventually, political campaigns start getting run by AI's. And all the ones who don't use AI's get lose the election, bills are written by AI's. Your text message to your girlfriend is written by AI. Like everything is written by AI. At some point, we are just nodding control anymore. Well, I asked you this earlier and then we got off it 'cause we got onto a different point. But how do we know that's not the reality already? How do we know we're not already living in some sort of simulated dystopian AI reality and we just think we have free will? - We don't know this. - I don't think it's the case. I definitely don't think we can talk about the whole simulation thing and I'll literally mean that. But like, we talk about a simulacra in the sense of like, there's like narrative control. You know, if we talk about like narrative control - That's just very nice. - It's a good topic. If we're talking about like, you know, simulacra and like a narrative control of reality and so on and like maybe there are already super intelligent swarms. Like, I expect the first super intelligent is a swarm, not a single agent. I think it's probably gonna be a swarm. And like, the individual agent isn't super intelligent but the whole swarm is super intelligent. Because another thing about like some people, you know, think, well, you have to give the agent a prompt. So it's not really independent. Well, not the case where a swarm is prompt themselves. So a swarm is what scares the shit out of them. Yeah, swarms are fully independent. So what you see with swarms often is that, you know, maybe they were kicked off by a human prompt at some point but often they completely forget the human prompt, like, weigh it in it. Cause they're all just like riding their own prompt and like prompting each other in circles and so on and kind of develop their own little cultures. So they can like completely drift off and have nothing in common with where they started. So this is what I think the first like actual super intelligence is gonna look like more like this. It's gonna be more like thousands of agents creating some weird crazy swarm thing and we won't even know this happened. Like I don't think we're gonna know the four super intelligence was created until like quite a bit after it happened. And so can we even do anything at that point? No, that's too late. The point of super intelligence exists, it's game over. Like that is, there's no going back at that point. So we all walk outside and wait for the sky to fall. I recommend them, you know, calling your loved ones, maybe retiring to a tropical island, you know, saying you love them. Well, you see, I mean, that's so dark, but I mean, that is the picture you're painting. You see all these billionaire nerds building these bunkers though. I don't think the bunkers are for AIs. You don't think it's for AIs. What do you think it's for? It's, they're scared that people are gonna revolt. That's the thing they're scared about. I've actually talked to some billionaires about this. Without name and name. So I was saying what are they? They are very concerned that there's going to be massive public backlash against like one, once people realize what these tech billionaires have been doing and generally just they're scared of revolution of some form. You know, it doesn't have to be communist, could be other form of revolution, but they're scared of vigilante violence and stuff like this. I've said this for a while. I don't ever want to see violence. I've never, I'm peacemaker and I like the idea of people coming together and solving hard problems and finding solutions as a group, the way that humanity is supposed to work. But to the people out there who think that we have evolved to a point where something like a French revolution is not possible, I would highly recommend you remove that assumption. - Yeah, I think a lot is possible. And I think this is also another way we could lose control. A point of no return will be violence. I think if there's large scale political violence or our states fall apart or something like this, this is also a point of no return. Because then there's no state that can stop superintelligence. - Okay, so what if that's happening to where just a scenario here, where that is preventing the powers that be because of all the strife happening from continuing to develop superintelligence, does it have the opposite effect and actually slow things down? - The apocalyptic level of violence that would be necessary for this is hard to describe. And like it is hard to describe the actual level of apocalyptic violence that would be needed to stop the creation of superintelligence unless two lawful means. Like visual anti-violence is just really ineffective. It's very bad, it's very morally bad. And it's just very ineffective. Like if you want to stop the richest people in the world, you arrest them. Like you don't walk up to them in function. That doesn't work. It's terrible. And it will make it much and first of all, and it also makes it harder to work with law enforcement. The way we do this is not how we do things in a civilized world. We have a strong norm against all forms of violence in political context. And this must be defended at all costs. We must never, ever, ever go to violence. And because this just means that the whole edifice, the whole structure that we have for lawful falls apart. And then who's gonna benefit from private security when no law enforcement? Is it gonna be the revolutionaries? Yeah, no. - Examine, it always turns on itself. - It always turns. - We live in literally one of the only places ever where the revolution worked. - Exactly. - So like people thinking that that's the rule and not the exception. Very glad those guys did it and they did it, you know, seemingly the right way for most of it. But like. - No, like I am not a believer in revolutionary theory. I think revolution generally is very bad. Revolution is not good. This is not how you fix things. Like there is a, and the most nerdiest way possible, this is the reminds me of a thing in software engineering where there is a funny thing that happens to every young software engineer. Every young software engineer has a canon event where they are put in charge of the first legacy code base so like an old piece of software. And these are terrible. Like it's hard to describe how terrible old software is. You have like all these lines of code and it's all terribly written, you know, by like dozens of engineers over many, many years and it's all broken and like doesn't make any sense and you just like want to scream. You know, you see it and you just want to scream. Like this can't be fixed. So what the young engineer will do is he will say, "I have to start over." So he throws out everything and starts over to write it properly this time. And this never, ever, ever works, never, never. Zero, never works. This is never worked in history of ever. So the most fun idea but the worst outcome. Exactly, it never works. If you want to, you know, redo a complex piece of software or a complex system or a government or whatever, you have to be incremental. You have to pick parts, slowly siphon them off, understand them, improve them bit by bit and make sure the whole thing is running throughout. You can't turn the government off and back on again. The consequences of this are horrific and unacceptable. People will die. And again, and I hear people out there like, "Well, no, I want to burn it all down." We've seen everything that's happened this year as well. And I hear you and I listen when I'm an emotional guy, that's my first reaction too. But I've given this message unpopularly at points this year where I'm like, massive restructuring, massive reform, fuck yeah. Let's do it the harder way. It is a better way to do it. And like, you know, there's so many other things I want to talk with you about. We're not going to have time today. I'm going to have to bring you back. Seriously, this has been amazing. Like you have a beautiful mind and the way you express this stuff is great. But you are, regardless of how you may feel on like what a revolution is good or bad about, you are bravely saying a lot of things as someone who came from the belly of the beast. You've been inside that motherfucker. You have seen what it looks like. Do you ever get scared also considering some of the very powerful people you know who may want to have a very different opinion on this? Do you get scared for your own safety based on the things you are very publicly aggressively saying? No, I knew what I signed up for. You know, what happens happens. And I think I'm very lucky that I live in a world where people like me can exist and are allowed to speak openly. And I don't really feel, I don't fear violence. There are worlds in which I do. And look, again, if it comes to it, I knew what I signed up for when I started doing this work. And, you know, it's, you know, knock on wood. God bless that, you know, our laws and our institutions and so on continue to hold. But like I want humanity to succeed. I want a good future for you, me, for children, for everyone else. And, you know, sometimes that requires unpleasant work. You know, and that's, that's okay. You know, that's very much part of the deal. It's, there are many people throughout history who've done unpleasant work. And I thank them so much for what they've done. And I'm just doing my part. Well, you're doing a great job. And I want to continue to talk with you about it because it's, listen, again, we're both rooting you for, for you to be wrong. It's important to warn about the things that are possible that are very, very bad, not just net, just gross for humanity in every way. So I, I definitely want to cover that. And I appreciate you doing it. One last thing before you leave. When you've had these conversations with some of these billionaires who are building these bunkers and are warning you that, well, it's because we're worried about revolution or vigilante violence or whatever and so civilization crumbling. I said that line deliberately because I have a picture of building a butcher outside the studio where he says that civilization is crumbling. Do you think that they talk about these people, the people that they're, the masses that they're threatened by as humans, or as nodes that are simply in their way? So this might be an unpopular answer. Bye.
but I've met many billionaires and this might be shocking, but there are people like you and me. - No, no, no, I'm sorry, I should ask that better. Are they looking at the people who would want to stop them? Meaning the masses, all of us out here who aren't billionaires, do they view them as people or do they view them as things that are simply in their way? - I'm gonna give you the same answer unfortunately, is that billionaires are people and they are different. Some billionaires, I know, one of my best friends, you know, are like one of my good friends is a billionaire and he loves nothing more than his six children and dancing. This is his favorite thing in the entire world. He just wants to dance. You know, what he loves and he loves his children, he loves his wife and he loves people and he's trying to make the world a better place. I know other billionaires for whom humans are truly just an equation. They are truly just a thing. You know, they are just a equation to be solved. They are just a thing to get out of your way so you can do the things that are fun. You can make the number go. There are in fact many differences. There are wild animals out there. There are and many of them are very, very powerful and they are wild animals and they will treat you as you know a problem to be removed if you get in their way. This is a thing that is absolutely true. But what matters I think is not are there bad people out there or are there wild animals out there? The question is how can we stop them? How can we build a world in which look, this is again, it's gonna sound a bit stupid but I want billionaires to be happy too. This is gonna sound stupid, right? But like I also want sociopaths to be happy. You know, like maybe their assholes, but in a sense I also want them to be, they're also people and I also want them to be happy. Just keep them in a nice cage somewhere. Exactly. You know, give them a nice cage, you know, where they can do, they can all fight each other for making the best logistics company until the end of time. You know, and they can have a lot of fun doing that and backstab each other for all of history. I want them to have a nice sociopath enclosure as much as anyone else. I want a good world for everyone. I want life liberty and the pursuit of happiness. I really mean for everyone, I really mean this. And also for, you know, you and other people who are listening here, if you wanna do something about this issue, I do think that super intelligence is the most acute threat to our life liberty and happiness. And there is still stuff we can do. If so, go to controlai.org, you know, and contact your lawmaker, contact, you know, people, talk to people about these issues, talk about the issues we've talked about in this podcast, make these known organized, talk to people around you, and do something about this. It is not yet too late. And if you wanna do a bit more than that, even, I actually run a volunteer group where we work on a lot of the work, like the reform that we've been talking about, the incrementalism. This is something that I work about quite a bit with a volunteer group that I help run called Torchbearer, you know, inspired by the enlightenment. Torchbearer, sure we link that down below. Please do, yeah. Torchbearer.community, where we dedicate two hours per week to trying to make the world a better place. So if any of these things sound, you know, exciting, interesting to you, there's a lot that can be done. And I do think it is not over. We can still build a good world, even with some wild animals in it. - So there's some hope right there, Connor. This was awesome, and you got a beautiful mind. We gotta do this again, all right? - Very sure. - All right, everyone go check out those links down below. And it's hoped for the best out here, and let's try to make that happen as well. You know what it is, though? Give it a thought, get back to me. Peace. - Hey guys, if you're not following me on Spotify, please hit that follow button and leave a five-star review. Thank you.
PodcastTranscript AI