A Stark Warning From a Former AI Doomer

An argument that I've been making for a long time now is that if we really want to understand what's happening in AI, we have to understand the futurist ideologies that are heavily influencing the top frontier labs, anthropic, and open AI. The problem is, is that these ideologies are confusing, and there's rumors swirling around them, and the excerpts we hear can sometimes be absurd, and we think that can't possibly be true. What I thought would be useful is to bring in an insider to explain to us what is really happening in these labs, and what do they truly believe. This is why I'm really happy to have on the show this week, Emil Torres, Emil is a researcher in academic who focuses on eschatological and utopian movements, but what makes Emil really useful this discussion is that they used to be involved in these ideologies. Emil comes from the world of futurist ideologies that believed that superintelligence in AI would either kill a solid, redeem humanity with a post-human, digital god. But then Emil's moral compass led them out of that world, and now Emil has become, in my opinion, one of the most astute chroniclers, expositors, and critics of these ideologies. Emil's going to join me here in a second, and they're going to walk us through what exactly are these different beliefs? How do they intersect? What is their role in the major frontier labs? And why should we be really worried about them? If you have any ambition to understand what's really happening in the world of AI, you need to listen to this conversation with Emil. It is going to quite frankly probably shock you, but this is a good shock. This message is one that we all need to hear, so let's get right into it. Emil, thank you for joining me on the show. Thanks for inviting me. Great to be here. I've been looking forward to this conversation for a long time. I think both your story and your research is being too overlooked in our current moment. I think it is very relevant to a lot of issues that people care about or are worried about, or don't understand about the current AI situation. So I'm excited to really get into both the meat of your work as well as the origin story, which I think is very important. But I thought we should start briefly with definitions just so that everyone is on the same page when we're talking about these various movements. So can you explain what the acronym Tess Griel means? Yes, I can. So this was coined by myself during a collaboration with the computer scientist Timmy Gebrew, which we initiated in I think late 2022 or so, and we were specifically interested in the underlying ideological drivers behind the current race to build ASI or artificial superintelligence. So Tess Griel is a clunky kind of unsightly term, but it does a lot of important explanatory work. One way to think about the acronym is it's basically just a name for the dominant orthodox worldview within Silicon Valley. That simultaneously highlights a certain intellectual history that is relevant to understanding that worldview. So it explains what the worldview is, where it came from, what shaped it, how it evolved, and so on. The acronym itself represents seven different ideologies, and the letters appear in roughly the same order as the ideologies emerge chronologically over the past 40 years. So the T stands for transhumanism, E for extrobenism, than secularitarianism, cosmism, rationalism, effective altruism, and long-terms, a bunch of like really big clunky polysyllabic terms. But these ideologies are really fringe within American society, and I think if you were to go and talk to the average person on the street, they would find them to be so bizarre that they'd consider the grand cosmic vision behind these ideologies as basically a non-starter. Nonetheless, they are massively influential within Silicon Valley, like basically everybody from Elon Musk to Mark Zuckerberg to Sam Altman to Dario Amadeh are transhumanists. And many people in Silicon Valley don't even call themselves transhumanists, but that's for the same reason that I've never publicly stated I'm around Earther, right? It's just like what else would I be, obviously I'm that. So transhumanism really is the water that people in Silicon Valley swim into the air that they breathe, and it's also the backbone of the whole test-gruel worldview. And so once you understand transhumanism, then all of the other ideologies, some of which are just variants of transhumanism, are sort of easy to grasp. Well, I want to ask you a history question here, because some of this is new to me. The history of this that I best understand starts with the E. So the history I've been writing or talking about starts with extropism becoming a pretty big utopian futurist movement. I guess like the '80s and '90s. From there you get rationalism, sort of simultaneously effective altruism, which starts with a bunch of different goals, embraces ideas from rationalism and long-termorism, becomes more focused on a sort of, more of a long-termorism, transhuman utopian anti-AI type of orientation. All right, so what pieces am I missing in that really simplified history? No, that's a pretty good overview. So when Gopru and I are talking about transhumanism, we primarily mean modern transhumanism in its libertarian form. That libertarian form, like the first organized libertarian transhumanist movement was the extropians. Right. And so this is a movement that, like you said, emerged in the late 1980s, early 1990s. There was somewhat famously an extropian mailing list that brought together transhumanists from all around the world. It's where people like Elio Ziegowski, one of the leading AI doomers out there, was able to exchange ideas with people like Nick Bostrom, who's another very important figure in the formation and development of the test girl world view. So you're absolutely right. And then, you know, the other ideologies right after E, in the acronym, like, "sincularitarianism," this is basically just transhumanism, libertarian transhumanism, with a focus on the singularity. And at this point, I feel like most people, listening probably know what the singularity is, right? That wasn't the case five years ago. So, you know, very quick definition. It's actually has several definitions, but the most common one these days is an intelligence explosion, right? Triggered by recursive self-improvement. So you get an AI system that's just clever enough or smart enough to develop the next generation of AI systems. And you get this positive feedback loop. And suddenly, through some kind of exponential or hyperbolic explosion of capabilities, you end up with a super intelligence. And then beyond that, you go out and colonize the entire universe, plunder the vast resource of the cosmos and create this, you know, sprawling, multi-galactic utopia among the stars. And cosmism, the next letter is just transhumanism that emphasizes the aspect of this worldview that I just articulated, which is, it's basically an idea of like, well, why stop at just radically re-engineering the human organism, which is the central goal of transhumanism to create a new post-human species? Let's go out and create technologies that enable us to re-engineer the entire universe. You know, whole galaxies, maybe intervening on the fundamental fabric of space-time to manipulate the world in ways that we find desirable. So it's kind of just like transhumanism with this like broader cosmic vision. And then rationalism was founded by Giacowsky, who was a transhumanist, extropian, singularitarian who hired the leading modern cosmos, namely Ben Gertzel, who worked at the Machine Intelligence Research Institute, which is originally the Singularian Institute founded by Giacowsky. So rationalism like emerged directly out of this movement. And basically the idea was, okay, if we're going to establish a cosmic post-human utopia at some point, we're going to need to develop these advanced technologies. There's no other way around that, especially superintelligence, but to develop them in the right way, such that these technologies do not destroy us, but rather enable us to establish this, you know, sprawling utopia, heaven among the literal heavens, we're going to need to optimize our rationality. So that is essentially the main thrust behind rationalism and effective altruism, as you mentioned, I mean, it emerged basically from rationalist blocks, like overcoming bias and less wrong. The focus there is not so much optimizing our rationality, but rather optimizing our morality. And although the story that effective altruism is like to tell is that initially they were focused primarily on global poverty and then animal welfare, and then this other cause area called long-termism came into the picture later on, really the effective altruism from the very origins of this movement were focused on long-termism, right? So long-termism is basically just cosmism, except it has a more explicit ethical foundation to it. And that ethical foundation, I know there are a lot of isms that I've mentioned so far, but that ethical foundation is utilitarianism, specifically total or totalist utilitarianism. And that just says that we have an obligation to increase the total amount. of value in the universe across cosmic space and time. How do we do that? Well, one way is if you have a fixed population, it's to increase the well-being or welfare of the entities that exist in that population. The other possibility is to just increase the population. So if you double the population and everything else is the same, you know, Keter's Parabas, then you've doubled the amount of value. So from the long-termist view, you know, Cosmism basically, I think one way to think about it is it's just like, wouldn't it be really cool if we as transhumanists radically re-engineered ourselves, became post-human, you know, uploaded our minds to the cloud, went out and colonized the universe? Wouldn't that just be like a really awesome thing to do? Long-termism goes a step further and says, actually, no, we have a kind of ethical obligation to do this. As Will McCaskill says in his 2022 book, what we owe the future, the upshot is a moral case for space settlement, right? And so you could maybe see why Elon Musk has called long-termism, quote, a close match for my philosophy, right? So that's in a little bit more detail, just fleshing out some of the details of your very good first pass history. - So it sounds like the tricky part, but both the simple part about understanding this bundle, that used to term that you and Timnet often use, is there's these little differences, which can be confusing to people where the Yakowsky and the rationalists would be, they were dismissive of a Kurzweil style, singularitarianism because it was too optimistic and he was more negative and the effective altruist are they're doing more like of these explicit utilitarian expected value computations where the rational, you know, so there's these little differences and they don't, they're not all the same group and yet they're really knit together by fundamental ideas about transhumanist, utopia, avoiding extinction and technology being delivered. But what I could never figure out and maybe you know, what were the cultural and technical forces that really made the transhumanist foundation emerge in the '70s and '80s and give birth to this movement that ended up essentially taking over Silicon Valley? - May I comment briefly on the remarks you just made before I answer that question, if you don't mind? So one way that I've explained it in the past is if you have like, let's call it an ideological map, right? And maybe it's like six by four feet. If you are right up close to, and so they're different, these ideologies are plotted geographically on this map. If you are right up next to the map, you will see that rationalism, effective altruism, for example, occupy different locations, right? If you take like five steps back, you find that all of these ideologies are basically co-terminus, right? They're like at the exact same location. And just for perspective, if you are looking from a distance at this ideological map, you would find that like my view, which is very anti-transhumanism, for reasons maybe we could discuss, or maybe like Tim Tekebrou's view, is like five feet away, right? So you've got this cluster of views, of world views that share a whole lot of ideological real estate and in perspective, you find that they are almost the exact, I mean, all of these communities have the exact same vision of the future. That vision is a cosmic post-human utopia through the creation of artificial superintelligence, right? And so this leads to my second comment, if I may, which is just that one way to understand how all of these different alternative, and basically alternative interpretations of test grill fit together, is that within the test grill movement as a whole. So everybody in the test grill movement is aiming to create this cosmic post-human utopia among the stars, right? Then there are differences opinion about the risks associated with developing this necessary technology, which is superintelligence, right? And so you have two camps, dooms and the accelerationists. And of course, it's a spectra of their people in between. But the dooms are represented by individuals like Yodkowski. And he is, it oftentimes presents himself as a prohuman and anti-ASI, but he is neither of those things. He is not pro-human, he is not pro-ASI. Like most people in the test grill movement, he is a pro-extinctions. Part of the vision, integral to the vision of establishing a cosmic post-human utopia is that our biological species disappears, right? So in the good case, if ASI is aligned, we disappear. This is the most important thing for people to understand. These AI companies are run by individuals who have extensive connections with the test grill movement and they are trying to bring about, trying to inaugurate the post-human era. And in the post-human era, our species has no place, right? Utopia is an exclusionary concept. Somewhat or something is always left out. So you always have to ask, like what is being left out of this proposed utopian vision? In theirs, it's our biological species, right? And that's for two reasons. I know I'm going off in a little bit of a tangent here. But on the side, we will become, we will become technologically, scientifically, economically, and politically irrelevant. We will become obsolete, right? In a world that is ruled and run by these digital post-humans that they imagine becoming, we just will serve no function because they will be, I mean, if these individuals are successful in realizing their vision, these post-humans will be superior, right? And so they will be able to do anything that we do better than us. But the situation isn't even worse than that because not only will we be useless, but we'll be in the way, right? So we will continue to suck up valuable resources that they could use for other purposes, like building a Dyson Swarm around the Sun, which Devas Haasipus says we may start doing in the early 2030s, or perhaps covering the entire planet in data centers, which Sam Altman has said on record is the most likely outcome of super teljes. Sam Altman also, by the way, has expressed pro-extinctionist ideas just like all these other people. 'Cause he says in a 2017 blog post that the only way that we will survive the advent of super teljes is by merging with machines. Well, what happens when we emerge with machines and become, you know, uploaded minds and digital posts, well, we're not gonna be human anymore, right? So he's essentially saying that the only way to avoid a certain kind of extinction is to undergo another kind of extinction, namely a radical transformation into the post-human form, right? And so I know I did surround your question a little bit, but you have these tumors and these accelerationists, and another really key thing to understand is that we have no place in the future. I mean, there are some people in the community who would argue that biological life as a whole has no place in the future. Will McCaskill, in the aforementioned book published in 2022, says that it's probably the case that most wild animals have lives that are not worth living. And hence, he concludes that it may be good, our systematic destruction of the biosphere, right? It may actually be good because the fewer wild animals there are the less wild animal suffering. So the goal is to ultimately bring about this digital world in which there may be, you know, not human creatures out there, but they're all gonna be simulated. They're all gonna be, you know, within a computer simulation that we live in. So it's just a deeply anti-life, anti-human vision. Yeah, that's the most important thing to emphasize. - These are the people who watch the Matrix and we're on the Matrix's side. I'm like, "Oh, that's a good idea." But okay, so here's a tricky part about this, which I think is confusing a lot of people. You could probably help us out here. The tricky part is, you have anthropic and open AI where, you know, op-documented this, you've documented this, are very strongly connected to test-creal. I mean, they're basically, they're formed up test-creal motivations, right? Like to bring about this sort of utopian future. But they're feuding, this what confuses people with other people from within the movement. So now you have a huge amounts of effective altruist and rationalist who are pushing back against open AI and anthropic. And I think it fools people in the thinking, "Oh, there's a lot of sort of people outside of that movement "pushing back on what open AI and anthropic are doing." But actually a lot of the pushback is coming from people within the movement who agree with the eventual goal. Yeah, we want to get to a transhuman utopia, but just don't think that Dario Amade or Sam Altman are doing it right, right? And I think this idea that, and I've been arguing, "Hey, and I've been talking to a lot "of, you know, congressional staffers," et cetera, saying, "Look, there's a danger in having this team "just fighting against other members of its own team. "That debate is only going to happen within." So is that more or less right? Is that there's a schism within the movement, but it's a really actually pretty small one. They're all on board with the same track. They just think specifically anthropic and open AI maybe aren't doing it quite right. That is correct. Within team post human, there are family disputes. And as many of us know, family disputes can be vicious, right? But there's this whole other team which is team human, right? That's the team that I'm on. And the debate right now, the public debate about the development of super-intelligent machines should be between team human and team post human. But team post human has more or less monopolized the conversation. They've hogged the microphone. So you're exactly right. So for example, I mentioned that Yadkowski not anti-ASI or pro-human. Even though if you read his book, if anyone builds it, everyone dies, you kind of get that impression. But he has explicitly said in interviews that once post-humanity arrives, then it may be unethical to continue to have biological humans. And he draws this quite offensive ape-list analogy between a post-human choosing to have biological humans or children, or humans continue to have biological humans in the post-human era, and someone right now intentionally choosing to have a disabled child. And in the same interview, he talks about how it might be good just to wipe all humans off off of the earth. In other interviews, he said that if the only and reliable way to create super-telegences, flitting about the universe, "having fun" is to sacrifice all of humanity, then he would make that trade-off. So he has this, he's on team post-human, not a team-human. The debate has to do with AI safety, which is exactly that dispute, the tension between the accelerationists and the doers. The question is how to go about developing super-telegences. Everybody agrees, including Gekowski, that we need super-telegences. As Nick Bostrom says, virtually all pass to a utopian paradise in the future, take us through super-telegences. So we can't not build it. The question is when to build it, and how, right? And so you have some people who are on the doomer side of the, in the doomer camp within the test-girl movement, who are screaming that we aren't yet ready to build super-telegences. We don't know how to align it with human values, but what they mean by human values is the test-girl worldview, right? So for example, if there were a super-telegences that were aligned with my anti-test-girl worldview, and if this super-telegences were to shape the entire future of life in the universe, which is how they imagine what they imagined super-telegences, ultimately, dooming, they would think that's an existential catastrophe, right? Because what they want is a cosmic post-human utopia, right? So it's not about aligning it to human values. It's really about aligning it to test-girlvas, anything other than that, anything anything. I mean, literally, a definition of an existential risk is any event that prevents us from establishing this post-human paradise among the stars. So if you have an ASI that's aligned with some non-test-girl worldview, that is going to be extensively catastrophic, right? So you have people like Yakowsky who are saying that we need more time to solve the alignment problem, to figure out how to control the super-telegences such that we can direct it to radically re-engineer the human organism, upload our minds to computers, and ultimately go out and plunder the vast resources of the universe. All right. So the people within Team Post-Human did our upset at Anthropical Open Now. It's because you don't have, you haven't sufficiently solved the alignment problem to proceed with our shared vision. And the labs are basically saying, no, I think we can do it. We'll figure it out on the fly. We can't really slow down now because of what have you China. But it's basically that's the bottom of the feud, is like, are we able to solve the alignment problem enough? Though it sounds like if you talked to these anthropic employees, what they're really saying is like, hey, we might not. But now we've got to go, we've got to follow through, because other people are going to start doing it now too. And so we got to get there first, because if we don't, then for sure someone else will do it, and it will be the bad version. It will be the version that kills us all. One thing to notice here is you will hear like the AI CEOs and others at the company say that there are really three reasons that they can't stop, right? One has to do with market forces. So just another brief, like, historical attention here. The AI race began, motivated almost entirely for these ideological, test-gruel-related reasons. If you look at what the founders and the people around the founders were saying, you know, back in 2010, 2015, et cetera, no one was talking about profit, no one cared about profit, because all of them realized that Utopia is so much more valuable than however many billions of dollars they might gain in personal wealth, right? This is about fulfilling some kind of grand grandiose eschatological vision. That is why the race started. I know because I was in the community back then, and we had these conversations. And, you know, I knew a lot of these people. But over time, market forces absolutely have crept in and become a major driver of the ASI race. So things have evolved. It started off very ideological. Now, the profit motive is a piece to the explanatory picture, especially after ChatGBT was released in November of 2022. The second reason they'll say is, well, there's some kind of like game theoretic predicament between the AI companies, right? And so, like, they all think that they're more responsible than the others. So, inthropics, trying to race ahead of opening eye, opening eye doesn't like that. They think that anthropic is on the wrong path. So they're trying to race ahead of anthropic. And then the third one, of course, is geopolitical, and that has to do with the race between, so zooming out the race between the US and China. But what the important thing to notice is that what all of these boil down to is anxieties about whether or not utopia, their version of quote unquote utopia will actually be realized, right? So, anthropic is thinking that if open AI builds superintelligence in the very near future, that superintelligence isn't going to be aligned. And the problem outcome is going to be an existential catastrophe, maybe human extinction, right? If China wins the AI race, then we also don't get utopia. Because, you know, whoever builds the first ASI will then establish a singleton, you know, like a world government, they'll have what Bostrom calls a decisive strategic advantage. And so, China will then realize its version of like a quote unquote communist utopia. Well, that's not the test girl utopia that they're envisioning. So ultimately, what all of these excuses, all these pretext for continuing to race ahead, boil down to, is deep anxieties about utopia. And maybe there's one last thing to mention, if I may. Yeah. Which is just that, you know, people have become aware. So the topic has exploded into the public discussion public consciousness over the past few weeks as a result of Jacob Coxon's resignation in Thropic, and an employee there retweeting him and saying, yes, we believe there's a 10% chance of human extinction in the near future. So it's really, really important to properly interpret the terms that these people use. Because oftentimes the way they use terms that seem to have a straightforward definition is defined in an idusyncratic manner, right? Safety, what does safety mean? Safety means building an AI, advanced AI that is aligned with the test girl worldview. What is the likely outcome of successfully aligning ASI with the test girl through? Well, our species will be sideline, marginalized, disempowered, and ultimately eliminated because we'll be of no use and we'll be in the way. And like Yetkowski says, maybe they're ethical reasons, right? So another term that contains hidden ambiguities is the term human extinction, right? So there are both human and extinction are ambiguous. So if you look at the academic test girl literature, you'll find that everyone defines humanity according to what I call for like a better term, a broad definition. On this broad definition, it denotes both our species and whatever post-human successors might come after us, right? So an immediate implication of this definition is that our species could die out literally next year without human extinction having happened, right? As long as there are post-humans to take our place, then humanity will persist. And if humanity persists, then that's sufficient for human extinction to have been avoided. The terminology is confusing because they count post-humans as humans, right? But the other ambiguity is if you take like a narrow definition of humanity as meaning our species in particular, homo sapiens, then the term extinction becomes ambiguous, right? So again, for lack of better terms, there is a difference between terminal extinction as I call it and final extinction. Terminal extinction is just a scenario in which our species disappears entirely and forever full stop. That's the only condition that needs to be satisfied for terminal extinction to have happened. Final extinction kind of builds on this, and it denotes a scenario whereby our species disappears entirely and forever and we do not leave behind any post-human successors. All the test girl people care about is avoiding final extinction. That is it. Many of them hold a pro extinctionist view according to which we should bring about terminal extinction without bringing about final extinction, which is just to say we should survive for long enough to create our post-human successors. And then once our post-human successors are here, we should debount. We should follow the dinosaurs of the dodo into the eternal grave of extinction, right? And so it is deeply frustrating to see the public conversation where people are like, you know, all of these like test girl guys like Jacob Coxon who has deep roots in affective altruism and rationalism, which grew out of the libertarian transhumanist movement, on talking about the importance of avoiding human extinction because most people sitting at home are interpreting that. It's like, yeah, we should like do whatever we can to ensure the continued survival of our species that is not what they're talking about. They're talking about bringing about [BLANK_AUDIO] surviving long enough to bring about the post human era, at which point who cares what happens to the 8.3 billion people on Earth, you know, we're just irrelevant at that point. And from a grand cosmic perspective, we're utterly insignificant, right? So our role in this whole eschatological scheme is ultimately to be the transitional species between the biological and digital realms, right? And that is our role. And once our eschatological duties have been discharged, there is no reason for keeping us proud. So this is a deeply, deeply anti-human position. I hope I didn't ramble for too long, but maybe that was useful. Useful clarification. No, this is critically important. And I think for a lot of people listening to it who are not anywhere near the Bay Area and not anywhere near Tuscalo, this seems almost absurd or it seems almost, you know, like who would actually hold a view like that, but partially what I think makes you such a good, not only critic, but under, you know, expositor on this movement is that you became a part of it for a while. And so I was hoping to ask you about that. So if we can understand what attracted you to it as, you know, incredibly bright academic, you know, I mean, what attracted, what attracted you to this and what did you see on the inside? I think this might help people understand how it might be possible that you have an entire essentially sector of the economy and region of the country that all strongly believes this. Yeah, absolutely. I mean, the, the absurdity of this, this grandiose vision of the future is part of the reason that Tess Grille is still kind of outside the over 10 window. I think that is shifting, you know, there are more and more people waking up to like learning about effective altruism and rationalism and transhumanism. I've seen those terms all over the place there in popular media articles like, like, never before. And so yes, there are many, many people in Silicon Valley who are true believers. And if you find this like whole story to be absolutely nuts, you are, you are getting it. It is completely nuts. And so for me, like, I grew up in a very religious household. And there was lots of talk about eschatological motifs, like the rapture. And, you know, I was convinced that Bill Clinton was the anti-Christ in 1992 when he was elected. I was like, overcome with terror. I was like, oh my god, the tribulations right around the corner, you know, and then in my early 20s, I, as many people do, apostatize my faith. And for, I took another, like, several years before I came across the work of Ray Kurzweil, who I later ended up being a research assistant to for his sequel, sequel to the book Singularity's Near, which is published in 2005. And at first, I thought that these singularitarian view, again, like singularitarianism is a version of transhumanism that just emphasizes the singularity. I thought it was completely outrageous. There's a point in the book where Ray Kurzweil says he believes there is a greater than 50% chance of us making it through this century without destroying ourselves. And he adds, but then again, I've always been accused of being an optimist. So that means like up to a 50% chance that we destroy. So you might say like, well, why develop these technologies in the first place? Well, Kurzweil's acquaintance, Bill Joy, who was the co-founder of Sun Microsystems, who's technologist, no, no, you know, technophob. He published an article, a famous article widely discussed in 2000 on April 1st, 2000 in Wired magazine of all places in all times titled Why the Future Doesn't Need Us, in which he argued that these technologies are too dangerous to develop. So we need to impose like broad moratorium on entire fields of emerging science and technology because the risk is just too great. The reason that these, the testral folks are plowing ahead with developing these technologies is because they are necessary to reach utopia. I alluded to this earlier, right? So if the definition of an existential catastrophe is any event or state, et cetera, that prevents us from reaching cosmic post-human utopia, then failing to develop these technologies would itself be an existential catastrophe, right? So there's only one way forward. And so initially when I came across his work in that that quote about the the risks associated with advanced technologies, I was, I was kind of mortified, but making a long story short, like I eventually got on board and it was kind of like, well, if you can't fight them, then, you know, maybe the best thing is to is to join them. The deeper I got into the transhumanist movement through this through the literature on existential risk, the more compelling I found it because it really perfectly fit the religion shape hole that was left behind. There are so many parallels between transhumanism and traditional Christianity from the promise of resurrection, right? If you're a transhumanist and you don't live long enough to live forever, as Ray Kurzweil says, you can always opt to have your body or just hadn't that cryogenically frozen so that you can be resurrected, not from the grave, but from the fat, right? All the way to the theistic aspects. Like, "Tescal is a I would consider to be a new religious movement." And I would consider this religious movement to be theistic, right? The only difference is that the the Tescal world view inverts the traditional relation between humans and God, rather than God creating us, we're creating God in the form of ASI, a God, you know, God like AI, a machine God, a digital deity, etc. So this is what attracted me. I thought, "Oh, here's like an ideology that is checking all the St. Boxes, all of the things that I miss." You know, meaning, purpose, eschatological hope about the longer term future of igniting the possibility of immortality, you know, of like transcending my biology, becoming this digital, you know, piece of software, living in a computer, something that Sam Altman himself has already taken steps to become by signing up with a company called Dectone, that has framed chemically preserved and then digitized. And, you know, ultimately just just like living in paradise with radical abundance and no suffering, that just really appealed to me. And so I, you know, it was really around 2009 that I publicly, by which I mean, I published an academic paper on on embracing transhumanism and, you know, at the time, like the early, earlier testicle movement. And I continued to be a true believer really up until 2019, at which point I defected and realized that once again, I had been completely wrong about, you know, dearly held beliefs. And so I since then have become a critic. And I mean, if one uses the term a bit loosely kind of like a whistleblower, you know, sounding the alarm about like just how extreme and radical the views are that are held by, you know, people at Oxford and in Silicon Valley. So I want to get soon to why you defected, but sort of two follow up questions about your experience there that I'm very curious about is one asking about the role of there's a sort of logical and sometimes even mathematical complexity to some of the views, which I've heard makes it very attractive. If you're smart, if you're an engineer, because you can very much believe like, oh, I can understand this is actually kind of complicated. This would explain why maybe the masses don't yet get this. But, you know, Dunning Krueger, I'm smart enough to like it makes sense to me that I'm I'm understanding this future. I mean, too, I want to ask you about I've heard that it's a very self reinforcing community. When you're in this world, there's a lot of support. There's a lot of social connection. There's a lot of money sloshing around. There's jobs to be had. And so I'm curious about those two aspects about being inside the test-griel world. Yeah, the community is very culty, right? That obviously ties into the new religious movements aspect. There are many cultish elements. I mean, people in EA are like encouraged to like, you know, get pursue career paths that are recommended by people in EA. They're, you know, they have their own like Lingo, you know, their own Argo, you know, technical charge. Everybody's into expected value theory and Bayesianism. And, you know, they oftentimes date each other. There are a lot of polycules. I mean, people are now discovering this with respect to rationalism. Because rationalism and EA, I mean, they're like twin sibling communities. Again, the really the biggest difference is one specifically focuses on rationality in order to do the most good, especially by creating an aligned superintelligence. EA is about optimizing your morality mostly by focusing on the very far future, which means being concerned with AI safety and AI alignment. So you can see there, you know, a lot of parallels. But I mean, even when I was in the community, I remember I in 2019, I spent several months at the center for this study of existential risk at Cambridge University and over drinks, I had conversations with, you know, at like a party with 15 people about whether or not we might be in a cult, you know. So it's, I mean, it was these conversations that that did end up playing a role. You know, it was one of many factors, but did end up playing a role in the ultimately defecting from the community. So yeah, I mean, I'm sure at, if you go on social media and you look, you search for rationalism, I mean, you'll find that like people are describing as basically like, like a sex cult. It kind of is, like, there are big orgies that they have and, you know, everyone participates. It's a very, very weird, very weird community. And there's a lot of money, too. I was surprised to see you estimate, it was a bigger number than I had heard, but you know, more than I do, that the EA community, for example, has something, you said $26 billion worth of donations have been pledged to that community, which allows them, you know, you go to Oxford, and it'll be like, one of the nicest buildings on campus is going to be an EA funded, you know, building. You look at anthropic. That's a lot of EA money went into fund that. There's these future life institute. These are really large, really wealth. And so as that, we're really talking about billions of dollars. I assume mainly comes from tech, centimillionaires and tech billionaires who like where this narrative puts them, that makes them the heroes of the world. But we're talking about huge amounts of money, too, right? Yeah, just as a quick sign note, I mean, long-termism tells billionaires exactly what they want to hear. It says not only are you morally excused from helping the poor, or focusing on animal welfare, you know, here and now kind of problems, but you're a morally better person, a superior person for focusing on the far future. And so, you know, again, like Elon Musk is a fan of long-termism. No wonder when like a key aspect of long-termism is that it's morally good. It's, you could say it's morally obligatory to go out and colonize the entire universe, conquer the cosmos. It's your absolute right that there's a huge amount of money. That figure, the 26 billion, that was before FTX collapse, right? So, same-backed freed was one of the most prominent EA long-termists. Right? He never had like a global poverty phase, like some EA's do. I mean, EA has been described by Molly Lieberman as a bait and switch, where it sort of draws people in by advertising that they care about, you know, alleviating global poverty. And then once people are in, they say, well, actually, there's a much better way to do the most good. If you can affect, like, possibly affect a tiny, tiny percent of the trillions and trillions and trillions of future digital people spread throughout the universe, the amount of good you'll do, like in terms of expected value, is just many orders of magnitude greater than if you were to lift the 1.2 billion people in multidimensional poverty today out of poverty. So, it's really, it's kind of always been folks. But so, you know, same-backed freed was a leading long-termist. So they lost some of the pledged money. But since then, I mean, they've, they continue to attract just truly mind-boggling amounts of money. And I mean, if you count, if you count inthropic as an EA long-termist organization, and you should, right? Dario Amade was one of the very first effective altruists, right? He was one of the first to sign the giving what we can pledge. I think he was 34 on the list. You know, Amanda Askel is a prominent philosopher at inthropic. She, her ex-husband is William McCaskill, right? So if you count like companies like inthropic, then there's even more money, right? If you start to include all of that money, and of course inthropic is about to have its IPO, which the figure I saw most recently, I think was two trillion, is what they might aim for. - It's coming down now because people are, their institutional investors want a little risk mitigation because they think they're sounding like crazy people, because if you're a pension fund, you're not on board with the test-cree altruists. You're like, whoa, wait, what do you guys believe? But a lot over a trillion, for sure. - Yeah, over a trillion. So there's gonna be this massive influx of money. And also just as a sign note, there is another, as it were using the term loosely whistleblower, who was heavily involved in the test-cural EA rationalist, et cetera, communities like I was named Holly Elmore. And she has known personally, some of the people at inthropic, and she's also out there sounding the alarm. One of the quotes, this is what I wanted to mention, was that the people at inthropic decided to pursue superintelligence to ultimately create utopia for post-humans, right? That I've already mentioned that. And she goes on to say that many in the company are concerned that the superintelligence that they create will just completely usurp, or supplant humanity. And that this is why it's really important to figure out how to transfer our values. Like basically, the, are ethical and axiological values to the superintelligence. So that it can go out and do what we, what we ultimately think is the right thing for it to do, ethically speaking, namely colonize the universe, create all these computer simulations, and maximize value. So anyways, effective altruism is bad to get an enormous amount of money. And this is a community of people who oftentimes present themselves as living humble modest lifestyles. And some of them do that, right? But there's also like a kind of televangelist aspect to this. So like McCaskill, when he published his book, he went out to a, of a vegan restaurant in, in New York City and paid. I think it was, it was more than $400 a plate, right? Also the center for effective altruism, well, technically effective ventures, which is the umbrella organization over it. Secretly bought a, I think it was 15 million pound castle. So a palatial estate, in Oxfordshire, two toast conferences and get togethers, et cetera. And when a New Yorker journalist asked them about this, they said, this is before FTX imploded. They said, well, we can, if you do the math, right? If you crunch the numbers and you, you know, figure out the expected value of this. If there's even a small probability that this castle brings in the next sand fragment for you, the investment will have totally been worth it, right? And that's not only, like there was a chateau, and I think the Czech Republic or something that the EAs also bought. So once you, you know, EA, like from a distance, like doing the most good, you could possibly do, that sounds great. And I'm all for that, right? And I'm all for like donating once income, you know, I'm committed to doing that as well. Like all of my disposable income to charities, but when you actually like get close and look at the details, it's just a deeply flawed view. I mean, the whole reason sand fragment for you went into crypto, I mean, he initially worked at Wall Street, was to follow this career path that EA's call earned to give, whereby you try to make as much money as possible, even if you work for, quote, immoral organizations, as Will McCaskill says, so that you can donate more money to charity, right? So this is what got sand fragment for you on Wall Street, then he worked for Center for Effective Autism, and then he got into crypto, and he was, you know, a hardcore utilitarian, and if your hardcore utilitarian, you believe that there is no act that is intrinsically wrong. Like that is, this is a key aspect of the ethical theory. All that matters are the consequences. So there's nothing intrinsically wrong with fraud, with lying, with cheating. So nothing intrinsically wrong with murder or genocide. If the outcome is overall good, you know, all things considered, then you have a moral obligation to engage in fraud, and lie and cheat and steal, and so on. So you can see now how bankered freed ended up, you know, justifying his own mind, his actions, 'cause, you know, he was a long-termist, and he was donated, I mean, FTX invested in the anthropic, right, so he was really concerned about ensuring that we build an aligned with the Tesco World View, super-intelligence, so that we could ultimately maximize value from now until the heat death of the universe. So how did you turn around on this? So you're, you know, we're in the 2010s, you know, 2017, you write a big book that's sort of explaining some of these ideas and a positive light. What was it that led to your second, you know, apostasy in your life? Why did you, what turned you? - Yeah, an apostasy is the right term, because I do think that I was involved in a new religious movement. So there were really a bunch of, I guess they were, there were like, mainly three considerations that resulted in this drastic pivot away from the, you know, what I considered to be my intellectual home for a decade, one was the realization of the poverty of utilitarianism, which really underlies like basically the entire Tesco movement, even some of the far-out accelerationists who argue that the ultimate goal is to climb the cartichev scale, which is a measure of basically how much energy a civilization uses to, to bring about the heat death of the universe faster. Like ultimately, it's a kind of utilitarian view because it's saying that there is something of value that we need to maximize and that thing is entropy, right? So it's not like happiness or pleasure as traditional utilitarians would say, but it's entrance, so it's still deeply utilitarian in flavor. And so utilitarianism, like consider its view of, its conception of persons, of you and I. Like we have no intrinsic value. There is only one thing that has intrinsic value in the universe and that is well-being or welfare, right? You and I are nothing more than the containers or vessels or vehicles of value. And so I matter as a means to an end, not as an end in and of myself. Insofar as I am bringing value like pleasure into the universe, then I am fulfilling my ethical obligations. And I matter only insofar as I'm doing that, right? And so this gets back to the idea of like maximizing future population because if we are just these funcible value containers, then if you create a whole bunch of new value containers, right, especially if they're digital because you create a more digital people per volumetric unit of space time or whatever, then you can buy logical beings. So if you create all of these these value containers and they all have like net positive amounts of value on average, then you're going to you're going to also increase the total amount of value in the universe. So like this view of people as as mere means to an end, it didn't sit right with me. I guess it did for a while, but the more I reflected on it, the more convinced I was this is just a deeply impoverished way of thinking about things. And maybe I'll mention the one other reason, which is that which is that I began to realize that the the vision of the future that these people had articulated is essentially technochapitalism on steroids, right. So that going out and conquering the universe, you know, subjugating nature, maximizing economic productivity. I mean, literally, Bostrom one point defines the essential risks, any event that prevents us from reaching technological maturity and technological maturity, he defines as a situation in which we have maximized economic productivity and fully basically subdued nature. It's a very big hone in. And one thing I noticed is was kind of completely obvious in retrospect. It's that this vision was the product of almost entirely the product of white dudes who were very privileged, you know, quite wealthy in Oxford and in Silicon Valley. And if you look at the test cruel literature, there's virtually zero reference to what the future ought to look like from alternative perspectives, feminism, indigeneity, disability, queerness, and so on and so on. There's a huge list. So I mentioned before that that utopia is inherently exclusionary. So I start to think like, gosh, like is the future that we are trying to bring about through in aligned super intelligence, really going to include like all of humanity, like who's going to get left out? Which, which, because you know, we're not like trying to determine what the future should look like in a democratic manner through some kind of democratic process by actually going out and asking people, what do you want the future to look like? There's a colleague of mine named Monica Bielskita. And she has this great line, which she specifically referring to the future. She said, you cannot design for, you must always design with, right? So from the very start, you have to include people or representatives from every tradition, culture, religion, gender, you know, socio-economic status and so on and so on, from the very start, right? And that is, that is the only way you can go about creating a genuinely good and inclusive future that benefits all of humanity. But the, the next step of this line of thinking was recognizing that, oh, actually, yeah, it's that no humans have a place in the future. Right. Obviously, like we're going to create these, these post humans. And like recourse will says in the singularities near, if you don't want to upgrade your genes or, you know, become post human some other manner, that's fine with me, but you won't be around for very long to influence the debate, right? So it's essentially saying become post human or just tie out, right? And so this, this seemed even more troubling than the already troubling conclusion I came to that like, yeah, our species just doesn't have a place in the future. And I'm not sure that's right. And furthermore, I mean, the only plausible way that we could transition fully from a human to a post human era is through some involuntary means because the vast majority of humans out there are not going to consent or give their permission to allow post humans to take over the world, right? And that's the whole point again, for post humans to take over the world to run, you know, the politics, the economy, science, technology and so on. So there's going to have to be some kind of involuntary forceful coercive push by those of us in the testicle movement I was thinking at the time, to actually bring about this transition. And well, that obviously introduces all sorts of profound ethical problems, right? Violation of rights, you know, there are 8.3 billion people on the earth. When you start to like list the cities, you know, like, okay, so like everyone, we're talking about like everyone in Toronto in Singapore, and you know, you go through, it's like, like, there are a lot of people around the world. And how heubristic is it for us to say, well, you know, we know best, we're smarter than everyone. And so, you know, these like the dumb masses, I mean, this is the way a lot of people think that the dumb masses, they just don't understand the stakes. They don't understand the ethical importance of bringing about this cosmic post human utopia. And so kind of like, you know, forget them, like kind of screw them because they're just, you know, as, as Yadkowski would say, they're NPCs, you know, non-player characters. So they just don't really matter. So that was the other thing that the consideration that really, you know, put a nail in the coffin and let me away from the community and encourage me to become a critic. Andrew, so, I mean, is this going too far to say, because this is what's occurred to me is that when I think about this type of long-termerist, expected value style utilitarianism where, you know, like an individual human's life. So you move away from that sort of fundamental natural law, religious principle at the core of, you know, all normative moral thinking of a human or infinitely valuable, a human life is infinitely valuable. When you move away from that liberal conception, immediately what comes to mind is the 20th century utopian movements. So if you look at far right fascism, if you look at the far left movements, the mid-century were completely willing for many millions of people right now to die because, yeah, but the utopia they were going towards was worth it. So if you're in Nazi Germany, you're like, well, yeah, maybe it's not the best we have to kill, you know, six, eight million people, but what matters is this is going to give us this sort of utopian state, and that's worth more. Or if you're, you know, the Soviet Russia at a certain point, like sure, like half of Ukraine is dying from starvation, but the point is you have to break a couple of eggs and make an omelet, and the omelet is going to be this sort of utopian, you know, socialist communist world. And so there was as soon as you move away from the precept of life as a human is infinitely valuable, and has their own natural rights. So as soon as you move away from that, we've always, we've tried this experiment. It ends in a place that we just universally look back and say that was really bad. Am I going too far to put this in the same ontological category within the world of ethics? In my opinion, absolutely not. The utopia, the thing to understand is that utopia justifies anything and everything, right? Because utopia is marked by infinite or near infinite or as test-grillists would say astronomical amounts of value. There is no act, however extreme, however violent, et cetera, that can't be justified, right? If you, you know, if you do the expected value calculations, like committing genocide might actually be the, like, like absolutely the best thing for you to do. There's a statistician who named Oli Haikstrom, who was generally sympathetic with long-termism, but he has this moment of, what I would describe, his moment of clarity in his, I think, his 2015 book, Here Be Dragons, where he talks about it. Somebody, you know, Nick Bostrom has said that reducing, because there could be so many digital people in the far future enhance so much value. Again, his estimate is like 10 to the 58 digital people at least, that decreasing the probability of an existential catastrophe by a tiny, tiny amount, which is just to say, it's just another way of saying increasing the probability of this cosmopolist human utopia by a tiny, tiny amount is worth the lives of billions and billions of actual living human beings, right? So Haikstrom says if there's a politician out there who, who takes seriously this line of reasoning and is told by, you know, intelligence agencies that there's a 1 in a million chance that some lunatic in somewhere in Germany, they don't know, they don't know where, is about to bring about some kind of existential catastrophe, destroy this, as Toby Ord says, vast and glorious future among the stars. Then the president might be justified in just duking all of Germany, right? So there's absolutely an enormous, and I mean, that was sort of hypothetical back in 2015, but if you fast forward to 2023, you find Yekowsky publishing an article in Thai magazine in which he says that the almost certain outcome of vis-a-line superintelligence is that everyone on Earth will die. And again, his concern here isn't that 8.3 billion people will die. I mean, he would say like that's a part of it, but much, much more important is the loss of what he calls our glorious transhumanist future, right? So all that future value, which is again, another way to say he cares about finding avoiding final extinction, not terminal extinction, right? And so he's not in a rub, not in a rub, but he, there's a tweet of his about this is going around that makes his stance much more clear where he said, if we had a fear that misaligned superintelligence was emerging, thermonuclear, global thermonuclear war, it was not a bad option. Sure, it would kill most people, but probably there would be some humans who would survive a thermonuclear war, and then they would have a shot thousands of years in the future of going for better align superintelligence. intelligence. So he said, I'm completely fine with killing 99.9% of the population right now. If I thought that was going to stop a misline super intelligence and the survivors that are thrown back to the Paleolithic, like again, a couple thousand years, they'll rebuild civilization and that we have another shot at getting super intelligence, right? So he explicitly said, yeah, I would discount the lives don't matter. It's exactly what you're talking about. It's the glorious future. Right. So it's exactly the kind of of a profoundly pernicious reasoning that Hextram highlights in the passage, right? So Yutkowski is saying, okay, there's a thermonuclear war, like probably it's probably not the case that everyone will die. And the best calculations that I've seen from the past six, seven years support that. Like maybe five billion people would die, but that would leave a reassuring three billion to rebuild civilization. So he's saying that, like, we should risk thermonuclear war. If that is what is necessary to prevent misaligned super intelligence, then we should do that. He was asked online, quote, how many people are allowed to die to prevent AGI, end quote, from being built in the near future, because it would be misaligned. He says, and his response is like, basically everyone as long as there are enough survivors to enclose proximity to carry on the human species to rebuild civilization. And then I'm quoting, then there's still a chance of reaching the stars someday. Well, if you, like, just consider like what that means, the minimum viable human population is estimated to be to be between 40,000 and maybe as low as 150 people. Right. So pretty, like nobody knows. But even if it's just 40,000, if you subtract 40,000 from 8.3 billion, you get just rounding off 8.3 billion. Right. So he's saying 8.3 billion people should be, quote, allowed to die to, you know, ensure that that super intelligence is aligned. Again, he is pro super touch. If we figured out a way to align super intelligence to the Tesco world view tomorrow, he would come out tomorrow and say, pedal to the metal, build ASI as quickly as possible. We'll create post humanity. You know, it'll be unethical to have biological humans. We'll wipe them off Earth. We'll replace them and so on. And so it's, it's just important to understand how like absolutely extreme that view is. But you're absolutely, you're completely right that utopia is just a, I think a very dangerous concept. And the other thing I would add to this is I have been speaking so far as if the real successful realization of the test grill vision is possible. Right. And I've been saying that, you know, if they successfully realize their vision, then the outcome of safe, safe super intelligence is that will become completely irrelevant. Or according to many of them, like Peter Teal, like we should, we should disappear. Right. I actually, in very skeptical that they could successfully realize this vision, but you brought up utopian movements of the 20th century. And examples go back many centuries before that. These utopian movements never reached their goal, their goal. But that doesn't mean that they weren't profoundly dangerous. They left a trail. It's so many cases they left a trail of destruction behind them. I mean, hit their promised a thousand year, right? You know, he basically was barring from, you know, Christian eschatological, the Christian eschatological narrative, which promises a millennial kingdom at some point in the future that one of the, the bloodiest battles in human history was the taping rebellion of the 19th century in China. And resulted, I don't know, like something like 30 million deaths. I can't remember the exact figure. But that was, that this was driven by the taping, having, having the root kingdom, which was essentially utopian apocalyptic movement, just like, just like Tesquirol. You know, terrorist, terrorist groups, doomsday, cults, you know, the Islamic state, you know, they were in apocalyptic movement. Well, from the religious point of view, the other side of the apocalypse is utopian. So it's just another way of saying that they were motivated by utopian visions of the future. And that's why they were trying to, literally, they were trying to start the, the Islamic version of the battle of Armageddon in this small Syrian town in, in Northern Syrian area, Aleppo called the Depeak, which is said to be the location of their version of the Grand Battle. So it's just, my, my main concern is that, you know, so far, the AI companies have left a trail of destruction behind them. I mean, some people would argue that they're like tearing apart society. They're flooding the internet with, you know, disinformation, deep fakes. They're working with the US military to engage in illegal activities like kidnapping, Maduro and bombing Iran. They're partnering with Palantir to sort of, like, build the infrastructure for totalitarian state. And so so all of these are, I mean, not, not to mention the massive, massive intellectual property theft that they're all guilty of. So these are all, these are like harms that we can point to right now. And I think the harms are going to get worse, but from their perspective, like, like, they're marching to Utopia. And sometimes the march of progress, like, it's going to trample some people. And also, like, from the grand cosmic point of view, all of these harms that they're causing right now, they are mere pecadillos, right? They're just small, like, little mishaps that Utopia will forgive because Utopia forgives everything. So this brings me to my, the last thing I want to ask you about before I let you go is trying to understand, okay, what's happening right now? And I'll say, I'm sure you've experienced this as well. But I find if I, if I write or talk about some of these movements, a very common response I get is like, you're just being ad hominin. Sure, these are weird people, but they were right. The things they were warning us about are starting to happen. So who cares if they're weird or not? They were right. Now, I have this alternate view. I want to run by you that actually it's a self-fulfilling prophecy, right? So that they're, they're talking about AI in the ways that matches the test greal sort of predictions because they are from test greal. And they're trying to bring those predictions about. And I also am increasingly believing, if you look at Anthropocropa and AI, is that they're specifically running experiments and developing this technology in a way that's creating dangers like you were just talking about the dangerous experiments, have hazard experiments, unpredictable experiments, exactly because they're trying to rush forward a movement towards their predicted sort of super intelligent future. The strong version of this argument would say, if we go to a counterfactual world where you don't have these two test greal aligned labs at the frontier and you just have other technology companies unrelated to these philosophies pushing forward generative AI, we would be having a completely different conversation. There would be no conversation about existential death, the sort of really unpredictable scary experiments, like what happened at Openair. The summer just would not have happened. No one responsible would have run those. And that so that we have almost, we're already seeing this damage on the Ralph Utopia. It's not that they predicted this right. It's that they're, they are trying to make their predictions happen. But I want to get your final take on this is, how do we understand what is happening right now, the things that are concerning people in the AI world, how that connects to the test greal philosophy? Yeah, I think you were exactly right. So one way I put it in the past is if you want to understand this kind of brings this full circle, if you want to understand the behavior of companies, virtually all companies in our capitalist milieu, you have to look no further than the profit motive, right? Why are the fossil fuel companies destroying the world for profit? Why pharmaceutical companies jacking up prices? Well, they just will, you know, want to increase the revenue and so on and so on. If you look at the AI companies, they are different. They are anomalous. And I can't think of any other examples, right? I said before that profit has become increasingly significant factor that explains kind of their their behavior, the dynamics, the race dynamics they're in. But if you look at the origins and if you look at the core of these companies, what is really ultimately driving them, it is ideological. It's it's test cruel. Test cruel is integral to an explanation of why there's an ASI race, why these companies arose in the first place. All of this is to say that because these companies directly emerged out of the test cruel movement, claiming that test cruel advocates back in the day were prescient for talking about AI risks and so on is is is a complete misreading of the situation. It's sort of like if if someone were to be impressed that, you know, I woke up this morning and I prophesized I will soon have a cup of coffee. And then and then amazingly, I got up. I made a cup of coffee and I drank it. So it was like, wow, that's that's incredible that you you accurately predicted what what what was going to happen. Well, yeah, because like, you know, I mean, deep mind got its funding because Demis Haasibis and Shane Lake, two of the three co founders gave talks at the Singularity Summit. Singularity Summit was founded by Yakowsky, Ray Kurzweil and Peter Teo. Yakowsky was funded by Peter Teo. The connections here are absolutely convoluted and and completely incestuous. I've written an article about this, but it's it's it's it's stunning. I mean, it looks like a conspiracy when you actually look at the details and you know, not all conspiracies are conspiracy theories. But it's it's just like an incestuous network of these people like exchanging money and supporting each other. So they found it the Singularity Summit. Demis Haasibis and Shane Lake were invited. They gave a talk about super intelligence because Shane Lake had just published his dissertation on super intelligence for rich he won a prize from Yakowsky's Singularity Institute for $10,000. And then after the [BLANK_AUDIO] some event. They followed Peter Teal back to his mansion, and Peter Teal is as test real as anyone else in this community. They asked him for funding, and he gave them funding. So I mean, the deep mind, which was the company that in a sense, you could say kind of initiated the super intelligence race, because OpenAI was founded because they were concerns about Devas Haasibis establishing, but they referred to as an AGI dictatorship. Basically, I'm just taking over the world. And so the whole reason that deep mind came into existence is because Shane Legg was deeply involved in this community. Devas Haasibis was into the idea of the singularity that they gave talks at this conference co-founded by Yad Kowski. So to say that Yad Kowski was clear-voyant in some sense for having anticipated what's going on now, is just to completely misinterpret the situation. You're totally right that this is a self-fulfilling prophecy. And so we have them to blame for the government predicament. You know, nobody has done more than the test girl people, including the doomers. Nobody has done more than the doomers to launch, sustain, and accelerate the race to build super intelligence. So rather than saying they're prescient, you should point your finger and blame them for putting us in this situation, because my personal view is that if there had not been a test girl movement, there would not be a race to build these like general systems that exceed human capabilities in every cognitive domain of interest. I don't think that would exist right now. So they are responsible for the unfortunate plate that we all now fight each other ourselves in. Well, I think, I mean, this is, it's vitally important information. It's confusing. So I'm glad we got to spend some extra time to try to pull this apart. And I've been saying this to my audience a lot. You have to understand this. This is not ad hominin. This is not, well, they might be right, but they're weird. It's the actions we are seeing by certain companies, the rhetoric we're seeing by certain companies, as you said, is in a almost unprecedented situation. I also try to come up with an example from, you know, American history, at least, where you had a really large corporation that was not being driven by the profit motive, but instead by a sort of philosophical sort of religious motive. You have to find, I guess, a sort of explicitly religiously aligned organization, I guess. So it's really important to understand what's going on. And you've seen it from both sides. So I appreciate you coming on. I want to point people towards all the excellent work you do on this. Your website is it's ex-riskology. Am I getting that right? Yes, but people should really go to my newsletter newsletter. Okay, go for it. Yeah. Yeah, which is, which is titled the real-time tech apocalypse newsletter. And my podcast is dystopian out. So it's sort of like the darker the topic, the greater the chances that I'll have something to do with it. I'm also going to push people. If you want a really good sort of article linked introduction to Emile's thoughts, check out the article against long-termorism. There was an A on magazine. And I think it's like a 6,000-word manifesto that covers a lot of these points, but very systematically. It's a great work of sort of modern applied ethics. I would also point people towards that as a good introduction as well. Yeah, thanks. And I think the only thing that's missing is I had fully developed these thoughts is the pro-extinctionist element that is inherent to the Tesco world view. So by more recent, like there's one-half pro-human manifesto is I think another like good, I mean, so far as anything I write is worth reading. I feel like that is very much worth checking out. All right. Well, you heard it here first. Check it out. You need to understand this world. Emile can be your guide to it. All right. Thank you again for joining us. Thanks a lot. All right. So that was my interview with Emile Torres. That was really useful. I've been writing about these ideologies for a little while now. I had a Big New York Times article. I've done a bunch of podcast interviews where I've given my best rendition. Even last week's Thursday episode of this show got into some of those details. But I think what I learned today that wasn't as clear to me before was just how extinctionist all these beliefs are. I think Emile is very good on this point that the goal, the ultimate goal of all of these Tesco role bundle of ideologies is a post-human utopia where biological humans are gone. And what we're left with are these exalted digital angels, these sort of digital post-human versions of humanity. But what as Emile points out, that means at some point these are tech overlards who are trying to bring this this future to fruition are going to have to kill all the other the meat biological people. Because most biological people in a world of eight billion people don't want to say I'll just disappear and we can have a select few go off into a transhumanist future. They're like we like our world as we have it. So if you really pull the thread, this was new to me and quite frankly it's disconcerting. But I think Emile's right about this. If you pull the thread on the Tesco role bundle of ideologies, you end up at some point killing billions of people on purpose because they're getting in the way of your dream of a digital utopia. All right, the other key point that came out of this that we both emphasized. Neither Emile or myself think that this vision is actually possible that you actually are going to if you have a good enough agent on your LLM or something you're going to end up with a super intelligent God. There's a lot of good computer science reasons to say that's kind of fanciful. But Emile and me both agree on this point. Pursuing utopia can be devastating. That's what we've seen. The Nazis did not get the thousand year right. But tens of millions of people died. The Soviet communists did not get their socialist utopia. But tens of millions of people died on the way. So we've learned this lesson well. The pursuit of utopias where you're saying the value of humans alive right now pale in comparison to the value of the utopia that I am pursuing. We have seen that exact configuration throughout history. No one has ever successfully got to their utopia. Every time it's been pursued with any sort of well-funded vigor, it's been a blight on humanity's history. So we really need to be woke to these ideologies and what's really happening here. I am a fan of AI. I think gender AI can produce any number of useful tools. And the whole thing is being solely to hijacked and made into this unstable dynamite bundle of terribleness by this weird utopian anti-humanist, pretty dark philosophy that's incredibly well-funded and the two top frontier labs are completely driven by it. So I think this is a very important message. It needs to be out there. This has to be a part of the conversation we're having when we're trying to understand the AI moment, but also when we're trying to understand what to do next. So Emile, thank you for coming on the show. Hopefully you enjoyed this. See you again next time. Hey, if you've made it this far, you must be ready to join my fight for depth in a distracted world. Now, the best way to do this is to join over 125,000 people who receive my email newsletter each Monday. You can sign up at calnewport.com/ideas. And when you do, I will send you a free guide to my seven best ideas about cultivating a deep life. Sign up today at calnewport.com/ideas.