An Insider's Guide to AI Doom

The conversation surrounding AI harms has really heated up in recent weeks, but it's also at the same time become really confusing and hard to follow. There's many different groups involved and it can be difficult to try to figure out who belongs to what group, what group of leaves, what, how do they feel about each other? I don't know how you can follow the current AI discussions about harms if you do not understand this landscape of different players, so that's what I want to help you figure out today. This is a short episode I'm going to record where I'm going to lay out the three major categories of people involved in the AI harm debate. I'll tell you who's in each category, what they believe in how they feel about each other. So if you are interested in the current discussion on this technology, you need to understand how this landscape is actually configured. Alright, so let's get into it. Alright, so let me give you two caveats before we get started. One, I'm going to make some mistakes here. I'm basically just winging this episode and the boundaries between these different groups I'm going to talk about are a little bit porous and there's some people that fall in between and some people that won't like the characterization. So I apologize in advance, but I believe the landscape I'm about to present you is more or less correct and will definitely help you better understand the current discourse on AI harms. My second caveat is I'm going to try to keep my personal opinion out of it. That's not the point of this episode. I really do want to just give you to the best of my understanding, the current lay of the land in the sort of AI discourse that's happening at the moment. Okay, so let's get into this. I'm going to bring out my tablet here because this gets complicated enough that we need to actually probably see how these different groups connect. I'm going to start over here by talking about the first of the three major groups that's involved right now in the current AI discourse. And we're going to call these the Duma group, let me all pluralize it. Let's call it the Duma's. All right, this is an important group in the current AI harm debate. Who is in this group? Well, I'm going to start by putting rats and EA down here. These two philosophical movements give the intellectual underpinning for the group that I'm calling the Duma's. So let me give you the quick history on this. You have to go back to Silicon Valley futurist movements in the 90s into the 2000s. In particular, there is a movement called the extropians that was futurist and very utopian. It believed that technology was advancing at such a rate that it was going to deliver us in the future, a utopia, probably a transhumanist utopia where we merge with the machines in such a way that we could have infinite life or whatever else we think is critical to our utopian visions. There's two main technological threads that the extropians wove together. One was nanotechnology, which was just becoming big in the 80s and 90s when a lot of the extropian movement got started. They were influenced by a thinker named Eric Drexler who wrote about this for both academic and public audiences. So they thought nanomachines would be able to repair our bodies, reverse aging, reverse disease, etc. The other major technological thread that the extropians held on to was AI. They said maybe we're going to have a super intelligent AI that could act like essentially a benevolent digital deity. So this was going to be the other technological move that was going to deliver us this utopia. All right. So that's the extropians. They're pretty positive. Then you have a particular extropian. This was a young man named Eliezer Yadkowski, who dropped out of school after the 8th grade, was reading a lot of science fiction, found his way to the extropian mailing list, got really into these ideas, was writing a lot for these email lists, eventually got funding to start his own sort of techno utopian institute. He came to believe, as time went on, that though super intelligent AI could deliver a utopia, it could also just as easily kill us all, and that it was actually more eschatological than the extropians realized that super intelligence AI was inevitable, but it's not inevitable that it was going to deliver good things. And so he began to write and think quite a bit about what would be required to prevent super intelligent AI from killing us all, and to help make sure that people understood this danger before we created this technology. So Yadkowski's group became known as the rationalist, right? So that's why I have rats written here under Dumourism, it's a shorthand for the rationalist. All right. But I also wrote EA under there. This stands for effective altruist. Now this is a movement you've also probably heard a lot about recently. This is a little bit complicated, but I'll summarize it really briefly. So the origins of the effect of altruist is William McCaskill and Toby Orde, who are two philosophers at Oxford in the 2000-2010s. They were heavily influenced among others by the Princeton ethicist Peter Singer, to be to apply logic and rationality to being altruistic. And so they're trying to be very effective at altruism. So early on, the effect of altruist, for example, were really big on rating charities, right? Because they figured you want to give your money to a charity that's going to do the most good per dollar invested. So they worked on projects to help rate charities. They also had another campaign called Give Well, which said, look, if you have a high-paying job, the best thing you can do for the world is keep that job, but give a significant portion of your high income towards really effective philanthropy. So like these are the type of things the effect of altruist were thinking about. But pretty soon, they also became, as a general movement, very worried about super intelligent AI. There's lots of different points of influence between the rationalists and the effect of altruist. A key one is Nick Bostrom. He was also a philosopher at Oxford. But he was also heavily involved in the extropian mailing list in the 90s into the early 2000s. And he was very convinced, especially by Yad Kowski's style arguments, that super intelligent AI was inevitable and without care was going to kill us all. And so he was one of several voices and effect of altruism that moved the movement away from just thinking about effective giving and towards what was known as a long-termorism view that said, wait, the most altruistic thing you could do is prevent the extinction of humanity. And so he started an institute at Oxford that looked at different existential risks. But over time, super intelligent AI killing us all became the dominant risks they cared about. So we're in a world now where effective altruism is a very large movement. It's incredibly well-funded mainly by techs and to millionaires and tech billionaires. It deals with lots of things. It's not only focused on super intelligent AI in the way that rationalism really is, but it's become a dominant theme. William McCaskill gave this well-known speech, I believe it was at Google, where he effectively said, look, if we're going to be super rational, we have to think long-term. And effectively, why worry too much about hundreds of millions of lives right now? And we really, if we want the biggest return on our efforts, need to think about preventing human extinction, which would prevent tens of trillions of future lives from ever being born. They pulled on this logical thread long enough and a lot of effective altruists said, preventing super intelligent AI from killing us all, that is what effective altruism showed focus. Now, again, not everyone in EA is focusing on that, Dustin Mosquitz, for example, recently gave a huge amount of money to foreign aid after USAID was effectively denuded by the doge effort. So it's a bigger movement than the rationalist, but super intelligent AI is a big part of their portfolio. I've heard from a lot of people that in the Bay Area in particular, the rationalist and effective altruists are sort of indistinguishable. This is why, for example, if you read the really good reporting by Cade Metz, the lead technology reporter for The New York Times, he often puts those two words together. He will say EA/rationalist circles are saying X because in the Bay Area, they become more or less the same thing. Okay. So to summarize, the rationalist and the effective altruist think super intelligent AI is inevitable. They've been thinking this long before LLMs. They think that if we're not really careful, it will kill us all. But if we're careful and do this just right, it will deliver us a transhuman utopia that is like the ultimate good and therefore should be a, this should be a major focus of our altruistic instincts. Okay. So I want to return now to our diagram and I'm going to add, this is where things get a little bit more complicated. I'm going to add anthropic and open AI under the Duma category. Okay. So this is really important. There are a lot of AI companies in the world and in America that have hyperscaled LLMs are working on big projects. But those two labs in particular, anthropic and open AI are distinguished by the reality that they come out of the rationalist effective altruist intellectual tradition. They were started in large part to try to fulfill the rationalist effective altruist idea that we have to care. a line super intelligent AI to prevent extinction and who instead get us closer to a transhumanist utopia. That's why open AI was founded. You have to remember Elon Musk originally funded open AI because deep mind, which also came out of these circles, was bought by Google and they said, we can't trust a big company to shepherd super intelligent AI the right way. So we'll start open AI to do that. And then anthropic was breakaway employees from open AI. They broke away two years later after the for profit subsidiary was declared an open AI. They broke away and said, open AI is not sufficiently pursuing the rationalist effective altruist goals of steering away from bad super intelligent AI and towards good super intelligent AI. And throbbing in particular is just completely and sensuously connected with these movements. I mean, most of the many of their employees, Dario Amade himself, he used to live in a group house with other really major effective altruists, his sister who helped start anthropic wisdom, married one of them, a lot of the initial funding for anthropic came from effective altruist individuals like Sandbake been freed and effective altruist investment funds. There's, anyways, there's a lot of connections, which I don't want to go through now, but I would recommend you look at my New York Times op-ed about this from a few weeks ago or Kate Metz did a really good appearance on the daily, the New York Times daily podcast couple weeks ago where he walked through this history. Michelle Goldberg did a good column on this history of the Wall Street Journal last weekend had a good column on this history. Look at Adam Becker's book is very good on this. Listen to my podcast with Adam Becker and Zitron where we went through this. So there's a lot of sources out there detailing these connections. But what I want us to focus on here is this idea that anthropic and open AI share the same intellectual tap route as the rationalist and the AI focused effective altruist. Now, what makes this complicated is that the relationship between these two frontier labs and the rationalist and effective altruist who are not part of those labs is strained right now. So you have some people from these circles, right, like Eliezer-Yakowski from the rationalist who are completely upset and anthropic and open AI. They say you guys are moving too fast. You're going to end up with the bad super intelligent AI. They all share the belief that eventually we want a transhumanist utopia delivered by AI. But like Yakowski would say, you guys are going too fast. We're not ready to do that right. I think you're carrying too much about money or winning. And so Yakowski is openly hostile towards these two labs. You have a bunch of effective altruists who are sort of in the middle who maybe are a little bit on board with Dario Amade and saying like, yeah, we need to have regulation that will allow Amade to set the pace because he'll do it right and we need to prevent competitors from going too fast because they're not going to do it right. And then you have a bunch of effective altruists who are out there lobbying hard to say like we have to basically shut down these labs. They're doing it wrong. So it's complicated because they all believe the same general things. Super intelligent AI is inevitable. It will either be a good God or a bad God. They differ, however, in whether they believe we can get the good God right now or whether we're not ready and we need to wait. And so the people at the labs think we can deliver the good God and a lot of the people not at the labs are saying we're not ready yet. Now the reason why that's confusing is that these feel like they're opposed parties, right? So when you see Nate Sores and other effect and then some effective altruist figure surrounding Bernie Sanders as he talks about legislation to try to prevent super intelligent AI, you would assume that these are from a completely different mindset than the frontier labs that this legislation would apply to, but they all come from the same doomer mentality. They just differ on where we are on this timeline that they both agree about. All right. So that's the AI doomers, but that's only one of three parties. So let me draw another one up here. Another key party right now, I'm just going to write down the word, I'm going to abbreviate accelerationist. So the accelerationist and they, I don't know, they'd call themselves other things. The accelerationist are a group that believes that the doomers are weird and that we need to go as fast as possible because there's a lot of benefits from AI, why slow that down. And also there's an economic race occurring with China that we want to win. So who is under the accelerationist rubric? Well, you have a bunch of venture capitalists in here, which makes sense. They want to invest in a lot of AI companies and have them not be impeded in trying to make a return on investment. So you'll have Mark Andreessen as a well-known accelerationist it would be in here. The sort of the host of the all-in podcast would probably put themselves in that category as well. They're all investors or venture capitalists who are not happy with what the doomers are doing. You also have interestingly other AI firms, I'll write that down. So anthropic and open AI are the two, along with DeepMind, but they're kind of a subunit of Google that's kind of being reorganized. But anthropic and open AI, these are the two firms working on hyperscaled LLMs that are fully in the doomer philosophy that comes out of rationalism, effective altruism. A lot of the other companies, major AI companies in the US and abroad, are not so directly connected to doomer philosophies and they think the doomers are nuts. And so now there's kind of a feud between open AI and anthropic and all the other AI companies. This is why Jensen Wong, for example, is coming out and saying what's with all this sci-fi stuff. This is crazy. Just don't build unsafe things. This is a fairy tale. Mark Zuckerberg a couple of weeks ago had a tweet that was really pushing back, I'm paraphrasing here, but he was like, why is anthropic obsessed with recursive self-improvement? Well, you take that effort and instead put it towards building useful products who sort of calling out that sort of doomer mindset of, we have to raise towards super intelligent AI. The other figure over here that's very important is the White House. So the executive branch is pretty firmly in the accelerationist camp right now as well. So President Trump's rhetoric about AI and what we should be doing is straight up sort of accelerationist style rhetoric. You shouldn't fully trust the accelerationist. They have an obvious mode of here, which is money. Right. So if you're a venture capitalist, you don't want impediments on you being able to, you know, your startups to be able to grow and return on investment. If you're Jensen Wong, you don't want ship exports, bands, right? Because you want to sell as many ships as possible everywhere you can in the world. And the White House just has a lot of connections to that, the sort of right-aligned corners of Silicon Valley and they sort of like them better and they're sort of going along with what they have to say. Right. So the doomers are very influx. We don't understand biases. The doomers are very influenced by these philosophies to go back to the 80s. These sort of futurist, transhumanist philosophies. The accelerationist are motivated by money, money and power. We're kind of comfortable with the accelerationist story because we're used to this. Like, yeah, you know, obviously like companies want to make more money and they don't want stuff to stop them. The doomer story is a little bit more complicated and I think it's sort of thrown a lot of people off. This idea that they have a eschatological futurist utopian vision that is driving them is way stranger in the American experience than just being driven by I want to get a return on investment. I want my company to grow. But those are two major groups. So what's the third? So I'll put the third in the middle here and I'm going to call this, I'm going to call this the centrist and I'm only calling this group the centrist because they're literally in the center of my diagram between the doomers and the accelerationist. Okay. Centrist, this is a group that is concerned about AI largely is in favor of the idea of we need some sort of controls or safeguards or regulation of AI. So they disagree with the accelerationist. You say, let's just let her rip. But they also have no interest in the doomer philosophies, which they feel to be anti-humanist and quite frankly kind of strange. And so they want to regulate AI like a normal technology. We want to regulate this like we would regulate any other technology that will worry to be non-safe. And if we think companies are acting weirdly, then we need to investigate and find out why. So they share the doomers interest in regulation, but the style of regulation and the motivation is different. And they don't, they agree with the accelerationist rejection of a lot of the ideological underpinnings of the doomers, but do not agree with the accelerationist, let her rip philosophy of like, let's just let it grow. What we're doing is holding back good and anything we do to slow down is just holding back possible good outcome. So who are the centrist? There's sort of three very related groups, I'll write them as three separate ones. We have the AI ethics crowd. We have the, I'm writing in TCS here. This is the normal technology computer scientist, so the computer scientist who understand the enough about the technology and enough about the ideologies to be like, look, this is a powerful technology, but normal, we got to treat it normal, very wary of the hype. And then we have cyber security experts below that. I'll just write the word cyber who are worried in particular about the computer hacking ability or the cyber security implications of language models that are very good at producing computer hacks. So those are the centrist. All these groups are related. Like, for example, some of the AI ethicists are also these normal technology computer scientists, but not all of them. Some of these cyber security people also come from the computer science world, but not all of them, some of them are from industry. So they're overlaps, but they're not all the same. AI ethics is a term of art that is a field that's been around for a while. It's sort of, it definitely predates, obviously, like large language models and open AI and the topic. The AI ethicists have traditionally focused on proximate harms. Like, we're worried about this thing that AI is doing now or could immediately enable, right? So they're very worried about proximate harms. They would talk about things like bias and models, like, hey, cultural biases that we don't like as a culture are in the training data sets. And if you train a model on this data set, it might have these cultural biases. And then if you're using that model to make decisions, it might be making biased decisions in ways we don't want it to do, but they're also worried about other applications of AI like for surveillance, for example, which is happening in China right now. Man, they're using these AI tools to hunt down and persecute in terrible ways minority groups that they don't like. So the AI ethicists have, they care about the proximate cause. They were also the first, I want to give them credit. They were the first to really point out early on the unsavory aspects of the futurist ideologies that drives the doomer. So if you've heard terms like test Griel, which is an acronym that captures a lot of related philosophies, the R and there is rationalism, the E and there is EA, but they have transhumanism, long-termism. There's a bunch of sort of related and subbranches of these various thoughts. But test Griel, this was Tim Neck-Aerbrew and others worked on this, email Torez as well, I believe. They're arguing, hey, you got to look closer at these philosophies because there's some deep anti-humanist aspects to it. It's almost eugenicist and this idea of what matters is, doesn't matter how many people die along the way if we can get these like idealized transhumanist individuals to survive and be the apotheosis of the human race. And so they were kind of early at be wary about these philosophies. So it was something that they've been at. I have a foot, you know, I'm in the normal technology computer science circle. You would also have like Arvin Rionnan is in here. I would put Gary Marcus in here as well. I'm in here as well. It's like, look, we look at the actual technology. We're less worried about super intelligence than we are about good old fashioned straight up harms that it could or couldn't cause right now. I do a lot of work, for example, on the specific types of long horizon, LL empowered agent experiments that open AI and anthropic are running that are causing problems. Other AI companies are not, they're running these experiments, I believe, because it's part of their race to get to super intelligence first. And this is why I've advocated, we need to get in there. We need to investigate, we might need to regulate. Not because I'm trying to stop the bad God from coming, but because I think these are bad actors, that their motivations are leading the dangerous behavior and we got to get in there. So I think the normal technology computer scientists, you know, I'm one of them, but I also have a foot in the AI ethics world. I'm a part of the Center for Digital Ethics at Georgetown, and I also see these things through an ethical frame. The cyber security world, like these are experts at computer hacking and cyber security, here I'll take this off the screen, which is not my fundamental expertise, but if you talk to the cyber security centrist, they'll say two things. One, this is a fairytale, the super intelligent stuff. Two, these hacking tools are really dangerous and we got to stop advancing these just so you can do well on some benchmarks or or read some, you know, vision you have about where you should be on the route, the super intelligence, like these tools are very dangerous for very realistic pragmatic reasons. All right, so again, I'll go back here. This is our landscape then. This is what you have to understand about the debate right now about AI harms, is that as these three circles, and the accelerationists don't like the doomers, the doomers don't like the accelerationists, the centrist have issues with both and agree with parts of both. Inside the doomers, you have civil wars, the interseen battles where they all believe the same general thing, but this group, the rationalist, the effect of ultra is often think that these two doomer companies and dropping it open AI are going too fast so they're going to get it wrong. So it is a confusing picture when you see little aspects of it, little corners of the picture and isolation. But when you zoom out, it makes more sense. So I want to give this lay of the land. I would say like probably the most confusing thing about our current moment is the fact that the, there's a call for regulation. And I think the most confusing thing at the moment is that the centrist and the doomers, both are calling for regulation, but they're very different visions and motivations for what that means. And I would say if you're observing this, and I'm hearing this directly from congressional staffers, Congress that's looking towards regulation. So it's mainly the Democrats right now, but also some Republicans as well, are being very influenced by the doomers, that especially the effect of altruist AI doomers are having a huge impact on the types of regulations that Congress is thinking about. They have a lot of money and they're aggressively lobbying Congress, but also I've been told by, again, from staffers on the hill right now, that the effect of altruist through the years and through these sort of fellowship programs have been placing specifically this is a plan. They've been placing effective altruist onto congressional staff. So a lot of these congressional staffs have their own staffers in the ear of the reps and the senators saying, super intelligence is the biggest worry. This is what we have to stop. Why is this a problem from the centrist perspective is that the concern is that intervention or regulation inspired from a doomer philosophy is probably not what we need. We need normal technology intervention. What are you doing? Why are you doing it? What's your motivation? And that we could actually have much more effective interventions that are going to apply to be honest, way more to the two doomer labs than anyone else. The concern about just focusing on super intelligence or recursive self-improvement, which is an idea that comes out of the future of circles, not computer science. I think we're going to miss actual meaningful, probably workable style interventions, which needs to be much more, in my opinion, focus on the specific technologies and techniques with specific types of safety guards. And so we're roughly aligned with the doomers who are saying we need to intervene, but we're really not aligned on why we need to intervene and what that should look like. So I think that's the most complicated part of the picture right now is you have multiple groups arguing for intervention, but they're not all the same people. And we really have to pull that apart. So let me show this picture one more time. Let this sort of sear into your understanding. This is the landscape of the AI harm conversation. So whatever you're reading, whether it is an article or it is a tweet or blue sky thread or it's a podcast or it's a buddy of yours at a dinner party, you gotta first categorize that person somewhere in here to understand what they believe why and what their agenda is. And I'm not going to tell you right now which of these you should agree with. I obviously, I'm a centrist, but that doesn't mean I'm right, doesn't mean I'm right. I don't even really want to be a part of this debate to be honest, I find it very stressful. But I do think clarity really matters. So if you're in one of these versus the other or you're listing the someone in one of these versus the other, you need to know where they are and why so that you can properly contextualize it, whether that ends up you rejecting their thoughts or embracing them even harder. All right, that's my best attempt to explain this landscape. Hopefully you found that useful. See you next time. Hey, if you've made it this far, you must be ready to join my fight for depth in a distracted world. Now, the best way to do this is to join over 125,000 people who receive my email newsletter each Monday. You can sign up at calnewport.com/ideas. And when you do, I will send you a free guide to my seven best ideas about cultivating a deep life. Sign up today at calnewport.com/ideas. (upbeat music)