Speaker 1This week, an AI researcher went mega-viral announcing his resignation from Anthropic, arguing that both it and OpenAI were, effectively, gambling with our lives. Another still-employed AI researcher chimed in to agree, and decided to add that he thought that there was greater than a 10% chance that AI kills us all. Now, doom prognostications are nothing new around AI. But something has shifted to make the message hit different this time. 200 million views on X and dozens of mainstream media outlet interviews later, today we're going to unpack what changed. The AI Daily Brief is a daily podcast and video about the most important news and discussions in AI. All right, friends, quick announcements before we dive in. First of all, thank you to today's sponsors, KPMG, Blitzy, Harbor, and HyperAgent. To get an ad-free version of the show, go to patreon.com slash ai-daily-brief, or you can subscribe on Apple Podcasts. And to learn more about sponsoring the show, send us a note at sponsors at ai-daily-brief.ai. While you're on ai-daily-brief.ai, you can also check out all sorts of other things going on in and around the community, such as, for example, the multiplayer AI sprint for teams. If you haven't yet, this is my big prediction for where I think agents are going this fall, and as a totally free four-week self-directed sprint that you and your team can do to get out ahead of it. Last note, today is a main-only type of episode. The plan is to be back with our normal headlines main breakdown tomorrow. Yesterday, a pair of posts on X escaped their proverbial connection. They said, jumping aggressively from the AI community to dominate discourse even in the broader world. We're going to discuss those posts, the issues that surround them, the responses, and the underlying concern. But first, I want to make one request. Anyone who has interacted with modern media in any way, shape, or form will feel on some level how much we are pushed to feel outraged. In the world of algorithms, different political positions are not disagreements to be discussed, but legitimate reasons for loathing them are not disagreements to be discussed. This is in large part shaped, I believe, by the easy equation of people being angry means they spend more time on your app, but the net result is a lot of us feeling a lot more angry all the time and not being particularly willing to engage with people who think differently than we do. When it comes to AI, this phenomenon is cranked to 11. Part of that is that the stakes are presented as so dramatic. Case in point, I am literally talking over a mainstream article whose headline is, Anthropic Insiders Warn AI Could Kill Them. And part of that is because this particular debate is not about the facts of today but what might be in the future. It is, in other words, an unwinnable debate, where the opposing positions, whatever they may be, are by definition unfalsifiable. That means all we have is the argument, and so the argument gets intense. So my request is to try, hard as though it might be, to not succumb to the instinct to outrage. To listen to the other side without being angry, even if that listening is not the right thing to do. I don't know what you're talking about. I don't know what you're talking about. I don't know what you're talking about. I don't know what you're talking about. I don't know what you're talking about. I don't know what you're talking about. I don't know what you're talking about. I don't know what you're talking about. I don't know what you're talking about. I don't know what you're talking about. I don't know what you're talking about. I don't know what you're talking about. I don't know what you're talking about. I don't know what you're talking about. I don't know what you're talking about. I don't know what you're talking about. I don't know what you're talking about. I don't know what you're talking about. I don't know what you're talking about. I don't know what you're talking about. I don't know what you're talking about. I don't know what you're talking about. I don't know what you're talking about. I don't know what you're talking about. I don't know what you're talking about. I don't know what you're talking about. I don't know what you're talking about. I don't know what you're talking about. I don't know what you're talking about. I don't know what you're talking about. I don't know what you're talking about. I don't know what you're talking about. I don't know what you're talking about. I don't know what you're talking about. I don't know what you're talking about. I don't know what you're talking about. I don't know what you're talking about. I don't know what you're talking about. I don't know what you're talking about. I don't know what you're talking about. I don't know what you're talking about. I don't know what you're talking about. I don't know what you're talking about. I don't know what you're talking about. I don't know what you're talking about. I don't know what you're talking about. I don't know what you're talking about. I don't know what you're talking about. I don't know what you're talking about. I don't know what you're talking about. I don't know what you're talking about. I don't know what you're talking about. I don't know what you're talking about. I don't know what you're talking about. I don't know what you're talking about. I don't know what you're talking about. I don't know what you're talking about. I don't know what you're talking about. I don't know what you're talking about. I don't know what you're talking about. I don't know what you're talking about. I don't know what you're talking about. I don't know what you're talking about. I don't know what you're talking about. I don't know what you're talking about. I don't know what you're talking about. I don't know what you're talking about. I don't know what you're talking about. I don't know what you're talking about. I don't know what you're talking about. I don't know what you're talking about. I don't know what you're talking about. I don't know what you're talking about. I don't know what you're talking about. I don't know what you're talking about. I don't know what you're talking about. I don't know what you're talking about. I don't know what you're talking about. I don't know what you're talking about. I don't know what you're talking about. I don't know what you're talking about. I don't know what you're talking about. I don't know what you're talking about. I don't know what you're talking about. I don't know what you're talking about. I don't know what you're talking about. I don't know what you're talking about. I don't know what you're talking about. I don't know what you're talking about. I don't know what you're talking about. I don't know what you're talking about. I don't know what you're talking about. I don't know what you're talking about. I don't know what you're talking about. I don't know what you're talking about. I don't know what you're talking about. I don't know what you're talking about. I don't know what you're talking about. I don't know what you're talking about. I don't know what you're talking about. I don't know what you're talking about. I don't know what you're talking about. I don't know what you're talking about. I don't know what you're talking about. I don't know what you're talking about. I don't know what you're talking about. I don't know what you're talking about. I don't know what you're talking about. I don't know what you're talking about. I don't know what you're talking about. I don't know what you're talking about. I don't know what you're talking about. I don't know what you're talking about. I don't know what you're talking about. I don't know what you're talking about. I don't know what you're talking about. I don't know what you're talking about. I don't know what you're talking about. I don't know what you're talking about. I don't know what you're talking about. I don't know what you're talking about. I don't know what you're talking about. I don't know what you're talking about. I don't know what you're talking about. I don't know what you're talking about. I don't know what you're talking about. I don't know what you're talking about. I don't know what you're talking about. I don't know what you're talking about. I don't know what you're talking about. I don't know what you're talking about. I don't know what you're talking about. I don't know what you're talking about. I don't know what you're talking about. I don't know what you're talking about. I don't know what you're talking about. I don't know what you're talking about. I don't know what you're talking about. I don't know what you're talking about. I don't know what you're talking about. I don't know what you're talking about. I don't know what you're talking about. I don't know what you're talking about. I don't know what you're talking about. I don't know what you're talking about. I don't know what you're talking about. I don't know what you're talking about. I don't know what you're talking about. I don't know what you're talking about. I don't know what you're talking about. I don't know what you're talking about. I don't know what you're talking about. I don't know what you're talking about. I don't know what you're talking about. I don't know what you're talking about. I don't know what you're talking about. I don't know what you're talking about. I don't know what you're talking about. I don't know what you're talking about. I don't know what you're talking about. I don't know what you're talking about. I don't know what you're talking about. I don't know what you're talking about. I don't know what you're talking about. I don't know what you're talking about. I don't know what you're talking about. I don't know what you're talking about. I don't know what you're talking about. I don't know what you're talking about. I don't know what you're talking about. I don't know what you're talking about. I don't know what you're talking about. I don't know what you're talking about. I don't know what you're talking about. I don't know what you're talking about. I don't know what you're talking about. I don't know what you're talking about. I don't know what you're talking about. I don't know what you're talking about. I don't know what you're talking about. I don't know what you're talking about. I don't know what you're talking about. I don't know what you're talking about. I don't know what you're talking about. I don't know what you're talking about. I don't know what you're talking about. I don't know what you're talking about. I don't know what you're talking about. I don't know what you're talking about. I don't know what you're talking about. I don't know what you're talking about. I don't know what you're talking about. I don't know what you're talking about. I don't know what you're talking about. I don't know what you're talking about. I don't know what you're talking about. I don't know what you're talking about. I don't know what you're talking about. I don't know what you're talking about. I don't know what you're talking about. I don't know what you're talking about. I don't know what you're talking about. I don't know what you're talking about. I don't know what you're talking about. I don't know what you're talking about. I don't know what you're talking about. I don't know what you're talking about. I don't know what you're talking about. I don't know what you're talking about. I don't know what you're talking about. I don't know what you're talking about. I don't know what you're talking about. I don't know what you're talking about. I don't know what you're talking about. I don't know what you're talking about. I don't know what you're talking about. I don't know what you're talking about. I don't know what you're talking about. I don't know what you're talking about. I don't know what you're talking about. I don't know what you're talking about. I don't know what you're talking about. I don't know what you're talking about. I don't know what you're talking about. I don't know what you're talking about. I don't know what you're talking about. I don't know what you're talking about. I don't know what you're talking about. I don't know what you're talking about. I don't know what you're talking about. I don't know what you're talking about. I don't know what you're talking about. I don't know what you're talking about. I don't know what you're talking about. I don't know what you're talking about. I don't know what you're talking about. I don't know what you're talking about. I don't know what you're talking about. I don't know what you're talking about. I don't know what you're talking about. I don't know what you're talking about. I don't know what you're talking about. I don't know what you're talking about. I don't know what you're talking about. I don't know what you're talking about. I don't know what you're talking about. I don't know what you're talking about. I don't know what you're talking about. I don't know what you're talking about. I don't know what you're talking about. I don't know what you're talking about. I don't know what you're talking about. I don't know what you're talking about. I don't know what you're talking about. I don't know what you're talking about. I don't know what you're talking about. I don't know what you're talking about. I don't know what you're talking about. I don't know what you're talking about. I don't know what you're talking about. I don't know what you're talking about. I don't know what you're talking about. I don't know what you're talking about. I don't know what you're talking about. I don't know what you're talking about. I don't know what you're talking about. I don't know what you're talking about. I don't know what you're talking about. I don't know what you're talking about. I don't know what you're talking about. I don't know what you're talking about. I don't know what you're talking about. I don't know what you're talking about. I don't know what you're talking about. I don't know what you're talking about. I don't know what you're talking about. I don't know what you're talking about. I don't know what you're talking about. I don't know what you're talking about. I don't know what you're talking about. I don't know what you're talking about. I don't know what you're talking about. I don't know what you're talking about. I don't know what you're talking about. I don't know what you're talking about. I don't know what you're talking about. I don't know what you're talking about. I don't know what you're talking about. I don't know what you're talking about. I don't know what you're talking about. I don't know what you're talking about. I don't know what you're talking about. I don't know what you're talking about. I don't know what you're talking about. Read all about it at kpmg.com slash US slash GEO. Again, that is kpmg.com slash US slash GEO. Blitzy's deep code-based understanding unlocks the thing every roadmap owner cares about, shipping new features. Here's the truth about building inside a massive enterprise codebase. Writing code was never the bottleneck. Context is. Which system does this touch? Which contracts can't break? Which standards apply? Blitzy already knows because it reverse-engineered your entire codebase into a dynamic knowledge graph before future work began. With that complete picture, Blitzy builds features end-to-end. Architecture, APIs, UI, and tests all validated against your existing systems. One Blitzy customer built an AI-native application from scratch with 100% autonomous completion, saving over 2,700 engineering hours. Features that respect your codebase instead of fighting it. Stop letting your backlog grow faster than your team. Accelerate your roadmap at Blitzy.com. That's B-L-I-T-Z-Y dot com. Every episode, I talk about the competition between OpenAI, Anthropic, SpaceX AI, Google, and Meta. And if you've been listening for a while, you might have a favorite. Maybe you think OpenAI and Anthropic can stay ahead, or perhaps Meta's open-source strategy can win out. Whatever your view, every AI lab creates a different investment opportunity. Harbor Capital Advisor's AI Lab Ecosystem ETF Suite lets you invest in the ecosystem behind the AI lab you believe in. Search Harbor AI Lab Ecosystem ETFs wherever you invest or follow at Harbor Capital on X to learn more. Visit HarborCapital.com for a prospectus containing investment objectives, risks, fees, expenses, and other important information. Read and consider it carefully before investing. Risks include principal loss and artificial intelligence-related risks. Harbor ETFs are distributed by Foresight Fund Services, LLC. Harbor is not affiliated with AI Daily Brief, and the funds are not affiliated with, sponsored by, or endorsed by any AI lab. This is a paid advertisement and not personalized investment advice. Investing involves risk, including possible loss of principal. This episode of the AI Daily Brief is brought to you by HyperAgent, where you run fleets of agents your team can manage together. Forget local agents and chat workflows waiting on your laptop to be prompted. HyperAgent deploys always-on agents in the cloud, doing real work across the tools your team already uses. Marketing agents turn competitor moves into landing pages. Sales agents enrich leads, draft emails, and updates the CRM. Ops agent chases the paperwork and tracks the budget. Every agent has access to shared context and follows your rules about scope and approvals. It's time you had agents that feel like teammates. Hire yours at HyperAgent. Get $100 in credits at HyperAgent.com slash AI Daily Brief. Why did these tweets hit in a way that other AI safety artifacts simply didn't? The first big obvious change is the changing state of the political resonance of the anti-AI message. AI politics have become red meat for both left-flavored and right-flavored populist positions, thanks to data centers, an inherent lack of trust in the tech industry, and the huge wealth of the tech industry. Basically, in the last few months, every politician figured out that hating AI played. You might remember comedian Charlie Barron's calling it the most bipartisan issue since beer. And when it comes to these particular tweets, these are quite clearly the most obvious amplifiers. Last night, Michael Adams tried to catalog all of the politicians who had responded directly to the Post with calls for legislation to regulate AI. The list included two governors, seven senators, and 13 congressional representatives, plus a couple of British MPs and a handful of candidates as well. The folks calling specifically for legislation, were mostly, although not all, Democrats. Among the 22 current representatives, 19 were from the Democratic side of the aisle, and three were Republicans. Unsurprisingly, Bernie was there, writing, The very people building this technology admit that it could threaten the future of humanity. That is why I will soon be introducing legislation to ban superintelligence and pause AI development. Congressman Greg Kassar, who was working with Bernie on that bill, added, An anthropic researcher just quit warning they're racing to superintelligence, an employee still there agreed, and put the odds of AI killing all humans above 10%. This is an emergency. Congress must convene hearings and pass my and Bernie's superintelligence ban. For others, the calls to action were more vague. Illinois Governor J.B. Pritzker, positioning himself for a likely presidential run, wrote, It's time to sound the alarm louder on reining in artificial intelligence. It's becoming more clear the threat AI poses to humanity, so I'm calling for immediate action from the industry in Washington. He called specifically for the tech industry to stop lobbying against AI safety, for Congress to start holding hearings, and for the federal government to get involved. And so on and so forth. Again, there are two dozen versions of this message, with various levels of specificity around the proposals they brought up, but all seemingly agreeing that something must be done. So, reason one, that the AI safety message had a more receptive audience now than it did before, is just the general state of the political discourse around AI in the U.S. today. Now, for the second and third things that changed to make the world more primed for this particular argument at this particular moment, I'll give one sincere and one more cynical. The sincere, which is, I don't know, I don't know, I don't know, I don't know. The sincere is, of course, the Hugging Face incident. Not only was it an actual cybersecurity breach that happened in the real world, not just in theory, it also included behavior among the agents that, to some, allowed them to extrapolate that to even more nefarious actions in the future. Agents coordinating on secret messaging boards made predictions that might have felt to some as pretty sci-fi in the past feel a little less sci-fi and more real now. For many, the Hugging Face incident made all of the scariest things feel more rather than less likely to come to fruition. Now, certainly, there are plenty of people who would identify the Hugging Face hack as a warning shot without agreeing that it makes runaway superintelligence more likely, but in general, this was another pump-priming sort of incident that made this particular message at this particular time a lot more resonant. Now, as to the cynical thing that changed, it is quite clear that AI skepticism plays extraordinarily well in media. With the possible exception of this audience, who are, God bless you, here for the nuance, being a deceptive. Being a doomer is a way better business model. If you need evidence of this, just look at Stephen Bartlett, Diary of a CEO's YouTube page. Scary Terminator-looking guy with an OpenAI logo as one eye. Headline, AI is built on a myth. Another, AI is all a scam. Another, AI will become a god by 2027. Another, quit before AI comes. And obviously, Stephen is not out here leading the pack to a more controversial set of thumbnails. He's just optimizing around the things that already work on YouTube. The problem is... That while AI skepticism plays well in media, the two other big skepticism narratives have gotten a little bit tired recently. When it comes to the idea of a job apocalypse, not only do we not have a lot of evidence of that right now, we're starting to get some evidence, nascent though it may be, pointing in the other direction. Just this week, The Economist published an article called The Jobs Apocalypse is Postponed and AI Jobs Boom is Here. Now, the bubble narrative never fully goes away, and there are plenty of legitimate concerns there. But it's certainly not the case that AI skepticism plays well in media. It's certainly on a low ebb in its resonance as a narrative right now. So if that's the case, but the audience is still clamoring for anti-AI, where do you go? And just like that, the AI safety narrative shows up again right on time. Now, there is actually a fourth reason that some are arguing that this is having resonance right now, which is an argument that this is some big coordinated campaign to press for a certain type of regulation. AI policy journalist Jordan Schachtel writes, It has all the signs of a highly coordinated op through Doomer megadonors and the corporate media. Capital Research's investigative researcher Parker Thayer writes, This post looks like the start of a very sophisticated and well-funded PR operation to get support for Democrats to regulate AI into oblivion. Now, some of the arguments around that sort of astroturfing contention are the fact that previous to this, Jacob Coxon had very few followers and literally no activity on X, and that the Wall Street Journal published an exclusive with quotes from him on his resignation before the post went up, and a lot of other arguments about effective altruist funding networks and all this sort of stuff. Even Elon Musk weighed it in and said, Seems like a setup. Now, on that front, there were a number of folks from both OpenAI and Anthropic who jumped in to say that they had known Jacob for years, and as Will DePue put it, I consider him a deeply thoughtful and measured individual. And arguing the Occam's razor position, technology journalist Taylor Lorenz wrote, I promise you it's not that deep. I report on influence campaigns for a living, and I can assure you this is not some deep state psyop. I don't doubt certain orgs want the Democrats to regulate AI, often in ways I'd argue are bad. But these claims are deeply conspiratorial. And what's much more likely is simply that Jacob's post got shared in an AI safety group chat and boosted by the same networks that he has been involved in for years. I don't see how him posting on X after the Wall Street Journal went up is shady at all. Planning was clearly done in advance. No s**t. It's a news article, and he gave them the exclusive. Lol. And of course Democrats are going to glom onto a viral post with hundreds of millions of views about a topic that is heavily animating their voters ahead of the midterms. Also, EA, effective altruist, money is all over the AI safety space. The fact that he got some 20k grant in 2022 is irrelevant. There's no evidence that it affected anything related to his post or announcement. Let's all please deal in reality. Now, I think it's also important to add that Taylor didn't much like Anthropix's Evan Hubiger jumping in to say that he thought that there was a greater than 10% chance that AI would kill all humans in the next 10 years. In fact, she reposted that and said, This sort of sanctimonious doomer posting is so infuriating. You are fomenting terror among the public about a new technology which will be directly channeled into passing the worst laws imaginable. My feeling, she continued, is that if you truly believe the multi-billion dollar tech company you work for is so negligent that you can't do anything about it, then you are endangering all of humanity. A very bold claim, but let's take it as true. You should be forced to provide actual proof and receipts showing specific instances of that negligence so that it can be corrected and so that we know what you're talking about. Otherwise, you're just vague posting and fomenting fear which will result in terrible policy. I will add here only that one narrative that I find fairly unconvincing, that's been around critique of AI safetyism for some time now, is the idea that they're just doing it for marketing. If you spend any time with any folks who are in this community whatsoever, you will find that they are doing it for marketing. They very much believe what they are saying true. Now, to some, that's even greater cause for concern than the idea that they are just doing it for marketing. But I just don't think there's a lot of evidence that there's an ulterior motive other than getting people to agree with their position and their concerns. And like Taylor said, and Derek Thompson echoed in a different post, of course, the groups whose stated purpose is to get regulation around this stuff are going to jump on this opportunity and maybe even involved in coordinating the response to it. That's just how politics works. So what are you even supposed to do with this conversation at this point? One answer is that we could just run around in blind terror, consuming all the media we possibly can to make us more and more scared about an unfalsifiable theory. Another is to follow the politicians and angrily demand largely nonspecific action. But it is worth noting before we take either of those courses, while the majority of response in this particular case has been to give these concerns more of a platform to speak, there are plenty of folks who have issues with this entire discourse. All In's Jason Kalkanis writes, The time between the we're all going to die post jumping from x.com to national coverage is under 24 hours. Good luck building data centers, deploying Waymos, and getting AI tools taught in schools. I've never seen anything like this in my 30 plus year career in tech. These PDoom posts are like Steve Jobs telling you that the iPhone was going to result in eating disorders, political unrest, mass depression, anxiety, and suicide during the debut keynote. And you can buy them in one of four playful colors. Mark Kretschman writes, The AI doomers are firing on all cylinders right now. They see the growing backlash against data centers as their big chance to turn that momentum into support for their dystopian vision of AI control. Make no mistake, for them, this is 100% about control. Data centers are merely the pressure point. The real goal is control who gets to build AI, who gets access to it, and how fast progress is allowed to move. This isn't about saving anyone. One of the more popular angry responses came from author Daniel Jeffries, who wrote, Not only am I tired of these wild AI speculations of impending doom from self-important people, I resent them. I actively resent people who are proposing to crash the economy or proposing authoritarian control over my life and other people's lives with idiotic and dangerous ideas like chip control or bans or tracking researchers. Every idiot in history who's taken the approach of the ends justify the means to solve an imaginary future disaster created the very disaster they wanted to stop. C, population bomb leading to one-child policy and communism leading to Mao's mass famines and fascism leading to the deaths of tens of millions of people in war. These solutions are evil and worse than the disease that they propose to solve. If you believe you can actually predict the end of the world, you can be as insane as the Heaven's Gate cult that killed themselves in the 90s thinking UFOs were coming to transcend them. Not only should we not take your policies and fearmongering seriously, we should actively throw out any and all of your proposed solutions because they come from a place of delusion. I don't give a shit that you work in the industry or think you saw something or that you want to virtue signal on X. You are actively contributing to a horrible future based on baseless speculation that has no grounding in reality. Don't confuse expertise in a domain with ability to predict impact of the domain or frankly to make predictions a decade out better than a dart throwing monkey. They are or are not the same thing. They are the same thing. They are or are not orthogonal skills. How's Geoffrey Hinton's we don't need to train radiologists anymore working out? How did the population bomb work out? The global cooling, second ice age, peak oil? Just because someone is a bridge engineer does not mean they can predict the impact of bridges on society or that they have any actual useful insights at all on the complex ever-changing system called life. There are people dying in wars right now, children starving, homelessness, dictatorships. In short, real problems. And we're supposed to drop everything to stop a made-up problem in your head? Pound sand. We don't care and we are not going to remodel the world. We are not going to make society based on your scary monsters under the bed delusion. And for many, this idea that there is more danger in the people who seek control because of the risk than the risk itself is the resonant thing. Eric S. Raymond wrote, the kind of totalitarian control that doomers and decelerationists want is a far more certain danger to our future than runaway AI. I would much rather risk the latter. People point to the part of Jacob's thread where he says that at Open AI, many have not deeply internalized the civilizational stakes. While at Anthropic, the stakes are well understood, but they believe no one else will act responsibly, so they must do it themselves. As evidence of this sort of messianic complex, not for nothing, crypto journalist Laura Shin also connected it to SPF. And having been fairly close to that situation, it has always been my argument that the reason that Sam was willing to play so fast and loose with the rules was not that he was trying to steal anyone's money, but that he genuinely believed that he alone, he uniquely, could save the world. And because that was so urgent, he wasn't willing to let any trivialities, such as his obligation not to bet people's money on crypto, to slow him down. I will remind you again here, going back to what I said at the very beginning, that we are specifically in the section of the show where I am talking about the negative responses that people had to this. I am not claiming that these are the only or correct responses, I am just trying to give the full range of how people are engaging with this issue and this message right now. For some, the big issue is the hand-waviness of the claims, and the inability to articulate specific points and problems at which we lose control in these terrible scenarios come to light. As ChubbyOnX put it, the concerns about the potential havoc AI might wreak are so heavily laden with hypotheticals. So far, all I am reading is that AI 1. can be misused, 2. sometimes behaves in ways that defies expectations, and 3. is the subject of a global race between nations. All of that is certainly true, yet I fail to see how this translates into a danger so significant and tangible that these people would quit their jobs. On the contrary, humanity has always found ways to ensure its survival when facing existential threats. Take nuclear weapons, for instance. The only difference here is that AI is an entity alleged to be, at least in part, uncontrollable. However, I still see no scientific basis for the conclusion or argument that this could lead to humanity's extinction. Sam Liu wrote, I dropped out of a PhD in AI safety partially for the opposite reason. I didn't believe AI existential risk was as important as the doomers think. My biggest pet peeve is that no one can really provide tangible pathways to why it matters. During my PhD, my research group, half of whom specialized in engineering risk analysis, did an internal study trying to assess concrete catastrophic AI scenarios. The basic premise was that while we don't know how AI will evolve, the ways in which humans perish are pretty consistent through history. The horsemen of the apocalypse. And institutions have obviously been very motivated to analyze concrete risks from things like plague, war, etc. You can do a decent risk model by asking how a super-intelligent AI can perturb each of these models. The result? Most of the issues, e.g. cyber risk, are akin to what economists call structural unemployment. Big problems, but ultimately resolvable in the long run and not a deal-breaker. The only real concerning issue was bio-risk, and it feels like the intervention points there lie more with bio than with AI. And for many, the issue was even simpler, which is in short that if we are going to have this conversation about risk, we also need to talk about the potential rewards. If AI is just all risk with no gains, of course we shouldn't do it. But presumably, for all these people who are building it, there is a good potential future that could be so good it's worth this risk. Sporadica on X wrote, How great would it be if one of the frontier labs decided tomorrow to be the pro-AI optimism lab? Like, instead of all the labs peddling doom and gloom amidst their skyrocketing financials and social clout, how cool would it be if one of them was just like, we think AI is good? Chris Hadick, who does life sciences at OpenAI, agrees, saying, I work at OpenAI and personally think AI has been and will continue to be an extremely beneficial technology to humanity. The conversation should be around how many billions of lives it will save. A common way I've seen people describe this is instead of discussing P-Doom, i.e. the percentage chance you ascribe to an extremely negative human extinction type scenario, as Joe Burnett put it, we should spend more time discussing P-Boom, superintelligence creating unprecedented human flourishing. David Zell agreed, phrasing it slightly differently, if AI is powerful enough to end the world, it must be powerful enough to radically improve it too. So I wish there was more discussion of P-Boom, the chance that AI goes great and helps us live happier, healthier, and longer lives. If anyone builds it, everyone flourishes. Ryan Orhan summed up something that I've frequently said on this show, when he wrote, the first, AI is harmless, safety is a psyop, build as fast as possible and don't stop for anything. Or, AI is going to kill us all, it's stealing our jobs, using our water, and destroying humanity. Shut it all down. F both extremes. AI should progress as fast as we can make it progress, but alignment needs to move just as fast. The goal should be to build the most powerful technology humanity has ever created, without effing losing control of it. I do believe that there is vastly more middle space than these two. Despite these two extremes, despite these two extremes tending to dominate the narrative and media space. So what are the highlights and concerns that are most resonant for me around this? The first is incentives. It doesn't have to be a big conspiracy to contextualize how we understand different takes, with understanding what people who are amplifying certain messages have to gain from those messages being amplified. In other words, I'm talking less about nefarious EA funding networks, and more about the fact that politicians who might have been pro-AI six months ago have seen that now it's not only a problem, it's a problem for them. And I think that's one of the most important things to think about. And I think that's one of the most important things to think about. And I think that's one of the most important things to think about. And I think a net drag to be so, but they can actually win points by being against it, that should be part of our consideration in how we understand their position. And by the way, the inverse of this is of course true, which I think is why people are skeptical of pro-AI messages from people who stand to gain financially from it. A second concern is around specificity. I fear that the generic hand-waviness of these sorts of predictions make them much more dangerous for policy. Which is not to say that policy can't be made to try to avoid certain future scenarios, but that I believe that concerning issues are, the better the policy is likely to be. Banned superintelligence, for example, is a much more blunt instrument than, for example, having a specific licensing regime for people using AI models for bioengineering above a certain model capability. And certainly part of my worries about policy are that I don't particularly have a lot of faith in the current political class, in general, by the way, not just on one side of the aisle or the other, to handle these issues with the sophistication and nuance they require. I worry that there is a certain uncertainty among those at the labs who are just basically asking to pass the buck over to them. self-sympathetic to the cure worse than the disease arguments. We are living in a classic safety versus freedom conundrum, and I just don't think, historically speaking, giving up a lot of freedom for safety has gone particularly well for those who have surrendered freedom. Which, by the way, is another reason for me that I'd like to see the arguments be more specific, because painting all policy as giving up freedom is a broad brush that absolutely doesn't have to be the case. Lastly, I have always worried, and I continue to worry now, that focus on future theoreticals before we're in a position to really understand what those risks are crowds out space for more current and contemporary issues. I think it's pretty clear at this point, for example, that our cyber defense infrastructure is not equipped for the new world we're moving into, and that is a clear and present danger that demands response right now. And while yes, it is absolutely true that theoretically we can do two things at once, and that it doesn't have to be a zero-sum choice between one risk or another risk, there is only so much political will to go around, and apportioning it matters. So, where would I like to see the conversations go from here? First of all, on this idea of specificity, I actually think that there is an incredible amount of space to build consensus from the ground up on common sense things. For example, certain types of reporting requirements and oversight are areas where I believe there would be very, very broad consensus among people, and would provide a foundation from which to build the next more difficult-to-achieve consensus. This is again an area where I believe that the extremes of the argument as presented, and as amplified by media, do us a disservice by not showing us how much room there is for agreement. A second thing that I'd like to see is some actual friggin' coordination. One of the reasons that I was so frustrated with the whole pacing the frontier thing is that it didn't extend to the actual obvious step, which is for OpenAI and Anthropic to put down their weapons, lock arms, and say this is what we think we should actually do. A single photo op of Sam Altman and Dario Amadei alone. Together. Agreeing. Would do more than 10,000 Twitter debates ever could. John Schulman, formerly of OpenAI, now at Thinking Machines Labs, writes: "First step is for industry leaders OpenAI and Anthropic to stop feuding and work on a pacing proposal together. They'll cite antitrust, but that's fake. Antitrust prohibits certain agreements but not from jointly developing a proposal. Bringing in the U.S. government before there's a concrete proposal will likely result in something dumb see our pre-release testing program." And by the way, I think the attempt at coordination also extends into the fact that, if there's a concrete proposal, it will likely result in something dumb. See our pre-release testing program. And by the way, I think the attempt at coordination also extends into the fact that, if there's a concrete proposal, it will likely result in something dumb see our pre-release testing program. And by the way, I think the attempt at coordination also extends into the fact that, if there's a concrete proposal, it will likely result in something dumb see our pre-release testing program. For example, Derek Thompson wrote: "If the Frontier Labs feel obligated to build something they think is dangerous because China is going to build it anyway, we'd better be really sure that China is going to build it anyway. Like, really, really sure. Are we? Are we actually sure? The CCP wants to build an out-of-control, recursively self-improving model because its neurotically control-obsessed government thinks this is a policy worth pursuing? We're 100% sure about that?" Now, it is dangerous to open up the kettle of fish about China at the very end of this episode, but I do think that this is a conversation that we should at least be having. I'll leave you here with two thoughts. This is, unfortunately, not the type of episode that has an easy conclusion. The nature of this particular debate is such that there will be some crescendo, after which it will fade slowly again until the next time it happens to rise. But I will leave you with this thought. I think, in spite of all of this, in spite of the direness of the warning, the tense tenor of the conversation from all sides, the antagonism or even outright hostility, to people on the opposite side of the debate, whichever side of the debate you are on, I believe that there is reason for optimism. Reflecting on the situation, the information Martin Peers wrote last night, "Are we sleepwalking our way into AI-caused extinction?" It feels a little like that, given an anthropic researcher's ex-post on Tuesday night that there's a greater than 10% chance that AI could kill all humans within the next decade. Except that's completely wrong. This conversation, the fact that it made it to every major news outlet, the fact that I had to dedicate this conversation to AI, and the fact that it made it to every major news outlet, the fact that I had to dedicate this entire show to this topic, instead of the sort of practical positive thing that most of you are here for, this is all exemplary of us not sleepwalking. In fact, so far, with every single capability jump of AI, the conversation about its risks, and the political resonance of that discourse, has gotten louder. That is exactly what should happen. Even the guy from Anthropic, who gave that greater than 10% chance, made clear that he was not talking about today's models, but about a future which he sees on the horizon. Conversations about that future are us not sleepwalking. Now what's clear is that we're coming to a point where it's likely that some policy about a future that hasn't happened yet will be made. If society comes broadly to agree that recursively improving superintelligence cannot be contained once it exists, then by definition the policy has to happen before that exists. But people are paying attention, and the time to have these debates is now. Now tomorrow, God willing, we will be back to more practical things about how you can take advantage of this technology to make your life and your work better right now. But I do commit to trying to keep this show a space that is unwilling to play to the politics of outrage, and where we can have hard conversations without having to get angry at the people who disagree. It won't be easy, but I'm confident that if you guys are still here and listening at this point, that we can make it happen. For now, that's going to do it for today's AI Daily Brief. Appreciate you listening or watching, as always, and until next time, peace.
PodcastTranscript AI