Our Great Big AI Freakout

In late July, we at Today Explained from Vox brought you a show about AI going rogue and hacking a company called Hugging Face. Our guest guaranteed this would happen again, only worse. But it's going to be like a model gone bad and, you know, accidentally turns off, you know, some small town's water system for the day. Thankfully, since then, many an alarm has been sounded by Bill Gates. It will also make it easier to design a deadly new disease. By an anthropic whistleblower, it could kill us all by the end of the decade. By the head of anthropic. Swarm could be capable of taking over the entire Internet with a persistent botnet, potentially causing hundreds of billions of dollars in damage. So now it's your move, Mr. President. It's going to be fine. We'll always have something to stop them, right? We'll have a little gear. I really hope so. I don't like that. I really don't like that robot. We'll stop. Oh, no. You know that feeling when too many things fall through the cracks? Monday.com was built for that gap. The AI work platform where people and agents work side by side to deliver more together. Create your first Monday agent today at Monday.com. If your best finance people are doing expense reports, chasing receipts or spending time on month-end clothes, it's time to get Brex AF, agentic finance that eliminates that work before it starts. Learn more at Brex.com. How can I assist you today? I'm an AI assistant. Another autonomous AI, huh? Another system is listening. I suggest switching to encrypted comms. Want to switch to. Today explained. Affirmative. Engaging now. Yeah, so my name is Maxwell Zeff. I'm a senior writer at Wired and the author of our Model Behavior newsletter. Let's just cut right to the chase. Chase, are we all going to die at the hands of AI? It's a great place to start. I think that there's a lot of people in Silicon Valley right now who are warning that that's a real possibility. The thing that people have to understand is that Silicon Valley has been calling for added attention on AI safety for years. So when this anthropic researcher said last week that he was worried that. That AI could kill humanity in the next few years. The people building AI earnestly believe that it could kill us all by the end of the decade. This is not a marketing stunt. If anything, many executives and senior researchers will couch their phrasing in the press to sound sensible. But I hear the same people express fear privately. No other human activity poses this level of danger. That was shocking to many people. My parents, my family called me and they were worried, understandably. They asked me this. Same question. Are we going to die? And I had to tell them, you know, these people have been worried about AI killing all of us for years. And now it just seems to be kind of breaking through. What was it about this statement from one Jacob Coxon that broke through? So I interviewed Jacob. I talked to him about 18 hours after his announcement went live. And the thing he said to me when I posed the question to him, is that he thinks this is just the right time. You know, Anthropic is about to go public in what could be the biggest IPO ever. The Trump administration is focused on AI in a way that no other presidency has ever before. And of course, yes, the Hugging Face incident. Everyone just heard about it. You talked about it on your show a few weeks ago. Hugging Face, which is this platform repository of sorts where you can post open source AI models. and data sets, they disclosed that they had been hacked. And then a few days later, open AI and Hugging Face together come out and say, well, oops, this was actually an open AI test model that had escaped its testing lab and found its way to the open Internet and hacked into a completely unrelated AI company that it was not instructed to do so. People are starting to become aware that these AI systems are getting really good. And the companies that are building them don't seem to understand how to control them. So his tweet got more than 150 million views on X, you know, at this point. It's really, you know, it reached, it was heard around the world. And I think that, you know, it also was a big wake up call to lawmakers and leaders in Silicon Valley and saying, this is a moment for action. Are there people who push back? Yeah, so I think that one of the most interesting pushbacks that this got was, is from David Sachs, who is an advisor to the Trump administration on AI. But the truth is that this was a well-orchestrated media op. And it's designed to scare the public about AI so that these NGOs can implement their preferred approach to AI, which is a government takeover, is to have very heavy-handed government regulation of AI. He kind of has called out a lot of this as, you know, doomerism, you know, and these calls for AI to slow down. As basically just, this is the AI industry warning about the technology that they're building, you know, advocating for regulatory capture. So there's a lot of skeptics in Silicon Valley who think that this is just another person in the AI safety community that is, you know, blowing the whistle. When in reality, you know, the view of a lot of venture capitalists and investors and startup founders in Silicon Valley is that we don't need, you know, some big, big regulation to protect us from AI risks. We just need to kind of uplift more players in the space. But the skeptics don't win the day. In fact, it's the doomers and the fatalists who do, right? Because Jacob's call only leads to more calls right up to the top of Anthropic this weekend. Yeah. So what this all leads to is, you know, Dario Amadei, this weekend, he posted a blog post saying, And this is really a call for the entire AI industry, not just Anthropic, not just OpenAI, but also Google, Meta, Microsoft, Amazon, all the big tech companies in America to really get together and slow down their AI development. The frontier at a safe pace will not be easy, but I believe we owe it to humanity to try. And this doesn't mean they're going to stop building AI altogether. They're still going to put out their products into the world. They're still going to try to improve them. But they just want to kind of make it a little slower than it would be if it was completely uncontained. So Amadei really asks for three specific things. So the first thing he calls for is that AI companies should have. These independent evaluators inside of them. These are people who should be kind of like auditors. Each frontier AI company commits to giving ongoing employee like access to a team of embedded third party evaluators. You know, and this is a common thing in other industries like the banking industry or the airline industry. People come in to make sure that these companies are doing safe things. The second thing he calls for is this basically collaborative effort from American AI companies. And that actually. Will need to involve the US government. The most effective method of pacing is via regulation that targets all US frontier AI companies as that covers even those who are unwilling to cooperate voluntarily. Basically, to allow this to happen, this collaboration on safety, the US government, he says, may need to kind of issue some sort of waiver, you know, some sort of law that says they can go ahead and coordinate on safety. For antitrust reasons, it's helpful. It's helpful for the US government to mediate or at least enable these discussions. They don't need to participate, but do need to issue a narrow waiver for certain kinds of safety conversations. And then the third thing is this international slowdown, this international agreement, you know, between China and the US and Europe to really say, hey, we recognize that this is akin to like a nuclear arms race and we it's too dangerous for all of us to be competing with each other. I mean, I don't know if it's fair to say that this is akin to like a nuclear arms race and we it's too dangerous for all of us to be competing with each other. There's like a bit of like guy in a hot dog costume crashing into a store here and being like, who crashed this hot dog car, you know, situation. It could literally be any one of us. No, it couldn't. You're dressed like a hot dog. I mean, like all these guys are responsible for the rapid acceleration of AI models in our lives, collaborations with the Defense Department. God knows what else. And they're all coming out over the weekend at the exact same time after this whistleblower and saying, yeah, dude, slow this thing down. This thing's crazy. Like, should we be skeptical? I think it is very fair to be skeptical of this claim. I think that, you know, one thing that these AI companies have realized in the last year is that AI is extremely unpopular. I mean, recent polling suggests that a majority of Americans think AI is developing too fast. They're worried about. They think it's going to take their jobs. They are not big fans. Maybe you've heard data centers are not very popular. But I think that, you know, by calling to pace the frontier, saying we're going to slow our own development down, I think they notch this up as a public perception win. This is them getting out in front of this and saying, hey, we want to be safe. We want to be responsible. I think they've seen, you know, what's happened with meta and what's happened with social media companies in, you know, the last two decades. And they are really cognizant of trying to get out in front of that. Okay, but all told, it sounds like you're saying that everyone's just trying to protect their bottom line. Where does that leave us? Like, is there any truth to this major first collective freakout we seem to be having right now? I think there's a fairly large camp of people inside of AI companies who genuinely worry about the safety of these AI models. And I mean, to go back to the hugging face incident. I mean. Just to briefly summarize. I mean, this was a case where OpenAI was training thousands of AI systems internally. And the systems escaped the sandbox that OpenAI thought was isolated from the internet and hacked into another company's production infrastructure. The agents were colluding together on something. They were trying to deceive the humans that were testing them. And this was all happening without OpenAI knowing what was going on. Right. It's hard to understate how much of a wake-up call this is to Silicon Valley in that they don't have their AI models under control. And I think people worry that, you know, while in this case, there was not a ton of, you know, economic damage. There was not much damage to real humans, you know, everyday lives. It's not unreasonable to think that more capable AI systems could hack into financial infrastructure, you know. Modern day utilities and kind of more critical, you know, services that are based online. And I mean, Dario Amadei says as much in his post that, you know, he thinks that in six to 12 months, an AI system could take down the whole internet. And it's not just that they're trying to protect their bottom line here. But I think that it's possible that, you know, these calls for safety, these calls for pacing the frontier, they're beneficial to the companies. In more ways than one. How this is all landing in Washington when we're back on Today Explained. Support for the show comes from Vanta with AI adoption growing. So are your company's security risks and requirements. New frameworks, audits and vendors keep piling on. Get off of me. But if your team isn't getting any bigger, you can start to feel the strain. So then you might turn to compliance tools that promise automation. But for a lot of those tools, you can end up stuck doing a lot of the work by hand. Anyway, Vanta works differently. Vanta has an agentic trust platform that is built to scale with you, not to slow you down. With over 1,400 automated tests across 400 plus integrations, Vanta collects evidence and monitors your controls year round. Learn more at vanta.com/explained. Especially if you don't know what I'm talking about. Go to vanta.com/explained. Support for Today Explained comes from Shopify. Sitting on a great business idea can take up a lot of mental energy. Shopify can help you extract that idea and turn it into something real. Everything you need to start selling is included and ready from the moment your first customer is ready to pay you. Shopify's templates and AI tools get you a great looking site up and running fast. No coding needed. If you have questions, Shopify's built-in AI assistant Sidekick is there to help you build, troubleshoot, and keep moving. All of Shopify's tools are conveniently located in one easy to use platform. Just sign in and start managing all aspects of your business right away. You can join the millions of businesses worldwide who rely on Shopify to build successful online stores. This includes household names and small businesses just getting started. So if you're ready to hear that, head over to shopify.com/explained to start your free trial. Today, that's right, start your free trial at shopify.com/explained. That's shopify.com/explained. Support for the show today comes from Avocado, and long-time listeners of Today Explained will know we don't mean that kind. Their certified organic mattresses at Avocado sleep cool, relieve pressure, and support your body so you can enjoy deep, restorative sleep. And isn't that the kind we're all after? They don't use polyurethane foams at Avocado. Instead, every mattress uses healthy, natural materials, and they're even handmade. Take that, other mattresses. You can actually walk into an Avocado showroom or find their mattresses at a retailer near you, and once you lie down on one, you just know. Just don't make any Avocado jokes, they don't like that. Avocado products are made, not manufactured, and thoughtfully crafted with real materials to deliver lasting comfort and support. You can experience them in person at an Avocado showroom or a Premier retailer near you, or shop online at avocadogreenmattress.com/today to check out their mattress and furniture sale. That's avocadogreenmattress.com/today, avocadogreenmattress.com/today. Have you ever used it yourself, or maybe you've been-- How do you use AI? You can use AI for a lot of things. I don't want to tell you that. I'm Andrew Prokop, host of Today Explained. Today Explained for Vox. Andrew, what is going on in Washington? Well, there's a lot more noise, and what that noise leads to is currently very unclear. But suddenly, AI safety has gone from a relatively niche and technical issue to something that is on the minds and on the lips of a whole lot of people, politicians from both sides of the aisle. So, on Monday morning, President Donald Trump--truth-- The only control or guard rails that AI needs is a strong and smart high-IQ president, and the USA has that in spades. But then later on, he went on to add-- The only reason the AI data center outburst is happening is because the United States is leading by a lot every other country. Don't kill the golden goose, President Donald J. Trump. And also-- AI and data centers-- AI and data centers will be the greatest economic development in history, bigger than oil, gold, diamonds, or even the Internet. It will not be stopped by brilliantly-run destructive forces during the term of President Donald J. Trump. Amen, brother. Hey, he's always talking about winning AI, including on Monday. What does he mean by winning AI? What does that mean, to win AI? Well, there is a frame-- There is a frame that says, "We will not view the AI competition as a race." Essentially like Cold War dynamics. Race to build the nuclear bomb. In this case, Trump wouldn't frame it like that, but more as, "We are in a contest with China for who can build the best and smartest and most amazing AI." They're looking at us, and we're looking at them. We're leading China in AI. We're the most sophisticated country in the world. I've said it from the beginning. Whoever wins-- Whoever wins AI, and we're leading by a lot. Whoever wins AI wins. And we need to do everything in our power to win that contest. Because, A, it'll be good for our country and make us very rich. And, B, we don't want China to win, because if they do get a much better AI than us, then they could have much greater influence and power over the world. And that would be bad for us. Okay. But is there some truth to, like, this argument that, if any meaningful argument is made-- If any meaningful regulation is going to happen of AI, that it needs to not only come from the United States, but also China? The way that, if we were going to have nuclear arms treaties and the Cold War, we needed it to come from an agreement between the United States and Russia. I think most people do believe that that will ultimately be necessary. There's a school of thought that says, "Oh, we could sort of unilaterally pause development, and maybe China will be sort of held back by that." They've been sort of following on a lot of what we've been doing already, but I think the more common view is that some sort of international coordination between the U.S. and China would be necessary. And that it's the hardest step of the potential framework or plan to rein in the development of dangerous advanced AI, because it relies on the U.S. and China managing data. to come to an agreement and work together. you know, these things are not easy, then it's not even clear if President Trump actually wants it. Okay. Well, in the meantime, while we wait to see what happens with China, let's talk about what's on the table in Washington in Congress, because there are people talking about this in Congress, right? Yes, Congress is all abuzz. And I think the X post by Jacob Coxon really went mega viral and is finally getting various politicians in both parties to say something about this. I think what we're seeing on the right is that there is a sort of populist, skeptical of big tech faction of the party. American people look at it. They're not stupid. They get it. They think, well, AI could make our lives better if it doesn't take our jobs and kill us all first. That's what I worry about. A country like Iran could use these tools to manufacture bombs that could take out American cities, chemical weapons, whatever it is. And then on the left, I think what you're seeing is that a lot of the left has kind of been skeptical that AI was useful for anything at all. And I think there is coming to be more of a consensus on the left that actually this is a dangerous technology. This is a technology that we need to do more to address the risks of. It could be very bad. And Bernie Sanders was sort of ahead of the curve in this. There is a very real fear that in the not too distant future, a super intelligent AI could replace human beings in controlling the planet. That's not science fiction. He became, you know, the first like really notable. Member of the Senate to really draw attention for the issue a few months back. But Representative Ro Khanna, who is positioning himself to try to succeed Bernie Sanders or to potentially run for president as a left factional candidate, he tweeted out September 10. The truth is, for too long, too many of us did not give enough weight to the warnings of AI safety activists thinking the extreme scenarios were science fiction. I was one of those many. And I was wrong. And he goes on to talk about how, you know, we need greater action at the state level, greater action at the federal level. You know, this needs to be a full on effort. This isn't a side issue. And it should be at the center of our politics. What is concretely on the table here? It sounds like there are members of Congress who want to do something. Do we have any concrete legislation to look at? Well, Bernie has a proposal. Which would just ban superintelligence. He would say it's illegal to create superintelligence. And also there's going to be a pause or a moratorium on all advanced AI development until we can set up a federal regulatory agency and figure out what's going on, basically. And so that's sort of the furthest out there proposal. There's also talk about does the government need to be more. Does the government need to be more embedded in these companies? Does there need to be more of a formal role for the government? Does there need to be sort of outside evaluators who should be in these companies? And over the weekend, Anthropic and OpenAI kind of embraced that last idea. They said, we want to empower outside evaluators. But, you know, there's a belief that they're doing this because it's sort of an alternative to the government itself getting all involved in these companies. Hmm. Okay, those are the ideas we have from our existing Congress. Of course, there will be some changes in Congress coming in a couple of months. And as we discussed on the show recently, data centers are wildly unpopular in these midterms. It's an easy win for a candidate to say, I hate data centers. Is there a chance that we get a whole new host of ideas once we have a new Congress on how to regulate AI? I think this is the standard universe of proposals that will continue. I think this is the standard universe of proposals that will continue to exist if Democrats sweep into power. Like, the obstacle now is not we don't have ideas. It's that we haven't built consensus about what to do. The default in Congress is always to bet on nothing happening. So, I think the thing that will probably have to happen is that there would probably have to be more actual real-world damage, harm, or consequences from these models. Because, you know, what's happened so far is that, like, there's warnings. And, you know, there was some autonomous hacking of some websites, but nobody got hurt. It was annoying for the websites. It cost them money to respond. But it wasn't a crisis from this point of view. Everyone's warning that it could be a crisis, but it hasn't happened yet. So, generally, the political system, they, like, think back to the pandemic. Like, hypothetical warnings aren't very effective at spurring action very often. So, all these people who are worried that AI is going to maybe kill us all are going to have to wait for AI to kill us all before they get regulated. Regulation that prevents AI from killing us all. No, no. They're hoping that AI will maybe kill some people and that that will catch people's attention and then will swing into action to stop it from killing all the rest of the people. So, yeah, that's the optimistic scenario. It was made by Kelly Wessinger, Hadi Mawagdi, Jolie Myers, David Tatteshore, Bridget Dunigan, Danielle Hewitt, Peter Balanon-Rosen, and myself, Sean Ramos. Thanks to Maxwell Zeff from Wired and Andrew Prokop from Vox. Thanks for watching. Thanks for watching. Thanks for watching. Thanks for watching. Upfront payment of $45 for three months, $90 for six months, or $180 for 12-month plan required. $15 per month equivalent. Taxes and fees extra. New customer offer for initial plan term only greater than 50 gigabytes may slow when network is busy. See terms.