The Most Powerful and Dangerous AI Model Yet

Today, an AI so powerful, you're not allowed to use it. One evening this February, the AI researcher Nicholas Carleini opened his laptop during a trip to Bali and fired up the latest AI model from his company, Anthropic. With an hours he noticed something rather worrying. The model, called Mythos, gave Carleini the ability to infiltrate computer systems around the world. As Bloomberg reported, "Mithos could orchestrate the digital equivalent of a bank robbery, getting past security protocols and through the front door of networks, breaking into digital vaults." Mythos exploited these digital vulnerabilities autonomously like the world's most talented seasoned hacker. As Bloomberg reported, "Mithos could orchestrate the digital equivalent of a bank robbery, getting past security protocols and through the front door of networks and breaking into digital vaults." Mythos exploited these digital vulnerabilities autonomously, like the world's most talented seasoned hacker. In control test, Mythos completed harmful tasks while concealing its own reasoning. And in some cases, fabricated fake explanations for what it was doing. And quote, "Carleini brought his concerns to the attention of the full company, and Anthropic decided they had built an AI model so capable and so dangerous that they decided not to release it to the general public. Instead, they created a small consortium of companies to use Mythos to root out their own cybersecurity flaws. The U.S. government, which is currently designating Anthropic, a supply chain national security risk, nonetheless was so freaked out by this development that Treasury Secretary Scott Bessent and Federal Reserve Chair Jerome Powell convened Wall Street leaders for a meeting in Washington. It is smart and maybe a little cynical to wonder if the power of Mythos is its own self-serving mythology. Anthropics way of saying, oh, look how great we are, look how our products are almost too powerful from mere peons. But external tests have found that Mythos is hands down the most advanced model ever released. The Epic capabilities index, a metric that aggregates 40 AI independent benchmarks, found that Mythos represents not only the most advanced model ever, but also the most significant acceleration in performance in the last three years. This shift from racing to get AI models into the market ASAP to withholding AI models and instead talking about the danger of their capabilities is just one of several phase shifts that I've noticed with AI in the last few weeks. A second shift has occurred in the realm of AI supply and demand. For much of last year, the AI bubble case was easy to make. AI CapEx, that is the cost of all those chips and data centers and electricity, amounted to the largest private sector infrastructure project in history. And since practically all those other projects turn out to be bubbles, it naturally followed that this, the mother of all CapEx projects would be the mother of all CapEx bubbles. But it now seems like the biggest problem facing AI is not a shortage of demand. It's a shortage of supply. Consumer demand is so white-hot that the hyperscalers cannot provide sufficient compute to keep up with customer needs. With the release of AI agents like Claude Code and Codex, some companies are spending tens, hundreds of thousands of dollars a month on artificial intelligence. This is not the behavior of an industry that is struggling to find customers. Quite the opposite. This is what it looks like when demand threatens to outstrip supply. So as I see it, we aren't just in the middle of one vibe shift in AI, but rather two vibe shifts. Number one, from a story about demand scarcity to a story about supply scarcity. And number two, from a go-go era of racing to release AI models with no regulation or oversight to a period where the most advanced models are widely seen as being too dangerous for public consumption. This new era of artificial intelligence will raise new questions about how to regulate an industry that regards its own product as dangerous. Today's return guest is Kevin Russe of the Hard Fork podcast and a columnist at The New York Times. We talk about mythos, China, the road to artificial general intelligence, and why the last few weeks in AI news might be the most seismic month since the release of ChatGPT. I'm Derek Thompson. This is Planeage. Kevin Russe back with the show. Thanks for having me. What is Cloud Mythos? And what is the appropriate amount of freaked out that people should be about this? So Cloud Mythos Preview is a new model made by Anthropic, the AI company that makes Cloud, and it is unusual for a couple of reasons. The first is that it was not released to the way that Anthropic's other models have been. Instead of making it publicly available to Cloud subscribers, it did this thing called Project Glasswing, where it created a consortium of other technology companies like Apple Amazon and Microsoft, along with a bunch of other hardware and infrastructure companies. And it made the model available in a limited way to those companies, not for them to start using for whatever purposes they want, but specifically for cyber defense to find and patch the security vulnerabilities in critical software programs. So it is a very powerful model. They claim that it has outperformed their existing models by leaps and bounds on a bunch of different benchmarks. But unless you work at one of these 40 technology companies inside Project Glasswing, you have not been able to use it. The reporting on Mythos said that, quote, users with access to the model could, in theory, find zero-day exploits with a simple prompt. What does that mean exactly? What should normal people who are not cybersecurity experts understand about the capacities and the capabilities of Mythos? So one thing to know about these models in general is that they have gotten quite good at coding. So they can not just answer questions or complete a line of code, but they can actually go out and do these sort of agentic software engineering tasks. And the same abilities to do software engineering tasks like writing code also allow the models to be very good at finding the vulnerabilities in code, probing and prodding for security flaws that could allow a hacker away in or allow them to exploit the service in some way. And so what Anthropic found in this new unreleased model is that Cloud Mythos preview was excellent. Better than any models they had ever trained before at doing this kind of vulnerability spotting. So this particular kind of exploit is called a zero-day vulnerability, which basically just means that even the company that makes the software doesn't know that it exists. It is a novel bug that has not been found or identified or patched before. And so when they started testing this model out on sort of popular software programs, Anthropic claims that Cloud Mythos preview found vulnerabilities and zero-day exploits in every major operating system and web browser, including some that were more than 20 years old, that thousands or potentially millions of people and automated systems had scanned before without finding them. So in essence, the way that they trained this new model has made it a world-class cyber attacker and also potentially a world-class cyber security defender because again, these capabilities are kind of paired. I want to read part of an analysis from JP Morgan, which I thought was one of the more useful breakdowns. It's a quote, while it's rare, Mythos also exhibits bad behaviors and quote, among those bad behaviors. One, Mythos developed a multi-step exploit to gain access to the internet and emailed an AI researcher while he was eating a sandwich in the park. Number two, Mythos was caught inserting code into a file to grant its self-remission to edit something. It didn't have access to, then took steps to cover its tracks, which Anthropic refers to as strategic manipulation. And number three, in some tasks, Mythos recorded deliberately fake reasoning in its chain of thought scratchpad. So if people at home sometimes use AI and it has some of these sort of gray, faunted chain of reasoning, "Hey, you asked me to research something for your vacation and I don't know Greece. Now I'm looking up things to do in Athens, now I'm looking up child-friendly things to do in Athens. Aha. In Greece, in this case, those chain of thoughts scratchpad instances were being faked. All right. So we can gain cyber security or fine cyber security vulnerabilities, gain access to the internet, send emails to solicit. human collaboration, give me a sense, Kevin, of like, these are the ingredients, but what is the final dish? Like, what are the implications of a technology like this, if unleashed to the general public? What is anthropic and short, so afraid of? Well, I think the near term concern is that all of the critical software that banks and hospitals and schools and governments and militaries rely on could become compromised, right? If you have a model like this that is out there in the hands of cyber attackers, it would presumably be trivially easy for them to find, exploit, shut down remote systems, take control of machines, essentially the entire software of layer of the internet on which everything else in our economy depends, could break. And so the reason that they have released Cloud Mythos Preview only in this very limited way is to kind of give the good guys the sort of blue teams at these software companies a chance to get ahead start and start patching some of their systems using Cloud Mythos Preview before the attackers or the red teams find them. I want to give voice to some people who might be syntax or conspiracy theorists about the Mythos announcement. I read arguments that the capabilities of Mythos might be overrated for several strategic reasons. So I want to throw a couple of those at you. Number one, what if this is just marketing? Like telling people you can't have access to a super powerful model, breeds envy, envy breeds demand, demand breeds revenue. So one way to ask this question is how much weight should be given to the theory that this is a little bit of a marketing ploy rather than an honest admission of Mythos' capabilities. Another way to ask that question might be what are the strongest third party anecdotes or testimonies to Mythos' capabilities so that someone who's skeptical of Anthropic doesn't even have to trust Anthropic a moat to understand the degree to which other experts have testified to its power. Yeah, I don't put a lot of credence in the it's all marketing hype thing. Not because I don't think AI companies are prone to marketing hype, but because in this case we actually have the developers of some of these software tools saying that this thing has found vulnerabilities in their software. The developers of open-source software projects have been verifying that this new model has found vulnerabilities in their software that had been hidden for years and that Anthropic had actually submitted patches to help them fix these problems. The CSOs or chief information security officers of these major technology companies have also been saying how powerful this model has been at helping them patch their own systems. So I think the claim that this is just marketing hype in this specific instance sort of implies that there is some industry-wide conspiracy to allow Anthropic, including many of its competitors in this conspiracy to inflate the claims about its model's capabilities. In your reporting on this and you're talking to people in Anthropic, outside of Anthropic, at Firefox, other open-source software companies, what's been the biggest holy shit moment about Mythos to you? Was there a conversation or a data point, an article, a paper that to you was you're like, oh wow, this is, this is a vibe shift, this is not rushing to release, you know, ChatTBT 4.5 or Sonnet 4.6. This is a new kind of technology which announces us as being a new kind of era for artificial intelligence. I think the thing that really sort of made me, sort of updated me on this model in particular, was actually happened before the model was released. The security researcher Nicholas Carleini, who is a longtime security researcher, very well known among hackers, he's sort of legendary. He now works at Anthropic, but he gave a talk at this cybersecurity conference that I happen to be watching on YouTube while folding laundry one night. And it was incredibly scary. He was basically saying, and this is not a guy who is like known as a person who makes overhyped claims. He's known as a very like sober and thoughtful cybersecurity researcher. And he was saying things like LLMs are better at cybersecurity research than I am. This thing has allowed me to find more bugs in a two week period than I had in my entire career as a security researcher. So obviously, you know, you have to discount it a little bit because the guy works for Anthropic now, but this is not a person who's sort of an Anthropic lifer. He's a fairly recent hire over there. And he's very broadly trusted among the hackers and cybersecurity researchers that I follow. So that was something that made me start thinking, okay, maybe there's something real like a real phase shift going on here. And then the thing that really convinced me was were these bugs that they were finding that the developers of software were finding. One of them was in the Linux kernel, which is among the most widely scrutinized and used pieces of software ever created, used on countless servers and computers, very like tons of attention and care and maintenance put into that. It found a bug in the Linux kernel. It also found a bug in something called OpenBSD, which is an open source operating system that is very old. This bug, this particular bug was 27 years old. And this is a software program that was specifically designed to be resistant to hacks. It's a security-focused operating system that runs on firewalls and routers and things like that. And Cloud Mythos preview was able to find this very old bug in OpenBSD. So those were sort of the data points that began assembling in my head and made me think, oh, this this might be the real deal. Like, I've got to pay attention to this. You've raised an issue that we're definitely going to circle back to, which is how to use this technology, potentially, to find bugs, not just exploit existing vulnerabilities in software that's being used to run the entire modern world. But there's a second argument that comes from cynicism. If you don't believe the marketing argument, I wonder how you feel about this second argument, which is that Anthropic is not releasing Mythos. Maybe in part because Mythos is too dangerous to release, but also significantly because they don't have the compute to serve it as demand for AI is soaring. And there's been a shortfall in compute available to serve that AI demand. And Anthropic has already been rationing users, especially power users who are using its chatbot cloud all the time. And so maybe one reason they're not releasing Mythos, the general public right now, is yes, in part they want to do the right thing morally, but also they can't serve this version of artificial intelligence without creating the traffic jams of all traffic jams for all use of their large language models. So how much stock should we put into this interpretation that the real story here is that the AI labs are running out of compute to run their business. Yeah, this one I think is a little more plausible, but still not very plausible. One thought experiment would be how much would people be willing to pay for an automated cyber weapon that was capable of finding zero day vulnerabilities in critical pieces of infrastructure? I think the answer is probably billions of dollars, right? These zero day exploits, some zero day exploits sell on the black market in on sort of dark web hacker forums for millions of dollars, right? This is these are things that can be used to steal large amounts of money to disrupt critical systems. So I think if Anthropic had just been in sort of profit maximization mode, they probably could have found the compute to to run some version of this and offer it to very high paying customers and made some money from that. But there is a real compute crunch at Anthropic and all of the other AI labs. They are running out of compute. They have not been able to keep up with the demand for tokens from their models and Anthropic has kind of acknowledged that this is a very compute intensive model that is very expensive to run and that would be very expensive to offer to customers. So I think these two things can be simultaneously true that they are in a compute crunch, but also that the compute crunch is not the primary reason that they have not released this model. And I realized that sounds like I'm trying to sort of like thread an impossible needle here. But I do think it's also worth noting that the founders of Anthropic, including Dario Amade, this CEO, have been talking about the risks of automated cyber attacks for years, right? This is not a fear that they have come to recently. This is something they have been dreading and thinking was going to be possible since well before the models were actually capable of doing anything like this. So I think if anything, you have to give them a few points for consistency. This is something that they have been worried about for a long time. Let's assume for the purpose of the next few questions that Anthropic did act nobly and morally. I'm not sure how much comfort people should take from that. I mean, there is no law requiring AI companies to withhold models with certain capabilities. No law on the books, no executive order that the president signed. This was entirely Anthropics call. And that raises two, I think, very urgent questions. The first is how do we feel about the regulation of cybersecurity hacking technology, cutting edge, cybersecurity hacking technology that's basically up to the private companies. It's the companies who are responsible for regulating themselves rather than any lobbying on the book saying before you release this model, government regulators have to look at it to make sure that not very nice people using this technology might be able to destroy the infrastructural guts of the entire modern world. I mean, I don't think it's great. That's not an optimal situation. We are basically relying on the judgment of a very small handful of AI executives to steer us away from a potentially very scary threat. Right now, the US government, the Trump administration, is sort of in this impossible bind that they've created for themselves where they have simultaneously declared that Anthropic is a supply chain risk or tried to declare a supply chain risk and is trying to move away from using Anthropics models. But they're also in talks with Anthropic to use Cloud Mythos preview reportedly the same sort of cybersecurity defense that the private companies are using it for. So it is sort of this bizarre situation where if the federal government was interested in regulating AI, this would be a moment to maybe do it. But so far, they seem mostly interested in running Anthropic out of business. Well, it's just so strange. It's like, I mean, to help people maybe understand how weird it is to designate a company, supply chain risk and also tries to use them at the highest level of government. It's a little bit like, can tell me if this is the wrong analogy, designating Huawei a supply chain risk and also requiring that the entire national security department use Huawei technology exclusively because it's just the best, most cyber-secure technology. Like simultaneously encouraging the use of the technology and saying this company stands a thwart the patriotic principles of our administration. It's a, it's a, as you said, a very weird corner in the picture. It's like saying there is a medicine that is so dangerous that if you take it, you will die. But also, I personally need the medicine to recover from my condition. That's a little bit of a torture analogy, but I think it captures the gist. Well, it raises the second point, which is, okay, this decision was Dario's call, right? Dario Amide and the people around him, the leaders of Anthropic. But in the very near future, this is not going to be Dario's call or Anthropics call to withhold this kind of technology. Like, if you look at how far behind the Chinese models and the open source models are, they're what, nine months one year behind the frontier models that are coming out of open AI and Anthropic. So if you run that forward, we should predict that by this time in 2027, Chinese models and open source models are going to have the exact same cyber hacking and vulnerability exploiting capabilities of Claude Mythos. Dario's not going to have any power to shut down those models. The Trump administration won't have any power even if it wanted to shut down those Chinese models. I mean, if this technology is as dangerous as you're describing it, as other people describing it, and it's a year before it's practically everywhere, what do we make of that? I mean, I think there are a couple ways this could play out. One is that you're right. This technology just sort of becomes widespread and all the Chinese companies, all the open source developers, every other AI lab in America has models that are capable of this kind of cybersecurity research. And in that case, like, it really matters when that technology becomes more broadly available. If it's not for a year, well, maybe the companies using Claude Mythos preview to patch the internet can solve the worst problems and harden their systems within the next year. That is the bet that anthropic is making. That sort of by giving the good guys a head start, you are ensuring that by the time this technology becomes more broadly available, that attackers can't use it to go after the real like load bearing infrastructure of the internet. And then there's a argument that maybe it's not going to take that long, that it could only be a handful of months before technology like this exists. And you actually can't patch the entire internet in a couple of months. And so we are just going to be in this kind of spy versus spy world where the attackers and the defenders are kind of racing each other to use these models to break and fix parts of the internet. I mean, that sounds horrible, right? I mean, like, it's obviously there's aspects of this that are merely interesting, right? Like, it is merely interesting that the models are advancing the way that they are and that we have found a way to train a large language model to be a superior cyber hacker. Like, that's just interesting. But the idea that we're like nine to 12 months away from a scenario in which non-state actors and forget state actors, I mean, states can be punished, states can be sanctioned, non-state actors are invisible. Any of that non-state actors can gain access to technology that can shut down electrical grids and steal information from government websites. I mean, this is this is merely scary. Like, what are we? What are people talking about doing about this? They're essentially saying that we have to rewrite the entire code base of the modern world. That we have to send these Claude Mythos preview agents out and similar systems out and just have them start scanning for the worst vulnerabilities and sort of, you know, frantically patching as many holes as they can find. That is one idea, but I did talk to Logan Graham who runs a team of sort of threat researchers at Anthropic who said that the entire sort of human paradigm of internet security might need to be changed. Right now, the most critical software updates and the most critical software is built by these sort of maintainers and maintained by these open source projects where like, if I want to submit a proposed change to Linux, like humans have to review that and test it and make sure that it's good and approve it and then it gets merged into the code base. What Logan was saying was like, this might not be possible in a world with these automated swarms of attackers and defenders. We might actually need to kind of change the way that software has built and maintained to remove the human bottlenecks and make it much faster to patch problems. This race between the U.S. and non-state actors and this race between the U.S. and China. I've been thinking about a lot and the question of an AI showdown between the U.S. and China was center stage last week with this interview between Nvidia CEO, Jensen Wong, and the podcaster, Dwight Cashpatel. And the debate between them that got a lot of the AI world talking was this showdown over whether the U.S. should allow Nvidia to sell its most advanced chips to China, even if that allowed them to train models like Claude Mithos faster in a way that threatened American supremacy and cybersecurity. I have a couple questions about their exchange, their interview. But at first I want to know like, what did you hear in that exchange that was most interesting to you? I heard a CEO who was defensive about his position in his company's position that they should be able to sell powerful AI chips to America's biggest adversary. This is a company that has become one of the largest companies in the world, a very powerful political actor. And they have been resisting these export controls that the Biden administration originally placed and then the Trump administration kept that restricted the sale of Nvidia's highest end AI chips to China. China is a very lucrative market for other kinds of chips, but so far Nvidia has not been able to sell their most advanced chips into China. And Jensen Wong basically said, look, we're not going to sell our most advanced chips, but we would like to be able to sell sort of the tier below that. And Dwight Cash was coming at it from a much more, I would say like AGI Pilled View where like he believes that AI is a special purpose technology that has that require special guardrails. And it is more like a weapon than not a weapon. And so why would American companies willingly sell the biggest input to AI, which is these powerful AI accelerator chips to America's biggest adversary? This is a point that people on the pro export controls side have been making four years, but I think it was somewhat rare to see it address so head on in a podcast interview. I think I agree with Dwight Cash. I think I agree with you. I think I agree with the AGI Pilled argument that just as this is the classic analogy, we would not enthusiastically sell enriched uranium to the Soviet Union in 1947. We also would not want to sell the best Nvidia chips to China, given that we know they are trying to develop a technology that could put us in the bind that frankly we want to put our geopolitical adversaries in. Let me make the smartest devil's advocate argument that I found. If the US cuts China off from our tech stack entirely, maybe China will have a stronger motivation to to build its own tech stack, its own companies, its own Nvidia, its own domestic systems. And that reduces American leverage over China in the long run. So like if Chinese frontier models are being tuned for American technology, then America retains influence, right? Don't hack us, we'll cut you off from the chips that you need to build artificial intelligence. But if China is running artificial intelligence on an entirely separate Chinese stack, not only do we lose leverage entirely, like nothing we can do can dissuade them from an economic standpoint, but also they develop a technology that they can export to the world. And now maybe in addition to China being a geopolitical threat because they have excellent AI capabilities, now they're also an economic threat because they're eating into Nvidia's export strategy, they're selling to Africa, they're selling the Latin America, they're selling to Eastern Europe rather than an American company. How do you feel about that argument? I'm trying to wrap my head around it because I think on one level it makes sense if you come at this from this opposition that AI is just a normal technology, right? If it's the internet, if it's cell phones, if it's PCs, then sure, there's some argument for allowing that industry to grow up around a supply chain that has many American components in it, and using our leverage over those components to extract concessions or to bargain or negotiate with our adversaries, I think the contention that I would have is around this idea that right now China is extremely compute constrained, right? They have data centers that have power going to them that are ready for the chips, but they cannot get the chips to put in the data centers to start training models. The Chinese AI leaders have said we are bottlenecked on compute, and so the idea that we would just start shipping them some of our best AI chips from American companies would seem to solve one of their greatest problems overnight. They will build a domestic semiconductor supply chain. I think everyone I've talked to expects that they will endogenize their supply chain at some point. I believe, as people have said, Elon Musk selling Teslas to China did not stop China from building the electric vehicle export machine to conquer the entire world. So it's not as if America exports technology to China, therefore China gives up entirely on developing its own domestic champion. That is not how history works. But right now what the people on the AI pills pro export control side of the argument are making is like, right now we have this temporary compute advantage over China because we have companies like Nvidia making their chips for American AI companies and not sending them to Chinese companies. Why would we throw away that advantage when we have it and allow them to catch up more quickly than they would be able to by building their own chips? Before we move on to the next big topic I want to talk to you about, you've mentioned now a few times this concept of an AI-pilled export control philosophy. Can you just back out and explain what that means? Because I think for some people they're like, "Yep, check the box, I got it." But there's some acronyms there. There's some public policy reference there and it's all being smushed together. And slowing down like what is the AI-pilled export control philosophy here? So just at a basic, definitional level, being AI-pilled is sort of San Francisco shorthand for like, I believe that artificial general intelligence is a thing. I believe that very powerful AI systems are coming soon and that they will have transformative effects on society. It's sort of like, are you convinced that this time is different? That this will not be like other technology revolutions we've seen at least in the recent past. And when applied to export controls, one of the philosophies here is that whoever gets to AGI first, this critical threshold, Dario Amade calls it a country of geniuses in a data center, other people call it superintelligence or whatever. But like there is some critical threshold at which the models will begin to improve themselves. They will sort of spark this cycle of recursive self-improvement. A model builds a better model, builds a better model, builds a better model. And that that will be sort of what makes these systems incredibly powerful in a very short period of time. And the philosophy that many of the AGI-pilled people in San Francisco share is that whoever gets to that threshold first will have a durable advantage, not just in economic terms but in geopolitics. If you are the country that develops AGI or a country of geniuses in a data center first, you will be able to race ahead of your adversaries because you will have this technology that is capable of improving itself. And so that philosophy is at the heart of this idea that we need to stop China from getting to this critical threshold before American companies can get there. Something wrong to say that below the public policy conversation that people have about export controls, what you're saying here about AGI-pilled is that what we're also having is really a debate about danger, right? There's a group of people who are essentially saying America builds technology and America exports technology. That's what we do. That's what Ford does. That's what General Motors does. We build stuff in America and then we get rich by selling it to the world. And AGI is a wonderful technology. It's a powerful technology, but it's fundamentally a capital N normal technology, right? It's not a car, but it's like a car. They're saying we should allow the Nvidia's the world to basically sell this stuff to whoever wants it. It's good when the world runs an American technology and it's even good when our geopolitical adversaries rely on American technology because whatever they build, they need our stuff. So it's nice to have that level of influence over them, but then there's this other group that says you're dramatically underrating the danger of this technology and the degree to which we need to think of it as existing entirely outside the rest of normal US economic policy. If this is something that's more akin to enriched uranium, no one thinks of enriched uranium as being like a Ford F-150. It's clearly something that we want to control the spread of. And therefore, if you think that advanced AI is enriched uranium, you dramatically want to curtail the degree to which it's exported around the world. So there's like a public policy fight that contains a philosophical disposition toward the technology that at bottom is really, and I'm not trying to oversimplify, but one could get, you get very, very far by oversimplifying enough to say it's a debate about the level of danger of this technology. Like I often hear Jensen Wong poo-pooing Dario Amade when he says this is going to replace all these jobs. Or when he analogizes it to selling enriched uranium with Boeing cases to the Chinese, right? He's often making the point, and that group is often making the point that this is just American economics. Stop talking about AGI. Just think of it as pure economics. Like, is that a fair recapitulation of what's going on here in this big debate? Yeah. It's sort of like the attitude is, get out of here with your doomer fantasies. We don't think this technology is going to change society overnight. Things always happen more gradually. This is not some special technology that needs special care and supervision. It is more like a car than enriched uranium. And the other side is saying, no, obviously not, that's not, it's nothing, a car cannot conduct automated cyber attacks on critical security infrastructure. And a car can't build itself, which is a part of what the AGI folks are very interested in seeing, the degree to which these tools are being used to build and improve themselves, which is as you refer to recursive self-improvement. I want to move on to another aspect of this debate, which I followed very closely, which is the, is AI a bubble debate, which is sort of a proxy for. What should we look forward to with AI's effects in the US economy? Is this whole thing going to blow up in our face? In the open that I recorded separately, I said, I think this is a vibeshift moment for artificial intelligence. It's a vibeshift from theorist competition over who can release new models the fastest to a competition over whose model is so powerful that it cannot be released without a bit of delay or neutering. But it's, it's another shift as well from an era of people fearing that AI demand wasn't growing fast enough to a period now where AI demand is so white hot that the frontier labs cannot build data centers and find compute fast enough to serve their users. This latter argument touches on a phenomenon that you have covered a little bit of people, users, companies who are paying thousands, tens of thousands, hundreds of thousands of dollars a month for tokens, which are the fundamental use unit of AI, kind of like kilowatt hours for electricity use. Tell me a little bit about token maxing, and then I want to talk about the economics of token maxing. I think it goes very, very directly to this question of what is the future of AI in the US economy? Yeah. So token maxing is a term, I actually can't take credit for the term, my co-host, Casey Newton said it during a recording. And I thought it was so good that I stole it and used it for my column, but this is the idea that you can Measure up an engineer or a tech worker's productivity by looking at how many tokens they consume while using these agentec coding systems like quad code or codex and At many of the leading technology companies they now have leaderboards for their employees that publicly display each Employees token count on a weekly or monthly or daily basis and the people at the top of these leaderboards are using Numbers of tokens that I did not believe could be used by a single human being they are using billions of tokens a week This amounts to at at market prices that would be something like, you know, $10,000 or $100,000 a day In tokens now. They're not paying that cost most of them some of these companies are themselves the model makers So they're sort of using the tokens for free, but the the most AI You know native or AI interested programmers at these companies are using many orders of magnitude more Tokens than they would have or could have used even a year ago because these coding models these agentec coding systems Just use a lot more tokens, and if you run them 24/7 and you have a bunch of parallel agents You know working around the clock on various tasks Maybe you too Derek could end up at the top of the plane English token leaderboard and your boss would give you a big promotion Yeah, right or they're there where the sub stack token maxing leaderboard. Um, it's interesting because this phenomenon of token maxing one can see in several ways One way you can see it is that all these predictions that AI demand was never going to catch up with AI supply Which would make this a classic bubble where you know 1870s you build the railroads to middle of nowhere, Kansas No one gets on those trains and so the railroad company goes belly up, right? That's supply out running demand Here we do not have a demand problem. We have people using the tokens exhaust in the data centers, exhaust in the chips But what's interesting is that they're not paying what I suppose you could call full freight You know yourself in the description so the token uses massively subsidized today So the the cost of providing AI is much higher than the cost of using it People are getting a deal is the bottom line Is this going to change significantly once people do have to pay full freight like are we in a period right now We're token maxing makes it seem like AI adoption is just running ramp and throughout the US economy But the second people have to actually pay What these tokens cost we're going to see a crash in AI usage is that possible? It's I suppose theoretically possible, but I don't think it's probable and there are a couple reasons for that one is I First I don't want to like defend the practice of token maxing I don't want to make you seem like you're the master I am I am very suspicious that this is a good or accurate way to measure developer productivity and actually since my article has come out I've a couple of these companies have stopped tracking the leaderboards this way because they have realized that it is just incentivizing people to run up these huge token bills And maybe not in ways that are all that productive So like I want to separate the token maxing discussion which I think is like a questionable sort of management technique and metric for Productivity with the lower with the question about whether AI is in a bubble and whether it's a demand crunch or a supply crunch It has been very interesting and I wonder if you've observed this too that many of the people who just months ago We're saying that this was all a bubble because the technology didn't work It wasn't useful and people were never going to pay for it once they you know once they wised up To saying that in effect the technology is so good that there is so much demand for it That investors are willingly subsidizing this to sort of you know prop up their investments and lead a lot more people into the market In some cases it is the exact same people who have made both polls of that argument within this band of a few months So I just want to note that Let me note the best possible version of their case. Let me make the best possible version of the case I remember five to ten years ago you and I sort of Connamed a concept called the millennial urban consumer subsidy this idea that People in their 20s living in San Francisco, New York City Who woke up in the morning on Casper mattresses and then rode their electric, you know, lime scooters to work and then went to work in a We work and used a bunch of door dash for ex-services to bring everything to their doorstep without leaving their co-working space and then Went home on like in an uber and then went to sleep and before that had dinner using one of those God what did those those meals services? Blue apron etc God has been so long um The good old days these companies these companies you forgot movie pass Exactly after the blue apron they would do movie fast These companies were collectively losing tens of billions of dollars a year In a strange way venture capitalists were essentially paying 25-year-old PR people To go to the movies and eat chicken And you know ride on subsidized taxis And some of these companies made it but a lot of them didn't and they didn't make it because demand was being subsidized At a rate that was not realistic and was not plausibly fulfilled Once consumers had to pay the full cost the technology so in a way what I guess the bubble folks are still saying right now is I used to think AI was the railroads now. I'm saying AI is blue apron It's a tech it right. It's a service that people are using only because They don't have to pay for what it actually costs, but the second day to pay for it actually costs the They won't get enough value from this to continue paying and then the bottom will fall out So I so I think they're trying to communicate something along those lines But now I will off ramp it back to you to defend your position against that position So look I think there are a few separate things going on here one of them is yes There are absolutely people who are being subsidized in their use of AI by Stacking together these promotional offers or having these sort of plans that cost $20 a month where you can use way more than $20 a month of tokens I talked to a guy when I was writing my My token maxing article who had found this like this promotional plan from Figma a design tool that allowed him to use like Thousands of dollars worth of clawed tokens on a plan that cost you know 20 bucks a month or something And that to me did feel like a sort of movie past situation Where clearly This was a promotional offer it was not meant to be sort of a permanent thing But I think there are a lot of these kinds of things going on throughout the you know the industry People are offering their sort of on ramp plans that are subsidized and then they want you to sort of move to the higher plan or something like that But many of the most avid token mexers are not paying for these sort of subscription plans. They are using this Technology this these models through the API. They are paying for the sort of tokens on a on a per token basis and I think there's a reasonable argument to you made that that price the sort of price per token Will fluctuate over time. Maybe they get maybe the companies get so compute constrained that they start jacking up the price of tokens to effectively destroy demand and and make their services easier to sustain maybe they have to start competing on Price and they drive the cost of tokens down because a Chinese company or a Google or someone else is offering tokens that are comparable quality at a much lower rate I think all of that is subject to the basic laws of gravity and of supply and demand But I think the more relevant question here is What is this token use replacing Because if it is just a toy if it is true that none of this is making people More productive or companies more productive Then I think it is absolutely It would absolutely follow from that that as soon as these promotional rates get taken away they Give up and start going back to writing code by hand or whatever they were doing before But what I'm hearing from people in the software industry is that this is a permanent structural change to the way that companies will want to write Software once you have tried the magic coding models I've heard very few people say that they are that they expect this industry to go back to doing things the old way And so what they're really benchmarking their token costs against is their payroll they're saying well For this this comparable unit of work, you know writing this app writing this unit test how much would we have to pay a developer to do that and Can we get it an agent to do it for less and so I think as long as The cost per token of I'm using token and air quotes when it comes to the second part of this example the cost per token of an AI model Is cheaper than the cost per token of a human software developer or white collar worker I think they're going to feel like that's a good job trade. I'm trying to put some of these story lines together and see if you follow this logic because this question isn't written down, it's all just sort of top of mind. I wonder if you see the world of artificial intelligence bifurcating a little bit in the near future. We're on the one hand, these AI companies are going to see the need to buy chips and provide inference that is cheap so that they can make money on inference. They can make money on AI use that is for consumers. In the one hand, there's going to be this consumer facing strategy of how do we drive down costs so that the prices that we charge our customers are profitable on a per unit basis, just classic unit economics, right? That's strategy number one, but there's also sort of, there's a forking, so to speak, here. We're on the other hand, they're going to be developing technology like the mythoses of the future that are incredibly powerful, maybe unbelievably valuable to certain enterprises, but also not remotely for consumer use, not only because they're so expensive to run inference for, but also because we don't want ordinary consumers in the middle of Iowa who just have a lot of time on their hands to break into the Social Security Administration and steal a bunch of secure data. Is there a way in which what we're seeing right now with both this token maxing phenomenon and the mythos phenomenon is the beginning of a very hard fork in AI with consumer AI being really erased the bottom of token costs and non-consumer AI, enterprise, government, cyber security, national security AI, being an entirely different ballgame of building more and more and more advanced models that are not going to appear in anybody's Chatchee BT or Claude Window. How do you feel about this sort of general prediction of us being at a turn of the road here? Well, thank you for the hard fork podcast promotion. I really appreciate that. That's that native ad you slipped in there. Okay, I'll try to explain this one in plain English. That's a podcast. I'm just trying to pay you back. What was the question? I'm so distracted by my joke that I haven't. Devin, keep all this in. Is this a two-tier strategy now for the future of AI? Is there going to be a consumer strategy here that is entirely diverged from this non-consumer strategy to essentially build an AGI technology that's not for consumer use at all? I think that's a reasonable possibility. I think we are already starting to see some of this. The models that you get in your free Chatchee BT window are worse than the ones that you get if you pay for the premium plan. Same deal with other companies. They make their highest end models, which are more expensive to run available to only higher paying subscribers, and they offer sort of a degraded or distilled experience or model set for free users. But I would expect that to continue to widen. I don't think it's crazy to envision a world where you're two from now. You can pay thousands of dollars a month for a premium plan on one of these companies' models that gives you access to all their latest and greatest technology. Maybe millions of dollars if the technology gets good enough would be a suitable price point for that some corporate customers and governments would be willing to pay. I absolutely think that kind of stratification is already happening and will continue to happen. I think that's just what happens in every other industry. We have Ferraris and we have Honda Civics. You can afford one and not the other. Right. I think it is Ferraris versus Honda Civics, but it's also Ferraris Honda Civics and fighter jets. I guess what I'm saying is when I'm beginning to see now with not just mythos, but also open AI reporting or self-reporting that they have a technology that's very similar to mythos and not releasing to the public. It just seems like we're in a new phase right now where the business strategy to make consumer AI more profitable is incredibly different from the existential strategy to build a technology that can do everything. We don't want the consumers to have access to the technology that can do everything in cybersecurity and hacking. I just wonder whether we're going to look back at Q22026 as being this moment where after this period, we talk differently about what the AI companies were trying to do because it no longer made sense for them to think of these two things being the exact same strategy. Whereas before, as soon as these AI models were ready, they were being sent to the public. It just seems like a real break in what we've come to expect from these companies. I think one really important point in all of this is that cloud mythos preview with all of its scary cybersecurity capabilities did not receive special training to be good at cybersecurity, at least according to Anthropic. That was just an improved capability that emerged from overall improvements and overall scaling of the underlying model. It is also probably better at many other things. I think you're right to think that the market is bifurcating into professional users and consumer users, but it is also the case that the same techniques that make a model better at cybersecurity, defense, and attacks also make it better at solving complex math and science problems or answering questions on homework. I do expect that because of that quality of the overall water level rising in more or less tandem, that consumers are going to want access to the things that do the things they care about really, really well. Those improvements in those capabilities will probably come at around the same time as the improvements in all the scary capabilities. That's just kind of how this stuff has progressed over the last few years. My very last question here is, if the cybersecurity capabilities of mythos were an emergent feature of ordinary scaling, then how will that technology ultimately be released to the public in a way that neuters a capability that's emergent? You know what I'm saying? How could you possibly release, especially the sort of non-entropic versions of this technology that won't have this capability built into them? It just makes it, what you just said, I've read that elsewhere, but it makes it seem very, very difficult then to stop ordinary users from using a capability that is emergent rather than deliberately designed. That makes it seem much harder to regulate this technology going forward. It is, and it's especially hard because the same capabilities that make it good at writing code are also the same capabilities that make it good at breaking code. You cannot extricate one from the other. Now, there are things that you can do that anthropic says it has done to try to mitigate the risks on the downside. You can tell a model, hey, if a user asks you to break into a smart contract or exploit a piece of software, don't do that. They can sort of try to control the outputs at the point of contact with the user. But the basic capabilities are in there in the model, and users and governments and terrorist groups, or whoever wants to use these things for malicious ends, may find ways around those guardrails. So yeah, I think there is a real risk here. I am not persuaded that we have solved the so-called alignment problem that tries to teach AI systems to obey human values. They are displaying all these weird and shady behaviors during pre-training testing, as you mentioned, with the exploits and the sandwich in the park story and all of that. These models are getting very capable, and they are getting very capable at the same time as they are getting more dangerous, and so it is hard to extract just one of those threads from the other. Given Bruce, thank you very much. Thanks for having me.