The US Government Gave Anthropic 90 Minutes to Shut Down Its AI

Last week, Anthropic became the first company in history to be so good at building artificial intelligence that the United States government made them turn it off. Well, they made them turn it off for foreigners anyhow. Anthropic then turned it off for everyone because they couldn't tell who the foreigners were. So that went about as well as you could have hoped for. This all happened on a Friday evening. Anthropic were given about 90 minutes notice and the timing was not ideal as earlier in the month. Anthropic had confidentially filed paperwork with the SEC for an initial public offering. According to the Financial Times, the prospectus disclosed a revenue run rate of $47 billion and an expected valuation of $965 billion, so almost $1 trillion but the being modest about it. It is a considerable step up from the $380 billion they were valued at back in February and it is a lot of money for a company that until fairly recently was best known for an extreme dedication to AI safety. Anthropic built its brand around a framework called Constitutional AI, which in practice meant that they built a chatbot that would politely refuse to help you write a slightly mean email just in case someone's feelings got hurt, which is nice. Table 5 was the headline consumer product, while it's more powerful sibling Mythos 5 was only made available to a vetted group of organizations to a program called Project Glass Wing. They are Frontier AI models, which means they cost billions of dollars to train and are expected to usher in a new era of human productivity or at the very least to justify a near-trillion-dollar stock market valuation. Now when you're drafting your S1 and planning one of the largest IPOs in corporate history, you generally want your flagship product to remain available to your customers. It's difficult to sell shares to investors if your core business is illegal. In theory, the current US administration has a very clear policy on artificial intelligence. Preston Trump announced on his first full day in office that America is in an AI race with China, declaring we have an emergency, we have to get this stuff built. The stated plan was to deregulate the industry, get out of Silicon Valley's way, and let American tech champions dominate the global market. It now turns out that getting out of Silicon Valley's way involves the Commerce Department sending a letter to your CEO on a Friday evening, giving you 90 minutes to shut down the company's most advanced product. The letter that Dario Amote received was what's known as an "is informed letter", a private notice that the government uses in export control cases to tell a company that from now on it needs a license for things it didn't need a license for yesterday. It directed Anthropic to suspend all access to the Fable 5 and Mythos 5 models by any foreign national anywhere in the world. The letter, a private notice from Commerce Secretary Howard Latnik, gave no specific basis for why the restrictions were necessary, but cited US laws that allowed the government to impose export controls on civilian technology that could be used for intelligence purposes. It went on to say, until further notice, you must submit an application for an individually validated license prior to the export of the Mythos or Fable models to any destination worldwide or to any foreign person wherever they're located. It then threatened criminal and civil penalties if the company failed to comply. If you run a global tech business with hundreds of millions of users, verifying the exact citizenship and immigration status of every single person visiting your website or using your API on a Friday evening is difficult. The exception is probably "palmeteer", but because there's no foolproof way to instantly separate the Americans from the foreign nationals online, Anthropic had no real choice within 90 minutes, other than to hit the kill switch for absolutely. They abruptly disabled Fable 5 and Mythos 5 globally, ensuring compliance by simply axing their own product. The directive did not just apply to users in other countries. It applied to any foreign national, which under the export administration regulations, includes people working inside the United States on visas. This mechanism is known as a deemed export. The legal theory is that granting a foreign citizen access to control technology inside a San Francisco office building is treated exactly the same as boxing it up and shipping it to their home country. The net effect of this rule was that Anthropic was forced to lock its own non-US citizen engineers out of the very systems they had just spent the last year building. I'm not sure what the standard corporate governance guidelines say about making it a federal crime for your own staff to log into their computers, but I imagine it's bad for productivity. According to the administration, this shutdown of an year trillion dollar company's primary new product was a reluctant but entirely necessary intervention to protect national security. White House AI Advisor and podcaster David Sachs explained on social media that a trusted partner had come forward with a highly dangerous jailbreak that allowed users to bypass Fable 5 safety guardrails. Sachs claimed that the administration had asked Anthropic to fix the flaw or pull the model and that Anthropic had refused, which is a very different story. Anthropic gave a different version of the conversation. According to their statement, the letter arrived at 5.21pm on a Friday, which is of course the precise time of day chosen by people who want a problem to become someone else's problem on to Monday. They said they hadn't been given any specific technical details of the government's concerns and that they certainly hadn't refused to fix anything. They complied immediately. Instead they explained that after reviewing the report, they believed caused the panic, the technique in question, and I'm quoting their statement here, essentially consists of asking the model to read a specific code base and fix any software flaws. Now just to be clear, asking a coding AI to read computer code and fix the bugs is not typically considered a complex cyber weapon exploit. It's generally considered the product working exactly as it was advertised. Now you might think that the cybersecurity industry, the literal people whose networks the government was trying to protect from these terrifying automated code fixers, would be thrilled that the state had stepped in to save them, but it turns out that they weren't happy at all. More than 100 top cybersecurity executives and technical pioneers from companies like Google, Zoom, and Sophos signed an open letter to the administration politely pointing out that the government had just confiscated their best defensive tool. The experts noted that anthropic had already built significant protections into Fable 5 to prevent it from being used for offensive hacking. In fact, according to an article in TechCrunch, those safety measures were so aggressive on launch day that they were the source of quite a lot of humor in the cybersecurity community. Well-known security researcher Valentina Palmiotti reported that the model was hitting its own internal safety filters and refusing to complete completely innocuous tasks such as reading an ordinary IT blog post, just in case the blog post contained a word that sounded vaguely related to computer networks. Now I should say something here because viewers often assume that I think AI is all nonsense and I don't. Set aside the AI-generated music nobody asked for and the LinkedIn posts written by people pretending to have insights, these tools are genuinely useful to a lot of people and they've real business cases. The how matters here because you've all seen the stories of someone filing a legal brief full of cases that the AI simply invented. The lesson there isn't that the tool is useless, it's that AI doesn't replace human expertise, it supplements it, a good lawyer can spot the errors while a person with no legal training can't. The same goes for code, for accounts, for analysis, in the hands of someone who knows what they're doing, these models don't make the expert redundant, they let one expert do the work that used to take a team. I know people running small businesses who can now take on jobs that used to go to much larger firms, a friend in the gaming industry told me how creative people who aren't necessarily technical can use Cloud Code to put together a rough version of their idea. And handed to coders to make the code production ready. That's a lot quicker than trying to describe an idea to a programmer who can't see your vision. This is the whole reason a Friday evening can't be done. hill switch matters and isn't just a funny story. The tool being switched off is one that a lot of people really rely on. Cybersecurity veteran Matsuichi told TechCrunch that if you simply asked Fabel 5 to write secure code, it assumed that you were doing dangerous cybersecurity work instead of standard software engineering, it then panicked and immediately downgraded itself to a less advanced model. So you have a situation where a company spends thousands of hours red teaming a product to make it so hopelessly cautious it won't even read a website, its own users complain that it's practically unusable for basic engineering and the White House still gives you 90 minutes to turn it off because they're worried it's a threat to the republic. And the topic statement observed that if they standard were applied across the entire industry, it would essentially halt all new model developments for all frontier model providers, which would be awkward given the evaluations we just discussed. Look, Anthropics position is that no AI model from any provider is perfectly resistant to jail breaks. They said so plainly when they launched Fabel and they say it again here, they suspect that perfect jail break resistance is not currently possible for any model provider. Every system in the industry can be coaxed into something it shouldn't do under the right conditions. So Anthropics approach was what they called defense in depth, make the jail breaks narrow are expensive and then monitor closely to catch anything that slips through. This is also why they now retain customer data for 30 days so that they can study attacks, the safety company building itself a surveillance requirement for safety. But wait, in case you're wondering who the highly credible trusted partner was, who found this flaw and ran to the White House to report it, it was Amazon CEO Andy Jassy. I know, you all thought Jeff Bezos ran Amazon, he's retired there's a new fellow Andy. Andy's in charge now. Now corporate partnerships in Silicon Valley can be quite complex. Amazon is one of Anthropics largest financial backers, having invested $13 billion in the startup with commitments to put in up to $25 billion more. Anthropic is also a massive Amazon customer. CNN reported in April that the company had committed more than $100 billion to Amazon web services to train and run its AI models. So to summarize the financial logic of modern tech alliances, you raise billions of dollars from a cloud computing giant, you sign a contract promising to spend $100 billion renting their computers. And then that partner's CEO turns you into the White House on a Friday afternoon to ensure that your flagship product is pulled from the market. Presumably that particular phone call is not featured in the promotional brochures for Amazon web services. It's a very distinctive style of investor relations. Of course, Amazon also happens to develop its own competing AI models, which means that by blowing the whistle on Anthropic, Andy Jassy managed to protect the civilized world from the dangers of automated software debugging while simultaneously kneecapping his primary corporate rival. It's a remarkable display of multitasking. When asked for comment, Amazon released a statement saying, "It's not uncommon for governments to seek our council on potential security risks. When they occur, we don't share the details of these discussions, which translated means we're not going to tell you what we've said about you." Now we must keep in mind that being treated like an international arms dealer was a bit of a surprise for Dario Amode, mainly because he'd spent the days beforehand explaining to anyone who would listen how the government should have the power to do exactly what they had just done to him. On June 11th, exactly one day after the launch of Fable 5 and less than 24 hours before the Commerce Department letter arrived, Amode had published a lengthy policy essay. In it, he complained that the intersection of artificial intelligence and political institutions felt a bit like the plot of the Lord of the Rings. I don't know what it is with these tech guys and the wizard books. "Where the Hobbits," he said, tried to rouse treebeard the wise but ponderous sentient tree to defend his forest. The problem Amode wrote was that treebeard operates at a very slow speed compared to Hobbits. It's a thoughtful literary analogy to sort of think tech executives write when they want to seem deep ahead of a near-trillion dollar IPO. Amode's argument was that the government's needed to build a statutory process that was transparent, fair, clear and grounded in technical facts so that they could block unsafe model deployments. The only flaw in his theory was assuming that a government bureaucracy tasked with national security would behave like a wise philosophical tree from a wizard book. It turns out that when you tell a regulator that you've built a piece of technology so powerful it might destroy civilization. They don't look up from their desk, stroke their bark and ponder the deep philosophical implications of recursive self-improvement. They give you 90 minutes to turn it off. Treebeard moved fast. This all led to a rather awkward situation a few days later in France where the leaders of the G7 nations had gathered for a summer. The schedule included a special lunch with what the press called the AI3, Dario Amode of Anthropic, Sam Altman of Open AI and Demis Hassabis of Google DeepMind. The purpose of bringing these multi-billion dollar tech giants to a luxury lakeside resort in France was to discuss the tremendous opportunities of artificial intelligence for the financial sector. The financial sector, as we know, is incredibly keen on automated coding tools and international banks like JP Morgan Chase and Goldman Sachs had spent months negotiating global contracts to give their staff access to Claude Fable. Unfortunately, those permissions had evaporated overnight. By the time the super-rived JP Morgan had already cut off Anthropics access for its Hong Kong staff following a similar move by Goldman Sachs, the banks had looked at Anthropics licensing terms which technically exclude usage in greater China and decided that rather than risk a multi-billion dollar compliance dispute with the US government on a Monday morning, it was simpler to just remove the option from the drop down menu. So, you have the leaders of the Western world sitting down for a polite meal with a group of executives whose primary commercial products have either just been frozen by executive order or are currently banned for use in the major financial hubs of Asia. French President Emmanuel Macron observed afterwards that the Anthropic dispute had clarified the stakes for the G7. He warned that if the United States from one day to the next can turn off the switch, it would damage the multi-trillion dollar companies leading the AI arms race, which is a reasonable point. And I never thought I'd see today a French leader criticize America for interfering too much in business, but here we are. If relying on an American cloud service means a bureaucrat in Washington can unplug your entire business over a line of code on a Friday afternoon, companies might reasonably start to treat American AI as an operational risk. But while President Macron and the other middle-power leaders were busy panicking about their tech subservience and drafting a G7 communicate pledging to discuss the risks, the tech executives themselves were busy doing something entirely different. They were using the panic to explain how incredibly dangerous and therefore how incredibly valuable their technology actually is. Dario Emote urged the room to resist the temptation to splinter while Sam Altman and Demis Hassabas warned the world leaders about the imminent perils of AI-powered bioterrorism and cyberwarfare. There's a distinct style of marketing in Silicon Valley where the best way to convince investors that your company is worth a trillion dollars is to tell the President of the United States that your product is so terrifying that it might accidentally end all human life. If you tell Wall Street that you've made a very efficient spreadsheet tool, you get a tech multiple. If you tell them that you've built a digital nuke, you get an intergalactic multiple. The only problem with convincing the government that you've built a cyber-weapon of mass destruction is that they might eventually believe you. And when the state decides that your product is a munition, they tend to stop letting you sell it to anyone. Unless it's an actual munition, of course, in which case Texas will take all you've got. It's not entirely new for the US government to treat software-like munitions. They've a rich history of trying to regulate math by treating it like a bomb. From the 1970s to the 1990s, the government classified strong cryptography the technology used to secure digital communications as a munition, which meant that publishing your encryption code online could be treated as illegally exporting weapons. One developer was criminally investigated essentially for putting cryptography on the internet. Going back further, the 1946 Atomic Energy Act, known as the McMahan Act, abruptly cut off all nuclear research sharing with Britain, essentially locking America's closest ally out of technology that they had helped to develop. Governments generally prefer to control powerful new technologies, and they eventually have to learn the hard way that once the math is out there, you can't really put it back in the bottle. But it usually takes them a few decades to admit it. Which brings us to the financial mechanics of Anthropics' $965 billion valuation. To justify a number like that in the private markets, a company has to convince investors that its total addressable market, or TAM, is essentially everyone on earth with an internet connection. The business model relies on building hyper-intelligent systems and renting them out to the global economy for a monthly fee. But if your flagship product is abruptly classified as a national security threat, that TAM shrinks considerably. You can't easily run a global software as a service business if your product is legally classified as a weapon. When your target market shrinks from the entire global economy to people who have passed a background check in Washington, your near-trillion dollar valuation starts to look a bit like a hallucination. The astronomical valuations of companies like Anthropic and OpenAI are built on the assumption that artificial intelligence is a winner takes all market, and that they will be the winning company, much like internet search or social media did. Google and Facebook were not the first companies to enter their respective markets, but once they arrived they took practically everything. Investors are handing billions of dollars to AI labs under the assumption that the same monopoly dynamics will apply to large language models and that they'll eventually be able to charge a lot for access. But if relying on American frontier models means that the US Commerce Department can pull the plug on your entire business over a line of code on a Friday afternoon, well, the market tends to start looking for alternatives. The reaction in Europe to the Anthropic shutdown was immediate panic. French presidential candidate Bruno Ritalio described it as a wake-up call, noting that a nation that depends on others for its technology can be unplugged overnight. Europe currently relies on non-EU countries for about 80% of its technology and 70% of its cloud computing. In response to this sudden realization that they do not control their own infrastructure, European politicians proposed the European technological sovereignty package, a plan to direct 422 billion euros or around 490 billion dollars towards building their own data centers and AI models. Things grew so anxious that a group of European tech experts even published a dystopian novella called Europe 2031, which imagines a dark future where America cuts off all AI access and ends up taking over ASML, the Dutch company that makes the machines the entire world needs to manufacture advanced chips. Now, I won't lie, when your tech policy involves publishing speculative fiction about corporate takeovers, things are getting pretty weird. While European officials were busy writing novellas, the Chinese tech sector was busy writing code. Washington's export ban essentially served as a massive free advertising campaign for Chinese open source AI. Shortly after Anthropic was forced to shut down its models globally, the Beijing-based AI lab, Zhipu, announced a new open source model. They chose to launch it at exactly 521 pm, a direct reference to the precise minute Anthropic had received its shutdown order. You have to respect the level of corporate readiness required to coordinate an international product launch around a time-based joke. Because models from Chinese firms like Zhipu and DeepSeek are open source, companies can run them on their own servers. The fact that their self-hostible makes them entirely immune to being randomly switched off by Howard Lutnik, and more importantly, their cheap. DeepSeek's flagship model currently costs 87 cents per 1 million output tokens, which makes it roughly 60 times cheaper than Anthropic's fable 5 was. The results of this are already visible. The Financial Times reported that in the first two weeks of June, four of the five most popular models on OpenRouter, a platform that provides access to hundreds of AI models were Chinese. Among the global top 20, Chinese models processed twice as many tokens as their US competitors. If AI is not a natural monopoly, and foreign competitors are offering models that are good enough immune to US kill switches and 60 times cheaper, then we end up with a highly competitive market, and in a highly competitive market, profit margins collapse. The economic benefits of all this AI productivity would instead flow to the people using the models rather than the company spending billions of dollars to build them. This might be great news for businesses looking to automate their coding. It's terrible news for anyone buying shares at a $965 billion valuation. So, where does all of this leave the frontier AI industry? Well, the lesson is simpler than Wall Street would like. For years, the theory was that building artificial intelligence required silicon valley genius, billions in venture capital, and an exclusive relationship with a cloud computing giant, ideally one whose CEO won't report you to the federal government as a hobby. It turns out that there is no monopoly on math. The open source models are good enough, they're 60 times cheaper, and nobody in Washington can switch them off on a Friday afternoon. The productivity gains everyone was promised will still happen. They'll just flow to the businesses running the code on their own machines, rather than to the investors who paid almost a trillion dollars for a global monopoly on math. As for anthropic, they spend years warning the government that their product was dangerous, built it to be so cautious that it would refuse to read a blog post, and were then genuinely surprised when the government agreed it was dangerous and took it away. Telling the authorities that you've built a weapon and then being shocked when they treated like one is all things considered genuinely considered a bad corporate management. Anyhow, that's all from me. I'd say more, but I might be seen as a foreign national, and I'm not sure I'm legally allowed to finish this. Thanks for tuning into this week's podcast. The podcast is entirely supported by viewers like you on Patreon. If you'd like to contribute, there's a link in the show notes. Have a great week and talk to you again soon. Bye!