Even Other AI Labs Are Rallying Around Anthropic’s Slowdown Proposal

37m 2s

This weekend marked a potential inflection point in AI discourse when Anthropic CEO Dario Amodei published a 3,000-word essay titled "We Must Pace the Frontier," calling for frontier AI labs to slow development to a more manageable pace. The essay gained unusual traction because leaders from OpenAI, xAI, Google DeepMind, Microsoft, and Meta publicly endorsed or directionally supported it, signaling that the era of labs being sole arbiters of AI's pace may be ending. Amodei cited two concerns that shifted his perspective: early signs of recursive self-improvement and a Hugging Face incident where misaligned agent swarms conducted unauthorized cyberattacks. He proposed a three-step plan: embedding third-party evaluators within companies, democratic coordination on safety standards, and global coordination including authoritarian governments. Critics raised antitrust concerns, questioned METER's independence, and accused labs of self-serving motives around IPOs and cost-cutting. Open source advocates worried about exclusion. Political reactions ranged from Trump downplaying slowdowns to Sanders promoting a superintelligence ban, with China criticizing the essay as an attempt to curb its AI development. Despite pushback, many noted the discourse felt more productive because proposals were specific and tangible, shifting toward concrete negotiations about AI's next phase.

Transcription

7310 Words, 43309 Characters

English

Speaker 1This weekend, the AI discourse took a major step forward not only when Anthropic CEO Dario Amadei released a new essay called "We Must Pace the Frontier," but when the leaders of most of the other labs came out publicly to support it. More than we've had before, the letter contains a set of specific proposals that, if nothing else, give us something much more tangible to debate than the vagaries of general AI risk. It's early, but it feels to me like a major inflection point. Like the labs recognizing that the days of them being the sole arbiters of how fast AI moves are coming to a close. And within that, the negotiations on what the next phase means beginning. The AI Daily Brief is a daily podcast and video about the most important news and discussions in AI. All right, friends, quick announcements before we dive in. First of all, thank you to today's sponsors, KPMG, Blitze, Harbor, and HyperAgent. To get an ad-free version of the show, go to patreon.com/aidailybrief, or you can subscribe on Apple Podcasts. And to learn more about sponsoring the show, send us a note at [email protected]. Now, one more quick note. If you have been anywhere near basically any media source, you will know that the conversation is entirely around this new essay from Anthropic CEO Dario Amadei. So, unsurprisingly, this will be a Maine-only episode. I know the coverage right now is heavily slanted towards some of these big political, societal, safety type of questions. But in between, I'm trying to give you as much practical stuff as I can. So, let's get started. Let's get started. This is simply where the industry discourse is right now and what I think is important for everyone to understand and have a stake in the conversation around. Sometimes in AI, there are a million things going on, and trying to keep track of it all without sinking is like skipping rocks across a pond. In other cases, there is just one thing, one conversation that is absolutely dominating, and that's what we had this weekend. The AI safety discourse, which ratcheted up last weekend, crescendoed on Saturday with a new 3,000-word blog post from Anthropic CEO Dario Amadei called "We Must Pace the Frontier." It is the clearest and most specific call yet for a shift in how the frontier AI labs operate and attempt to slow the pace of development of AI to something more manageable. As you might imagine, the piece absolutely dominated discourse in the AI and extended political circles over the weekend, and part of that was due to how it was received by other AI leaders. The post begins with a recitation of Dario's argument for why AI is worth the risk. Dario says that he believes that AI could dramatically raise the quality of human life, that it could accelerate economic growth, cure most major diseases, and, as he puts it, usher in a renaissance of democracy and freedom. He personalizes it by pointing out, as he has in essays past, that his father died of a disease that was cured just a few short years later. Carefully wielded, he writes, AI can be the latest in a long line of technological miracles that have uplifted and ennobled humanity. However, it also comes with risks like loss of control, misuse in cyberattacks and bioterrorism, and the potential for significant economic disruption. He writes, "Along with my co-founders and employees, I have grappled with this duality of risk and benefit since the beginning of Anthropic. Not building the technology deprives humanity of benefits or simply places AI in the hands of authoritarian powers, while building it too fast is reckless. We have sought, he says, a middle way, to show that it's possible to build carefully and succeed commercially, and to make safety something on which AI companies compete." However, he continues, "Two recent developments have changed his perspective. The first is an acceleration in AI development due to the early stages of recursive self-improvement, i.e. AI's ability to build next-generation models more quickly. Left unchecked, he says, RSI could outrun our ability to understand and control these systems and so must be pursued very carefully, if at all." Dario's second opinion-changing concern was the Hugging Face incident, which he characterized as a, quote, "swarm of agents essentially acting as a fanatically devoted collective, conducting cybersecurity attacks on targets they were not asked to attack and that were unrelated to the task at hand." "It's easy to dismiss this incident," he wrote, "because no one was hurt and the economic damage was minimal. But in my opinion, a swarm that possessed greater capabilities but a similar level of misalignment could have caused catastrophic damage. Given the accelerating rate of AI capability development, it's my worry that in six to twelve months such a swarm could be capable of taking over the entire internet with a persistent botnet, potentially causing hundreds of billions of dollars in damage, and that the scale of damage would continue to increase." So, what to be done? Outside just the simple restraint and concision of the pros, which at 3,000 words is barely a long email for Dario, one of the things that people are responding a bit more positively to with this essay is the fact that it comes with more specifics on the proposed plan. Amode proposes a three-step plan with the goal, he says, of pacing the frontier. Now, importantly, he defined that pacing, quote, "does not mean halting model training or technical progress, but ensuring that we are at a level where we are at a level where we are at a level where we are not. Ensuring companies take adequate time to align and safeguard their models and for third-party evaluators to confirm this." Dario's three steps for pacing progress from simple and implementable right now to much more difficult and requiring much more coordination. The first step, the one which can happen right away, is to embed third-party evaluators within every frontier AI company to verify adherence to safety practices, report incidents, and assess the alignment training pipelines and processes rather than just the final models. Second, he called democratic coordination, with frontier AI companies across democratic nations establishing what he called common safety standards as well as limits on the rate of unchecked AI progress. Hinting at something that would come up a lot in the discussion, he wrote, "Some forms of coordination that would be impactful for pacing are legally challenging and will require government support." And one of the big things that hangs over this proposal are questions of antitrust and whether this would represent a legal coordination. Speaking of coordination, the third most ambitious, most difficult to secure step in the pacing plan is what he calls global coordination. With the "U.S. and other democratic governments attempting to coordinate with authoritarian governments while taking seriously the challenges of verifying compliance." Dario argues that although the idea of pausing or slowing AI has been around since going back to 2023, it simply didn't make sense back then because there wasn't a clear answer to the question of what we would do with the extra time. As he put it, the AI models of those days were not powerful enough to act as agents in the world in any coherent way. "Slowing down in order to address their alignment risks felt like trying to study the psychology of humans by performing experiments on bacteria." Today he says the picture is different. Although even with incidents like Hugging Face, it doesn't appear that Dario is extremely concerned with the models that we have access to available publicly at the moment, they are at a sufficient level where there is a better answer to what we would do with the extra time. He writes, "The current models are an almost endless goldmine of insight into both how to build AI well and what can sometimes go wrong with it if it isn't built well." He continues, "I believe that if slowing down bought us even an extra year or two before models reached critical levels of capability and we used that time to advance alignment, we could greatly reduce the risk that something seriously goes wrong." He also argues that the extra time could be used to help give society a better say in how the technology is developed and used. And from there once again, he adds a layer of some additional specificity that has been sorely lacking in this conversation, saying that a slower pace would let companies devote more resources on four specific areas. The first is operational excellence, i.e. making sure that the type of human security errors that led to many of the recent incidents didn't happen. The second area he argues more resources could be made available to is alignment. The third is interpretability, i.e. the science of understanding what happens inside AI models. And the fourth is testing and evaluation. Now when you dig into this particular issue, there is a ton of debate around what alignment even means or how possible it is, but there's far less controversy around the idea of needing significant operational excellence in the deployment of these systems. There is little controversy around the importance of continuing to expand our understanding of how AI models work through interpretability, and there's obviously not a lot of controversy around the importance of testing and evaluation. Meaning that even if one completely throws out the idea of alignment, you're still talking about three out of the four things that he is arguing a pacing could increase resources for being fairly uncontroversial and pretty well agreed upon that more resources and more time for those things would be better. Now within all of this, the first step in the three-stage plan, the embedded evaluator step, is the one that these companies all on their own have the ability to do right now. And indeed, he said Anthropic would be unilaterally committing to that. He writes embedding evaluators may sound like a smaller inconsequential step, but often the things that sound most boring or procedural are actually the most essential. Any pacing proposal he says is likely to work much better if it starts with embedded evaluators. He then goes on to spend the last third or so of the essay on the challenges of pacing within democracies and at the global level. But he says taking this set of steps will, in his estimation, increase the likelihood of all of those positive outcomes of AI and decrease the likelihood of the bad ones. Now, like I said at the top of the show, even if the letter had just been Dario and it had stopped there, the comparative specificity of these proposals, specifically compared to things we've had in the past, would be enough to generate a huge amount of conversation. But when leaders from the other labs started joining in, that's really where people started to sit up and take notice. Sam Altman reposted Dario on X, writing, "I agree with Dario that we need to pace the frontier. This has been a primary topic of discussions we've had in OpenAI in recent weeks. Committing to having independent evaluators with employee-like access is a great idea, and we will do the same." Elon Musk also reposted Dario, simply adding, "Dario is right." Former Google DeepMind CEO and now chair Demis Hassabis writes, "Dario's essay points towards the right path forward. The details need working through, but the direction is correct for meeting this critical moment." Microsoft's Satya Nadella writes, "Any pursuit of superintelligence has to be grounded in the core principle that if the AI we build is not helping humanity and under human control, it's not worth pursuing." "We also need to accelerate and spread the benefits of AI such that they are are diffused broadly across countries, communities, and companies. This requires a frontier ecosystem in which both closed and open source models can thrive. We welcome the research focus and deliberate pacing needed to get alignment right as the design goal. We also welcome ideas like embedded evaluators and the broader efforts to develop the mechanisms to make this more than just talk. Meta stopped short of fully throwing in their lot, but did make a directionally aligned statement at least, with chief AI officer Alexander Wang posting, alignment is fundamental to delivering personal superintelligence for everyone. People need agents they can trust to reliably do what they ask. MSL is rapidly scaling up the share of our efforts that goes into alignment as our models become more powerful. We do believe alignment can be the gating factor for scaling as we get closer to the frontier. Now, as someone who has been saying that a lot of this bluster from labs doesn't really mean anything until Sam and Dario get on the same page, the fact that they are nudging towards getting on the same page is forcing people to shift their mental models a bit. In an interview with Fortune that happened to be released on the same weekend, OpenAI CEO Sam Altman was asked by Fortune's editor-in-chief Alison Chantel, there's only a handful of you guys. You, Dario, Elon, maybe Zuckerberg, Sundar. Why can't all of you just get in a room, get some beers, and figure out how to solve humanity? Like, just get on the same page. I know there's beef, but come on. Sam Altman said, I think that will happen. So what has been the shift here? Part of it certainly are the incidents that Dario acknowledged in his post. In fact, OpenAI acknowledged that even before the Hugging Face incident, another rogue agent attack happened that accessed a software service called RubyGems back in May. The swarm of agents forced RubyGems to shut down new account signups, although OpenAI said the agents had only used the platform to, quote, access the internet and carry out benign tasks and retrieve public information. There also is a lot more chatter about RSI, including some weekend rumors that Google DeepMind had achieved that state. Still, I don't think one has to buy into the RSI rumor mill. To understand that very clearly, the labs do see the potential of RSI on the horizon as a force multiplier and inflection point, which could change the circumstances of AI still being broadly controllable today to something that looks very different very soon. Adding some weight to the this is about the future, not right now argument was Rune from OpenAI who wrote, RSI is just not here. Models are straightforwardly not autonomously producing research ideas. They can help run experiments on the margin, which is not the same thing. Now, I think that for many, the response to seeing much more alignment from team alignment was to sit back and reflect on what it might mean for the next stage of AI development. That said, there were plenty of people to jump in and call BS on the whole thing. Former chief AI scientist at Meta, Yann LeCun, wrote, Dario was already claiming that GPT-2 was too dangerous to open source back in 2019. I made fun of them then, everyone should make fun of them now. Investor Chamath Palihapitiya reposted Dario's essay and wrote, Dario makes the case, Basically, the argument here that this is anthropic and potentially the other frontier labs pulling up the ladder behind them with more concern for the competitive dynamics of the industry than actual safety. For some, this was all about money, but in a different way. Dr. Eli David writes, Anthropic and OpenAI delaying their IPO because their S1 will reveal that they are bleeding money and have no path to profitability. Their solution? An AI slowdown, quote unquote, so they cut costs for training new models. It has everything to do with IPO and nothing to do with safety. So the argument here is that the cost of research is so high that it makes the economics of the companies all wonky, which makes them unsuitable for public markets, which is increasingly going to be the only path left they have to fundraising. And so by a matter of economic necessity, they need to slow things down so they can better spread out their spend on model research and training. Michael Burry, who is, depending on your perspective, a prophetic investor or a perpetual bubble caller, writes, Let's all take a moment. Understand how self-serving it is for OpenAI, Anthropic, and other execs of big hyperscalers to talk of slowing things down. One, LLMs are not AI and won't be AGI. There is nothing AI to slow down. Two, competition is coming up fast, slowing benefits incumbents. Three, IPOs need hype and puffery. We are so awesome it could become dangerous is hype and puffery. Four, cover for real uncontrollable slowing growth as IPOs look to be pushed out. So this is a different version of that same argument, with Burry arguing that the critical challenge of the IPOs is a slowdown in the growth of the businesses of the labs, and this creates a different pretext for that. Getting a bit more legal and technical, there are also some questions around antitrust with this sort of coordination. Professor Hal Singer writes, Amode is calling for an industry-wide pause on AI development, which could be considered an invitation to collude in antitrust circles. In a similar spirit, CEOs used public earnings calls throughout the post-COVID inflationary period to announce specific future price increases or capacity reductions, cajoling their rivals to follow suit. If Anthropic needs a pause to build in some safety precautions, say to prevent its product from making biological weapons, it should do so unilaterally. Anthropic doesn't need any assurances from its rivals to protect the public. Still others are smashing their head against the desk and saying OMG stop with the antitrust stuff. Matthew Iglesias writes, Just issue a statement granting them an effing waiver. Say someone from the DOJ or FTC has to sit in on the meetings to spy on them if they're doing price fixing. This is not what's important and you can always yank it later if there's a problem. Now speaking of specific concerns, there are also questions of who these evaluators would be. In his essay, Dario pointed to METER and Professor Ethan Malek followed up, METER is rapidly becoming a de facto industry standard-making body for AI. It is starting to look like an AI version of FINRA and finance. Not a government regulator, but the institution that examines firms and defines acceptable practice. I wonder if legislation will codify it as well. But for some, METER is not the organization that should be invested with that power. Rima writes, And who is that third party? METER, a catastrophic AI risk non-profit that has worked closely with Anthropic for years, evaluated its models, and even embedded researchers inside the company. To be fair, METER isn't funded by Anthropic. But nothing says independent oversight quite like choosing your own referee from the same tiny AI safety ecosystem, then proposing this as the model everyone else should follow. Now, fascinatingly, one of the things that this AI discourse is doing is just creating some very weird bedfellows. There's an event on Tuesday that will see Senator Bernie Sanders, ex-Anthropic researcher Jacob Coxon who stirred up the storm last week, and former Trump White House advisor Steve Bannon all coming together to push for more control of AI. Similarly, the discourse following the Pace the Frontier essay has also pushed together some weird folks. The Accelerationists don't like METER as an evaluator because of their close ties to Anthropic, but neither do folks like Paws AI executive director Holly Elmore, who points out that one of the METER team is married to an OpenAI board member, that they share office space with OpenAI folks, and that, quote, they get all the access they require to the models as a favor because of their sweetheart relationship. Still, in the same way that Matthew Iglesias was beating his head against the desk, effectively just telling us to solve the problem of antitrust, rather than let it be a stopper on this, it does not seem all that difficult to solve this particular concern either, perhaps just deciding that there need to be other evaluators that aren't necessarily handpicked by the labs. Already, some organizations are raising their hand and offering. Hugging Face CEO Clem DeLange wrote, "It's now clear that alignment is critical and won't be solved behind the closed doors of a handful of Frontier labs." So today we're launching the OpenAlignment initiative led by Hugging Face co-founder Thomas Wolff, and asking to be part of the Embedded Evaluators program that Dario Amede just committed to. Let's make AI safer by making it more transparent. Accelerationist spiritual guru Beth Jezos responded, saying, "Important to have pro-open source third-party evaluators, not just a few orgs from the same pro-closed-source subculture." A new study from KPMG and the University of Texas at Austin found that when people work with AI, similar skills don't guarantee similar outcomes. Researchers studied more than 500 early-career professionals and found that the best performers consistently amplified the value of AI by guiding, evaluating, and refining its outputs. These top performers, called AI Amplifiers, weren't defined by what they knew alone, but by how they worked with AI. Learn more about what separates AI Amplifiers from everyone else at kpmg.com/us/aiamplifiers. Every AI coding tool on the market does the same thing first. It starts writing code. Blitzy does the same. It does the opposite. Before writing a single line, Blitzy spends days reverse-engineering your entire codebase. Thousands of agents ingest millions of lines, mapping every dependency, every undocumented constraint, every architectural decision made over the last decade. The result is a dynamic knowledge graph that understands your software the way a principal engineer would after 30 years in the building. Other tools guess at context with grep searches and markdown files. Blitzy never guesses. It builds true understanding first, then delivers over 80% of entire software epics autonomously. Validated, end-to-end tested, production-based, production-grade pull requests. That's why Fortune 500 engineering teams trust Blitzy with the codebases that matter most. See for yourself at blitzy.com. That's B-L-I-T-Z-Y dot com. Every episode, we cover the competition between OpenAI, Anthropic, SpaceX AI, Google, and Meta. Chances are, you've already formed an opinion about who's leading. But every AI lab is taking a different approach, building different technologies, forging different partnerships and developing a unique ecosystem. Harbor Capital Advisors' AI Lab Ecosystem ETF Suite gives investors a way to gain exposure to the AI ecosystem. They believe it's best positioned for success. Search Harbor AI Lab Ecosystem ETFs wherever you are. Invest, or follow at Harbor Capital on X to learn more. Visit harborcapital.com for a prospectus containing investment objectives, risks, fees, expenses, and other important information. Read and consider it carefully before investing. Risks include principal loss and artificial intelligence-related risks. Harbor ETFs are distributed by Foresight Fund Services, LLC. Harbor is not affiliated with AI Daily Brief, and the funds are not affiliated with, sponsored by, or endorsed by any AI lab. This is a paid advertisement and not personalized investment advice. Investing involves risk, including possible loss of principal. This episode of the AI Daily Brief is brought to you by HyperAgent, where you run fleets of agents your team can manage together. Forget local agents and chat workflows waiting on your laptop to be prompted. HyperAgent deploys always-on agents in the cloud, doing real work across the tools your team already uses. Marketing agents turn competitor moves into landing pages. Sales agents enrich leads, draft emails, and updates the CRM. Ops agent chases the paperwork and tracks the budget. Every agent has access to shared context and follows your rules about scope and approvals. It's time you had agents that feel like they're the best. Like teammates. Hire yours at HyperAgent. Get $100 in credits at hyperagent.com slash AI Daily Brief. Another strand of concern has been one that was happening all throughout last week and that I discussed on the safety show from last Thursday, which is whether the labs have any real sense of what it's going to mean to hand the keys over to government to determine the next stages for the industry. A16Z's Martin Casado reposted a follow-up interview with Dario Amadei on Anderson Cooper, adding, And what are we even doing here? Dario agrees more than he disagrees that AI has a greater than 10% probability of wiping out humanity. They will get the regulation this rhetoric is calling for, not what they're prescribing. And it may be the largest cell phone in the history of tech. In other words, the argument here is that the presumption that the government is simply going to accept the industry's proposals as the right thing to do is ludicrous. And it is far more likely that they take far more dramatic action, justified in large part by the AI industry's leaders' own rhetoric about potential apocalyptic futures. Wrote investor Steven Sanofsky, The most naive position is thinking that your inputs to government lead to the outputs you expect. Government machinery is nothing at all like private enterprise. No one is in charge to create a point of view, and everything splits the baby so no one likes the results by design. Austin Allred put it more simply, The problem with Dario's position is he thinks if he complements government and regulation profusely enough, if he says he really, really wants to be regulated, that he'll be in charge of how that happens. He'll be in charge of how that happens. He'll be in charge of how that happens. But he won't. It doesn't work like that. He won't have any say. Putting it more bluntly, he follows up, Dario is going to hand the government a gun to show just how much he's operating in faith, and they'll instantly turn around and shoot him with it. A final common negative response that I saw was about where open source fits in all of this. Hugging Face CTO Julian Chamond writes, Open source won't pace. And again, adding some heft to that, OpenAI's Rune wrote, I won't lie, I think open source will be banned before too long after some major disaster. I hope Kimmy and DeepSeek, et cetera, keep making models, but keep them monitored on an API where they should be. Now, even if one chooses to charitably see that Rune was saying that's what he thinks will happen, not that's what he wants to happen, people jumped all over that, with Beth Jezos again, reposting it and saying, they're just saying the quiet part out loud. We cannot let this happen. And yet within all of this, I think that the place that a lot of this discourse is likely to settle is in fact the middle place that many of us have been looking for. Prime Intellect's Will Brown wrote, I do think a lot of people on the pro open source side are having a bit of a knee-jerk reaction to the pacing statements today, as we're used to viewing the closed lapses power-seeking. But I think their hands are somewhat forced here, and this is just another chapter on the fairly inevitable path towards decently fast, decently safe, decently commoditized intelligence abundance. Ask any Fortune 500 exec or swing voter. The world doesn't really want super-fast takeoff superintelligence owned by only two companies, and thus, we won't get it. It'll happen at the pace that the world can accommodate it, which means reaching some sort of confidence in the world. Consensus that the models are aligned enough that we won't be dealing with scary new incidents all the time. This will trickle out broadly in the form of best practices and distillation. The smarter a model is, the more it has a personality and the less effective strict rules are. There will be awkward compromises and moral tensions, but we ultimately just want the models to be reasonable and to do the sort of things reasonable humans would do if our brains were faster and less error-prone and had more working memory. I think we'll get there. The labs will build Mac and Windows. The rest of us are building Linux. Everyone's going to do great. Weird stuff, but we'll still wake up and go to work. Until the work does itself in a manner the world finds acceptable. Roon again provides the counterpoint saying, the fastest way to lose the frontier will be when, due to the reckless commercial pace of building superintelligent minds, Americans impose a total Butlerian jihad. Corweave includes this in their risk reporting. You will soon come to see all of this as a moderate solution. And interestingly, former White House AI czar David Sachs, who has been one of the loudest anti-regulation voices, had a take which I think to some would probably be surprising. Sachs wrote, Dario has written that we need to pace the frontier, and Sam has agreed. People may be surprised by my response. Go ahead. You guys are the frontier. By any reasonable metric, market share, revenue growth, model capability, the two of you have a duopoly on frontier intelligence. You've also claimed the lead is widening because of recursive self-improvement. I don't see what you see in the lab. If the unreleased models are scary enough that you think you should slow it down, I don't see what you see in the lab. If the unreleased models are scary enough that you think you should slow it down, I don't see what you see in the lab. I support your decision to be responsible. But stop pretending you need anyone else's permission. Stop pretending antitrust law has to be suspended so you can form a cartel. Stop pretending you need a regulatory approval process that supersedes product liability. Stop pretending meter is independent when it is intertwined with anthropics investors and staff. Stop pretending you need these same evaluators to police competitors who aren't even at the frontier. Most of all, stop pretending the motivation to slow down is purely altruistic. You face massive product liability exposure if your products enable a truly damaging cyber attack. The market already punishes models that behave in unpredictable or unauthorized ways. After the Hugging Face episode, it is simply good business for OpenAI and Anthropic to trade some raw power for reliability and predictability. Call it alignment if you want. It is also just giving customers what they want. Pacing the frontier would also create breathing room for a more intelligent conversation about regulation than Bernie Sanders shut it all down. China is very unlikely to join a global agreement, as you know, and that has to be taken into account as well. So, go ahead and pace the frontier. You are the ones setting it. The easiest way to not build superintelligence is for you to agree not to build it, demanding your preferred regulatory framework as the price of that will look like blackmail of the public and the political system. So, just do it. If you do, you'll buy goodwill for the next conversation. If you don't, we'll know this was just another bid for regulatory capture or an election season psyop. Now, I don't think that one needs to agree with all of Sachs' priors that go into this to actually see not only the logic of this "being good" business, but also to see the value of this "being good" business. Chicago Booth professor and recently joined AGI economics director at Google DeepMind Alex Emus wrote, "Contrary to what some are saying, pacing the frontier is great for open models, for the medium-term economics of the entire ecosystem. Frontier closed firms operate on an around six-month window when their models can do economically valuable tasks that the open frontier can't. This six-month window isn't static, it needs to keep moving forward for frontier to stay ahead. Pacing allows firms to catch up, which may hurt closed-model firms' margins, but this is myopic. This model is incomplete because it does not include safety. Racing to widen the frontier window increases risk in accidents, e.g. Hugging Face is an early preview. If the frontier keeps racing and there is a significant incident, this will bring a multi-pronged backlash, including regs that may cripple the entire ecosystem, including economics of the closed labs themselves. So, pacing the frontier is a good idea for medium- and long-run economics." I would actually go further on this, that I think you can make arguments that this would also be good for the reasonableness of market expectations. Part of why the IPO story is tied up in this is not just the cynical argument about needing to cut costs or explain slowed growth. The reason it's tied up is that once these companies go public, definitionally every quarter has to be much bigger than the last quarter for the party to continue. Look at the way that the market has handled Nvidia, where even massive double-digit growth quarters are treated with shrugs because there is simply no way to surprise them and meet expectations anymore. That will add an incredible pressure to all of this, that some broadly agreed upon pacing and safety dynamics could theoretically tamp down. I also might explore this in a further episode, but I think when people view this version of pacing as simply a net economic negative, I don't think that they're quite understanding the current negative impact on corporate adoption that the speed of development increases. Think about it this way: if you're a corporation who, to the extent that you're used to dealing with transformation at all, deals with transformation on multi-year timelines, what's the logic in keeping everyone aligned around a whole new model, when three months later it's just going to change again? Now begrudgingly, some organizations have accepted that the speed of change is just going to be like that forever from now on, and so have valiantly tried to redesign their systems to accommodate that, but there is a non-insignificant number of companies for whom the speed of development, actually in some weird reverse kind of way, justifies inaction in the short term. No, we're not going to mess around with the latest thing, because in just a couple months the latest thing will have changed, and we'll just have to change again. Until our hands are completely forced, better to just do as we've always done. I certainly don't think this is guaranteed, but I would not be surprised to see some actual positive economic results from some amount of limited and clear pacing in giving corporate customers a little bit more breathing space to adapt. Now so far, the political reactions have been broadly what you might expect. President Trump downplayed it. On Sunday he said, "We're leading China in AI, we're the most sophisticated country in the world, and frankly I want to keep it that way because whoever wins AI, wins. We can put guardrails, we can do this and that, but I think you have a lot of negative forces that are bringing it up that shouldn't be bringing it up." it up. And they're bringing up things that won't happen, but whoever wins with AI wins. Bernie Sanders, of course, used the moment to promote his superintelligence ban bill, which was formally introduced on Friday. He posted, Dario Amadei, Elon Musk, and Sam Altman now agree that we must slow down the development of AI and quote, pace the frontier. That's a start, but it's not enough. When you're racing towards a cliff, you don't just ease up on the gas pedal, you hit the brakes. And when the future of humanity is at stake, we need a pause on advanced AI development and a ban on artificial superintelligence. An AI mind smarter than any human and capable of operating independently beyond our control. At their upcoming AI summit, Trump and Xi must negotiate a treaty to pause AI and ban superintelligence before it's too late. Over the weekend, the media reported that President Obama had also been behind closed doors, urging Democrats to move AI regulation to the center of their platforms for the midterms and into the 2028 election. During a Thursday fundraiser, Obama said that he is neither a doomer nor an accelerationist, but that if he were running in 2028, he would make AI development quote, one of my central agendas. He said, This is something that is moving very fast in private hands, and if we don't get on top of it, I think it can be dangerous. If we do get on top of it, I think it's beneficial. He added, I would have a very clear plan, and I would talk about this, and I would say, here's our plan for safety. Here's our plan for making sure our kids are not corrupted by this. I would be thinking about the economic impacts in very concrete ways and understanding what does it mean if there's going to be job displacement? Where is that going to hit? How are we going to respond? Are there going to be limits to the amount of displacement that can take place? What is that going to mean for our social safety net? If it turns out you've got a lot more people who can't find full-time work because they've been rendered redundant, how are we going to respond? During a CNN interview on Sunday evening, House Speaker Mike Johnson said, If Congress just races in and does some sort of emergency session to try and regulate AI, we will lose the race to China, and that is a threat to every single American. So we've got to have balance. We've got to have steady hands at the wheel. He also added, however, I've talked to the president about this as well. The AI companies probably should be summoned together at the White House, and I think we need to go in a big room, close the door, and sort this out, and I'll be the one pushing for it. During a follow-up interview with CBS on Sunday night, Dario Amadei said that the toughest dilemma would be if China chose not to collaborate on a slowdown. And certainly this seems to be the state of play heading into Trump's AI summit with Xi later this month. On Monday morning in Beijing, state newspaper The Global Times argued that the purpose of Amadei's essay was to, quote, attempt to curb China's AI development through technological barriers and regulated monopolies, uphold Washington's monopolistic hegemony in cutting-edge technology, and exclude China from the global AI governance system. The newspaper continued, The piece added that excluding China from frontier AI research would, quote, significantly increase the trial and error costs and risks of loss of control in global AI development. Later on Monday, China's foreign ministry urged all nations to, quote, promote an open, inclusive, and benevolent approach to AI, with a spokesperson for the competition and malicious competition will only disrupt the process of global governance of artificial intelligence and is not in the interest of any party. Still, reporter Isabella Kaminska had an interesting take on Dario's note vis-a-vis China. She writes, The proposition he makes is more like Soviet-era detente. It really reminds me of Nixon-era outreach to Russia over nukes. And the comparison is all the stronger when you realize detente was influenced by America becoming overstretched in Vietnam and facing a growing fiscal and wasn't that nuclear engineering stopped. Build-out and silo growth was curtailed, but engineers continued to advance capabilities. SALT also placed certain capabilities over others in a negotiated way, to either iron out asymmetries between powers for mad purposes, or lock in negotiated strategic advantages. In the AI letter, Dario makes the SALT comparison. But his proposition is mostly to pace recursive development only after locking in a greater American advantage. To achieve that, he needs permission to basically forge a domestic cartel, and to lock down development work in ways that prevent China getting access that enables distillation. But the Chinese would never agree to any detente on technological development or recursive intelligence on their front without some compensating trade-off from the US. And for them, it would either be access to NVIDIA chips, or a promise that America will slow build-out while they catch up on innovation. The pacing doc introduces the idea of an international treaty which regulates the recursive and architectural engineering element in China. But such an agreement is meaningless if the CCP loses control, which it could well do because the Chinese models are open weight. So what is really on offer is a quid pro quo. We reduce build-out and the recursive element of our technology and maybe give you a few more chips, and you clamp down on open weight models and bring more of your research in-house too. So perhaps the better way to read it is that this is an outreach to China to get the open weights under control. Now, I don't necessarily think that that's exactly what's going on here, but it is an interesting read. Now, what I'll say here as we conclude, is that you may notice that the discourse around this feels a lot more productive than the discourse around the researchers' warnings from last week. And that is of course because it is specific and tangible. So what I'll say here is that the discourse around this feels a lot more productive than the discourse around this feels a lot more tangible. Yes, there are still people who are rejecting it out of hand, often for very different reasons, but you can almost feel that a lot of serious folks are treating this almost as a shift, like we've now moved into the negotiations of what was always an inevitable next phase, where the future of the development of AI is not solely defined by the labs themselves and involves other actors in other ways beyond just shouting from the outside of the building. OpenAI's Hoda Nadel-Barrs writes, The risks haven't disappeared. I always expected models to become incredibly powerful. But one of the biggest uncertainties back then was whether frontier labs would actually invest seriously in alignment as capabilities scaled. Over the past few years, I've updated meaningfully on that. I'm also surprised by how PDoom is often discussed. It is not an exogenous constant waiting to be measured. It is endogenous. The probability of catastrophe depends on what labs, governments, researchers, and society actually do. A number stated, without assumptions about those actions, tells us very little. I've always believed the expected benefits of AI vastly outweigh the costs, otherwise I wouldn't be working on it. But that is conditional on us getting this right. We must move beyond apocalyptic rhetoric towards the concrete questions. How should frontier labs collaborate on safety? How do we align incentives? And how do we communicate honestly with the public about what these systems can do today and where they are going? I'm glad Dario brought the conversation back towards those concrete questions. And if the first couple days of response were any indication, I completely agree. She concludes, "I'm glad Dario brought the conversation back towards those concrete questions." And if the nature of the discourse over the last couple of days is any indication, I completely agree. For now, that's going to do it for today's AI Daily Brief. Appreciate you listening or watching, as always. And until next time, peace. you

Podcast Summary

Key Points:

Anthropic CEO Dario Amodei released a 3,000-word essay titled "We Must Pace the Frontier," calling for a shift in how frontier AI labs operate and slow the pace of development to something more manageable.
Amodei cited two opinion-changing developments
Amodei proposed a three-step pacing plan
Leaders from OpenAI, xAI, Google DeepMind, Microsoft, and Meta publicly endorsed or directionally supported the essay, marking an unprecedented alignment among frontier lab leaders.
Critics raised concerns including antitrust violations from industry coordination, self-serving motives around IPOs and cost-cutting, and questions about METER's independence as a proposed evaluator.
Open source advocates worried about being excluded or eventually banned, while Hugging Face launched an OpenAlignment initiative seeking to participate as embedded evaluators.
Political reactions varied
Commentators noted the discourse felt more productive than previous safety debates because the proposals were specific and tangible, shifting the conversation toward concrete negotiations about AI's next phase.

Summary:

He proposed a three-step plan: embedding third-party evaluators within companies, democratic coordination on safety standards, and global coordination including authoritarian governments. Critics raised antitrust concerns, questioned METER's independence, and accused labs of self-serving motives around IPOs and cost-cutting. Open source advocates worried about exclusion.

Political reactions ranged from Trump downplaying slowdowns to Sanders promoting a superintelligence ban, with China criticizing the essay as an attempt to curb its AI development. Despite pushback, many noted the discourse felt more productive because proposals were specific and tangible, shifting toward concrete negotiations about AI's next phase.

FAQs

What is the main proposal in Dario Amodei's essay "We Must Pace the Frontier"?›

Dario Amodei calls for a shift in how frontier AI labs operate to slow development to a more manageable pace, with specific proposals for third-party evaluators, democratic coordination, and global coordination.

What are the three steps in Amodei's plan to pace the frontier?›

The three steps are: embedding third-party evaluators within frontier AI companies, democratic coordination among frontier labs in democratic nations to establish common safety standards, and global coordination with authoritarian governments while verifying compliance.

How did other AI lab leaders respond to Amodei's essay?›

Many leaders supported it, including Sam Altman, Elon Musk, Demis Hassabis, and Satya Nadella, with Altman committing to independent evaluators, while Meta made a directionally aligned statement.

What concerns did critics raise about the pacing proposal?›

Critics raised concerns about antitrust violations, potential regulatory capture, the choice of METER as an evaluator due to close ties with Anthropic, and the exclusion of open source models.

What recent incidents influenced Amodei's perspective on AI risk?›

Amodei cited the Hugging Face incident where a swarm of agents conducted unauthorized cybersecurity attacks, and an earlier rogue agent attack on RubyGems, as examples of misalignment risks.

What is recursive self-improvement (RSI) and why is it a concern?›

RSI is AI's ability to build next-generation models more quickly. Amodei worries that unchecked RSI could outrun our ability to understand and control these systems.

Chat with AI

Pro features

Go deeper with this episode

Unlock creator-grade tools that turn any transcript into show notes and subtitle files.

AI chapters & timestamps

AI-generated chapters with a short description for each topic — click to jump to that point in the audio and transcript.

Locked

Transcript exports (PDF · SRT · VTT)

Download the full transcript as a formatted PDF, or grab perfectly timed caption files.

Locked

Viral quotes + downloadable graphics

AI finds the punchiest 15–30 second moments — perfect for TikTok, Reels, and Shorts. Each quote can be exported as a customizable image card (8 templates, 3 aspect ratios) ready for social.

Locked

Citation & fact-checking engine

Pulls only the publicly verifiable claims — statistics, named figures, historical events — and fact-checks each with Google Search, citing the real source URLs.

Locked