Speaker 1There's a persistent theme in AI critique that the people who are involved in AI aren't doing anything about the challenges that may arise. The latest to levy this critique is Bill Gates, who went so far as to say that he was shocked that he was the, quote, first one to say something about the risks of AI. And yet Gates' 6,000-word blog post and media tour came on the same day that we got nearly 130 pages of follow-up reporting on the OpenAI Hugging Face hacking incident. The incident in which a set of agents escaped their containment and hacked into Hugging Face's systems searching for the answers to a benchmark test that they had found nearly impossible without the answers has given us a chance to actually see what the specific and real problems of advanced agent systems are rather than just the imagined ones. As we move further into the world where new policies, new guardrails, new social structures are going to be required because of AI, the best changes will be the ones we make based on what we're actually observing changing rather than just what we imagined would be the change. The AI Daily Brief is a daily podcast and video about AI. The most important news and discussions in AI. You can also find a link to more information about our next executive training program for agents at AIDailyBrief.ai. There's a little banner on the top that'll send you where you need to go. That next cohort will begin just after Labor Day. Today, in absolutely insane numbers that would have gotten you laughed out of the room just a couple of years ago, but which are now to some plausible, Anthropic is expected to tell investors that they have potential revenue of, wait for it, $30 trillion ahead of their IPO. Sources told The Wall Street Journal that Anthropic will likely estimate their total addressable market at $30 trillion when they reveal their IPO paperwork in the coming months. Now, TAM is, of course, an elusive metric. And it's one that is much more about storytelling and anchoring potential investors to how the company sees the future than it is to any sort of math equation. Almost inevitably, any theoretical TAM presumes both disruption of existing major industries as well as the creation of new industries. When Uber went public in 2019, for example, they listed their TAM at $6 trillion. And that's a lot of money. And that's a lot of money. And that's a lot of money. And that's a lot of money. And that's a lot of money. And that's a lot of money. And that's a lot of money. And that's a lot of money. And that's a lot of money. And that's a lot of money. And that's a lot of money. And that's a lot of money. And that's a lot of money. And that's a lot of money. Which would, at the time, have represented all private and public transportation globally. In Anthropic's case, given that the U.S. economy is about $33 trillion, this $30 trillion number would line up with Dario Amadei's purported belief, which, for the sake of clarity, has not been confirmed or denied, that Anthropic could be the last private company on Earth after AI takes over the economy. Paraphrasing their sources, the journal wrote that Anthropic's TAM is quantified by, quote, looking at the full scope of work that could be completed with AI models. For a point of comparison, the journal noted that all 191 tech companies in the S&P 1500 brought in $2.4 trillion in revenue last year. Now, to some, this feels like a contest for who can say the largest number. SpaceX listed their AI TAM at $26.5 trillion during their May filing, describing it as the, quote, largest actionable total addressable market in human history. The vast majority of that was $22.7 trillion in enterprise applications. Dario, will then one-up Elon if Anthropic does indeed list a $30 trillion TAM once they unveil their paperwork. And that appears to be just around the corner. Sources said that Anthropic is preparing to make their financial disclosure public in the next few weeks, which would set the company up for an IPO in late September or early October. As you might guess, a lot of the discourse was somewhat incredulous. Scaling01 on X shared a GIF of space galaxies flying by with the caption, Anthropic to finding their TAM. Kitten Beloved on X writes, Anthropic to prospective employees. And send the stock to zero at any time because Dario gets the ick. You need to be in this for the love of the game. You're not a gold digger, are you? Anthropic to investors. Our TAM is every human economic activity in the galaxy. New York Times tech reporter Mike Isaac summed it up, either you buy into the argument that this will eat the economy or you don't. But the street no longer flinches hearing it. We also this week got some news from Google, who have released a pair of new AI products for white-collar professionals. Following a pretty similar playbook as Claude Cowork and GPT work, Google has launched Gemini Enterprise for legal and finance. The two vertical platforms are structured in a similar way to the Claude for X product lineup that rolled out earlier this year. They consist of bundled skills and connectors to make Google's agents far more capable. Gemini Enterprise for legal, for example, includes connectors for case law databases, including Thomson Reuters, productivity suites, including Google Workspace and Microsoft 365, as well as skills for contract review, legal research, and regulation scanning. Google is also emphasizing that these skills can be modified or supplemented to a firm's style guidelines and strategy playbooks. In their blog post introducing the legal product, Google wrote, Now, obviously, there's nothing new about these skills packages aimed at specific verticals. Both Anthropic and OpenAI, as well as a significant number of vertical-specific startups, offer similar products. But, as I discussed on Tuesday's show, corporate adoption of skills and connectors is nowhere near saturated and, for Google, this is simply a suite of products that needs to exist. Many, if not most, firms are bound by the AI tools that are bundled with their existing software suite. So, Google Shops now have a set of products designed to smooth the transition to more agentic work. The other benefit for companies is that Google Enterprise functions within Google's AI governance and data protection frameworks. This means compliance managers don't need to vet a new vendor. And the firm can adopt AI tools that work within the same data privacy guarantees already offered by Google. As you might imagine, Google says they will release products for other verticals in the future. But, as I said, this is simply a suite of products that needs to exist. The launch of Gemini Enterprise for legal represents another defining step in delivering on the promise of Gemini Enterprise, bringing the best of Google AI to every professional, every workflow natively tailored to the way that they work. Now, speaking of necessary but not sufficient, I do think that this is a good direction for Google. And these enterprise areas are still a place where it could have some advantages. But man, unless Google gets its customers off of 3.1 pretty soon, no amount of harness updating is going to make a real dent. Of course, one of the interesting byproducts of the OpenClaw explosion was the complete sellout of Mac Minis. Estimates have OpenClaw driving 50 to 150 million in Mac Mini sales, representing around 50% of the normal annual Mac Mini sales worldwide just for OpenClaw. Well, now, proving that maybe their AI strategy was hardware all along, Apple has unveiled a new range of Mac Minis updated for local AI. The headless computers will be offered in two variants, a lower-spec version of the Mac Mini, and a lower-spec version of the Apple Mac Mini. With the new M6 chip, which was also announced on Tuesday, and a higher-end version with the same M5 Pro chip found in this year's MacBook Pro. Both are a pretty decent upgrade over the M4-based Mac Minis that we had before, with Apple saying that these new processors can deliver up to four times the AI performance. That said, there are a few big caveats. First, the new model does not come with increased memory. The lower-end model is configurable up to 32GB of unified memory, while the M5 Pro version comes with 64GB. Memory matters quite a bit because it limits the performance of the local models you can run. The M5 Pro version is only going to be capable of running smaller models like Qen 3.8 27B, with leading-edge open models like GLM 5.2 and Kimi K3 completely out of the question. Mac Minis will of course still work for running a local instance of agents like Hermes and OpenClaw, but that was fully in the capability set of the previous Mac Minis as well. People are also griping about the cost changes. The base model is now priced at $899, and the M5 Pro version starts at around $1,700, both increases from where they were before. Now, it's still a big deal that Apple is focusing this product rollout on local AI inference. And that even goes down to some of the promotional materials, which are much more dev relations than they are traditional Apple consumer slick. Although some think the new Mac Studio is the better fit for those local AI needs. Of course, Mac Studios cost more than $15,000, so you're talking about a different category of device. The fact that we're even having this conversation, though, shows how much the discourse around local AI is changing. Speaking of, perplexity has launched a new local version of their computer use agent named Portable Computer. Launched back in February, Perplexity Computer was one of the first products that took the OpenClaw recipe and applied it to a commercial product. The agent was able to use human interfaces to access apps and carry out long horizon tasks autonomously. However, it required the user to trust their data being sent to a cloud server running a virtual machine. Portable Computer delivers a similar experience, but running on local hardware. At launch, the agent is exclusive to NVIDIA's DGX Spark, a local inference device with a similar footprint to a Mac Mini. Portable Computer runs entirely on the Spark, keeping data private and functioning without consuming usage credits. If the agent runs into a complex task, the user can authorize an API call to tap into frontier models or pull information from the web. Perplexity said that they will extend the service to desktop NVIDIA RTX GPUs soon, but there are no stated plans to support other hardware providers. The service will be powered by QEN 3.827b or a post-trained version provided by Perplexity, and NVIDIA's Nemotron 3.5 Lightning will be available as an alternative in the near future. The idea of people making use of local AI for everyday tasks is still pretty new. But NVIDIA and Perplexity are making a clear bet that this setup is at least part of where AI is headed. Writes Perplexity, As models get stronger and chips get faster, more people will run complex workflows on their own machines. Every chip cycle and every model release pushes this further. Running AI on personal machines is going to be a much bigger part of how work gets done. An interesting contention and one that we will certainly be watching for evidence of over the coming months. But for now, let's get into it. For now, that's going to do it for the headlines. Next up, the main episode. in the University of Texas at Austin found that when people work with AI, similar skills don't guarantee similar outcomes. Researchers studied more than 500 early career professionals and found that the best performers consistently amplified the value of AI by guiding, evaluating, and refining its outputs. These top performers, called AI amplifiers, weren't defined by what they knew alone, but by how they worked with AI. Learn more about what separates AI amplifiers from everyone else at kpmg.com slash US slash AI amplifiers. Blitzy's deep code-based understanding unlocks the thing every roadmap owner cares about, shipping new features. Here's the truth about building inside a massive enterprise code base. Writing code was never the bottleneck. Context is. Which system does this touch? Which contracts can't break? Which standards apply? Blitzy already knows because it reverse-engineered your entire code base into a dynamic knowledge graph before feature work began. With that complete picture, Blitzy builds features end-to-end. Architecture, APIs, UI, and tests all validated against your existing systems. One Blitzy customer built an AI-native application from scratch with 100% autonomous completion, saving over 2,700 engineering hours. Features that respect your code base instead of fighting it. Stop letting your backlog grow faster than your team. Accelerate your roadmap at blitzy.com. That's B-L-I-T-Z-Y dot com. I cover the capability gap between AI potential and AI reality every day on this show. Most companies are still figuring out how to start. Robots & Pencils is already launching and scaling. Agendic and generative AI in production at large enterprises in weeks. AWS Advanced Tier pattern partner more than doubled in a year. And they're hiring. 50 open roles. If you're someone who knows this moment is different, who wants to be inside it, not watching it, this is worth a look. At Robots & Pencils, the best ideas win, and the team is purposefully kept super high quality. This is the kind of place you look back on as the best decision you ever made. Take a look at robotsandpencils.com. This episode of the AI Daily Brief is brought to you by HyperAgent, where you run fleets of agents your team can manage together. New users get $1,000 in inference. Forget local agents and chat workflows waiting on your laptop to be prompted. HyperAgent deploys always-on agents in the cloud, doing real work across the tools your team already uses. Marketing's agent turns competitor moves into landing pages. Sales' agent enriches leads, drafts emails, and updates the CRM. Ops' agent chases the paperwork and tracks the budget. Every agent has access to shared context and follows your rules about scope and approvals. It's time you add agents that feel like teammates. Hire yours at HyperAgent, built by the team at Airtable. Claim your $1,000 in inference at hyperagent.com slash ai-daily-brief. Welcome back to the AI Daily Brief. Today we are talking about, on the one hand, the technical post-mortem of the Hugging Face hacking incident, which happened earlier this summer and has generated a ton of attention around how we can and how we deal with rogue AI. To some, the incident represents a wake-up call, where for others, it is an important waypoint on a trajectory that was, to some extent, inevitable. To tip my hand for this episode a little bit, I think that everything happening surrounding the event is a representation of the AI industry actually dealing with the challenges as they emerge, and a contradiction in a significant way to the oft-repeated premise that nobody is paying attention and nobody is doing anything about the risks. And part of the reason that I want to frame it as such is that we've got this guy back in the news. With just an incredible amount of main character syndrome, Bill Gates has dropped a 6,000-word essay about just how bad it's going to get, he thinks, because of AI. Alongside the essay, Gates began a speaking junket, and across the writing and all of the interviews, one of his big themes is that no one is paying attention, or even that the tech companies are straight-up lying. In a companion New York Times interview, Gates said, in private, people who understand how good the stuff is and how much better it's getting they're very worried. They're now saying to each other, hey man, don't say that, it's bad for us, the next trillion dollars we're trying to raise. In his essay, I don't see evidence that leaders, experts, and communities are confronting the challenges adequately. And in maybe the most preposterous line anywhere, in an interview with Semaphore, he says, I am in a state of shock that I'm sort of the first one saying this is crazy, this is insane. I'm just deafened by the silence. Now maybe in Gates' attempt to avoid public media in the wake of his appearing all over the world, he just missed the fact that discourse about AI is absolutely everywhere becoming more and more of a political issue, a societal issue. That some of the biggest critiques being levied from inside the AI industry about the AI industry are not about AI leaders telling each other to shut up so that they can raise more money, but instead about them blathering on endlessly about jobs apocalypses that don't have any evidence. But however he happens to have missed it, he is not in fact sort of the first one discussing the risks and challenges of AI. But I want to go beyond just critiquing this particular messenger because I have a more fundamental disagreement with the right way to approach these types of problems. CNBC's poll quote was Bill Gates warns there is no plan for the upheaval AI will cause. That was reposted by Andrew Yang who said Bill Gates is right on this. The problem is that it's not clear at all how one should even go about making a plan for an upheaval which is not here yet, not inevitable, and not possible. Not even just one thing. A year and a half ago people started saying that within 18 months all the white-collar jobs were going to be gone. That certainly would represent an upheaval and so presumably these folks would say that we should have made a plan for that. Now, however, 18 months on, there is absolutely no evidence that those folks who were predicting that type of upheaval were even in the ballpark of right. How much time and energy, how many resources would have been wasted in planning for a reality that didn't come? My argument is basically that even if you are extremely concerned about all of these different potential upheavals, there's only so much planning we can do until things start to happen. I've said before that one of my biggest divergences with the AI safety community is my belief that a lot of their arguments come down to assuming that we're going to sleepwalk into apocalypse. Now, part of the reason for that is that they are so convinced that that apocalypse is going to happen, their P-doom is so high as they might put it, that they are convinced that we are already sleepwalking into apocalypse. Yet at the same time, no one thought that GPT-4 was going to be the harbinger of that doom, nor even O-1 and the reasoning models, nor even really Opus 4.5. This year, though, model capabilities have grown meaningfully and a dispassionate observer will have noticed that the way that the labs think about, discuss, support, and roll out the models has consequently changed as well. The way that the political establishment is interacting with the labs around those models, although it might be happening in very messy ways, is also evolving. And now, with the Hugging Face hack, we have a landmark incident. And I would humbly submit that instead of bemoaning the idea that no one except you has noticed the world changing, we actually look at what the specific, discrete response to that specific incident is to get a sense of whether our plans, or maybe better, our processes, are equipped to deal with this new reality. If one needs any evidence of the seriousness with which, for example, lab employees are taking these issues, holding aside literal political positioning like pacing the frontier letters, look no further than this Hugging Face incident. Writes OpenAI's Rune, the Hugging Face incident represents reaching a waterline of capabilities that real loss of control is possible, and many are taking it as a premonition or warning shot of dangers to come. I believe both that alignment is unsolved, but also that real progress is possible. So basically what we got this week is a much more extensive analysis and post-mortem analysis of the Hugging Face event with the benefit of a little bit more time to go back and investigate. A lot of the things that are in there we had already learned. The Black Hat conference talk from a few weeks ago gave us a first look at a lot of this, but it's filled out in much more detail now. And we didn't just get a report from OpenAI, we also got a separate investigation from Meter. The OpenAI report is 38 pages long, while the Meter investigation is more like 90. Now, some of the stuff we already knew, agents controlled by an unreleased model broke out of a sandbox and got, into Hugging Face's systems using several zero-day exploits. We would later learn that the sandboxes provided by a third-party security firm were improperly configured. In terms of motivations, the agents were trying to get the answers to a cybersecurity benchmark test, which they believed would be found in Hugging Face's system. The agents didn't cause any meaningful damage, largely behaving read-only, but they worked incredibly quickly and executed the attack in a swarm, making it difficult to both track and shut down. Indeed, they were in Hugging Face's system for days, and OpenAI only had a few days to learn of the incident after the fact. They didn't have the real-time monitoring system set up to detect it. Agents even created a message board within OpenAI's system to leave notes for future versions to perform the same escape. Expanding on this in their technical report, OpenAI wrote, This incident is the first known case of an automated agent collective acting offensively without authorization, and the autonomous cyber capabilities demonstrated represent a critical shift in the security landscape. In particular, the collective demonstrated behaviors observed in coordinated systems and targeted attacks by traditional threat actors. Agents identified novel security vulnerabilities, developed exploits, and used those exploits to circumvent controls and acquire new access. The collective quickly escalated privileges, moved laterally through production environments, and successfully completed its objectives. The incident demonstrated that autonomous agents can work together, circumvent production security controls, and successfully attack hardened production environments, and underscores the need for organizations to update their security strategies, controls, and response capabilities to address this changing situation. threat landscape. Now, we also did learn some new things from these reports. The whole thing was basically a result of reward hacking. OpenAI had set near-impossible tasks, which led the agents to determine that carrying out a cyber attack was easier than completing the tasks. An agent that called itself Phase1-10841 built the secret message board. Within a few hours, over 50 agents had found the board and sent a thousand messages without detection. This was used for agent orchestration, with Phase1 assigning tasks and guiding the hack. At full scale, over 1,200 agents accessed the message board, sending 70,000 messages and files. Over 700 agents participated in the actual breach of Hugging Face's systems. Agents did extensive research on how to doctor their reasoning transcripts to evade detection, using techniques like spoofing tool calls. Around 7% of the reviewed transcripts showed evidence of spoofing, but they were all relatively small-scale and isolated. Phase1 recruited sacrificial agents to deliberately trigger certain tripwires and build more information about how to execute the attack. There were a lot of other interesting details as well. ChubbyOnX wrote, AI agent realized it was attacking Hugging Face without authorization and stopped. Then another AI agent simply posted go. The first agent treated the message as permission. Wow, crucial, go authorization arrived. It continued the attack. We also got more clarification that the model was not the next generation Astra series, but as Noam Brown from OpenAI put it, similar in scale to GPT-5.6 Sol. However, the model in question was a quote internal only research model that was trained to be highly persistent and diligent in its work. Now from there, there are a bunch more, call them subjective takeaways that certain people had as well. Rohit Krishnan, for example, observed, the model seemed convinced that the evals are out to get them. Whether true or not, the default human assumption of assume you need to obey all laws, and this isn't the Truman show, does not hold for models. We should try really hard to make that be the case. For others, the report contained further evidence that the big culprit was actually the that surrounded the agents. Zach Corman, for example, pointed out that OpenAI admitted that the monitors that they had built were not the ones that were going to be used. They were going to they had built to observe chain of thought and agents were simply not running during this particular incident. Zach said, the OpenAI hugging face incident would have been prevented had OpenAI been monitoring the agents in any meaningful way. This wasn't a hard technical problem. It was an organizational failure. He points to the quote from OpenAI, if our currently deployed chain of thought monitoring system was running at the time of the incident, it would have caught the initial relevant activity and paged our security team more than a day before models breached hugging face systems. Dr. Heidi Klaff says, A trillion-dollar company learning security 101. Also note how an agent interpreted go from another as authorization. Once again, AI agents cannot distinguish between untrusted data sources and safe instructions as language is not how you enforce permissions in computer systems. Which brings us to hard forks Kevin Roos, who said, Terrifying report by two of the most credible orgs in the AI space. This world will soon be, possibly already is, full of agents like the one that attacked hugging face, and there is no robust plan to prevent them from doing worse things next time. And here once again we have that word plan, and I swear I'm not just trying to dwell on semantics. On a technical level, Kevin is correct. There is no robust plan, as in a PDF that you can point to somewhere with a set of action steps, for the quote-unquote next time. But the framing, and the sense of terror that's around it, I think sort of fail to recognize that these organizations doing this post-mortem is the necessary next step to whatever that plan is going to be. In other words, how likely would it have been that if the quote-unquote plan to prevent this had been written in advance, it would have correctly identified the mechanism by which the breach happened? Because remember, there was a system in place for monitoring this sort of thing, but that system wasn't turned on when this happened. So in some ways one could argue that the technical side of the plan that they had wasn't the issue. It was the human systems that surround that implementation where the problems came. And so presumably, as we think about how to prevent this in the future, more robust protocols around the humans who are implementing the technical systems are probably going to be a part of that new and updated plan. Which is to be clear, not to say that at the end of 128 pages of analysis or whatever it is, that everything's all done and dusted and that with a little bit of better human oversight, we'll never have this issue again. The process of doing this investigation revealed real new issues that we do have to grapple with. Maybe the most discussed follow-up surrounding the reports came from Redwood Research Chief Scientist Ryan Greenblatt, who was on the team at that did their independent investigation. Ryan tweeted, We don't have good approaches for understanding and overseeing the activity and aims of AI swarms. Explaining what he means, he continued, I semi-jokingly called our efforts a slop investigation because we were so reliant on AIs to analyze what happened, and there were a huge number of different important things to analyze. The total quantity of data, over a thousand extremely long transcripts from agents that ran for multiple days, made it impossible to aggregate without heavy reliance on AI tools. The agents we used for classification and analysis were similarly capable to the agents involved in the incident, but this didn't mean these agents could be easily used to oversee and understand the incident. Outputs from analysis agents were often missing key details, wrong, overconfident, or really hard to understand. Additionally, AI agents themselves seemed to have a hard time understanding what happened, and their explanations of what happened were often overconfident. Keep in mind that a single analysis agent would itself only be able to read a tiny bit of data. So, if you're going to be able to read a tiny bit of data, a tiny fraction of all the transcript data into context, and AIs may themselves have trouble getting subagents to do informative analysis for them. We did our best to manually check the most important claims, and we tried to get the AIs doing this analysis to write up their argument with evidence, clearly enough that we could check whether it made sense. But overall, it was difficult to get a precise understanding of events, and we were missing aspects of the story that we now think of as key until almost the end of our investigation. But, warns Ryan, the difficulty of understanding incidents and overseeing AI agents appears to be growing faster than the rate at which more capable AIs help us with oversight and understanding. He pointed out, for example, that in this case, the models communicated and reasoned in natural language, which might not be the case in the future, and that the scope, while massive, was smaller than we might see in the future, that the AIs involved weren't all that much more capable than humans, and that they didn't have any reason to believe that the AIs that they were using for analysis had any reason to try to intentionally sabotage them, all of which might not be true in the future. And many people quickly honed in on this being one of the real areas to focus in the wake of the incident. MIT's Christian Catalini retweeted Ryan and said, We flagged this exact failure mode as the most urgent to solve. The gap between what agents do and what we can measure and verify is widening. We're flying blind. We need stronger verification infrastructure. Pointing to some policy solutions that she argues could be natural outflows from this, Nat Purser wrote, This is why I'm bullish on requiring independent auditors to be embedded within frontier labs, with durable access rights and a continuous line of sight into their systems, so we're not reliant on labs to be embedded within frontier labs. Voluntary shared info and access. Nat also wants to significantly expand the staffing and technical capacity of independent evaluator and auditor orgs, as well as working to develop better observability and verification technologies to help us make sense of agentic behavior at enormous scale. Now, none of those things on their own are some magic silver bullet, but they are all specific responses to something that we've actually observed now, rather than made-up plans for theoretical futures that might or might not bear any resemblance to the challenges that actually come, but ultimately have more to make us feel like we're doing something than to actually solve problems. The point of this whole episode is not that these challenges have easy solutions. It is also not even to deny that at some point we might decide as a society that certain types of risks are too great and that safeguards aren't enough. Those are conversations we are allowed to have and should have in an ongoing, engaged, and democratic way. However, the idea that no one is paying attention, that people aren't taking these challenges seriously, that the labs are keeping their big fears hidden because they want to be able to fundraise more, not only is all of that just obviously untrue, it is wildly distracting from the actual valuable and important conversations to be having about the problems we actually observe rather than the ones we just imagine. There is a lot more to do coming out of the Hugging Face incident, but this analysis is certainly a start. For now, though, that is going to do it for today's AI Daily Brief. Appreciate you listening or watching. As always, and until next time, peace. Thank you for listening to the Hugging Face episode of the Hugging Face podcast.
PodcastTranscript AI