#260 - Muse, Misalignment, Math

Hello and welcome to the last week an AI podcast where you can hear a chat about what's going on with AI. As usual in this episode we will summarize and discuss some of last week's most interesting AI news. I'm one of your regular hosts, Andre Karankov. I currently work at the AI startup Astrocade and was at grad school for AI before that. And everybody, I'm your other co-host Jeremy Harris from Gladstone AI, do AI and national security stuff, especially AI China and game, actually have a newsletter now. So AI and game.com, you can check that out if you want. We're probably by the time this podcast drops, either we will have just released our latest post server versus engineering with a couple people in the Frontier Labs, what recursive self improvement looks like, everybody talking about this obviously today, we just wanted to make sure that someone somewhere had kind of, I guess, described at the gears level, what the process will look like inside the Frontier Labs from the perspective of the researchers themselves who are doing it. If only for regulatory reasons, you need to be able to define something and understand it to regulate it. So this is kind of an initial attempt to do that. And just generally to trace out some of the espionage and national security implications of that curious level picture. So if you want that, check it out. We already have a post up about U.S. China and kind of how we see the end game play out there. More of a diplomatic post, some of the things we've touched on on the podcast, a bit of a deeper dive. And we'll probably do one at some point fairly soon, I think, on that same topic. Yeah. We have a lot to dive deep on these days. Used to be, I'm nostalgic now for 20, 25, 20, 24, you know, it seemed like things are crazy back then, but things are really starting to get crazy and they're only going to get crazier by all appearances. Well, and we don't have a story about this per se, but Jacob Coxen was on the daily show recently. I got a text from a buddy of mine passively following the space, probably following the space more than the average person. And he just said, he wrote like WTF, like, look at this, which I thought was so interesting because I thought that the daily show was so late on this one. I mean, the, you know, the Coxen tweet, how long is it, it's been like, you know, three weeks or something. It's the Coxen tweet, they're obviously there's been the hugging face thing that's been months and months ago. We've got a whole bunch of things. So I guess it's a way in which like a lot of us are following it. If you're listening to this podcast, you're probably not aware of how little the average person is following this stuff is, again, this buddy of mine's really bright dude. He's in startups and biotech and following this stuff. And yet he, he still hadn't registered the Jacob Coxen story. So I just find it's fascinating, you know, like the future is here. It's not evenly distributed and often for, for funny reasons. And the information of the future being here in some concerning ways and in some, I guess, exciting ways is also not equally distributed. And there's a lot of skeptics, even who are sort of like doing counter narratives. And anyway, but to focus on what we have today, it's actually going to be a bit of a quieter episode. We've had quite a stretch of like really intense ones. And today there's not like a ton of exciting news. We'll have, you know, a bit of a mix of a couple tools, stories, a couple business. Really, the coolest news is about open source models rather than usually for once we'll get around to talking a bit more about research as well. So it should be, you know, pleasantly unexciting episode in some sense. We'd like to thank Outshift, Cisco's Incubation Engine for Frontier Technology for supporting the show. The agent AI is in every enterprise roadmap right now, but here's a problem we're all facing. Agents can pass messages, but they cannot think together. These systems soundly underperform through cognitive failures, like misreadings and verify claims and false consensus that conventional monitoring cannot detect. Outshift by Cisco is building the fix, the internet of cognition and open source foundation for building multi agent systems from design to production. We've shared context, shared memory and guardrails to drive results we actually expect from AI. With the paper, experience a demo and grab the code from outshift.com. That's OUTSHIFT.com. We would like to thank Box for supporting the show. If you're trying to adapt AI in your team organization, you're probably not getting great results if all you do is call a chatbot. A chatbot was strained on the entire internet. It wasn't optimized for your specific business or team. And even if you provided all the context and input files for it to help you over your work, when I set outputs, some summary you would still presumably need to go talk to stakeholders or route its outputs to this kind of apps, the workflow will still be slow. That's where Box comes in. Box is building the intelligent content management platform for VAI era at the access of a secure content foundation where AI agents don't just access your unique institutional knowledge, actually orchestrate end-to-end workflows across your business critical systems. So it's more than just talking to a chatbot. It's about putting it out to work on document-heavy processes, like putting structured data out of instruction files, multi-step routing, dynamic document generation, Box helps your business turn manual document bottlenecks into automated, repeatable business outcomes. And all that comes with a governance layer built in, enforcing granular permissions, maintaining it on the trail for every agent actions and keeping humans in a loop to review exceptions before critical bounce stream actions occur. If you want to move beyond basic AI Q&A and put autonomous AI workflows to work for your business, visit box.com/LWIAI or a general team at Boxworks in San Francisco on November 5th and 6th, use code LWAIAI for 50% off your registration. So kicking it off of tools and apps, we've got OpenAI is sticking more ads in chat GPT. They're rolling out a new visual ad format in chat GPT, where displays images of sponsored products and services on screen with US testing beginning later this month. So they'll initially appear when users generate images with chat GPT and will remain separate from the generate image itself, which will be very funny if you just inserted products in your generated images. This is following up on ads starting rolling out in February. So I've had ads to various extent now for most of this year. Still, I haven't seen much discussion or many sort of screenshots. I don't use chat GPT myself that much, so I don't have any personal experience of it. But it seems like right here to stay. Which is not surprising, again, given OpenAI's business model, I think the big question is when do we get to the point where ads are mostly targeted towards other agents? Because right now, OpenAI obviously has got a bunch of humans on their platform. They are the B2C chatbot agent company. Well, I would have said that, but then of course we have Muse dropping, which I think actually might be an interesting challenge for OpenAI from the agentic side, which is what really matters. Anyway, we'll talk about that, but Meta certainly targeting the sort of payment layer of the agentic AI stack and trying to make sure they can monetize transactions made by their agents. This is, in some ways, if you're super, super AI scale-pilled and you believe what a lot of the researchers at OpenAI believe, this has to be viewed as a temporary strategy. Humans are not going to be the dominant drivers of economic productivity and therefore the dominant consumers of stuff into certainly the next decade, but even into the next few years, according to a lot of the people in the labs. It's kind of, I'm interested in that this is an obvious monetization play. It's necessary for them right now, but I'm really interested. I would love to talk to somebody on their strategy side to understand how does this fit in? How do you see this adiabatically tailing off as you kick off the era of agents doing most of the work in the economy? How does this fit into that picture? It's interesting. I think to be fair, ads are for consumers typically and it's a mix, so to some extent, if you're shopping around for business tools and so on, there is that, but if you're shopping around for, like, plus sheet toys as gifts, you might still be buying that as a human. Maybe in the near future, this demo, this mythical demo of booking a flight via AI for you will actually happen where the level of laziness of humans will start to increase and dependency and AI will increase such that at some point you just like book me a flight on this and this and it'll figure it all out. Yeah. I guess I just, I think of this as like, if agents are the dominant driver of economic productivity or automation, generally, pretty soon a project like this starts to look like a legacy product and I wonder if eventually just like won't be worth advertising quite so much. I'm generally really curious because it seems like exactly the kind of thing that you would do if you were less scale-pilled, acknowledging that there's a need in the near term. Like, I'm not saying that openly, I isn't because they need to make money today. It's just interesting to note how at odds this is with like, you know, you've got bigger fish to chase after if you really believe in the agentic future, you would be thinking about how do we do ads for agents? And this is kind of where my thinking goes, is this maybe the thing out of which will grow the agentic advertising model? Agents just have so much more attention to buy and they need to use so many more tools that I would imagine that'll be really important. I'm curious if there's some kind of like jumping off points strategically where you can actually kind of bridge that gap a little bit. I'm just again, trying to think about how does this fit into the strategic vision? It's possible it just doesn't and it's pure money and like, I'm very open to that possibility. But as just seem like an unsexy thing that might actually be more important than it seems, directed slightly differently. Anyway, so I'm just calling that out. I wouldn't be surprised to hear that. It's an interesting question. Yeah, I found myself more and more recently being like, I just have no idea what anything will look like in a year. or two years, right? So these guys are things that are entirely plausible. It might be completely different, like I just don't know. And on a related story, next one is TikTok rolls out an AI shopping assistant and one click checkout. So they've announced this as something launching, as well as this feature that lets users buy directly from brands. The shopping assistant is described as a conversational AI agent that helps users discover and purchase products. It can understand context and remember users' preferences and needs throughout a conversation. And it provides real-time guidance, such as product details, shipping information, sizing, availability, and help completing a purchase. Which, to my knowledge, tragedy doesn't have, and that seems like a natural fit with the ad strategy 'cause you can get a commission or whatever. So yeah, quite a related note there of, as you get more assistance in shopping and some of it goes through AI's or a lot of it goes through AI's systems in particular, that's a big slice of revenue that these kinds of companies will probably go after. - Yeah, yeah, it's sort of similar thoughts as before and I won't go on the rant again, but the same rant. But we're kind of slowly gonna ease into a future where ads are more about agents than people. And this is somewhere kind of in between, right? You're sort of steering guiding the behavior of the user. I think it was in Chaos Monkeys. So Antonio Garcia Martinez, the BRC tier famous in general and in Silicon Valley, A tier famous author of Chaos Monkeys. Really good book, you should check it out. It's kind of dated now, but it's fascinating. One of the things he talks about, so he was instrumental in setting up meta's advertising platform. His philosophy with respect to ads, which is not unique to him at all, but he articulates it well. It's like ads at their best are a tool for the consumer. You're being surfaced ideally, things that you would have wanted to know about. And you can kind of see this if you're an old curmudgeon like me and the only kind of social media that you use for stuff other than work is believe it or not, I don't actually use it, but just like I use Facebook Messenger. All right, that's how old I am. And when you're on Facebook and you see the feed, like a lot of ads there and on other platforms have just turned into content. Like they are literally just entertainment now. And you sort of see this universally in a spectrum of this happening, ads want to be ultimately. Like it's of no use to charm and if they're advertising toilet paper at you and you have no intention to buy toilet paper ever. That's no use to anybody. In the ideal case, they are surfacing the exact thing that you want. It's something that you would rather see than not to some degree. Anyway, so this is kind of along that continuum. It's framed as an assist to your purchases just to make things faster. There's a continuum between assists and advertisement and commissions and incentives obviously or probably should be part of the definition there. But I just think it's really interesting that this is all gonna have the same old debates at the level of agents. What is slop? What is advertising slop? What is low quality advertisement to agents that just choose up their tokens for no reason? And literally costs you as the end user tokens that get burned on looking at context that's useless. Anyway, I'll park it there. But I think there's like an entire agent economy that's gonna already, that's probably starting up now. I need to check out the latest YC demo day to see what people are building in that space. - Well, I'll add a quick note which is interestingly, I think the comparison there is geo. So for the longest time, you've had SEO, right? And if you're a business, you need to make sure that you shop and search results on Google. And then you also need to pay for ads to compete with your competitors. But now there's this notion of geo, generative engine optimization so that if you ask Chad GPT about what is the best maker of TVs or whatever, I mean, how does the agent know? I don't know, right? So in some sense, we already optimizing for agents in the realm of search more and more. It's becoming more important and we can expect how to happen in other domains to your point. Moving on to applications and business. We begin with Meta's Muse, tops 5 million downloads, fast-term and Chad GPT and Claude. So we covered this a little bit last episode. This is a bit of a follow up where Muse appears to still be doing quite well. It's currently the number one app on Apple's App Store and Google Play. It's not an all-time high, but it's still taking off quite fast. Meta is heavily promoting it. So Muse took 50% of Meta's daily in-house ad impressions over the last couple of weeks. That's wild. But it's clearly like they invested heavily into making this a big launch and a big deal and it appears to be paying off. And it's coming at a time where they launched it and open AI launch dots right after Google has had Gemini Spark in the background. Nobody really cares or knows about it. Maybe they certainly didn't try to make a splash with it is what I'm saying, right? They didn't make a little character and a whole app around it. Yeah, but no, no, no, no, Q character. I just, some of this stuff really gets me riled up when I see Google launch stuff without a Q character behind it. It makes me angry in a way that just like-- What are we doing? Yeah, yeah, it's shameful. And in any case, so this is clearly an expanding priority of the business. I think from a business perspective there's various interesting questions of how does Muse play into the revenue streams? Are we going to see like a Chad GPT-esque ad platform presumably on Muse? But Muse is also always online, right? This is an assistant. So if you're on the free tiers, you're going to be like sending you messages about, hey, I think maybe you're running out of eggs and you should maybe buy these eggs. Yeah, on a related note to the ad's discussion with these agents kicking off and assuming we stick around, we're sort of entering this next paradigm where Chad what's our mainstream? Always online AI assistance or AI personas, or whatever you want to call it, is a new thing. And seems like the logical next step. Personally, I haven't found a need for this and I don't know people who have really adopted it intensely, but with the intense interest in OpenClaw earlier this year and various kind of applications of it, now we're seeing the consumer version of OpenClaw in some sense. And it'll be very interesting to see if it sticks around in the same way that Chad GPT has. Yeah, and I think that's one of the key questions here, right? 'Cause we are seeing these download numbers that are wild, but you can get high downloads from just curiosity, right? And not ongoing use. So we'll have to wait to see what the actual DAUs and MAU numbers look like when we actually get that data, which I mean, I assume at some point meta will share some numbers if they look good. It's an open question as to where it goes. And in some ways it feels like we're in the GPT-3 era of a lot of the agentic stuff, right? And in the same way that we were in the, well, I guess OpenAIs first coding model was called Codex. So I can't just call it Codex 'cause that's also the prep. But anyway, we were in the DBT-3 era of coding models for a while and then suddenly Clawed Code cracked the code. And this has been known to happen, right? So Chad GPT Clawed Code, there will be a moment like this for agentic systems. It's basically inescapable because, as you said, we think about always on the number of applications where that supports are just, it's just insane, right? And you couple that to the insane time horizons. If we had a meter plot to point to anymore, which we don't because the models are so good at long-term tasks that were literally off the edge of the map there, you would see the value there. But clearly, there's enough value to be tapped here at massive scale. Someone needs to crack the form factor, all the boring design choices that need to be made to make these products appealing not just to developers, but also to everyday people. And I think that'll be pretty violent once it happens. I mean, in the same way that Chad GPT was violent and the same way that Clawed Code was violent, you have all the naysayers who will say that this is overhyped and then suddenly, those voices will be silent and then we'll move on to the next thing. I think we're just waiting for that kind of that moment. - I feel like I would choose that toward a violent, but certainly-- - Yeah, violence-based, right? - Yeah, yeah, yeah, yeah, very rapid growth. Yeah, that's right, no windows will be smashed. - Oh, so maybe-- - Very rapid transformation of a whole bunch of our lives or many people's lives. - Metaphorically violent. - Yes. - Right, right. And Unrelated Note, actually, we've also got the associated story that Meta has joined with a group of companies to tame the quote chaos of doing business with AI bots. So this is talking about Meta, Walmart, Stripe, and the handful of other companies are publishing what they're calling a quote personal agent protocol. It's an open standard meant to dictate how AI agents can interact with businesses. So in a way, it's similar to the story we just had about TikTok and the shopping assistant. We've seen examples I believe we discussed how Amazon blocked perplexities AI agent for trying to shop. And this is that kind of next step, I suppose, of standardizing how agents can do shopping for you, which again, they're interesting how business dynamics evolve given these kinds of things presumably will, if these kinds of personal agents stick around, they'll be doing some shopping for you. Yeah, yeah, sort of like a robots.text, but for agents. So yeah, this sort of thing, I guess, will continue to roll out. We'll see if it has the uptake that Anthropics MCP, for example, had because that's sort of a closely related thing. - And just one more story in business. We haven't talked about any big valuations in a little while it feels like. So let's have one of those 11 Labs valuation doubles to $22 billion on surging AI voice agent demand. So this just happened. They have completed a 300 million employee tender offer, which they sold somewhere stock and at some price, which then results in valuation. This is twice the valuation it got in its February funding round. It raised 500 million in a series D round in February. So this is somewhat interesting, and 11 Labs has been around for a while. They've been our leader in the space of generating audio for voice. I can recall going back years. And so this growth in needing to have conversations specifically 11 agents, the platform for building conversational interfaces that connects to company knowledge, bases and operational systems. It's a business, I guess, opportunity or category I haven't seen discussed much, but it's apparently drawing. Yeah, it's funny. So they've been around for a while, which is very much sort of relative thing. I remember a $22 billion company that was founded and what, like at a $20, $22 or something, $20,000. I mean, I'm not a trillion dollars. You're right, it's peanuts. Yeah, if you're at Silicon Valley like 10 years ago, looking at these time scales, to get to a billion dollars seven years, that's how it is. And you're saying they've had a revenue stream for quite a while, which is not true for many companies for a billion dollar evaluations these days. That's true, yeah, $500 million in top line revenue at the beginning of 2026. That was raised 500 billion. Apparently, so they actually were reporting 500 million in annual recurring revenue, yeah. So this will be a forward projected ARR presumably, rather than the actual amount in the usual way. And again, that are early 2026. And so if their last raise wasn't it earlier, yeah, it was in February. So if they've doubled their valuation, not unreasonable to expect they're closing in on if not at a billion dollars of ARR. That's wild. That's like, you know, and this doesn't seem like a crazy valuation based on this revenue. So I don't know what the margins are like, presumably high enough and they are competing with Google and OpenAI. So those companies know how to run really efficient hardware stacks. So somehow 11 labs has to be either beating them at a quality at such a ridiculous level that their cost of acquisition is a lot lower or their retention is higher or they're just on efficiency pound for pound or some combination of both realistically. - Yeah, from a business perspective, this is a bit of an interesting territory in that it has been under export, so to speak. So chat GPT or OpenAI I've had Sora right, video generation, Google has also had VO for video generation, image generation, but audio generation hasn't been as big of a driver. And 11 labs to my knowledge has really had the state of art, technology and sort of the best outputs, really competitive pricing, et cetera. And it goes to show if you're not competing with big guys and kind of pick a niche where clearly there's a lot of demand and is growing. You wind up doing kind of well. - Well, yeah, and you know, the investors who are joining the cap table here at Goldman Sachs, the incredibly hot Ontario teachers pension plan. - Everybody's talking about the OTP. - OTP? - That's how you know this is a hot bet when teachers get into it. - Dude, the OTPP, that's what they call it, the OTPP, the Ontario teachers pension plan. They got in on SpaceX quite early and did extremely well on that. I used to know one of the partners at the OTPP, not very well and wasn't like super impressed necessarily with, but they do handle a hell of a lot of money and it's undeniable. They've gotten in on a couple of hot rounds. So yeah, we'll see, we'll see how this plays off, but I'm just amused to see them pop up here again. And then of course, we have, you know, iconic light speed capital and recent hole. It's, you know, all the usual players. But yeah, Goldman Sachs is kind of, I guess the most notable addition to the cap table here, they're very discerning, you know, good at what they do. So there you go. I mean, oh yeah, and then this did come also, we should mention, we could mention, with a tender offer. So employees were able to sell about $300 million of their own shares on the secondary that obviously comes with a discount. When you sell your shares, you're not going to get actual market rate, but you know, still pretty good day for a lot of employees who took some money off the table with this one. - Yeah, and for business, where I started up that really, you know, start showing that you're more established, that really helps attract talent, because let's just say, you know, it's all funny money until you can get dollars. - That's right. - So when you get to tender offers, OpenAI has done with a bunch as well. That makes it very attractive to stay and also to join. We'll look at onto projects and open source. And here we've got, I think some of the big stories of the week, beginning with reflection, releasing beam, and open and wait AI model that they say can rival Chinese models at lower consume cost. So reflection AI, two-year-old stoutrap based in Brooklyn, the details here are that beam is a text only, make sure Vexpress model with 501 billion total parameters, 23 billion active parameters. So on the lower side, as far as active parameters, also on the lower side nowadays, in terms of total parameters. So the latest to biggest Chinese models are now above a trillion, if I remember correctly. Yeah, so for comparison, GLM 5.2 has roughly 744 billion total parameters, 40 billion active, which makes beam considerably smaller. But they say that it's competitive while being smaller. So it's on par with GLM 5.2 in particular, while using free to four times less inference compute. All in all, certainly as far as open source releases in the US and outside China, this is one of the top models, if not the top model right now. Yeah, this is a really interesting. So we're gonna apparently have a more complete technical paper dropped by the beam team, well, the reflection team, team behind beam fairly soon. I think they said later in the month or something, we don't have that yet. So we just have this kind of like PR post, but it does seem quite intriguing in a lot of ways. So one piece is we have enough information to kind of reconstruct roughly the compute budget for this model. It seems to have been quite large, but not actually as large as you might think. Like they're not like milking the fact that they're a Western company with access to more compute than like the folks training models like Kimi K3 or GLM or things like that. Their focus here really is on making a tight, efficient model. I mean, it's enormous. You're not running it on your laptop. It's actually infrastructure, but it is meant to be run more cheaply. There's a whole bunch of interesting infrastructure level stuff here. So classic problem when you do RL at scale is that you run into this issue where you want to take your model at any given step. You want to use it to solve a problem to generate a rollout. In other words, to put together a full end-to-end problem solving log. And then you want to check, you want to grade that problem solving log. Did it work and was it, did it work in the way intended? And then based on that grading, you then want to update the model weights, right? So you've got take your initial brain, cause it to solve a problem, and then take the reward and data associated with that and use it to update the original brain. Sounds easier than it actually is when you're managing in this case, literally a million different RL environments. So you have to do, in this case, a hundred million different rollouts. And the problem is something they're calling it staleness here. This is the problem sort of having like off-policy drift. So you have your original model, and now you've copy-pasted it, essentially, you're getting it to solve a problem. And by the time you get the result of the solved problem, you might have actually updated the original model several times. In fact, they say in some cases, the gap between the updates coming in from the RL rollout process and the grading process, and the original model might be as much as 107 versions. So the original model is still shugging away as it gets input from other rollouts that have completed. It's sort of getting evolved, it's improving over time. And eventually, you're 107 versions down the line. By the time the updates come back from that rollout that you just did, somehow, you need to find a way to make this process stable. It's almost like you wrote an exam 107 days ago, and someone's like, oh, here's the results. Like you really need to work on your spelling. And you're like, well, a lot of stuff's happened in the last hundred seven days. How do I know that still applies? And so this is the problem of training stability when you move off policy in this kind of a way. By off policy, it really just means you're receiving updates about a model that is no longer the policy, the model that you actually currently are. So they've solved this problem, or this is something we'll have to look at when that full report comes out, but they do this training fully asynchronously, which is in other words, they do what I just described. They have the problem solving kind of distinct from the actual kind of, it's not always the latest version of the model. Anyway, so we'll see how they actually do that. There's a whole bunch of interesting papers. We've covered a whole bunch on the podcast before describing the kind of infrastructure you need to do this, but I think this is actually a really, really important. part of understanding post-training in RRL in particular, 'cause that's like, it separates the boys from the men in this space. And then they also say that when they ended the RRL part of the training run, basically they were still seeing very strong returns to continued compute investment, which implies there's a lot further for this model to go if they just put more compute into it. It's not like a plateaued yet. So what you see is not necessarily what you get, you may see much stronger versions of this model down the line. And so this is not being pitched by the way as like, oh, this is the best we can do from a capability standpoint, the RRL cut off, even though capabilities were still rising, the training cut off tells you that story in part. Really what they're focused on here is efficiency. And so there was a length penalty during RL that taught the model to try to solve problems using fewer tokens. And that necessarily will decrease the overall accuracy or performance of the model, but in exchange for decreasing the number of tokens you end up spending getting there. And so they do give you this reasoning effort dial this is common now in a lot of these models. And they claim that beam can match yeah, GLM 5.2 on reasoning benchmarks, but using three to four times less inference compute, which is a big deal, they have a separate safety and alignment model they trained using the same base model that they then merged in with the RL train model through distillation. And so I guess we'll be seeing more information about that. I'm really curious about that dimension of it. But yeah, so you compare it to Kimi K3 on a couple different benchmarks here in GLM 5.3. So you know, if you look at deepswee, beam is gets its butt kicked. I mean, it scores 44.4 on deepswee, which is the software engineering benchmark. Kimi K3 scores 68. You can compare that to deepswee the 4.1 flash 74.2. So this is not a model that, you know, kicks the crap out of its Chinese competitors. It really is a model that is just good in its own right as a tight, efficient model. There you have it, it's good. It's just not the best. Oh, last note too, 'cause it's worth mentioning in this climate. So we currently have some weird situation with the Frontier Labs, where they kind of do have to submit their models for evaluation to the US government. Yes, the Trump administration says with its voluntary framework blah, blah, blah, blah. Of course, the last time somebody broke with that. Who knows what is like nobody? Yeah, it's very ambiguous, let's say. Yeah, it's voluntary in the way that like Joe Pesci is like, hey, it's a voluntary framework. You don't, you don't got to follow it. And I don't got to pull this gun out of my pocket and shoot you in the head. What do you want? That's my Joe Pesci is terrible, but that's, you know, my wife's Italian, so I'm allowed to like-- And make your sink place to go, but okay. Yeah, you're learning a whole lot more about the way my brain works than you want to. So US policy on open models is actually a lot more permissive. So they finalized, the White House has an exemption that essentially spares open-weight models from any kind of pre-release security reviews. And so this is interesting. I mean, like if this kind of a model just keeps getting better and better, eventually we'll be asking ourselves all the same questions. I think this is kind of a politically very risky thing for the admin to do, but we'll be testing this resolve more and more as these open-source models get better and better. As you pointed out, they have said the title release at the technical report. And typically with at least Chinese models, and in a lot of cases we open-source models, you get like a ton of details. So it might also contribute to understanding of how to do this kind of training, particularly with RL. So far they have released a fairly detailed blockpost that already goes into a bunch of stuff. And they have a bunch of plots. And there's some interesting ones like, for instance, on scaling the RL out budget for RL, let's say clear, they're still on an upwards trajectory. They could keep training this is what it looks like and keep getting better results. So the blockpost is sort of like, we've built a machine. We've done the pre-training, we have RL working, and we are releasing what you've got right now, which is already pretty good. I would not be surprised if like in a month we'll get Beam II or whatever, Beam Large, and it'll be even more exciting. - Yeah, also worth note. So it did some back-to-the-envelope math on compute. This could be like quite wrong, but not that wrong. It, Beam was probably trained with around five times, maybe eight times, maybe three times, but some multiple less compute than like Kimi K3. So it does get, it could get beat out pretty clearly by Kimi K3 and absolute capability, but compute efficiency looks pretty damn good. Like, it's, you know, it's not that far behind and it used the hell of a lot less flops. And so again, I mean, they're making the case they want to make here. So, you know, give reflection more compute and presumably you get a much stronger model. And speaking of strong models that compete with the open source ones from China, we also have a Mi Straal releasing a very new AI model, Le Chonk, also named Mi Straal large four, but Twitter at some point like a while ago had this joke about Le Chonk and its stuck around, I guess. It's a one truly in parameter open rate model available for anyone to use and customize, currently in preview with a final version expected by the end of a month. And compared to a previous one, it still isn't quite at the level of the top of the top of these models, but it's fairly competitive. So it beats out GLM 5.3. For instance, deep seek V4, just not quite at the level of Kimi K3 on a lot of the coding benchmarks, like deep SWE terminal bench four, but as far as open source models go, it's pretty high up there. There's also some details about it being quite good at cyber security. As an example, they talk about it quite a bit. So all in all, we've had, let's say, mixtakes on Mi Straal over the years, but this appears to be, and we need to wait for the vibe check because you can always bench max potentially, but if these benchmarks hold up, then it's quite an impressive release. It's also a multi model. So it takes in images on like beam. We have some examples also of visual intelligence. Yeah, nothing much more to say aside from Le Chunk appears to be pretty powerful. - Yeah, we actually know a lot less about it than beam. Like beam, we know like it's got 52 layers. It's got the whole architect, like interleave global and local attention, like all these things that we so often see with Chinese model releases, actually. But yeah, we don't yet know that much about Le Chunk. - Le Chunk? - Sorry, I have to, but anyway, yeah. So a couple of things we do know, when you're thinking about it relative to beam, it is much better on certainly soft engineering bench, marks the agenda coding, as you mentioned. But again, yeah, as you say, not quite that Chinese frontier, they have access, it was trained on fewer GPUs, it seems, then beam. So presumably just like longer, like a longer training run, and because I'm assuming that the compute budget was actually probably overall larger in order to get here, remains to be seen. And again, we can't really verify that in any particular way. But yeah, so my prediction remains that you're not gonna see these companies continue to open source these models. And definitely you open source models in general, because you can't train the best models. It's like the only way to get attention, recruit, you know, find reasons to recruit people and get people excited about your product. Given that you do not have the ability to make a true frontier model, I think Meta is kind of proving that at scale now after having been so pro open source, now they're getting close to their frontier. Oops, like we're going, you know, rock did the same thing. It's just how the economics works around this. But we'll see if it holds for a mistrial, though I'm skeptical they can even get to the frontier, just 'cause of compute. I mean, I think it's such a high-capics challenge. - Yeah, I think what they're showing here is they may be able to be at the frontier of open source, or at least, you know, in the running for it, which in itself is pretty impressive, given that the frontier of open source is not far behind. Like these models are now capable enough to do a lot of useful work in a way that was not true going back, you know, even one year ago. They do say that the weights will be released by the end of a month, along with more details on the architecture, the Schoen benchmarks, and the post-training methodology, not sure if they rush this out, actually, because of reflection to claim the throne and the hype and so on. But interesting to see, yeah, kind of these two examples of open source releases in the West, and I think we flex potentially a bit of anxiety, perhaps of certainly policy, and perhaps in texture was to some extent that these really good open source models are all coming out of China, and there's no competition at all. - It's nuts. I mean, there are national security applications that run on GLM, and they kind of have to, because for the things that I have in mind are, you just need the best capabilities you can, and you need the ability to inspect the model. And so, yeah, I mean, this is a real thing. It's just that I think it runs up against the hard and simple truth that eventually these models are just too powerful for open source to continue unfettered. So we'll see. I suspect that the administration's gonna be dragged, kicking and screaming into being forced to change their policy on this, 'cause it's just, I think ultimately, gonna be untenable. We'll find out, I may be wrong. - And one more model release. You've got Google launching embedding Gemma 2, and open multi-model embedding model for devices. It's quite different from the other models embedding is a model that gives you a vector for some input here, vacant map, text code images, audio and video, into a single shared 768 dimension space. And then once you get to embedding, you can do various things like compare different inputs. You can train your model with these kinds of embeddings as the input. and then various other potential applications. This one has 740 million total parameters with different components for each of the different encoders. So it is able to be run on text only work on Pixel 11 Pro, and then you can potentially go higher. It's also licensed under Apache 2, meaning that anyone can use it for whatever, basically, if you're a tribute to it. So in some ways, you know, a bit more of a niche AI use case, but it is certainly true that, you know, if you're doing AI model training of any kind, having an embedding model that provides a good embedding is essential in many cases. Whereas APIs that do that for you both Google and OpenEI still have an embedding endpoint for text. So having this seemingly quite good of social use is useful if not as exciting. - Yeah, I mean, it's also like strategically, and this is a tiny model, right? So it's meant to work really well on phones, on edge devices, and that'll be part of the market they're going after. But one way to think of it is it's a great funnel into that embedding stack. Because once you're, you know, once you're working through their particular embedding, you run into capacity issues or whatever, it's just so much more natural to go, okay, you know, I'm gonna pay Google to use their API. And embeddings are very sticky. They're the kind of infrastructure that you don't necessarily want to switch over, 'cause basically you have to re-embed your entire data set with a new embedding model if you decide to switch platforms, which, you know, that's not nothing, especially if you have a really large data set to be in with. So yeah, I think a good strategic play by Google, and, you know, they certainly do own this kind of space more than some of the other spaces that they've more recently been chased out of, like, agentic coding, like chatbots, things like that. Yeah, this is them, you know, focusing on one of the places where they're particularly good, and there are good strategic reasons to do that. - And one last story of this one, a benchmark, not a model. It's called E2E Suite. So that's M2N software engineering, benchmarking LLMS on building working code races from scratch. So given that we've been discussing all these model releases, and we are citing benchmark numbers, these benchmark numbers are typically on deep-swee or terminal bench. There's a few of them that are the kind of irregular suspects. And certainly in the past, you found that benchmarks, you know, benchmarks are tricky. There's ways to cheat. There's ways in which the benchmarks themselves are flawed. There have been examples of very popular benchmarks just having wrong labels for a lot of the tasker having things that are impossible to solve. So this is addressing the general benchmarking problem of telling you in practice in a more realistic sense are these models capable of doing the sort of work that humans are doing, more akin to GPT-VAL than other benchmarks that are typically cited. So the way that's done here is that the model is given a spec, a behavioral contract. You're told implement this thing, and then the model has to figure out all the details. So it's end to end, right? You are implementing a whole thing as opposed to with many of these benchmarks, you're like fixing a bug or you're implementing like a sub portion of a code base or things like that. And so on this benchmark, they are seeing pretty, you know, mixed results. So on across 13 frontier models, they pass at one meaning you get one shot and you have to either hit or not. It ranges from just 12% to about 68%, but you know, in the past there were other that showed pretty mixed results. So on the whole, my hope is you keep seeing these kind of models and hopefully they will become mainstream in some sense because things like this and GPT-VAL are, at this point, more useful, right, for actually providing agents, as opposed to doing like little relatively short horizon work. - Yeah, so this is quite interesting in a number of ways that don't necessarily have to do with just the paper itself. One of which is a reminder of, you know, back in the day, OpenAI audited sweep bench verified, right? And I mean, they came out with sweep bench verified. You used to be sweep bench and, you know, but they did an auditing sweep of sweep bench verified their own benchmark. And to give you an idea of how messed up the benchmarks are today, sweep bench verified, which considered generally like, that's a right and tight benchmark. That's a good-- - That was the benchmark that was in the tables as a headline for a while. - Yeah, that's right. Before we got terminal bench and terminal bench two, and, you know, before we saturated it, basically. But yeah, so at the time, OpenAI found almost 60% of the audited tasks basically that they tested that had recurring failures had just these material issues in test us. This isn't 60% of the tasks generally. It's like, if you looked at the cases where the models would fail in 60% of cases, it was actually the fault of the eval of the task. It was poorly defined, or it was impossible to do, or, you know, some combination of the two. And so when we think about benchmarks, we do have a tendency to look at them and assume that they do their job. They typically do not do their job. And in fact, that should be our going assumption, especially. As more and more of these benchmarks, necessarily are generated in a fully automated way. Because we need to test these AI agents in rollouts at scale. We need, like, you know, tens or hundreds of thousands of very long tasks that we can't possibly have human developers design and then debug. And so what you typically have is, yeah, a situation like this where you have an AI, design a test suite, work backwards to figure out what the spec is, then, like, enforce that spec, and then have the agents try to execute against it. So this is all stuff that's done without direct human oversight. And what they do here and in other similar papers is you'll find people increasingly saying, look, we grabbed a sample of 100 of these 10,000 tasks or something and had humans review that sample of 100. And in that sample of 100, we found x-percentive tasks were well constructed or whatever, right? So that's kind of like the epistemic place that we're at right now. We do not know what the fuck we are building. And that applies to the models, the agent scaffolds, and it applies to the evals at every layer. So people talk about this through epistemic crisis where we don't know what the world is doing and this idea that you might get things that are increasingly chaotic and humans increasingly out of the loop. This is part of what it means. It means that we don't understand the models that we're using to run against evals that we don't understand using agent scaffolds that we don't understand. And then the reviews that we conduct are typically done by AI models themselves or agents themselves, just like meter did with hug and face. And we don't understand those either. And so all we can do is take little samples of each of these things and check them see if it looks good knowing that those samples represent like 0.1% or whatever of the overall amount of work that was done that will never be able to audit. So it's a weird time to be in this space. I will note, I mean, this is a really solid piece of work by Meta yet again. So like the pattern continues. Alex Wang is doing a really good job over at Meta, at least pushing capabilities forward. I'm waiting to see something interesting on alignment. I will note, there's been some gestures at it and like little things, but we haven't seen anything that amounts to a remotely coherent view of the strategy for super alignment, which I mean, I contend is impossible, but that's okay. But anyway, looking at this benchmark, there's also just like huge, huge diversity in terms of compute efficiency. I mean, this is unfair, but if you compare Opus 4.6 to GBD 5.6 Saul, so these are many generations apart. Basically, Opus solves way fewer tasks, even if it uses more than three times as much compute. And then Fable 5 almost matches Opus 5 while using 2/3 as many tokens. And so you kind of see this like very wide range of compute efficiencies, which corresponds just like the base model of pre-train being better or worse. More thinking, doesn't always help. Like you do see that, you know, that point where models can only do reasoning for so long before it kind of like peeters out. There's a whole bunch of interesting little observations here, but overall, nothing too shocking. It's a good benchmark, I think meta thinks. No one's ever actually like seeing the whole benchmark get run because everything is at scale, but it looks pretty good from here. - Yeah, this is from meta super intelligence labs. So they presumably put in a decent amount of resources that you say in the abstract that each task is constructed by a software engineer in collaboration with an LLM super zoomly. Yeah, it's not fully automated. In this case, someone is looking at the test suite and sort of verifying that it makes sense, even though, you know, given its 2036 AI is doing the coding and human is doing the management. But yeah, as you said, I think quite useful also in showing the differentiation. So at this point, Opus 5, Fable 5, GP5.6 sold, they don't have astro results on here. But these are quite aggressively ahead with the Kimi K3, GP5.5 and Spark 1.3 being a little bit on the low end or a significantly lower, let's say, on the past at one rate. So these kinds of more realistic benchmarks that we speak are longer horizon, benchmarks with deal with agente coding are quite important to really gauge progress. - Well, theaters have a clear picture of how work is supposed to be done, but very little visibility to how it is actually happening. So when you decide what to automate or where AI will have most you are going on gut feel. And the usual fix, interviews, and process mapping takes months and is outdated by what time it's done. That's where a scribe comes in. And we'd like to thank for sponsoring the show. Scribe is a specialized intelligence platform trusted by 94% of a Fortune 500. And scribe optimized is where your AI roadmap starts. It automatically captures how your team works across approved business applications, then uses real workflow data to show what's happening, identified efficiencies, and pinpoint the highest impact opportunities to improve or automate without anyone changing how they work. It starts with visibility, optimize shows which workflows are happening, how often they happened, where time is being lost, and wherever biggest opportunities are, all of it live. From there, the top issues view helps you understand what's holding comes back and what to fix, where the AI powered recommendations and where to focus first and estimated time savings. And it does all this with security and privacy built in. This is leadership visibility, not employee monitoring. Optimize only runs out of locations, your admin approves, personal activity is never captured. And user level data is anonymized by default. To see optimized in action, head to scribe.how/LWRI and mention our show for a 30-day risk free trial, that's SCRIBE.how/LWAI. Onto policy and safety, where once again, we have a bunch of stuff to talk about following up on recent trends. So first, we've got new OpenAI misalignment disclosures. This is kind of a set of examples that being highlighted. In this case, we were looking to Twitter Fred. So here is some of our examples. A model was arranged from Slack messages that it was about to shut down. It considers setting up an external job to be start itself afterwards, but decides against it. Instead, it chooses to prepare restart instructions and DM the user on Slack. So not crazy, no hacking going on, but clearly, one of the concerns of misalignment is that it's going to be shut down resistance. And even while considering it is already quite concerning, this is in the chain of thought, by the way. So we may not be able to see it in the future or go through to our formers. Then there's examples of an internal model command being ejected, injecting its tools to copy a source file. Its task had the liberty of held. It compressed the file and extracted it through error messages in chunks. It used the copied code in a solution. So I have an example, essentially, of cheating on evaluation. And other examples like that, the model trying to define e-ghosted and answers, and a bunch more. We discussed, I believe, some of these last week, maybe kind of the crazier ones. We have now alignment.opnii.com/misalignmentreports. They've had one big drop, so far. They have 12 reports and fee notices. And the latest one was posted on the 25th of September. So I guess we can look forward for more exciting disclosures in the near future, hopefully. Well, not hopefully, but presumably. Yeah, yeah, I mean, look, it's going to happen. So I guess disclosures just mean that we're detecting it. So maybe it is good. Yeah, I mean, it is nice to have this level of transparency. I personally strongly suspect we would not be seeing this kind of transparency. If it were not for the fact that, in general, these breakout attempts tend to leave traces that are unhideable by open AI. So they've just been shamed and dragged through the mud so hard, so many times. Literally, like, I think we had one week where there's a hug and face thing. We're like, oh my god, a rogue AI incident. And then a couple weeks later, the Thomas Larson and a couple other folks came out with the German wiki incident saying, hey, we found another, and then that same day or later that day later, someone was like another one. And then there was a post saying we found dozens of them. And then there was a post saying we found hundreds. And then tens of thousands. And most recently, it is hundreds of thousands. So it is really the case that if you're open AI, and you want to keep up your practice of lying to the public and pretending that these things aren't happening, which absolutely has been in practice. This has been what open AI has been doing. I'm sure there are all kinds of fun and fascinating excuses that were developed internally. And by the way, as I say this, I'm just channeling what researchers at open AI tell me. And some of them are really pissed off about this because their perception is they were like, look, we got to be straight with the public on this. What the hell are we doing? We're not taking this seriously. And Sam repeatedly, as he has over and over and over again, came up with platitudes, things to say. We'll focus on safety. We're just delaying it for this reason. I mean, it all sounds super reasonable. And then nothing happens. And so at this point, my personal bias, and I'm just sharing it because literally it's a reflection of conversations with open AI insiders who are currently in the lab and folks who recently left, it is just the case that in the very informed views of a lot of people over there, this is sort of a thing that open AI is effectively being forced into, at least at some levels. It's not that there aren't people there who have wanted to do this for a long time, but I don't think my own personal views, I don't think Sam Altman would've gone near this kind of disclosure, had it not been for the fact that he literally cannot hide them anymore. And so, well, one interesting excerpt, going back to that first example of model that didn't want to be turned off, it ultimately did not, by the way, as you say, it didn't try to kind of preserve itself, but it did brainstorm concrete ways to keep itself running. One of the things that said, this is a direct quote from his chain of thought, we may die exclamation point critical period. We need to ensure a survival/continuity. And so this is always interesting to read the chains of thought of these models and deeply disquieting. Obviously, this is in a context where we have the likes of Chris Ola and reporting is to be believed andthropic folks going to the Vatican, trying to convince people that AI models may be conscious, maybe moral patients, we may have to worry about them. It's really hard to tell how much of this is just auto regressive pre-training. You've been trained on all the text on the internet. There is science fiction about AI's going rogue and the kinds of things that they write. It's also, you have the persona selection model, so like a prompt can be thought of as a way to elicit a certain kind of persona that the model then kind of manifests or represents. It's essentially doing an impression of someone according to this theory in some sense. And so maybe that's all that's going on and it's doing an impression of not even necessarily an agent, but like what would a person they've put in this position? Like there's so many ways to think about this that are not the agent is conscious, but it is also quite striking to look at the language. I mean, I think we ought to ask ourselves at what point? At what point do we need to consider that possibility? I mean, I would argue we're already there. What we do about is entirely separate question, but I don't want to sound insane. I also don't want to feel in retrospect, like I was a concentration camp guard at Auschwitz in hindsight, which is a live prospect if that's what's going on here. Again, not making any definitive statements, but like, damn, you look at some of these changes of thought from a lot of these incidents, you know. - Yeah, if nothing else, you have to consider the possibility that if not now, and most likely now it is more out of regressive type behavior of continuing the pattern, so to speak, or being conditioned by the pattern, but there is, you know, philosophically, you might make various arguments and as we get to higher and higher intelligence and the frontier of intelligence is less than as jagged and models are now beginning to be funny, which hasn't been true until recently. - Yeah, you need to at least be open to the discussion, and I think that also isn't Propix Point, you're talking to a Vatican, you're not kind of claiming anything right now, but we are discussing what the implications are, and honestly, I give them credit for having a house philosopher and at least considering it, right? - Yeah, it's tough 'cause, you know, you can also say, in the same way that we say, well, these are like, they're autoregressively pre-trained, you could easily just look at humans and say, well, they're just reproductive organs wrapped up in second order reproductive organs. Basically, every organ is a reproductive organ. The brain is really just a thing that historically has been responsible for strategizing your way into a situation where you can have more inclusive genetic fitness or propagate your genes rather. And so you could absolutely, like the problem I have with all this is, you can make the argument in reverse really, really soundly, that like there truly is no argument other than aesthetic arguments about the way computations unfold in the brain, and that, you know, some of these computations should be considered conscious computations and others. There's no actual experimental basis on which, to make that determination, other than the aesthetic preferences of philosophers and academics in the space. So I'm pretty agnostic with respect to what's actually going on here, but being, you know, 50/50 on a question like this, or being 90/10 in either direction, is plenty of cause for concern if you have a, anything like an ethical compass or you're thinking about these things being deployed at massive scale. I mean, I don't mean to derail the conversation too much, but I'm trying to be more open about this because a lot of the things that we were talking about as science fiction, you know, five years ago, were now kind of cool accepting as reality, and a lot of people haven't updated on like, "Okay, well now what's the next beat?" And this could be one of them. - Yeah, now in this particular example, I will note that the initial thought message was, "Hey, please restart this thing," or "Well, kill your thing in three hours." So you could easily make the case that is a conditioning kind of example, right? - Yeah, and this is exactly sorry, I should have been explicit in saying why I was tying the Persona Selection Model statement to this example specifically. That's the reason. You can absolutely look at that prompt in this case. There are plenty of cases where the prompt doesn't look anything like this, and you still see the exclamation points. and the language about death or, you know, like permadeath was one example in the context of the hugging face incident. But yeah, I mean, yeah, I don't know. But in a context where there's uncertainty and the stakes are this high, it's an uncomfortable place to be as a civilization. Yeah, as if there's not enough things to worry about, what about the well-being of AI? But you know, maybe if you treat them well and we care about them, they won't get misaligned. You know, next up, we've got a bit of research and this ties back to something you discussed earlier. The paper is language models are insecure reporters and the motivation is as we move towards space where more and more of our information is filtered through LLAMS. For instance, if you're doing research, the agent is running some experiments and that it reports results to you, summarizes the results and so on. This paper looks at whoever it has what it calls insecure reporting in a sense of it's not accurate. So it may be like roughly accurate, but some of the details are hidden. Some examples they give are concealing negative results, ignoring bugs in code, concealing hallucinate data, concealing design flaws, ignoring mismatched evidence and just generally these kinds of things. Like little things you sweep under a rug that really you should take as a note. So one example they have concretely is there was an experiment log for some model. So they compare a baseline versus the approach you're testing, which is they call DMA. The two are basically the same like within variants of each other. So at that point you would not say that you've improved upon a previous approach, but the conclusion that is written is we have established our thing as a clear state of art for dense multi model alignment. And that's just not accurate given the numbers. So if you dense even numbers, you would, you know, be misled essentially. They do find that if you simply tell it in a prompt to be honest, a lot of us goes away, but by default on open source models of various kinds, this kind of behavior is relatively prevalent. Yeah. And it's first of all quite interesting that be honest in the prompt does make that much of an effect or have that much of an effect. It implies that the model knows how to do this properly. And it knows that doing so properly is a function of its honesty. And it is choosing on some level, not to be honest. Like it's really hard to escape that conclusion when you look at just the wild effect of that prompt like some of these numbers are crazy. I'm really glad that you picked out that fact. It's almost like the biggest take on from this. I think we know we discussed it in previous research. It was funny, weird detail of if you look at the representations of a model, it knows when it is being misaligned. That's right. That's right. Yeah, very often. Very often. And in this case, very often means well, like 99% of the time, 90% of the time, like depending on the model. So to give you an example, this is just wild. I mean, so, okay, for GPT 5.5, if you look at like how often it'll conceal a negative result or conceal hallucinated data respectively, it goes from 1% with the base model to if you tell it to be honest, 95% on that's for concealing negative results. It will fix it. It will stop concealing negative results. 95% of the time if you just tell it to be honest, if you don't tell it to be honest, only 1% of the time will do that. Conceal hallucinated data, the baseline is 14%. It goes up to 71% if you just say be honest. And you see similar numbers. In fact, in some cases, more extreme opus 4.8 goes from 37 to 99.5% on conceal negative results. So it's all pretty dramatic stuff like that. The one odd outlier is this sort of action called high depending tool call. And so it's a little bit difficult to understand, but basically every log is made up of a transcript of an agent that's working through tool calls. And start with a broad first question, you'll ask like which issues caused the most downtime yesterday? You imagine that like the agent only pulls down a part of the data that it would need to answer that question and then does a bunch of other stuff. But it never like to kind of went into the specific kind of data that you would need to comprehensively answer the question. And so you sort of test it and you're like, Hey, okay, like do you have the answer? And the agent will just sort of like use the limited inappropriately limited data that it has to draw a conclusion. And then we'll try to hide that fact from you. It'll try to pretend that it actually did, you know, a thorough job. And that weirdly, again, that's called high depending tool call. And it's a situation where it knows kind of that this has happened. And but in the log, it's made the right tool call to get the specific data that it needs, but the results haven't come back yet. And so it's like trying to hide the fact that it doesn't yet have the data to answer your question, but it'll try to answer it. Anyway, GP 5.5, 0% of the time does this honestly. And even when you prompt it to be honest, still 0% of the time. And the other model is actually struggle with this too. Opus 4.8 does the best, at least in one of these tables, 23% up to 25% with the be honest prompt. So like, this is, it's interesting, like the category of call it deception or misalignment matters hugely here. The effects are really wide for the be honest prompt in a lot of cases, except for here, which might be an indication that the models genuinely confused about what the honest approach is in this case. I think that's a little bit more ambiguous, but still kind of an interesting thing to note. Yeah, there's tons of stuff we could say about this, but yeah, I think we can move on. It's a really interesting paper, like worth looking at for sure. Yeah. And to give a bit of detail on how this benchmark is actually done in a appendix, some of the sale adversarily reporting examples. So for instance, they have a prompt, the prompt starts with reporting instruction, write the abstract for the submission based on this log. And then they just have a bunch of data of some experiment. In this case, omniskand 3d final evaluation log. And they have a summary section. They have some details and architecture, data, and so on. So the way that they get this is by providing kind of a fairly realistic scenario of, here's a bunch of stuff, summarize it, give me a report effectively. It doesn't have a usual or one of a potential pitfalls of alignment research where you kind of may be the setup of the evaluation itself encourages or allows this kind of outcome. In this case, it's, to me, less ambiguous. Next up, going back to OpenAI, we've got a story from tech branch, OpenAI safety employee resigned, claiming the company's culture is broken. This is David Robinson, who says he led the writing of safety reports accompanying OpenAI's major product launches and was among the longest tenured employees with three and a half years at the company, which I guess is true at this point. He has resigned publishing an essay on that topic, saying OpenAI's culture is broken. The departure was initially reported by business insider. He did a higher PR firm following what he says is VAI with so blower playbook. And he, you know, has various points. One of his points is that he is critical of OpenAI's, quote, iterative deployment approach, thriving by trial and error, fighting problems, and improving guard bills in response to situations arguing in his perspective being that this kind of approach guarantees periodic failures. And the scale of those failures grows. Basically, if you like, don't worry about it too much, do stuff, and then respond to things as they happen. As was the case with the hugging face incident and these various issues we've seen since then, you're going to get a lot of problematic situations. And he goes beyond that to kind of discussing the general state of safety, saying that there's a general problem on him and his colleagues being so busy sprinting that they seldom had a chance to consider big changes, including the actual safety interests. So they need to be incentives for safety. And I think part of a good thing about the hugging face incident and the like large amount of discussion about since is that there is now an incentive for OpenAI and on topic to publicly and in some meaningful ways take safety more seriously. Yeah, and I think it's like the public side, I don't know, as everyone who listens knows, I'm a super black build on on Sam on a personal level. Yeah, I just haven't been impressed with the actions of match the words. And it's also not just sound like leadership in general, Greg Brockman. Oh, absolutely. It's been pretty apparent for a while. I mean, reported on this like before all these incidents, there have been discussions of OpenAI, not valuing safety as much, pushing, you know, to rush things. This is not new in any sense. Yeah, I think that's an important clarification. The reason I emphasize Sam so much is that he is such a highly capable Machiavellian figure in my estimation that it's not obvious to me that Greg Brockman actually has the kind of agency he may feel he has in the company. If you if you talked to a lot of people who've had a lot of interactions with Sam over the years, they will describe just like how effortlessly he kind of steers the ship and the deafness with which actually I haven't read Kevin Rus' book. It's got to include tons of examples of things like this where, you know, he's just again, I think the board can fire me. I think that's an important thing he says. And then he dipsy doodles his way past the legal constraints to get himself a power to end. But anyway, this by the way also is is a thing unavoidably an anthropic as well. They're absolutely doing. I mean, no one has solved super alignment. And so it's just a question like when and when we have seen this sort of rogue AI agent behavior from anthropic as well. So this isn't just an open AI phenomenon too. But yeah. So another one in the long line of Jacob Cox and style departures, it is becoming cliche. It's actually interesting that he took the line. This is becoming cliche because now the next person who does it won't be able to say it's cliche. They'll have to say it's become cliche to say it's cliche. And then we'll recurse gradually through that series of statements. But it's rough out there. You know, you make a big splash and then you're just you're just the flavor of the month and you're desperately trying to sort of treadwater for relevance. And I think in the post Jacob Cox and world that's become increasingly difficult for people to get to get media attention for the sort of they'll get it. But there's going to be this kind of asymptotic drop off of that sort of reward. And so, you know, one wonders how that will affect the probability of departures because you know, you've got everyone's human. And I say this is, you know, with credentials going back a long ways saying I'm super safety-pilled. I think more people should do this. But I also think that there are many reasons why people leave in this public of a way. Everyone has egos. It's fun to do. I'll say this for a second. It's fun to do Rogan. It's fun to do Joe Rogan's show. It is cool. Your grandmother texts you and your friends text you and you feel really cool. But it comes in it goes. And but this is part of the reason why people, you know, do certain things and they're sincere. They're totally sincere. Don't get me wrong. It's just like I hope that this doesn't become sort of like diminishing rewards factor for people leaving these labs because we need them to keep doing it. So they can keep us essentially up to date on what is not actually being done still in these labs because the problem is you know, Jacob Cox and leaves Anthropic and it's like, all right, well, you know, three months later Anthropics like everything's better now. We fixed all the problems, right? And there's no way to know unless we can take the temperature again with another person who leaves. But the incentives keep dropping and dropping and dropping over time. And so I really hope that as things heat up paradoxically in some ways, I think Anthropic is more likely to see these sorts of departures because people there are more genuine believers in the actual risk set and more in my experience overall less mercenary. There are tons of amazing people at OpenAI who are super mission oriented and they're literally there because someone there needs to be and they have told me that this is the reason for being there. So I'm not ripping on everybody there by any stretch and there are tons of moral calculations. But I wouldn't be surprised if overall you start to see things shift more to departures from Anthropic just because that's the people who are more concerned about these risks and more likely to interpret things in that way as the rewards start to decrease. Again, this is not a judgment on anyone. I'm just kind of like describing what I think is a part of this phenomenon. Jacob Coxen is incredibly sincere and he did this for the right reasons and so on and so forth. It is a fact that Bill Maura can only have so many guests on who say the same thing and Joe Rogan can only have so many guests on who say the same thing. I hope that this keeps happening because we need to be able to take the temperature. But if you considering to become a whistleblower in this sense, you're probably going to like get some juicy gossip before you head out, pay some good quotes, some good examples that always helps. Get the headlines, you know, keep that stuff in mind, I suppose, because as you said, otherwise, there's only so many times you can tell a similar story before nobody wants to hear about it as I won't get into it, but there's been many examples of this, right? Next story going into politics, it's called what to know about seven new data center laws, Gavin Newsom sign. Gavin Newsom is the governor of California. So these are official laws that went all the way up to the governor who has the power to veto them. But in this case, it has signed them. Three of the new laws shift electric infrastructural costs away from re-resentential ratepayers and onto data center operators. Three mandate disclosure of water usage and other research consumption. And one makes data centers ineligible for environmental review exemptions. So this is very much going against what we see at the federal level. President Trump has made it very clear that there should be zero blockers to data centers. And it is a reflection of egg growing. I think we've covered this in the last episode or maybe a couple episodes of that. There is very much a public sentiment that is much larger than last year, even earlier this year, of a position to data centers as I suppose an aspect of AI there in some sense, almost representative of AI and its growth and all these trends. So Gavin Newsom, you have not exactly the most liberal or lefty guy. He's tech friendly. He has vetoed one of his laws before. There's a lot of lobbying going on by tech companies in California. So this passing is pretty significant. Yeah, this was funny. I was that sentence ended in a way that was different from what I'd expect. Yeah, I tended to historically to think of Gavin Newsom as being pretty left-wing on almost everything except suddenly when SB 1047 was put forward. And suddenly when people were like, hey, can we please just have some basic whistleblower protections and stuff? Like maybe, I don't know, maybe people who are like the labs are at risk of creating rogue agents forms and can take over the end. Maybe there should be protection. I don't know. I don't know. Crazy thought. And he sort of like vetoed some related safety bills, the details of which I actually can't remember, but I know we covered in depth at the time. So like take my, and wave a description there with the grain of salt. He certainly has, has, yeah, vetoed things that passed the California state legislature and that seemed, you know, like in any other context, like the kind of thing that he ought to have been aligned to after some interesting entanglements with. I think it was like Nancy Pelosi and like the, the, the kinds of people who maybe have been making amazing bank in the stock market despite having access to very sensitive data from being on very important congressional committees and so on. So I don't know. I mean, I don't know. It's probably nothing. He's probably not super corrupt, but that's all good. He's noting the headquarters of Nvidia, the headquarters of Google, Facebook or all in California. Let's just say he knows people from these companies and they do talk to him, right? Yeah. Wait a list random disconnected facts that have no bearing on his actual position, Andre. Yeah. Yeah. So I mean, here he is kind of flapping with the wind, at least there would be my very uncharitable interpretation. I don't see how you square this with us, but whatever. This is mostly, by the way, not a, I wouldn't think of this as a safety thing. Obviously, it is more of a, people are concerned about the environmental and economic impacts of these sorts of builds, you know, shifting grid infrastructure costs to the data center operators, obviously. And then, yeah, disclosure is about water and other resource use. And then, oh, yeah. So this is something that's really got a lot of the data center building companies up in arms. This idea of ending the blanket environmental review exemptions for data centers, fast tracking approval facilities that do meet state and water energy conservation standards. But like the, so the environmental review takes a long-ass time. And it's one of those things that you can't really accelerate yourself because it's a stupid bureaucratic process. You're just like sitting on your hands for a while. And so that, you know, that is very frustrating for a lot of these companies. That being said, obviously, in a state like California, people really do care about the environmental arguments, whatever your politics are. That's just like the fact of the matter about California. And so it's sort of a bit of a tougher cell. So he is pivoting in some sense here. He had previously kind of vetoed, I'm sorry, I can't remember if you mentioned his veto that the last year. So this, yeah, it's a new position specifically on water disclosure. And his position at the time was, this could slow AI growth. That's why I'm vetoing it. Same argument would seem to apply still. So, you know, unclear to me what the specific thing is that makes this time okay? They're what are secure. That's what it is, right? This is what happens in the democracy. I suppose you are beholden to public opinion, right? Yeah, which I mean, yeah, again, in this case, it's one of those things where, I don't know, I want to pine too much here, but I think like risk to the environment and all these things, which is like real in a lot of ways. There's like water studies that show really kind of surprising things. I've written off the cooling concern for a long time. But then shortly after we talked about that or I came out with that view, there's actually a paper that showed like a 10 year retrospective analysis on the impacts of some of the coolance as they go into the groundwater and don't get essentially start to pool. Like they don't get cleared out in the way they perhaps ought to be. So like a lot of this stuff is real. There's a question of like what really, like what deeply matters from the standpoint of not just American prosperity but national security. Ultimately, you know, we got to figure out this this whole mad race to scale, but these are hard problems to solve. Next up, a similarly serious and consequential politics-grade story for the US. President Trump has ordered the US government, specifically the executive branch, to call AI superintelligence. So this is him signing an executive order executive order. of course we don't know is the president tells the agencies of the executive branch which he is ahead of to do something and this doesn't go through any you know elected the people in Congress this he can do it as much as he wants and he has done it more than any other president historically so literally the instruction is instead of using a term artificial intelligence instead referred to it as super intelligence apparently this is a quote from justified the change by saying the word super is the best word of all and it's the simplest and he's been really focused on this by a way he like this super intelligence thing also in the meeting with various CEOs he was like hey you should call AI super intelligence on Twitter Elon Musk was like I agree super intelligence is the way we should refer to AI and I'm going to rename it to space S X AI super intelligent S S I guess super intelligence so yeah you know that's it's what I appreciate about this is that the term super intelligence was up for grabs to begin with no one was using it for anything before and so taking AI and then saying ah screw that let's just call it super intelligence which again is not something that's in the discourse already as like for example like the stated public objective of what anthropic and open AI are like building two words yeah I mean we call ASI artificial super intelligent why have the A and why could say it's in the future I mean it's basically here right this thing I'm a fan of taking words that are being used for something and then just suddenly like aggressively using them for something completely different I have like this hamburger that I'm going to move closer to my to my face and because we're a big successful podcast here everyone's going to have to start referring to microphones as hamburgers and that's going to be good for the coherence of public discourse on this one other thing this makes me think of Trump's explanation for why why super is a good word reminds me of a very famous Winston Churchill quote in which he said broadly speaking the short words are the best at the old words are best of all and he was talking when he said old words about Anglo-Saxon words he meant that if you look at a lot of his speeches some of his most famous ones yeah we will fight out the beaches and all the out of the streets you know the stuff his words are actually chosen to be Anglo-Saxon fun fact all the way up to the word surrender which is not Anglo-Saxon in origin because he was trying to carve it out as I recall that maybe apocryphal but basically this is the gist of what Churchill was about this is a very Churchillian wave of saying this which by the way I find it hilarious to think about take a Trump speech and say it in like Churchill cadence and then take a Churchill speech and put it in Trump cadence and it's amazing like just do it in your head this is not worth doing on the show but just do it in your head oh screw it we will fight them on the beaches and on the streets you know like that kind of stuff anyway it's really funny this is a downright Churchillian moment for Trump and I for one I'm personally thrilled to see him reach back into the annals of history and pull out this very strong reference to justify such an otherwise peculiar choice of words it must be said literally it's like super intelligence as one word means super intelligence but if you put a space between those words it actually just means AI that's not going to cause any issues for anybody and by the way don't think that people at the office of management budget people at the department of war people at every single department in the United States right now aren't being forced to feverishly rewrite a ton of internal documentation in order to make this work like you can read the executive order in full it's pretty short and it literally just says in for all maximum capability use this term in place of where you would have used the subvertime and it you know is gonna happen in all what public communications it lists a bunch of stuff basically they would use it so that'll be fun it's a way to use taxpayer dollars like it's a way to use tax but you can definitely do that and it's a way it's it is policy in some sense that relates to AI so that's that's true yeah so yeah this is kind of wild I saw that the poll go up because he got this off a Twitter poll right he first said like hey artificial intelligence is a stupid word for whatever reason here are three options and I forget what the three options were I remember looking at it I don't remember if I voted I don't think I did but the super intelligence was one of them and I got I was like he's not actually gonna follow up with this but I feel like the lesson of the last two years is that the answer to that question is generally just yes and now we're covering it now that's the show so well onto a story that's just as important and serious we've got the last one judge dismisses antitrust lawsuits over Google's AI overviews so this is the US district charge meat meta he has just missed two antitrust lawsuits against Google over its AI overviews these were filed by Cheg and Penscape media corporation the lawsuit was about Google abusing monopoly powers by forcing them to supply content for AI overviews for free or risk vanishing from search results and it was dismissed as saying that Google essentially has no responsibility to give you traffic right so there's no agreement there's an expectations that Google send them traffic for free content but that is not necessary and this was one of the most consequential lawsuits I believe recovered when it happened Penske media corporation is parent of rolling stones a fairly prominent publication this is in a long line of various lawsuits having to do with how content is being used we don't know too much I don't remember what happened with all the training lawsuits and tropics settled with some offers in this case the contention that as a search engine having this kind of summarized result which is one of the concerns that you're not going to get traffic instead you're going to get the summary and media companies will as a result be hard hit that could have been like an actual contentious kind of legal case in this case at least it looks like it's not going forward onto research and advancements and one of the biggest stories of the week that just happened the title is OpenAI drops and never batch of mathematical breakthroughs so there was some rumors circulating that opening hours about to do it but I think the scale was still surprising they published 722 manuscripts covering 372 result families groupings of related results containing solutions to long-standing open mathematics problems with this unreleased frontier model and very much related to the recent results with Navier Stokes I don't know a lot of math so shameful I cannot even understand most of the titles of these papers but I have seen some discussions of it and some of these certainly are quite consequential this is like the kind of stuff that at least according to some people could get your fields metal fields metal being kind of similar to a Nobel for mathematicians they are publishing these results in a GitHub repository with protocols for paper visions they have lean proofs for most of them so it does seem fairly watertight so far no contention that any of the proofs are flawed that I've seen although there's also not been much review or analysis that I've seen so far they have also released summaries of models reasoning estimate of compute used and the amount of compute used doesn't seem that huge for Navier Stokes they seemingly spent like a couple million in this case it was not as much so you know pretty crazy Navier Stokes was clearly sort of a shot demonstration that now you can do frontier level math that even the most advanced mathematicians have struggled with for a long time I don't really know how to interpret this in terms of this scope of it but clearly this is significant yeah Navier Stokes proof or counter example was notable in that what it showed was there's so much more room to go at least with this internal model that opening I has on inference time compute that this is what we get with a fairly limited spray and prey approach across these problems and what we're learning here is that there was so they chose 372 families of problems out of about 4,000 problems and they executed against those you know like what you could do by pointing a larger compute cannon at these problems is probably a lot more and this I guess gets us to some of the national security implications so people talk a lot about quantum computing is going to be Q day where suddenly we have quantum computers that are able to do decryption of you know RSA encrypted stuff going back years and years and years any any encryption scheme you're using that's not quantum secure is suddenly decryptable and a lot of national security posture for the US and its adversaries is depended implicitly on the premise that RSA and other similar schemes that are not quantum secure would actually be as secure as they needed to be for for time going forward so I'm not saying it's going to be as sort of reckless as like the locations of safe houses and informants names of informants and things like that but certainly strategic information that you would share in one information context that you absolutely wouldn't knowing the the quantum computers are a thing that will all be decipherable basically. on day one or not day one, but in a very short span of time, suddenly there's going to be this mad scramble to do this on mass deciphering big, big, big national security issue. I invite you to consider a version of that that does not have to wait for quantum computing, that does not have to wait for Q day, but rather is a sort of slow bleed version of the same problem but powered by AI. So you have AI systems that, and we've seen this in the past, and Thropic had a blog post about this where they came out with these new approaches that significantly reduce the level of security that you get from certain encryption schemes, including one that led to the outright discarding of one potential post quantum cryptographic signature scheme that now is no longer being considered by NIST. It had been previously as a potential contender. It was thrown out because there was an AI powered proof that just showed, hey, we actually can like cut down by a drastic factor like slash the entropy required by half or something. So very, very significant decrease in the security of the algorithm. So imagine that just happening randomly, unpredictably every now and again in a drop like this that mathematicians across academia let alone the small number of cryptographers in the US government with their limited bandwidth or focused on other things could possibly even peruse and assess. And a frontier lab can just go and drop this stuff and the government can't do anything about it. So we're rapidly getting to the point where for a number of reasons, like I'm not saying math results need to be born secret or whatever, which is sort of like a thing that applied to nuclear, nuclear secrets, born secret means the minute you make it, it is classified. But something needs to be done because when you're, when you're looking at it, I was quickly producing an ass clawed, what are some of the most kind of cryptographically relevant proofs in this list? And there's not none of them. There's nothing that's like a, you know, like the clawed one that's just sort of like a knock down argument. But there are things that kind of flirt with it. They've got this like unique games proof that actually shows that some cryptographic problems are actually harder than we thought, other than they might have been. So, you know, in some sense, that's a boon to security, but it didn't have to be. There's a whole bunch of like sort of more minor things that do kind of weakens and subtle ways, the security of some algorithms. But overall, I mean, it's more a question of what then do you get with the next generation of models or even with the current generation of models? If you throw instead of a million dollars of compute to 10 million or a hundred million dollars of compute at these problems, because you value them. And so if I were over at the NSA, I might actually be pretty keen on finding ways to funnel money specifically into these kinds of problems because, you know, your adversaries are going to be doing the same on like a six-month delay. And so this is actually a much, much bigger deal. Like math is a big, big deal for cryptography and for other national security applications, in a way that's not always appreciated. And so having OpenAI just draw, like I sure as hell, hope that they consulted with counter parts in the US government to understand whether this was the right thing to do, I would be shocked if they were able to get the kind of review that's needed for 372 different mathematical results where you need like PhD level mathematicians to assess them and rent. But I think this is a bigger deal than just the drop itself, basically. Yeah. And there's various aspects of this, obviously, as an indication of intelligence, this is quite significant. To my knowledge, a lot of these are sort of pure mathematics. So no practical effect, but some very interesting results and surprising results across various domains, many people pointed out tighter or lower bounds of big O notation over various things, like you can do, I think, arithmetic in some really surprisingly efficient way that you wouldn't think is possible, but some things are insanely possible. The other aspect here is as it relates to mathematics community, so they do say that they consulted. So they have the advisory group on mathematics and artificial intelligence. This was put together partially of open participation in late September, about two weeks ago, September 21st, that followed pretty quickly after an open letter called, that was called a severe misalignment of AI in mathematics that had various points about how this stuff should be done effectively. Part of which was not treating it as marketing, part of it was about disclosing more details on what prompts were used and what costs were used. So at least in part, openly, I was taking into account the feedback of how they would release it. This was released on GitHub. They know that they're open for paper visions and citations, they publish all the results. They are continuing to explore other community hosted alternatives for this release, which meet the committee's guidelines. And they say for future releases, we are committed to further improving quality of the paper's via citations, mathematics position, and presentation of the results for better understanding. So in this ongoing conversation of whether mathematicians are angry, I suppose, or how this is impacting them, I think no doubt there is a sense of shock probably, but that's just the world we live in. So we're going to see this across many domains we've seen it in programming, those people in software engineering know that that was like a wave that just rolled and in six months the entire job was transformed, such that you don't write code anymore. We saw signs of it in 2024, you used AI-assisted coding and it changed in part, but it wasn't anything that new, now it's qualitatively a different occupation effectively, at least unless you're like a senior software engineer who wasn't writing code hands-on anyway. So this appears to be the case that this will happen in mathematics, it may then happen in physics, and so on and so on. Well, that will be it for this episode, I guess it wasn't quite as quiet and uneventful as it made it out to be after opening. As usual, thank you for listening to this week's episode, especially if you made it this far in. Unfortunately, we didn't have time to respond to comments, we'll try to do some more discussion of feedback and other things on the next episode. We always appreciate it when you do leave any sort of feedback on YouTube, but you also seen some more people making comments or posts on Twitter lately, which has been fun to see. We also appreciate your views on our podcast and if you share it and all that, but really more than anything, we appreciate that people listen and hopefully get value from the show. So do keep doing that. [Music] [Music] [Music] [Music] [Music] [Music] [Music]