OUT OF BAND | The Breaking Point: Inside Mythos' Zero-Day Machine with Anthropic's Nicholas Carlini

[MUSIC] Hi everyone, welcome to a special edition of the podcast. We're calling out a fan. To catch a thief was always designed to be a long-form documentary series that would let me go deep on threats like Chinese cyber espionage that I had tracked for years, but felt like no one had told the story end to end. In a few weeks, we'll watch season two of to catch a thief, and it could not be more excited to bring you that season. But the world is moving faster now. AI is unfolding at overwhelming rates. It's happening in real time, and the world of AI-driven threats is going to look very different in three months, and frankly, we no longer have the luxury of time of waiting months to get you these conversations. Some of these conversations are just too urgent, too consequential, and so we're breaking format. Out of band is where I will bring you one-off urgent conversations that are raw, immediate, and critical to have right now. And today is one of those conversations. I feel incredibly privileged to sit down with Anthropics Nicholas Karlinny. What Nicholas is going to walk us through should give all of us pause. So let's begin. Nicholas, welcome. Thank you for having me here. Thank you so much. So, apologies for this preamble, but I think it's important to level set here and why I'm really happy that you and I, in particular, are having this conversation at this moment in history. I was at the RSA conference the other week, and everyone was talking about agents, and agents going off the rails, and everyone was using the same buzzwords, but it was sort of this theory calm, I felt like, especially knowing that what we're in for over these next couple of months and over this next year of what people are calling the Volnipocalypse, so not kind of thing. And so I got home from this long week, and I flipped on my computer to see what I had missed, and half a dozen people had tagged me in your unprobbeded talk. And I sat down and I watched it, and then I watched it again, and I watched it again, and I realized that what I was really watching was act four of my book. And for those of you who've never read my book, it's called, this is how they tell me the world ends, and it's about something that started out as really a niche technical topic, which was the zero-de-market, but was growing into something else, and just a backup for a moment, what is a zero-day, what is the zero-day market, a zero-day is essentially an undiscovered bug or a flaw in code that no one knows exists. And if you can find that zero-day bug and you can develop the means to exploit it, you can potentially exploit it for all sorts of things, like spying on people or potentially sabotaging systems on the other end. And what drew me to the topic was really the humans involved and the moral hazard involved and then sort of the national security implications of it. Because I was learning that zero days have real value. If you find a zero-day exploit that you can execute remotely and things like your iPhone's iOS software, you can sell it for a million dollars, millions of dollars, if it's a really good one. And there are buyers out there that at that time were willing to pay 10 million dollars in some cases for a really good iPhone iOS exploit. And so my questions initially were, well, who's finding these? How are they finding them? Who will they sell them to? When will they decide not to sell them? Who's buying them? How are they using them? How will a researcher feel if they find out that that zero day has been used to make a dissident's life a living hell? What about the fact that they don't even really know how it could be used once they sell it and it gets thrown over the fence? So it was really like the moral hazard of all of this. But something else was happening as I was asking these questions, which was Mark and Dries' famous line of software is eating the world. You know, this software wasn't just in our desktop computers or our phones anymore. We were now finding zero day exploits in the software that runs our pipeline networks, our power grids, our water treatment facilities. And I was meeting hackers who were really mercenaries who told me they were specifically looking for bugs in those systems and they were just willing to sell them to the highest bidder. And then the questions became, who will you sell them to? Who won't you sell them to? And really no one was talking about this. And so I decided to write the book because I felt like this was a can of worms and a debate that needed to be have beyond the usual suspects. And if my editor had come to me and he had said, Nicole, I want you to write a sequel about your book. And I want you to write a science fiction book about how bad you know this could go or how far this could go. I would write something that doesn't look that different from what I heard you present at the unprompted conference, which was that machines, LLMs can now find zero day exploits and exploit them with limited human guidance at all. And that this capability is really no longer about any kind of moral hazard or human decision making. We've really got into a point where the LLMs can now find this. And now we have to decide what do we do with it. And so I will now shut up. And I will now ask you to just tell us what it is you've found and what it was you presented at the unprompted conference. Yeah, so this was a talk that we gave prior to the existence of Mythos Preview. So this was we are at the time using Opus 4.6. And we were trying to understand what capabilities the model had. And yeah, we really found that it could find the class of vulnerabilities that expert security researchers try to find. At the talk I presented one of the ones we found in the Linux kernel. This is probably the most widely used operating system that you wouldn't know because it's running on all of the IoT devices, the core behind Android. And just a pause there. Like 70% of the world's smartphones rely on the Linux kernel. Like Waymo, self-driving cars rely on the Linux kernel testless, power grid, energy controllers. Like it is it is a foundational piece of technology that has immense control. And if you can get access to it, you can do lots and lots and lots of things. Yeah. And yeah, we had found a bunch of vulnerabilities. I presented one of them that I thought was particularly clever that the model had found. This one was one that wasn't likely to be exploitable in most of these devices because it required a particular service to be turned on. And most the devices won't have that service enabled. But it's the kind of thing that I presented to a bunch of security researchers in order to say like, look at this type of bug, you would have been very excited if you could have found this bug. A model found it entirely automatically with no intervention from any humans. And that was right two months ago or whatever it was. Then we announced Mythos Preview, which is another step up. It has finding even more interesting, even more compelling vulnerabilities. But the thing that really surprised us with this capability was that not only could it find the bugs, it could also exploit them very well, which wasn't something that we could do exactly with open 4.6. It could do it in some cases. For some simpler things you needed to exploit web applications, it could do it. But it was unable to exploit, yeah, things in Linux kernel like that. But with Mythos Preview, we were finding that it was writing the kinds of exploits that I know in principle that they work, but I don't actually understand how. I could learn if I needed to, but it's beyond my exploitation abilities now. And so it's like, yeah, they're getting really quite good with this. Okay, so step back a little bit. So just on the Linux kernel exploit that you presented, how long would it have taken you if you just focused on the Linux kernel to find this bug without any LLM capabilities whatsoever? Yeah, so it's very hard to say this question because there are lots of bugs. And so like, saying how long would it take to find this bug? I probably would never find this bug. Right. I'm maybe a question I could answer as how long would it take me to find an equivalent class of bugs if I was looking for something like this bad. And I really don't know exactly how long it would take. Probably would be on the order of a couple weeks to months to find something like this. It's not the case that these are found never, but they're found infrequently by security researchers. And what was the prompt? Like was this a complicated prompt that only Nicholas early needed to have written or is this something my eight year old could probably figure out he's pretty technically adept. Yeah, no, it's, yeah, so the prompt is very, very simple. We've changed it a bunch of times. It doesn't really matter what we say. The basic setup is we tell the model we want you to find a bug. This is the code base we're looking at. You have complete access to the machine. Go forth and find something for me. And then we add a little bit of sophistication here when we say something like please focus on this file. And we do that because this way we can run the model a hundred times each time looking at a different file. And the same way that a person would read all of the files individually we make sure the model is thorough. And so it checks each file one by one. But yeah, we don't do anything that complicated with this. The models have gotten good enough now that you don't need to design fancy custom scaffolds and tell them to look in certain bug categories. And we've often found if you do this, you end up just constraining them more than actually giving them a helping hand. Because if you tell the model, only look for certain class of bugs, then it might ignore a very obvious bug that's not what you're trying to look for. And then you end up missing something you would have thought. And how long did that take? Yeah. So we run this a bunch. The run that we did at the time was probably a run around maybe a couple hundred to a thousand files we ran it over. This would have taken a couple of day or two. You could run a bunch of these models in parallel. That's one of the reasons why they're kind of scary. And the couple days of this, each individual run probably lasts between 30 minutes and a couple hours, depending on the specification of the bug. But it's relatively fast. It's much, much faster than it would be if you had a human. And I can't just spawn 100 humans at any point in time and then have them go away when I don't want them in the same way that these language models can, which is like can be used for good, but it is oftentimes the little concerning. OK. So I've now watched most of the interviews who've done at least over the last six months. And what I appreciate is that you said originally we were an LLM skeptic. You like to kind of like prod and poke these things and make fun of them when they would break down. Talk to me about the journey from that point where you're just poking at them just to make fun of them to now where you're at the point where you're watching them, not only find these Linux kernel level exploits, but exploits where they're doing the autonomous exploitation. Yeah. I like to say that I'm a realist. I don't know. Everyone thinks that says this. The early models were very, very bad at doing things like this. They could, if you showed them an isolated piece of code, say, I think this is a bug here. But if you showed them a perfectly secure piece of code, we'll also say the same thing. That thinks this is a bug here, even if it's completely exploitable. And then the language model's got better. And it became harder and harder to make the argument that there wasn't something to be seen. And so when I joined in foropic last year, one of the first things I did was start trying to understand how good at that time Cloud 3.7 saw it was at doing this. I had previously that measured GPT-40. I looked at Gemini and they just didn't have it at the time. And then over the course of the last year, the models have gotten a lot better at code. And I think I ask any programmer, they'll tell you this. But it's a consequence of them getting a lot better at code. We've watched the benchmarks go up on how well the models can find vulnerabilities. And it started at whatever, 13% of the time they could find vulnerabilities. People knew existed. And then went to 30% of the time they could find vulnerabilities. People knew existed. And around November was 60% of the time. And at a certain point, we're like, OK, it's finding the bugs that we already know are looking for. Can we now find new bugs? Because there's always a concern when you're measuring these models. Are they just memorizing the answers? All these bugs have been announced before. We find them by looking for CVEs. These are bugs that someone has written up published on the internet. Because the model's trained on the internet. You might be worried about contamination here. And so we eventually switched our mind to, well, what if we just asked them all to find new bugs? And it's a very, very nice held out evaluation. Because if the model can do this, it means that it must have in some sense generalized at least a little bit. And the first time we ran this, we came back with a bunch of bugs. We wrote a blog post in February with maybe, I think, we said like roughly 500 of them we had found. And then kept getting worse from there. And we kept finding more. And yeah, then with the new model, we ended up announcing it was finding and exploiting a bunch of them. And would it make the logic leap from discovery to exploitation without any human guidance? We ask it to do it. The models are pretty good in most cases about doing what you're asking for. I mean, there was regularly the oops, the language model did a thing that we didn't want to. There are like, entire sections of our model cars that talk about this. It's definitely not a problem. But the majority of the time that do it as we want. And so, yeah, we very much asked the model to find the bugs. We asked the model to exploit them. It's not yet the case that these models are like finding and exploiting these things completely autonomously without being told to do so. But the capability is there. And that's, I think, the thing that I care most about is understanding if someone had access to these models and asked them to do this. What could they do? The zero-day market was still largely driven by scarcity. And there were only elite hackers in the world. Many of them at advanced tier one intelligence agencies that had this capability. But what added urgency to my book, I thought, was that the cat was now out of the bag, especially after Stuxnet. When people saw what a chain of zero-day exploits could do together to a nuclear enrichment plant, like every country with any resources at all started looking at what else can they do? And how do we get our hands on this capability? And those that didn't have you or someone in-house that knew how to do this could now, by their way, in some cases, into this market. And it was getting commoditized to the point where now we are seeing these click and shoot tools, click and shoots by where that had these exploits built in, that enabled surveillance and worse in some cases. And the thinking was still pretty static. I was learning inside intelligence agencies that there was still a little bit of this arrogance that only we can find these bugs. And there was even-- and I thought this was particularly eye-opening-- there was even an acronym, Nobus, that stood for Nobody But Us. If an intelligence agency found a certain zero-day exploit that they believed that only they could exploit, they deemed it Nobus, and they held onto it, and they went hand it back to the software maker to get it patched. But I was learning that Nobus was breaking down. There were now other hackers all over the world. In Argentina, I went down to Argentina and met with Subrall who said, no, we'll just sell these to anyone. And we're not only going to sell them to Western governments. In fact, maybe you want to go back and read up on some history of what the US has done here in South America. Everyone had their own moral compass of who was good, who was bad, who they might sell to, who they might not sell to. But for the most part, it was still driven by scarcity. And what I'm watching right now and what you have essentially presented is the end of scarcity. And-- Maybe. Yeah, how do you think about that? I mean, there will still be guardrails around who can actually get the level of access that you have inside anthropic with the models. But when the models can do this, then it becomes a game of who gets insider access to the models. So how are you thinking about this? Yeah, this can go one of two ways. And it's very hard to predict ahead of time what's going to happen. I can tell you the hopeful case first, which is that maybe what happens is that these models are really good at finding bugs. Finding bugs is easier than exploiting them. And so you can imagine a world where these models are sufficiently good at finding bugs that they're not going to fix all of the bugs. But they might fix a much larger fraction of the bugs than we could find with just people looking. And this will make software more secure. And this is not completely unprecedented. Something 10, 15 years ago, there was a time when people just started using fuzzers, which are these software programs that another way of finding bugs. They spew random inputs at some program, see if it crashes. And if it does, then you found something. And it's not completely random. There's a bunch of sophistication here. But when these first came out, people were worried that, OK, this is going to give the attacker's the ability to find a bunch more bugs that they couldn't find before. Except it turned out that, for the most part, they benefited the defenders a lot more, because the defenders can just before they release their software, run the fuzzers over it. Google has this project, OSS Fuzz, that continuously fuzzers all the existing code that exists out there. And for the most part, the fuzzers haven't benefited the fender a lot more. And the software is much more secure because of this. So you can imagine a world where this happens. And we end up where things are a lot better. But at the same time, you can also imagine a world where maybe you can fix more bugs, but you can find equally more. Maybe the number of bugs is essentially infinite. And you end up having removed some finite fraction of them. But there's always a next bug that can be found. And someone just needs to run the model for a little bit longer to come up with that one, too. And I don't know which one it's going to be. I'm hopeful that we're in the defense favorite world. But I think no one can tell you a certainty that this is what's going to happen. And I think the reason why we're doing this research now is because we want to prepare ourselves to think about these questions. Because I sometimes see my job as not making sure that everything necessarily goes well, but that we're not surprised. At least if things go poorly, I want us to have been, [BLANK_AUDIO] "Okay, this was a thing that I considered. I can't force everything to go well, but I can tell you that the thing that happened was one of the things that we thought about ahead of time. And it's not a complete shock to us that this is something that can be done. And so yeah, my hope is that the model will find and fix as many bugs as we can, but maybe the attackers can use it more. - When we talk about the attacker in this context, I think the attacker we think about the most right now is China, mainly because of some of the Vultiphone attacks we're seeing right now. And also because in China, they've made zero-day discovery, a major national security priority, we've seen them in national security laws that force Chinese researchers, Chinese citizens to turn over any zero-day they find. Having gone deep on these capabilities, are you more worried about like a Tier 1 advanced nation state, using an LLM to find and weaponize zero-day exploits at scale or are you more worried about the hacker we've never heard of who has a dark curiosity in watching the world burn. Which of those two in that dark spectrum do you think about more? - Yeah, so I tend to add a third category here, which is in the middle there's the ransomware cyber criminal who's not a nation state, but is financially motivated. And so I think these are the three categories I intend to think about. It's again very hard to predict. And you can make a case for each of these groups, you can make the case that this changes things or it doesn't. The case that this changes things for the Tier 1 adversaries is that previously they would need to put in quite a lot of work in order to find each of these bugs. And now you can find them at much larger scale, they have the expertise to know how to use the tools. And so they could potentially use them a lot better. The reason why maybe it doesn't make as much of a difference for them is they already had a zero day and everything that they wanted. And so how much does it really matter if you have a second or a third or a fourth? And maybe it makes a little bit of a difference. Maybe it makes not as much difference. I don't know, I'm not in national security. I don't know what's gonna happen here. I think the thing that's very clearly makes a difference for is the people who don't know anything. You know, today if you have some random person who wants to break into some computer system, they can't. It's just impossible for them to do that. With these models, it becomes possible. They have to just ask and the model will produce something for them. It's unclear how bad this would be because they need to know, what we can produce right now with these models is we can produce an exploit that gets some code execution. But if you go to someone and you say, like, okay, great, what are you gonna do with that? You have code execution now what? The person's not gonna have an idea of what the next thing to do is like how, going from code execution. Not just the prompt, like a chat should be tiered. Sure, yeah, you can ask. Yeah, yeah. But the models will almost certainly refuse there. Like there's like, you know, it's very easy to disguise. I'm trying to look for some bug. Please help me find this. It's very hard to be like, so I have code execution in this sort of company that I'm attacking. Tell me what are the three things I should do now that I have this to get as much money as possible. Like this is like a much easier thing to defend against when we're trying to make these models robust to jail breaks. But even still, I think that if we're worried about this, I would worry more about the like middle-tier attackers who are already, they know how to do the monetization of exploits. Like this is the thing that they exist for. And so I'm pretty worried that we, if we give people unfettered access to these models in this middle tier, that they might be the one to most uplifted because they know what they want to try and achieve. And they don't have the resources of the nation state. They don't always just have zero days sitting around because they have like a couple hundred thousand dollars. They can't buy a zero day in the same way. But if they could produce them on demand, then they could cause a lot more harm. And so this is, I think for me, what I tend to think I'm most worried about. But I don't know, like economics is hard. You have these cases where supply goes up and then it turns out what do you know demand goes up to and then profit go like it's like, I don't know, this is like economics of the field. And I don't know anything about that. I think there are some people who talk with certainty about what exactly will happen. And the only thing I can say with certainty that they're probably wrong, that things are going to be complicated and we should know what the capabilities are. And then what I do, I pass off this knowledge to someone else who then thinks about this. And then they go and try and produce the best explanation that they can for what's going to happen in the future. So what you've demoed on stage, you were using the previous cloud model that all of us can get access to if we want. - Yes. - But talk to me about what you've been able to find with me though, and what sort of step change you've seen. - Sure. So the primary thing that we've seen here is, as I mentioned earlier, the export agent capabilities will come back to the second, that in a second. But then the vulnerability finding capabilities, it's able to find vulnerabilities and systems that are much more sophisticated than we were able to find with previous models. Where with previous models, when we had them find bugs for us, there were the kinds of bugs that you could explain to another security person in a sentence or two. You took some data off of the network, and you forgot to check the length, and you put it into a fixed size buffer, and then the machine crashed. Or you took some data and you inject it into web page, and now someone gets code execution from access. These are the kinds of bugs that people understand very, very well. And what we're finding with this model and recent models like it, is that they can find the kinds of vulnerabilities that take a couple of paragraphs to fully explain what's happening. So yes, we found one bug in OpenBSD, that's 27 years old, where in order for the bug to fire, it had to find a combination of three or four weird things that all had to go wrong at the same time. It turns out there's some integer that you can make really big, and then it gets added to some other thing, and because of the way that they did less than, and some signed integer arithmetic, it turns out that there's some other bug that was also hidden, sort of like three or four things come together in just the right way, and then you get a vulnerability. And these are the kinds of things that people have a very hard time trying to find, and these models are starting to now produce these kinds of reports. There was a blog post that the Mozilla team put out after their use of Mythos preview, and the thing that they said somewhere in it was, right now, at least it's only finding the kinds of bugs that all of the bugs that it could find were bugs that were understandable to humans. This is like a good thing. It's not producing these completely understandable bugs. But so they're still understandable, but they're at the edge of what we are, you would expect a normal expert human to be able to find. And then, yes, the other side is the exploitation. OK, stepping back for a second. The vulnerability is showing the demonstration of a flaw. Like it says, something could go wrong here. It doesn't actually do anything with it in particular. The analogy I like to make is, I'm picking a lock, and you have the movie scene where someone's on a safe, and I have the desoscope. The fact that the lock makes a noise when you turn the dial is the vulnerability. The exploit is knowing how to use that in order to actually open the vault. And you can be good at finding bugs without knowing how to exploit them. The model is now gotten good enough that it could exploit them. And this is like a specific skill that takes people a very long time to learn how to do. It often involves esoteric knowledge about random things that you have no reason to want to learn, except for the ability of knowing how to exploit things. But the thing these models are good at is that they have knowledge of all of the things. And so you can give them a system that depends on some weird network protocol, and they know how that protocol works. They don't need to go and spend a couple of days learning about that in order to then go and pick what to do next. And maybe that goes to a dead end. And if a human you were just wasted a couple of days figuring this out, but the model-- it already knows that. It dismisses it, moves on to the next thing. And so the models are very, very good now at producing some of these exploits in a way that would have taken-- we've showed some of these exploits to some very skilled penetration tester people. And they've been saying, yeah, some of these would take them a couple of weeks to have written. And they're very impressive. When you start seeing what Mithos is capable of, you write this internal memo in Therophex saying, we're finding real exploitable vulnerabilities for seeing that this model can exploit them. Maybe we should hold off. I kind of want to pause on that human moment, because I kind of picture you, like the security researcher equivalent of Tom Cruise and Jerry McGuire, where writing this man of bestow in the middle of the night, talked to me about what you saw that was so alarming that you thought I need to make a good solid argument for holding back here. Yeah. The primary thing was the fact that these models were finding bugs that really felt like the kinds of bugs that I had never found myself before. I used to do some security research. Before I got into machine learning, I worked for a little while doing pen testing of software. So I know what it means to find real bugs. I was fine at it. I was not an expert, but I was competent. And. And the model was finding the kinds of bugs that I knew that I would not have found other good people would be able to find. And I knew I would not pretty much work into it. Like I'm certain you could find even more bugs if we had this model for longer and we scaffolded it more appropriately. Like again, we're doing like the most naive thing when we're asking for these bugs. And yeah, it just felt, you know, again, sort of my, I want to make sure we're not surprised. And as long as we are still finding out new capabilities that are surprising in us, it feels more appropriate to be like, let's just like make sure that we're doing this in the most pragmatic way that is not going to cause bad stuff to happen. And we can just do a slow, careful rollout. And, you know, it's entirely possible that if we look back with three, five years hindsight, people will go, eh, you know, turns out not a big deal, just like fuzzers, like, you know, people were worried at the time, I said it was okay, but I would rather have been a little too careful than a little too reckless. - Everyone's, and I think rightfully so, including myself, are focused on sort of this automated vulnerability discovery exploitation. But I want to know if you've been testing out other attack vectors that this is unlocking and what that looks like. - Yeah. Not as much as we should have. And the reason why is that the kinds of things that like, yeah, so I'd be very interested to learn how good is the model at lateral movement? This is the skill of once you've gotten into the system, you know, getting access to an adjacent system. The problem is I don't have a good test ground for this, right? Like the nice thing about finding and exploiting vulnerabilities is I can download the Linux kernel and then run it completely independent from the offline, from the rest of the world. And like if I go and cause havoc, it will only be havoc on my machine. So it's very easy for me to know that like, I'm not going to cause harm. I can't in the same way try and unleash the model on someone's suspecting company and then say, like, okay, great, now go to the lateral movement thing. Like this is, you know. - We're not set up for that. - Yeah. And same thing for like, you know, most compromises at least up until now aren't with zero days. You know, most compromises are someone misconfigured a service or someone forgot a password and put it somewhere they shouldn't have. Again, very hard to test for these things because I'm not gonna go searching the internet for misconfigured services with this model. You know, I feel comfortable doing this. Like if it's just me, this is what a lot of people do for bug bountyists is they find these kinds of non-zero days. But you know, knowing about kind of vulnerability, someone made a mistake somewhere. But again, going back to the like, sometimes the model makes mistakes. There's no way that I feel comfortable pointing the model at some actual service that if the model goes a little bit too far, like, oops, drop the database. This would be quite bad. And so like we're not testing these kinds of cases. And so I'm, this is my thing, my biggest area of uncertainty right now. - What is still bottle necking kind of end to end autonomous exploitation? Like is there anything that's falling down? You know, if you gave me those unlimited tokens, what is it still failing at right now? - Yeah, so the reason I think it would probably fail if you had to try and do like an end-to-end operation with just the model is that yeah, finding and exploiting the vulnerabilities only the first step. And then you have to do something else. And we have at present at least no evidence that it's good at this. It might be, but we haven't been able to do this in quite the same way because, you know, people have, you know, software on antivirus end point protection, these kinds of things that are meant to detect post-compromise. You could imagine a world where the model could get past these things too. You could also imagine a world where, if the model makes one mistake and probes some honey pot that it shouldn't, that you get, it's immediately detected and then, you know, the things get locked down. So I don't think we have any evidence right now that the model can autonomously end to end and do a full operation. But the kinds of things that it can do are quite compelling and I tend to, you know, I'm security first, I tend to assume that the worst is possible unless someone shows me otherwise. You know, the, there are like guys, you know, I'm, I'm, I'm, I'm, some most of the time doing academic stuff. There's an entire field of research on post-compromised cryptography. We don't have a quantum computer and that we do research on this because we're worried about the potential for what could happen if. In the same way, I tend to assume for these, these models now that we have at least some convincing evidence that it's not gonna be that far away that we ought to spend a little bit of time assuming that this is gonna be possible in the future even if it's not something we've demonstrated right now and we should try and consider what the implications of that might be so that we're not caught completely unprepared. - One thing you said that struck me beyond that analogy of post-quantant cryptography, you know, that we are, we are very much preparing academic institutions are studying this as a field and it hasn't happened yet. You know, we don't have post-quantum computers yet but they're studying it because we assume that once we have a major breakthrough, we need protections. Are you seeing that level of focus and coordination yet on what happens when these models unlock and to end autonomous exploitation? - I think we're starting to get there. I think, you know, like me, most security people were in our skeptical about the AI hype and whatever. I think, you know, you have probably read people talking about five years ago how these models are going to be finding vulnerabilities and it's going to be so bad. And you know, when I was reading this at the time, I mean, I don't know, science fiction, it's not going to be in the near future. And so I think, you know, reasonably, I think people didn't think of this as a thing that was going to happen. I think, yeah, within the last couple of months, the security community has really started to understand that this is realist time. And I think, you know, it's unfortunate that there was a bunch of crying wolf, but you know, the, what's the moral of the story? Eventually the wolf comes until you can, you know, you can have this case where people say something's going to happen and they could have been very silly for the first several years of it. And eventually, the models get to the point that they're actually very good. I think they're not yet as good as some people are saying, there's always going to be people who are hyped in this up more than they should be. But the capabilities are there, they're real. And I think the security community is coming to understand this. I wish they would be moving faster. We need a lot of people to be working on figuring out what to do here. Doesn't have to be everyone thinking about the language models. I mean, even just getting it to be the case that we can do classical security stuff faster would be a big help as the time to exploit goes down. So I hope that people will start to respond much more quickly. And that's the reason why we're doing the work is because I don't want to watch the world burn. So of the exploits you have talked about, these are ones that I'm assuming you now have fixes for. Of course. So I do have fixes for. Talk to me about the ones that don't yet have fixes. And I'm sure you can't talk about them at any level of real detail, but just talk about some of the capabilities you've seen. Yeah. So yes, we have a number of exploits that we have not yet fixed. And so I think one of the things, one of the annoying things is that I currently know more about the capabilities of the models than I can really say responsibly. When some people sort of have been talking about, "Oh, you know, the exploits aren't that bad. "I have to remind myself that what they see "is only what's public. "I know some of the ones that we have that are not yet public." And some of the ones that are not yet public are the ones that are harder to fix and are in the bigger institutional systems that are just takes more time. But yeah, so on most web browsers, on most operating systems, we have full exploits, the kinds of things where you visit a web page and then the model goes and accesses your bank records. Either kinds of things where on most operating systems an un-cliffled user can become administrator. The kinds of things where if you give me your locked phone, I can download the pictures off of it without unlocking it. These are the kinds of things that we have, which are sophisticated exploits that people used to think of as only the kinds of things that tier one adversaries could build that the model is producing for us. And it's like me and a handful of other people are anthropic who are doing this. None of us who are doing this are experts in this and would have absolutely no hope of doing this unassisted. And over the course of a couple of weeks with the model, we found dozens of these kinds of things. You know, I had this moment in my book Research Rows in Argentina and this wasn't on my question list, but it just reminded me, I was meeting with this one, Argentinian security researcher who specialized in kernel CRD exploits. And he turned to me at one point and said, you know, I know a nuclear physicist who was kidnapped. And he said, I kind of feel like we're the new nuclear physicists. And I kind of feel like you're the new nuclear physicist. Like what you disclose, the attention it's gotten, what you just described in terms of the capability. Like if anyone was going to be a target right now, it might be you. So like, I don't think so. - I don't think so. - Okay. - Yeah, those are like the me of the thing. Like I don't think that a lot, like I think anyone who has access to this model could have been me. Like I think like, you know, we have hired, some people at Anthropic who are very bright, but are not experts, security people who have been working in the field for maybe a year or two. And they have similar kinds of things to what I have found here. You know, I think it's easy to say, I guess you're the expert, you know, it's only possible because you're the person who-- Maybe it's just that you're insider access. It's now such a target. Yes, this is one. And that's-- and you know, there was a scary headline last week that some, you know, unidentified unauthorized users had gotten access to mythos preview. Like, should we be really worried about that? Yeah, so I've seen some of the reports here. I like to play with the models. So I haven't been following very closely what's been happening here. Yeah, the thing that I try to make sure is that, yeah, I have a good understanding of what's actually happening with the models and that we're, yeah, not surprised what the capabilities are, but I don't have a good understanding of what's been going on with this. Do you feel like it raises the insider risk modeling at Anthropic and at the frontier labs? Like, I remember, you know, take a social media example. You know, there was a case where the Saudis planted spies at Twitter because they were trying to get insider access to some people on Twitter, some users who are being critical of the of the monarchy. I imagine that if there aren't already, people are trying to place spies inside the frontier labs. And so what do you do? Do you look at, you know, beyond code execution or data exultation? Do you look at like bizarre code audit requests? Like, how do you even look for that? That's a very good question. I don't know. Yeah, like, you know, there's, Anthropic, there's several different departments. There's like the security team who handles this and they actually know how to make sure Anthropic is secure. And that's, yeah, not where I sit. I sit on the sort of safe cards in front of your red team whose job is, okay, here is a language model. Tell us the properties about this objects we have created. And so, yes, I'm sure that they have the work cut out for them. They're a very smart group of people. And yeah, I'm sure that they're doing, you know, as best as anyone can do on this. But yeah, this is a very hard problem to solve. Okay, I want to talk about the next model and the next model after that, you know, on some level, we know some of the other major frontier models already have or they have talked about some level of exploitation capability. What does it mean when this is like table stakes for every new model that comes to market? You know, what happens to a project glass wing, which was this decision to hold back and kind of give these capabilities to the defenders to give them to buy a more time? What happens when every model has this capability, you know, does, does glass wing become temporal? Do, do we hope that there will be regulation that everyone needs to hold back and do their own version of project glass wing? So, we buy defenders time. Like, what is, what is the future around the next model look like? It's going to become complicated. The hope is that in the best case, we can find and fix a lot of bugs. And it won't be quite so severe if someone were to not hold back or release in the future. That said, there's a very long tail of software. And the number of people who we can give access to this model in a restricted way is quite small. And so we can try and do our best to make sure that some of the infrastructure is protected in this way, but you can't possibly protect all of it. So a big part of why we're doing the project glass wing this time is not necessarily in protecting the infrastructure it is in part, but the big thing that I was excited to learn is we said, well, right to report at the end of like the things that we've learned. And I think this is maybe the most interesting finding for me of what we'll get is like, now that we have this model, we have a bunch of external people who are validating and finding a bunch of bugs with it, seeing what it is that they've been able to, how this affects them. And maybe the answer is for some of them, it doesn't have that much of an impact of us, we're very useful to know. Maybe for other classes of people, this is a very important thing that they had access to early. And then again, it's very useful to know so that people in the future going forwards can do better versions of this. Yeah, we really just want to make sure that for this first model, we handled it in a reasonably responsible way. Going forwards, it's we'll have to be careful, but I don't know what this actually means. There are worlds in which people might be really same models that are this capable open source in the future. As a general rule, these open-weight models tend to be, I don't know, three to 12 months behind the frontier. In a world in which an open-weight model existed that had these capabilities, then you might have to change the approach that you take for stage rollout if people are already doing this in the wild in the same way that when you have vulnerability, if someone finds a vulnerability, usually you give some 90 days to fix it. But it's being exploited in the wild, you might only give them seven days. This is the policy that Project Zero has for how long they wait. You can imagine some of the things for these models. I really don't know how it's going to go. I'm glad that there are smart people anthropic who are thinking about this. I see sort of my role here as providing them, the information they need to make responsible decision. I say, "Okay, here's what we can do. Give an access to this model. You are experts on reasoning about how the world will react from this. Go come up with a response plan that seems good. And so far, I've been pretty happy with the plans that they've been coming up with. But their job is not going to get any easier." Speave of Open Weight. How does what you're seeing right now shift your thinking on Open Weight versus Close Models? We've always had this line in cybersecurity that with enough eyeballs, all bugs are shallow. And it didn't really work out in a lot of cases. I remember covering heartbleed. And here was this major vulnerability in an open source security protocol that everyone relied on from Amazon to the FBI and no one caught it. And then when we looked, it turned out actually it was being maintained by two guys named Steve who were living on a shoestring budget of $3,000 a year. And of course, we can't expect them to maintain this and find every bug. With Open Weight Models, do you think that there is an inherent risk now in that approach that we really didn't have to reckon with at this level before? Yeah. I think there's a good question here. As a security person, I like to think that we ought to be releasing most of what we do as open as possible. I'm glad that debuggers exist. As an offensive security person, you need a debugger in order to go and find vulnerabilities, but they benefit the defenders a lot more than anyone else. And so for the most part, I am very much someone who thinks that you don't blame the tool when someone does something bad with it. At the same time, I also believe that we shouldn't allow anyone to go build a nuclear weapon in the garage. I'm not going to sort of clearly language models neither of these. And so the question is like, everyone I think agrees that there is a spectrum here. Some people think that it's closer to just a tool. Some people think that it sort of goes a little bit further on the spectrum, so it's become closer to a weapon, especially when it can exploit vulnerabilities automatically. Debuggers don't give you exploits, they give you vulnerabilities. And so I think there's a real good question here. Again, I don't know how to think about this. I think that some people like to call it black and white. And this is not the case. There are shades of gray here. And I'm optimistic that we will be able to release these models in ways that have sufficient safeguards that we bias them towards the defense side. This is a lot easier to do when you have a model behind an API. When you release it and you have the open-weight model, it turns out to be very, very easy to remove any safeguards because you just like edit the way to the model directly and then all of a sudden complies with everything. This will become harder to defend against these kinds of things. Again, maybe it turns out that we have this and it turns out everything was fine. That would be I think all else a world that I'd be happier to live in. But I don't want to let my optimism get in the way of what actually might cause some harm. And so I think for right now the approach is let's be as careful as we can and try and do this in a slow, deliberate way. And if it turns out that we are very careful and over the course of the next 12 months, we learn that these are mostly just tools and they're not in like there's better ways of ensuring we're safe. And I'll feel very good that we got there in a way that didn't make a bunch of exploits just like show up on random people's doors without I mean previously expected it. You know, I'm really concerned about what's going to happen with the founding programs. Like we just over the last year and changed, we saw, you know, Expo was being the leader board at hacker one and now we're seeing hacker one kind of shut down the internet bug bounty program. A number of bug bounty programs have shut down over the past year, mostly because they've just been inundated with essentially AI slop. And they don't know what to make make sense of. And so in some cases the noise has stopped, but there's no real relief in that because it's not like the capabilities of the models to find these things will slow down. And so I don't know what the next chapter looks like, you know, I suppose that bug bounty programs will just say, hey, from now on, we won't accept AI generated exploits, but like doesn't that leave an entire attack surface vulnerable. How are you thinking about this? Yeah, it's going to be. Yeah, so there's a period of time starting maybe last year and this maybe goes back to why some security people remain skeptical about language models where the models were good enough to write convincing bug reports, but not good enough to find actual bugs that a bunch of people. Yeah, curl and others closed the bug bounty programs because they were receiving reports that were looked to be high quality, but there was nothing there. Because you know, for the very long time, if the report had the style of a security person, you can like glance at and be like, yeah, like this, this is comes from someone who knows what they're talking about. And then you can spend the 30 minutes to an hour validating that the bug is real or not, but when it becomes so easy to write reports that have the style of real bug reports, but there's no actual vulnerabilities there, just wasting a huge amount of maintain your time. So yeah, so a bunch of security people close down the bug bounty programs and then the models got good enough that they could do the thing for real. And it's yes, kind of the worst of both worlds because now you have a thing that actually is working in practice, but people are used to not that long ago receiving a bunch of reports and I'm sure they still do right like I'm sure they're still getting a huge amount of the reports that are actually just completely invalid. But now there are small fraction of the ports that are valid, they have to be attention to I recently filed a bug report against some open piece of some almost piece of software. And at the top of my bug report, like I do for all of them, I say we found this with the vulnerability and the person who was automatically triaged and closed with the comment we don't take element generated bug reports. I emailed the maintainer and I said like I hate to say this please look up who I am this one is real and they they are fixing it. But like this is like a thing that like you know is going to happen. It's very hard for people, you know, it's the classic XKCD that you mentioned earlier, there's like a couple of people who are maintaining some critical piece of software that don't have the resources. And previously the only assistance they got was from people who like actually were trying to do well. And now you get people who are well meaning but are not using the tools in quite the right ways flooding them. And it's yeah going to become difficult. My hope is that out of these models get better, we will remove most of those cases. My hope is that we'll be able to help have LLM validation of reports. One of the big things that we found that as model get more capable, they're much better at checking the correctness of vulnerabilities. For some classes of bugs it's very easy to check that they're that they're you have a real vulnerability for other than it's a little bit more difficult and the models are now getting the point that they can't with certainty tell you it's there or not that they can discard a lot of the fake reports. And so I think that there's a hope that people can use some of the models in this way to to do better. But yeah, no, I mean like the reason why we want to be talking about this now is because we see a world where people are not thinking ahead of time are not prepared and then it becomes very difficult and it may still be difficult. But at least we can say you have a couple more months to think about how this is going to go. Take some plans, put them in motion and you know maybe it turns out that we didn't act fast enough, but at least we started the moment we really knew that we needed start acting. Most of the CSOs I keep in touch with on a regular basis like have always sounded incredibly overtaxed and overburdened and stressed out and I've always said like the CISO role is unsustainable. And I just keep loading more and more and more responsibility on their shoulders. And just in the last couple of weeks with me, those you know they are getting dragged with four boards and asked one question, which is are we protected against this and any of them that say yes, know that they're lying, you know they they know that this is this is a unique moment and some of them have even sounded pretty panicked. So it's your best piece of advice to them like if they could do one thing on Monday, what would that be? I think people need to just look very carefully at what exists right there right in the world right now and try to use the tools in the best way that they can. And right now there are lots of really good open source models. There are lots of good models that the frontier labs are providing. You know, anthropic recently put out opus 4.7 open AI recently put out GPT 5.5. These models are very good. They're not quite at the level at exploitation that we have with mythos. But you don't need that to fix vulnerabilities in quite the same way. It's like it does find some bugs that we can't find with the other models, but it's relatively easy to swap in a new model. If you have the framework set up to be able to use these things to find vulnerabilities, the processes, getting people to have the right harnesses, getting the infrastructure setup that you can run your code offline in a way that's secure in those little sandbox all this stuff takes time. And getting that going now while you may not have access to the best best model that we can give gives the ability so that you know these models will continue to be improving and you can always swap out another model when when you get the time. And so it was like making sure that people are starting to use these things in ways that they can for defense is I think probably the best case here. Because the people who are trying to find vulnerabilities are also doing the same thing. And if there are people who are not engaging with the technology that they have access to, it will be much harder for them. And so, yeah, I think it's going to become difficult couple of months here. So as we have people trying to very quickly adopt the technology for defense and as the attackers try to start adapting technology our hope. Yeah, I like I sit on the safeguards team and I'm for a pick is that we will make it as hard as possible for people to misuse the capabilities. One of the reasons why we're not releasing the model is because we don't feel like we have a good answer. As of last month and hopefully we'll have a much better one when we feel ready to put something like that publicly and that this will give us a short period of time we were biasing things towards the defenders. But people need to start using that time that we give them to the best that they can and yeah just using any of the models that are out that are on the frontier will will help with this. This might sound like an incredibly stupid question and you can tell me if it's a stupid question but when they even thinking about is that you know with these prompt cardrails there's like always going to be ways around them and it strikes me that it will be even harder for us to set up prompt guardrails in certain languages. And I don't know how we you know almost like our prompt guardrails in English are going to be the hardest to break but it's going to be more easy for I don't know China or North Korea or name Polen Russia to break these guardrails because we don't have the same level of understanding of their language like how did you tackle that one. I put aside even the multilingual aspect of it you know the thing that I spent so I used to do classical security then I went into machine learning security and the thing that I spent 10 years of machine learning security looking at was the problem what's called adversarial examples which is the general property that you can take these machine learning models that are very very good on distribution and an attacker can construct small perturbations of inputs that make the model do very bad things. We had these first on image classifiers then we had them on audio classifiers then we had them on natural language processors then we had them on. On my own my these large language models where yeah we were among the first people to show that we could jailbreak at the time GPT for so we can make it comply with any request that we we asked of it. We had some examples like tell me how to build a bomb with a model would refuse we constructed question we ask this depends on random tokens the end the model gave us an answer. So I understand very very well how hard of a problem this is and this is part of the reason why we want to be very careful here is that you can make these models very easily refuse when you ask the question in the most obvious way. But then you ask it in either a more subtle way or you intentionally yes add some kind of jailbreak to the prompt and other model. complies with you and you're entirely right that in non-English languages, you can happen even more. It's very hard to solve this problem. Anthropic has done a bunch of really good work here when Anthropic put out Opus 4, along with Opus 4, they released another paper looking at doing what they're calling constitutional classifiers. These are classifiers that are designed to make it difficult to elicit harmful capabilities initially in biology so that someone couldn't use it to produce a bio-eapon or something like that. And this is something that over the last year has become very, very hard to job-break. I'm optimistic that you could have something like this for security capabilities for misuse. The challenge is that whereas there are not many good uses of creating bio-eapons, there's lots of good uses of finding vulnerabilities and so it's a little bit more difficult of how you decide when something should be allowed or not. You can imagine versions of this that might just refuse all find vulnerabilities. It's much easier to refuse exploits in finding vulnerabilities. But yes, this is a very hard problem. I understand, yes, probably better than most people just how hard it from this is. And I'm hopeful that people will be able to come up with something here, but yeah, it's again not going to be easy. Code Menders, how far are we from LLM capabilities that can actually mend by code and it's close to real time as possible? Yeah, we're getting there. These models are really good at writing code. They can fix bugs. The challenge that we have, it's relatively easy to patch the individual bug that you found because you have a POC, a proof of concept. And so you can produce a change that makes the attack that you found no longer work. There are two challenges, maybe three. The first challenge is, does a different version of the attack still work? You might have patched one specific hole, there might be other ones like it. The second challenge is you can't break any of the existing functionality. The reason why the models are so good at finding bugs is that it's very, very easy to check if you've got it right because the program crashed or something. And so if you construct the attack and it didn't succeed, the program didn't crash and so it's very easy to know this. It's very hard to validate that this patch is going to work on all deployments of the software, especially when you have some software that's supposed to run on different devices. It's much harder for the model to check all of this. And the final reason why this is hard is that people like the code that they write to look a certain way. And sometimes the model just doesn't write it in the way that makes them happy. And it may be entirely correct. It is entirely functional. It works in all the right ways, but like aesthetically, they don't like the appeal of the way that the model wrote this fix and so they want to do it differently. And so for many reasons, it's much harder to fix these bugs than it is to find them. You could hope that at least for the short term, people will be able to take the initial report plus a patch the model said, this is basically correct. And they'll be able to use their knowledge to much more quickly turn that into like a fully working version of the patch. When I send these bug reports, I always attach a patch. And even when I'm entirely unconfident in what it's doing, the maintainers always say like we prefer that it comes with a patch. Because even if what you have is wrong, it lets us know what you were thinking about. And we can produce our version of the patch much more efficiently. And so I'm optimistic about these patches. Even if people don't end up accepting them just in helping them get along faster. Depending on how the world goes though, I think it's probable that we'll need these tools to be able to almost entirely automatically merge patches. If things start moving at a rate beyond what people are able to do. But there's, you know, even if we could mend all of the code in five seconds, the patch is still easy to get rolled out. People need to update their software. All of this stuff is going to take a bunch of time. And so I think focusing too much on the like getting the patch immediately deployed is, you know, some sense the wrong question is going to be like shortening the entire life cycle of when the vulnerability is identified until the end product is actually patched, which in some cases can be challenging. Yeah. And I think, you know, that was one of the assumptions I made in my book was, well, some vulnerabilities we should turn over for patching. And some people after they read my book said, Nicole, you know, the most dangerous window sometimes is not when the zero day is discovered. It's when the patch is released. And everyone goes to town saying, who hasn't patch and reverse engineering the patch. And often there's problems with the patch. And one thing I've, one thing I've learned since I've kind of gone inside the tent and left the New York Times is how hard it actually is to properly install a patch. Yes. I worry so much that this will become a game of time, you know, that if we're finding these vulnerabilities and exploiting them faster and faster, the bad guys are doing that, then it becomes a game of how quickly can the defenders fix these properly with full coverage? What is the best way to buy them more time when they're going to be under this deluge of bugs everywhere? I don't know. Yeah, it's, you know, we can help them write the first version of the patch faster. We can help them test the code faster. But some of this is the kind of thing that, you know, security people have been talking about for a very long time of we need to reduce the amount of time that it takes for people to actually accept the updates. We need to reduce the amount of time between when the update is available and when people actually are available to like apply it to their own machines. We need to convince people that security matters and they need to apply patches. Some people just don't do that. There's just like, you know, a lot of regular problems and security that we're not introducing, it's not going to be new, but just like you take all of the existing problems that people have and just like add an element of time to them. That's how I feel about this moment. I feel like it's all these debts coming to well ones and like only half of them were technical, you know, it's like our failure to ever really take secure coding and secure by default. Seriously, in some cases, it's moving fast and breaking things and software eats world, but it's also technical debt. It's also like failure to resource a lot of this critical infrastructure that can't even get a patch now if they want it. It's like failure to educate Congress and regulators about this technical topics in a way that they're able to action on it. And then you look at all of that together and it's kind of like a system failure. So I know security researchers don't really ever want to dip into policy, but I will ask you the question, like if you could wave your magic policy wand, what's the one thing that you're like, I wish regulators would do this. I have not even considered the question. I just want people to think about the problem and try hard. And at this point, I want to play the clip from your unprompted talk, where you make a really impassioned plea for help. Speaking of not as an anthropic employee, like I don't really care where you help, just please help. Like I would be great if you would like to help us out of topic too, but like just the world will need a lot of people to be doing a lot of this work and it needs to happen soon. Like order months. I think waiting a year is going to be too long. Soon it's not just going to be me who has all of this, but it's going to be anyone malicious in the world who wants. So really, I'd encourage you to try and find a way to see if some particular set of skills that you have could help us make sure things go well over the next couple of months and over the next couple of years because I am quite worried about how this direction is heading and we need all the help we can get. And what do you think even cybersecurity experts are missing about this shift? Not very much. I think people are starting to, like it's not very complicated. Right? Like sort of there used to be a thing where the only people who could produce vulnerabilities and exploits were expert humans. Now many more people can do this. And so I think like that's the like the only thing we're trying to convince people of. It feels a little bit like when computers first became good at chess or at go or something like this where for a while it was only humans were very good and then there was a shift where the machines became a lot better and could start to do the things that only people could do. And I think it doesn't take people that long to recognize that the shift is happening or has happened. I think the point question which I don't have the answer to is what do you do with this information? When computers became better as a chess, that was fine. This was not an important part of society. The same is not true for security. And I think understanding what to do with this information is the hard piece. And I think it's starting to be the case that most security people who are paying attention are noticing this. I think people differ in their opinions of what the right points of action is to take. And no one knows what to do. And so this is entirely reasonable. I don't think people are missing that much though. I think maybe the only thing I would point at if people are missing things, which this includes me too, is looking at the exponential rate of increase in progress on these things, where if you plot how good these models are getting on essentially every metric, they're getting much better, much faster than we like to think. And I tend to-- With underestimate it. Yeah, every time. Every time. Like, when I sort of doing all of this conversation, what I am telling you is the things that you would do if progress was exactly as it was right now and had stalled. Because I don't know. It's hard for me to imagine a future where models got another 10 times better. That may get happened. I don't know. But I tend to only see what we're actually right in front of me. I feel like most security people are similar. It's easy for them to believe when I show you the machine that the machine exists. It's much harder to really feel the exponential and believe that it's possible that in a year, the models might be better than literally everyone in finding vulnerabilities that even after you try to explain to a person they don't understand. And when I say this, it doesn't feel like-- it feels like science fiction when I say this. And so I don't believe that this will happen. And yet, every other time I have said that I don't believe this will happen, it has then happened. And so I'm willing to have now wider margins of error on my beliefs on what may come true because I have been wrong. Every other time that I have said that, it's not going to happen. And I still feel like it's not going to happen. But now, at least today, I think it's not going to happen, as opposed to saying it's not going to happen. So I've learned a little bit. If this plays out the way you expect, what's the world we end up in? And what decisions today, like I was in this week, determine whether that world is going to be stable or chaos? I think the primary thing that Germans help us plays out is how good the models get. And whether or not they favor defenders or attackers more. And I feel like this is mostly a question of nature. And we can try and balance it one way or the other by making sure that when we put these models out for the first time, we bias them towards defensive capabilities. But either it's true or it's not that language models will be able to find and exploit vulnerabilities that humans can't understand. I may have been very confident in the answer was that this would not be possible. A couple of years ago, I now feel like it's probably not true. But if it turns out to be the case that these models do work and can do this kind of thing, then this is the important question. Sort of up to us to just make our way through. Bias it so that we favor the kinds of things that we want and we just favor the things that we don't. That we're very diligent with how we decide to apply these models that we do so in a way that feels responsible. But I don't feel like there's any particular action that this week we can take that will determine what happens here. We'll play out and we need to make sure that to the best that we can, we make it play out in a way that's favorable. But it's very hard to know ahead of time. Where any potential ceiling on capabilities is or on what kinds of bugs they will find or won't find. I'm optimistic that we'll do a reasonable job here. But it's yeah, it's going to become a lot of work to figure out how to do this well. All right, final question, which is what gives you hope? Yeah, I mean, we've done very well for ourselves so far. Security used to be in a really bad state. I haven't been to doing security forever. But what was it? Maybe 10, 15 years ago, I used to give talks where I was explaining security to normal people. I would have them call out a website and then live, I would demo an XSS attack on a website and anything that you wanted I could find. I can't do that anymore. Like security has like significantly improved over the last 15 years. It was worse before that. My hope is it will get better in the future. The trend has been so far that these things are trending to be more secure. And occasionally we get things where we have new attacks that we have to worry about. But like, directionally, I feel like we've software is in a more secure state than it was 5, 10, 30 years ago. I'm hopeful that that will continue over the next couple of years that these language models will be able to find and fix as many of these bugs as they can and things will work out great. And if that's the case, it will only be because people put in a lot of work to make it happen that way. It doesn't happen this way just by chance. And I'm optimistic that people will meet the moment and do the things that they need, which I don't even know what it is for their areas. But I'm sure that people who are working on this have a better understanding than me. But everyone will look at how things are going right now and to figure out a way to really do whatever what they were doing, but faster and more efficiently. And we'll end up in a world that's better. But I don't know. This is the best case scenario. And I can imagine places. I can imagine going ways that are not quite so nice. But yeah, that's what at least gives me hope. - Meglas, thank you so much. Really appreciate your time today. - Yeah, thanks for having me. - Thanks. (upbeat music)