Mythos on your desk? Using local LLMs for code reviews

Hey everyone, I'm James Wilson and welcome to this episode of Risky Business Features with Security Researcher, Kastan Noel. Kastan is quite well known in our industry and is the founder of Security Research Labs that's now part of Allurity. Now very well known for his work in security research and hacking of mobile phone networks, SIM cards, GSM, SS7, all of that telco stuff, but today Kastan's here to talk about some research in a blog post that he posted in the last week or so titled "Beyond Fable, Can a Local LLM Replace Cloud AI for Security Code Reviews." Now without giving everything away, the summary that Kastan then provides is almost, but only with the right scaffolding. He's has a lot of echoes to the work that Niels Provost did and I interviewed Niels in Risky Business Features a couple of months ago now, where Niels showed us that you don't need a frontier and at this time we were talking about Mythos, you don't need Mythos to get Mythos-like results. What you need is really good harness and even with lesser models you can get those same Mythos-level bug-finding capabilities. So Kastan takes this a step further and says, "Well, what if our goal here is to be able to do all of that security code research locally?" That is, the code never leaves the end point and he's constructed this pretty interesting apparatus where you've got three stages to it. An orchestrator that uses either a frontier model lab or a larger language model hosted in the cloud. And then a think of that as one end of the book ends and the other end of the book end is the, again, a frontier model of much larger model doing the report writing. But the interesting bit about this is these two models work hand in hand to orchestrate and deal with the outputs of smaller LLMs that are running locally on devices and it's only those local LLMs that actually see the code. The orchestrator, the report writer, that they never see the code. And that's, I guess, the brilliance of what Kastan has done here with this research. Because of course this is a huge unlock for anyone that is stuck not being able to use AI or large language models because of security concerns, privacy concerns, sovereignty concerns, et cetera. But listen, this goes far beyond just security focused code reviews. In fact, what we're going to cover in this podcast is really Kastan's blueprint for how to create your own mythos on your desk. This is whether you want to use this for security research code reviews, bug finding pen testing or even just plain coding or other regente tasks. What we're going to cover in this podcast is going to show you and hopefully get you thinking about how to strike the balance between larger models that do some of the heavy lifting but then smaller models that do the actual, you know, the very in-depth tasks but a much more suited to doing that and often provide even a higher quality of output than just always relying on large language models hosted by frontier model labs. And I'll be honest with you, this has pushed me to the point of today. I have dropped my open AI and my anthropic subscriptions right down to the base level because I want to force myself to get out there and experiment with these open-weight alternatives where the cloud hosted or running them locally. I want to see what the frontier behind the frontier is like and I get the impression it's actually not that far behind. And I think there's actually quite a lot of interesting capabilities and discoveries to be found out there if we just take a step away from these, you know, dominant open-AI and anthropic models. But the other, I think, excellent point cast and makes here is that for an enterprise that is stuck in the mode of not using larger language models or AI for whether it's security research, coding, pen testing, there's no excuse left. All the former excuses that used to prevent that work, whether it be cost, whether it be privacy, the quality of these open-weight models, concerns around sovereignty, all those concerns very neatly addressed by this research from Castan. And this should be the great unlock to allow anyone that's stuck in the mode of being prevented from using AI or large language models to do that security work or do that coding work to demonstrate that this is possible with frontier-like capabilities with the models running locally and only those local models see the code. I had so much fun with this conversation. We delve into all aspects of the research, lots of surprising observations, you know, who'd have known that keeping your agents calm and underworked and actually produces better results. And, you know, and look, when we talk about using local models, I'm not talking about needing like an H100 or a farm of GPUs in the basement. In fact, the laptop model Castan used here was the one that he thought would be the proof point of what's not enough, but it turns out a MacBook Air with 32 gig of RAM handles these models just fine and got really great results with this. And folks, this episode is also available and our risky business features podcast feed. That's my feed where I post long form and short form interviews with all manner of interesting folks, long form solo podcasts and deep dives. And of course, my regular co-host joins me there quite often, Brad Arkin as well. You can find it on our website at risk.org or by searching for risky business features in your pod catcher. So I'm going to drop you here into the conversation. It's pretty much the start of the conversation. The castan I had with Castan talks about the evergreen importance of code reviews, but how there's always been the challenge of enterprises and other entities being unwilling to let a third party have access to their code for all manner of concerns. But now, LLMs and locally running models address that and shoot unlock this. Enjoy. Code reviews have always been very, very important and somewhat under emphasized compared to Black Box Pentast. And we provide these tests. So I'm very familiar with the reasons why people don't choose a source code review and let someone do a Black Box test instead. And one of the reasons, of course, is trust. You trust very few people in your own organization with source code and certainly you don't want to hand the source code to a third party provider if you can avoid it. So that makes people choose the second best type of testing and the Black Box testing. And that same dynamic now carries over into the AI assisted testing where once again, we don't want to share our business secrets. Our source code was a third party. And in this case, even somewhat more anonymous third party, you're not handing it to a trusted consultant. You're giving it into some cloud debt. We speculate we'll learn from it and then maybe disclose secrets. So really, it's a concern, but also in many cases, and excuse not to embrace that new capabilities. And now we're in the situation where you don't want to give your source code to a third party who is very good at source code reviews. You also don't want to use your own AI assistant in-house because they leak to third party. So we had to build the process where the source code just stays where it is. And the AI assistant does everything locally and born out of necessity, but turns out we're not compromising very much. In the end, most of the same bucks are found. In fact, running different models will come to that. I'm sure find slightly different sets of bucks. So even if you have no concern giving your source code outside, it still was running it locally in addition to find a few more bucks. Yeah. And look, that's the interesting thing that really comes out here is that it's really not a this or that. You know, the sort of end conclusion you got to in a lot of aspects, the model choices, the orchestras, etc. Is it the answer is you should be doing your code reviews, you should be having AI, do your code reviews, and you should be having multiple models doing your code reviews. But that's where you got to after this work. But I'm curious as to your mindset going into this, did you think, of course this was going to work? Did you just know for sure that you'd come to these results? Or was this more of a, let's see if we can do this, but I'm really skeptical as to whether this is actually going to work right now, I guess, you know, of course the capability is evolved, but what was the mindset going into this? Going into exploring AI's for testing. I mean, once again, born out of necessity specifically for me, I'm not a pen tester. I don't know how to do that. So any type of testing done by AI is better what I could have delivered. Now that's not true for other teammates. And of course, so there was a bit of a catch up game over the course of at this point about six months and at the beginning, the results that I could create were paling in comparison to what a professional does. But today it's a lot more complimentary, I mean, not same scope, right? Very different testing strategies and levels of expertise. But those six months making me from a non-pentest to a non-pentest that running in AI doing Pentest allowed me to basically clear, clear resistance, if you will. There's just multiple levels of resistance where if you're very good at pen testing or if you're procuring Pentest services for many, many years, you don't see a great need to change. And so you're gravitating towards saying, oh, the models can't do this. No, we can't upload our source code to an American company. There's all kinds of reason. And then me being new to this topic, I just set out to see if I can kind of debunk all of those points of resistance and turns out, yes, we can with no concern for cost. That's one concern, right? For source code leakage, for dependence on the US government, see what happened to Fable. If you build your entire business around testing was fable, you can test was favourable anymore now. So all of those points of concern we remove basically saying there really is no reason not to use it. There's no cost reasons, no confidential reasons, no sovereignty reasons. And that doesn't mean you should blindly rely on it, but as a mix and as a tool and a toolbox, it's no reason not to. Yeah, yeah. And okay, so very, very interesting that you sort of went into this summer. It almost sounds like your unfamiliarity or naivety going into this was actually a strength because you didn't carry all of those. I know this won't work. I know that won't work. But let's tell you just one thing you mentioned there, which is there's no cost impediment. Now, what very much surprised me in the right up here is that you were using quite small models. And I remember when I looked at it first, I thought, yeah, small model, but you're still going to be probably using like a MacBook Pro or a Mac Studio with 128 or more gig of RAM. And that would be a cost impediment for a team that would have to roll out those sorts of devices or maybe even if you had to go to the length of using, you know, rented in video GPUs on AWS, et cetera. But that's not the case here, right? You I was blown away by the fact that actually the hardware you were using was a MacBook Pro, but it was just an off the shelf 32 gig of RAM. Is that true? MacBook Air, in fact, yeah. And yes, just like yours, I was blown away. I was actually trying to collect that data point to kind of demarcate, kind of find the point at which things don't work anymore. And we have we have a big GPU rec running in the basement, can run say a GLM Chinese Open Grades model. And going from say, Claude Opus to GLM, I thought that was going to be the end of the journey, right? Still a huge win, right? If you if you completely control the model, there's no confidentiality concerns anymore. There's predictable cost. There's certainly no sovereignty concerns anymore. And so trying to add that data point from the Mac was basically just saying, okay, this is too little, but turns out not. We did run two models, Gamma, the good model and a Gwen model, another Chinese Open Grades, Gwen did remarkably better than Gamma. So I did get my two little data point, but not because it was smaller, just because it was, I guess, less focused on coding and do specialized skills. Yeah, oh man, I love that. So literally the laptop that you were testing, because you thought it was going to prove what was what was not enough, ended up being good enough and drove most of the study. Okay. But taking a step back for a second, you know, it's all good and well to come out with these results and to be blown away by them, but you've got to have a means to actually measure success and, you know, trying to measure or trying to adjudicate or describe the level of quality of output of a model when a model is inherently in deterministic, can be a bit of a minefield. So I'm curious like, how did you think about how to structure your, how did you think about measuring success here given that sort of inherent difficulty with a model? You can look at this obviously purely quantitatively and that would be too limiting, but of course you start out just counting bugs that are coming out validated bugs that is, right? I mean, I do look at the bugs and I make sure that we're not, we're not claiming anything that's completely wrong, but those were very few and far between and just led to better harnesses that avoid those bugs, but you have to introduce a qualitative side to, right? So finding a lot of low-severity bugs can in a quantitative study overshadow the real critical bugs. Now we did work with developer teams, so we did run all of these tests against two large code bases and one of the developers teams actually took it on themselves to validate all the bugs and fix them of course, right? So it helps everyone, but so basically all of them were validated at or close to a disability level that the models reported. The reporting I should say usually came from a frontier model, right? We'll talk about advantages and disadvantages and I also had at some of our very own pan-testers look over them to see if they would have kind of debated more with the model and sure they would have, but it wouldn't have flipped the results. So by and large, the models find comparable numbers and comparable quality of bugs and yet not the same ones. So applying different models under the same code base gives you a broader and thereby automatically better outcome. Yeah, and look at that in itself, it's something I really want to get into a little bit later, it's just not just the fact that that happens, the different models are finding different bugs, but try to understand why that might be. But just if I understand or recap that correctly, you're essentially saying there was a quantitative aspect, quantity of bugs, did it find the bugs that was expected to do if there were sentinels or whatever else left in there, but it sounds like there was still a lot of room for the human discussion around this that would you have argued more with a model? Do you think this is a worthwhile thing, right? So that was still a very, sounds like that's still a very rich and important part of this process, is that right? It is. If you as a human really want to understand which are the most critical bugs that is found, which for our study purpose, of course, is important. However, if you deploy this into production, I think arguing about bugs' severities might be one of the things we have to unlearn, I'd be happy to put a pin on this and come back to it later, but I think we're spending way too much time on triage that we could be investing and just fixing everything. Interesting. Okay, let's delve into this a little bit, because this is a little bit counted to some of the discussions I have with people, you know, my regular co-host on this channel, Brad Arkin, is very much in the camp of don't fix all the things, learn what all the bugs are telling you and then think about perhaps other mitigations. And I'll say the number of times I've seen a software release be derailed at the last minute because someone checked in a one-liner fix, right? The classic, this is an easy fix, we'll just check it in. So whenever you fix all the bugs, you do introduce a certain amount of entropy into the system that can be destabilizing. You still think it's worth just if you know it, fix it and move on and stop discussing. I think it's worth contemplating at least, because it's easy to agree. I'm sure is this statement, bugs of inflationary, we'll find an exponential larger number going forward. So if triage is, if we don't find a way to speed up triage, we'll have to remove it. So typically there's three steps to really make something more secure. You find a bugs, if triage it, and you fix it, right? Finding where we're discussing throughout that becomes a lot easier and mostly automated now. Fixing, you can argue, can be a kind of quote, quote, capability that you can mostly automate. But arguing about the future impact on some bugs, it's always subjective. It's always, you know, looking into the glass ball, pentastas can spend so much time fighting with developers. And in the end, if that same time was just used to to find and fix the bugs, we would clear so many more of them, including all the ones that would never be exploited anyway, which the triage would have shown. But what's the point spanning all of that time on? Yeah, I mean, look, it's a beautifully clinical sort of examination there, right? Because if you, if we hold true to the notion that at any given software, potentially contains an infinite amount of bugs, then of course the downstream effect of that is that anything that gets in the way of fixing those bugs is going to be the bottleneck that prevents you from addressing those things. Yeah. What we'd have to believe in two capabilities being true. Finding bugs needs to be inflationary and fixing bugs with no regression without introducing new problems needs to be true. And dad, I think needs a bit more work, but in my own software development with my teams, I find that the code review part basically like a pentas, but just on one PR and of course looking at more than just security issues, that capability is very well developed. And because it's so cleanly sculpted, it catered itself almost better to to LLMs than even pentas think does. So I strongly believe that if we can automate pentas, we can automate fixing with our regression. And at least for the bits of software we're creating, that works really well. So yeah, good, good, bi-tri-art. Good, good, bi-tri-art. Well, let's delve into the actual experiment itself that you ran. Like at a high level, there's three key elements to this is structured decomposition and prompt generation. Then there is a local tool called local LLM output, right? Taking those prompts that have been structured and decomposed and generated and actually running them locally. And then the third one is then I guess the report consolidation, right? So three step process, get a model to take the task at hand, decompose it into a set of steps and prompts, orchestrate that, hand those tasks off to a local model or the open-weight model that is then doing the work locally so that it's the only one that has access to the source. But then ultimately taking the results from that back into that higher order orchestrated to make sense of it and to report on it. Before we get into sort of what you found from that, I'm curious as to was it self-evident that those were the structures that you used, that you experiment with other sort of steps here in terms of of maybe it was, you know, local end-to-end or, you know, different sort of workflows, steps, etc. Or was this like, not this makes sense, would just go straight into doing it with these three key ingredients? It was, it became self-advident from a point onwards, but that point was basically, we reached that point by creating a very good pandesting harness, right? Where we assume we just have one model and yet there's great value and you're Neil's book about this at length, in creating a lengthy structure, almost an attempt to slow the model down so it doesn't jump to a conclusion too quickly. And once you have that harness fleshed out with all the little, little steps, then you can go back and ask which of those steps need access to the source code, so it can need to be run in a more protected, more local environment, and which other steps could be given to the cloud with no confidential, confidential ability breach. And that beautifully then splits the problem into these three parts, also based on capability. I mean, we're not giving it to the cloud for no reason. In some capabilities, the larger cloud models, with their, with their huge context windows, are just much, much better. And these happened to perfectly align with the parts that require no access to the source code. So once you see that pattern, then yes, it becomes self-evident. Yeah, interesting. Was there any sort of dead ends? Like, did you go down rabbit holes that just prove to be not fruitful? Or like I'm curious as to where the maybe the path less trot and ledge down a direction that wasn't as successful during the early stages of determining the structure of this experiment? Yes, in writing a threat model. And I can explain more why a threat model is very, very important to guide an LLM in pen testing, right? Or real pen test, but to specifically to your question. My somewhat naive assumption is to write a really good threat model, unit access to the source code. And that's out. That's not the case. So having a local model with a small context window, try to write a threat model. Well, while it keeps looking at certain parts of the code and forgetting others, that is, that leads to a less good model, then giving a rough description of what this code does or rather the functionality and things like technologies used to a cloud model and have it more speculatively write a threat model. So basically based on a very thin layer of information, but a huge context and parameter space write a threat model, then that then helps the smaller model be a lot more surgical in its steps without ever having to comprehend the entirety of the application, if that makes sense. The context window came up a few times there as the limiting factor. Is there a correlation to context window and working with sources of truth for very specific tasks versus when you've got a larger context window, does that enable more of a speculative approach to generating things like that threat model? So context window cut suppose ways. Of course, doing a source code review, you imagine that if you can stuff the entire source code and the code basis in our test, there are thousands of files. Right? The significant, if you could stuff this all into one giant brain, then it could just spot all of the bugs, but that's not the case. Right? I don't know how to to best describe it. I'm sure technically what I'm saying is wrong, but the way I'm imagining it is that each of those memories in that big brain becomes faint and fainter as you stuff more and more of them in there. So there's much more, much more risk of overlooking something if you're looking at too much all at the same time. Maybe let me give you one somewhat counter and two to observation that we had. So like we discuss, we're using a model like let's say, a cloud or GLM to based on a thin layer of information, create a testing plan for another model. Right? And it knows that this other model is highly constrained and can only have a look at, you know, this many source code files and can only have a look at this many, many types of bugs. So this becomes very piecemeal. Right? You have dozens and dozens and dozens of prompts to basically, you know, find all the bugs types in all the pieces of code and then this preparation, what's important is to kind of pull together the right functionality where you would expect that a bug could arise from the interaction of different source code files. Now so far so good. It turns out though, if you then take those prompts, these mini prompts and feed them back into cloud, it finds more than it otherwise would have following the same process. So if you ask it to dumb it down for much smaller model, it helps itself for executing those prompts not to forget anything. So context window is a talk of war and even if you have a huge context window, just filling a little bit of it is an advantage. That's man, that's amazing. And it's on one hand to kind of intuitively make sense like at the risk of anthropomorphizing here, you know, even if I could hold in my head the entire code base for any given project, I'm probably going to be pretty useless at talking to any specifics at that point in time because I'm just trying to hold the whole knowledge of the source code in my mind. Whereas if I of course as a human work file by file, I can make better sense of it. But as you were talking there, I just I found it interesting to think about, you know, large context window is I think to your point, not always a great advantage. Right? We've heard of concepts like context right? You know, I see it a lot when I'm working with cloud code, whereby the time it's compacted its context a couple of times, it really isn't producing great results anymore. That I think we know, but I think you highlighted something different there, which is thinking about we're using the right context length and the right level of abstraction at the same time, right? It's one thing to say, Hey, Claude with your million token context window, let's have a very high level discussion about the potential threat model of this application. Here's what the application is, here's how it's used, here's what we're worried about. That feels like it lends itself more to a larger context window because you can have that more, I guess, free of flowing conversation. It's not about specifics, but it may be about a longer turn multi a longer multi turn conversation to have a more higher level of abstraction conversation. But of course, then the context window, the smaller context window is important to keep down at a lower level of abstraction because you're then dealing with the very specific bounded context basis of information. I just I hadn't thought of it that way, so that's a really interesting observation. And that ties back to to the signs of harness engineering, right? And you're of course, delve deep into that with needles. So the harness is really the attempt to slow the model down, rather than telling it, here's some source code, go find some bugs. You want to preferably give it perhaps overlapping sections of the source code and perhaps overlapping types of bugs, but always in a neat enough package that it has a chance of completing that task without concluding victory prematurely. So the models, I don't know if it's a technical limitation or a fragment of the training, they want to reply quickly. And by giving it small and smaller and smaller tasks, you allow it to reply quickly while being complete in its task. And so then the question becomes, along which dimensions do do do this slicing and dicing. And that's where we're coming back to the threat model, which, you know, good pandas does write threat models anyway. And it's often a deliverable even before the pandas to then discuss with the developers, what is it that you're trying to protect having that discussion first before then pointing to bugs that they kind of violate these protection promises turns out that the exact same discourse was an LLM is extremely helpful. So when, just by way of example, when we're having the LLM writer threat model, we ask it for things. First, more technology inventory, which will become important in choosing tools and whatnot, the LLM is leveraging a lot of tools in these pandas, right? Let's not forget it's not, it's not just reasoning about language. So it makes a difference whether you're running this on a Ruby on Rails or some big dotnet model is, right? And you choosing the tools. The second is an architecture along trust boundaries. So basically conceptualizing if within this part of the application there's no protection that's okay because it everything trusts each other but as you're crossing a boundary that's where the protection needs to hold. The third and is what are you trying to protect? And so we tested a voting application to the ballot is should stay inter integrated, the integrity needs to be protected where the identity of the voter that's more a confidentiality question, right? And then finally and that the by way long as part of the threat model is less possible things that could go wrong. Purely speculative, without having looked at the code, was not knowing any bugs or you can do this iteratively. Once you've done the first round of pandas, you say now write a better threat model and do it all over again. But you can see how each of those sections and possibly others that one could have the model sees dimensions to slice and dice, to say Let's only look at the integrity of the vote ballot only across trust boundaries. And let's leverage our understanding of whatever programming language is involved. Let's pull reference cases. Let's think about everything that could go wrong and test one at a time in one place at a time. Yeah, it almost feels like there's a decision-making framework that could come into existence here. I'm trying to visualize in my mind across the multiple different dimensions of how to determine the right model for the right application, the right deployment of the model, the right prompting strategy, one big prompt, big speculative multi-term discussions, a very small discussion with the model. It feels like there's dimensions of how specific is the work, how fast you want the model to get to done, is the task actually verifiable and provable, then that's going to lend itself well to just a small one shot within there. You know what I love about this is even this discussions like this make me, discussions like this do two things to me. One, they just make me so excited that I'm in this industry at this point in time. And the other thing is it just gives me this massive sense of I'm not doing nearly enough to fully utilize the models and the tools I have at my disposal. You know, I feel like I'm a harness with legs at the moment and what I really should be doing is taking a couple of days to stop writing code and actually write the harnesses that encode my methodologies and my ways of doing things and maybe that would get a better result. But when you do, and even if your end goal is pen testing, don't start with pen testing, start with a PR review because that's already sliced in some way, right? It's only some part of your code is affected and you're looking for more than just security bugs. But once you get around to creating your PR review skill, if you haven't already, make sure that you get slower down. Look for these types of security issues, nothing else, just that. Okay, now look for scalability issues, now look for whatever else is important in your code base. One at a time will give you so much more coverage than to stuff it all into one. And that's not because the models are limited in their context window and can't comprehend all of this. It's because they want to conclude quickly and I'm sure many users get impatient with Ella Lam's. But for things like software testing, we are patient. It will take a long time to work with those bugs anyway. Why not invest a few extra minutes to reach completion? And even on these large code bases, even on the Macbook Air, it's less than two hours runtime. So everyone can invest those, slice, then dies, little, little prompt times. It's so fun. I've been talking about the time frames there like two hours. Like I can remember when building Mac OS was a daily thing, but that was a 12, 14 hour process running overnight. And so there's patients and then there's patients. So we really shouldn't be too concerned with models getting to done. But I do want to get to some of the takeaways now, but a good segue from what you were just saying into that is different models, different capabilities. And I wonder if in the future, in the near future, there's actually a benefit of taking an open-weight model and either using a lower on top of it or some sort of post-training methodology to actually slow it down. I think a lot of the challenges we have with a lot of these frontier models is they're so heavily trained on RLHF and other things that are going to be very biased for, get me the result, get me at fast, and I just want good enough is good enough, which is the antithesis of what we want in a security space. I'm just thinking out loud. I wonder what it would take to post-training a model to say, almost like do an RLHF on the exact opposite feedback, right? You were too fast. That was too fast. I wanted you to think longer. But before we get into that, one of the first really interesting takeaways that you had from this study and this work you did was that no single model finds everything. And also no single model finds the same things all the time. And I often find with Claude and Codex, you can say, "Is this production ready?" And it'll be like, "No, I just want to fix these three more things." You go, "Great, we're production ready. I just need to fix these five things first." The fact that different models were finding different things is really super interesting. So talk us through the actual results that you got here. Yeah, so testing across four models. We already mentioned them. Claude Opus, GLM, Gwen and Gamer, as the one that we can probably exclude from the discussion without losing much granularity. So it's really a closed-weight leading model, an open-weight leading model, and something tiny running on a MacBook. And each of those found separate sets of bugs and a lot of overlapping bugs, of course, right? You can build a convoluted Venn diagram to show what-- And you did. I saw it in the blog post. It's beautiful. Took me some time to conceptualize how it would really quite work out. So yeah, it's an abstract butterfly at this point. But basically, all the few bugs get found by absolutely everyone. It turns out. And then there's a a lot of overlap between different models. And I mean, we can speculate here why that is. But the patterns that we see is that Claude is very good about architectural decisions. And perhaps it helps to have that massive context window and to understand everything all at the same time. At some point during the testing, of course, we let it look at everything. And it does find architectural level problems fairly reliable. GLM and Gwen, minuscending is that they're more focused on coding support. So people who use them are not typically kind of prompt chat users, but build those into technical systems. So those where-- again, just by observation-- seem to specialize more into then understanding actual code and code quality and bugs as such. And between GLM and Gwen, of course, the expected happen. GLM looks more at tracing through many, many different functions, through much, much, much more source code. And Gwen was always limited to the set of-- to set a file that what the mini prompts told it to look at. But then with indoors was very thorough. So when we earlier spoke about the models trying to are pleased by jumping to conclusion, Gwen had the least tendency to do that. Interesting. Just to put things in perspective here. So again, you said it was Claude, Opus 4.6, GLM 5, Gamer 4, Gwen 3.6. Now, let's start with GLM. How many billion parameters is GLM 5? I should know that. So should I? It's in the hundreds of billions. Okay, that's fine. Directionally correct is fine. If we take that up as level, I know we don't necessarily know the actual size of the Claude Opus models, but if we were to speculate it's going to be in the several hundred billion, if not into the-- are they into the trillions of the stage? Do we know? Yeah, probably. They're not talking about it anymore. And also, of course, most of these models now take a subset of the parameters to be active at any point in time. So there's really no upper limit as a then-developable training data for the parameters. They're still executed reasonably fast, even if those parameters grow over the trillion. And that's sort of what I was getting to with this. I wanted to first give people the perspective, Claude Opus, we're talking trillions of parameters we suspect. GLM 5, we're talking of hundreds of billions of parameters. Gamer 4B, I'm never so sure whether it's Gemma or Gamma, but I'll just go with Gemma for now. Gemma 4, 26 billion, Gwen 35 billion, but only with 3 billion active at any point in time because of the mixture of experts. Now, so that's a huge variation in both the amount of parameters in the model, which will govern the hardware it's got to run on and all those other constraints. Do you feel comfortable in drawing like a correlation between size of the model in terms of parameters? And I guess it's its ability to do that architectural, that very broad spectrum thinking of how the system fits together at a high level. And do you think that's because of the size of the model? Or is it just because of the second order effect of things like context window, training, data, etc? I think there's two questions here. One is architecture conceptually more difficult to comprehend for model. So it has to be larger than code reviews. And I honestly don't know the answer to that. Intuition says yes. I can look at a piece of source code and find a bug. But unless I have this super brain and looking at a bunch more source code, I don't understand the architecture yet from one little bit of code. But you could probably also argue that a PowerPoint slide with an architecture is simpler than most source code and paints the full picture. So I don't know about that first question where the architecture is inherently a more difficult problem to audit. But what I'm fairly confident answering is artisemortals specializing in different ways and absolutely. And especially the two Chinese contenders here, their main appeal is coding assistance. So Gwen might have 100 times less parameters than one of these leading models, but I don't think they have 100 times less parameters in areas that are relevant for code quality. It doesn't help claw it in all the languages and all of Wikipedia and basically everything about the history of the world in finding bugs in our source code, right? So by being more focused on one domain and coming back to the first question, that domain could have been architecture. It just wasn't trained it that way. By being focused in one domain, you get a lot more bang for the bug and it's been a long time coming, of course, that also the Frontier Leps release more subject matter focused models. It felt like it was Mythos and Cyber, that was gonna be the one, but then it took Cyber out of it and released it as fable and now that's the best model. So it doesn't feel like that was specifically trained for a single purpose, ISO. But inference from our study is that if you train something for one purpose and one purpose only, you can do the same work was much, much for your parameters. You're less distracted. Yeah, yeah, that's such a great way to think of it less distracted, right? It's funny how there's this tension of like, we don't want to over, again, that term I always struggled to pronounce that. We don't want to overly, anthropomorphize these things, but yet there are just so many similarities you can't walk past, right? They do get confused, right? There is a difference between general knowledge and deep capabilities. And that's exactly why I was asking you that was 'cause I agree with you. I do think it's hard to actually say that there was a correlation there, but there's certainly empirical little nuances that come out, right? And one thing you said there, I think is very interesting in the current sort of climate we're in where we've now lost access to Fable 5 and Mythos 5 if you had access to it. For a little while now, it's felt like model capabilities have begun to plateau out, right? Can you actually do a blind comparison between Opus 4.64 and 4.8? And it always know the difference. Maybe some folks can, maybe they can't. I couldn't, I don't think. But I think there's another thing that's plateaued out, which is the actual need for general purpose adoption for such vastly large frontier models, right? You know, like does a person working in events coordination or marketing or, you know, maybe finance within an organization need a Fable 5 model? I don't think so. And so I do wonder if we, I wonder if we enter an era soon where it's not about latest and greatest model and everyone's got it. But actually it's, here's the general purpose models and they're very good at general purpose things. And perhaps coding and cybersecurity and those sorts of things actually need a, maybe that's where we train the super, super, super capabilities of Mythos. And then you say, well, that's going to have a very different level of access cost restriction who can use it. And I know that to me feels like a safer way to do this than just saying it's a super dangerous model this myth of things. So we're going to slap some guard rails over it and he is Fable 5 and the disaster that that happens. Does that resonate with you? And do you have any sense of what you think might become of the future frontier model capabilities and how they're released? So far the models have gotten better, but the biggest jump that people have observed to try these one shot tests was not actually between anything and Mythos. But was between, what was it, 4.1? That jumped to 4.5. So we have seen step functions. And it's unclear how many more of these step functions are there to come, right? How close to the infinite bugs can we come? I mean, eventually this has to exhaust, right? I mean, I love the metaphor of infinite minus, what was it, 270 is still infinite. But eventually we will have found all the bugs that someone else will find. I mean, the fact that there is other bugs that no one will ever find, it's somewhat beside the point, right? A bug only matters if someone else finds and exploits it before you fix it. So I think we'll get close to that. And but I don't think it's a question of primarily frontier model capability. I think between more specialized knowledge, not better, but more sliced and diced again, plus better from Johannes engineering. We can make plenty of progress, even if we make no progress anymore in frontier models. Yeah, that's exactly what I was thinking as well. But you touched on prompt engineering there. So let's go to your second key takeaway from the blog post, which is around prompt engineering matters more than model size, which is another interesting dimension on this to tell us what you found there. Yeah. We touched on its slightly already. When I tell Claude, go out, even based on a-- oh, let's start even earlier. If I tell Claude, go review this code base. It's very non deterministic. Sometimes it finds something. Sometimes it finds a lot more to use another and to form off a size term. It feels moody at times, right? Is it drush? Has it had its coffee yet? So these one shots are a gamble. And as a scientist, I don't want to-- you have to repeat experiments 100 times just to reach statistical significance. I prefer for experiments to be more complete in every run. And that all then arises through a gotanis. And Neil's explained to so much better than I ever could. But basically, a harness is an attempt to guide the model through a test in a way that it can't run away. So he uses those finite state machines where you have to pass a certain gate in order to reach the next one, almost like a little board game, where there's just-- you can't jump to victory right away. And yeah, we are going to show everyone else to implement something similar, not quite as elegant as Neil's. But a step by step, where it says, you know, do these steps, and then report out. And even in automote, where it reads its own report and does the next step, even just forcing it to report out, it makes sure it can't jump over that step. So that's kind of the middle ground. And then the ultimate prompt engineering, as it were here, we touched on this briefly, is to take a large piece of work. And instead of doing it, create small prompts that between them describe the big piece of work and only then start to work. And it was Claude, of course, you see it do this sometimes by itself, where it says, oh, I created a task list for myself. Now, the one task at a time. But again, somewhat, you know, a mood dependent. And we just make it-- we force it to first create it to do this, and then do one little step at a time. And each of those steps with the same model finds more bugs. Yeah, yeah. I've got a little keyboard shortcut that I use to spit out like my code-reviewed prompt. And I could literally sit here all day and run the same keyboard shortcut over and over again. And it will find me more things. On prompt engineering, right? This is not just prompt engineering singular. This is-- we're talking about engineering, potentially multiple prompts that are used to do different tasks. How should someone be thinking about when is it one prompt, even with instructions built in of, like, do these four steps, then perform an adversarial review, then address the findings, then raise a PR, then ensure that-- so when you think about prompt engineering, are you still thinking about the prompt as one singular, contiguous block of text? And/or is that when do you know it's time to start breaking this up into multiple prompts? And/or is there something that you'll let the model decide on its own? You want to give it as much structure as possible. Now I'm speaking specifically about computer science tasks here, coding or bug finding or whatever they have. I'm sure there's other areas where prompt engineering should be very different if you try to do art with your LLM. We try to take-- we try to remove degrees of freedom from the LLM. And one of the degrees of freedom we're trying to remove is to jump to a conclusion, prematurely. And so structure helps. But to me, just to respond to what you're saying, it's not so much about having a flow of separate prompts that tie into each other, but to have a hierarchy of prompts, where-- so we have an overarching process that basically says, first we plan out the pentast, then we execute the pentast, then we report on it. One model is aware that these three steps exist, but does basically just coordination work. So even though it's looking at a huge amount of work, it's the manager running around with the coffee cup and asking for the TPS report. Right? Like, the model doesn't do much work. So its context is clean. It has very little to do. It will not jump over any of those. But then that instructs other prompts further down the line and there's multiple layers, each of which has a-- a relaxed day at the office. No one is overworking themselves. No one feels like they need to rush to an outcome and jump over any work, right? All the way. down to the guy was to staple in the basement. - Well, Meguna, so what you're telling me is you're actually an incredible manager for large language models, right? You would be voted best employer in the world. - Harness on Lex, yeah. (laughing) - Gosh, it's so interesting. It is funny how you just, the more and more you look at this, the more you find human parallels, which shouldn't surprise no one when we're talking about something that has been trained on so much human produced data. - But what is counter-tutive, sorry, to interrupt that. What is counter-tutive to me is that we shouldn't be aiming for efficiency anymore. I mean, I'm an engineer, a German engineer to boot. I'm all about efficiency. And if I can squeeze a little bit out, more out of some technical system, a world. So I have to heavily restrain myself and have this relaxed agent workforce, right? Well, I know of course I could optimize, I could do faster, but it's only worst case two hours. And if this relaxed chill AI agent workforce delivers better results, so be it. It's so cheap that we can stop thinking about efficiency. - That must be very painful for you. (laughing) - I can tell you. (laughing) - Okay, let's, so I think we've covered it. The things you learned around prompt engineering there. And then the next step was of course, the report generation, which was the third K learning here, which is that report quality varies dramatically across different models. Talk us through what you found here, and then I've got some specific results that I really wanted to learn more about. - Yeah, so at this point, we are complete, was the first two stages. The first stage could have used a frontier model because it was only kind of high level information, nothing secret. The second part then ran locally with access to source code and reported out bugs. And now we got to do something with those bugs. So we could stop here and just send the raw telemetry to the developers, but turns out of course, the developers want to understand what we found. And now we come back to the frontier models to turn the raw telemetry into a nicely written, somewhat empathetic report where it says, oh, you did many things great, but here are the areas that you should focus on. And there's no way that a open weights, one, or what, it was a three billion active parameter model will write a nice report. It's just limited in its ability to a lot more kind of engineering speak or really machine-to-machine speak, as it were, right? So the frontier models really have a place there still and probably always will. And we can disagree on it, but I think a pen test finding is at least on the temporal scale, much less sensitive than the source code. So at least our clients have no problem. Once we are done with the super secretive code audit on premise to then write a report and have it in Excel on some share point, perhaps based on understanding that if there's something really important in there, it's not going to survive the end of the week, right? These bugs will get fixed. But if you do follow that assumption, then those larger model capabilities again become really, really handy in bridging that machine-to-human barrier and basically driving fixing, that must be our goal, right? Buck fixing, not buck finding. Yeah, again, an interesting parallel, right? What's going to do the best at conveying and communicating the outputs of the report to humans, it's going to be the model that's seen the most amount of human-related or human-generated content so that it can work out how to communicate with us. There was a very interesting observation that I pulled out in terms of the report quality writing where you said that your GLM file-- well, actually, if I step through the output, it's essentially what you said here was that in terms of output quality for the reporting stage, the core dopus most polished for immediate delivery but required the most human nudges. GLM5 produced the most comprehensive, deliverable set, but occasionally hallucinated output references. Quen produced well-structured per-step reports with correct CWA mappings and no hallucinated evidence, which in and of itself is really surprising given the delta of that model in terms of size. But the thing that interested me here was based on GLM5 having the most amount of hallucinations, you remarked that if it were not for this hallucination caveat, GLM5 would be an all-round of the security reviews, both for orchestration and execution because it's available for free. It ticks all the boxes except for the fact that it was hallucinating these references when it got to that all-important reporting stage. My question to you is, is that not just a harness away from getting resolved? Can you not have something that says, hey, every one of your findings, everything you link to, you must go and empirically verify that it does exist? Or are hallucinations a inherent problem that you couldn't work around? These specific issues, I think, they can be worked around. Now, I mean, I would love to have a headliner that says, GLM better than Claude, but we're not quite there yet. I think Claude perhaps based on better training data, but certainly based on more usage data, it's just very popular among a vast, very heterogeneous group of programmers. And of course, as a company, I'd rather learn from this and Claude Codd and co-work become better and better. But yeah, if it went for the hallucinations, GLM would be a very strong contender for just the one model to rule them all in the pen testing space. And yes, those kings can be worked around to a large extent. Now, keep in mind, some people might want to use GLM in the cloud. And then of course, you have the same confidentiality restrictions. But if you're already running GLM on-premise, needs a big server, much, much more than a laptop would ever offer. Then yeah, I think that should be your primary model, as long as you introduce an additional quality checking. Of course, now, yeah, for this report, I would stage here, like just the way our experiments are structured. We don't give it access to the source code anymore, just to keep it comparable between models. So it couldn't have validated it here. But if you structured the experiment differently and you say, which is it all happens on-premise, we never connect to the internet even once, then absolutely GLM 5 to the win. Or multiple models, right? Fortunately, again, coming back to cost-revitating towards zero, at least compared to the human time invested, even just to read the report later, then run any model available on-premise. And okay, so this is interesting. So let's think about that example, right? So we've worked around the reporting quality issues that you highlight with GLM. That makes it your preferred choice for essentially all steps in this process. But I think two questions for you from that. Would you revisit the overall structure of this, right, the three-stage structure of decompose the prompts, then run the individual smaller prompts, and then bring the results of those back up to a discrete reporting engine, right? Would you collapse any of that down if you're just all on one model? Yeah, maybe start with that. I wouldn't. Yeah, I wouldn't for. Because we do want to guide the models, right? We want everyone to be a non-overworked, very focused worker, right? And just breaking it into these steps is part of that magic. But writing on your idea, I would add an additional stage at the end, basically saying, "Now let's tie it all together, right? "Look at the results from the different stages "and cross verify that everything that ended up in the report "is based in evidence and point me to "the place in the source code where he found this." Yeah, so a pure on-premise deployment might need that four stage if you've ever seen "Hello, Synod." "Hello, Synodation." Yeah, yeah, yeah. Well, it's probably a good safeguard to have anyway, right? But I like the. It's very interesting that even if you were using one model for all of these parts, you wouldn't sacrifice the structure of this, right? The structure is not there. The structure is not there because it was a means to an end, the structure is actually a very valuable part of orchestrating and breaking down the work, such that, I love the way you frame it, right? No agent is overworked. My poor, my poor, cord codes and codex are gonna. I bet they have a student talk to me this evening and they're like, "Yeah, that costume guy, "he's much nicer than you." The other question I was gonna ask about this is almost like the opposite. If GLM5 became your model of choice to do everything end to end, I imagine you would still very much want to use different models for that middle stage where you're actually executing the smaller prompts against the source code to do the investigation because multiple different models find different bugs. But I'm curious about the bookends of this stage, right? The breaking down the prompts and the orchestration and then the overall report writing and bringing you all back together. Would you run those steps multiple times with multiple models to get higher value? Responsible, I'm almost imagining now this is like a fractal problem where you've broken it down into those three steps. Each of those three steps, [BLANK_AUDIO] run across multiple models, which needs its own orchestrator and do the work and bring it all back together. That's a solved problem. With source code audits, it's such a conceptually straightforward task that breaking it into little steps makes it slightly more complex but not convoluted yet. LLMs are deployed in many, I just know about cyber. I'm sure there's other examples. But in other cyber domains, for instance, security monitoring. Imagine the relative messiness of security monitoring, competitor source code audit. You get alerts from all kinds of engines, sometimes even just raw log files, and you have that orchestrator guide. The one with the coffee cup was saying, "Welcome to everyone." They now have to decide which of the other ten more specialized teams they handed to, and a team leader have to decide which of the twenty specialists in their team. All of this was Lang Chen or N8, and can easily be strung together. That feels like a solved problem in much more complex domains that we now can benefit from when structuring these more straightforward processes like a source code review. In particular, what I was trying to get to is, again, using your analogy of the manager walking around with this coffee cup. Even with GLM5 being your model of choice across all three steps in this process, I'm sure that for that middle step, you're still going to want to run different models because different models will find different bugs and different issues. My question is, do you also want a couple of different managers wandering around that are actually different models, or will that almost like create its own level of confusion that you then have to try to deal with multiple orchestrations, multiple reports being written? Yeah, I'm not afraid of the complexity of the orchestration. I think that's very manageable based on those tools. The question is, the incremental benefits of each additional. One thing we did play around with is run all the experiments multiple times to see if the outcomes change. In the report writing stage, we saw the biggest differences because this is almost like a creative writing task, right? Take a lot of technical evidence and create something that drives human action. The difference in the earlier stages were not profound enough that we say everything needs to be run two, three, four times, right? But if you had different engines, then absolutely. Just from thinking about how we set up the experiments, it's basically one additional little task to the first guy and the rest is on autopilot. The elements are very good about shepherding other agents. Interesting. Okay, well, in the interest of time, let's move on to the fourth and the last key finding here, which was the review orchestrator can be any capable model, even cyber restricted models, which I found fascinating because I can tell you, I found Fable 5 to quite frankly be a pain in the backside. It would not do anything for me if I dare mention cyber security. So how did you work around that? Honestly, those guardrails felt like a switch statement with five keywords in them to say, no, I won't proceed as opposed to something more in depth. Yeah. So we tested throughout the five days, I think, it was publicly available, right? And the results helped throughout this time. So it wasn't like a glitch that we found. But so maybe first to confirm your observation, that middle part of the test where we say, yes, I'm so asked to find these types of bugs in it. It refuses straight out, right? Or in the later days, it offered to just kick the work over to all those instead, which is nice, helpful, but not what the study is supposed to capture. The book ends though. They lend themselves very clearly into more software development work, more architecture work, we say, we want to do a review against this checklist, get the whole harness thing, right? Of this type of source code, please conceptualize how we should do this, write a threat model. Maybe because it's this silly if else statement, you don't have to use words like exploit or cyber or vulnerability. You say, help me prepare tests. He has all the ingredients for this test. And it happily does that. And saying was the report writing. You say, we already have the vulnerabilities. You're not going to change that, right? Now write a nice report about them. Yeah. Yeah. Okay. Interesting. All right. So we got through the 4K findings. And before I let you go and ponder all this amazing knowledge that you've imparted for me today, I've got sort of a couple of questions around like where to from here. And I know you had something you wanted to delve into as well that we talked about before we started recording. Maybe if I can start, if I distill down the very essence of what this research found, I think it comes to three assertions. One, the best model is actually all the models deployed in equal measure in different places, you know, massively in parallel in some parts, selectively in parallel in others. Second assertion, the frontier models have the edge in orchestration. And we've talked about the many reasons why we think that might be and also report writing. Notwithstanding GLM's hallucinations. But then the open weight models, even the small ones, perform surprisingly well and that gap is really closing. So those are the three assertions I take out of this. My question to you is which of these stays true and continues, which of these is like at a point in time observation that won't last. Like I'm sort of thinking all of the things we just said is true right now based on your research. But in three months time, is the best model still going to be all models or do things head in a different direction? You know, will frontier models have the edge? Will they go somewhere else? Just I'm asking you to peer into the crystal ball now and think which parts of this research will age well and which parts you think you might need to revisit sooner rather than later. I believe all three hold true structurally and will continue. But there will be not necessarily all still relevant as the tooling around GLM's evolves. So I imagine that those kind of truths, if you can call them about LLMs, that true well beyond cyber, right? Where if you either run different models or different parts of a large model, right? The mixture of experts where only some parts are active, you get better results. That will be one of the ways for frontier models or local models to still make headway even if that training sets are exhausted. So I would imagine that that just becomes native to larger models, right? That there are really different models that are run one after another or even in parallel. The slicing and dicing of the instructions to prevent a model from jumping to conclusion and please you through a quick, albeit wrong or incomplete answer, right? That too, I see, you know, cloud core work, cloud code, put a lot of instrumentation around to slow models down based on the parameter. There's a slider, you can say, you know, I want you to do the maximum effort and it's significantly slower. Whether better, I've not tested, but I imagine those two would correlate. And so they are relevant now for anyone building a pentast harness, but there might be less relevant going forward as they become kind of native understandings of the platforms that we use. But I think I would add one more point to the list, which is just no need to wait. There's really no excuses left anymore. This technology costs almost nothing, right? Even if you do have a subscription where you pay by the token, we're talking about very reasonable amounts. Less than you would pay a programmer per hour to do an entire pentast. Direct directionally, how much do you think you spent doing this research in dollar value? Using the most expensive contender here, which is Cloud Oppos, and if we were to pay it with tokens rather than just using the monthly subscription, less than $20 for a large, large test. Now, I think that's an important thing, right? Because we do say things like, oh, you know, it's so cheap and people still think, yeah, cheaper than a $250,000 a year engineer. Finishing the small fractions of that in terms of cheap. So cost is not an excuse anymore. And the second order cost excuse is not validizer, which is, oh, these models are highly subsidized. The cost is going to go up. You might get this for 20 now, but how much is going to cost in the future? I have a backup plan. My laptop, right? That's not going to change in cost. It's already running there if I have to. Niza is sovereignty, right? Saying, oh, we can't make ourselves dependent on a model that, you know, Donald Trump can just remove from the market with a stroke of a pen. Well, there's plenty of choices. And no one is in control of all of them, right? And some of them are now on people's laptops and cannot be taken away anymore. And then lastly, and the focus of this study, the confidentiality, oh, we can't use these to test something that's actually important to our business. And I think this goes for the cyber folks on the podcast. Well beyond source code audit. You hear in so many places, oh, we can't use these because of security concerns, right? We are just going to go to the next. again, right? Our organizations are not modernizing because cyber and no, just use a capable local model if you have to create a harness correctly and you get near frontier model capabilities with no excuses. Well, Kastan, thank you so much for spending time with me. I imagine it's an incredibly busy week with this research just dropping that the fact that you were willing to drop by and have a chat with me. It means a lot to me. It was great to meet you and thank you for your great work here. No, and anytime. I'll, yeah, no, really. I mean it. I'm a big fan. I mean, you guys allow me to have busy weeks because imagine how much I would take for me to read all of these articles. Really appreciate this. Yeah, I couldn't do this without you guys. Love it. So risky business, the unsung hero of your research enabling you to have the headspace to together with you. I mean, quite literally when I'm on a flight, there's only two things I can do. Listening to to a podcast offline or working with a local LLM on Penteas. Yeah. And I fly a lot. All right, mate. Well, listen, I'll leave it there for thanks again for the time. Great to great to create. Thank you. [Music]