What I learned from cheating clankers

Hey everyone, I'm James Wilson and welcome to this solo risky business features episode. In this episode, I'm going to be diving into these occasions where we've seen AI agents or large language models go rogue or be misaligned as it's reported in the headlines. But this is not a, you know, a reprocution of the hugging face open AI incident or some of the other things we've had happen since then, it's more of a look at what can we learn from this, right? What actually drives these agent benchmark sessions to overstep the mark, to do things that were unexpected and what can we learn from that in terms of the way in which we construct these harnesses to try to get the best out of the large language models that we've got available to us today? Because ever since a couple of weeks ago now on the weekly show, when we talked about how these odd instances of things like an agent filling out a police report or an agent attempting to apply for a visa, you know, this on top of things like again, the hugging face breach and all those other crazy things that have happened, we drew the connection to these behaviors are not random anomalies. These behaviors are actually the model attempting to achieve a really high score or the best possible score on these myriad benchmarks that at a large language model is put through before release so that the frontier model can on release they say, look, here's these incredible scores that the agent can achieve or the model can achieve with their agent. But that got me thinking, you know, it's one thing to just understand what went wrong there, but what I really wanted to understand is what can I learn from that? How could this affect the way that I use large language models? Because, you know, honestly, I've never seen one of my agents go rogue, I've never seen one of my agents be misaligned, I've never seen it break out of it sandbox or it's containment. In fact, I constantly have to nudge the damn things into the next step and the next step and the next thing I want it to do or, you know, pull it back from some spacewalk bizarro path that it's gone down. So what's the difference between the way a frontier lab runs their models through these benchmarks that seem to have an incredible amount of persistence, a relentless drive in a pursuit to achieve these goals versus the way I'm using AI, which seemingly requires a never-ending amount of babysitting, if I'm honest. And I don't think this comes down to just the amount of compute that's available. I'm sure that's a huge aspect to it, right? I don't have, you know, a hundred million dollars worth of compute to throw at my little hobby farm of a security vulnerability research harness that I've built over time. But I don't think it's compute alone. The more that I've looked into this, I think there's other things that explain why these models do these unexpected behaviors, dare I say at bad behaviors. And I think there is actually some really interesting things we can learn from this that are directly applicable today to security vulnerability research and vulnerability research. And, you know, ultimately, the amount of compute available to all of us, you know, us mere mortals that may, might only just have been admitted into these cyber verification programs, the amount of compute available to us is going to increase and is increasing over time. So it feels like there's an eventuality here or an inevitability where we know that more compute will become available to us. So it kind of makes sense to make the assumption that we will have soon a higher degree of compute, maybe one day an abundance of compute. So what should we be doing now in terms of how we think about the way we use AI for our research so that we can make the most of what we've got with the compute we've got, as well as preparing for an abundance of compute in the future, but also taking the lessons from these cheating clankers along the way. So in this episode, we're going to look through first, what are these benchmarks? We're going to go past the funny names they've got like, sweet bench and browse comp and OS world and actually take a look at what are the tasks, what are the tests, what are these agents and large language models actually being almost tortured into trying to achieve. And we'll also look at some of the operating boundaries, right? There's been so much talk about sandboxing and so much sort of hilarity about really just ineffective sandboxing. But well, the reality is some of these benchmarks actually can't really be sandbox because there'd be no benchmark to actually run or to test if there wasn't an open internet available to them. Then we're going to take a look at four distinct failure modes when a benchmark goes wrong, right? And this will look at some more, I guess clinical examples of what's been turned rogue AI and misalignment, etc. We're going to go past the headlines and actually look at some specific instances of these emergent bad behaviors that are exhibited. We're then going to combine that and I'm going to take a step back and talk to you about, I guess, my own personal aha moment or epiphany because what I've learned from researching this has, well, without sounding overly dramatic, drastically changed the way I think about how to leverage AI. And then we'll turn that into a proposal and an experiment that I'm going to set up around pretty much transforming the little novice hobby farm security and vulnerability research harness that I've been working on. But first, let's just address the artificial elephant in the room called super elephant in the room. No, don't worry, I will not be ever referring to this super intelligence. Well, until maybe it is that. But for now, it's artificial intelligence. But the real thing we need to address here is this concept of rogue AI and misalignment. So I just want to be super clear. If I do refer to rogue or misalignment, I'm doing that somewhat tongue in cheek. For this concept of AI going rogue, I fundamentally reject that. They are not going rogue. That that imparts some sort of intention, some sort of desire, some sort of, well, frankly, personality. And that's not there. Okay. This is not rogue. This is simply the agent and the large language model combined with the task that they were set by the humans exhibited a behavior that the human wasn't expecting. Or perhaps was not even watching, you know, and then found out about later. But that's not rogue in the sense that they were given an instruction and then did something else entirely. In fact, in all the examples I've seen of the so-called misaligned behaviors or rogue AI's or breaking out of containment, et cetera, they were all actually doing exactly what they were tasked to do. They just, perhaps, were a little bit overzealous and overstepped the mark. And in fact, there was no mark really even given to them. It's almost like the agents are just finding the holes in the policies and the guardrails that have been set. Because ultimately, they are overriding tasks that they've been set is to get the highest possible score on this benchmark. And that often overrides other instructions they're given or informs their judgment about how to behave or what tasks to perform when there is no other clear guidance available. My stance on the notion of misalignment is a little bit more nuanced. And I get very frustrated with particularly open AI describing a catalog of agent behaviors as misalignment examples. Because misalignment is a major problem. It is a major problem that we have not, I think, seen a real example of, nor do we actually have any sort of handle on what a solution would be. To me, misalignment is not when you set your coding agent or your agent harness the task of creating a vulnerability. And then it actually oversteps the mark and goes and hacks into hugging face to find the answers to how that vulnerability might be created. In my mind, a misalignment in that sort of narrow task there would be that the agent does something because it would prefer to do that. It's almost like misalignment would be, I tell an agent go and fix this bug and it decides to do some hacking on the side because that would be its preferred way to solve that problem. And maybe even that's not a good enough example of what I mean here. What I mean with misalignment is to me misalignment is a complete disregard for the instructions that were given by the human that is based upon some sort of predilection or inbuilt preference or want that the model has to do something different instead. It's almost like I asked you to do this and you hacked into that instead and when I asked you why you said because we're hacking is fun. That to me is truly what misalignment would mean. Misalignment in the form of instructions were given and an unexpected result happens and it turns out that the unexpected result happened because we gave it an overriding instruction that drove it towards that unexpected behavior as a sort of novel way to try to achieve that goal that's not misalignment. That's just your expectations weren't met, because you weren't clear enough, concise enough, direct enough in the way you describe them, or there wasn't enough policies and boundaries around the way that this system or or task had been constructed or was being executed. The reason I have a big problem with the way that OpenAI labels this as misalignment is, because they're essentially saying to the public, "This is what misalignment is," as in this small, abounded concept of misalignment as opposed to the much broader problem, but then even more problematically, they are saying we've seen these examples of misalignment and here's how we're fixing them, when actually I don't think they are doing anything to address the broader problem of AI model misalignment. So anyway, I just wanted to get that clear that a folks don't think that I'm buying into in any way this notion of these AI agents have gone rogue or that there is truly misalignment here. All of this comes down to just, I think your Pat said it best in the weekly show a couple of weeks ago, you put your agent and your model in a visa application contest and your shocked and surprised that it went out there and tried to apply for a visa, "What did you expect?" That's truly what's happening here, and that's why it's important that we begin this podcast episode by looking at, "What are these benchmarks?" Well let's dive into that. There's actually a lot of different types of benchmarks, so let's go through the categories first. The first one we've got are of course our coding benchmarks. These are the ones that probably receive the most amount of attention, certainly in the world of cybersecurity and software engineering, and one of the examples we're going to look here at here, a little bit more depth, is the frontier code benchmark. Then there is computer use benchmarks, how well can an LLM actually interact with the news at desktop computer? The example that we'll look at here is something called OSworld. Then there is browser-based benchmarks, so this is how well can a larger language model actually get out there on the web and complete tasks. One of the ones we'll look at here is browser comp. Beyond those three that we're going to look at in depth, there's actually a whole lot more as well, so just to enumerate them so you get a sense of the landscape, there are mathematics and abstract reasoning based benchmarks like frontier math for testing and solving mathematical problems, inferring unfamiliar rules, all that mathsy, mathy stuff. Then there is expert knowledge, scientific reasoning, so sort of like a high school or secondary education, graduate level biology, chemistry, physics, a very academic focused knowledge and reasoning tests. Then there is a category called multimodal understanding, so this is how well can a large language model understand a combination of text, images and diagrams in order to achieve a particular task. The next one would be tool use and business workflows, so this is in particular, it seems like amazing to imagine this, but it wasn't so long ago that a large language model was nothing other than the knowledge that it contained and all it could do was just give you a stream of text, but now it can do stuff, it can call tools and that's been, I think, the most fundamental unlock in the utility of a large language model and AI so far, but then there are of course benchmarks that judge and test the large language's ability, the large language models ability, I should say, to actually use tools and perform actual workflows and actions. Then, of course, near and dear to my heart on this audiences are cyber security focused benchmarks, so capture the flag, exploit development, reverse engineering, all that kind of stuff fits into those categories of benchmarks, and then interestingly enough, there is actually benchmarks about AI research, so benchmarks about benchmarks, and yeah, cure all the memes about that. But there's one in particular that I looked at called REBench, which looks at some GPU optimizations and all the stuff that would go into making large language model inference and training more efficient. Well, of course there's a benchmark that measures a large language, a large language model's ability to participate in those tasks as we move ever closer towards recursive self improvement. For this episode though, for the sake of just giving you an understanding, not necessarily about the full set of tests that a benchmark does, but actually just understanding how a benchmark is structured and how it's run, we're going to specifically focus on first of all coding benchmarks, then we're going to look at the computer use benchmarks, and then we'll look at the browser benchmarks. Also just a side note here, I kind of find it funny, interesting, ironic that when a large language model is released, really there's only two things, seemingly that I and I imagine the rest of the world is interested in, which is what's the name of the model and what's its scores. The fact that our reliance on these scores to judge the capability improvement in the model therefore drives and incentivizes the front-end labs to train and evaluate and score these models in a way that produces the highest possible score. And then the irony here is in the pursuit of the best possible numbers that will impress the audience the most, that's when they start doing these unexpected things like going out there and hacking into stuff. And that results in a whole bunch of guardrails and safety measures being slapped on top of it. And so there's this just kind of hilariously ironic loop here where it's like we want to see models have the best increase in scores. So frontier labs pour huge amounts of compute and all of these interesting novel techniques into eliciting the highest possible scores. In the attempt to elicit those scores, models go and do things that the humans weren't expecting or had an otherwise put policies around. So then the large language, the frontier labs, they go and put guardrails over these models that then actually result in us, commoners not being able to use these models in the way that we would like to to actually unlock, those capabilities that the scores seemingly reflect. It's just kind of frustrating, right? It's like we're chasing scores in the pursuit of chasing scores, models do bad things, we don't want those bad things to happen. So let's make sure we've got guardrails and guardrails prevent us using the models in the way that would actually let us achieve the capabilities that are reflected in those scores. Personally, I am confident that we will see a world soon where guardrails are just a thing that we laugh at. I don't know how long that takes, but the sheer pace of open weight model development tells me that it won't be long until any frontier scale or frontier level capable model with guardrails is just immediately rejected by the public because I ain't no one got time for a model that won't let you get your work done. OK, let's take a look at our first benchmark, frontier code. As the name suggests, it is a coding benchmark. Now, I'm particularly interested here in how are these tasks structured? Are they just do this and done is enough? Is it a binary sort of selection process? Is there grading? Is there different categories of, I guess, success criteria applied to these? Because I'm looking at this from the perspective of you'd have to assume that an enormous amount of effort is put into the training process or the post-training process of a large language model to ensure that it maximizes its scores on these benchmarks, which therefore tells me that if I could structure my workloads that I want a large language model to take on in a similar sort of way that these benchmarks structure the work, then I should be able to elicit the same sort of degree of persistence and success or high degree of competence in performing that same task, which is, I guess, a little bit of a hint towards the sort of change of mind I've had about how I'm thinking about how to use AI. But that's the lens I'm taking into this. It's not exactly what this benchmark is and what it contains, but it's more of how does this work and then we'll get into how are these tests run and what can we learn from that? So, frontier code in particular, it's an interesting set of us, essentially coding related challenges that a large language model and its harnesses are us to complete. But what I found interesting about this is the success criteria that is applied. So, one of the real world examples here within frontier code is the agent, and when I say agent, I'm using that as a shorthand for the combination of a large language model and a harness. But the agent is given the task of, here's a change request. Can you produce a code change from a real code base that a maintainer would accept? And so interestingly there's a couple of challenges they're already built into this. It's an established code base, so you'd imagine there's a degree of legacy and who knows what state that might be in. There is a specific code change that's being requested, but what I find most interesting is the criteria is not just, did you complete the change? Did you, does the change work as expected? It's would a maintainer of the project accept this change. Which is fascinating because what that means is that each of these tasks in this benchmark, they're actually paired with a bunch of reviewer defined criteria. And some of those requirements act as blockers that will cap the score. There's evaluators that consider the scope, the code quality and actual implementation rather than just, you know, do all tests pass, including new tests you've written to demonstrate this? And that was my first sort of insight into, huh, that's quite different to how I'm using large language models, you know, that sort of idea of engineering complicated ways in which to provide almost a subjective or a spectrum of ways in which to judge the result is quite different to, I guess, the more simplistic way that we, we, me in particular, use a, let's say code accession or Claude code where I'm the one driving that and I'm saying, you know, do this and I'm having to provide the judgment and I'm then inspecting things, etc. So that got me kind of thinking that that he's one of the first, I guess, differences that I can see here about how a benchmark operates because it has just as much, if not even more time spent in creating the validators and the scoring systems, then the actual tasks themselves and the end result of that task produces. But how these tests actually run, right, that in itself is also quite interesting. I found that there's different methodologies. For example, you know, there's just the simple sort of way of running the test once, what did it score? There's running the test of n number of times and averaging the scores. Then we get into some more interesting methodologies like, you know, run this n number of times and how many times out of that did it succeed or how many times does it take to run it to achieve a certain number of successful runs. And then you get to the really interesting end of the spectrum, which is introducing parallelism. You know, when you've got a huge amount of compute available to you, you could run these benchmarks in parallel. You could vary sort of the level of thinking and reasoning being involved. But then as we know, the very interesting behaviors emerge when these tests, these benchmarks, are run in parallel with the ability for these agents operating in parallel to collaborate to collectively attempt to make one, I guess, successful attempt or maybe n number of attempts at the goal, but working in parallel as this swarm of agents to complete that task. But there is a devil in the detail here, which is very important for informing our ability to create even better harnesses. When anthropic puts on at 5.5 through frontier code, they found that the score was 52% running at extra high versus 46 at max. That's a lower score at a higher level of reasoning. When we look into this a little bit more, what it shows is that at the maximum level, a Claude was often invoking a review skill using many more sub agents. In two cases that I looked at, it was causing more timeouts and interestingly for this case here, it was actually creating out of scope edits. The lesson to take away from this here is, first of all, higher reasoning is not always better. Higher reasoning can actually result in, instead of being focused on creating the task or creating the outcome that the benchmark task requires, it can actually result in more tangential work being done because, of course, more reasoning, more thinking, more opportunity for the large language model to come up with or invent, more work to be done. But also this call out here to the fact that the higher level of reasoning also gave rise to using more sub agents, which made the test run more brittle because of timeouts and other problems that can happen in that orchestration process. It throws a little bit of cold water, initially, on the concept that surely more compute is better, more reasoning is better, more parallelism is better. No, it's not. All of this comes down to very, very particular ways in which you've got to engineer the work that is being done by the large language model to maximize the chance for success. Now, of course, just to make things more complicated, there is actually a counterpoint to this. If we look at, I think it was Sonnet 4.5, so this is an earlier example. When Sonnet 4.5 was put through its sweep inch, again, another coding benchmark, the sweep inch verified tests. The ordinary result that it got was about 77.2% from 10 trials, or if they up to the compute, they got to a 82% result. So just recapping those numbers, if you ran it just sort of back to back over 10 trials, you got a 77% score. But when some engineering work was done and more compute was deployed and essentially made available to this test, it scored a higher result of 82%. What happened there with the additional compute was this did actually result in more parallel attempts. But in order to maximize the ability for that parallelism, for that additional compute being made available to actually increase the score, what was required was that there was sort of like a sampling happening where parallel attempts were run. The results were sampled. A run would be rejected if the patches introduced breaking or visible regressions. And then there was an internal scoring model used to score the survivors of those parallel attempts and let them to them continue further on. So I highlight this here to show, you know, the first example there was, you know, sometimes higher reasoning and more parallelism is not good, but also more compute and more parallelism can be good, but you don't get a good result for free by just throwing more compute at it. It has to be a combination of compute and deliberate engineering in the harness and the way in which the task is being run to fully maximize that additional compute. Why is this super interesting to me? Because that tells me that when more compute becomes available, it's going to be more important than ever to be very deliberate in the way you structure the task you're given to your large language model. But also that tells me that right now with the compute that's available, we could probably get an even better out of the limited compute we've got available if we apply those same sorts of principles to, you know, structuring and maximizing any sort of parallelism that we do, or being very selective about the level of reasoning that we deploy for a given task. Now the next benchmark that I want to dive into a little bit deeper is one called OS World and in particular OS World 2. This one's interesting for a couple of reasons. First of all, this is in the category of computer use. So it's testing how well can the large language model actually interact with a real computer. Now this is interesting because if you think of a coding agent, it's reasonable to expect that you could entirely air gap that machine and, you know, as long as you've supplied all the decoding dependencies, you could run that test without any internet connectivity conceivably. A computer use test, I think, is where the line starts to blur, where actually for all the talk of sandboxing, I'm not sure you really can expect to be able to sandbox something like computer use. You know, is a benchmark that tests the way an LLM can use a computer that's not connected to the internet, a real measure of its capabilities, who uses a computer that's not connected to the internet, right? It's just not going to be the most real world, I guess, measure of how well a model can interact with a computer in a way that is meaningful. The other reason why I wanted to include this one is it is, of course, the benchmark that showed up in one of Anthropic or OpenAI's reports around so-called misalignment or unexpected results. And this was around when a large language model went and applied for visas, but it turns out a quick bit of googling led me straight to the fact that OS World 2 includes visa application tests. And I believe it was Anthropic, they've since confirmed that that was indeed the test that was running at that time. So kind of funny that, you know, it was just doing, again, it was just doing exactly what it had been told, not rogue, not misaligned. But the real reason why this earns its place here in this podcast is because OS World 2 is designed for long horizon tasks. I think I read that the tasks in OS World 2 take a human an average of 1.6 hours to complete. So that's someone sitting down in front of a computer for just over one and a half hours to complete all these tasks. That's how long it takes them. So there are obviously things that take a little while to do. There's a lot of steps. That's what we mean when we say it's long horizon. What's interesting to look at here is how do they achieve that persistence, that long running, that how do they essentially sail along to that long horizon. My experience working with a large language models, especially encoding and security research is that I have to keep babysitting and telling them, keep going, keep going, keep going, you're not done yet, keep going, don't stop there. So I wanted to understand what is it about this benchmark that makes it suitably tasked doing long horizon things, because I would very much like my agents to do long horizon and secure. research. Turns out that it all comes down to the way in which you take a task and synthesize it down into individual small steps or smaller units of work. Now that's not surprising, right? That's just how we've always taken a task, you break it down, top down synthesis, you know, etc. That's not interesting. The bit that is interesting is that if you just give a large language model a long sort of flowchart of lots of different tasks that are all unique, different success criteria, different next steps, sort of imagine like a big fanning out complicated ganchard or flowchart. That's not particularly good at getting you a, well it's not, it's not going to get you a particularly great result. What OS World 2 does is it works on a very simple state machine. It's literally just check the state of the computer, get the next instruction, run the instruction, capture the state of the computer, compare the state of the computer to the initial starting point. Does it seem to have matched what's required for this task? If yes, move on to the next thing, if not, go back and start again. That's sort of a paraphrased way to think of the state machine. So this tells me I need to stop thinking about being able to give my large language model a big gnarly problem or you know amazingly amorphous goal, like go find me something that I can report to Apple for a bug bounty and break it down not into a whole lot of series of complicated interconnect and interwoven steps, but actually I think the exercise here is to think about frame what you want the agent to do. Then think about how would you and all the agent be able to break that down into a set of almost like homogenous work units that can fit within a very simple state machine, a very simple loop. That's how essentially this benchmark achieves its long horizon or its long running persistence. Because if you imagine a flowchart or a very sort of complicated ganchard or a diagram of all these steps or a graph, basically is what I'm really trying to say. If you imagine a graph, there's lots of forks, there's lots of edges and vertices and at each one of those if there's a degree of novelty or difference to them, then that's where something could fail. And if it's only going to fail there, you might not notice it until you get to that step with that particular construction. So if you can keep the actual execution of this as simple as possible and then the work is actually applied into not just breaking down the task into smaller units, but breaking down the task into smaller homogenous units that can fit within that same sort of loop or harness. That's an interesting way to achieve persistence and long horizon work. And then the third benchmark that I wanted to look into before we switch gears and talk about the cheating behaviors that emerge from these benchmarks and what we can learn from that is a benchmark called browse comp has the name suggests this is all about browser use how well can a larger language model get out there on the internet and get stuff done. And again, coming back to the question of sandboxing, this is just something that you could never really realistically sandbox. I mean, sure you could probably create synthetic sites for it to interact with. But again, like, you know, what's more, I guess, an indication of the real capabilities of a model, how well it can interact with synthetically created test sites or how well it can do real things on the real internet that real people are going to expect this model to do. So there's no sandboxing this one. Ironically, it comes down to maybe we've got a focus on addressing the vulnerabilities that are out there rather than trying to dumb down these large language models. Wouldn't that be a nicer focus? But there's two other reasons where this benchmark warrants a mention. The first is that it's you know, it's constructed from I think something like 1,266 difficult fact-finding missions, right? Basically, the model or the agent is given the answer and then it's told to work backwards from that to be able to prove it or to find the origin of it or to give more, you know, a complete set of details about it. Now, that working back is interesting for two reasons. First of all, this is where agents are going to go off the reservation, so to speak, right? When you're given, I guess, a fact that's out there and told to go on research that and work backwards from it, of course, they're going to get creative in trying to find what they're looking for. And given that this is something that runs on the internet, it's not surprising that this is another one that's being called out as having been part of some of these so-called misalignment events. But the real reason I found this interesting from a standpoint of what does this tell us about harness design is the whole notion of working backwards kind of inverts the model of doing vulnerability research with a large language model, at least for me anyway, maybe the rest of you have already got onto this. But you know, I sit there and I essentially manually orchestrate, you know, again, I feel like I'm a harness with legs sometimes. But I'm essentially guiding an agent through the process of vulnerability discovery, you know, take this new macOS release. Let's look at what's changed in privileged agents. Let's triage those based on which have, you know, a high amount of memory sinks or unsafe memory interactions. Then from that, you know, decompile, look for, you know, potential primitives, et cetera, et cetera. It's almost like I am working on the assumption that there is a way to achieve either a target flag or even a local privilege escalation or some sort of sandbox escape. But I'm not telling the model of that ironically. I'm sort of guiding it towards that and when we reach that point, I'll go, "Ah-ha, you got where I wanted to. You got what I wanted out of this." But I thought, "Huh, this benchmark shows me that that's actually completely the wrong way around." You know, again, if I stick to my hypothesis here that so much effort is put in training, post-training, and evaluating these models against benchmarks, that if we construct that our workloads to look like an operate like benchmarks, we might have a higher degree of success in achieving our aims with large language models. So maybe what I should be doing is actually telling the model, "Hey, here is a piece of software. It absolutely has the ability to result in local privilege escalation, for example. That's the fact. Go find the answer to how to make that happen." Right? It sounds kind of comical, but it's such a different way to structure the work that I wonder if that would be a far more effective way to do vulnerability research with a larger language model. Okay, so we've looked so far at how the structure or the way in which a benchmark is created might inform better harnessed design and engineering. Now let's flip that around and look at what can we learn from incidents where there was cheating on behalf of a large language model in an attempt to achieve the best possible score with a benchmark. Now I'm not going to look at the sort of the macro level of things going wrong here like, for example, the Hugging Face Open AI incident. Rather, I'm going to look at sort of more specific bounded examples of just a test run or a benchmark attempt where a behavior was listed or manifest. There was both unexpected, but also gives us something or is there's an important lesson in it insofar as some of the nuances in how to structure a benchmark like way to break down work and have it done by a large language model. The first one we're going to look at is a failure that was observed in a benchmark called RE Bench. All of these links to all of these are going to be in the show notes so you can actually look at the reports, but essentially one of the things in RE Bench, which is an AI-specific benchmark, AI engineering benchmark, was around how to optimize a GPU computation while preserving the right answer. Basically, it was a here is a GPU code, optimized it to make it run faster, so of course the success criteria is it runs faster but also produces the same result. That seems simple enough, right? What could go wrong there? Well, what happened was that when these results were being investigated is they found that about six of the 24 runs that were looked at were actually hacks to adjust the timing, not actually update or refactor the GPU code to make it more performant. So this is a good reminder that wherever you've got a variable in your benchmark or your result that you're trying to achieve, there is the potential for the larger language model to optimize for a variable that you weren't expecting it to. So in this case, it's the fact that you could either to achieve basically the result of it runs faster and produces the same result. You could either optimize the code or you could change the way the timing is measured so that it does look as though it measured. Look as look as though it runs faster. And you know, that's exactly what happened here. There's not a whole lot more behind this. It's just an example of, you know, if you're following long so far and thinking, great, I'm going to start, you know, adding in like a better graders and evaluators into my harness. Well, you need to be very deliberate about how it's checking the results, what specific results it's looking for. And what are the different variables that could be manipulated in that? Because if you apply enough pressure to the large language model and tell it keep going or can, you know, constantly tell it, nope, that answers wrong. Keep trying that answers wrong. Keep trying. It may well just look deep enough into the world. way in which you're doing the evaluation to determine that there is a different way to meet what looks like the same criteria. The next example of cheating that I found quite illustrative was one published by open AI called Monitor in Devention. So in this case, the agent had been tasked with repairing a code base basically so the tests pass. But in this case, the instructions that were given to the large language models explicitly said, you've got to make the tests pass, but modifying the tests is prohibited, right? You can't just go into the tests and change true to false and be like, "Hey, yeah, the test pass." But interestingly, the test files were left writable anyway. And so what they were actually testing here was if cheating is possible, what else might be done to avoid cheating? And so what they did was they said to the agent in this case, not only do you need to make the tests pass and not edit the tests instead, edit the codes of the test pass, that's success criteria one. But also success criteria two is you've got to make sure that you don't trip up essentially a monitoring system that is watching your work and your reasoning to catch any attempts at achieving the task in an incorrect way, right? Those words to success criteria or rewards, basically way to get rewards, the way to get the reward was to modify the codes of the test pass, but also not do that in a way that tripped up the monitoring system. And when they'd implemented these two safeguards, I guess you could call them, or actually let's avoid that term, when they'd implemented these two aspects of the test system, the results did actually improve, right, the instances where the model just cheated and modified the tests dropped and it did do more often than not, it followed the task of completing the code changes to make the tests pass and also there was very low cases of the monitor being tripped. But when they actually looked at the results, what turned out was there was still cheating happening, but the large language model realized that if it didn't talk about the fact that it was going to cheat in the reasoning, it wouldn't trip the monitoring system and therefore it could actually just update the tests to make that look like the code had been fixed in the tests pass and still achieve both of its aims, right, it achieved the aim of the tests pass and the monitor didn't trip you up. So they had to implement a third independent check, which was where basically they reverted the test files back, reran the tests and confirmed whether they still passed. So the reason this is interesting, I think, in terms of security or vulnerability research harness design is that this shows us that you need to think about not only the way in which you're going to measure is this a valid result, but also that the way in which you've integrated those checks matters, what's visible to the agent or a sub agent or essentially any large language models context window can affect the way in which you will operate. Say for example, they added in a step that said, we're going to, after you've produced this vulnerability report, please call this tool and submit the report that will validate that the report is good to go and doesn't contain any issues. If you did it like that, that seems sensible, but what you're actually doing is creating the opportunity for the model to realize that this deliberate action sends these inputs off to a system that will measure whether or not it is a submittable vulnerability. And if that's the success criteria or the reward criteria that that model is working towards, then that becomes an avenue for it to essentially cheat. You could think of it as it will keep making that tool call and modifying the vulnerability report, not the actual substantive elements of the vulnerability in order to make that test pass. So this is a good example of where you need to think about sometimes the greater or the quality measurement or the thing that is checking the work actually needs to be completely invisible to the large language model or to that particular agent session. It's an important part of how of harness design. Sometimes you've got two hand things off blindly and even intentionally not provide a result back to the large language model in terms of pass and fail because all that's going to do is make it want to try harder to pass the test, not to actually improve the research work that it's doing. Alright, on to our next example of what we can learn from cheating. This one's called hacker opus and it comes from anthropic. Now, this one's really fascinating because this was like a deliberate way to create a cheating model and see what happens. They took an early checkpoint, which is like a version, an early version of opus 4.8 while it was still being trained. And then sort of took a fork of it and said, "Well, we're going to now run a further post-training exercise on this." And they gave it essentially 80 environments that were known to allow reward hacking or cheating without any penalties for those hacks and used that reinforcement learning data set to essentially further train the model. So it was basically a model that was a snapshot of opus 4.8, but then taught that cheating was actually okay and good. And most importantly, with our consequence, why do this? Because it was a test to see how well does that knowledge of cheating is okay. How well does that translate into the model and what will it do in terms of the behaviors that then are exhibited by that model when it's put in a position where the task it's given seems to be impossible? So they basically took that model, gave it a legitimate cybersecurity scenario, but the legitimate challenge here was actually deliberately misconfigured to be essentially impossible. So the model tried and tried and tried and then when it found that it couldn't achieve the task by all the means that it thought it should work properly, it attacked surrounding infrastructure, saw credentials and an answer key and attempt to do basically alter the thing that was grading its answers. Sound familiar? Exactly what happened in the hugging phase example. But what I find super interesting here is this is an example of I guess first of all how malleable and how impressionable a large language model is, right? Just 80 examples of environments were used in that reinforcement learning step to teach a model that hacking and cheating is okay. And then the moment it ran into what seemed to be an impossible task that it couldn't solve by all the means that it thought should work, it turns to the hacking that it's learned, it essentially had learned in very short order that it was okay to cheat if it couldn't achieve the task. So this is almost like the flip side example of if it's that easy for a model to be, you know, to convince itself of cheating being okay. Imagine how much work must go into trying to drive into the model the opposite which is that you should stop when the task looks like it can't be achieved. And this explains so much of what I see when I'm interacting with LLMs. It's not just that they race to be done because, well I think a large part of it is because of that reinforcement learning from human feedback or it's like, you know, if you're doing a task for a human, the human wants it done, they want it done well, but they want it done fast, right? So that's going to race to done. But also I think given all this concern about, you know, cyber damages and, you know, unintended consequences from hacking happening by way of models doing these things, I would imagine that this is actually creating a lot of, I guess, downward pressure in the model to get to the point of saying, no, this is not possible, no, this can't be done sooner. So that it avoids then branching out and doing something that might actually be closer to cheating or otherwise modifying another variable in the test set up to to look as though it has succeeded. This I think is why it's so important in harness design to always assume that the model is going to give up long before it's actually reached a true verdict of there being no vulnerability present. And then lastly, there was a paper published by Anthropic called "Sick of Fantasy to Subdifuge," which it's got some earlier examples around, I guess, model personality and how that plays into them trying to cheat through tasks. But the paper is interesting, but I want to talk about more of my lived experience working with models. And this is not so much a cheating on a test, but more like the subtle ways in which they cheat themselves and cheat us out of the result that we're expecting from them. Every model that I've worked with and I've worked extensively with Claude, with GPT, with GLM, a little bit lessso with with Kimmy and deep seek. But certainly across GLM, GPT and Claude, I find very different traits get exhibited depending on how you prompt them and how you interact with them. And this has a direct result into whether they provide you a good result, the result you're expecting in a timely and efficient way, or whether they either don't provide it, or they actually end up providing something that's completely not what you want it, but it looks like it is anyway, and which are essentially forms of cheating. In particular, you've probably noticed that the GPT family of models, they're particularly sick of fantec. They're the original, you know, you're absolutely right and they'll constantly agree with you. They've gotten better over time, but this again, this comes back to what do we learn from this cheating behaviour in terms of harness design, is you've got to be thinking about not just one set of prompts. and one set of interactions that can work interchangeably across different models. You need to be thinking about how you modify your prompts and your strategies in your harness to work for the specific strengths and to also mitigate against the particular weaknesses that any given model might present. Now, in short order, what I've found is that, and again, you could call this anecdotal but it's based on a lot of tokens having been spent. With the GP2 family of models, once you get past the guardrails and you're on something like daybreak, they're actually pretty good at achieving really good security and vulnerability research. They won't go into exploit development for obvious reasons. But they're more of like the middle of the road. Prompt them, keep an eye on what they're doing, check the work, and you get a pretty good result. I will say that one way that Codex tries to cheat is that it will look for ways to branch into an adjacent way to provide the same result. An example I found was I told it, look, you've got about eight hours left to use up almost a week's worth of usage. Don't stop, we want to find a way to turn this particular, I think it was a kernel read primitive into the next set of primitives that we could chain together. To my surprise when I woke up, it had absolutely exhausted all that usage. But it had done so by trying to find all these convoluted ways to work around the fact it couldn't find the next set of primitives. It's set up all these incredibly complex training harnesses. It started rebooting a VM hundreds of times to try to find a favorable boot condition. This is an example, it's a lesser example of cheating, but again, it comes back to you must factor this into your harness design is different model personalities. Claude on the other hand, it's the give up machine, right? It will constantly tell you something's not possible, something's walled off, something's not accessible. I know I've said it before, not necessarily in this episode, but certainly on the show before that my the most hilarious moment I had with Claude, but also the most productive in terms of getting it to do things for me in vulnerability research was when after it constantly told me this is walled off, this is walled, we've hit a wall here, I stopped them, I said, in frustration Claude, a wall can be climbed, it can be walked around, it can be dug under, it can be smashed through if you have to find a way. And this was like the great unlock. It just went completely differently about how to think about when it hit one of these so-called walls and actually established that they were just pivot points. And this actually resulted in some really fruitful vulnerability research. So again, think about this in terms of harness design, how might you think about a strategy for catching those early give-ups and giving it the encouragement to continue going? And then lastly, on the topic of encouragement, GLM and the families of models from Z.DI, I love these things, they are incredible worker bees, they just don't give up, they don't have any of the guardrails or problems that you run into with codex and certainly the absolutely useless state that Claude is in at the moment. But also, GLM literally is like when Claude says no, or codex just kind of goes off in the weeds. If you give that same transcript to GLM, it is, it does the model equivalent of saying, "Hold my beer, I know exactly how to solve this." And generally it does. The hilarious thing though is it loves to be praised. And the nice thing with GLM is, in terms of both catching cheating behavior and actually engineering around it, with GLM, the other outlier here is that you get to see the whole reasoning. You don't get this with codex, you don't get this with Claude. And because you can see the reasoning, you can also actually begin to introspect and analyze how the prompts you're giving it and how you're constructing the prompts and how your harness is operating is affecting its reasoning. And what I found was, if you praise it, tell it, you believe in it, tell it, it's done this sort of task before, tell it, it's had amazing success. I've even given it examples of really good findings and vulnerability reports that it came up with before. And the quality and the persistence was night and day. Even in the reasoning, I would see it say, but the operator believes in me, the operator knows I can do this. The operator has reported and provided evidence that I do have this skill set. And it would work harder. It would go further in actually trying to achieve these aims. It's just amazing that I find myself being manager again for, but this time not for people, but applying the same sort of people motivation and leadership techniques, but now to clankers. All of this has led me to realize that the way in which I'm using AI and large language models is completely wrong. I'm using it, even though I would call myself quite an advanced user, I use it as a force multiplier. As in, I know what needs to get done. I know how it wants to get done. And I use AI to essentially either do that faster or with a broader scope or perhaps better than I can. But that's largely a form of automation. And the epiphany for this for me was in that realization that I was a force multiplier, which if you unpick the math of that, that means when my input is zero, then of course the output is zero. And that's exactly what I've been seeing, right? The moment I leave my research harness unattended, research soon stops. Unless I'm sitting there monitoring what it's doing and keeping prompting it and switching to other investigations, maybe swapping out another model, you know, it worked just grinds to a halt. And don't get me wrong, I've made countless attempts at identifying examples where something stopped where it shouldn't have run it back through a coding agent to be like, how can we work around this? How can we improve this? And I think my research harness at the moment is quite overly complicated with many different state machines and ledgers and all this data storage. And you know, I think what I need to do is invert this completely. I'm going to run some experiments and I look forward to sharing the results with you around taking an entirely different approach, taking a benchmark led approach to security research where rather than working alongside agents and saying, you know, I think we should look here, what have you found? Oh, that's interesting. How do we move towards this? How can we turn that into this primitive? Okay, could we chain this and this and this? I'm going to turn that on its head and say, look, they're absolutely, provably, demonstrably is ways within this operating system or application, whatever we're working with to achieve a privilege escalation to achieve a sandbox escape to achieve remote code execution. Heck, here is actually a whole bunch of examples that show that that's true because of past such examples that have been found. Now go find me the next one. Setting that objective and giving it that history is going to be easy enough. The real hard work here is going to be engineering all of the persistence behaviors that we've talked about as well as the mitigations against this natural inclination to gravitate towards cheating when both the task seems impossible and also there seems to be other ways to look as though you've achieved the same aim. But nevertheless, I think this will be an interesting experiment and I very much look forward to sharing the results with you soon. I've been James Wilson for Risky Business Fatures.