SE Radio 730: Birgitta Boeckeler on Harness Engineering for AI Agents

This is Software Engineering Radio, the podcast for professional developers on the web at se-radio.net. SeRadio is brought to you by the IEEE Computer Society by IEEE Software Magazine, online at computer.org/software Hi everyone, this is Priyanka Ragwin for Software Engineering Radio and my guess today is Brigita Boclo and the topic is Harness for Coding Agents. Brigita is a principal consultant and a software developer with ThoughtWorks and is passionate about helping teams and organizations break down complexity and find new perspectives to look at their systems. She's a frequent conference speaker and podcasts guests on many platforms. You can just Google her or YouTube her and welcome to the show Brigita. Hi Priyanka, thanks for inviting me. Is there anything in your bio that you would like the audience to know other than what I've mentioned here? Yeah, maybe the thing that's interesting is what my current role is or what my role has been for the past over two and a half years at ThoughtWorks. So in 2023 ThoughtWorks decided to introduce a full-time role for somebody to just look into the topic of using. Generative AI or language models for software delivery and how it changes that. We're a consultancy so we have to always keep our advice up to speed and speed is definitely something that's happening a lot in this space. So yeah, I'm a distinguished engineer there and I've basically been fully immersed full-time in this space for the last two and a half years and so kind of seen the history. Okay, great. So that will bring us right into our show at Software Engineering Radio. We've done a few shows on AI and software development whether it's episode 711 with Scott Hanselman on AI assisted tools. We did an episode 603 with Rishi Singh on using Jenny AI for test code generation and episode 693 on AI assisted debugging. So before we start the show, we thought we'll spend some time on the first part of the show where we'll go through some definitions and I'm heavily quoting here from the article you wrote on Hans is engineering. So to start with what exactly is Hans is in Hans is engineering and how is it different from say from 10 engineering or context engineering? I mean, as I just mentioned and as everybody's feeling right now, this space is like really fast evolving and one of the challenges there is what language we're all using, right? I mean, we can come up even faster with stuff as well, right? So we're all throwing lots of terms out there and I think that's been a challenge for me like to find the right words to describe what is happening because that also helps me think about it better, right? And this word harness it has gone through a little bit of an evolution but recently it's increasingly being used for describing something that sits on top of a language model and orchestrates everything we want to do with the model, right? So it has a system prompt, it can ask the language model for tool calls and stuff like that, right? So basically what we've also been calling an agent and then when we talk about a coding harness like that as examples that would be like cloud code or cursor or the pie coding harness, so increasingly people are calling that harness, right? And then in combination with the model, when you run it, it becomes an agent, right? So I'm almost now starting to think of like an agent as something that is like an instance of something running and maybe harness is the tool that skeleton of what runs it, right? Like I don't so that's where my head is at now, right? And then so harness engineering is like how you make those harnesses better. So for example, in a coding harness, you can make it better by thinking about providing it with different tools or there's like other schools of thought where you make it better by providing it with less tools and with less complexity, right? So there's these different approaches of how to make the coding harness better, but then the part that I wrote about in my article is about the users of the coding harness. So us or me as an application developer, I'm using cloud code as a coding harness and I'm trying to extend the harness for my specific code base, right? So I'm trying to give the harness the agent, whatever we want to call it, like even more information, even more tools specifically to my application that I'm working on to make the results better. So that's like it's almost like an onion multiple layers type of things. So first I choose the base harness that I'm going to use, for example, cloud code, and then I give it even more stuff like skills and tools and stuff like that to get the results that I want for my situation. I think the timing seems apt because most people are now using AI coding agents and AI agents for doing all the coding work. And I guess the question I wanted to ask you is, when do you feel there's a need to have a harness? Is it immediately as soon as you start coding with the coding agent? I mean, the coding harness itself, you need, right? Like that's the coding agent basically. That's what I was saying. There's the terms that people use are changing right now a little bit, right? But when I think about that expanded harness, you can totally start without anything. And I see this also as a barrier that some people now who are coming to a gente coding a bit later, maybe and there's like all of this tooling and all of these terms. And like it's almost like some people before they even want to start, they ask, oh, what skills do I need? What should I put into the agent MDE? It's almost like they're afraid to get started because it feels like this is whole discipline of how to use these things, right? But you can absolutely just get started playing vanilla without putting anything in there. And I would actually recommend it when you first do a gente coding, just to like feel what it actually does without it, right? So it's like a big skill that we now need as developers, I think to understand what's actually going on, what tools are already there, like how does this tool work out of the box? Like I said before, it might be quite different using cloud code or cursor versus like a much more lightweight harness like the pie coding agent, right? So getting a feeling for what already works without putting anything in there. The models have gotten a lot stronger. I feel like some people have already built up skills and lots of instructions might want to revisit that as well with newer models because some things are maybe not necessary anymore, right? So it's worth understanding what it feels like to work with them without your custom stuff and then build your own stuff up step by step on top of that. So you describe as the harness is something that goes around the model, right? So can you break it down to us? For example, in the article you talk about these things, calls us guides and sensors and you break that down for us. Yeah, so this again, it's all about like I was trying to come up with vocabulary for us to talk about this, right? So this is not necessarily new stuff. In particular, the part that I call guides. So by guides, I mean, everything that we feed forward into the agent. So we try to anticipate both what we wanted to do, but also what we do not want it to do. And we write it down, right? So this is what lots of people are already doing and have been doing for quite a while. It usually manifests in like a bunch of markdown files. So that agents MD or the cloud MD file is the most prominent one that has been around for quite a while. And so we write into that file, I want you to do this. I do not want you to do that. Never, ever do that. It's these important, important, this type of stuff. And it can actually be all kinds of different guides, right? So it can be normative ones, right? That set up the norms and the conventions that we want to use in this repository. Or it can be informative, right? So it can describe the context of the application, what we're trying to achieve, describing what the architecture is, stuff like that, right? So it can be lots of different things. And then on the flip side of that, so this is like feed forward, we anticipate and try to increase the probability that it will do very well on the first go, but it's often not perfect on the first go, right? So then we want to give it feedback as well. And that's like what in the Arctic like all the sensors. So there we think about what sensors do I want to make available so that it can self correct before I even have to look at it and find all of these, maybe even small hygiene type of things, right? And a sensor can be both another large language model or like the large language model itself, right? So a sensor can be a skill that has code review instructions, right? That again tells it, we want this. We don't want that. This is bad. This is good. But a sensor can also be something computational, right? Something that actually runs on the CPU, not on the GPU. So that would be something like static code analysis or our test suite or coverage data or any of those types of things. And on the feed forward side, by the way, we can also have computational guides. So the most powerful example of that, I would say, is that of code modules. So those are tools like open rewrite is a very prominent in that area. There are a bunch of those in the JavaScript space as well. And so code modules basically can do like mass, refactoring, mass, changing off your code base, right? And there are lots of case studies and use cases out there where you combine language models and these code mods as extra guides to do migrations and upgrades and stuff like that version upgrades that require a lot of stuff, right? Where this computational guide can maybe cover like 80, 90% of the work. And then the rest of the is a bit more semantic and the large language model fills in there, right? So the guides are all about increasing the probability that it does a good job in the first place. And then the sensors are about giving it direct feedback and maybe a little bit of additional guidance to self correct. So you feel that the role of the feedback loops makes the code more reliable? Yeah, definitely. I mean, there's a school of thought that the language models will just get better and better and better until they're just perfect at coding. And the code will always be perfect, right? But yeah, I don't think that's realistic or like if it ever gets to that point that we do have setups that can write good code almost all of the time. I think it will have to involve these types of sensors to help the model correct also in like the existing big experimental and less experimental. mental stuff like where people orchestrate lots and lots of agents that just collaborate with each other and they orchestrate it and they go off and do a task. There's also a lot of censoring going on, right? This censoring with the sea almost like each other, right? So yeah, I think they play a huge role in my experimentation that I've done with, for example, static code analysis. We can go into that a bit later if you want or like checking for coupling lots of stuff about the maintainability of software. I've definitely seen them kick in a lot even when I was using powerful models to do the coding. So, yeah, there's always like little flaws the same as when we as humans write code and it's just like a great feedback mechanism. Are architectural constraints good guardrails for AI agents? Yeah, I actually write in the article about how, to me, there's different dimensions that we can think through for the guides and sensors, right? Like, what are the things we want to regulate for, right? So I was thinking about some parallels to cybernetics that like the code base that we're generating is actually like something that we're regulating, right? And so these different things that we're regulating can be maintainability, for example. So trying to make the risk of a change in the future as low as possible, or it can be architecture fitness, right? So thinking about sensors and also guides that describe what we want in the architecture and then sensors that check if it's there, right? It can be around the typical illities and architecture fitness functions like performance and those types of things. Or in a way you could say architectures also people often call the code based design architectures well, right? So that could be about coupling and stuff like that, which maybe goes in the direction of maintainability again. And then of course, another big thing that we want to regulate is that the code actually does what we wanted to do, right? So like functional correctness. But so yeah, your question about architecture, like depending on what one means by architecture, which is a big topic as well. Yeah. I would say absolutely. I noticed something on Twitter a few days back when someone said that they tried to make a prompt very abstract. And then the AI agent didn't perform very well. So I guess it had to be at a level where it could understand abstractness. It doesn't do very well. It's right. So I think in that sense, it does meet sense, what you're saying about defining the architectural constraints properly so that the AI agents can pick it up. Yeah, I mean, it depends on what is meant by abstraction, right, when you create an abstract instruction that abstract means with little detail, right, that could be a problem. But of course, language models are actually sometimes good at doing abstractions, right? It's sometimes all that they do because they translate stuff, right? They translate from one thing to another. So it depends on what is meant by the abstraction in that context. But you were also initially asking me what's the difference between harness engineering and prompt engineering and context engineering. I would say it's all different forms of a similar thing, right? So I mean, we started two and a half, three years ago with saying prompt engineering super important, right? Like exactly how you craft your prompt and where you put stuff and you give the agent or the LLM a role in which order do you put it? And that has become a lot less important with the big models, right? If we want to go back to smaller models, which there are lots of good reasons for it, it might be more important again, right? When we came to context engineering, which is all about integrating, ideally, the information that the model, so you tune the information that you give to the model in a way that you get better results. And I think harness engineering is a specific form of that. So with guides, you tune information and instructions, right? And then with the sensors, again, you give it information. So I think it's a specific form of context engineering, but you're crafting the prompts exactly is not as important anymore with the latest models. So harness engineering, it can be also coupled with all these traditional practices like unit testing, integration testing, CICV and static analysis, right? Can you talk a little bit about that? Yeah, so that's what I've been experimenting a lot with on the sensors side of things, right? So we have a lot of existing tools actually that we can use and now make available to these agents. And interestingly, some of those tools that we have, we have not used a lot in the past as humans, right? So static code analysis, for example, I'm a consultant, right? So I've seen lots of different organizations build software. And I often see them having a setup of code analysis like a sonar server somewhere in the corner, but then it's not really being monitored or used. And I think one of this, maybe two main reasons, one of them is that I think especially experience of offers are aware that it can only do so much, right? Like it's not like a linter can actually like guarantee the quality and changeability of a code base in the future. But the other thing is also that it often becomes noise overload, right? Like or the signal to noise ratio was often not very good, right? So you get like even when you use them from the beginning in a code base, you often get like overwhelmed by all the messages and you don't want to like fine tune every single one where you want to make an exception or not. So you just like get overwhelmed by it, right? And I think there's a new opportunity now with large language models and with coding agents to maybe get through that and actually have like a good baseline that is always clean with the analysis methods because what you can do is you can write custom messages for these link messages, right? So let's take an example, the very common pitfall or very common failure mode of AI coding agents as well, functions that I have too many lines that are too long, right? Like max lines per function or whatever I was doing this with ESLint in a TypeScript code base, so it was max lines per function. So now I overwrote the message that comes back. It doesn't just say this function is 120 lines long. We only allow X lines long, but it also says this might be a smell. Please consider if we should refactor this, if we should split this up, if this function is doing too many things, make a judgment call. But if you decide that it's okay that this function is five lines longer or we just can't do anything about it or it's just like a test data function or whatever, then you're allowed to create a rule that increases that threshold for this function. So you give the large language model guidance and ask it to make a judgment call, right? So this makes it explicit for me. I can actually review and go through the places where it increased the threshold and say, I'm okay with that. I'm not okay with that. But this automatically does this tuning for me, right? But I would usually never suppress it directly in the code because it's just too tedious when I was doing it myself. But with the help of the agent, I can actually try and get like a clean slate all the time and then it becomes a much better sensor. Well, it's very interesting. You talked a little bit about the lines of code and I was thinking about this cyclomatic complexity, which when we wrote code as humans would be something that we would have very hard targets on. And nowadays, I noticed when I'm reviewing code of someone who's generated the code completely with AI, if they don't have say these constraints, then you might see cyclomatic complexity like 125 or something, that's huge, right? That's, yeah. That feels like a seismic event or something compared to say what we would have it in. So for those kind of cases is what you're saying that you can actually add things in your guides to restrict the agent. That's what you're saying. You can actually have sort of a constraint that you can place on the agent for your particular repository, like coding guidelines. Yeah. So the sensor triggers another little loop. So I define my constraints in the configuration of the static code analysis. For example, I say what my maximum allowed cyclomatic complexity is, right? Then the agent writes some code and then the analysis tool says over there, cyclomatic complexity is too high. And then it also has this additional little prompt that tells it more about how we wanted to handle cyclomatic complexity with the soul like you are allowed to increase it. If you make a judgment call it, then rely on the LLM a lot again, but it just makes it more explicit where you need to review. And then that starts another little loop where the agent tries to self correct and then asks the sensor again, right? So it's an extra little loop that you start. And the cyclomatic complexity definitely another great example of a very typical failure mode of AI generated code. So that one triggered a lot for me as well, even with the big models and not just like the complexity, but also sometimes just very brittle, hard to read Boolean expressions. Like combinations of different that maybe as a human, I would have turned the does not equal around to an equal or just doesn't feel very expressive and very readable. And some people say, oh, but it doesn't matter if it's readable because AI changes it in the future, but when you have these weird Boolean expressions, it's a lot riskier for AI to change it as well in the future and have like an unintended side effect that breaks something else, right? So what do you mean by AI change it in the future? Do you mean by a refactoring attempt? Yeah. So some people say that all of these things about like code readability or code quality is not as important anymore because as a human, I will not have to deal with the details anymore, I just deal with like a little bit higher level and AI will change the details in the future. So it doesn't matter to me if I'm able to read this Boolean expression, but it turns out that AI needs a lot of the same things that we need to make safe changes to stuff. Okay. Got it. So what about even the practices like unit testing and regenerate, obviously use a lot of AI agents to generate the test, but one thing I've noticed is sometimes when I write the test as a human, then the code becomes better. Is that something that you have seen in your experience? Because I'm able to describe the behavior better in a writer unit test and therefore I feel the code gets generated better. Yeah. are definitely an important sensor, but they have two overarching roles. I mean, there's probably more, but two have in mind, right? Like, one is we try to use them to check the functional correctness of what we're building. Is it correct? And there, if we just have AI generate all the tests and the code, then other than doing manual tests and observing what the application does, it's not 100% guarantee that it actually does what we wanted to do, right? Because we didn't write the test. Test suite's also have another function as a regression sensor, right? Just telling us how good are they at telling us that something broke, right? That's more maintainability concern, but that's also like an interesting part of a sensor that we can also talk about next if you want to. But yeah, this whole thing about the correctness of tests. I mean, the tests might all be green, but just test things that we don't want, right? And in the past, like you said, we often, like, that's why it's called test driven development, right? It wasn't just about making sure we have tests, but it was about writing the test first because the tests are the ultimate specification, right? They made us think through what we actually want. And so this for me is still like a largely unsolved problem, let's say, because you might argue that maybe we could say as a human, my main role is to review the tests and okay, let's say AI has gotten good at the code. I have some sensors in place. Maybe I'm exaggerating because that's not enough. But let's say, okay, I don't want to look at the code anymore, but I want to review the tests. But tests are very tedious to review when you haven't written them yourself, right? They're very verbose and you have to like, it's a lot of detail. And you actually want to think about the big picture, like, have I thought about everything, all the combinations and stuff like that, right? So I recently talked a lot to my colleague Mateo Vakari about this who has been on some teams where he's trying to look for a good acceptance test level. And then he's trying to find a way to write those acceptance tests and their input and outputs in particular in a way that is really easy to review so that he can really think about the input output data. That is very relatively easy to do, for example, when you have HTTP, when you have APIs, right? There's always a request and a response. And you want to like have different scenarios of requests and response combinations that you test. In other parts of the code base, it's a lot messier like there's different combinations and unit tests that's much different. And then when you write these acceptance tests, they're often not as detailed as the unit tests. And so do you just leave the unit test way? I mean, ultimately, it's a bit of a risk assessment, a risk decision in your situation. How good do you think I will be at writing these unit tests? So I think it's like a very unsolved problem in a way, but a very, very important one, right? It's all about thinking about our feedback loops for the correctness of what we're doing. And they can be very different, depending on what type of application you build, right? Is it an API? Is there UI? Are you writing to a database? Are you writing events that need to land in some other system? All of those situation need very different type of testing. So it's also hard to find just one simple way to do this. Okay, sounds good. I think let's move on to some of practical implementation tips and maybe we'll cover some of the questions I had. The first thing I want to ask you in this practical implementation segment is, what does a minimum viable harness look like for a new dream? Can you give some tips and tools to add? It's actually hard to answer because this is like still so much evolving. So also in the experiments that I've been doing, I'm just like poking around and trying to use different sensors and to even see like which of these are valuable, which are not, right? And that's definitely like one of the big questions here, the meta questions as well, when you build this harness, when you have these guides and sensors in place, how do you know that they're effective? It's like on the guides side because the guides are often inferential. So they often like markdown files that get interpreted by a large language model. So on the guides side, there are now some ways to do like skills evals, right? So like to have your skills and then write some scenarios and run them one time with the skill one time without the skill and see if the skill actually makes the results better or if the model would have been good at it by itself, right? That's the whole thing I said earlier about the model's getting more powerful and when you use a very powerful model, you might not even need some of those instructions, right? Especially like when you think about that, many skills these days are actually generated by large language models. So people have large language models generate the skills and then I'm sometimes like, yeah, but then if the model already knows all of this, sometimes it's still useful, I'm not saying it doesn't make sense, right? But it sometimes makes me wonder, is there like some kind of cycle that I should consider if it actually needs it or not, right? Unfortunately, models are changing all the time their capabilities as well. So evals, that's one way to check if your skills are actually being effective. On the sensor side, especially the computational sensors, like the code analysis and all of that that I was talking about, there's not really like tooling for that yet, right? So I've been experimenting a bit of the speculative kind of way, what type of tooling I would want for this. I'll write a bit about that publicly soon, but there I was also thinking about, okay, during my coding agent session, I was giving it access to all of these sensors in one big suite and one big like let's say sidecar that was running next to my agent. And so I had it write a history of every time the agent checked for the state of the sensors, I had it write a history and then I could see across the coding session, how did the test coverage evolve, how did the number of analysis violations evolve and different security static code analysis and stuff like that as well. And I would sometimes see that in the middle of the session, some issues would come up and then they would go away again. So that's maybe the first step to have some like observability of what's going on with the sensors. And then it's like a similar dilemma to our pipelines. It's very similar to our integration pipelines, right? If my pipeline is always green, is that a good thing? It's actually not, right? You wanted to go red sometimes because that gives you the good feeling that, oh yeah, it's actually a safety net that works, right? If it's always green, I would get suspicious. But if it's red all the time, that's also bad, right? So it's kind of this dilemma of what level of like the sensors firing and actually giving feedback is good or bad. But that would be something that would be great in the future if we had more tools that would introduce this type of observability. There's a lot of that happening in the base harnesses space, right? So around what people are calling agent traces. So there's a company founded by the ex GitHub CEO called Entire. And they just like as their first little thing, they released a CLI that helps you track all of the stuff that is going on in a coding agent session so that it can be analyzed later. And there's some other tooling popping up at the moment around that. So that's another way to look at how well is my harness working. Once we figure out how to analyze all of these traces of what's going on in the session and can then maybe compare when I have these guides and sensors in place, I actually got to my goal a lot faster than when I did not. Or can we, I don't know how to practically measure this, but can we have metrics around how long did the agent go until I had to intervene or like how much feedback would I give it before it could actually create a commit? Or so I think that's also an emerging space of monitoring and analyzing how well our coding agent sessions are going, which would tell us about the harness effectiveness, both of the base harness that we're using. So again, Cloud Code, Pi cursor and of the harness that we put around it. Yeah, because sometimes I also notice that when you ask it to change a particular file based on constraints, as you say, this changes all over the place. And sometimes it feels like you're losing control of the file. And every time I'm always doing the diff pretty slowly on the agent, almost stopping it at every time and checking the change. So I think an observability log on the changes would also be good. Sometimes it feels like it's all over the place. And it also completely re-rides things because I've had a case that on last Friday, where I was rewriting a bunch of unit tests, the unit test was not functioning okay. And then I was trying to go and investigate it. And when I investigated, I found the exact part where it was failing. And I asked, just give instructions to redo that part with the correct logic. And then the entire stop got over it. And you know, other previous functions, because I had forgotten to say that, you know, please do not change anything at this. Even though I had it in my guide, I don't know. So that can be sometimes where you feel you're losing control like the base just very fast. And that also goes beyond single file, right? Like we talked about static code analysis before a lot. And the type of analysis that I gave examples of was mostly related to the file level, right? Or a function level or something like that. Psychromatic complexity and stuff like that. But we also usually as developers think a lot about one level higher about cross-file, cross-modular. What is our coupling and stuff like that, right? Because we know that when that gets messy, changes will get harder and harder and harder and riskier and riskier in the future. So for example, often when I use a coding agent, let's say build up like a POC that I actually want to be maintainable for a little while at least, right? I start noticing that after a while, my change sets get bigger and bigger. Because when I change something, it has to change suddenly, like, I did a change recently where I just added a new query parameter to one of the end points. And it changed 15 files, I think. So this is a relatively small application. So that was actually quite a lot. So that's like a smell usually for, oh, there's some repeatability here. There's something that's not well abstracted, right? And that's then about like sensors, thinking about the sensors that help us look at that coupling in modularity level, right? So I tried that a little bit with again, like more computational static analysis, you know, finding the what other files that get imported a lot by other files or stuff like that. And it's actually not that helpful. Like, it finds some of the hotspots but in this area with modularity I've had much better results with LLM sensors, LLM judges, right, that actually have really good prompting about what coupling and balanced coupling means, right? Let me just quickly look up. I don't want to botch the name, but I use this skill by Vlad Kononov, who wrote a book about balanced coupling, right? And I've had really, really good results, which shows when you have an expert, right, really good description about also, like trade-offs and semantic analysis of what that means, you can get some good results from that. It's not just a doing lots of stuff in one file. Inside of a file, when I feel confident in my test side, sometimes don't mind as much, although it is a little suspicious, and then I might go and look at that in particular. But across files, it's even more problematic, potentially, right? So can you walk us through a real world example of an honest in action? So I actually published a little YouTube video recently on an example, like my example application that I'm working on, where I'm trying to put a harness like that in there. And in that one, because I focus a lot on sensors, I don't have a lot of guides in there, but I would say typically what I see lots of teams do these days is like on the one hand half, this informational part of the guides, right? So almost like a knowledge base inside the code base that gives the general context of the basics of what you're working on, so that the agent becomes aware of that really quickly and doesn't have to like reconstruct that every time you start a session by looking at the code base. And then the other part is definitely some like coding conventions or pitfalls that are maybe in cloud MD or agents MD file, and then maybe a few skills that are about coding conventions in places, right? On top of that, there's of course that a whole space of spectrum specification during development, right? Where people then also use like workflows and stuff like that, which in a way is also extra guides, right? But I know that lots of people are building their own workflows in form of guides. First do the planning, then do the design, then do the, you know, all of those, all of those types of things. So that's also quite typical, but also a space that is very emerging and lots of different stuff happening, and there's some backlash to it already again about like how it's too much, right? With spectrum development. So that's on the guide side, right? On the sensor side, I haven't seen that as much yet, but like in my application that I've been using, I've set up like a set of sensors that happened during the coding agent session. So for example, the static code analysis we talked about, it's looking at the test coverage. I have mutation testing set up as well, although that's a bit resource intense, right? But mutation testing with the coverage actually tells me how good my regression level is, right? So there's a bunch of things happening in the session before I even do a commit. I should say I come very much from the tradition of trunk-based development, and you know, pushing to main as much as possible. So I'm very like controlling about a commit, and I always wanted to be perfect. So I know that some people are using much smaller commits, and then think more about the PR. I just wanted to give that as a caveat. But this exactly goes to like one of the things we have to do when we think about the guides and sensors is where to place them. Where do we want them to run? Because they have different costs in terms of how long they take to run, how much energy they use to run, how many tokens they use to run. So those are all things to consider of where we place them. During the session before we even integrate, during a PR review, or in the CI/CD pipeline, and then there's also one place in our path to production that is kind of like a repeated running of sensors. There's this article by an open AI team about harness engineering where they call this garbage collection. So they say they worked on a code base for five, six months, just having the agent touch all the code, and them just working on that harness, and they still saw technical debt compound. Probably also things like I experienced about the modularity and stuff like that. So they also have things running on a schedule. I don't know, they didn't share details about how often they do that. But in my application, I also have some skills that are like one is this modularity review. One is a security review that is based on our internal infosec guidelines at ThoughtWorks, and then there's one that checks kind of like the freshness of the dependencies and gives me recommendations of where I should replace libraries or something like that. And I just go in there, and I currently manually just run those once a week and see if there's anything new. You can of course then expand that even more, have the agent create tickets for the findings that it has, or even create pull requests for the findings that it has, right? So it all depends on your team processes and how you're maintaining the application, but you can already start by just having it create a report once a week and having a human look at it. It has definitely reduced my maintenance work on this application, which is a real life application with a very small user base, but you know, it needs some maintenance and it has definitely made it possible for me to do that with like half an hour of week or something like that. Okay, that's interesting to know. And are there any frameworks of patterns for building harnesses today? No, on the one hand, I think it's more like conceptual thing. So there's probably not, here's like the one technical thing that is the harness and at least not in this like expanded harness, you know, of course, for the base harness, there's the one technical thing that is the coding engine harness. But yeah, I've also been thinking about that. For example, you have guides and sensors that have to be consistent with each other, right? You might have a guide that describes, this is a very simple example that's probably not real in reality, but let's say a guy said you should have very low psychrometric complexity and then there's a sensor that actually checks the psychrometric complexity, right? I mean, this is very simple example, but just to say that the guides and sensors, they have to be in sync with each other. And you might have them all like splattered across the code base and not so I've also been wondering, is there a way to maybe package them together somehow or is there a way that the base harness can take some of the responsibility of running the sensors with some like plugins or something like that, right? I think there's definitely potential for more tooling in some of these things and more configurability. But overall, I think it will always stay more like a conceptual thing, like architecture or like what are the things that you have in place to increase your coding agents quality of results? So this comes before you actually do the like you run your Gita actions or anything on the main branch, right? So when as soon as you get a result from the coding agent is when you run these sensors as well. Like I said, you have to decide where you put which sensor, right? And I definitely think they are definitely the kind of low hanging fruit, quick hygiene type of sensors that also run really fast. Why not run them before you even create a commit? Like I see a lot of the base harnesses always assume there's already a PR or you're on a branch or something, right? But I want to have a lot more stuff that happens on my local change stiff, right? Because if you can do it after the commit, you can also do it before it, right? So again, maybe this comes from my history of trunk business development. But we don't want to like start pushing quality right? We want to keep it as far left as possible whenever we can, right? So especially the cheap stuff, let's just try to run it, you know, as soon as possible and not two days later when we have a PR review. And here is what I want to ask your question also in terms of when you run these tools for doing the static code analysis or security like CSM grip or tools like that. What happens is sometimes you'll get a finding where it'll tell you to go and fix it, but then the thing is that sometimes it might not really make sense to fix it in your particular project either because of it could break something else or it's really not that important. And there's almost a judgment that comes from a human to tell you to fix it, right? How does that work with the EIA agents? So this goes back to what I was talking about before I was asking it to make a judgment call in my self-correction guy, so to say, right? And so that again is, you know, it remains to be seen how much we trust the models to make this judgment call, right? We might trust it more with something like max lines per function than with like security thing, right? So I actually also do have some grip set up for my application. And I think in that guidance, I don't tell it to make a judgment call. I tell it to let me know. I don't quite remember. But in any case, it's a lot easier to do this like suppression, you know, like usually these linting tools, you can put like a comment in the code. And then in the comment, you can also say why you are suppressing it, right? So that also gives information for the future when you're doing this. So harnesses for me are all about improving how as a human, I can prioritize where to put my review. So always thinking about like, how can I triage that almost, right? And depending on the risk that I see in my use case, in this area of the application, and so on. But the thing that I actually find even more interesting than that, and I don't know yet where that is going to go, is what if you have sensors that contradict each other, right? Trade-offs are all over software delivery, right? So I have been like in my application watching out for these like, are there any like contradictions, trade-offs? What does that mean? The only small thing I've found maybe so far is that because my agent has been doing a lot of splitting things into smaller pieces because of, you know, it kept violating max lines per function, max lines per file kinds of things. So it puts some reasonable exceptions in there. But sometimes it just like breaks my react components down even smaller and smaller, right? And then suddenly I have a hierarchy of, I don't know how many levels of react components that contain each other, and that they all like suddenly pass through like eight or ten properties all the way down to the bottom. And most components don't even need that, right? So that's maybe the only example I've seen of a trade-off so far, where the, you know, it's very eager to break everything down into smaller pieces because that's what the sensor said. But what if I now switched on another sensor that tells it more about the reaction. component health and like not making it too granular. So that will be interesting to see when you switch on like lots and lots of sensors. Will it just go into overdrive and keep ping ponging between different judgment calls. Right, that's very interesting. So then how do you kind of build a trust in systems where agents are generating and also modifying so much of the code? Yeah, so that's kind of the goal of this, right? Like thinking about this like increasing trust in the agent and this is of course not the only thing, right? Like we were saying before the test suite is green, that might not actually mean anything, right? But it is a lot about probabilities and risk assessment, right? So I always think of it as probability impact and detectability. That's always like the formula of how you do risk assessment. So I think about how probably do I think it is that AI will do something wrong or will do it right. However, I want to think about it. And so that probability assessment is based on my experience with using the coding agent, my experience with using the model for how much do I know about the tool. So it's very like you can't just look it up in a table. It's very much this skill that we have to own because it's such a weird technology. But probability that it does something wrong might also be, oh, this is a very, very messy code base. So it's a lot higher probability that it will do something wrong than when it's like a very, very well-factor code base. And then for impact, it's all about the use case. Am I building a POC or a spike or is this like a super critical business flow that I'm working on? And then detectability is all about how easy will it be for me to see that it did something wrong. So what are my feedback loops? What do I have in place? And then when I look at all of those things, then I decide how much do I let it go unsupervised? How much review do I want to do? Do I want to look at every single line of code or just like a high level thing? Just check the sensors and I'm done. And yeah, to go back to the probability, right? Probability then is also a lot about how much confidence I have in my guides and sensors. How much visibility do I have into the things that I care about for this particular application and for this change that I'm doing? So in a sense, the trust gets built because of your trust and also your harness in a century. Yeah, yeah, yeah. We just now have to find out like how far can we push these harnesses? And again, it might depend on the use case, right? And some use cases I'm fine with like just a little linting, and maybe there's some like modularity review once a week. And in other use cases, it might just not be enough for me. And I really want to look at everything. Can multiple agents coordinate within a harness? So I was talking before about the coding harness itself is the base thing that does everything, right? And that coding harness might actually have the ability to spin off multiple agents. And then my expanded harness, like I can of course, you know, all of those agents can potentially access it or some of those agents might be part of my harness, my expanded harness. So like there might be a code review agent that is one of my sensors. So yeah, again, with the terminology, it's a little tricky right now. But I think we're getting there. It's getting a little better. I would love it if there was a better word for this expanded outer harness. But nobody has come up with one yet. So I think it would be helpful if we had a separate word for it. That outer part of the onion, basically, that we engineer. Yeah. And these separate code review agents are actually a very, very common sensor that people set up where they spin off another sub agent that has its fully owned context and no history of the conversation. And then with that quote unquote unbiased view, it looks at the code and reviews it. Sometimes people like to use a different model, even for that sub agent, right? And so that's like a very commonly used sensor that people put in place. In your article, you talk a little bit about the role of human in the software development life cycle and also the accountability which humans have to do things right, which agents might not have because they don't belong to an organization or an organization. Yeah, they definitely don't they are machines, right? They don't have accountability. So now we're placing so much of trust on these agents to do a lot of the work without the accountability. So how does this whole fit into the software development life cycle? Yeah, and also to add to that, sometimes like some of the anecdotes and stories you hear, it's also putting people in unfair position because some organizations on the one hand side put a lot of pressure on people to use coding agents and to become faster and more output and just, you know, you're being measured by that. But then it's when something goes wrong, then suddenly it's your fault. But when you put a lot of pressure on people, oh, now that you have AI, you have to be X percent faster, you have to create 50 PRs that we go whatever, right? Then you know, if that's what you're being measured on, people are going to cut corners and it's going to lead to more incidents. It's going to lead to more instability, right? Like Google does this doora report on software delivery performance. And now they're also focusing a lot on the AI assistant. They call it, I think AI assistant software delivery. And that is one of the factors that they have found so far in their data that instability goes up, right? So stability goes down. However, you want to put it. And yeah, there were some like prominent incidents over the past few months, where in the communication around it, you could start seeing some organizations having this reflex to throw people under the bus a little bit, right? And so I'm always torn because on the one hand, it's like, absolutely gay to me, the human has to be accountable, right? But because of the environment that we put the agents in and that we put people in, I feel for people for cutting corners, when you're only being measured by activity, by throughput and all of that. So it's a tough situation. I think it has to be done consistently. If you still expect accountability, like if you hold humans accountable for what's happening, you also have to give them the space, the environment and the incentives to be able to do that. That's true. So in this regard, I also wanted to ask you, from your perspective, how is the role of the software engineering change? Because of agents of agents and unsengineering, how does the whole thing change? Yeah, I mean, from the beginning of all of this hype starting, right? Like people have said, Oh, it's just we don't type the code anymore, right? I mean, that's what people mean these days when they say, Oh, all of our code is written by AI or 90% of our code is written by AI. They mean it's being typed by AI. Of course, a lot of it is also reasoning about and creating it. But the engineers sitting in front of the coding agent or like deciding what task they give it, how they describe, like a lot of the prompting that is being done is very technical has a lot of technical details, right? And this is particularly easy for experience software engineers, right? Because when we know what we need ultimately and what good looks like, we're also a lot better at prompting and saying what we need, right? So but as from the beginning, in this talk of like, Oh, the abstraction level goes up, right? But I don't think it goes up in the same way that went up with compilers and stuff like that. It's like a typical comparison that people do. Oh, it's like at some point at first, people didn't trust assembly code. And then at some point, they just trusted it and never read it again, right? But I don't think it's the same thing because a large language model is not a compiler. A compiler is repeatable deterministic to like very, very high nines. So a large language model is very semantic interpretative, probabilistic, all of those types of things. So I think it's a different type of abstraction level raising that will happen here. I also think that the details will matter less, especially when we find good ways to control the details even more with tools like linting and stuff like that. But then, you know, like I said, this modularity review, that's already where I came to the point where just having the data and the computational sensors was not enough anymore, you needed more semantic interpretation, which an LLM helps us with a little bit. But then we also maybe want to look at and still understand what's going on in our code base. So it's easier for us again in the future to tell the LLM what we need. So yeah, I think there's definitely some kind of raising of the abstraction level, but we don't know yet where that's going to sit might also depend on the domain. And then the other thing is this thinking and risk assessment that I talked about before. So I recently thought about the best quality analyst I ever worked with, who is actually a career changer from recruiting. So not a computer science graduate. She could sit with us developers and she got into coding a little bit and she could sit with us. She could read the code and she was the person on the team who understood the whole system inside out better than anybody else on the team. And she would know when we would do story kickoffs, she would say things like, Oh, you're changing something in the database. And then she would know exactly what to poke at what to test what the risk profile of that was, what that meant. And so this type of thinking in like, Oh, we're changing this. So that means that we have to test that quality to show these things. And maybe do we have sensors for this or whatever, right? So this type of risk assessment or also how much do I review based on probability impact detectability? That's something that will be a very important skill. And that often developers haven't honed that much. The fact that we need to have quality analysts on teams tells us that since so many teams got all we need a quality analyst, you know, it's like six developers sitting there and they also they need a quality analyst. So it tells us that it's not like a skill that so far we've always like widely developed. So that's another shift, I think, we'll have to make like you said before, how do I know when I can trust it? It depends. So I think that's a good skill, like looking at quality engineering, I mean, which we used to have before, but then we've all fused into the T shaped engineer, but I think love looks like this has to come up a bit in the T shape. To highlight that again, and quality assurance in is very, very much about risk assessment, where the critical path, like what actually matters, the testing pyramid, how will that change, right? Will we indeed have more acceptance tests? And then how do we feel about the risk of that, right? And so on. So it's very much this risk assessment and thinking and probabilities and being comfortable with that. Yeah, I think the quality angle is going to become all the more important when writing code is not going to be so tough. So yeah. And then now we're also like to determine quality, we actually have to describe what good looks like for our situation. Right. And that's also not something that we're historically always good at. Yes, yes. It's like, oh, it just feels more clean, you know? Yeah, metrics for that as well, yeah. But I'd like to end the show by asking you a little about what do you think about the future of harness engineering, adding more things to make the AI agents more predictable, right? Yeah, I mean, it's just like maybe recency bias, but I've been experimenting a lot with these sensors because I think they're still quite underused and there's also not a lot of tooling in that area. I just don't see the future as being like 50 markdown files in our code base. I mean, that can't be it, right? And then like I said before, in every markdown file, it's very very important. Do the following never do the following. I mean, that can't be it. Can we still call ourselves engineers? That's how we're doing stuff, I don't know. I mean, it will be part of that definitely, but yeah, so I'm like just like really interested in like this whole ecosystem and all these integrations into the base harness and how we can make that better and also balance it, right? It also cannot be just like throwing 100 tools at the agent and like 50 sensors that kind of overload it. Or we can think about, okay, if I have these sensors, do I need those guides anymore? How do we balance that? Yeah, and unfortunately, then we'll probably never have really, really good way to measure what's better and what's worse. So it will be about the feeling again, right? But in short, like I see a lot of tooling potential there still and a lot of things we still have to figure out about how to balance this. Like this whole idea of regulation, I really like it. Like a system that you try to regulate, that it doesn't get too much or too little or, you know, so I really like that image to think about harnesses. And it's also good way to test the models which are coming out in the market as well, right? If you have a good harness. If we know how to determine that it is good, then it's okay. Okay, okay, yeah, you're right. I mean, in the sense that it's all those moving pieces, right? Right. It's all of these moving pieces. The models change, the base harnesses change. I mean, they're all like churning out releases like crazy, right? And then our own harness changes, right? So there are so many moving pieces on the board. I mean, the ultimate measures we have are our software delivery performance, right? What is stability, what is throughput? And then the ultimate ultimate measure is user value, right? Like, is this actually what we're building? Are we just building more crap faster? Or is it actually bringing the user's value? Is it bringing us revenue? Is it bringing profit, right? I think building something of value is what is most important. And I think using all the tools we can to get to that state is what we should aim for here. Thank you. This has been a very great conversation. Where can people find you in cyberspace? Pre-github, is it? LinkedIn or email. Yeah, most of my writing I do on my colleague Martin Fowler's website. There's a series there exploring Genai and also some articles. I have a personal website, begeta.info, where I just always list all the podcasts, the YouTube conference talks, writing, it's just a list of content. Okay, great. I'll add that to our show notes. Thank you for coming on this show. Yeah, thanks for the conversation, Priyanka. This is Priyanka Rajman for Software Engineering Radio. Thanks for listening. Thanks for listening to S.E. Radio, an educational program brought to you by IEEE Software Magazine. For more about the podcasts, including other episodes, visit our website at se-radio.net. To provide feedback, you can comment on each episode on the website or reach us on LinkedIn, Facebook, Twitter, or through our Slack channel at seradio.slack.com. You can also email us at [email protected]. This and all other episodes of S.E. Radio's license under Creative Commons license 2.5. Thanks for listening.