This is Software Engineering Radio, the podcast for professional developers on the web at sc-radio.net. SCRadio is brought to you by the IEEE Computer Society by IEEE Software Magazine, online at computer.org/software. Hi everyone, welcome to today's episode of Software Engineering Radio. Our guest is Mark Brooker, weepy and distinguished engineer at AWS, and our topic is Back Driven Development. If you try to use AI to write code, you probably noticed how productive web coding can be, but also that thing start to break pretty soon. Spec Driven Development aims to correct that, so rather than treating prompts as a main artifact, it treats specs and intent as a source of truth, and uses them to drive the code, the tests, and reviews. Welcome to the show today, Mark, so great to have you here. Is there anything you'd like to add either to the topic or your bio before we get started? Yeah, well thank you, thanks for this opportunity to talk about this topic. I've been building software in various ways for the best part of 30 years, maybe slightly over 30 years, and this is so exciting to have this new set of tools in my toolbox as a software engineer and as somebody who's really interested in building software and more broadly building systems to serve customers with software. So you said you've been doing this developing software in different form or fashion for a long time. Could you help us by summarizing the current SDLT best practices? Yeah, well I think obviously that varies from organization to organization and team to team, but the way that I see that is when I think about a high performing software team. This is a team that is iterating quickly. They're very close to the customer in gathering requirements, they're very close to the business and gathering requirements. They're working to get feedback from the business and from the customer about what they're building as quickly as possible. Often they are operationally close to running the things that they are building, especially their cloud services or software as a service, and they are working closely across multiple disciplines between software teams, designers, product folks, customer facing folks, and so on. We can talk about getting to the more kind of software development part of that, but for me it is really about structuring teams and organizations to be able to iterate quickly and be close to their customers and close to their businesses that lead to good outcomes for software development. So you mentioned a lot about iteration and feedback loops and structuring teams, so so far a lot of teams have been aligned with the scrum or similar form of iterative dev process. As we start to introduce AI into the mix, what are the things that you have been observing? Probably the biggest thing that's changing is the multiplier of overhead of process goes up. It becomes so much more of a big impact. So if I think about, hey I used to be spending 70% of my time building software, writing code and testing code, writing designs, and so on, and 30% of time on overhead, if the speed of the software building portion has gone up by 2x or 4x, it's a 2x, right? Now I've suddenly shifted the ratio from 70/30 to something that's closer to 50/50. And so as that software portion speeds up or the code building portion speeds up, so we also have to figure out how to speed up the rest of the software development process or we don't get a ton of the benefit. So that's been this kind of second order effect I've seen with teams as they have embraced AI and embraced the faster speed of coding that have gone with that, is they've also had to pay a bunch of attention to how do we build better processes? How do we make those processes lighter weight? How do we use AI to accelerate a lot of those processes? And that's something that there is a ton of conversation around now that I'm excited about. Then I think the other part of that is another big enabler in taking advantage of the speed of being able to build with AI is having great infrastructure for testing and for knowing that that code works. And so the teams that I've seen get the best benefit from AI are the teams who have made investments in testing, infrastructure, in testing, in knowing what their software should do and therefore can move really fast and move really fast with confidence. You spent a lot of time in focusing on all the things that surround the actual writing of the code. Though a lot of the hype in the last year has been just on the coding with wipecoding being a term that nobody could not have heard of. How did you make that journey or how did you observe developers making the journey from wipecoding to now understanding that's a small part or not the whole part of the whole experience. I think that's something of a natural journey and so I start off by coding and feels great right like it feels so productive and I get so much done and I generate so much code and then I realize what I've actually got to get this out into production you know if I want to have an impact on my business and my customers I've got to get it out into production. Now I need to be able to test it now I need to be able to validate it now I need to actually be able to know that those features that I built are the right features for the customer. So it's this natural thing that happens to teams of saying hey we're going to pick up these vibe coding tools it's going to feel great we're going to feel super productive and then this sort of and then happens and it's like and then while I need to maintain this piece of software I need to test this piece of software I need to listen to customers and feedback in what they're saying and improve the piece of software. And that's where it becomes really obvious that vibe coding as great as it feels and as useful as tools it can be isn't everything and doesn't solve the end to end software lifecycle it solves a small part of that or accelerates a small part of that and even if you accelerate that substantially you haven't yet accelerated the whole end to end process of what it takes to build a successful software system. But if I'm building something for my own use little hobby project that's okay right it's fine and I get a huge amount of the whole into end lifecycle down just with vibe coding if I'm building a piece of a critical service or an application that I need to put in front of my customers and meet at the safety bar and a security bar and a quality bar. Well then I need something more and even more so over the long term if I need to maintain that software for weeks months years decades then I need a more it's a sustainable process than the one that does kind of pure vibe coding approach offers me. You mentioned maintaining a couple of times and that's key but what about even reading the code before that how does that work how do core reviews work. Yeah I think this is one of the key questions that's being asked in the industry right now is where do we go with code reading where do we go with code reviews code review from my perspective has historically been this incredibly useful tool for driving software quality for driving alignment for driving API quality I'm not super optimistic about it being a way of preventing bags but I think it does drive up quality it does improve code design and has these good outcomes but as we accelerate or what is the role of that kind of code reading and I suspect where we're going to end up with for a lot of code bases is not all but a lot is code reading is going to go a little bit of the same way that reading assembly code did 50 years ago we're going to stop building in these high level languages and people are going to say well if you're going to build in a high level language you have to read the compiler output and check it's right. And people did that right they did that a long time ago and then over time we stopped doing that because it just didn't feel like a good use of our time and we felt that reviewing the high level language was much more useful and much more productive and led to better outcome and I think that's more or less where we're going to end up with AI powered software building is where we're going to look at a lot of the code that gets generated almost like it is the assembly code right it's is an implementation detail and we want to be reviewing in page. Reviewing and paying attention to is the specification right is the writing of what should the software do and how are we checking it does that that key in terms of spec driven development is how are we checking what the software is supposed to do and how do we provide that in the first place is that the key goal like can you maybe take a step back now and explain what is spec driven development from your perspective. Yeah good to pick a step back on that for sure I think you know as long as we've been building software as an organizational thing right as a team sport part of that has been figuring out what should the software do and writing that down and sometimes we've written that down as formal documents sometimes we written that down as a series of tickets sometimes we've written that down as a bunch of napkin sketches and some stuff on a whiteboard and all of those things are specifications. And I think in this new era of AI what we're finding is that writing down what your software can do and we're going to call it this word specification that implies a level of formality that doesn't necessarily exist.
that is a description that is as crisp as it needs to be about what the software can do and what a successful piece of software looks like. And then once we have that artifact, then we can start using that to do two things. One of them is build the software, often with an AI assisted process. And if you look at the specification driven development flow in something like Kiro, this is an agentic process that takes that specification and starts building it with code. Then the other thing that we can do with that specification is build tests. And this is where techniques like property-based testing come in super powerful. I can extract properties from that specification and build tests that assert that those properties are true about the piece of code that I've built. But crucially, that specification isn't a static artifact. As I build the code, as I test the code, as I talk to customers, as I talk to my business, as I listen to the various stakeholders, I'm going to keep improving that specification. I'm going to clarify it. Maybe I'm going to generalize it in some ways. Maybe I'm going to make some pieces of it much more specific. Maybe I'm going to add in some code snippets or UI mocks and improve that specification. And then that is going to feed into improving the code and improving the tests. And so this is a iterative software development approach, a short-term iterative software development approach. But one that is very explicit about writing down the goals of what it is that we're trying to build. And then really using AI tools to accelerate the building of that. So you mentioned Kira. Can you just talk to our audience about what Kira is? Yeah. So Kira is essentially three things. It's an IDE. It's a CLI, a command line interface. And most importantly, it's a set of agents that we've built here at Amazon. That we use for our internal development. And we can talk about some of our successes with that. And is available to customers to it's a set of tools for agenteic development. It supports the vibe coding mode. It supports the CLI based step-by-step mode. But most importantly, in my mind, it supports this specification driven development approach. Where alongside the Kira agent, inside the Kira IDE, you develop a specification of the software. Then you use that specification to develop the software. Use that specification to build the tests powered by those Kira agents. And then that feeds back. Then you go out to the world. You show the world your software. And then you feed that back into improving and sharpening that specification. There are a lot of tools out there today. So I'm just curious what led you to build a new one. I think we had two pieces of vision there. One of them was when we started this journey of building Kira and building the Kira agents. The world was starting to move on from the kind of using AI a smart auto-complete, which was a nice interim step to this vibe coding mode of prompt-by-prom development. And we had used that internally at Amazon and at AWS. And what we were finding with that prompt-by-prom development is that it felt really good early on. But it wasn't scaling super well to more complex code bases. And most importantly, it wasn't scaling well over time. Right? Like we would build software prompt-by-prompt. And we would write a prompt to add a feature. And it would go and undo a bunch of the stuff that it did. It sort of forgotten requirements from earlier on. And that got a bunch of us in different ways thinking about. What if we could solve that requirement over time problem? And that's where we ended up with specification-driven development. To say, "Hey, let's write down all of these requirements. Let's keep all of these requirements handy." So as we iterate on a piece of software, we can refer back to them and say, "Does this piece of software still meet these other things that we were doing?" Right? It's like having a map rather than having turned-by-turn directions. So building Ciro allowed us to really experiment with and then really invest in the specification-driven development. The other thing, the other investment that we've been making, is in code reasoning and neuro-symbolic AI, where we use symbolic tools, solvers, and so on, these more formal side of AI to reason about code. And we wanted to bring the power of those tools to software developers in an easier way too. And by building those tools into Ciro where they can do things like power some of our work on property-based testing, we can bring the power of code reasoning tools and symbolic reasoning tools to programmers in a way that it doesn't require them to be experts in formal methods or automated reasoning and so on. Could you take an example and walk us through the steps of how somebody would use Ciro and I'm really curious about an example of a property-based test that you mentioned a couple of times to. Yeah, let's pick an example. One of our recent releases, maybe not that recent anymore, we built a Rora DC-Qual, right? This is a great scalable, multi-region, active-active-sequel database. And what the team realized that customers needed was a set of drivers to make that database easier to use from within Java. So we could write a specification that said, "Here's the driver that we need to build. Here are the properties that it needs to have. Here are the ways that customers should interact with it, sort of an API sketch, and then use that property that flow to generate the implementation of that driver. And then when we think about a property-based test, a property-based test would take out something from the specification. Like, every connection attempt contains an authorization token. And then build a test, the test in an automated way, lots and lots and lots of permutations of using the driver's API, and make sure that that property of every attempt to connect contains an authorization token is always true, no matter what permutations of the API. So either I put things in the API and I get a valid request out, or I put things in the API and I get a helpful, failure or helpful error message out. And so that's quite it in the small, relatively modest-sized project, but a very useful one. Another bigger project is that we have been building a new, fundamental piece of our inference infrastructure, also using Kiro, also using specification-driven development. And that's more of the kind of cloud service type rather than end driver. And so there the specification is about, this is a cloud service, it does these things, has this architecture, it has this API, but it's written in Rust, and so we want to use all safe code, for example, and then generate the code from that, and then we get these specifications or properties, again, like if a customer calls this API, either it passes on the request in this form, or routes the request in this form, or it gives the customer a useful 400 or 500 error message. And what the property-based test will do and what makes it so interesting, is it isn't going to test five or six cases like a human developer might. It is in an automated way going to go off and test hundreds, thousands, millions of different API permutations and make sure that property holds in all of them. Can you maybe walk us through the stages potentially for, based on some of the things you said, I potentially may not be applicable to all projects, but when I think of this SDLC and from some of the pointers you had, the spec is the first thing, but could you use AI to refine the spec is the question, then what about the architecture, documentation, the design documentation, the code Gen will follow that, and then talk about code review too. You did say the generated code, you could start thinking of it really as assembly and not really necessary to read it, but would you still potentially get a different model perhaps to validate the code by some form of code review? You mentioned a lot about test, but maybe just walk us through all the stages for a complex project and what would you then not necessarily do for a simpler project? Well, I think step one for every project is figuring out what problem you're trying to solve and for who, right, like who is the customer, what do they need, what are the requirements of the business? And AI helps there, it can help me pull in a bunch of requirements, it can help me write documents, it can help me figure out what questions I need to ask different stakeholders but doesn't fundamentally change that step of the software lifecycle. And that's going to vary, right? If I'm writing a super important piece of code for a new AWS service, I'm going to go wide with that, I'm going to talk to a bunch of people, I'm going to really understand the economics of that. If I'm just hacking on a software project, that's something I'm going to do in my head in a couple of minutes. And then we have the step of writing down the initial version, let me say, of the specification. And then yeah, I mean, I can do that manually, or I can have the AI help me with that. That can be, I start off with developer specification based on these prompts, based on these customer use cases, based on these customer anecdotes. And then I can read that specification, say, well, no, that's not quite what I meant. Let's iterate on that together. And so that can be anything from me writing it down all at once to kind of vibe specifying, right? Like, hey, I'm going to go piece by piece, I'm going to go anecdote by anecdote and check that those meet in the specification. But what's important is that I'm reading that specification, I'm thinking about it, is it the right thing, and it's only an initial specification. So we're going to close the loop on that in a second. Then again, working with the
the AI, I am going to turn that specification into a set of design decisions, right? Like is this a web service? Is this a microservice? Is this a library? Is it a UI application? Is it a web application, right? Those are important design choices that, again, are going to be driven by the needs of the business, needs of the problem I'm trying to solve. They aren't kinds of decisions that the AI can make for me. They're decisions I'm making based on my understanding of the problem I'm trying to solve. Then we're going to make a bunch of implementation choices. I'm going to choose a programming language. Maybe I'm going to choose some frameworks. Maybe I'm going to choose some libraries, right? If I'm building a web app, I'm probably going to use TypeScript. If I'm building a new component for an AWS service, I'm probably going to use REST. If I'm building a library for people to consume in Python, I'm probably going to use Python. If I'm doing some AI science and data analysis, maybe I'm going to use Python again. What frameworks do I want? Do I want to use internal or external frameworks? I want to use open source things. Then I've got my specification. I've got this high level description of my software design. Then I've got this low level description of these implementation choices. Then I'm going to step into, "Okay, let me go and generate the code, the first versions of the code and the first versions of the tests." I'm going to do that working with the agent step by step, stepping through the specification and at each step, making sure that I'm moving forward and implementing more and more of this specification as it comes. Then we get to the crucial part where I've generated the code. It's passing all of the tests. Now, I'm going to see does it actually meet the needs of the customer or business? Does it actually do the thing that I'm building the software system to do? Maybe it does initially, but then people say, "Well, it'll be really cool if you had a feature to do this thing or it'll be really cool if this button was over there or it'll be really cool if you redesigned this to be clearer in some way." Then I loop back. I loop back to that specification. I loop back to those design choices and I iteratively change them. Then we rebuild the code alongside the agent. We rebuild the tests alongside the agent and then run the software in production again. It remains this iterative process, but it's an iterative process where the core goal of the iteration is keeping a specification up to date and downstream of that is keeping code and tests and so on up to date. Another benefit from what he said is it gives you an ability to check along the way rather than just handing off the spec to the AI and then getting the final code generated. You'll actually have a checkpoint where you read the design doc that is generated, make sure it's correct or any details of the spec itself and make sure it's correct and so on. Is that right? Yeah, that's exactly right. How long you let day I run probably varies from application to application and goal to goal, but it does give you this checkpoint of checking in of, "Am I headed in the right direction? Am I driving to the right place?" So you get into an autonomous taxi and you say, "I want to go to this place and you watch out the window and see, "Oh, are we actually going there?" Or maybe I've changed my opinion and I want to go somewhere else. But yeah, and it does give you these interim steps of, "Okay, before I go off and generate all this code and spend all of this time and spend all of these tokens, am I actually building something that is going to solve my customer's problems?" A question about the models that you use. Is there any benefit that you observe from maybe using a different model for some of those stages, perhaps for code review? Yeah, so that's really interesting. So actually we need to talk about the code review question. I think tentatively, yes, right? It certainly is useful to have at this moment with the technology where we are and things are moving so quickly that this might not be true by the end of the week. But from what I've seen at this moment, it's really useful to have, it's a separate agent doing things like code review rather than letting the code build a self-supervised. And so you want this agent that's going to look at the code as an artifact and say, "Is it meeting my goals? Is it well designed? Does it have good internal APIs?" All of these things that we think add up to good code maintainability over time. So a different agent, differently prompted an agent with different goals. And that's what's important with agents. Is it helpful to use a different model for that? I think tentatively, yes, again, I think we've seen some good indications that doing that code review with models of different sizes or models of different capabilities is useful. But I wouldn't say that anywhere near is useful as the sort of fundamental multi-agent architecture of saying, "Hey, I want my code review to be done by an agent whose goal it is to drive for great code quality rather than the agent whose goal it is to implement the specification and build the code initially." Earlier, we talked about how you came up with this methodology. Do you happen to have a story where you can compare and contrast projects that were built, wipe-coded versus falling a spectraven methodology? I don't know if we've done any kind of side-by-side between vibe coding and spectraven development. But what we have found is let me say that Amazon has a very, let's say, service-orientated architecture, kind of microservice architecture. And we have literally thousands of services within the company that provide certain things. And as we've adopted AI, what we've heard from the teams that own those services, they really started with this kind of vibe coding and what they were finding was they would pull their code base into an IDE, they would prompt for a change. Often they would see that change successfully implemented, but they would see also a regression. At the same time, the AI would kind of forget one of the things properties about the code and be like, "Well, I've optimized this, but that breaks the API and makes the service not work anymore." And so what we're seeing internally at Amazon is going through the process of how I've got the service code, I'm going to extract from that a specification, even if it is a kind of method-by-method code-to-text process. And then I'm going to use that specification in a spectraven development flow, makes that agent that is building the next feature much better at not undoing goodness at the past. And a separate agent that's building tests, much better at thinking about which tests should it build. And so I don't think we've done a lot of side-by-side, but I think what we have found as teams have embraced specification-driven development, that specification abstraction is a step that helps them move a lot faster because it stops this two-step forward one step back dynamic that can happen with existing code bases with vibe coding. I think that's a good point and we'll come back to that. I'll just talk a little bit more about providing context, but before that I'm curious. So spectraven is a methodology, Amazon-built Kiro, but I'm sure spectraven could be applied to other agent tech IDEs, is that a fair assumption? Oh yeah, absolutely. It's a methodology that you can use with any of the popular tools, but it's not as ergonomic as we've tried to make it in Kiro, but it's a methodology with general goodness. So when we recommend to our teams who are building software, whatever tools they use. Thanks, Mike. Let's go back to providing the context. I think you brought up a good point is that unless the spec itself or what you told the LLM had the context of the code, they were always regressions. Can you generalize that? Is it then true that vibe coding is more effective for new projects or for open source projects? I don't know about open source, but I think we have seen at least anecdotally that if you build a code base with AI from the beginning, the AI finds that code easier to understand. Why that is? I don't really know, but anecdotally that is true from my experience and similar to what I hear from my colleagues. I'll also say that for me, the easy and fan feeling of this kind of pure vibe coding works best in the kind of green field new project situation, but as soon as I have a significant existing code base that I'm working on, then I don't find I get as much of a benefit from that kind of interaction. And so yeah, I would say that that vibe coding flow is just maximally awesome and productive for getting from nothing to a great first prototype, but after that it is less scalable, it kind of peter's out, right? Whereas the specification flow loads a little bit more work up front and so there's no hiding that, but what you get out of that is a much more sustainable ongoing process and ongoing process that is much easier to maintain much easier to iterate on, much easier to add features to without regressions, much easier to get great test coverage, much easier to get great QA coverage. And so by loading a little bit more work, not a huge amount more work, but a little bit more work at the beginning, what you end up with is a process that doesn't drop off, right? A process that remains effective and productive over the long term. So in terms of context, you talked about extracting the context and having that provided as part of the spec, but is that sufficient? I mean, the codebase could be pretty large, you certainly could do that all the time. What other ways are there of providing context? I really like that question and I think
for two reasons. Like one of them is, I think in a lot of ways, 2026 is the year of thinking about context management for AI agents that really has become one of the most important problems in the world of agent building. So this is something that everyone across the industry is going to be giving a bunch of thought to and a bunch of investment in this year. But then if we think about, okay, how do we think about that in this particular case of software building? And what's really interesting to me is so many of the techniques that we've developed to make software and code bases understandable to humans also work really well in this context, right? Like we have always, you know, from the early days of kind of structured programming, kind of going all the way back to, you know, 60 years ago, thought about how do we build interfaces? How do we build modularity? How do we extract functionality into libraries? How do we extract functionality behind APIs? You know, whether that's a library API or a local service API or remote microservice API? You know, we have made software modular. We've built APIs. We've built contracts. We've built protocols, right? These are all ways to take a extremely complex piece of software, limit the interactions and allow access humans to reason about the way that that, you know, piece of software works. And in a lot of ways, software makes the difference between long term successful software and unsuccessful long term software is their ability to be built in a way that they can evolve without every change being able to change everything about the way they work. And that's been what is driven the popularity of service orientated microservice architecture was driven the popularity of libraries and modularization for the whole of the history of software. And so we can kind of think about that. Well, in AI, also what we need to manage the context and help in AI understand our code base is modularity is good design is good APIs is good documentation of the APIs and their contracts and and what they mean and what you can assume and what you can depend on. And then once you've done that, then you can layer on top things like semantic search, right? I'm looking for the library method that does this. Well, here it is. And so I don't have to understand everything about the entire system. I can say this library call takes these inputs and has this effect on the system. And so I can reason, you know, more locally about the behavior of the system. And so for me, kind of context management for software agents is very much like context management for human developers is converting the impossible problem at scale of reasoning about every line of code in the system and their interactions into this more local reasoning thing of like, can I think about this one library call this one change and the more that we can design our software to make changes as local as possible, the less context you need to successfully make changes, the more you can well test changes and the more productive you can be. And I think that's true of of AI agents just as it has been true of humans over, you know, six plus decades of of software engineering practice. So you mentioned one way of, I guess two ways, one was extracting context into the spec and then semantic search where you potentially have to create vectors for a bunch of code in your code base. What else is there that you've experimented with? Do you have any guidance on somebody that are getting started now in this area? What they should do? Yeah, I mean, I think there's some really interesting work going on around indexing code bases and around code understanding and, you know, extracting understanding from code bases. But what I would encourage people to do who are starting out, you know, getting into this kind of coding is to, you know, think about software design and think about internal interfaces, you know, think about types, think about APIs and think about building your software in a way that well encapsulates the functionality of each component. And that makes it much easier when you say, you know, to the AI or you, you change the specification to find the lines of code that are going to do things, right? Like it's, you know, hey, UI dot, you know, TS or whatever, rather than, hey, this is just spread out through my code base. I can go and find the lines of code. I can go and change the lines of code and they are as local as possible. And so, you know, one of the things to maybe optimize for and pay attention to as you're getting started with this is you look at, I'm going to make a change to my specification, maybe to add a small feature. How many of the modules of my system does the AI need to touch to make that to make that happen? And if it has to touch every module in my system, well, then I've got a design problem that is going to make my AI driven development less productive over time. You know, if it can make a, a small feature change by touching, you know, one or two of the modules in my system, one or two of the libraries, one or two of the services, however, I factored things. Well, then I've got a design that is likely to work over the long term and scale up well without, you know, blowing up context windows without requiring, you know, sophisticated techniques to index and so on. And so, if I'm seeing that number go up that kind of complete or number of modules I have to to change to build a feature or to add some capability go up over time, maybe that's time to step back and then work with the AI to do some refactoring to kind of tease apart modules to tease apart APIs. And the good news is that, you know, AI is, and, you know, these agents, and especially, you know, once you've invested in great testing, they're extremely powerful at doing, you know, refactoring tasks. Let's talk about your comment on blowing up context windows. When does that happen? And the related question is when is there too much context, which actually harms outcomes? You know, this varies model to model and it has become less of a problem as we've had, you know, some slightly better, you know, better model capabilities, but it is still a real problem, right? And so, you know, if I go into a coffee shop and say, what would you like to drink? And I say, well, I would like a latte. And by the way, it's also 18 degrees outside and the sea hawks went yesterday and, you know, not only is that going to waste a bunch of time, but it's going to confuse the person who, you know, it's like, why is this relevant? Why are you telling me all of these things? Is there some way that you want me to customize your drink based on all of this extra information? And so it confuses humans. And I think, you know, almost by analogy, you know, it's a bit bit risky to reason by analogy with AI, but almost by analogy, putting a bunch of irrelevant stuff into a context window makes those AI outcomes worse. And so, you know, this is where, you know, context management becomes so important. And there are two aspects of it. One of them is all of that design and modularity stuff that I, I just talked about. And the other is the AI tool, whether that's Kiro or one of the many other AI-powered development tools, have to do, you know, have to manage what's in the context window, making sure that there is enough context to give the AI the power to at least discover what it should be doing. But there shouldn't be too much context. If it contains a bunch of irrelevant noise, we're going to have a worse outcome. You know, and so if you think about, like, what do agents do? Well, one of the things that AI agents do is build their own context window through a process of discovery. They're going to read that file, search this index, search, you know, use this MCP tool to do a semantic search over the documentation and try and build their own context window. And again, one of the things that makes agent development challenging is making sure they build that context window as reliably as possible, containing as higher ratio of relevant to irrelevant information as possible. And so there are kinds of three things there. It is, how do I build my code base and maintain my code base to make it kind of context friendly for humans and agents? How do I, as somebody who's building development tools, think about, you know, context management? And then how do I, as a developer of agents more generally, think about guiding those agents to build their context windows effectively? And then, you know, the tools I give to the agents, what do they return? Do they return the right stuff? Do they give the agent the ability to request only the facts that needs and so on? What about the process to make sure that the way you provide the context, the source of that is if it's not the code directly is kept up to date? Is that something outside of the dev process or it's kind of inbuilt there? Yeah, you know, and this has been again, not a new problem in the world of software. How do I keep my documentation up to date? And, you know, one of the real challenges of the kind of pre-AI era is how do I keep my requirements and specification up to date? And specification driven development has made that much easier by making, you know, everything flow down from the specification and, you know, has helped with that specification, freshness problem, a huge amount, but there's still a documentation, freshness problem. Well, how do I keep my documentation up to date. What I did in
And the previous AI era is, I would spend time from my development tool or technical writers who would look for changes in the code and go off an update documentation. Now I can fatigues very effectively accelerate that process with AI and have an agent in my kind of set of agents that is doing helping me with my software development process that is going off and keeping my documentation up today, keeping my read me up today, keeping those API docs up to date. And that feeds back into the next round of my process where I want to make a change. How do I know that that API doc is up to date because I have continuously kept it up to date as part of my software development flow. And so even when I'm vibe coding, even when I'm not doing the kind of specification development process, I always make sure that my prompts or my steering documents are directing the agents to keep a read me up today to keep a piece of documentation up to date, both for my reference as a human and my customers reference as they use my software, but also for the reference of the agent itself as it makes the next set at steps. - Thanks, Mark. I'd now like to spend a little bit of time on, how do you make sure that the configuration and the model itself is effective? Initially, you roll this out to the team and now the core group that's maintaining the config, the contacts, the different workflows that somebody has to do, how do they keep them up to date? So let's say there's a new model that comes along. How do you make sure that it hasn't regressed the flow in any form or fashion? - Yeah, and so I think the formal answer to that is, we have an evaluation process and that's where we will take an agentic workflow and probably starting off offline, run a bunch of examples through a new model or a new configuration or a new prompt and compare the success of the old versus the new. And that could be with a human looking at that, that could be looking at other tests passing, it could be looking at acceptance rate or it could be using a pattern like LLM as a judge where it uses another agent essentially to rate the agent outputs and say these ones are better and these ones are worse. The next step of that and the one that I think is actually most powerful is online evaluation where okay, I'm going to start experimenting with this new model in production, kind of A, B testing. I'm going to send it 5% to traffic. I'm going to send it 10% to traffic and then I'm going to feed back all of my success signals whether those are LLM as a judge, whether that's human feedback, whether that's latency information, test success rate, to look at am I doing better with this new version? If it looks like I'm doing better, if those initial experiments are looking good, while I'm going to ramp that up, eventually I'm going to get that up to 100% and then I'm going to be ready to do the next experiment. And then continuously in production, I'm going to continuously be evaluating the outcomes of my agent again. The metrics I use for that depend on the task and I'm going to know over time is my success rate good. And if my success rate suddenly changes for whatever reason or trends downwards, well then I need to understand why that is. Maybe my tasks have changed, maybe my customers needs have changed. Maybe I have some bad piece of context. Maybe that read me contains some false information that's leading my agent's off in the wrong direction. But the really key thing here is to have from the agent builders perspective, and here I'm not talking about the IDE user. They don't need to really worry about this stuff, but me as an agent builder, I need to think about how do I have a robust set of evaluations so I can run a really data-driven improvement process around agents, around the models, around the prompts, around the context, around the way the context is managed. And the more I do that, well the better a quality product I can offer to my customers, the more reliable it will be. But it also helps me do things like, hey I'm doing something with a really big model today. Can I lower my latency by using a smaller model? Answering that question in isolation? Man, that's next to impossible. Answering that question when I have a robust set of evaluators is much easier. And so this is a process for agent builders, but it really goes back to the beginning of the whole question of what are we trying to achieve? What is the outcome that we want? What are our customers need and what are our business needs? What is the outcome and then what is the success rate? But maybe just specifically what kind of things should team be measuring? Should they be measuring bug rates? Is there human correction needed during human code reviews? Do should they be measuring the divergence of the core from this back? Could you share some of those thoughts on what metrics team should focus on? Again, like the most important ones are the end-to-end ones. Our customers seeing bugs in production. Our customers seeing regressions in production are, our customers happy with the in-production performance. Now, those are the most important metrics. They're also the hardest to measure. They're also the ones with the highest risk because we want to catch things before customers are reporting bugs in production. We're going to choose some earlier in the process kind of proxies for those metrics. That's where am I seeing failures in regression tests? For example, what is the test rate of failures in regression tests? Am I seeing a. If I'm using human code review, am I seeing a good pass rate for that human code review? If I'm using a separate code review agent, am I seeing a good pass rate for that code review agent? Is it giving me good success? If I'm using a pen-testing agent like the one that we announced at ReInvent that does last mile security testing, is that finding security bugs before things go into production? If I'm seeing bugs being caught at my last pre-production milestone, I know something's not going well earlier on in my software development process. And that could be an inadequate kind of early in the software lifecycle testing. It could be a symptom of bad design and interfaces. It could maybe be a symptom of some bad implementation choices I need to go and fix. Or it could be a misalignment between the requirements of the tests that are going into the test and the requirements that are going into the software building process. And so if one of these things says you need to return a result to the customer within 10 milliseconds and the other one says 10 seconds, well obviously I'm going to have failures. And so there's this tension in these metrics of the most valuable metrics, all the ones all the way at the end of my software development lifecycle because they are what I really care about, they're what my customers see. But that's the most expensive place. And so I want to push as much of that catching of things in those metrics sort of left in my pipeline as possible. Again, classic idea of software engineering, right? Catch bugs early. You know, we've been software engineers been saying that for four or five decades and that hasn't changed. Thanks Mark. Now let's talk more about how do you roll this out to the team? Are there any prerequisites? Is there a mine shift that has to happen? Or do you just have a meeting? You introduce the tool and tell everyone, hey, go use it. Yeah, I mean, I think that varies. I think the biggest prerequisite is almost cultural, right? So what we've seen with adoption of new development practices is you need a team that has great understanding of their customer and business and high standards, right? You can't accept, I don't know, slop, I guess, is the term that people use these days, right? We don't want that in our software processes. But we also need folks and see the foster transformation with teams that have great curiosity who are really interested in picking up new things and great bias for action of like, hey, we're going to go off and try stuff and experiment with stuff. And so it's that tension, I think, within teams that I've seen really drive success here. Is the teams are like, well, we're going to go off and we're going to try new things and we're really interested in learning and trying and adopting new tools. But at the same time, we're not going to settle for bad outcomes, for our customers, for our business, for our engineering teams. And when you put those two things together, I think that's really where magic happens. Beyond that kind of cultural goodness, there is a, we also shouldn't expect that teams can pick up a new tool and know immediately how to use them well, right? You need this kind of communication and sharing of saying, hey, we've tried it this way, these are the things we learned. These are the things that we want to pass on to you. You want to bake those into communication that you have between your teams. And then if you're building tools, you want to bake those lessons into your tools. And so again, I think another piece of success is making sure that as a team learns how to use the tools better or to improve the tools, the whole organization learns. And ideally, the whole world learns and we can kind of drive everything forward. And sometimes that's big learnings of saying, hey, we want to introduce specification driven to the world.
development because we've learned that that scales better than prompt by prompt vibe coding. And sometimes it's little things of saying, hey, we found that indexing our code this way or structuring our code this way works really well. We found, for example, concretely internally at Amazon that building AI using AI-powered development with a language like RAST with a strong type system and a bancher compile time checking is really, really useful and helps accelerate development by catching bugs earlier in the pipeline. And so that's a message that we want to send out our teams and say, you know, RAST works well, TypeScript works pretty well. You know, the type annotations in Python, they're worth investing in. And you know, so it is that spreading of learning across an organization that is super important. And then maybe the last thought on that is, wow, you know, no surprise to anybody. This whole world of AI is moving so fast, right? There are new techniques, there are new tools, they're pretty seemingly every day and every week. One of the things that you need to balance for success is, you know, you want your teams to be paying attention to those new tools to be curious about them. But you also kind of forward to go and chase, you know, every week's trend day to day. And so there's a balance to be found there if we want to adopt new stuff, we want to adopt the best things quickly, but we don't want to spend all of our time just threshing on trying the latest, you know, the latest and greatest or almost fashionable tool of the moment. So yeah, there's a lot to put together into organizational success. So certainly, of course, curiosity is critical, but bias for action seems to be more important based on your comment. So with this specifically with AI specifically, there's a little bit of fear as well. How do you address that? What kind of fear can you say more about the fears? I mean, with AI doing more and more is different things expected of the developer or less expected of the developer as we proceed on this journey. Well, you know, I think that the job of software development and, you know, more broadly the job of software engineering is changing and our jobs are going to be very different in, you know, two, five, ten years from the jobs that they are today. And, you know, with any major change, there is going to come uncertainty. People are going to think about, you know, what does this change mean for me and for the reasons that I went into this field and the things that I love about this kind of were. And, you know, I'm super optimistic about that. I think AI-powered development is freeing us up to spend more time on the things that matter. And, you know, whether those things that matter are, you know, being closer to customers, being closer to the business, really understanding the big picture of the systems we're building, or, you know, getting really deep on the, you know, various parts of the design, getting really deep on protocols, getting really deep on efficiency. And, you know, less time on the, sort of, undifferentiated, busy work of building. So yeah, I mean, it's a time of really fast change in the industry and I think that is always going to come with some level of anxiety. You know, I'm super optimistic that it is going to, you know, free us up to do a job that is more fulfilling and more valuable. And, you know, we've seen over decades, you know, how powerful a economic force software has been. And over decades and, you know, for the whole lifetime of software, it has been supply constrained. The economic impact of software has been constrained by how much great software we can build. And, you know, I strongly believe that if we can drive down the cost of software development with AI tools by 10X, there's going to be at least 10X economic impact of software if not 100X economic impact of software. And so, you know, I think software and software engineering is going to be even more important in the future than it is today. But it is going to be a different job. It's going to have, you know, different things and we're going to need to optimize in different ways. And I think it's very reasonable for people to feel challenged by that. I think it's very reasonable for people to reflect and say, "Hey, what does this mean for me?" You know, I can't say anything other than personally as someone who's been excited and passionate about technology, I think the future is just super exciting. Thanks, Mark. So, starting to wrap up now, a couple of questions. What about the steps beyond the development of the software? What about deployment, incident management, etc? Is that tied to anything that you're thinking of with Kiro or, you know, beyond that in AWS? So, you know, at Re and Venpack in December, we announced our DevOps agent, which is an agent that is designed for, you know, doing these DevOps things in the cloud, helping out with incidents, helping out with tickets, helping out with infrastructures of service, maintenance, and so on. So, yeah, another part of the work that software teams do, we announced our security agent, which, you know, is for helping out with that pen testing and other parts of the kind of security testing lifecycle. You know, we have AWS transform, which is a set of agents for, you know, taking, moving systems from one implementation to another, you know, whether that is small patching tasks, like, hey, I want to pick up the latest version of Java to major software, kind of re-architectures of now, when it get off a mainframe and get into the cloud. And so, you know, it is vastly beyond just the work of building software. We're going to use agents to accelerate the deployment of software, to simplify the operations of software, to simplify, you know, software maintenance, and that's a huge one, right? I don't know many software engineers who are excited about, you know, making sure I can migrate from Java 11 to Java 17 or 21 or whatever, the latest one is these days. But it's critical. It's critical work for security, for efficiency, for all of these, you know, other things. So, you know, that's going to be a big one. The ability to change back end implementations much more easily and with lower risk is huge for engineering teams, right? Like, hey, I have all of this legacy. I built these things. Maybe I wrote all this code in Fortran. It is the crown jewel of my organization, because it's the only expression of my business logic. You know, how do I bring that into a modern implementation? How do I bring that into a modern architecture? Again, you know, this is something that AI is really going to power for us. Testing, validation, UI testing. You know, there's so much of the process around software, you know, even updating tickets. All of these things, I think, you know, these are areas where we're investing in heavily in AWS, because, you know, we have paid a bunch of attention to where, you know, software developers both internally and in our customers are actually spending their time, software teams and organizations are spending their time. But they're not as shiny and cool as software building. And so, you know, all the buzz is around, you know, here's the thing that helps me write more code more quickly. And I think that's okay. But really the big picture is, here's something that allows me as a software organization, as a leader of a software organization, or to deliver value to my customers more quickly and more reliably. And that's going to take much more than just writing more code. But still, you are writing code as you're building these agents, you're writing code to build these agents. Is there special considerations for generating code for agents itself? Yeah. I think that is an area that is emerging. So if I look at the kind of first few generations of agent building, you know, very, you know, bunch of Python, bunch of kind of ad hoc techniques that's been, you know, very valuable. And over time, you know, what's emerging is, can we mix in tools like symbolic reasoning? Can we add more structure? You know, and so, for example, we built this really cool set of features into strands, which is our agent framework called strand steering that uses a neuro symbolic technique, combining kind of models and symbolic reasoning to nudge agent trajectories in the right direction. You know, we're seeing more tools that pose that optimization and evaluation loop with improving tool descriptions and so on. You know, we're seeing tools like the policy that we built into agent core, which allows you to take a kind of plain text description of a security constraint or an operational constraint, turn that into a crisp piece of authorization code in a approval language like cedar and apply those to all of their agents. And so I think what we're going to learn and are actively learning is how do we build software development practices and tools that go around agents that allow us to take advantage of the flexibility and power of agents, which is what we need, right? We want our agents to be flexible. We want them to almost be creative in their ability to solve problems, but we also need to do that in a way that is secure and meets the business needs and doesn't do things that are dangerous. And so policy helps balance that strand steering helps balance that. You know, we also need to be able to do that.
need to reduce cost. And that's where things like model customization come in. And so agent building is in this super exciting moment of a huge amount of both existing impact, but also a massive amount of innovation. And so I think what we're going to see is over the next few years, building reliable agents is going to become even easier and easier and building cost-effective agents with great ROI is going to be easier. And so that's been our focus at AWS. And it's certainly a focus across the industry of, how do we take agent building from being as fairly ad hoc, almost kind of bunch of spousals to being a real engineering discipline? Thanks, Mark. So we've done a lot of times. So I won't ask you about this, that you mentioned a few times, the Neurosynbolic reasoning, but I'd love to get some references to add to our show notes for readers. Yeah, we'd love to share. Is there anything that we miss today that you'd like to spend a few minutes on? I think the overall message for me, for software builders, is if we think about code, it encodes what it does, right? Like very, very crisply to computers. And we haven't historically been reliably in the habitat software builders of writing down why? Why is the code like this? And I think specification-driven development gets its power from making the why is the code like this explicit, both to humans and to models. And so I think that is going to be this huge transformation we see in software, is that the details of the how are going to maybe be less valuable, and the artifacts explaining the why. Why is the software designed this way? What is it trying to achieve? Who's it trying to achieve at four, is going to become more and more and more valuable? That is going to be true if you're doing specification development. I think it's true if you're doing vibe coding. And I think it's going to be true of whatever the next waves of agent-powered or AI-powered, or even human development practices look like. Thanks, Mark. How can people contact you and keep up with your work? Check out my blog that's brokodotseo.se. And I'm on LinkedIn and X and various other social platforms. I love to share the work that me and my team are doing here at AWS on those platforms. And always, you know, I love to hear thoughtful questions from folks and have great discussions about the future of software engineering. Thank you so much, Mark. This was a very interesting discussion and a very interesting topic. Great. Well, thank you. And thanks for the opportunity and fantastic conversation. [MUSIC PLAYING] Thanks for listening to SE Radio. An educational program brought to you by IEEE Software Magazine. For more about the podcast, including other episodes, visit our website at se-radio.net. To provide feedback, you can comment on each episode on the website or reach us on LinkedIn, Facebook, Twitter, or through our Slack channel at seradio.slack.com. You can also email us at
[email protected]. This and all other episodes of SE Radio is licensed under Creative Commons license 2.5. Thanks for listening. [MUSIC PLAYING]
PodcastTranscript AI