#753: Amazon Bedrock Mantle and Developing at the Speed of AI

This is episode 753 of the AWS podcast released on January 26, 2026. Hello, everyone. And welcome back to the Adabis podcast. Sunlishy with great to have you back on drone by a super special guest. I'm joined by Joe McGhermov, who is VP and distinguished engineer at Adabis. Joe, welcome so much to the podcast. Thanks. I'm really excited to be here today. It's amazing to have you here. You've been at Amazon for 20 years, not many times on me, Amazonians that have had more tend to them myself, but you well into there. You've seen some stuff and you've written so much code and lead teams and help teams write a lot of code that I'm sure many of our customers use each and every day, whether I know to or not. Just to give us some context, I mean, 20 years at Amazon, what are some of the things you've worked on as an engineer during that time? Oh, wow. It might be easy to say what I haven't worked on, but it's going to have interesting. My time at Amazon kind of has two halves. So I spend the first half, the first 10 years working on all the systems behind our Amazon.com retail website. So I worked on some of our shipping systems, some of our payment systems, a number of systems behind our marketplace. And about 10, 10 years ago, I transitioned to working in the cloud where I've worked, I've worked on our computer networking services. So I worked on services like VPC or load balancers, and that gateways and all other types of gateways would have been networking. And then I also worked special time with our container and serverless services. So ECS Lambda, you can ask. And now the last last year, I've been working on on bedrock and inference platform for Amazon. And we worked on project Vantil, which is something we released this year and something we haven't super excited about. Yeah, just got mentioned and called out in reinvent as well. I guess just to spend, although that subset of things you work on is your mention. I mean, I think that the characteristics there are there very much about scale, robustness and reliability. I mean, you're not working on stuff that, well, if it's an error, there's an error, it's okay. You know, the user will retry. This is like, you know, trillions of packets, real time processing, mission, like this is the stuff that keeps you up and on if you get it wrong. I'm guessing. That's right. Yeah. It's, you know, reliability and scale is at the heart of a lot where we do it on Amazon. Clearly scale makes a lot of things more interesting and more challenging. So you constantly have to think about not just only how do you get the scale, right? But also how do you get it right in the way that doesn't add too much complexity because complexity tends to be the end and the reliability. And so there's this constant tension behind engineering for scale versus engineering for reliability. And you have to get an navigate that tension navigate, getting things right, but not too complicated, not too complex. Yeah. It's, it's, it's one of those classic things if you know when you see it, but you know, it only takes a whole bunch of years of experience to figure that out. But also something else I want to call that that, at least as a currency, but I can see because I can see Joe on video here's behind Joe is, I don't know what the collective now would be, but I'm going to call it a watch of patent puzzle pieces. So that was only forget a patent. We get a puzzle piece, which is very exciting and great. And so it always could do that. And I've always been proud of my own patents. I've nine, my small number of nine. Joe does not have nine. Joe, how many patents are sitting behind the head? There's a lot. It's hard to say. It's about, it's a cube of about, let's see, uh, six by six by two. So I guess in around 72. Yeah. That's a lot. Proximate. It's impressive. Yeah. Yeah. The funny story at Amazon, uh, Procorks, you don't know at Amazon patents are, you get a puzzle piece, like an actual puzzle, a three, three dimensional, acrylic puzzle piece. And while I, you know, people have different opinions on software, positive software, uh, patents, what truly gets me is a love puzzle pieces. And so this whole concept of stacking things together and building out of them. Uh, and you hear a lot about it is I'm a dog right hearted. I'm a building star. And so the whole concept of stacking those pieces and building it is really what it feels to me. And that's, um, I, um, I'm afraid to admit that's, but by far the most fun part of getting a self-care out of it. I think you think the hardest possible way to get puzzle pieces to make a puzzle, but I love the dedication. I think it's fantastic. So we talked about this thing called mantle, which is part of bedrock. Just tell us, I understand, I guess what problem are we solving? What it's for? And then we're going to get into the guts of how you built it, which is, I think really a great unpacking of the current way of using I.R. for software development. But before we get to that, let's just talk about mantle and give it a cheer. Yeah. And so mantle mantle is a new inference engine underneath Amazon bedrock. So of course, starting bedrock is, uh, inference service for Amazon. It's publicly to be a service. Uh, and, um, it's a service that's seen tremendous girls, tremendous amount of, uh, uh, customer adoption. And, uh, um, we've learned a single two operating bedrock. So as we've operated it for a couple of years, we've learned how customers use it. We'll learn how inference behaves. We'll learn how, how inference scales. And so with mantle, it's our realization that, um, at the heart of its, um, inference is not quite a web service, but more of a scheduling system. And so request comes in and there's a lot of concerns you are typically seeing in schedule system, things like prioritization, things like fairness, things like placement, things like, uh, uh, uh, cold placement where you want to have multiple requests at the same time. And so there's a lot of those concerns come in. As if you look, if you step back and look at the whole ecosystem, uh, and if the bedrock you realize that it's, well, what it does is a giant scheduling system. It's our, um, it's an, and in an effort to accept that reality to make sure that the bedrock and off were the best customer experience. We've, uh, built, uh, this new inference engine underneath the bedrock. So if you're using, uh, bedrock today with models like minimax, a GBT OSS or a mistral, uh, mantle system is what's actually plastic and executing, uh, those requests. And of course, from the customers point of view, there, the benefits are two-fold. The first one is we, we want to offer customers the best possible customer experience. So the best latency, the best performance, um, the, the, the, the high number, highest number of features that the customers want to. But on our side, we also want to do it at, we want to operate this fleet efficiently because it's a large fleet. And, um, as everybody knows, this day's inference is, it's, it's, it's a demand constraint space where there's enough demand for the customer. And so if you can't quite utilize your resources efficiently, then you're not, uh, serving all the customers who want, who want to, uh, who want to use the service. And so on our side, we also want to make sure that we do it as efficiently as possible. And, and so mantle was combination of kind of observations of how, uh, inference behaves. And that's building the inference system that allows us to, allows us to, uh, server customers, serve them, uh, well, and keeping, keeping our utilizations, uh, uh, and our, our, our costs, uh, it's good as possible because I didn't want to be able to. The most possible in terms of our customers, eh? That's right. Exactly. Exactly. I think, I think one of the great things about that challenge is because of the proliferation of different models and being able to cater for all these kinds of models and kind of be able to run inference efficiently on that is a, is a non-truvial problem. Um, and, and I think what's interesting is that you and the team decided when you were going to build this particular, um, service or part of the service that you were taking an AI first development approach. And just before we, we gone to the show, we were having a quick chat. And we tried not to chat too much before the show because then all the good conversations happened off the show. Uh, we were talking about the fact that, you know, what is the way to develop using AI, uh, today is not what it was three months ago. It's not what it was. Twelve months ago. Wipey ordered it is in six months. It's, it's, it's changing all the time. But what really appealed is, is, is you and the team have taken an approach and then you've blocked about this approach and use data to talk about what happened, what you've learned. So we're going to get into the guts of this because I think there are folks who are hungry to hear about this. So firstly, you know, you're, you're a distinguished engineer. As I said, you got the wadge of patents. You've written more code than most would have yet you've taken an AI first approach. What was the thinking here? Uh, let me kind of tell you a little bit because I think everybody had a slightly different path in how they started trying, trying to use AI based at all. But let me walk you through my path. So of course, you know, five, five years ago, four years ago, chat to PD comes out. Ella lambs become a mainstream and, um, and one of the things they, one of the capabilities they had was writing code. So folks, a lot of interest in that industry, a lot of folks trying it. And, um, probably roughly about two years ago for me, I started trying to write code using LLAMs. And of course, you know, the results for pretty much back then. You can try. You can maybe have it write a little piece of code. Occasionally we'd have to get things right. There's Twitter is full of all the jokes and all the comments making fun of the code produced by LLAMs. Uh, and as long as they were getting better and better. Um, and for me, there was about an inflation point where I started being able to use it, um, on actual prototypes. So I could literally have, um, an idea I could use a model to write a little prototype, try it out. Uh, but yet wasn't, I wasn't quite convinced yet whether this is, this is something that could be used in production. So I would totally take the prototype and then turn it into something that I would build. Um, it was using maybe more as a gimmick or as a tool to help you learn rather than something that would make me more, more, more effective. productive. Yeah. And then maybe about six to nine months ago, I started noticing that I was doing that a lot less often. All of a sudden the code produced by the model was getting good enough to wear like, yeah, I still needed to to make modifications. I still needed to constrain it, but all of a sudden it was it was solving the problems in a more robust way. And so you can have this state in March of progress. And at the same time, we were also, you know, we were also trying to figure out the, how do we, how do we can apply, apply models, how can apply this learning to actually turn into real production code? What needs to change in our industry? What needs to change in our approaches to where we actually not just using it for toy exercises and PSCs, but actually using it for real stuff. And so within mantle team, we had a couple of ideas, couple of those. They're not, not, they're not that fancy. And one of the, one of the kind of one of our realizations was that at then the day, the model is a tool. And it's a tool that accelerates an engineer. And so one concrete rule would happen. I think that rule worked out really well for us is that at the end of the day, any line of code committed into the repository has a human name attached to it. And the human is ultimately responsible for what the quality of the code. And so, and so if you look at it that way, it's probably more analogous to a compiler or to programming language other than to a fully autonomous agent that runs around and modifies the code. But I'm not saying there aren't patterns that could benefit from fully autonomous agents. But in our case, we made a decision that a human is the ultimate author of the source code. It's kind of the continuous extension of you build it, you run at the top thing. It's like, exactly. Exactly. For all your code, however you produced it, you know, you could have got a barrel of monkeys to make that code. It doesn't matter. That's right. It's still Joe's code. That's right. That's right. And once you have that accountability, you now start a lot of right, a lot of right tensions happen where the engineer responsible for writing code has to have to figure out how to make the model produce the right quality code. For me, everybody has a slightly different approach, a slightly different pattern. What I do is I I give a model a prompt and we can talk a little bit about it and that process was interesting. And then I let the model produce the code and I review it. I review it. I decide, my first decision is like, well, the model solved my problem in the right way or not. Do I agree with the solution? And more often than not, I do. I think I decide did the model, you know, did a model solve the problem to my liking as a path that did it used the right practices that used the right libraries. He's overly complicated or did it miss edge cases. And then I iterate on that on that project the way I would do it myself. I fix bugs, I fix issues. And at some point, I think to the code directly at that point, or are you prompting the AI to do the coding? What's the balance if we both what I found out works the best for me is that if the model is almost there, but missing maybe a line or two or maybe missing an edge case or two out, I'll just take over and finish it. Just quicker, faster. I don't mind doing it at all. Occasionally I find that well, that's quite how I would have done it. And then we go for a second round. And we know different how you would do it with the engineer. You have a, you would have a, you provide feedback. Perhaps change data structure here. Perhaps let's use a different algorithms and we kind of iterate on it until we go. Earlier on though, and this is part of the learning experience, I, I would actually and this is me getting into a little bit of kind of have my approach of all in the early days, I would literally stay glued to my screen and I would look at lines of code appearing and that would even stop the model in the middle. So like no, no, wait. Yeah, yeah, you go in off of the tangent. Let's change the direction. Let's try something different. I still have gain more better intuitions and gain a little bit more confidence about how the model works. I started going becoming more and more asynchronous where like it's literally I send the task core with the fence. I wait for the results. I valid results and every people will. And almost universally, it's a multi-step date process. We iterate, but I've got to the point where I've also calibrated my prompts enough to actually get, find out what's just the right level of complexity to ask the agent to do that results are going to be something to my liking. And I think at the moment, that's one of the sort of techniques or tweaks that's really important. And before, before we get into that, I'm going to call out what we're going to talk about in a minute because we're talking about this, but we have data that shows 10X development velocity. So we'll get into that. So I want folks to listen carefully what we're doing here because this can really help. So the prompting is a huge thing. I think at the early stages of quote unquote, vibe coding people like you know, write me Twitter and expect the thing to do stuff. And we've come a long way in terms of understanding, well, the better you prompt, the better you, the outcome can be how big should something be, et cetera. It's almost like the old microservices conversation, we have this habit should mark service, but so tell us a bit the prompting approach you're currently using. And I'll preface this on your behalf by saying, this is the approach you use today. It doesn't mean it's going to be the best approach tomorrow, but it's what you've learned. Yeah. And I think I think it hits something important. When you said, you know, don't ask the model to write me Twitter because reality would be like just like anything else in life. You need to calibrate yourself on how to most effectively use the tool, right? Like when I first, when I first started software development, I learned I started with C and I'm terrified to look at my first code. I didn't know what I was doing. I was likely, I was likely using wrong idioms, bugs galore asking asking the language to do too much or something that's it wasn't meant to do. And as you, as you hone your skills as your practice, you can have get this intuitions about which approaches are likely to work and which approaches are not. And one of the intuitions that I find that is super helpful to build is finding the maximum supportable request from the model, meaning that something, you know, you ask too much and the model is likely to fail. You might run another context window or a may too much of the theory. It does weird stuff. Yep. Ask it for too little and well, you're not getting quite the speed ups because you still have to be constantly in a group and asking. And so one of the intuitions you build up and one of the reasons I think that actually doing and trying things is so important is that you build up that intuition of what is that maximally supportable request. And changes over time, it's not it's not a static saying it changes with the models, not models abilities, projects, domains, but having that intuition has been incredibly, incredibly helpful. And so for me, I've learned, a couple of observations. One is the observation of having that kind of sweet spot of the request is the maximum helps be get the maximum kind of acceleration, maximum speed up. The second one is ambiguity and this is where one of the interesting places is where, you know, we work in the field where many different approaches can solve the same problem. And sometimes you don't care and you can just just pick me once other times you may have strong opinions based on your past experiences as an engineer based on your kind of what you're trying to accomplish. And so helping the model, the simple, you aid also what you want to do is turns to be super helpful as well because it's since it sets up the guard rails for where what you want the model to operate, which is by the way, no different than then dealing with junior engineers where you want to start guard rails, you want to start absolutely shared expectations and work from that. And so a large chunk of what I kind of do, how I operate is just trying to think through what are the appropriate guard rails, what are the appropriate constraints, what are the high level approaches I want to do. And for example, it's super common for me to actually brainstorm a problem with the model first before we even start implementing. And so just recently, I was working on something where I didn't quite have a good intuition myself yet what what I wanted to do. And I literally started with like here's the problem I have list list list solutions. The model listed one immediately became obvious that one of those was really not going to work. So we went the other way we we worked back and forth to the point where I finally felt confident that this is going to work. The model understood what I wanted to do or this the context was there. And then I flipped from the brainstorm more to okay, let's go make it happen more. And so it is, you know, it is the tool is extremely flexible. It could be used in different ways. And you kind of want to take the maximum advantage of it. You want to not just, and not just tell what to do, but sometimes also use it to help you figure out what to do and how to do it. I think I think that's a really important insight because it's, you know, if you think about how these models are trying to train on huge corpses of code, some goods I'm not so good, but lots of code. And so it has, it doesn't have an opinion. It's a statistical model, but it has access to way more code than we can fit into our heads. And there's a lot to be said for, hey, here's the problem domain I'm trying to work on. And simply one of the things I've found is really short with the bombs are saying, ask me questions, you know, like prompt me to tell you what you need to do to get to a better point. And you say, you start that dialogue. And you're still driving, but it's It's taking around lines that you may have not even considered. different approaches that just weren't in your mind because you know, had had your first coffee yet, you weren't sort of really thinking clearly. It's just it's fascinating to see like you say it's like working with a colleague to some degree of having an interaction. Yeah, yeah, and that's exactly right. The model, you know, the model sitting every single implementation of Beatre out there. I have not. Sometimes it's just brainstorming. I still want to be in a driver's seat. I still want to be the one that makes the ultimate decision the way we go. But using the model as that sounding board on on problems is oftentimes that I found it as a useful first step. And a lot of times helps me help help me decide what I won the model you before I even go to the because you're still deciding this comes back to you on the code your your is not you know, hey model figure something out and then go implement something I have no idea about it's like you still yours. And even if it's proposing something maybe you're not that familiar with I'm assuming you probably do a deep dive yourself and say well, actually, yeah, what is what is in this for me is this an algorithm I haven't seen before is just an approach I hadn't considered what I want to know to understand the risks. Yeah, absolutely. And you still you still review the code and you know, you still you still have to keep your judgment on when you want to you know, what complexity when I introduce and when you want to go to try to test it way versus trying something more performant or more more more experimental. And you have to kind of you have to still own the decision and the day you have to verify what's being produced. But having that conversation is is I found it so often useful to just because of the day, the clarity clarity of thought clarity, what you want to do is what provides the acceleration. The more you have clarity in your own head, the more you kind of dealt with ambiguity, the faster you're going to go. That's been true before. That's true even more so now. So the models are a fantastic tools to just help you gain that clarity as well and kind of brainstorm ideas and trying new things and then you know, all days we would use Google, we'd use this days. It's just a new way of doing research in a way. Exactly. And learning very, very true. Well, even things like the fact that you have the idea of a same CPP with access to all the documentation just serves as a time I'm looking up the dark. Even with Amazon, look at the documentation. Oh, absolutely. In fact, I found out the models are they know your services better than me sometimes. And it's crazy because like just asking questions and asking about behavior has been tremendous timesaver. Let me touch on context windows briefly because certainly what we're seeing is that if you're running a heavy full context window, things start getting squirrely. And so certainly my own personal workflow, I'm using that the frequent intentional compaction approach and finding great results have almost become you know, resolute about, you know, if it's once it gets over 40% to 60%, I'm compacting because weird stuff happens. Are you seeing that? Do you manage your context window particularly closely? Oh, like there's no tomorrow. And for for for for listeners who are familiar, so models have context and love windows which is the ultimate kind of limit constraint resource when dealing with a model typically this range from 100,000 to a million tokens. And as soon as the context window reaches the maximum the model cannot do working anymore. So if you have to use techniques like compaction or reduction or kind of long term memorization to just start start working around that. And so it is it is one of the things that us an engineer need to actively model and manage. And what I found out what I do, I do a couple of things and there's there's an interesting conversation we can go into what happens, how the tools need to evolve. But I clearly context window between every request. And so what I tried to use I use a lot of files as a long term memory for the model. So concretely some some requests fit into a single context window where you want to fix a bug, make a change and you you working it to finish you clear the context window. A lot of things that work on they take more than you know they take they require multiple iterations. And there what I find I find works really well is starting with a file that describes a high level approach we wouldn't do. No different than what I would do if I was working on a large problem myself. Yeah. And then we break it down right you break it down. Curial that some of that for you with their it's been driven development. I tend to use the command line tools a lot and I would start with the model I started prompt with okay here's our here's the design we agreed on we're now in step two out of seven or two out of however many. And here's what we're going to do in this step and this step we're going to be doing blah blah blah. And then we'll work just these things. And then when we finish it we commit the code and then we want to clear the context so you forget everything of done. It's not durable durable file that keeps kind of the track of our intent and the last commit and we continue from there on. And so currently a lot of it is things I do manually by hand and this is just that's the match really well how I work myself as well. I start with the end to end goal but then I break down problem and it's just generally how humans tend to work right you you want to you want to break down the problems the smaller problems then go after those. I can imagine longer term this is going to become part of the built in tools and this is going to become this work was going to be a lot more natively built in. But yes the context window is something in your kind of in your back of your head the back of your mind and you are effectively managing to it as a constraint resource and breaking down the problems as much as you can to the point where you take maximal advantage of that resource. And the funny thing is both of us being quite you know experienced let's say practitioners in the field is that there will be a time in the next few years where we'll sit back together and go do you mean when you had to manage context window this. You young kids these days you don't know what it was like. So Joe you mentioned you mentioned in your blog post 10x development velocity. Now that's a classic you know well it's got to be marketing gump that's right now where you can prove it you know come on. Yeah anyway anytime somebody says use a round number you have to yeah you have you should have access to be 7x. Yeah yeah yeah. So tell us about tell us what what that looks like in your team in that team. Yeah well so I don't know exactly what the number is is it 9x or 11.3x but I'll tell you from my personal experience and personally you know our field when we think about venture productivity that's a that's a topic for many other podcasts because it's such a deep topic and so many strong opinions but in the day the way I view it is like well we engineers we we don't solve for the self-cost of our problems and then at the end of the day to don't solve for you have to write code and you have to write high quality code but you have to still write code. And for me I've always you know I've always enjoyed writing code times always been a challenge as I become a senior but I would always find an hour or two a day to write some code to do to do a little bit of engineering work that just enjoy doing it and with the kind of switch with agent first development I just find it I just accomplished so much more in the same amount of time and for us you know for our team again commits are not the not the whole story they're just the slice of the story but we've we've written we've written on average lost to 10x lost to maybe even more than 10x number of commits across the whole team and it's just not even it's not even just one team average not just be it's every team member on the whole team the whole team and in your blog post just for folks to understand there is a great commit graph that shows that the velocity of the same pre and post and it's I was going to say unbelievable that's wrong because it's believe it's the data tells the stories like it's a this dense packed amount of committing going on but as you mentioned so I want to I want to read a rate still owned by the individual developers still responsible for that but you're shipping a lot more and you talk about it like driving at 200 miles an hour yeah yeah I think it's a good analogy well in fact just even as a joke maybe to drive the point home as you know talking right now I prompted the model to write some code I haven't I haven't seen the results yet so I'm not actually paying attention but I'm literally writing code as we speak and can I can I can I share a DDC good I have Kiro writing some code for me as well yeah yeah yeah yeah this is the modern modern nerd we code even where we're doing other stuff now it's different and that's and that's and really the the reality of it is the the enablement the amount of quarter of everything would not have been possible you know the world's not just because you know it takes more time or I don't have enough free time he just I fundamentally would not have had enough continuous time where I could sit down and read the glitch code right because there's a lot of demand on my on my time there's a lot of things coming in and so having this it's it's not just the velocity improvement but it's also switching from a synchronous motor writing code to a synchronous motor writing code where I can actually write code and not necessarily have an entire entire my attention span focused on that until the end where I want to go ahead and review the results and so that not only that that in itself is what's been so enabling for me is that My family jokes around is like, I've got into the point where I love giving a model something over night just because I love kind of it bothers me that it's wonderful. Yeah, I can, you know, I can just give it a prompt and wake up in the morning and maybe maybe it was still ambitious. It doesn't work, but it's like, let's just keep working, right? Like why? Nothing to lose. I don't exactly. I don't need to be up. So might as well, might as well use the use the compute cycles to produce some code and see what you know, see where it goes. So if you're generating all this code and you're still responsible for it, how are you ascertaining the code is of high quality? Are you doing a testing? Are you seeing an increase of the velocity of bugs along with the increase of the velocity of code? Yeah, well humans are going to be humans and so bugs will continue being in the issue in that industry kind of until there's a breakthrough in how we do validation, how do verification of code I suspect we're going to have to be dealing with bugs. And so that's not different with the code reason by me or by the model. And so the thing that's changed and the thing that that our team had to work through and had to navigate is that even if the rate of bugs was lower than what would have been with the human, they still happen. And when they happen when you when you write a lot more code, you're going to have a lot more bugs just just just just a little bit right? We're all a big number. Yeah. But what's even worse about those bugs is that now it's your bug density. Once you have to deal with them, they impact the whole team, right? Like you checked in, I checked in a bug equal today. It's going to potentially break other engineers workflows are in other engineers codes. You can have almost like a tools down scenario as everybody strikes the chase down what's changed what's broken. And so we've learned very early that we have to and not only this new our industry has been paying a lot of attention to how to how to improve the testing, how to improve the verification of software. But we find ourselves that we need to be raised the bar even higher. We need to focus on how do we catch as many bugs as possible before they check in before they get into the production source code before they get into into a beta environment where where they could impact our engineers should do in testing. And so a lot of the thought that we've been putting a lot of the energy we've been putting is how do we set ourselves for success in the way that we don't constantly stumble and introduce bugs. It was again like a lot of it our things are this and not novel ideas ideas been tried out we've done them in the industry but all of a sudden they become even more important than they were before because at this rates of change if you don't you have to you have you need to have a way to curtail chaos or else it just becomes to explode yeah explodes it's its human's ability to reason with it. And so a few of the things have done and this is maybe kind of worth through if I had to pick the one thing that I think made our team successful it was less about you know having folks who knew what we do a lot of us this is new this is a new space for all of us you know our whole industry is trying to figure out how yeah how things are going to work but having folks who are when faced with obstacles look for solutions and so it's very easy to say yep I introduced the bug model that we know Joe and model together what about that made it to production we should slow down and not and not continue not like that's all happening right and it's a lot more satisfying but the other the same time a lot more difficult to say like okay what do we need to change to make that not true anymore and so for our team one of the things that we've been putting a lot of attention is how do we how do we we accepted that using AI assisted genticoating is the way we want to the industry is going to work what needs to change you know build systems in our test systems you know our development workflows you know operational workflows to make that a reality and so I would probably not as iteration to say that 25% of it seems energy goes into that aspect not just feature development but just thinking about development practices thinking about operational best practices and sometimes it subtle things like a very concrete example of that our build system you know our build system is been around for a while it solves a lot of Amazon needs but it's it's not fast and that made a ton of sense in the whole way yeah it didn't have to be because a human if the human takes a week to build to write software or where human takes a couple days to write a feature quick care of the takes 20 30 minutes to to build and test and run all the integration test but in the world where the velocity is is sufficiently high that workflow now becomes can become a bottleneck and so one of the things that we worked on is working with our build systems with our partners in builder tools and how do we actually speed up this workflow to the point where we can get an answer in a few minutes we can't have their model run all the tests run all the builds and catch integration bugs much much faster and you can't it's not one big things but it's a lot of this little attention to details little like you see a barrier you knock it down you figure out how to how to pave the way and it just takes takes work takes attention to details takes a kind of almost like stubborn persistence to keep insisting that it's possible to get a 10x or 12.6x speed up let's figure out how to get it well I think it's interesting because yeah yeah pointing out that as you as you unconstrained one thing you discover new bottlenecks and similar to when we're writing services and deploying services at scale you know the the attention to detail once you get to scale is vital to get the benefits and suddenly if we're talking about the scale of a software developed by suddenly becoming that's used 12.7x is a number now yeah 12.7x suddenly it's really important to focus on all the other efficiencies around that person what they're doing yeah yeah because that's going to be the the big the big problem that tend to all this up that's right that's right and a lot of it is just like you said attention to detail and just desire it's desire to go fast and desire to uh and bottleneck yourself while still keeping up the same bar on quality reliability exactly and that can't go with value yeah absolutely I can't go at all now you talk about also in your blog you talk about communication and communication bottlenecks I think if you know all of us working IT know that you know the ultimate world of IT is often me myself and I working on my own thing no I don't have to talk to anyone else life is perfect you know no overheads and the minute you add one other person it gets more complicated the minute you add more than one other person gets exponentially more complicated but if you want to go far you need lots of people so talk to us about communication and what you found particularly working this velocity yeah there's uh there's a famous beam of you know you have it plays on the graph theory but you know if number of nodes is n their number of connector edges is scales based on n squared and so because like I said team grows communication becomes like this proportional uh this proportion of uh percentage of your overall time um and so yeah our our hypothesis early on was that it is really hard to move fast without communicating a lot and you can kind of decide and you know Amazon is famous for the service or in architectures where we use service as boundaries uh of communication right so a service A can operate without spending a lot of time talking to service B other than a few times they have to change the interfaces and that's works really well at the large scale organizational levels but when you're working within you know a single platform a single service uh communication is oftentimes one of the necessities or for speed right share that having shared that shared understanding have that shared understanding about goals about approaches about trade-offs intentions and so our hypothesis early on was that it will be really hard for us to move fast for mental team to move fast without uh communicating frequently at high throughput and with high fidelity and so our hypothesis was that um you know remote is not gonna work and I know folks have also the book opinions about remote work but for us we have a ties that we will need to be all sitting together across to each other be able to constantly communicate uh within the mental team just because um the speed at which we we we thought we could move is just gonna be really hard for everybody on the team not to be on the same page and so uh we will sit uh in fact the whole team sitting right outside my door uh right now probably listening to me speak but we'll see um so we all sit in a in one-spot area uh we we have a lot of communicate we have a lot of interactions we have a lot of discussions we have a lot of debates um we we uh generally our mental model has been if we have a question or a decision to make we don't schedule a meeting we just walk over to each other's desks and having quick conversation sometimes we resolve the the discussions quickly other times we realize that even covered a fundamental decision point of on the middle of trade-off and then we have to spend a little bit more time wide boarding the idea talking to this um but I probably wouldn't exaggerate if I said on a typical day I at least spent one hour just talking to other engineers in a high throughput face-to-face interaction out of the wide board just to just to work through the details uh and how do you find that works in terms of obviously um we're folks getting to quite a quiet flow um it can be frustrating if someone sort of taps you on the shelves says hey can I kind of borrow you real quick how are you balancing that tension between yeah between those two things um probably not too well for For me, I just. - You generally find myself. - Well, you get to say, hey, this is what we're doing first. - Yeah, well not only that, but also find myself that. I prioritize shared understanding of over else. And so if somebody comes to me with the problem or space of decision they have to make, I actually think that probably is most important thing I could be doing at the time. Yep, I might be in the middle of writing some code, I might be in the middle of debugging something, unless it's an emergency, I will prioritize shared understanding because at the end of the day, that's the cost of not having that shared understanding that you're gonna go build something wrong, and then you're gonna have to double back and redo it. - It's more expensive in the long term. - It's more expensive than the long term. And so over time, I think I've put myself to just always, you know, with my headphones down or put my computer off and go spend that time. But at the time, you know, one of the cool things about kind of having small, tight knit teams is like, you also learn each other's styles, and you know how different folks operate. And so for example, I know if a person has their headphones on, it's probably because they don't wanna be disturbed. Somebody else is always just like me and they happy to be the jump-off from the table and talk to her. And so you kind of learn each other's practices, learn each other's habits. And frankly, that's to me, that's always a sign of a healthy team of the team of folks, which is respect each other's boundaries, respect each other's approaches, and know enough about each other to work, to work kind of in a way that's most effective for each other. But for us, you know, we generally, you know, for small discussions, it's pumped up in the shoulder where we have a big discussion. So I wanna make sure that we take a little bit, you know, we take a little bit more structured way where we would bring it out there, and stand up and say, yep, we're gonna have this discussion next where we're, needs to be part of it, come join it. But you and the team also, I think, have been very convicted about, is this something I know a lot of software engineers listening will be like throwing their hands up, going, yes, please let that be me. You've been very convicted about removing meetings and having little to no meetings in the diaries, which I think is fantastic. Just unpack that a little bit, 'cause that's, I know from a productivity perspective, it's a huge thing for developers. - Yeah, well, I'm not quite as successful as still. You know, I still probably spent 20%, 25% of my week in meetings. - Yeah, it's, look, it's, the meetings, the meetings, the runs very deep, our industry's been talking about, you know, meetings and discussions. One of the things that, you know, even before, before we got to, yeah, assisted coding, one of the things that always, I took prioritized, that's how I could find those, couple hours per day to do coding is that, I only wanted to be in a meeting where it's either absolutely necessary for me to add value to the meeting or it's absolutely necessary for the meeting to teach me something that I need to learn. And so a lot of it already can actually to meet to reduce the number of meetings I take per week and prioritize just doing hands-on engineering work, even before that. Now, we took a little stricter stance. We wanna make sure that the engineering team can focus on building, building mantle building, the best infrastructure we could build. And we took a stance with a team of, a science from this whiteboard conversation, a science from, you know, actual discussion pertaining to building mantle. Truly asking yourself, is this the meaning that we need to be in? It's a lot of times the answer is yes, right? You have to go meet with them, a team to figure out a solution to some technical problem, you have to go agree with an approach. But if you could, it looks to that critical lens, a lot of the meetings that you go through, may not be that important, right? And you kind of have to sink through, you kind of have to make that cost value of like, is this moving, is this getting, is this helping me solve the cost of a problem is a new unique way? The thing that I would, you know, maybe add to this that I personally found works with that, having folks who are senior knock who actually understand that trade off, that super is super important. And having, you know, there's a, there's a famous, what I find often helps, there's a framework that my line, a vice president of, at AWS, as written is, it's a principles, it's Chicago's are the principles, roles framework. And it kind of works through different role as senior technologists can play in a conversation. You can be a sponsor, a person who is, and I'm gonna put you a little bit of the expansion, but you're a person who is, pushing an idea forward, you could be a decision maker, you're the person who is deciding on something. And a couple of other roles. And what I found out is that super helpful for me is looking at the meeting as deciding, am I one of those roles that are critical? Am I a sponsor, am I a decision maker, or am I the person who's gonna be doing this? And the answer is no, usually it's probably a meeting that would be okay to skip, and you still have to apply judgment, you still have the needed decision, but going to that framework of the outcome that you are bringing to the meeting is oftentimes a useful framework to decide whether something is worth while you're attending. And if you look at it, turns out that, you don't probably have 40 hours what the meetings in a week, you probably have a lot less. - Yeah, I like it, I like it. So you touched on the fact that, you do a lot of asynchronous prompting and that sort of stuff. So what does the human do during that time? Like what are you doing, and what are your team do during that? Do you sort of kick back private affairs goal? I'm guessing that's not the answer. - Take a pop, do a podcast interview. - Yeah, do a podcast interview, exactly. Now what are you doing in the gaps? A lot of it, sometimes, sometimes those gaps are perfect time to have those kind of whiteboard conversations. Where you go, are you talking about what you're gonna do next? Sometimes you actually go grab a coffee or go take a meeting or go have a conversation. Increasingly, the idea I've been playing around with is actually like, "What can I use that time to spin up a second request?" Do a second change. And now you can start gonna see how you can go even beyond 12.6x and maybe you can get 20.22.6x. But a lot of it is just, a lot of it is just, yeah, it's using that time to do something else. And to be honest, it's not a surprise to anybody. Hopefully that software engineering is going beyond coding. You have to do a lot of other things, right? And so this would also be oftentimes the time I'll spend on other things that require building software. - Major attention. - Yeah, maybe we are, there's an operationally, should I need to look at? Maybe there's a deployment I need to test. Maybe this is a customer question. And you kind of have to structure a day a little bit differently and you kind of have to think through, you start playing the game of like, okay, let me just get this request and get this prompt out a little quick before my next beating. So I can don't waste the time have the mile actually do something. - It's funny, say that I've talked to a lot of folks who are like, yeah, as I'm walking to my lunch, I'm making sure I'm hitting into before I leave the desk for the prompt to happen. Or, yeah, I'm at home and I'm between chores and I'll quickly pick off another prompt because I know I can come back and air and start. - Yep. I have this sense and it's probably a thought I need to flesh out more that the whole concept of task management, of work management for particularly for software developers but all knowledge workers is gonna radically change because suddenly we're trying to track all these things in parallel. I don't know, but you, but once I've got three AI sessions going at once, it's tricky to maintain your contests correctly because as human beings we're not designed to do that. We're supposed to be tunnel vision and not content switching. So there's I think a lot more abstractions gonna have to happen to make that a little easier for us. And at the moment, we'll take the lift because we know the benefit is there but it's not the natural way to operate. - Yeah, we have our own context windows that overflow and we need to actively manage it. - And they're not a million characters, that's for sure. - Yeah. Yeah, absolutely. And I think this is where there's a lot of opportunity for innovation and tooling to help us manage this. Like for example, Kuros, you know, Steak Driven Development is a great approach to like how do you break down the problems and kind of help the human manage the overall workflow that the model needs to produce in the way that sort of harmonizes how humans and models work together. And I suspect that there's gonna be a lot more innovations in this space, a lot more innovations that help humans interact with models, help humans model, manage models. As also the models are gonna become more powerful and maybe able to run for longer periods of time with more complex tasks. - And some of them are starting to have that ability to spin off other models and all that sort of stuff. It becomes so if, yeah, it's turtles all the way down. - Exactly. And so yeah, I think we are kind of probably sounds a little cliche by now, but we are seeing big shift in the industry and a lot of our things, a lot of how things work are gonna change. Are we gonna have to learn? We're gonna have to discover, we're gonna have to pioneer some of those things. And I suspect there's gonna be multiple approaches that work just like with any other shift and you're gonna see a lot of cool new innovation coming out, a lot of cool new tool coming out that help humans help us. - So it makes it interesting. - That's right. It's exactly that. It's what makes it so exciting and what makes it so interesting. - And so what would you say the thing that, working this way, you've done it seriously now, production code, customers are using it, you've got a time, this is real stuff. What surprised you the most about working this way? Like did something leap at it and go, "Wow, I didn't expect that." I mean, the trivial answer is like just the fact that it works as well as it does. But it works. Yeah, yeah. And it's still, you know, if you think about it, you know, there's just model, you know, there's hundreds of billion, you know, truly, it's truly in floating numbers that somehow produce functionally-drag this stuff. Right? Like that's, I suppose we all have to pause and think about how all of that works. But if I told you that was going to happen 10 years ago, you were to say, I don't know what's not. Exactly, right? I think, I think sometimes it just worth reflecting the complexity of something and how well it works. But if I had to, you know, if I had to think that the thing that probably took me most by surprise is that a synchronous nature of software development is the fact that all of a sudden I don't actually need to be present or even like paying attention for, you know, 80% of the, like a cycle of making a software change. And then I have to kind of, I have to, well, hey, figure out what to do with myself. I don't actually have to be glued to the screen watching every line of code producing. I think to me that was a little bit of a surprise because I was still envisioning, and maybe this is just new thinking small, that hey, we are sitting together, we're actively peer programming. Yeah. And I'm making changes while the model works. And switching to any synchronous mode has proven to be a big enabler, a big, a big, frankly mindset shift, more so than just having model produced code. Interesting. And the other thing that's been pretty surprising for me is just fun. I think of the day to just make successful, like we have to almost like going to grassroots, a lot of things I talked about, you know, small routines for everybody in communication. That's how we used to work 20 years ago. So what are the fundamentals? Yeah. Building sophisticated test harnesses, paying attention to deployments and builds, but those are, you know, those are invariable in time. And they always matter. They just matter a lot more now. And so seeing that sort of full circle and coming back to things that matter, things that make you more productive, has really been, I would say, surprise, but been, been an interesting observation. And, you know, a colleague, a colleague of mine said it well that, that AI, AI driven development, it's going to impact well-functioning teams much more than it's going to magnify well-functioning teams much more than it magnifies it, not so well-functioning teams. That is a really interesting insight. That is the yes. I like that a lot. So that's, which comes back to, you know, in my experience, well-functioning, I'm saying to do the fundamentals exceptionally well. Like athletes, you know, pro athletes, you're the talented, the trained, the set, but you look what they do. They do the basics over and over again, perfectly. Yeah. So it is. Exactly. So, so we've got lots of listeners. There's folks who listen, go, oh my goodness, Joe, sign me up. This is the future of my team. I'm good to go. What advice would you give to someone in that mode? To enthusiastic, right, Lisa? Yeah. But about to embark on this journey. I'll probably give a couple of advice. I would say try build. There's just no substitute by building. Nothing I say right now. Nothing anybody else says. Nothing you're going to read online is going to teach you how to apply the tools to your problem space to your domain. Just fundamentally, I would say being a builder is a lot more valuable now, a lot more important now than it's ever before. So just start by building, trying, winning, and experimenting, learning what works, what doesn't, forming your own. Fawning with that edges. Yeah. Find your edges. You know, build muscle memory, build intuitions. And then the second kind of the second advice I would give is that it pays to lean in. The thing that made mental teams accessible was the fact that we all believed that it was possible. And we knocked down the barriers and we changed our own mental miles and our own approaches to make it so. And so I'd say starting with that mindset of not that, well, is the egg going to work or not, but more like, well, yeah, works. What do I need? What do I need to change about my systems, my approaches, my software, development approaches to make it so? And just going with there has been incredibly powerful for us because frankly, you know, earlier on, we run into bottlenecks, we run into problems and it took a lot of perseverance and a lot of attention to detail, a lot of kind of trying things to get to get to get to get to. It is hard to do. And it's hard to change your mind too. And it's hard to let go of things that you did previously or things you think are good and it comes out to our old friend, we stubborn on the vision book flexible on the details. So this is what this is. This is your like, you knew you wanted to go faster, you knew you wanted to automate, but my goodness, you had to change a lot of detail. Yeah. So if you want to change, you have to solve a lot of problems, you have to learn a lot of new skills and you have to focus on your basics. Yeah, yeah. That's how it is. Joe, this has been fascinating. I know a lot of folks will have enjoyed hearing this from the quote, quote, real world. If I can put it down, what is it? Well, be on vibe code here. Just to remind folks, you're using this code right now in your life. It's happening. Joe, we'd love to have you back sometime to share more about the journey because I know that in six miles time, your work will be completely different. So thank you so much for coming up. That'll do. And thank you so much for having me. I really enjoyed the conversation. Always a pleasure. And would you love to get your feedback at our best podcast at Amazon.com is the place to do it. And until next time, keep on building.