Why Coding Agents Keep Making Your Codebase Worse

This episode is brought to you by Google Chrome. You think you know a browser, but Gemini and Chrome? That's new. It can help you with practically anything on the web, like restoring a vintage motorcycle from a 50-page restoration block, or finally break down that long article you've had open for weeks. Gemini and Chrome is here for it. Ready to make anything online makes sense? There's no place like Chrome. Check responses set up require compatibility and availability varies 18 plus. - When you need to build up your team to handle the growing chaos at work, use Indeed Sponsored Jobs. It gives your job post the boost it needs to be seen and helps reach people with the right skills, certifications, and more. Spend less time searching and more time actually interviewing candidates who check all your boxes. Listeners of this show will get a $75 sponsor job credit at indeed.com/podcast. That's indeed.com/podcast, terms and conditions apply. Need a hiring hero? This is a job for Indeed Sponsored Jobs. Oh my God, I ran out six Codex subscriptions in a week. Isn't that amazing? What did you ship, dude? A Sloth Code bench? Oh, it's incredible. Has anyone seen this? Do not mortgage your code base in the hopes that in a year the model will come out that's smart enough to fix all your problems. I'm never letting anyone merge anything with this pattern in it again. Am I 99th percentile? Am I being left behind? Do less things in parallel? The same thing that was always true about engineering is still true and all work. - Today I'm joined by Dexhorathy. CEO of Human Layer, the man who coined the term context engineering. We talk about most of what you see online for agents is completely fake. While token maxing doesn't work in the real world, how benchmarks work and which one you should look at, how much faster you should actually aim to be with agents. And it's not 100x. And at the end, he shares his agentic engineering workflow. What is automated? What is a software factory? And where does he still spend this time? (upbeat music) - I saw a bunch of people commenting as well that like Astra got much better at computer use. I got much better at like blenders and like game slop and this kind of stuff. But it didn't actually get that much better at like writing maintainable code or like controlling complexity or like, you know, removing things or like changing abstractions or whatever it is. And I think I think that was my feeling a lot. And like a lot of people had to vibe. And so one of the things we're talking about tomorrow is sort of this benchmark I discovered called Slopcode Bench. It was made by the lab of Gabe Orlanski. He's like a researcher at the university. It was content. Have you seen this? - No. - A Slopcode Bench. Oh, it's incredible. It's an entire benchmark designed around proving like models make code bases worth worse over time. - Really? - Yeah. So up until that point, there have been like, I would say like three flavors of benchmarks. There's like the core like sweet bench, benchmark, which is like, here's a tiny PR from a real open source repo. We'll rewind the repo to before the PR was made. We'll like pull the tests out of the PR bases. Hey, model, here's the issue that was filed 10 years ago on this random Ruby project or whatever it is. Go like fix the issue. And then we would take basically any changes it made to the test, we would throw them out. And we would drop new test, like the test in from the benchmark case. And then we would run those tests against the models code. And then if it passed, we would get a reward and if it failed, it also has to like do it without breaking any other tests. But it's very, very like small targeted, like can you get the test to pass? - Yep. - Obviously these are benchmarks that are different from verifiers, but it's like all the benchmark. Oh, this model score is 95% on sweet bench. Coding is solved. And it's like, that's not what real coding looks like. - So you had this other chapter of things where it's like, okay, cool, instead of solving this one little PR, it's like rebuild all of it, Microsoft Excel from scratch with every single feature. Or rebuild all of Redis from the specs with every single feature. So we have these like marathon benchmarks. Those two are from one called a sweet marathon. Actually, it was like four to 400 hour tasks. And then cognition introduced this idea of like, yes, okay, we're gonna use much bigger tasks. And also we are going to have a judge model run afterwards and give like contribute to the score on the exercise of like, did this thing break any of our code based rules or conventions or patterns or whatever it is? So it's sort of like telling it like, hey, look, in C++, we can't do logging like this, it has to be like this. But none of those really map to how real software is built, which is they all give the model the entire problem up front. - Yeah. - Here's the thing, solve this thing. And what Slopcode Bench does is really interesting is it gives the model a thing to build Greenfield. And then it says, cool. Now take that code base and add this feature. Now take that and so they have these like challenges, each has like three to 10 checkpoints in it. - Yeah. - And so the problem is like what it's trying to demonstrate is like as models continue to work unattended in the same code base without knowing the whole problem up front, it's like, you know the whole problem up front, you can architect forward. But without knowing the whole problem up front, you have this problem where like the code gets worse and worse over time. And it kind of shows that like the number of failing tests just like trends up for all the models. And like I think GPT 5.5 codex when they ran it in May got like a 15% on this, yeah. - And I just ran it. I've been doing some like independent research 'cause they're, they're a lab they're not super funded. So we've been kind of sponsoring some research there. And we did it for Astra and Astra got like 16.5%. - Okay. - At one point higher. The ability to maintain like complex code bases over time. So anyway, we're gonna talk about that 'cause it used to be a vibe. And now they're like, okay this, I mean it's, you know, it's one benchmark out of one lab. It hasn't really been adopted by a lot of model cards. I think partially 'cause it's really slow and expensive to run. But I think it's a fascinating sort of, you know, proof that models are still not great at maintaining like a code base over time. - Do you expect that to change? That models will get better at code bases over time. Or will we solve that with regards to harness engineering? - Well, so here's the crazy thing is like, we used to do this thing in the like Cloud 3.5, 3.7, maybe like Opus 4 days where you'd say, like cool. We're just gonna, yes, I know the code is sloppy. But by the time that matters, that'll be GPT 7's problem. Well, we're on GPT 6, and we haven't made a lot of progress there. So it's like, you can't guess when that will happen. It will probably eventually happen, but it seems like the labs are much more interested in like hill climbing on other sorts of things, like computer use and other like frontiers that like the last 10% of software maintainability may be like as hard or harder than the whole first 90% that we spent the last three or four years on. - Yeah, interesting. What are some of the problems where you think the industry is headed? Like I heard a lot of conversations yesterday about remote agent execution, a lot about multiplayer. I still don't know exactly how that's gonna play out in practice, like the theory makes a lot of sense. - Yeah. - I get it running off my machine, get it running somewhere locally or centrally, where other colleagues can also look at the context and contribute in essence. But haven't seen a company kind of succeed with that. I feel like that's quite frontier stuff here. - Yeah, and I mean, a lot of the cloud agent vendors, like you look at cursor cloud agents and cognition and Devon and things like this, they've made good progress there. But I think the challenge, we work with a lot of like larger enterprises our whole thing is like, okay, if you have a hundred or more engineers, that's where like we have plenty of five person teams that love human layer. But if you have a hundred plus engineers, that's where human layer has like the biggest impact because it is all driven around this collaboration and those teams tend to hit a lot of friction when building with cloud agents because they have 10 years of history and hundreds of repos and lots of like opinions and asterisks and special things about how Devon environments are run and how can the agent test its work and things like this that become quite complex. So I do think the future is kind of remote background agents both for the collaboration angle and also for the like, hey, I want to be able to close my laptop and do this. Or it's like for small stuff, if I can kick off a copy change from Slack and I don't have the bother to develop her with it. Like, but I think setting up a like ticket to PR or like Slack to PR workflow is pretty straightforward for the small stuff, right? Like ramp and bricks and all these companies, they've all like published papers. I'm like, hey, we built this background agent and it solves 50% of our PRs or whatever. I actually think it's like not that exciting 'cause I've talked to people who've built that in, you know, a weekend or a hack week or something. So the bigger question that I think, I'm especially excited about trying to figure out a human layer is like, how do we enable people to collaborate meaningfully on the bigger pieces of work where it's like, yeah, I could tell the agent to do this thing and then I would get a PR out and then it would probably be somewhat right but it would have a bunch of issues and then I'm talking back and forth with the agent on PR comments or back in Slack thread or wherever it is. But it's like, okay, you went and wrote the code and it's not good and it's a lot harder to like, basically like fix what was already built. Agents kind of already committed to a thing. There's a lot of context to think through. And so our whole thing is like, what teams have been doing for 30 or 40 years back when building a feature took hours or days, not five minutes is they would like get together and they would align on what's being built either with the agent or with each other or both. The metaphorical whoever's actually gonna build the PR is gonna talk to the senior engineers that have been here and just like have a conversation and like, okay, here's the approaches we're thinking about and this usually happens in like Google Docs or Confluence or whatever. But this whole shift left idea of like can we, and like you're never gonna one shot it, right? There's no amount of effort where it's like, okay, we're gonna spend like five hours in a meeting building a perfect plan and then we're gonna give that to the agent and the thing that comes out is gonna be perfect. It's like no, there's still gonna be issues with it. Like you can't capture everything and the source of truth should be the code, not some specification or plan. But can you? in this planning process, eliminate as many failure modes as possible. Can you just, you know, from a 25,000-foot view of the problem, like a one-pager, can I eliminate like 50% of the bad things that the agent might have done? Not necessarily going to do, but could have done. And then can we zoom into, you know, the 10,000-foot view and eliminate another like 80% of bad solutions to this problem? So that by the end, when the code comes out, it's more likely to be, you know, 0 to 10% rework required rather than, you know, 25 to 50 to 75% rework, which is common with, like, if you try to one shot most larger agent PRs, like people don't just say, "Hey, Claude, go do this thing." They're sitting there working back and forth if it's large. You think you know a browser, but Gemini and Chrome, that's new. When you need to build up your team to handle the growing chaos at work, use indeed sponsor jobs. It gives your job posts the boost it needs to be seen, and helps reach people with the right skills, certifications, and more. Spend less time searching, and more time actually interviewing candidates who check all your boxes. Listeners of this show will get a $75 sponsor job credit and indeed.com/podcast. Need a hiring hero? This is a job for indeed sponsor jobs. Which ingredients make it a successful plan in the end because you mentioned, okay, what we've already been doing when features are kind of more expensive or more time intensive to build. We're doing that now more and more. I feel like leaning towards or experimenting with spectrum frameworks, which I think is good as well, but when we're talking about, okay, is this problem actually worth solving? Is that then part of the plan or how we solve it from an interface perspective? Is that part of the plan or more part of the spec? Where are the lines? Yeah, I mean, I think the words plan and spec are kind of mean different things to everybody. So I don't want to get up and be like, the spec is this, and here's what it looks like, and here's what it's for. And the plan is this, and here's how it should be formatted, and here's how it should be used. I think every team's going to kind of come to their own kind of taste on this. I think I have some opinions on what a good starting point would be, but we always, when we work with customers, we always end up like, okay, cool. Let's add a little bit of customization to your, we call it a design doc rather than a plan because it's like, we separate out like, okay, cool, where are we going? What's the end state versus like, how do we actually get there? What are the changes we're going to make? But our design doc often is like, we'll put some customization in there for users. We'll work kind of forward deploying with them and just say like, okay, cool, your design doc also needs these four questions answered at the top before we get into like everything else that it kind of comes by default. Yeah. I don't know if that really answers your question. I think personally does, as long as it's fine, right, if it's fine, that teams will eventually define what is the plan and what is the spec and what works for our environment rather than following some standard in essence, they have to be effective as a team, and I think it's a very context dependent, also organizationally, so in the end, I think that's fine. You mentioned human layers good when the engineers are like in the hundreds or even beyond at a bigger scale. And for me, I am wondering when a genetic engineering kind of breaks or at what scale, so if we're talking about 100 engineers or in my context, I have a thousand engineers, we're trying to figure out how to kind of enable people in terms of a genetic engineering. Yeah. The pain points I see is that we make models available, we make onboarding as seamless as possible, give people access to internal services through their agentic tools, and what I still have a challenge with and I don't know how to solve yet is how do I prevent everyone from using the biggest, most expensive model for everything, not just for where it's actually necessary, but for everything. Yeah. I mean, yeah, we work with plenty of teams where Fable has just been banned. Yeah. Like figure out how to make a work with Opus, and like you have to submit a special request for like temporary access to Fable for like a specific thing, but you've got to have to sell it. It's straight to GL Fable. Yeah. Yeah, I mean, I think the reason why we care about this space is one like I've been building software factories my whole career basically. For my first job, I was like, I was just a back in platform engineer and I immediately identified like, oh, that guy over there, like one of the first engineers of the company that maintains the deployment system and the rollout system and the staging environment and how like engineers create preview environments of like, hey, here's my feature work in progress. Like, hey, PM, here's a URL on the internal network. Like you can go look at what I'm working on kind of thing. That seemed to be the most like impactful person in the company. And that was kind of the work I was always drawn towards. And what we're finding is like a lot of tools work great in this space that if you're just building like a next JS app or you have a mono repo with everything in it, like there's a lot less friction, but I think like there's a lot more like flexibility and thought called for in the case where it's a larger team with a lot of baggage and opinions and dev environments are complicated and hundreds of repos and like, I don't know, we had a, we had a, I won't call it a complaint, but like a, a pain point that some people were having is like, well, we really want to get our, our PMs are using AI as well, product managers to build better PRDs. And of course, like someone who doesn't have a lot of skill with AI, they say go write this thing and they get a 10 page document out and they're like, amazing here, let me go give it to the team. The whole team is like, oh my God, what is this like AI psychosis slot that you're making me try to read through? Yeah. And part of that is just like giving the model better steering and skills of like, hey, here's how to keep it concise. Make sure you're writing for humans, not for whatever. And so make it clear and make it visual, but it's also it would find that a lot of the things proposed in these PRDs were so detached from reality in terms of like, how does the code base work and what are the systems? And so like a weird like tactical thing we found that works really, really well is like, give a product person who's technical enough to like, check out the repos on their box and things like this or like set up a GitHub MCP to go get repo contents. And have them, you know, drop in a two sentence prompt and then let let like a kind of pipeline take over like cool. Here's the thing we want to build go, go research everything, go find across all 200 repos, find the seven that matter to this product feature, pull all that context together in one thing, and then like interrogate the user, the product person in this case of like, cool, what problem are we solving? These are all the things that go into a good product requirements document, what problem are we solving, how are we going to measure success, like how are we going to know that it's working or if we should throw it out in a band in this thing, because everything, these bigger companies, everything kind of is an experiment, right? And then going through like cool, how should it work, should it work like this, should it work like this, sort of the like, grill me style thing. And so like this is just what example of like, that's not really built in. And like sometimes we'll build that internally and they'll come up with these things slowly over time and we're trying to help people do is like speed run that process. So if you don't have any opinions about how to build good product specs with AI, here's a thing that will work for, you know, 80% of teams, no matter how many repos you have, no matter how old your code base is. And so it's basically like, it's as much as anything it's about raising the floor, that's why like staff principal engineers that come to human layer and say like, oh, like, I like this. Like I can kind of see this is maps on to kind of how I was working, but I had no idea how I would ever be able to take how I was working and give it to everybody else on the team. Yeah. I like that. You highlight that the decision making in the end, that's still what a human does, right? You're doing all the context and it's baked in the harness or people are self-solutioning kind of something similar. And in the end, it tries to provide whoever is the user with all the context necessary to try and make decisions. And likely if they don't know the decision, they can still kind of spar ask questions or kind of in a real me way get to an answer in the end. Yeah. For me, when I heard that dark factories is kind of in my bubble, a lot of organizations are striving for dark factories. How do we get there? How do we do that? And I still don't know if that's like a realistic outcome, because especially when someone gives me a magical answer and it's not what I want, I'm just going to say, well, it's not what I want it. But if I created this magical answer based on my decision, like I'm building my own conviction that this is the answer. Yeah. Yeah. The buy-in is very different. I was sitting in the meeting yesterday. It's all hype. It's everyone wants to impress each other. Everyone wants to claim they figured out the dark factory thing. I think most of it is BS, sitting in a meeting yesterday with like the head of a department for a very old company, thousands and thousands of engineers and talking with kind of principal engineers on the ground who are like drowning in slop PRs from people who, you know, either just don't care, don't have a strong software engineering background. And he's like, so you're telling me that the code that is written by these people using AI is bad. And the two principal engineers are like, oh, yeah. And he pulls up a PR. It's like, look at this 30,000 lines of slop or whatever it is. Like, all right. Well, I got to go talk to my friends at like really big company and other really big company who have claimed that like 90% of this org's code is written by AI now. And I got to figure out what they're talking about because you're telling me that that's like completely infeasible and like, I have to figure out what they're doing that special or I have to like figure out why, what is the agenda? Anyways, the point is, like, there's this whole, like, I don't know, it's like a posturing game among all the top executives at all the top companies and everyone in big enterprise, everyone's like, am I 90th percentile? And this is like a thing that this conversation at every executive meeting that happens is people are just like everyone writes down their number of like where do you think we are from zero to a hundred among all other companies in our kind of zone or size? Yeah. I talked to on the best love today actually. Yeah. And he coined something which I'm going to steal. He said this is kind of the Instagram phase. Yeah. Right now. Everything is fake. Oh my god. I love it. Everything is bells and whistles. Yeah. And if you say, well, actually for me, it doesn't work. People say skill issue because how dare you kind of trying to disprove my accuracy of what I'm portraying online. This is incredible. I mean, a lot of it happens on Twitter. But yeah, it's the same. It's the same thing is like everyone is running around talking about how cool their factory is and how many things they have and how much adversarial review they're doing posting videos of the cool thing they built. And I don't know. For a while, there was a thing of like how many tokens they were. People have gone off on this a little bit, but it was like the token maxing thing of like, oh my god, I ran out six codex subscriptions in a week. Yeah. Isn't that amazing? And it's like, what did you ship, dude? And it's like, oh, I made like a bunch of games for myself that no one will ever play. And I'm like, yeah, so I thought I've also done that. It's fine. And it's trying to be like, hey, I made this cool thing check it out. I don't think it's okay to be like, if you're not backsing out six codex subs, then you're a bad engineer. I think that's like hurtful to the industry. And it's like, it's already, there's already a lot of noise and hype and confusion. And my, my sort of crusade has always been like, okay, what actually kind of works? What is practical? And like, what can you, what understanding can you build up from like first principles, rather than like working backwards from what you see on the internet? Yeah. If we give productivity a multiplier. Yeah. Is 10x then realistic? Or when should people be happy? Is it like 2x, 5x, 3x? The other number? I mean, if you think about it, like in terms of like economic impact in the world, I think every engineer going 2 to 4x faster would be transformational to human society. The amount of value that we would get the extent to which our cities would improve our, you know, the value of things would improve, like just life would get better would be insane. But everyone's going to miss out on that because of too busy trying to get to 100x. Yeah. And I think 10x is possible. Sorry, what are you going to say? I was saying people are trying to go infinite. We don't need engineers anymore. Yeah. People are going to go infinite or 100x or 10x or whatever it is. And it's like, figure out how to go 2 to 3 times faster. And then you can go 2 to 3 times on top of that. And 2 to 3 times on that. But if you try to throw out everything we've done, I mean, there's something to be said for like, don't get stuck and shackled to your traditions and like some things that you used to do may not be necessary anymore. But if you go and throw out everything that we've been doing and try to like rebuild all of engineering from scratch, not going to work, it's just like, and when an engineer sets to do a project and they're like, oh, yes, I'll architect it to be able to do this, this, this, this, and this. When all you needed was this one small thing, but what is it that jagged me? You ain't going to need it? It's the same thing. It's just the same story over again as like, people want to have this perfect band. And there is a time when you're like, cool, we've kind of like taped this together enough. But now we understand the problem. Now we can go build the next thing. And too many people are trying to skip that step. Trying to skip the like, okay, let's just get faster and faster and faster and incrementally over time because that compounds exponentially. But I was talking to Daria this morning. And he said from his perspective, what makes his team at AMP really productive, has nothing to do with AI, is they don't do pull requests anymore. Yeah, I heard about this. Yeah, everyone just pushes domain. Exactly. So no pull requests fatigue. No, no one wants to do code reviews anymore. Still do code reviews. I'm like, huh, maybe it's time for me to run an experiment to see if it's actually now's the time to do something like that. I think it's worth trying. I mean, the thing about pull requests is like every team's pull request culture is different. Like on my first job, everyone had merge on main. And there was no pull request requirement that was enforced. It was a cultural thing. It was like, you don't merge without getting a review from somebody else on the team. And then culture beats process every single time. And so if you have a team where like, okay, we have a cultural thing of like, you don't push slop to main. And it's like, you don't, you don't shirk responsive. Well, the reviewer reviewed it too. So we're both guilty if it's bad. It's like, no, you own it. You are the only person who owns it. And if someone finds it and like realizes that it's bad or you realize that it's bad later, like it's still on you. It's just, uh, we, you know, we're changing, we're changing the process of like, how do we guarantee quality and what are the outcomes and how do we manage risks and things like this? I feel like to get to that culture, that could be a fun experiment, right? You remove the pull requests and see if, because you're doing that, you can get to a culture of, we own what we push to production or to main. Yeah. We don't push any slop, especially with agents. How easy it is nowadays. You take ownership. See if that kind of is a cycle that then starts the role. Do you did, did he mention like, is, is it that pull requests aren't required or like pull requests are not just not afraid? This is killed. Because I could, I could see a world where like, okay, look, I don't require review, but I want a review on. Yeah. Yeah. We have this for doc review, right? Like, I don't recommend anybody say like, oh, design doc reviews are required. For you write the code, if you didn't get through design rock doc review, the PR is not relevant. And it's like, no, I'm incentivized to send my co-founder, who is the CTO is like the kind of slop, the slop gatekeeper and make sure that like the code base stays maintainable over time and things like this. I send him my design docs because I know it's a lot faster for him to be like, no, no, no, on, it's easier for him and easier for me for us to have that discussion on a one page or a two-pager. And then when we get to PR time, like, after I've polished it and I've got the test working and I've maybe like done some stuff that wasn't in the plan of like, okay, let's just change this UI and like riff a little bit over here. And then he says, oh, this entire architecture is wrong or like, I don't like that this was a separate table and it should be this instead. Like, I think there are incentives to get feedback from people as early in the process as possible. And like, it's easier to fix things before you merge it than after. Yeah, it's in there. Then it's like, okay, cool. Now we're like fixing bugs in production. Yeah, 100%. How's your thinking evolved when people said we're no longer looking at the code or doing code review specifically? We have so many deterministic tools that look at code conventions and now we even have built-in verification layers where the agent just sends me screenshots or sends me videos about the functionality that it's being built. So I don't necessarily look at the code anymore. Well, there's two reasons to look at the code. One is to make sure it's correct and one is to make sure it's maintainable. And it does a bunch of other reasons. It's like alignment with your team and things like this. But like in that world, like the video and the screenshots tell you whether it's correct in terms of like the features there doesn't tell me anything about whether the code is maintainable. And I have seen many, many deterministic editors. We use a bunch of them. I'm going to talk about some of them actually tomorrow or my talk that help like block large categories of slop. And you can do things like psychoanalytic complexity that is like tries to like keep you from creating functions that are too chaotic and hard to grok for a human. But I have never seen a prompt or a deterministic tool that can like drive code-based maintainability and at the beginning like fight the model's tendency to write un maintainable code in any meaningful way without a human in the loop. Do you think we can close that gap with deterministic tooling? You think you know a browser. But Gemini and Chrome? That's new. It can help you with practically anything on the web. Like restoring a vintage motorcycle from a 50 page restoration block or finally break down that long article you've had open for weeks. When you need to build up your team to handle the growing chaos at work, use Indeed Sponsored Jobs. Listeners of this show will get a $75 sponsor job credit at ND.com/podcast. That's ND.com/podcast. Terms and conditions apply. Need a hiring hero? That would be, you would make a lot of money if you could figure that out. And to this is an interesting company in that space because they're doing deterministic tooling for correctness, but not for maintainability. No, but you won't maintainability. You won't maintainability. I use MapHawk has a skill called improved code based architecture, which I use quite a bit. And it does help. And there are times where I'll just run this skill and then like cool, I'm not even going to read the discussion. I'm just going to make the first three fixes there because I'm sure it's probably better than what was there in the first place. But I think in general like that, the thing that makes that skill useful is having a human in the loop view. Ability to be like, no, I don't like, I don't want to refactor it that way. I want to refactor it this way. And like, I think humans are good at this. The skilled experience offer engineers are good at this because they have like been bitten by bad architecture so many. They've been up at three in the morning trying to fix an issue and been like, oh my god, I hate this pattern. And you do that hundreds and hundreds of times of the course of five, 10 year career. You build up this taste. And we haven't figured out how to put that taste in the models. And so for now, humans have to do it. We might try to figure it out. Literally the last model came out. The asterisk came out. And my buddy Ian, who ran a built a coding agent startup that got acquired by Expo, he posted, he's like, I am begging the RL researchers at the lab. Please start including tasks that have like, deletion of code as the goal. Or like, hey, let's take these three things and refactor out a good abstraction. Like it's just so obvious that like either we don't know how to do it or we don't care to do it in the in the RL like layer So I it could get cracked. I mean anything can happen But again also is like cool. There's all this hype and what is the future and no one knows when it's gonna happen and my advice is always like Do not mortgage your code base in the hopes that in a year or six months or 18 months a model will come out That's smart enough to fix all your problems. Yeah Someone listening might think I fully agree with the taste aspect But I don't have 10 years under my belt. I don't have 15 years under my belt pre-agentic tools or pre-agentic era How do people still build taste from your perspective nowadays and how can they do that? I mean, I think I think it comes from from burning yourself and stubbing or tell I think it can also come through mentorship and this is a thing I think about a lot and I don't have a good answer for which is like how do we bring on the next generation of software engineers? It used to be through pull request review. Yeah, right? It's like and like you think about like the goal of it The worst pull request comment is like this is wrong. I rewrote it for you. Here you go like the person learns nothing And they learn that if they write bad code someone will just come fix it for them Yeah, so a good pull request for a good senior engineer manager will be like Hey, I don't think this is quite right. Can you like look at how we do this over here? Yeah, and like do that pattern or why did you do this? Yeah, like try and probe exactly what's the logic behind this? Why why not you know whatever whether you give them the answer or not is like okay? Why do we do this? Can you is there any other places in the code base word and that forces you to learn right? You have to go read other code either. Oh, that's okay. Cool. That's cool. I like that and it like you reinforce And now if you give comments like that that used to be really helpful They're not because I I mean I talk to people all the time and we're like yeah I'm coming on this PR and I know that the person is not they don't understand the code they wrote Mm-hmm, and they're not really going to understand the comment and they're just gonna paste the comments to their AI and it's gonna Make the fixes and at that point like I'm doing every part of your job that matters You didn't think about or understand the code before you committed it and shipped it You just love the agent do it and then like you're not even thinking and under you have the option some people will But like you have the option to sometimes it just feels like I'm just talking to your agent Yeah, at which point like I might as well just have my agent do it zero value on top. Yeah So I don't know. I've seen Mitchell Hashimoto do this actually a couple times on on ghost DPR's where he doesn't even say Hey, this needs to be like that over there. He says ask your clawed why this is wrong That's it. He gives no hints or anything. Yeah, he's just like I will tell you that this is wrong I won't tell you why I'm forcing you to whether by understanding the software and building taste and software engineering or by Understanding agent to coding and being good at using agents to understand the code base whatever it is like you need to go You need to go do some more homework. Yeah, so I think I think that's Directually correct. Mm-hmm. I also think there's going to be a lot more the same way we collaborate on code and pull requests right now as the as as design docs and plans become more Standardized and formalized or at least their purpose parts of them become more universal There will be a lot of training and coaching that happens during design Mm-hmm. Because design doc is basically a prompt and so two people five people commenting on a design doc You're teaching the author how to prompt better. It's like okay cool. Like I don't like this section like you know You need to go find other patterns or whatever it is and that there's still a flavor of this word It's like okay. I'm just gonna give it to my agent and we're gonna like I'm not gonna learn anything But like I think like learning how to build a really good design doc Which again learning how to build a really good prompt for the whatever session is going to be building or in that session is Going to emerge as well as like just as important if not more important than code review I feel like I learned a lot through kind of defining solutions in a team Early in my career because I came from not a traditional computer science education I got rolled into a team because I said I want to do engineering not just operations because I was responsible for operations And then I learned the process of refinement and in that team specifically we had the convention of we also define the solution direction Not always but most of the time and sometimes as part of executing on the work that needs to be done The person executing with define the solution direction and align on the on the way But for the ones where we did that in advance I love just listening and learning and asking questions on why x y and z and Thinking back now likely whatever we captured back then in jira would have been great food for an agent to then execute on And even the person picking up that task of the work that needed to be done didn't have to do too much thinking anymore of exactly Kind of how to execute or what to execute maybe how still because you always have new findings But mainly why and what was answered which I loved and I learned a lot from Now it's kind of up to the individual I feel like and my fear is that because it's so easy to be on an island with your agent and go into Execution mode create slop or create something valuable. It's really up to the user that you lose that that kind of collaborative as effort and also interaction I feel like we'll miss that if we don't put extra effort in that Yeah, I mean we're talking about building a feature in the human layer where you can you can call someone on your team and do a voice call And then the transcript becomes an artifact attached to the task Because that's the last that's the last piece, right? But I think the AI native forage is much more than GitHub that manages code. It's going to be the code It's going to be the artifacts and it's going to be the sessions themselves all kind of linked together at a like the top level Yep, and the last part is like the human conversation and so like maybe that's that's part number four is like can you rayify those things for the agent But yeah, totally agree is like a lot of the learning happens in those Human conversations with the people who have that learning and like if we can't rely on the model To absorb that taste that Things and we have to find other ways to give them to people. Yeah As a last thought I wanted to zoom in on kind of what your agentic engineering workflow looks like nowadays Yeah, because I feel that people listening they might see something online all Instagrammified I'm gonna keep using that and keep echoing that because I think it's quite accurate and if they ask a question So I might say a scale issue. I have no clue. What's real? What's not real? What works for you nowadays? Yeah, so we do and I did a pretty fairly deep breakdown on this on I'll give you a link to put in the show notes Or whatever, but I basically have a page that is all the podcasts We over all the all the written content and video content we've done and I've done a couple deep dives on our process with screen share and stuff Awesome, I would say probably at least 50 to 60% of our work is like basically one-shotted Not one-shotted like two perfection, but like we don't do any planning upfront and it's literally just like If it's gonna be under a couple hundred lines then um, don't we just do that about 10% of our PRs are merged by You know, I call them like janitor bots or whatever that run overnight my co-founder gave a talk on like control loops And how we build all of this, but we obviously have like Things that we want to be true about the code base like desired end state And then we have like a process to progress the code base that direction. It's kind of like a thermostat, right? Okay, I wanted to be 71 or whatever the European version of 71 is uh, and it's 69 cool. We're gonna turn on the heat Um, and so it's like we wake up every morning to a handful of PRs along different dimensions uh Of cleaning up something in the code base refactoring fixing a reactor rule migrating an endpoint from the old framework to the new one Everything we do automatically in the factory is like very small because wanted to be digestible if the PR is too big and never gets merged Um, so that's probably like 10% of our PRs again about 50% are just like hey, I want to make this small fix go do it and then I'll test it and maybe like send a couple more prompts on it Um, does your day then also start wake up look at the PRs that ran overnight were created then yep Okay, we don't merge them every day. It has like a can-man thing We're like if it doesn't get merged the the thing doesn't run the next day. Oh, sure There's always only for each type of PR of which there were like three or four or five maybe six now Um, there's only at max one open at any time um, and then we use labels to basically say like okay If there's already hope we PR with that label don't try to fix the thing because you might might just try to fix the same thing again. Yeah Um, and then probably about 30% of our 30% of our like code shipped It's probably fewer PRs, but about 30% of what we ship is done doing Through a like specific process that we use called RPI or like the new version of it is like crispy. Although like now it's like QR Dio i I don't know anyways. We've lost uh, I stopped using acronyms, but it's all I would count it all in like the RPI family Which is like go do a bunch of code based research um take what the user asks for go build up query plan Then do another session that doesn't know what the ticket was have that do the research and then like one or two or three sets of like planning sessions If it's a very ambiguous like large thing. We'll do like a product requirements first Otherwise, we'll get straight into the technical design and then if it's fairly small If it's like it's it's too big to one shot because it just needs a lot of context from a lot of places But it's like fairly small we might skip the actual like here's the steps. It's like cool. Just go implement this design Or if it's quite large and I want to check along the way. I'll work with a model in a third type of document It is like breaking down what order are we going to do this in yeah And like how am I going to check it along the way because like it's a lot easier to ship 10,000 lines of code if you check it every thousand lines And like before it gets out because if you get to 10,000 lines of code and it's broken Then you have a massive surface area to debug across But if you can get it you can get a 90% in the first thousand and like fix that and then do the next thousand Okay, that's okay. Then the next thousand fixed 10% it's like much less overall work And I think much less mental effort as you go you were like check pointing along the way To get to the final solution. Sometimes we even do them as individual PRs like cool What's the plan for stacked PRs on this and like what's the first merchant bit that we can do yeah And then we go implement and like use we use sub agents. We've sometimes we'll use like, you know, fable orchestrating soul or we use soul orchestrating fable or we use just Opus as the main thing and Opus as the sub agent. Like, kind of depends on the task. I use soul for almost everything now. - So you use your chooses? - Yeah, the user chooses what the parent model is and then there's a couple. There's a couple like less obvious hacks you can do to do like cross provider. Like if you want fable to orchestrate soul sub agents, you have to tell it to use the code XCLI right now. We do have a custom harness that we'll support that soon 'cause we just think it's a cool pattern that works well. But yeah, and then from implementation, you know, it's review the code as you go or review the code at the end. Make sure it's good. We have a kind of an internal cultural role is like do not send anyone code for review that you yourself have not read. Like do a pass on it. Don't just make me read your agent's lock. Sometimes I read the code and I have to change 50% of it. Sometimes I have to read the code and I actually throw it all out and go back to the design 'cause I'm like, oh, it's Kyle's time. We're gonna approve this. They're like, okay, what's a simpler way to do this? - Yeah. - And sometimes I read the code and it's perfect. - Yeah, I think people underestimate that you can just throw out the output and try again. - Well, that's why also I love this like incremental like zooming in planning process, like product, tech, and then like the plan of the steps that we're gonna do because at any point, you're getting more and more specific. At any point, you can be like, oh, I don't like the, okay, throw this out, we're gonna go back to the design, we're gonna change some stuff because if we have to do that, then I don't wanna do it. The oracy world where, because this is also kind of in my bubble what people that are in leadership teams are envisioning, that a lot of the day is working on more of the planning or specification, how that is defined within the org or the team, and then overnight the work executes. - Yeah, I mean, I just, I don't wanna wake up to 10,000 lines of code, just like not, like if I'm, if I'm planning on reading the code, I think the work is a lot more human in the loop than that. - Yeah. - They're very specific things that lend themselves to hill climbing overnight. But I think a lot of the work you do, you're just gonna, you're going to enjoy it more, you're going to get more done, you're gonna actually gonna have more through, because that also requires you to do a ton of paralyzing. - Of course, it's like, plan 10 things and then kick them all off overnight. I know what I'm gonna spend the first four hours of my day reviewing code. Like, no, I'm gonna plan two things. I'm gonna start them now. I'm gonna like watch them as they go and I'm gonna get changes landed in production today. It's much better to do two things, two things, two things, then like, okay, I have 10 things in flight and they're all kind of going. And oh, I forgot about two of them. And by four days later, I did ship two of them, but the other three are still in flight. It's just the same thing that was always true about engineering is still true and all work. It's like, do less things in parallel. It feels more efficient to like, I don't know how to say like to, to like try to like multitask really hard, but you're end to end throughput of like value delivered to the customer will be faster if you do less things like a little bit slower and just like take a little bit more care with them. - Interesting. I wonder if the way we get access to models, like if I look at myself, my own license with cloud code, I spend 200 bucks a month, at least. I have weekly limits for my models. Able has a little bar that I want four or close, otherwise I don't feel like I squeezed enough juice out of my license. So when it's Friday and I know it's Saturday and it's reset time, and I still have a lot of work to do, that's really when I kick out these night processes where sometimes I don't really care about the code. I want not parallelize, I want to maximize my usage. - Yeah. - I wonder if that way of thinking or that way of execution has also primed then and kind of catapulted us into, maybe this is what the future of engineering is going to look like. The only way we got here is because of these limitations on how we can access models in the first place. - And this is also the biggest disconnect of everyone who's posting their maxed out all six subs on Twitter versus the real world is like no enterprise has subscription plans. They're all paying for token. And so there's this whole cultural thing of like, if you can get all $3,000 of spend out of your $200 thing, then you're winning, I don't know what you built and what value it is and who's going to use it, but that's how you win. And it's like, no, the skill issue is not, couldn't use enough tokens. Because that doesn't apply to most people who are doing serious work. We have companies like customers with 15 engineers and they are all on non-subscription plans because they need security requirements and you need data retention requirements and you all this stuff that you can only get on an enterprise plan, which means you can only get paying for token. So yeah, I think that we're going to look back on the era of subscription maxing. I don't know if it's like, you know, the intelligence is going to get too cheap to meter, it's going to get so cheap that they won't need subscriptions. You'll just pay for it and it'll be good value. Or the subscriptions are going to go away at some point because all it is is a loss leader and like a way for them to get data on you and like train the next model. Of course. So I think we'll probably look back and we're like, oh yeah, all that token maxing stuff was just because of how the model, I think you're right. It's like because of how we access the models, because of how they're distributed to like, especially like independent hackers and builders that it became this thing that seemed much more important and got a lot of attention and then one day it was just like nobody cared. - This episode is brought to you by Google Chrome. Ready to make anything online make sense? That's nd.com/podcast, terms and conditions apply. You need a hiring hero? - Yeah, everyone, it feels like there's a lot of FOMO if you don't max out whatever plan you have. And the best way to do so if I work during the day is gonna be that I do my thinking and I let it run overnight. - Yep. - But I don't think that's how actually I want my day to also be. - I do think there's value in blocks of thinking and blocks of coding, right? My thought is when you run a human layer, it does this research step. It's slow, it's in a large code base, it can take 10, 15, 20 minutes. And so I might kick off two or three of those things and let it run and go get a cup of coffee or something. But it's like rather than me going back and forth with the model and tell it, okay, now go look over here. Okay, now go look and then I notice, oh, you didn't understand that thing. Yeah, we also have to go look over there and see how that works. And like you're constantly like catching the models lack of understanding versus like, I'm willing to like async for 15, 20 minutes and just guarantee that it's like gonna have 90% and 99% of the context that it needs to do this task almost certainly and anything that it missed is like very easy to go find. That is worth a little bit of async. And then while those are running, I'm like reading another plan and going back and forth on that. But I try not to paralyze too hard. And when I do paralyze, I have one most important thing and it's not like, okay, launch three tasks. Okay, get this one to design. Now get this one to design. Okay, now move this one forward. Now move this, it's like when the top, when the most important thing is unblocked, we drop whatever we're doing even if it's like, I'm in the middle of writing a prompt, I will go back to the most important one because the goals should be like land one thing. You should always have one most important thing and your top goal is to get that thing landed. And the other stuff you're working on is just like stuff to fill the time while you're waiting for that thing to be unblocked. - Yeah, you should always know what the most important thing is. - As has always been true at engineering. And everything really. - The feeding of parallelization is very cool though. And I haven't done any measurements. But overall, if you look at the most important thing that needs to go out and you parallelize along the way, I wonder if that thing would go faster if you just wait and just focus on that one thing. - I wanna like traditionally. - I wanna see a study, it would be very sad. - Yeah. - I wanna see a study of like, you know, they hook up like the brain scans and like watch someone's brain as it's working. I wanna see someone running six cloths in parallel. I wanna see one person running one coding agent at a time. And then I wanna see someone sitting in a casino cranking the thing. And I wanna see. - Slot machine, yeah, exactly. And then maybe the person doing Slot machine and then maybe someone else at like the poker table. Whereas like a poker, you play one hand at a time. Any focus. - Yeah, good stuff. Thanks so much for coming on Dex. This was great. - Yeah man, this is super fun. I'm really excited for the show. I think you've had some really dope people on here. So hopefully, hopefully I don't tank your rap man. (laughing) - Should be good, thanks, thanks again. - Patrick, that's it. Your heart can tell you a lot about your health. Apple Watch Series 12 measures your heart rate every five seconds with the most accurate heart rate sensing and awareable. So your vital zap now with heart rate variability can tell you when something is off. And your readiness score can let you know when to rest and when to push. Here are the story in every heart beat with Apple Watch Series 12. The features described it for wellness purposes only and not for medical use. iPhone 11 or later required. Based on Apple conducted study of heart rate accuracy August 2026. Visit Apple.com/AppleWatchSeries12.