33: Ship Happens (When Agents Code)

A lot of these migrations, every time a human would run it, you would expect the same output and we expect the same thing from our LLM. So it becomes like useful for us to use something like an LLM as a judge. And when I say LLM as a judge, what I mean is we take the generated code of the agent and we compare it to our initial prompt. And we basically have a different model, evaluate whether the generated code changes actually do the thing that we requested in the prompt. Welcome to Nerd Out at Spotify, where we bring you behind the curtain of the world's most popular audio streaming subscription service. Machine learning, open source, clouds, tabs versus spaces. We'll talk to Spotify engineers about interesting tech issues, big and small. I'm Dave Zolotuski, principal engineer at Spotify. Today, we're diving into something I'm really excited about, background coding agents. Build on top of our fleet management system, these LLM powered helpers have started transforming how we work by taking on increasingly complex code changes, dependency bumps, migrations, cleanup work, all the tedious error prone stuff that we used to do by hand. Across Spotify, we emerged over 2,000 AI generated pull requests from these agents. That's given us a front row seat to what works? What doesn't? And what happens when you ask an LLM to make changes to production code at scale? Joining me are three engineers who have spent the last few months deep in this space. Joe Kelly Fenton, Alex Mittich, and Max Carris from Spotify's core infrastructure team. They have been wiring multiple LLM's and Anthropics Cloud code into our tooling, making sure things stay safe and reliable and wrestling with all the fun failure modes you get. Like when an agent decides that the best way to fix a failing test is to just delete it. We're getting into what these agents actually are, or probably solve, what makes them tricky, and how they're already changing the way engineers that Spotify ship and maintain code. I'm super happy to have all of you on today. I guess let's start just by introducing yourselves and getting to know each other. Joe, let's start with you. Can you tell me a little about your old Spotify, what you do here, how long you've been at Spotify, things like that? I'm Joe. I'm an engineer in the core infrastructure team. I've been here for a little over a year. And I'm working on the background coding agents for the last eight months or so with Max and Alex. Hello, everyone. I'm Alex. I've been at Spotify just a bit more than four years at this point. All my time here has been spent in core infrastructure or main infrastructure or working with various projects. Most recently, with the gang here, on background coding agents. And Max, I'm guessing everyone listened to all of our other episodes and all who knows you, but just in case someone missed one, can you tell us about yourself? Of course, hello Dave. Max cares here, also engineer, also in core infrastructure, and have also been working on these background coding agents for eight nine months now. I see. So the group we've been hanging out a lot for the past eight to nine months. We have, yes. All right, so tell me a little bit more about these background coding agents and what they are and why we care and why so many have been working on them for so long. We have a system internally called fleet management, which allows us to do code changes on many repositories and software components throughout the company. So if we want to say, do a dependency update in a pump file, a maven pump file, then we have the system where we can essentially write a little script and it says update whatever. Let's pick something like SLF for J from version X to version Y. And then we can run this over our entire fleet. And it's been great, I think close to over half of our PRs. Now are using this system. So we're doing a lot of automated dependency updates. But actually writing these scripts is complicated. It sounds easy when you just have to parse an XML file. There are many details. They can be malformed. And when you have thousands of repositories that become hard, we thought in the beginning, maybe we could just add an element to this. So that's how we started. Tell me more about that. You said that XML files is a little bit easier. Something something, other stuff, LLM fixes other things. So I guess why are XML files easy? And then maybe that'll tell me why other things are hard. And then we need to do something, something LLM to fix it. Different file formats have different difficulty parsing them. So actually XML is a good example of a file that's pretty easy to parse because it hopefully has a pretty good schema that you can parse and update. But it turns out when you have thousands and thousands of components, even well-defined things can be malformed, or indentation can be different. And there are small variations that when you do an update over these thousands of components, it becomes very hard. And if we take a more difficult example, actually updating code is very difficult because then you actually have to parse the code itself to update code. So over thousands of repositories, there are so many variations that you have to think about. It's actually writing at their deterministic script, like a small shell script or a Python script, or actually a Java program that parses the input files like a program itself can be very time-consuming and hard. And when you do it locally on your computer with a single example, it turns out it's pretty easy. When you do it for two things, it might still be easy. But when you hit a certain inflection point and just scale, then it becomes very hard. And there are so many corner cases and weird things that you have to care about. LLMs are just good at this. They're good at interpreting fuzzy information with text and being able to do changes based on just a simple text instruction instead of having to write a complicated program to do it. So I guess before we get into the LLM part and all of the really hard cases, you said that's something like half of all of our PRs are not created by fleet management. Tell me a little more about how well this thing works and what we've got today, or maybe slightly pre-LLME. So Max mentioned this fleet management program we have. And this has been the foundation for a lot of the work we have done through background coding engines. It's this system that allows us to scale any transformation we want to do to thousands and thousands of repositories. So take the example of bumping dependencies. This is a very common task that we have to do all of the time. So what the system allows us to write they script that finds the dependencies, bumps them to the right version, and we can target, let's say, all Java components as Spotify, which is thousands of them. The system would then spin up a job, a Kubernetes job that clones the repository, runs this transformation, and produces a PR. In most cases, when you bump a dependency, there's not much more that needs to be done than just bumping the version number. So this is a PR that, in many cases, we can automatically merge. So if you put these things together, what is different now is that script is an LLME, the system that allows us to create thousands of PRs. In cases where it's mostly producing valid changes that we can auto merge, you quickly get to these numbers of 50% or more at the scale of Spotify. Many of these changes right now are quite simple. Like, there's small configuration changes. But very important, that dependency bump is very important for our security posture and stuff. But they're simple in nature. The LLMs allow us to tackle more complicated changes simply. That sounds great for a bunch of these simpler changes that maybe are just kind of a version bump or a simpler thing. But what are the problems that you had with that system that you now want LLMs for to fix? We have a lot of migrations in core infra, right? So we take a lot of migration pain from our customers. We manage a lot of these more complicated version bumps as well, like we are in a lot of the third-party dependencies. And so we're constantly doing migrations. And an example of a migration that might be a little bit too complicated to try to encode in a deterministic script is maybe a bumping of a library that has a breaking change in the signature. Because now you have to try to get the application logic. We have to find a variable now to plum through to this new signature method. And so what we're finding is that people-- they're finding it a lot easier to actually use fleet management with an LLM because the barrier to entry to actually run these migrations is a lot lower. Figuring out how to write these scripts that work is really hard. Yeah, so tell me a little bit more about that. What are some of the models we've used or some of the kind of interesting Genai tools we've been using? And are there ones in particular that have worked really well? Or like step changes you've seen when switching models? Oh, yeah, for sure. So we started. When we started using Claude code, we noticed a massive step change in our performance. For example, it was way better at managing the context window. It did a lot better targeted file searches. And it basically was really great at planning how to tackle a migration up front. And so a lot of these issues that we had early on went away when we started using Claude code, which allowed us to focus a lot more on things like the verification system. With the settings file, you can manage the security aspect. The context window is kind of handled for you. You can use sub agents or things like that to delegate some of the context window to different agents. And we could focus on the things that were more on our end, which was verifying that the code changes are correct. So what sort of things have we been able to do now that we hadn't before? Just recently, we worked on a tech health project, which we called our context prop migration. And so that basically meant that we had to plumb some context through from our services to our clients. And so this meant that we had to update a whole call chain throughout our services and plumbed these objects through. And there was a bunch of edge cases. Maybe you would handle the context a certain way if it was a Pub/Sub handler. Maybe you would handle it a different way if it's an RPC handler. And so it got pretty complicated. So we ran an experiment. Can we actually do this faster than what we can do when we're trying to do these changes manually? We actually ended up getting, I think, 160 of these changes through where we migrated 160 different services to this new context framework, basically. Let's just talk through an example of some of the much more complicated things we can now do with an LLM, as opposed to the simpler dependency bump type things that we could do with just standard fleet shift. Do you have a good example we can talk through? Before in the old world, where we had to write scripts to do the code changes, we were using a open source B64 library, and now we moved into the JDK actually provides a B64 library. So we wrote a prompt to the LLM to migrate from this open source library to the internal JDK one. So that requires us-- if you think about a classical Java, a maven project or a basal project requires you to remove a dependency. But only if it's not used anymore. So in this case, you have to remove the dependency, remove all occurrences of its usage, in this case, the B64. But maybe there are other methods that are non-B64 related that are used from this dependency. So you would kind of, as a human, you would rewrite the code for the occurrences where the B64 code is used to this new built-in method. And then you would probably try to remove the dependency and see what happens and recompile. And if everything is good, and you just assume that this dependency is not used, or you have some kind of automatic mechanism to discover that. But all of these invariance and specific corner cases are very hard to encode when you write a program. But the agent can mimic the behavior of a human. So we can rewrite the code, and then it can try to remove the dependency. It can try to compile. And in the end, if it's happy with the result, then the system then opens a PR. I see that. So you don't have to be nearly as perfect in the explanation of all the things that have happened in these kind of much more slightly varying in every code-based type scenarios, and then the L and could figure a lot of that out. Yes. And specifically, what we do is we provide, like, before and after examples. That's essentially the granularity that you have to describe. It's like, this is how it used to look before. This is how it should look afterwards. And then you give that to the LLAM. Like Alex would give this task to me, or I would give this task to Joe, is the same thing. This is how it used to look before. And this is how it should look afterwards. And these are the thousand repos that you should go and do this on. And then we just say, yeah. I just maybe tell me a little bit more about that. So you-- I guess you're just prompting the thing. So it's a matter of figuring out what these prompts are. And so you're saying what before and after, you're really just giving it little bits of code. So normally the development flow goes like this. You start with a very simple prompt that says, move from this library to this library. Here are some examples. And then you run this over 1,000 repos. And you don't even create PRs. You just get information back how the change went. And you look at these changes, you squint. Then you say, oh, you know, now 30% of them pass. And the 70% of them don't pass. And then you look at this 70% cohort. And you try to figure out what's wrong. And in many cases, you can find a specific corner case where the agent is confused. So you find that corner case. You try to then rewrite the prompt to capture that corner case, put it in the prompt and rerun it again. And then maybe you get up to 70% and then you do the same thing. And you gradually increase until you're roughly happy. And the agent understands what to do with most of your cases. That sounds super cool. And I guess before I get into a bunch of the quick thoughts that are in my head, I'll just more open-endedly ask, what are the problems with this thing? It sounds magical and I'm sure it's not. Yeah, I mean, at every level of our development, there's new unique problems that arise. One was security. So it's different if we take a step back when we're running our agent in Fleet Shift, it's fully autonomous, right? So it's a lot different to running it locally, like our local developers might use their various products. And the thing about it running autonomously is that we need to make sure that it's not going off and executing a bunch of random commands and pushing to get master or something ridiculous. For example, when we first started hacking around with this, we had a pretty vanilla container and we found that our LLM would just start installing dependencies, left right in center to just get things to work. Pretty quickly, we realized we had to lock it down. We made sure that it was run as a non-root user. We very much restricted the types of commands that our agent could have access to. And then the second problem that we had, this was, I think, more of a problem early on before, I mean, our agents are getting more and more sophisticated now. But in the early days, we would pass a prompt to our agent and say maybe find references to these libraries and update them in the code. And the agent would just literally pass the entire repo. And then we would find that our context window was filling up and the agent became quickly confused and unhappy. So in the early days, we spent a lot of time fine tuning, like how we can reduce the context window that we used to run our LLM. And then like in our local development case, when we write code, we test it, we compile it, we run our linters, and then if there's any issues, we feed that back into our code and we make it better. And so we wanted to basically try to reproduce this feedback loop with our autonomous agent so that we could reduce the types of failures that we might see in a PR. So our agent was able to actually verify the changes by running the tests. And if a test failed, it could feed that back into the agent, the agent would improve and go again until it worked. - How hard was it even to get the agent to be able to do a lot of the things that you, as a human, just simply do on your local dev environment, like run tests and things like that? - One initial approach was to let the agent do its own verification. So it has access to the command line. You can try running basal test or whatever build system is there. That sometimes work, sometimes it doesn't. We've seen funny cases where the agent tries running the verification two times and when it doesn't succeed, it just proclaims success regardless and continues. It says I'm done, it's all great, right? So I do that, yeah. You do that very often, man. I think the key realization for us was, we have this principle we're trying to operate under, which is that we wanna verify all dimensions of correctness automatically. So a clear one is this natural engineering process we're talking about of writing code, running the tests and then iterating. So we're making sure that the LLAM runs in this feedback loop always, and there are ways to solve this. A lot of the popular agent coding tools have concepts like hooks or other deterministic interceptors basically you can run that ensure something happens. So we spend a lot of time guaranteeing that our agent will write some code and then we will always run a build, a test, a compile cycle, feed that feedback back to the agent and let it iterate. And there are, of course, cases where it cannot solve that and that's fine, but there is a value in knowing that we were not able to produce a succeeding build because we then we don't have to open a PR, we don't have to show this to a user and we prevent this AI fatigue. But in many cases, you can run two, three cycles and then produce a good peer. - So one kind of quick thing I wanna get back to you, I think Joe, you mentioned context windows a few times and I think you talked about decrease in performance as a result of issues with context windows. Can you just give me a quick, what is a context window and what were the issues you're in and do and why is that a problem with this sort of thing? - All of our agents models, they basically can work at a max number of tokens. When the tokens exceed some limit, we reach a point where we've basically sent too many words back and forth from the agent and it can break, but there's also like a performance degradation that starts to happen as you start to approach this max number of characters that the LLM can actually handle. So we don't wanna reach the point at which the model breaks, but we also just want to make sure that we're only using exactly what is needed to get to the coding solution. So an example of this is when we run our tests, I don't know if anyone's seen a build output, but I assume if you're listening to this broadcast, you've seen a build output, but it's huge. And there's a lot of mumbo jumbo that the LLM doesn't need to see. If we just fed that entire thing back to the agent and said, hey, there's a problem fixed this issue, that is going to rapidly consume a lot of your available tokens and for not much benefit. In fact, it's going to do the opposite of that because the LLM is gonna focus on everything basically in your build output fairly equally. So what we do is we actually parse our build outputs and try to get the essence of what the failure is in the smallest number of tokens possible to then send back to the LLM. - That's a super interesting failure mode that I think with other development tools I wouldn't have thought of. But I guess I'll talk a little bit more about other failure modes of this thing. What are the other ways in which we've had trouble with this are just like ways that it fails? - Yeah, there are a couple of interesting failure modes. And I'm gonna focus on the failure modes from the recipient. If we imagine that we use an LLM in this fleet management system and we operate over a thousand components, there is normally a team than on the receiving end of that PR that has nothing to do with a migration itself. So the receiving a PR and they don't know exactly what the reason was for that PR, they might be completely uninterested in moving from this basic 64 library to the internal one. So they're receiving a PR. So we focus on the failure modes from their perspective. There are three interesting failure modes. So the first one is that the agent is incapable of solving the problem. And in that case, they actually don't even receive a PR. It's mostly annoying for the person that wants to do the code change over the whole code base. So in the basic 64 example that we talked about, it could be that when 10% of the cases, the agent just gets confused and doesn't understand exactly what to do and no PR comes out. So the receiving end doesn't even see the PR. So that's not really a problem. And if you want to get rid of this basic 64 library and the entire company, you just have to take those 10% and do it manually, like we did before LLAMs. Then there are two other failure modes that are a little bit more tricky. So the agent can produce a PR, but it fails in CI and continues integration. And that is because we try to verify the changes as much as possible before opening a PR, but we generally can't verify everything. So it might be integration tests as an example. So if we don't run a specific integration test, there is a chance that the agent can edit the code for that specific integration test and it then fails in CI. So this specific failure mode is a bit annoying because if you're on the receiving end of this PR, you get a bot creating a PR that is red. So the build has failed. And then you're left thinking, what am I supposed to do with this PR? There's a bot created this PR. They gave it to me and it's red and what? It's not great. So that kind of detracts from the reputation of our AI tooling and we really don't want that. So that's why Alex said, we want to verify every dimension of correctness as much as possible. And then the last one, so this is the worst one. So we have the agent and it produces a PR and it passes CI, but it's functionally incorrect. That sounds the worst. You just said a bunch of bad things about the previous one, but that sounds worse. This is worse. Yeah, this is for the worst one. Yeah, so the third one is the worst one. This is when it creates a green PR, it passes CI, but it's functionally incorrect. So I don't know, instead of doing base 64, it does a two-string, you know, the built or like this is great. Because we had no tests on that. And then the person merges this and then there's an incident. This one is the worst, mostly because obviously, it causes an incident, but from our perspective, building these LLM based fleet management tools, it detracts from the trust of AI. And we really don't want to get into that situation. We really want to make sure that we are babysitting the agents of the LLM to do the right thing, because we believe, and this is kind of a hunch, that if we get too many of these third cases with a green builds, people will kind of refuse to merge RPRs. Yep, no amount of hype will get you back in the game here. No, exactly. If you get paged in the middle of the night, five times, then you will probably stop merging those PRs. Yep. No, that makes sense. And so is there something special you do for each one of these? Is there like one silver bullet that gets you past all of these? Are we only through one and not all of them? Like how are we dealing with these three escalatingly terrible failure modes? Yeah, I wish I could say there's one solution, and it's all solved. But there's not-- there's many different perspectives we're working on this from. One is, of course, as we talked about. We try to do as much verification as possible within this feedback loop. There's also the long-standing, right, more tests, right, better tests. And this is something we've always wanted within engineering. But I would say what we're trying to do is if I, as a migration owner, I want to do, let's say, the base64 change. The responsibility is on me to handle this migration. So I need to be the domain expert. I need to make sure my prompt is as good as possible. I need to make sure that I run this migration a few times so that I understand the edge cases that are possible to understand. But we're also trying to make it easy for me, as a migration owner, to review 1,000 PRs. So here we have built simple tools that would allow us to categorize the failures across PRs. So if I have 1,000 PRs and 200 of them fail, it's unfeasible for me to go through all of these failures. So we open peers in draft mode so that the code owners wouldn't see them and then our systems run some form an analysis on the built failures. They categorize them and this increases the chances of me producing a good PR. But at the end of the day, we still have a human reviewing it. And in many cases, it's enough for the migration owner to review it and that gives a lot of context. But in many cases, it's both a migration owner and the code owner. So we're tackling this through different dimensions. - So let me first jump out of the last thing you said. So on every one of these PR and you talked about 50% or potentially 1,000 PRs, there's a human involved every single time. - We have a lot of peers that are created with our fleet management systems that are changes that we trust. So a dependency bump, for example, that we can auto merge. But these more complex changes, which we're just now learning how to do, there we have humans reviewing them. - And I think the key here is whether or not an LLAM is involved or not because before in our fleet management ecosystem right, there were deterministic scripts. So we could write tests for those scripts and see that they did a specific thing. But now with the LLAMs, we can't trust the LLAM and the agent always do the right thing. So that's why we specifically for these LLAM based fleet shifts, we do review them by hand. - I say so, the bar for human is required is was an LLAM involved. - Right now, yes. But I mean, this is something that I think the industry in large is trying to figure out. And we are maybe a bit conservative here and we are doing manual reviews on the LLAM based changes. - Yep, no, I think that makes sense. I mean, I guess especially the thing you said about like we can write tests for the regular fleet management changes, but tell me more about why we can't test the stuff the LLAM does the way that we can test stuff but whatever we call not that a fleet management can. - I think as the changes get more complex, what you are testing is more complex as well. So let's take the example of changing the signature of some function because dependency made a breaking change. So we can change the signature, the LLAM can write the code, but maybe some underlying assumptions in that new function change as well. So there is a performance optimization or the way it interacts with some other libraries different. And this code path is only exercised during certain conditions. So the more complex the changes become, the more of these cases you start hitting where actually testing it is slightly trickier. So it leads us to higher order testing which is also harder to do. - So it sounds almost like it's less that an LLAM was involved. It's that when we involve an LLAM, it's a more complex change and that more complex change is harder to test. Is that valid? - Definitely. It's the same thing, right? If I were to give you a PR that is super complex, it would be harder for you to review as a human as well. And the same applies for the LLAMs. - I mean, another way that we try to validate that the changes are correct is through an LLM as a judge. So we've explored using an LLM as a judge in order to catch these failure modes that Max mentioned, which is like the CI passes, but the changes are like maybe not what you requested. And for our migration cases, we have a very predictable output. We're not saying go and implement this big feature and it has to make heats of code changes. A lot of these migrations every time a human would run it, you would expect the same output and we expect the same thing from our LLM. And so if you think about what might pass CI builds, disabling tests. The models are getting a bit better. They don't do this as often now. But if you just remove all the tests, great. Your PR build is gonna pass. Your agent's gonna be like super happy. And so we need this all the time. So we need this extra layer, which is like asserting that the migration was actually completed correctly. And the LLM is the judges, the thing that catches these kind of quirky changes that are not easily caught with the verifications the appresists. So having the LLM as a judge reduces the kind of non-functional crazy things that you might see in the PR. - I see. So I guess you give a different model, the prompt of here's what I asked to do. Does this do it basically? - Yeah, exactly. To be honest, we haven't really invested heavily in fine-tuning our judge. And it's something that we keep coming back to a little bit more than we might work on in the future. But we know that the judge catches maybe I think 25%. It basically fails 25% at the time. And of that 25%, I think the agents are able to course correct, I think maybe half of that. So it is definitely catching issues and helping us get to a better solution in the end. - Yeah, that's super interesting 'cause in my head I just go to code review, it's supposed to be like, whatever we call it, four eyes, like two humans look at this. And now you're saying in an LLM change, we just need two LLMs to look at this before it's merged. Burn more GPUs in the LLM as a judge, because I think the prompt for the LLM as a judge literally starts with, you are the most talented code reviewer in the world. So that is essentially what it does and it just gets the diff from the original prompt and tries to score it. - Have you tried prompting the first one with you are the most talented software engineer in the world and just skip the code reviewing? It might or may not say that. - It's not my thing, that's exactly it. - I see. So then does it actually need to be different models? If the prompt is significantly different, can the same model review itself or if you found that doesn't really work? We don't know, to be honest. The main problem that we've had when building this infrastructure has been scaffolding and integration. So when you have a complex ecosystem like ours, you need to do a lot of scaffolding and just getting everything to run and all of the permission set up and things like that. So we actually haven't had enough engineering time to do evaluations, but like Joe said, it's something that we want to do. If we had evaluations, I think it would be easier for us to answer the question whether or not this model is better than that model. But right now I think our honest answer would be, we don't know. - I see, so for now it's just, you went with using a different model and that's working. So let's run with that for now. - You mentioned this removing tests example and I kind of want to believe that's a joke because that's an easy example. Are there like real things that it's done or does it actually just remove tests? - Yeah, so it doesn't really remove tests anymore, but unfortunately I think it has one so twice in the past. Another example is like, it might be changing a bunch of things in a Java class and it moves the constructor of a GRPC client from the class constructor to a method. And now what that means is every time the method gets called a new class is created and a new threadpool is created. And so you're actually not seeing, I mean, functionally it's correct, C.I. passes, but then you deploy it into production and you've got like a bunch of inefficiencies that you've just introduced. - Well, there was another example where somebody ran a shift over like thousands of components than most of them were okay, but then this one component, it just hallucinated and invented photos. And it just added like whatever, 5,000 lines of photos because it felt like doing it. And the agent convinces itself that it needs it, probably because it gets confused by something in the build output or who knows why, but the LLM as a judge is much more constrained. So it looks at this diff and says, wait, the prompt, the original prompt says, nothing about photos, why are you inventing photos, right? And then it's kind of funny, they have this multi-agentic conversation between themselves, where the judge goes bad agent. Like, you don't have 5,000 lines of photos and it goes, oh, whoops, you're absolutely correct. And then it goes back and forth. - Another classic is you might ask it as part of this migration to bumper dependency. And then that dependency breaks the test. So it's great, I'm just gonna revert the dependency and it finishes and you're like, hang on a second, that side that's defeating the whole point of this migration. So that's a really good one for the judge to catch. - Yeah, so it does it, that causes the problem and it solves the problem by just undoing the thing. - I agree with that. - Yeah, I see, that's great. I mean, it's very creative, the agent's very creative. Nice. So are there other kind of fun problems you've had to overcome that are like unexpectedly, unexpected fun things that's done to you? - I think one is like, it shows sometimes, it's like a really good at pointing out, like or lacking knowledge of a change. So some of the proto examples are, like you would review a diff and you would see, like it's at a thousand and thousands of lines of code, like you just don't understand. And the easy thing is to go like, oh, the LLM hallucinator or whatever. But then you sit down yourself and you try doing this migration and you realize, wait, this is impossible because of this one constraint here or there. And can we really blame the LLM for that? If it's not even feasible for us humans to do, those are the moments where you sit there and then you're like, ah, dammit, I blame the LLM, but it's actually me. (laughs) Do you have a good example of that or? - The one I think of is the proto example where the prompt was effective. Add another call to another service. And in most of the cases, we would assume that the schema for this other service was transitively included through other libraries. And that was true for most cases, but not for the one out of ten. And then you would review these diffs and it would like hallucinate proto that look similar but are not really the thing and you just not understand why and then you try doing it yourself like, ah, wait, no proto to generate a stamp from you. And I guess one of the traits that we see with the agents is they really try to solve the task and they're not really good at figuring out when they're not able to do it. So they just continue and continue. The vague or your prompt is the more likely are to end up in those scenarios where it just tries to please the original prompt, but in fact, the task might just be to ambiguous or vague and it just doesn't know what to do. And then it does other fun things. - Yep. - So it also teaches you a lot about exactly how to give instructions to things. - Yeah, it's like the YouTube videos with writing instructions about making peanut butter and jelly sandwiches, have you seen those? - That is exactly the example I was wearing the collar like that or just the things you do when you're starting to teach kids to code and before coding you just get to like, I don't know, give me instructions to whatever, make a peanut butter sandwich, walk out the door or something and they give you instructions and they miss, I don't know, open the door. - Exactly like that. Tell me a little bit more about how this thing actually works and maybe in that context too, kind of fun constraints that gives you. So we have fleet management, the system that lets us run semi-arbitrary but very deterministic changes. What are the things that we had to build on top of that to make us be able to like, jam an LLM into fleet management? I think Joe mentioned some open source thing. I think you guys have all talked about a few bits and pieces like connecting it to CI and whatnot. What does that actually look like as either external things or things that the group is up to build or anything else? - Yeah, I think the main thing that comes to mind is that when we run our build system verification so we run our compile and our test, in a lot of cases that doesn't work in our existing fleet shift infrastructure out of the box, right? So what we do is we run our container in like an ephemeral Linux VM and we're running our coding agent in a Docker container. And so if you take the example of maybe we want to run integration tests to verify changes but actually our integration tests, they also use Docker. And so we get into a fun little issue where we get an exception that we can't start a Docker image within an already running Docker image. And so we basically had to work initially to try to filter out the integration tests from running to verify changes. But now you're like reducing the scope with which you can actually automatically verify. And so you're just increasing the chance that the PR build might break because the integration tests can't run until the PR's created. Another example is that we obviously have our client app and our client app runs on Mac hosts. And so we've required to actually build our and compile our client's monorepo on Mac hosts. So that doesn't work within our Linux VM as well. So now we can't also verify our app changes, our client changes. So we solve this problem by not trying to recreate our CI within our ephemeral VM environment but actually to utilize the pre-existing one that's been purpose built for exactly this, like running the test and has all the permissions and everything that we need runs on the right platform. So we now run our coding agent in one environment and then we delegate to our CI system to actually run the changes against whatever the difference that the agent's produced. And then we feed that back from the CI process back into our agent runtime. - Yeah, so I guess if I any second, that's effectively if you're running a remote build, like if you're doing the change on your MacBook, but then instead of building it, running it, you can send your diff to some cloud CI farm and it'll build it for you there. - Yeah, that's pretty much the essence of what we're trying to do. - That's very cool, but that sounds like we can also use all the things you just talked about on iOS and Android code. - Yes, exactly. - Which is super cool. - We hope to demo this internally actually in a week or two where we're delegating to this client build system because we haven't been able to run the background coding agent on the client ecosystem and iOS code. - Oh, so all the stuff that Joe just talked about is brand new. - Yeah, it's happening now. - Yeah. - Yeah. - Awesome. That is even cooler. We're breaking news on the podcast. - I know, right? - Watch that number of merged PRs climb rapidly. - So that's super cool. I guess that's a great new thing we've done. Also, I noticed that we've been today talking about how there's 2000 or 2000 plus merged PRs. And I think when we wrote a blog post about this, like days or weeks ago, it was only 1500. So how quickly is the ecosystem moving? And should I be expecting that number to exponentially grow given it took a while to, or it didn't take that long to get to 1500 but the gap between 15 and 2000 seems really small. - We wrote the blog post a few weeks ago and now we're at 2000. So it's, we're trying to interpret the curve of number of merged PRs from this ecosystem as a linear or exponential. But the last couple days we have this really interesting spike where it's going up very quickly. I would say the thing that's holding us back right now are the things that we can't verify client code as an example. So there's a large user cohort that we haven't really addressed yet internally. And we really hope that when we do that and this type of background coding agent is going to be very popular. And hopefully we see a hockey stick on the curve going up. - Yep, but it was some number of months to get to that 1500, right? And then it was weeks to get to 2000. - Yeah, totally. I mean, it's been the system took three months to get a working prototype of maybe. There was a lot of like infrastructure plumbing that we needed to do and that was in May. And then from that on it started ticking upwards and it's been kind of linear until now. And then now I guess it's like spreading word of mouth and people are just picking it up and starting doing things with it. - Yeah, I think we built it for us in core improv to solve our issues. And then we felt like it started to spread through the word of mouth as likes said. And so we got this like natural adoption of the product and we started to see that number start to grow even though we were like mid development of the thing. So we've been developing it whilst the customer base is increasing, which has been a fun experience. - Yeah, we're servicing the engine while it's flying. It claims flying. We're also seeing this use case of people using it for we're having a chat on Slack and like, "Hey, these dogs are wrong." And then we go triggered agent to do that in the background. So not the most impactful and biggest of changes but these are tiny and they add up over time. - Wait, we haven't even spoken about this. You can trigger the agent for a Slack. - Oh yeah. - So we've talked a bunch about migrations. We started with core-in-free migrations and I wanted to ask how we get to thousands of core-in-free migrations. But maybe I'll cover that in a minute. Tell me about the Slack thing. We quite early added this ability to just add our agent in Slack and that would trigger a run and it would get the context from the thread and it could even discuss with you. So you could ask you like, "Hey, is this the rep you want to target or?" Have you really thought this through? And we see people using this for all kinds of changes. Like, you're in Slack, you're in some support channel, someone comes in, points out that there's an error in the docs or this dataset got deprecated. This field is now renamed. So you have to do that. We've also used it to develop our agent as well. So someone puts up an issue. We tried to see what it does. We're also thinking about what are the other surfaces where this type of usage of background coding agent is useful. So through JIRA, through other automated workflows. So there's a lot of untapped potential there. A cool example I heard I was in the London office the other day and I was talking to a colleague of mine and he was saying that his PM would always ping him to enable feature flags in the code and now he doesn't need to change context and do feature flag enablements because the PM can now just add background coding agents in the Slack channel to do these changes on the behalf of the engineer. - Oh, that's super cool. So yeah, so then it's not only is it doing much more than migrations like I thought, but it's also kind of empowering a bunch of people that art engineers would normally be making code changes to make, I almost had simple, but like not necessarily even simple, just to make code changes that are within the scope of something the agent can do. - Yeah, it's also interesting. It's not only about maybe non-technical or less technical roles like a product manager, but also as an engineer, you're certainly not comfortable with all different disciplines of engineering. So if you work in a cross-company initiative and you need to do client code updates, then you need to do some back-end stuff and then later on you need to do a data pipeline to do your analysis. Most engineers including myself, I'm not comfortable with all of these disciplines. So we really hope that this background coding agent can allow engineers to kind of weave in and out and if you just need to like Joe's example, activate the feature flag or update a minor variable or accepting on something, then you use the background coding agent to do that if you're not completely familiar with that code based or discipline. - I see, that's really, really cool. And you completely stole all the thunder from my how the hell did we get the 2000 just migrations question? Because now I totally get it and I will not ask you that. Alex also just as an aside mentioned that you use some of this stuff to develop the agent itself. So like you ask the coding agent to fix itself when you see issues, is that what you're saying? - Yeah, definitely. And then we've ran this project initially as an inner source type of thing. So there's a lot of excitement and a lot of people that want to contribute. And what we see quite often is someone wants to add a feature. They're not really super familiar with the project but they know things here and there. So they write a Slack message. They describe the feature they want and then they add the agent and then the PR pops up. Sometimes a perfect PR, sometimes you need to do some things but it definitely cuts down the time. And a fun example I have in mind is from today where we were discussing doing like a few tasks and doing planning. And we spent maybe 15 minutes discussing how this one task is super simple and we can just do it in a day. And it's so easy and we were just describing, oh, you just need to do this and just do that. So someone said, let's just write that down and ask the agent to do that. And then at the end of the meeting we had a PR. So there's a lot of that happening. - That's incredibly cool. That does make you feel like the numbers are going to keep going up and up and up. That is, yeah, that's awesome. So I want to ask how people are reacting to it but it sounds like people kind of love it and it's going super well. I mean, are there interesting reactions you've had aside from just, it's awesome that I can basically get code out of a Slack conversation. - Yeah, the reactions have been super positive. Of course, there's still the T-things, zones of the failure modes that Max talked about. And I think one thing is when we enable, Fleet Shift is like super powerful, right? And LLMs are super powerful. And when we put them together, you have the opportunity if you write a pretty poor prompt, you could send out a lot of PRs that have maybe quite a bit of nonsense. So one aspect that we're kind of hitting now is like we have to train our engineers to write good prompts and look after the PRs so that they're creating reasonable changes and we avoid producing too many PRs that maybe cause a bit of AI fatigue across our code owners. Some examples of that is if we're running a migration, we might do the migration on a repo once locally. And then we actually do that collaboratively with an agent as part of our local development process. And then at the end, we ask the agent, okay, if I was to do this migration on 100 repos, can you write me a prompt to actually do that? And what we find is like the prompt that the LLM generates when we do that process is far better than the one that we write. It covers a lot more of the edge cases. It's an LLM writing a prompt for an LLM so it's speaking the same language. And so training our engineers to use these kinds of methods and putting in these types of guardrails for them is something that we're also thinking about. - I think a cool example is like, we have this little specific tool for the agent where it can perform a commit. So the agent doesn't actually have access to Git, but it has a tool called the commit tool. So it's instructed that when it's done a piece of work like any engineer would do, then it should perform a commit. But the agent itself is not allowed to write the commit message, but rather it invokes that tool. And then the tool itself has an LLM again that looks at the diff and tries its best according to the prompt and the diff to write a good commit message. So we spent quite a lot of time like writing a good prompt and the tool that provides like it does conventional commits. So it talks about which component it is in or what type of commit it is, how long the commit message should be and which should be verbose, but not too long and it should be easy to read. And there's a lot of intricacies that goes into just writing a good commit message. So now the autonomous or the background coding agent is really good at writing commit messages. In fact, if you look at the average commit message in the company, it's probably better than the average commit message, but it's not as good as the best person that writes commit messages. And I think this kind of illustrates what we think about these background coding agents. On average, they might even be better than most engineers, but they're definitely not going to be the best specialized things. But if you just want to do a small thing, you just want to do a small thing at a discipline code base that you're not completely familiar with it, then they will probably do it quite well and they will do a very nice commit message. The PR will be nice and they will follow your instructions to their best ability, basically. Have you tried prompting it that it is the best? I mean, I think we say that it is the best in a lot of places, right? I'd say it still just isn't quite the best, okay? No, no, no, it's it. It does a lot of mistakes. All right, I'm also worth a shot. So then tell me, what else are you going to make this thing do? I mean, I guess you said there's a bunch of rough edges and things, but what are the next couple things you're doing on this? I mean, the really cool thing we want to do is we want to hook this up into other workflows. So like one example, we want to take our page your duty incident or a page, and then we want to trigger the autonomous coding agent to provide a PR. We're not comfortable merging these automatically, but you can imagine if you get a page in a system and you wake up and if you use page your duty, you get this into the page, and then you can have comments in the page. You can then imagine that the background coding agent has actually left you a PR saying, I think I fixed your problem, and then you can just go in and merge that. Those type of more automatic workflows that need human review is one of the next things we want to look at. I mean, there's a bunch of those things that happen that we as engineers have to pick up all the time, like incident is one that there's other which are not as urgent, like security vulnerabilities that need to keep your code base up to date. We might get crash reports that you have to look at and understand, and in many of these cases where you might need to do a coding task, instead of just presenting the engineer with the problem, we can also present them with the proposed fix. - Yeah, that's super cool. All right, well, that was awesome. Thank you all for being here and hope to see you all again here soon. - Thank you, Deb. - Thank you. - Thank you for having us. (upbeat music) Outside of the things that we're working on here and we just talked about, what other things do you nerd out about? - I think they've, last time you asked me, I was nerding out about Diablo 'cause I was finishing the good old 90s Diablo one. Yes, that sounds right. This time, I'm just about to start nerding out on something. It's zero for Universalist five. It's a history game. I don't know, all of the paradox fans out there, you know what I'm talking about. Of course, kind of a PhD to even get started with. Crusader Kings is one of my favorite games of all time, and it just, I don't know, you need like weeks to even get into the game, but then I get stuck. - I think that comes to mind, it's cleaning. - Ooh, tell me more. - Marie Kondro. - Yeah, I don't know, like after you spend your hours every day thinking about code, trying to get your phone to work on, do not disturb. You just, you know, you go home, you take a little bottle of water and you clean something, it just helps the mind. I wouldn't say I'm nerding out on it, but I'm definitely using it as a way to break from all of it. - Love it. - Can I have a non-technical answer as well? - I purchased a yo-yo a couple of weeks ago, and I've definitely been nerding out on trying to use that yo-yo. If you buy a yo-yo, you can't just use it out of the box. That's what people don't realize. If you've got to take it apart and you've got to like adjust this string, you've got to loosen the string. If you want it to what we call sleep, which is like where you roll it down and it just kind of like hangs at the bottom, you need it to sleep to be able to actually do the tricks. And so you've got to take it apart, you've got to loosen the string. Sometimes you've got to change your bearing. If you want to actually level up your yo-yo skills, you can't just be buying it and using it out of the box. You've got to do a bit of YouTube being to figure out how to engage the high performance modes. - I have some really cool tricks. - Yeah, I'm going to get the yo-yo out after this. - Thanks for listening. To learn more about our journey with background coding agents, check out the engineering blog links on the show notes. Nerd out at Spotify is produced by Spotify, Sarah Tazari, and by C-Play and Armada, who wrote our finally tuned theme song. I'm Dave Zalvituski. Thanks for nerding out with us. (upbeat music)