The Future of Dev Experience: Spotify’s Playbook for Organization‑Scale AI

Hello, and welcome to the AI Engineering podcast, your guide to the fast-moving world of building scalable and maintainable AI systems. Unlock the full potential of your AI workloads with a seamless and composable data infrastructure. Brewin is an open-source framework that streamlines integration from the command line, allowing you to focus on what matters most, building intelligent systems. Write Python code for your business logic and let Brewin handle the heavy lifting of data movement, lineage tracking, data quality monitoring, and governance enforcement. With native support for ML and AI workloads, Brewin empowers data teams to deliver faster, more reliable, and scalable AI solutions. Harness Brewin's connectors for hundreds of platforms, including popular machine learning frameworks like TensorFlow and PyTorch. Feel then to end AI workflows that integrate seamlessly with your existing tech stack. Join the ranks of forward-thinking organizations that are revolutionizing their data engineering with Brewin. Get started today at AIengineeringpodcast.com/brewin and for DBT cloud customers, enjoy a $1,000 credit to migrate to Brewin Cloud. Your host is Tobias Macy and today I'm interviewing Niklas Gustafson about how Spotify is scaling AI usage and engineering and product work, so Niklas, can you start by introducing yourself? Yes. Thanks for having me. Yes. I'm Niklas. I'm, she for the tech Spotify. I've been here for, yes, coming up on 15 years in the summer, so quite a while. And one of the areas that I've been focused on for the last few years is very much around our developer experience and how we leverage AI in our development workflows. And do you remember how you first got started working in the ML and AI space? Yeah. My background is actually not in the ML. My primary background before Spotify was in backend and the distributed systems and things like that, but of course, as I joined Spotify and we started heavily investing into machine learning, yes, more than a decade now. It's always been close to what we do. It's deeply integrated into all our product development and so on. So it's always been very close to me. But then of course, as Transformers and LLMs came about a few years ago, that got much closer to everything we do. So that's where I really got deep into our AI strategy and has ever since been a lot of my time on that. And digging into the main topic here, before we get too deep in the weeds, can you just start by giving a bit of an overview about the general engineering structure and some of the developer experience investments that you've made independent of any AI tooling, maybe in the years before GPT came out of the scenes? Yeah, absolutely. So it might be good to know as I think we'll probably come back to some of this as well. Spotify, we run a highly distributed architecture. So we have many thousands of production components that might be, you know, backend services, mobile components, machine learning, pipelines and models and so on. And these are designed and owned and operated by a fairly large number of development teams, something like 800 development teams. So in a sense, there's both a lot of attention about our architecture and the way that we work, but it's also highly distributed. So there's some level of chaos to it, which is both fun and challenging. This is a very collaborative environment. We do a lot of back and forth between teams, something like 40 percent of our pull requests are from a non-owning team. So if I own a backend service, a lot of the changes to that backend service is actually going to come from some other team. And this is always going on in between teams. So like there's lots and lots and lots of collaboration. And a few years ago, we shifted a little bit. So it used to be that we were a super autonomous company, like every team had a very high flexibility around what they built and how they built the things. And we just didn't see that this worked very well with the scale that we were at. So a few years ago, we started efforts around an increasing standardization of our code and practices and also made an investment into better ways of managing our entire fleet of software. We used to be that every team would own their own components and when we needed to upgrade some language version or library or whatever, we would tell each team that, hey, you need to move to the latest version of Python or Java or whatever it would be. That just didn't scale very well. So we made all of these investments into instead of being able to manage our entire fleet of software as one thing that we could sort of mutate. And I'm mentioning this because this would come back in terms of where we were also focusing when it comes to AI at the moment. So that's roughly the backstory for us, like a few or number of thousands of production components, 800 teams, very high scale for what we do at least compared to my previous experiences. Like we have millions and billions of users connected at a given point in time. We have lots and lots of traffic. So it's a lot of fun and always been very fast-paced over here. And one of the key aspects of developer experiences that you're trying to keep engineers in that flow state, keep them focused on the problem at hand and not get bogged down by all of the various roadblocks and things that are technically necessary but don't really contribute to the actual product goals. So things like making sure that your linter checks pass, make sure that your CI runs in a timely fashion, make sure that you have good test coverage and I'm just wondering as you were investing in that developer experience, what are some of the main pain points or struggles that you were dealing with prior to the introduction of AI and then we can get into some of the things that you're trying to use AI to solve for? Yeah, so I think one of the main challenges that we saw was just a fragmentation of tooling for our developers. So it used to be as a developer. It's a Spotify you have to jump in between many, many different tools that either we had a small resource tool we were using or we had built internal tooling but in a very fragmented. So that was how we came about investing into what we now call backstage that we've also made available outside of Spotify and it's this like single pane of glass that developers interact with and it's very, very present as a developer at Spotify. You do a lot of your tasks every day within backstage and that's been I think if I would point out like a single productivity unblock that we've done maybe prior to AI, that's going to be the one. Like that, that was the one that really simplified the life of a developer. Now I can go in within backstage, I can look at any of our thousands of software components and I can see all the actions that I can take on them and all the data and information that is available around them and that was for us very, very different from the fragmented state we were in before that and I think this is true for many other companies as well but it's been a particularly particular focus for us. And now digging into the introduction of AI in the engineering workflow, there are various shapes that that can take. There is a never ending set of tools with new ones being introduced every day and I'm wondering as you were starting to see some of the potential for using AI to accelerate software development, what was the overall process for determining how and where and why to apply some of those technologies? Was it more of a bottom-up and organic growth situation? Was it more of a top-down mandate or maybe some hybrid structure and just wondering kind of some of the ways that you plot through that overall process and investment? It has been, I'm going to say, largely a bottom-up driver within 45. We started out with a very strong internal demand for co-pilot back in the days and we went through the journey of being able to clear that so we could use it. It was a bit of an early day so there was a lot of legal questions and security questions and things like that that we had to figure out but we were pretty early on jumping on co-pilot and then the journey has just continued from there, like people. We had an incredibly fast adoption of co-pilot within the company and then cursor came along and then Claude came along and we've continued to seeing these like extremely rapid adoption, nothing like nothing I've seen for any other tool that we've introduced. So there's been an incredible bottom-up demand. We are of course trying to balance that with our being somewhat intentional and strategic about this but I'm going to say we have gone very heavy on experimentation and sharing learnings and being pretty humble about that we don't know exactly what's going to happen tomorrow to your point of new tools being released all the time. So we've been pretty open and allowing for a pretty broad set of tools and then encouraging our engineers to experiment and share as much as they can and we have some very very active internal Slack channels and repositories with Claude skills and things like that where people are experimenting every day. So it is amazing to see at the moment and I imagine that this will continue, like the pace has only been increasing soon. And to that point of the rapid pace of change, the rapid introduction of new tools, the evolution of existing tools, particularly from the perspective of developer experience and investing in the aggregate velocity of the organization that variants can allow some people to move faster but maybe at the expense of the broader organizational velocity whereas standardization, such as what you were mentioning with the backstage investment of this is the way that you get to the thing can make sure that everybody is moving in the same direction and at roughly the same multiplier of pace. And I'm wondering how you're thinking about some of that tension of wanting to standardize the tool chain builds around some of those capabilities and share those best practices while at the same time enabling people to do that pioneering exploration of figuring out what are the capabilities, what are the boundaries and how do we determine what best practices to bring to the rest of the organization. Yeah, it's a great question and I think that's putting the point very clearly on some of the trade of stuff we're having to make. So much like you describe, like we are betting on standardizing a lot of our underlying foundational pieces so that will be the infrastructure that we have in place like the tools that we make available to our coding agents and so on and on the code level. So as I mentioned before, we've been on a multi-year journey to drive that standardization and we're continuing that now and accelerating that and we're doing that through various means like we're one of the investments that we we currently have ongoing is to move all our code into Monorepos which allows us to have a much much higher degree of standardization than we have before like where our code lived in thousands of smaller repositories owned by each and every team. So we were kind of lucky to make these like standardization, the fleet management and the Monorepos investments as AI happened to us that so we're in that sense like feel that we're relatively well set up but it is also moving incredibly fast. So we're we're trying to just as we go along. And to your point of some of the tools that are available to the agents, some of the coding standards that can be enforced using things like linters, CI/CD, etc. That is definitely one of those core engineering disciplines and I'm wondering what are some of the other foundational kind of first principles that are effective regardless of the specific harness that you're using for a given LLM and some of the ways that you are investing in that kind of platform layer of enabling the agents to be able to get the context, understand the understand the tasks and maybe just some of the ways that the introduction of those agentic coding capabilities even shifts the focus of the people who are guiding and responsible for that work. Yeah, it comes in many shapes and for us. One that you mentioned in there was we need to guide those agents. So we're spending quite a bit of time at the moment to make sure that we have the right skills and rules set up for the agents. So they are able to follow those and that we have the right where we need it, the right sandboxing as well. So we're doing it in a secure way. We're also investing into improving the verification loop. So one thing that we've seen is that the better testing we have in place, the better both agents and in particular these like fleet white agents that we haven't really talked about but I'll come back to work. So let me dive down that rabbit a little bit. So I mentioned before that we made this investment a few years ago on what we call fleet management which is all about like managing our entire fleet of software as one thing that we can make changes to. The challenge with that was that the type of changes that we could make were fairly simplistic because as it turns out code is very complex to change like it has a very wide API surface in a way and so we could make code changes but they were fairly simplistic and shallow in a way. So one thing that has fundamentally changed now is that we can use AI agents to make those changes for us and then orchestrate and roll those out across our fleet. And that means that we can significantly raise the bar for how complex changes we can make to our fleet. But that means that we really need to have a good test automation a way to verify that those changes are correct and safe. Again luckily we did some of those improvements as we started rolling out a fleet management stuff a bunch of years ago but we think that we need to improve that further now. So that is one of the investments we're doing. We're both revisiting our practices around how we do tests and also our test coverage in the fleet. Luckily agents are actually an excellent tool to also write some of those tests for us. So we think we can get some benefits from that. So those are some examples of these like more foundational capabilities that we're building out but there's lots more. Like we built out an MCP infrastructure internally as well for example to make sure that we can provide many different MCP tools and do that in a secure way. Your comment on fleet management also brings up an interesting nuance to the overall engineering challenge is that there are different styles of engineering within an organization where you have your application engineers who are focused on writing the features making sure that the end user experience is fulfilling the overall organizational product goals and then you also have data engineers, ML engineers, operations engineers, platform engineers and all of those roles are going to have some set of code that they are interacting with that they're responsible for that they're going to be managing but all of that also has to be deployed and maintained and monitored in the production environments which is its own set of engineering challenges that has its own set of agentic capabilities that are being developed across the ecosystem and I'm wondering how you're seeing the adoption and application of these agentic capabilities differ across those different styles of engineering or domains of responsibility. So I agree with the sort of classification of different disciplines that you outline but I think they've started to meld together a little bit both as we've been again like doing these investments in thinking about our entire fleet so it used to be that as a developer you were at Spotify you were very close to the code that you owned or your team owned but now we're in more in a mode or like yeah you still own some piece of or some number of components within your team but many other teams in particular are more platform teams will also make changes to that code so code has gone from being this thing that like I own to Spotify owns and in a sense like everyone can make changes to them there are still access rules and ownership around them but the principle has changed quite a bit here so so I feel that some of those like the distinction between a we would call a feature engineer like the app engineer and a platform engineer used to be very distinct but is much less so today. That being said I still think that the way that we apply AI tooling into this there's a lot of similarities but also a few differences so again the typical feature engineer like working on one of our mobile apps or let's say on our playlist systems they will be fairly focused on those systems and like building new features and rolling those out to our users and and then operating those those components and that tends to be that then you're focused on a like a small-ish set of code and you're very like deep expertise on that particular domain and that means that you will use AI agents typically like I would say in the sort of mode that they were primarily developed towards like what most engineers would do and then if you look at our platform engineers they are very focused around again like managing our entire fleet of software so they will then use more scheduled jobs and and things like that so one of the things that we built out this part of our AI agents is something we call home it's just kind of silly name but it's it's really way for us to and again using this sheet management infrastructure that I mentioned before really way to take like a AI agent in currently Claude code under the hood to apply that to our entire like millions and millions of lines of of software and that's been incredibly powerful in terms of being able to drive changes to our software and you know do code modernization and things like that so and that's really something that like I'm going to say mostly our platform engineer mostly but not exclusively our platform engineers are focused on so under the hood they're using the same tools and the same many of the same practices it's the same way of prompting it's the same way of working with agentic tools and whatnot but the focus for their efforts tend to be a little bit different so that's the main distinction that we have we don't have you mentioned like operations engineers we don't have separate operations engineers like it's the same teams that build a known our software that also operates it so we don't have some of those categories that you mentioned but I would say that's the big distinction that we have and then in terms of the shape of the engineering work I'm wondering how you're seeing the distribution of work change as far as what stage of the software lifecycle more of the time goes into now where for most of the history of writing software you had maybe a product manager or an architect who would be responsible for saying this is how everything is going to fit together these are the features that the end users are going to be delivered and then I'm going to write all of the tickets and then I'm going to hand that off to an engineer who's going to write all the code and test it and make sure that it gets deployed effectively and monitored and with agentic coding in the mix it changes that equation a bit where the writing of the code part can sometimes move from taking weeks down to minutes depending on exactly what you're doing but making sure that it's generating the right code is really the challenge now and I'm just wondering how you're seeing that change in dynamics shift the time investments where maybe rather than it being that middle point that takes up most of the work of actually writing the different lines of code you spend more time instead making sure that you understand the problem and the specification and then it also puts a higher burden on the review and deployment stage because you need to make sure that the code that was generated is actually accurate and correct in doing what it was designed to do and just how that's changing the dynamics of the engineering or gets modified. Yeah it's a great question and many of the things you mentioned are really a reins through so let me talk through some of them so first of all I think our product development process has always been a little bit more collaborative than what you described so it hasn't been those like clear handovers of like the product manager writes down a spec and hands it over to the developer that's always been very iterative within Spotify but we do have those roles we have like product managers that will think deeply about our product strategy and then engineers to your point to will then design and build and operate the software to manage that and yeah and like you said like that part of turning those ideas into code that is obviously dramatically changing now like it it is just like you said going from potentially weeks to to hours and at the work for the work for for us and me as a developer has changed dramatically as well like I used to be someone that loved the you know hands-on keyboard solving problems through code type of challenge and I haven't literally haven't written a line of code now in like two months because all of that is being done for me with the agents that I use and that is a very very different way of thinking about it and I me personally I really enjoyed that transition actually it turned out that it wasn't that handled keyboard thing that I enjoyed I enjoyed being able to solve problems and shipping those to users more than the the hands-on keyboard this is obviously going to be personal and different for different people but for me it's been a pretty fun from experience in in so many ways and it is shifting some of the dynamics that you mentioned like it is shifting the work for us as developers more towards how do we effectively translate these ideas of things we want to do into input for the agents that we use and then to your point then reviewing the output from those agents and we are leaning heavily on applying human judgment in in both of those cases like being making sure that we provide the right set of inputs for the agents so they do a good job together with all the stuff that we've talked about before with making sure that we have the right foundations in place like the agents just like humans knows what the code standards we expect then what levels of conventions and security and so on that we expect and then we put a lot of accountability on the developer to then review the output of the agent both in terms of me sitting in front of the agent but also as part of our peer code review so that is remains super important to us but it is also challenging right because now we're able to produce much more code changes and we and a lot of that happens purely like in this agent loop and much much more quickly than before but we still need humans to review all of those code changes so it's putting a lot of pressure on the code review stage before things goes into production and that is something that we're working on figuring out right now so putting more we already have but we need to improve the AI assistance that we provide for code reviewing and we're going to change a little bit the policies that we have around how we do code reviews so more people can contribute to those reviews than is the case today for example and then the lost reflection I'll add in is that you mentioned the again like going back to those different roles that we have with like product managers designers engineers the similar to what I said before in between the different disciplines within in within engineering we're also seeing that those boundaries between what it means to be a product manager and an engineer are loosening up as well like we see a lot of PM spending time in cursor and Claude code and you know building prototypes contributing to our code in a completely different ways that was the case before so I don't necessarily expect that these different disciplines will completely disappear because there are different expertise there but these like handover boundaries that that you started with like those I think are going to look very very different in the future and I fully expect that where we have congestion points today within that process like you mentioned like the code part having taken a lot of time historically and that now moving away that's just going to create new congestion points so one of the things that we're preparing for is being able to detect those quickly and then being able to make decisions so we can iterate our way out of that I think this is going to be a big part for us in the next few years just a transition through this and we don't have a perfect you know view of the future of what is going to look like but yeah I mean everyone are using these tools today we have a ton of adoption outside of the engineering cohort for both cursor and Claude code and like everyone's trying to use them to for many different things you would be surprised about that like wasn't necessarily what tools were intended to from from the beginning but people are having a lot of good luck in in like using them for for other things as well and the the world of just like the world of an engineer has has been completely revamped in the last few months I expect that to happen with all disciplines within Spotify in the next year or two there are a couple of different things I want to touch on here and I'll just note them now before we get too far down the path but I think that one of the things I really want to dig into is some of those other ways that you're seeing some of these agentic utilities being applied but before we explore that in terms of the validation and review stage I'm wondering how this speed of code generation and the need for context cues and rapid feedback for the agents is changing some of the ways that you think about what languages to use how to invest in things like type annotations and just some of the overall tooling that you're investing in for linting testing validation canary deploys etc just some of the ways that it's shifting you're thinking about which languages and which toolchains are best now that you don't necessarily have to spend all of those extra key strokes on putting in those type annotations and just because you don't have the benefit of engineers being lazy but also the the potential downsides that that can have as far as not wanting to put in the extra work of oh well I don't need that type annotation right now I know what I'm trying to do but maybe the llm does it just some of those trade-offs yeah I'm gonna say that as as a company with lots and lots and lots of code as I mentioned before in many many different teams and we have a fair I didn't touch on this before but we also have a fairly it's very often that like ownership of components will move around within the company like I built something but it's now closer to the business problem that you were trying to solve so I will hand over those back-end services or whatever it might be to you so given this and this has been the case for many many years as well if I am given this we have always been pretty like pretty strict around like what languages do we use what conventions do we have in place and so a lot of these things we had to solve many years ago prior to AI taking over and I think it has turned out that a lot of those things that were good things at scale when you have thousands of engineers also turns out to be good for AI agents at least that's with the current generation of those agents so I'm not seeing us making it ask me again in six months maybe maybe my view has changed but at the moment I'm not seeing us making a ton of changes in those like very foundational decisions on what languages we use or what not what we are doing is that one of the big shifts that we're going through at the moment is a lot of the standards that we've used like the the way that we've used to promote the standards within Spotify has largely been guidance towards engineers so we've had tech radar and various documents that tells you how to build a ML pipeline within Spotify and that's been reasonably effective but we don't think it works very well for for agents so one of the things we're doing now is really to put more of those like closer into our codebase as a rather than being guidelines they're going to be links instead so that is one of the changes that we're doing at the moment and we're shifting a lot of those things into into our actual codebase and this also goes hand in hand with the investments that I mentioned before in of like moving to more of a monorepo structure because one of the things that've been really hard for us historically has been I mentioned having thousands and thousands of production components they lived in thousands and thousands of git repos and trying to manage that and like reasonably enforce our standards in all of those tiny repositories with hundreds of teams owning them has been very hard so one of the things one of the there's many challenges for us to move to monorepo but one of the benefits we're getting is like we have this one place where we can put all of these standards in place and all of the lengthy rules in place we've done this for our clients for many many years they've always been in in monorepo but not the rest of our software so we're sort of catching up with that now so that gives us much better tooling for doing some of these things as well and also one of the things that you mentioned is identifying what are the points where you are hitting bottlenecks in the path to delivery and that's definitely an area where the overall industry has been investing time recently is in figuring out how do you identify those bottlenecks using things like doorometrics I think is the most widely adopted one but just wondering what are some of the ways that you are quantifying some of those aspects of the engineering life cycle and I in order to be able to identify those bottlenecks and hotspots yeah so we have very I'm going to say very good instrumentation of our developer workflows so we capture data from our production systems and whatnot then that goes into BigQuery in our case and and you can go in there and and make queries around pretty much everything that goes into our developer workflow so in that space we're in a pretty good state to and we have lots and lots of metrics that we track like door metrics but also much more fine-grained metrics that we that we catch and track all the time so we're in a pretty good state there unfortunately that is not true for the broader production sorry product development process that we have so we do have pretty significant gaps around being able to measure and quantify let's say how our pms are working or how our designers are working so as we're seeing these congestion points shifting around I'm reasonably confident that we'll be able to see them within what our engineers do so I mentioned before the code review we're starting to see somewhat that popping up in code review and we could clearly see that in our metrics it's not as obvious how we're going to see that in a quantifiable way when it comes to the broader product development process I think we're going to have to rely much more on you know user research and surveying and those types of things to capture some of those I honestly think that's going to be a bit of a challenge for us over the next years to really detect those early enough before you know as they become a real problem for people then it's obviously going to bubble up and we'll be very aware of them but I would not love to be able to identify them earlier than that and but that I think is going to be challenging for us in some cases and so digging now into some of the broader I guess side effects of introducing agentate capabilities into the engineering life cycle and curious some of the ways that you're seeing those capabilities applied beyond just the stage of I can generate a boatload of code as fast as I want and just some of the ways that it's leaking into other aspects of engineering work product work and just some of the maybe new code that it enables because of the fact that it is so fast and easy and also maybe some of the ways that it changes some of your patterns around documentation and diagramming yeah so let me actually go a little bit broader than that to start out so one of the very eager set of users that use a group that we've seen has been our our product managers that we've been talking about a little bit before here and they are real like many of them are really picking up these tools and again like they are using the same tools that our engineers are doing so they're trying to figure out the ways around like have to how to bend the cursor and clothe code and it's you know many different ways of improving how we write PRTs or how we evaluate the experiments and doing insights analysis and like many many different things and we've had if a number of internal tools and websites pop up for for that target audience of you know again managing like the life of bringing a product manager it's modifying in a better way so so that is one one way that I've been particularly impressed with some of the innovation that's come out of come out of that group and I think we haven't talked about vibe coding yet but I think that's been one of the things that they've really leaned into and compared to before where they would need to go to like they might have had some of these ideas around like these tools before but they weren't really able to build them because they would have to go to an engineer and come in stem to spend time on building out some web tools for that idea they had and now that's you know a few prompts away so we have a lot of these experimental tools popping up and people having ideas around how to again like improve the non-engineering parts of how we build products within engineering what we've seen is like we see this sort of coding part of this which we've been talking about one of the interesting parts with coding is that it's actually a fairly small part of what engineer spend their time on so it's roughly out of your eight days as far eight hours a day it is roughly one hour that engineer spend like hands on keyboard writing code this seems to be fairly consistent across companies so it's not unique for us but like there's been a bunch of companies sharing that research and it's true for us as well like we're able to measure that and then there's a broader set of development tasks so this will be you know debugging operational duties code review those types of things that builds up to roughly three or four hours of the day and then the rest is spent on planning coordinating with other teams reading writing documentation those types of things so it is easy to like I think overly focus on this like one hour a day and trying to make that as quickly as possible which like the tools that we've been talking about today has been amazing at but we're really even if we're 10 next thing that share of the work that engineers do to the previous point so just like finding a congestion point we won't actually make Spotify tonne faster by just optimizing that so we've been focusing on like what are other use cases outside of that one hour where we can apply AI early on and it used to be that like if you would have if we would have talked a few months ago you would have heard me talk about like the various strategies that we had for that of like how are we bringing AI into being on call or managing incidents or whatever it might be and that is still very much something that we think about and have strategies for but also where we've been out innovated by our employees so there are now all of those tools available internally people have like our multiple tools that's been built for how to for example help people managing incidents and we've taken one of those and made them like the sort of canonical tool that we now provide to the company but other people are like still experimenting with other ways of doing that which is amazing so I think like we're seeing this spread out like what I imagine again if you would have we would have talked a few months ago what I would imagine was like we had this like the coding toss that one hour a day and then we would gradually spread the word the use of agents out from that because that was really what they were optimized for I think what we're seeing in practice is that they're now being used for everything people do during their full work days like everyone will use them for I mean I'm I use Cloud Code to write documents today or like prepare for hiring interviews or whatever it might be like it is it is something where it's become these tools have become ever-present in every toss that we do as not just engineers but more broadly as well yeah speaking as a technical manager I was very pleased the first time that I realized that I could use things like co-pilot or cloud or whatever to take the fuzzy sketch of these are the things that I need to do and then stick them into the issue template so I can actually write comprehensive and useful issues for being able to assign to team members yeah yeah it is it is amazing and I think again like the amount of experimentation and innovation that we're seeing from people just being able to take these tools which again like we're built for a kind of different purpose been being able to reuse them for every problem they're faced with has been amazing to see and continues being amazing to see and one of the aspects of these agents being a double edged sword is that well they are immensely powerful and can allow people to move much faster they can also amplify problems where if you have somebody who doesn't properly understand the problem domain or doesn't properly understand the code base they can guide the agent in a direction that is counterproductive and then the other aspect of challenge in terms of the deployment of agentic capabilities is the cost factor of how do you make sure that you don't accidentally end up spending the entire company's budget on a question and answer session that ultimately does not provide any actual return on investment yeah so the first part I think honestly I think the first part of that is is is the most important and that is really to create a culture and practices where you have that human judgment in the loop so you don't blindly trust whatever is coming out of the agent one good and recent example on that that we've done is we have an internal tool for for essentially essentially being able to prompt our data so I mentioned before that we have for example data on the the engineer the developer process and there's a bunch of data sets for that that can be pretty daunting if you're not super familiar with SQL or our internal data sets and whatnot so one of the ways that we're trying to bring that capability to more folks within Spotify is is to allow you to prompt for that is definitely prompt the type of problem you want to solve and that tool would generate the SQL for you to try to answer that question that is great but it's also pretty prone to introducing errors into those SQL queries that will be very hard for a less experienced user to detect so that's a tool where we have spent a ton of work to make sure that we have the right validation in place of those queries before we show them to the user one of the this got otherwise lead to you know us taking decisions on incorrect data and things like that so we really want to be careful about that so that's one of the cases where we've spent a lot of effort to make sure that we do the validation before we again like show it back to the user and we're trying to do the same in all of these tools and like I said we're trying to make sure that we have places within the processes where we make sure where we still have that human in the loop remaining I'm sure that will change long-term and there will be more of these cases where quality of the output of the agents are of high quality enough that we can remove some of those but at the moment like we're still very aware of that and and trying to be very careful about on the cost side of things we've certainly seen cases where things goes a little bit haywire but generally speaking in most cases the ROI of using these tools have been very strong for us so we're not trying to constrain people too much by putting cost guardrails in their face we do have those guardrails in place behind the scene to make sure that yeah you know what you said like in terms of some single use case going completely haywire but we've seen very few cases of that in practice. Another interesting aspect of Spotify as an organization and the incorporation of these AI and ML capabilities in the engineering life cycle is that as an organization you've been investing very heavily in various ML features and capabilities in terms of your core product and I'm wondering how some of the expertise and lessons learned in the process of developing that operational capacity has fed into some of your overall adoption process for these AI utilities broadly in the organization in engineering and otherwise and also some of the ways that the lessons learned through that experimentation within the engineering team and across the different organizational areas has fed back into some of either the operational excellence of the ML features that you provide as well as some of the maybe more generative capabilities that are being introduced in the product. Yeah it's gone both ways just like you say so like like I said early on like we have more than a decade of experience of building machine learning products at scale and there's been quite a few of those things that we've been able to bring over to to our to the productivity side of things. So one such case is for example how we do eVALs of LM-based products so to take a very concrete example we have this internal chat agent called iKa and it's it's all about providing our internal Spotify knowledge to our employees and very much based on how we had experienced working with our traditional machine learning consumer facing products we reused a lot of that eVAL practices and infrastructure for how to build that chat agent and the same is true from many other things like how we run experiments and maybe testing for internal tools is there we're reusing the same tools that we use for our consumer facing products for example. I think it also to be honest allowed us to not do some things so one decision that we were faced with I don't know two years ago maybe was do we need to train our own model on our code or will the rest of the world sort of catch up and be able to do that well enough without us having to make that investment. I think that was one of the case where we had a fair amount of experience from training both our own traditional machine learning model at models and transform based models where we could better predict the future and we actually avoided making that investment. I know some of our friends in industry went down that journey of trying to do that and then had to back out of it because the tools we've been talking about today just became better than what they had internally and then thinking about it in the other direction so you mentioned this but one of the one of the I think interesting examples there has been we've now started talking externally about how we're moving towards a future with more control for the user over the recommendations that you get and that has been very much informed by like how we're able to prompt these types of tools that we're using every day now and then imagining like what would that look like in the Spotify consumer experience and you can imagine that many of those things that you're able to do with let's say Claude code today would be as applicable to be able to do within the Spotify experience so we already like I said we always started shipping some of those products and you'll see us talk more about this in the future. Yeah I know that as a user of Spotify I was pretty happy when I saw the natural language prompt to generate a playlist and being able to experiment with that and I think your point of being able to inform the type of recommendations that you want versus it just being a black box that I as a user get whatever comes out of it is an interesting way of exposing more of these AI capabilities beyond the typical you know the the trope that was first introduced when chat GPT came on the scenes of hey here's a chat box we can talk to whatever but it's not actually useful as an end user it's just a gimmick. Yeah we think there's lots and lots of opportunity in that space so we're very excited about what that will look like for Spotify as well and as you have been introducing these capabilities investing in them building some core tooling around that overall life cycle what are some of the most interesting or innovative or unexpected ways that you've seen your team bringing these agentic and AI capabilities to bear. Yeah there's been many many many ways we talked about a few before like how people are using these tools now to build strategies like one of the things that I if you would ask me a year ago that I would imagine being like one of the things where it would be hardest to apply these types of tools but turns out that they're incredibly good at like you know spitballing and working with you and brainstorming back and forth and then in the end you have a product strategy that is as strong or stronger than we could produce before for example we talked about incident management and how we're able to apply it there and in our our designers has been incredibly what's the word I'm looking for creative in terms of like how they're able to use we have a bunch of designers to spend time to like figure out how to use cursor and end up building these tools for other designers for like how to prototype things or how to one very concrete example if you go into a Spotify app and you go into our DJ feature which is another one of these like more agentech features that is like this visualization and it's actually like driven by a bunch of parameters on on how that behaves and we actually have a designer again like by coding something with cursor where you can go in and like play with those parameters and make new visualizations based on that so we've seen a ton of those examples including like people using these tools for making their taxes or whatever problems they're faced with so people are innovative around how to use these tools and in your own journey of understanding the use cases and capabilities and best practices around how to integrate these agentech workflows into your engineering organization what are some of the most interesting or unexpected or challenging lessons that you learned in the process yeah there's been a lot a lot around what we've been covering before in terms of having better verification both in our in our code basis in terms of testing and so on but also in that agentech loop so we mentioned before this tool that we call home which is the agent that we use to manage our entire manage changes to our entire code base and one of the real breakthroughs we had in developing home was really when we added in lm as a judge loop into that so you can imagine like if you would be using clawed codes to make some changes but but but before those changes are shown to us users there's there's another sort of independent lm that takes a look at those and takes a look at the problem you were using and saying like is this a good change or not so you're like you have that review loop always being built in and that dramatically improved the performance of of that tool when you're using clawed like interactively that's not a big problem because if clawed does something wonky you'll just tell it like change this thing like I do that a hundred times every day but when you're authoring changes that it's going to be sent out so that's pull requests to like a thousand different repositories you don't want people to then have to iterate on those in 90% of those repositories you want to catch those most much earlier so having 80 or 90% good output is just not enough in that scenario you really need to get to that like 99% plus and that was one of those cases where we were able to do that and really like make that jump in quality that was transformational for that particular tool for us. And I think to one of the key things in the adoption of these agentic capabilities that we've touched on throughout is the challenge of a lot of these tools being very much designed around a single user interacting with it directly so it's more of the single player mode but it has broader ramifications at the team and organization level and I'm just wondering what are some of the key ways that you've found effective in terms of being able to break out of that siloed mode of interacting with these utilities and turn it into more of a collaborative experience and making sure that some of those useful patterns get captured without necessarily the engineer having to click the button to say yes this is the right thing or just maybe figuring out what are some of those implicit signals around oh this is a useful prompting style this one let us down a dead end things like that. Yeah I'm going to say I think we're still very early in that journey both as an industry and it's Spotify. Today the patterns that we're able to use for that are I find still pretty manual in many ways of making sure that we update the skills or like Cloud MD calls files or whatever that we I would love to be in a place in the in the future where just like you said like these tools are much more collaborative so if you and I are working in one of these agents like the things that you do and like the agent can learn from that and then reuse that when I work on the same code base for example so that we build out that build up that collective memory in the agent without that having to be in this very explicit way that we that we do today I very much imagine that this is something that we'll see coming in the tools in the in the near term future and I'm very excited about that because I think that's going to make often often we talk about these agents as being like another human in the team and that's not what they feel like today to me like they're incredibly powerful don't get me wrong but they don't get that collective learning unless we encode that into into skills or whatever so I that seems to me like one of the things that will be coming I know that there's projects out there with people where people are experimenting with with exactly those things today and and I imagine a lot of that will be coming in the near term future and I think will be pretty powerful and when you're working with your engineering team and you are tackling a given problem what are the situations where you decide that a human is the better operator and you issue agentic utilities entirely I don't know if there's going to be a lot of cases where we completely not use agents there's definitely going to continue being cases where the human will be in the loop and pretty closely so again like I mentioned code reviews before which we think is are those one of those important things other things will be like working with working with our production data and things like that but that is not something that like much of that has been automated since many years ago prior to AI so it's not a very frequent task that we do so I don't think there's going to be a ton of cases where we don't use agents at all I think there's going to be cases where we still want the human to be in the loop and then there's going to be cases that I think increasingly over time we feel more and more comfortable more or less fully outsourcing to to agents and as you continue to build the Spotify product work with your engineers grow your capabilities around these agentic utilities and make investments in leveling up the broad capacity for bringing AI to bear on your engineering challenges what are some of the things you have planned for the near-to-medium term or any particular projects that you're excited to explore? So I really want as we were talking about before I want to broaden the set of use cases where we get that like the agentic impact that we have for coding today I would love to for that to be deeply integrated into how we manage incidents because in particular I think that's going to lower the time to recover for those incidents which is going to be important from a user experience point of view I think it's like I said before it's going to integrate a lot of the work that we do within our product development process so like bring together more of our disciplines working more closely together so as a PM I can visualize more clearly through prototyping what I want us to build and then as a developer I can pick up from there like those types of things I'm pretty excited about and really to zoom out and look at that full product development workflow rather than like only be looking at the coding tasks where we've been primarily focused historically here so I'm super excited about that world then of course there are a million things we need to get right on the foundational layer that we talked about before and a lot of that we're working on at the moment are there any other aspects of your work that you're doing in the investment for your engineering teams and sharing best practices and investing in platform capabilities for agentic coding that we didn't discuss yet that you'd like to cover before we close out the show I don't think so this was a broad and fun discussion we went into many many different areas there are surely other things that we're doing but I think we covered most of the most of the ground on the interesting stuff we've been involved all right well for anybody who wants to get in touch with you and follow along with the work that you're doing I'll have you add your preferred contact information to the show notes and as the final question I'd like to get your perspective on what you see as being the biggest gaps in the tooling or technology or human training that's available for AI systems today that's a big question but so but continuing within this scope that we've been talking about here today like I would love to see these tools getting broader into more of the that full like development lifecycle and product lifecycle even though we're able to use them for many of those purposes today I think there are still a fair amount of gaps that would make them much more impactful within companies like Spotify going forward so and I 100% think 100% believe that that's going to be happening I would just love to have that today rather than tomorrow all right well thank you very much for taking the time today to join me and sharing the work that you're doing at Spotify to make sure that everybody is able to take advantage of these new and exciting capabilities and all of the investment that you're putting into that and some of the lessons learned step-to-lay very interesting problem space that all of us are trying to come to grips with so I appreciate the time and effort you're putting into that and I hope you enjoy the rest of your day thanks thanks for having me this was great fun thank you for listening don't forget to check out our other shows the data engineering podcast covers the latest on modern data management and podcast.net covers the python language its community and the innovative ways is being used visit the site to subscribe to the show sign up for the mailing list and read the show notes and if you've learned something or tried out a project from the show then tell us about it email hosts at a iengineeringpodcast.com with your story