Matt Zelesko and the Future of SRE

[Music] Welcome to season 6 of the podcast, Google's podcast about site reliability engineering and production software. This season, we met with SREs in person to hear what's on their minds, to explore the importance of psychological safety, and to learn what's coming next for SRE. And of course, the most important part is the friends we made along away. Happy listening and may all your incidents be novel. [Music] Welcome back to the podcast, Google's podcast on SRE and production software. I got a special guest in person today. I'm Matt Seaguller, this is. Matt Zalisco. What are you? What do you do here? I lead SRE teams for Google globally. How long has that been now? It's been a little over four years. That's right. And you were here just 10 minutes ago. Yes. And the SRE book is now 10 years old. It's amazing. And in 10 months so much has happened. What has happened in 10 months? Yes. It's never been. Well, first of all, the SRE book is 10 years old. We've had small things like the invention of cloud and hyperscalers. And now AI that have all happened after the book. So I think we're really mindful that the industry and the landscape has changed a lot. And we need to change with it. What has happened over the last few months is a real shift in how we interact with AI and large language models. This was generally a dialogue-based interaction. You were talking to the model. You were chatting with the model. It has now shifted over to agents and agent workflows. And that's really only been over a few months, but we are already seeing really dramatic impacts as a result of that. And I'll say, and I probably said this 10 months ago, you know, it's hard to kind of guess how fast this is moving. But I think it's very clear that the role of software developers is changing and changing right now. And the role of SREs is changing and has to change right now as well. I like how you put that. Last time you were here, you called AI the Buddy. The Buddy sitting next to the SRE, maybe like in the passenger seat. But now it's like the Buddy's become a lot more autonomous. Let's reflect on what SREs role is in this increasing velocity of that Buddy now. What's Buddy doing now? Yeah. So I think you're absolutely right. It's going from sort of human centric work to human supervised work in a lot of ways, which means that Buddy, or actually it's like, buddies that are doing a bunch of work on your behalf. And, you know, what this means is that the pace at which we are creating code, the pace at which we are changing production is increasing dramatically by, let's call it an order of magnitude. And that changes a whole lot of the stuff around the role and what we have to do as SREs. And look, I always think that there's going to be a human element here. You know, we have a judgment and an oversight and a wisdom that is still very much required. But a lot of the other work can be done by agents and we're seeing more and more of that happen. Yeah, you mentioned last time we talked SRE will always be in the operations role. And AI is just shifting what our operations role can do, maybe changing the scale, maybe changing when they take action. Yeah, that implies a lot of complexity. It does. It increased complexity. I know that SRE role and mindset is to be comfortable in that complexity. Do you think that's a role suitable to this new world that we're entering into? I think that's what you're saying. But I'm saying a few things there. I think number one, you are absolutely right that there is a notion that SREs pride themselves on having really deep domain expertise. And I think in the future, we are going to prioritize generalist capabilities. So I think we'll still have expertise in areas, but I think we're going to really emphasize this need to be able to generalize across a bunch of different domains to not be as restricted in terms of the expertise we have today. At Google, we have really deep domain expertise simply because the scale is massive. And we almost have to. But I think as agents take on more and more of both the software creation and the production operation, we start to expand out and we each need to have a lot more domain, broad domain expertise as opposed to specific. Yeah, this expertise point, it's interesting to me. Let's do some before and after. Like think about zip back a few years, not all the way back to the beginning, but to sure enough, enough back in time where it felt like expertise could be focused. And you could still get your job done and that would be enough. But I think you're talking about maybe broadening one's experience. Let's think outside of Google, maybe industry is a whole, what would you like people to be skilling up maybe? What would they need to be doing to prepare for this? Yeah, well, one thing I would say is we've already seen a transition like this inside of Google. And we may have talked about this on the podcast, but there's been a real concerted effort over the last five years to get on to common production platforms for things like roll-outs. And observability, capacity management, incident response. Because prior to that, we had many systems that did those things. And there was real power that was unlocked when we said, hey, everybody's going to get on to the same systems. That in and of itself helped us expand the knowledge base of the typical SRE and the systems that they could actually manage. So if everything is running on the same production platforms, that means if I'm an SRE in one area supporting YouTube, I can go and support workspace because I understand all the platforms, I understand all the tools. So we've already had a model here where we've seen a real uplift in our ability to be productive and productive across multiple domains when we start sharing common tools. And I think this is another example of that as well. In terms of what I would suggest somebody in the industry does, right? Start using these things as quickly as possible. It's what we implore our teams to do. And so what we've really encouraged is start using this thing. Just use it in the things that you do every day. And we try, you know, there's a responsibility on us here, leadership in saying we are going to grant you the time and space to do that. We understand that adopting anything has a bit of a productivity dip at the beginning. And then you presumably get a lot more productive. And so we've got to give the team space to go and explore and experiment with these. And I think we also have to provide the team with safe and useful default ways to do it, but particularly when you're talking about managing production. Yeah, I hear the specialty of this exploratory. It does feel like although we want something right now, we're still trying to figure out what the thing it is and I think it's changing. How are you helping your teams figure out how to control the variables on this, like explore this, but also you still need to get your work done. Try this out, but also you still need to roll out safely. Like what's the, what are the parameters here? How do other engineering managers and their directors and their CEOs make good decisions about letting that into their system, but also confine it so they can explore this, because they know they're going to have to deal with it later. And they'd like to adopt it, but not all on a go. Yeah, what's the strategy for this? So I think there's a bunch of different aspects. And we've actually tried to break it down into some of the different parts of the role and how you apply it there. So when you think about making changes in production, you have kind of the roll out and supervision around that roll out. And then something goes wrong. So you have sort of an investigation phase to try and understand what's going wrong. And then you have a mitigate phase where you take actions to actually fix it. And so we are thinking about each one of those elements separately. And so investigation is something that you can do relatively safely. Right? You're not mutating production state. You're not doing anything else. And so we want really broad experimentation and adoption with investigation, having AI-assisted investigation. Mitigation where you're actually going and changing stuff from production, we definitely want to human and elope at this point. And we also have fairly limited in the types of things that we're going to allow agents to do in production today. So we're trying to take different elements of the job and the role and say, here's where we think you can really lean in. And here's where we've got a lot more work collectively to do in order to make those things safe and reliable for managing production. And I really mean we, in the sense that I don't think it's a central team that builds these things. I think it is really everyone in SRE figuring out what are the right sets of skills that work for them. And then starting to learn and generate kind of common skills that everyone can use as a result of the learning that we have on the ground. Yeah, I like how you called attention to this common skills because one of the things that SRE as a concept, internally externally, is this mindset of being able to really be down in the body of the work and having intuition for, oh, I think this reminds me of something that bothered me yesterday and that kind of skill that deep intuition of this deep in the stack and is concerned that letting more and more automation alone, automation, not just AI, take hands of the wheel, will blur if not dead in our ability of this intuition what do we do to preserve this intuition that we still want as the hands on the wheel on occasion because we get too much automation and eventually we'll be unable to address the complexity. We'll be, oh, it's reached the state I can no longer address. The machine is handed back to a human in this loop and they go, actually, I don't have any experience here. I don't know what to do. We don't want that to happen. So how do we keep these skills sharp? Yes. So you definitely still want to retain knowledge about the production infrastructure. as I said it that maybe abstracted up a few levels from where we we have that knowledge now, and that's, I think, okay, right? That enables us to look at a much broader landscape. So you still want to preserve some of that. I actually think that AI can help us with that because it can teach us about what the architecture is. It can, you know, sort of keep a repository of the information that we need and have it accessible whenever you need that. So I think there's actually ways in which AI can help us in being better experts at those things. >> Yeah, so actually there's some tension here. Roll clarity and roll specificity is something that SRE is claimed to have for quite some time now. And it's actually identified itself and going to conferences, I was at SRE CON a few weeks ago. It was very delightful to see people celebrating that. And also the idea that you can put someone on incident response, even in another company. And with a little bit of time actually makes sense of what's going on. And that, as you mentioned earlier, even within teams or cross projects, is an incredible superpower for an organization. And Google, that is certainly true. But as you described, this sort of proliferation of different kinds of production surfaces, maybe everyone has their own unique way of solving the problem, different kind of software stacks. That's a return to the way things were before. And I don't think that's initially what we want. Maybe in the service of innovation that would be delightful. How do you see working this tension out? What are we going to do? - Yeah, I mean, we're talking a lot about the risks, but I also think about, what are the huge opportunities? - Okay. - You're right. And I think here's some of them. There's a lot of times that we choose not to refactor code or rewrite code because we just understand what a giant task that is. And it could be something that would help the reliability dramatically, but we're just not willing to invest in it today. I think that is going to change. Like our ability to rewrite code, refactor code is going to increase dramatically. I think the same thing, when we run something across Google, we tend to call it a horizontal. We have an initiative where we go, okay, we need every team to adopt this certain thing. And we try and have a role that like if you have a horizontal, the central team whoever's running the horizontal is doing most of the work. So that we're not putting work on all of the teams across Google that need to implement this horizontal. But the reality is all the teams wind up doing a lot of work. I think we have an opportunity here to say, you know, if we ship a horizontal, we're going to ship skills with it so that every team can just use agents to do the work to satisfy that horizontal. - For those audience listeners, what those things are, that may not be familiar with the skill and agent, how they work together. - Oh, so skills are essentially just capabilities on top of the coding harness, whether like we use anti-gravity, which is Google's product and skills are essentially very specialized capabilities that you can write. It's essentially ways that you can create new techniques or uses from just the base AI harness. And so, you know, give you an example, we have recently, a lot of people have probers because it's a big part of SLIs and SLOs. And we want to deprecate one kind of probe or in favor of another kind of probe. Historically, we would have gone every team and said, okay, you've got to change your code and here's the guide of how you should change it and go and change your code to do that. Now, we can ship a skill that you just run on the code base and it changes everything for you, right? So I think this opportunity to get things done and get things done at scale actually goes up really dramatically. The other big thing I'm excited about is, we aren't always in the room at the right time and particularly in the room, early in the process of designing a system. And so, a lot of the reliability knowledge and expertise isn't always considered at the points where the architecture is being set or the software design is being set. Now, with AI, I think we have the ability to essentially have things that are automatically, you know, comparing designs against our production principles and catching reliability considerations much earlier and much more upstream of where we tend to catch them today. I was going to ask you that, but I'm so glad you brought me to that because last time you mentioned the SLOs were just a trailing indicator of reliability rather than leading and how do we retool S3 for this role which is effectively almost a product management role which is defining reliability, but by default reliability of the default, I guess it's expression. How do we get them earlier into the design phase and using these techniques that you just described right from the start and using the ideas to use AI to help actually indicate where they're missing. Absolutely, because I think as we look at, you know, the agentic impact on software development, it means that our software engineers are going to spend a lot more time defining the best spec they can for what they are trying to create and the agents are going to create a lot of the code. And so when we are investing that much in that spec development time, there's a real opportunity to inject reliability and reliability thinking into that spec as well in a way that I don't think we do today or at least we don't do it uniformly. And so, you know, at the time that we are creating the specifications that our agents are going to use to go write the code, having additional skills that go in and look at the spec and assess it for, identify risks with it, right? The big, if availability is the trailing indicator, risks are the leading indicator, right? We were trying to identify the risks associated in the system and we have some great mechanisms like STPA and other things that, you know, are ways for us to do that today. Today that is fairly human intensive. But you could imagine, you know, risk identification agents that are just running all the time and looking at this and trying to find things in the code. So whether it's at spec time or even at commit time, they are going and assessing the production risk of the changes that are being made. >> Yes, this is a very design heavy conversation we're having. Building architecture or even revising a very large foundational architecture or, you know, moving along its life span. I imagine there are large architectures that have designs that heavily departed from where they are in reality right now. And that's not, that's just what happens in life. >> That's right. >> And this is an opportunity to go back and see what's happened and maybe go identify that as an opportunity for revise. Now, is that happening now? Are we actually going and finding out, oh, this is not what's really going on anymore? In fact, we've just discovered this. Is this an AI opportunity? >> I think it absolutely is an AI opportunity. And it is one where, to your point, it's not a point in time spec, but it's ideally like a living diagram and understanding of the dependencies. And I talked about this before a little bit, but I think dependency mapping in this new world, particularly with much larger numbers of services, much larger production surface area, you've got to have a real time understanding of the systems and how they work together and how they depend on each other. And I think AI can do a lot of that for us. >> Yeah, I'm going to be repetitive on this point, but it is of course the the podcast. Why again, do you think SRE as a collective, as a culture is especially suitable to this role? We have so many development engineers, and SRE has been training for this particular job, apparently. And why are you trying to host, position us for this? It seems like you are. Tell me why you think that we are culturally suitable for it at this moment. Like make your sense. >> So a few things. I will say number one, and this went back to the original SRE book. You know, SRE's job is to automate ourselves out of a job, right? It has always been about identifying what are the difficult manual steps, the toils from steps, and essentially automating them away into the next thing. So when we say automating ourselves out of a job, it takes on a different tone today, I think. And so we've been talking a lot more about automating ourselves into jobs. And, you know, I think in that case, it is all about like, what are SRE's the best at? They're best at reliability engineering, having really good understandings of how to build reliable, resilient systems. And yes, we also do a bunch of toilsam work, and we are on call, and, you know, those are aspects of the job, but I think at its core, we are reliability experts. And in any agentic future, you still need reliability experts. And so in some ways, I feel like this shift to agents and agents writing software is just the next step in the automation of a lot of the work that we do. And SRE has been doing that from the beginning. We understand how to identify, you know, toil or tasks that are ripe for automation, and then going off and doing that. And I think that's going to be the same and very true over the next 10 years as well, as we look forward. - I think you pointed out some areas of production that even a small shop can take action on, especially in the anomaly detection, and mitigation, not necessarily in the rollouts, but like these are things that even the smallest of shop can do right now. Yeah, what are things that are surprising you right now that you've seen happening around you? Or what is something that you did not expect that is happening? I did not expect how fast we would make this shift. So, and I suspect I'm not alone there, but just the pace at which we went from, as we talked about 10 months ago, sort of, you know, AI as the companion, helping us out to AI, really taking the lead role in a bunch of the work that we are doing today. And I've been frankly shocked by its capabilities. I was a software engineer. I think we actually, I actually even talked about that on the first podcast. But I hadn't written code for a while. And as I've implored the team to spend more time with these things, I spent time with the system. I went and picked a bug that had been sitting there for a while. And I worked with anti-gravity to go through and fix that bug. And it was probably the first time that I had created software in a really long time. And what surprised me was how good that collaboration and interaction was and how good the results were that came out of it. And also, I got a little bit of that sense of wonder and fascination with things that are possible. It really opens up a whole new world of possibilities. And I think until you start to play with it, you don't quite understand how amazing it is. And how about about where you going from here? Like some places that you stream dreaming big. What's something extrapolate way out? What would you like to see in 2018, 2018? - Well, look, as I said before, today we are looking at sort of like what are the safe, non-mutating parts of production where agents can have an impact? I think that is a period of time. And so you step back from that and say generally, SREs are going to be on call a lot less than they are today. And why that's exciting is because we have structured so many things around SREs being on call. Even down to the way we structure our teams, right? So if we create an SRE team for a service, that is two sister teams and two geographically distributed locations so we can do kind of round the clock very easily. Like what is this unlocked for the organization if on call isn't the grounding for so many of those decisions we make? That is, that starts to get really exciting, right? And it doesn't mean that we aren't still the stewards of production and we aren't responsible for the reliability and resilience of the systems, but the way that we do it, I think, can be a lot more empowering. I like the sound of that. Maybe we won't get so many pages at two in the morning. All right, well, that's all the time we have from the Pittsburgh Office of Google. Matt Zalisco, I'm Matt Zagler for the podcast, Google's podcast on SRE production software. Thank you for joining us. Oh, happy to be here. You've been listening to the podcast. Google's podcast on site reliability engineering and production software. Visit us on the web at SRE.Google, where you can find books, papers, workshops, videos, and more about SRE. This season is brought to you by hosts Jordan Greenberg, Steve McKee, Florian Rathkeber, and Matt Zagler, with contributions from many SREs behind the scenes. The podcast is produced by Paul Gulliamino and Salim Virgi. The podcast theme is "Telibot" by Harvey Belchrin and Jordan Greenberg. page from Telebot.