Assumptions as code: SiriusXM’s approach to platform prioritization

I'd realize that you can break assumptions down into three faulty states, I would say. There is assumptions that can be in conflict, right? So that's when two product teams or two engineers have assumptions that are different from each other. There is assumptions that are invisible. That's when a team might not even know that an assumption already exists from another team. And then those assumptions are stale. Welcome to the Engineering Enablement Podcast. I'm your host this week, Justin Ryak. Before we jump into the episode, I want to tell you about the AI Measurement Framework, a research-based set of metrics designed to help you understand the impact AI is having on engineering productivity. To learn more about this framework and how to apply it, head over to getdx.com/ai-framework. I'm Justin Ryak, WDCTO@DX, and I focus a lot on our research and software community thought leadership. Thanks for listening. Today on Engineering Enablement, we're joined by Eleanor Millman, Senior Staff Product Manager of Platform Engineering, and Minato Wadros, Associate Director of Platform Engineering at SiriusXM. And we're going to talk about how SiriusXM uses data to prioritize platform improvements, how AI has helped to augment this, and their unique assumptions as code framework that provides a systems approach to decision-making at scale. So before we get started, Minato has a little bit more about your role and background, and then we'll do the same with Eleanor, and then we'll get into the questions. Finally, the product over here on Platform Engineering at SiriusXM, I actually started off as a game developer, according to some games back in the day, got into the entrepreneurship space, and then I kind of fell in love with product management. So I've been doing product management for 15 years now. I start off in consumer products, so education, retail, healthcare, finance, and then about halfway through my career, I kind of fell in love with developer tooling. So we started working a lot more in the developer tooling space on security tools and on other productivity tools, and that's where I learned about the one-of-world of Platform Engineering, and that's what's led me to where I am today. That's interesting. Eleanor, how about yourself? - Similar to Minato, I started out in a more technical area. I actually studied physics and school, and I was an Android developer, a little bit of Java and Python for a few years as well. But I noticed that as much as I enjoyed building tech, I even more enjoyed thinking about people, so thinking about the developers and what made life easier hard for them. And so I transitioned into product management, and there I was able, they did a little bit of non-technical products, but mostly I focused in my career on technical products. So I've been a VMware, Google, and I've really realized over the years that I like thinking about developer tooling and infrastructure, and just ways to make developers lives better and reduce their pain points. So now I'm at Series XM. I've been here for almost three years. I joined the Platform Engineering, or built up the product team for Platform Engineering. Spent some time promoting internal AI use, and I'm still here helping Series XM developers have an easier time with it. - Now that's great. Thank you both for joining us today. I'm really eager to dive into this subject and learn from both of you. So today we're talking about what I think is a really common sort of crisis problem in Platform Engineering. Specifically, you've got five teams, there's limited resources, you've got a million requests coming in. How are you deciding what to prioritize? - That was actually the first problem I tackled. I was the first product manager in Platform Engineering. I was hired by Jared Willinsky, who actually was on the show a few years ago. And when I came in, it was very clear that my engineering counterparts were doing such a great job of prioritization, but as the needs grew of our users, of our stakeholders, we were supporting folks over, of course, the whole SDLC, 700 plus developers, it's just what they were doing for prioritization was not scaling. And so they really needed a more robust method of prioritization. Of course, they felt very comfortable prioritizing at the team level, but once you get look over five teams at the whole Platform Engineering org and beyond, that's where we really needed something stirdier, broader as you will. So we looked around and there was just nothing that existed at the time for Platform Engineering. And as probably most folks here know, Platform Engineering is really a non-traditional product space. I feel like in most product spaces, most products, you can generally look at revenue as a way to prioritize, but as we know, Platform Engineering is twice removed from revenue because we built things for the people who built things that eventually generate revenue. So it's a complex space and we had to improvise in the end and how to prioritize. - I think when Eleanor brought me in, I think at the time I was working at a consumer-facing group and we have end users outside the organization, they pay money for our products. So we had a crutch, almost, right? Like you're looking at revenue, you're looking at ARR, you have all these metrics that I'm going to build for years. So then when Eleanor kind of showed me the world of what they were building, I was like kind of shocked at like the framework that I think you'll go into a little bit about how relational each item is to each other and why that's needed in internal, whether you have these different stakeholders but you don't have like a bottom line metric that you're aiming towards. But it was a pretty nice surprise to see kind of the level of maturity that Eleanor was able to bring the team on in terms of prioritization with an internal product team. - It gets hard sometimes to connect the dots to revenue when really what you're doing is making, more frictionless systems and you're encouraging throughput for the business, but it is hard sometimes to really quantify that in terms of revenue. Have you found a good way to kind of justify that up to executives? Do you feel like there is unification up at the top level of leadership that what you're doing is directly tied to throughput in the end? - Well yes, so this framework we worked hard to put in all the different ways we could be impactful as platform engineering and looking at all the things that leadership did care about. We always knew, I would say it's fairly obvious, as a platform team that folks really care about development speed, of course, execs want features to be delivered faster. We also had runtime reliability factored in because of course we want end users and in our case at Series XM that's listeners, we want end users to have a fabulous experience. So low latency, few errors, et cetera. And then of course cost reduction and secured in compliance, we knew all those matter to stakeholders and we were able to find a way to work that in. Over time we actually figured out other things we needed as platform engineering to be more successful, such as being more data driven, making ourselves more efficient and really building the trust of our users. And so by mixing all of these impact factors into this prioritization framework, and then we were able to tune it, we kind of changed the way it's depending on what signals we were getting from leadership. Maybe one quarter the company OKRs were really focused on this huge launch coming up so we really had to increase development speed and well keep in quality high. And then a few quarters later, cost reduction was a focus once we had hit our launch, we had all this traffic running through our AWS platform. So this framework allowed us to be responsive in a pretty detailed way based on leadership signals. And I think what I'm going to add here, Eleanor, where I think it really stood out to where it really helped is a lot of these, I think like a cross team efforts. And I think like when you have a global store for a priority, you know, the reality is like I've worked in startups. If I'm working on a five person team, we kind of know what we're building. We have a pretty good idea. We could align the five individuals. I think what was difficult when I joined platform engineering and where the framework really helped is, OK, now there's a product or the piece of work that involves two teams. How do we decide that this is actually globally important for platform engineering? And I think the weights and what you added there was something that I found really valuable joining to try to help with those cross team conflicts. Yeah, that's a great point. Me and I can throw in an example. We have our DEVX team working to make local iteration easier for developers. And we have our delivery team working to make the pipelines run really well. And in both cases, they to finish their projects, they need help from our Cloud Foundation team to set up the needed AWS permissions. And before this framework, it was really unclear which project should be prioritized. And of course, our Cloud Foundation team has its own projects. This framework allowed us to align on the org level as well as on the team level. And it just made things a little more clear, really, to everyone in the org in platform engineering about what we wanted to work on and why. So let's dig into that framework a little bit. You started with kind of that standard off the shelf. You rice framework for our audience. That's the reach impact confidence and effort model. Why did you have to kind of evolve past that? Why wasn't that framework enough for kind of the high performing platform team that you were trying to build? We started with rice. It's such a common prioritization framework and product management. But immediately, it didn't feel right to us. One thing is that we had urgency for a lot of our work. And let me be clear, as a product manager, always you should try not to use urgency as a reason to prioritize. You really always want to look at impact and building the most valuable things for users. But sometimes there are real urgent, like real deadlines, like for us, we had this enormous company-wide launch in December 2023. Like we couldn't ignore that. or sometimes you have a vendor's contract that's up and you cannot use it. a tool past the end of the contract and you have to do whatever you got to do to mitigate that issue. So rice doesn't have urgency, so we put urgency in. And then secondly, like, impact is part of rice. Fine, like we kept that there, but that's where I would say we did most of the changes in that, or we maybe fleshed that out, that what does impact mean and for which users. So as I mentioned earlier, basically we started out with the four impact factors of developers speed, runtime reliability, cost reduction, and security and compliance, those were quote unquote the obvious ones I'd say. And then over time we realized we did really need that having platform engineering be data driven, having platform engineering be more efficient and helping our users trust us because that helps us long term. So we really iterated bit by bit just as I would recommend for any product development. We iterated on this framework to just make it work for us. And so we had the impact factors in, we would put in the we waited them based on what signals we were getting from the company and our users and it made a few tweaks to the formula on the way. And in the end, we've ended up using this formula for almost three years at this point, very much changing it as time goes by. And we've prioritized, I've been saying over 200 projects, it's probably even over 250 projects or something, 300 at this point. So it's really lasted the test of time, but with significant changes over the last few years. And maybe let's talk about that iterative process a little bit. So you mentioned these impact factors. Did you give us an example of maybe okay, I see a very high quality score in one of these areas. How does that help you then prioritize like the next area of improvement for the platform? So all of these factors, as I said, we wait them. And for example, I mentioned how we needed to reduce costs at one point. We wanted to focus on cost reduction. Well, if we increase the weight of the cost reduction impact factor, something else has to go down. And at least at first, when you look at all the factors, you feel like you don't want to give anything up. This is where actually really being data driven helped us. So we had we used DX, the DX snapshots. And so when we're figuring out what to reduce, we were able to see that we had a very high quality score, especially relative to other industry benchmarks. And so we felt comfortable reducing our runtime reliability waiting, not to say that we don't care about runtime reliability, but just that we had confidence that we already had a pretty high quality software systems. And we looked at another metric from DX that showed us that we're having pretty few significant incidents. And thus we could reduce runtime reliability waiting to give more weight to the cost. Another example would be leading up to our really, really big launch. We weren't as we built up technical debt within platform engineering because we were so focused on increasing developer speed. After the launch, we wanted to take some time and make ourselves more efficient because of we're more efficient, plus platform engineering, we can help our users more. And so for instance, we were able to see that code review scores were not as high as we would like for platform engineering. So we were able to prioritize more heavily weight platform engineering efficiency and then prioritize some code review projects so we could build up more automated tests and other ways to help code review go more smoothly with the platform engineering. And I thought the cool thing, if I could like recollect back to like when those decisions were made, I don't really like, I think that's like one thing in an organization when you have like, there's a lot of these weights are assigned by our leaders, sort of worked with through our leadership because they have a good idea of like what the weights should be this quarter for kind of business objective reasons. So I think in a typical world, often what happens is like, a leader might say, it's, you know, this is the quarter that we are going to focus on, you know, developer speed. This is the quarter going to focus on cost. It's very different when we then ask that leader and work with them to actually assign a weight to it. Because if you up the weight of cost, that does mean there's going to be other work that doesn't get done as well. So we come from world we're like in product management, we do these things, we use the stack rank priorities, we select sit executives in a room back in our consultant days and force them to choose, you know, where are you going to put this sticky? Is it going to be up here? Is it going to be down here? And I think the weight in or of being forced to actually add a numeric value to one of these dimensions acts as the same kind of force and mechanism to not just like to say the thing, but to actually see what that impact would be for your roadmap. If you decide to change the weight of the importance on one of these factors versus not. As I thought, I thought it was a very interesting way to like take what we do day to day of like executive leadership through words and turn it to something very actionable that would immediately change the roadmap for the quarter if they decided to kind of pull that weight up or down. It may sound hard when we set a stakeholder or a leader like you have to give us the exact percentage weight that you want to weight cost or developer speed, something like that. What we found, this was part of the iteration that we would say let's put a weight in, let's choose values for all parts of this prioritization formula. And then let's see how the 70 projects we have in queue right now how they line up. And early on we would definitely look at the ordering and saying you know what, this does not feel right to us. Our gut feels like project B should actually be above project A. And then we would talk about it. It really generated a lot of great conversations and we would sometimes we would realize okay, we forgot a whole impact factor. You know crap, we actually really want to be data driven and we didn't have a data driven impact factor at the time. Okay, we actually need to take into account the value itself of being data driven as platform engineering. Or maybe we just had misestimated that actually there's a whole other group of users we weren't really thinking about who would benefit from this project and we need to take that into account. And lastly, sometimes we would talk about it and think you know what actually this formula is really leading us to realize the true value of the ordering we have and that's the right one. I mentioned this because a lot of times I early on when I introduce this to folks I hear criticism of it's really heavy weight, it's really numerical and I totally agree like the onboarding cost at the time was not great. We'll get into later how onboarding costs to this framework has been reduced with meaners work. But early on folks would say it's heavy weight, takes time to onboard, totally agree, you really have to learn it. But the good news is that a lot of it for us was aligning with our gut and having these really good conversations and landing in a place that felt good to everyone where it felt like what was in their gut could be accurately described by what was in the spreadsheet and the formula. So it sounds like this framework has allowed you to sort of balance that need for the autonomy that you have as a leader who wants to be able to improve the platform but not have to ask permission for everything but also maintain alignment between the needs of the business. So now you have this framework that allows you to scale that prioritization that we've been talking about almost like a high quality score in one area gives you permission to then move and focus on another area of the platform without having to take that through multiple layers of approvals. Is that a fair statement? So I would say that the instinct at first when we think about prioritization platform engineering is to really simplify it. Let's lump everything together into one or two values kind of like rice when you start out with it, that original prioritization framework. But the issue with that is that it's very, very hard to quantify. And now that we've teased out various impact factors and weights for them and ratings for each project within, it then gives you a place to put data. So exactly as you said, we had a place where we could say we've got this quality score, high quality score, that empowers us to make this decision. Jared, a leader of our org could then use that in case we were getting pushback from higher folks I are up. You could say, look, this is why we're making the decisions we're making. It's the conversations that happen as well to treat the, it's not just the store. I want to be clear about that. It's the same way we think about like point in stories. It's not the fact that you're assigning points to anything. It's the conversations you have to get there. It's not necessarily just that we were able to rank this project in 19.2. It is more so that the team had a conversation about the value in each of these dimensions and had an idea of the assumptions that they were actually leading into around that project. So the conversations behind the stores, I would argue, also led into what you're mentioning like the feeling of alignment comes out in these conversations. When it comes to forcing function to go deeper into the analysis and make that more of a collaborative exercise, I think that's really interesting. So Eleanor, you sort of hinted at this already that there was a way that you simplified some of the onboarding onto this process. And in a pre-show, we talked about something that I thought was really, really unique, which was this concept of assumptions as code. So being able to, a lot of times, the many, many assumptions that we have in an organization are kind of hidden amongst various communication channels, but you've actually come up with a framework where you can treat this as code. So, Meena, you've talked about this assumptions as code framework. Can you just, for the benefit of our audience, just really give us your very concrete definition of what this framework is? And then Eleanor, certainly, how has this helped to enhance the way that you're able to think about product and platform? The assumptions code framework is taking the spreadsheets and the documents and trying to increase onboarding. So what we're doing here is our developers and our product managers use these code and agents. In terminals is their primary use case. So essentially by calling back and still from these code and agents they're able to go ahead and try to build. Let's say they are the example that I think that is you're typically trying to build a product case or product brief or a PRD or you're trying to define an EPIC. And you would actually give that EPIC into the code and agent you would give it a GiroLand or give it a description or give it a PRD. And then the agent would actually reach into kind of a repository where all of our assumptions are stored across the organization. And what it's going to do is going to validate that EPIC or that PRD against all of those assumptions and might bring up some assumption you might not have realized actually exist. So as an example, if you're building something related to deployment times, it's going to go ahead and say, hey, this could actually help 33% of users who use Java in this case, in this repo on this team in this domain. And it's going to ask the user to go ahead and validate or invalidate that assumption or possibly add context to that assumption. And what that will generate is a possible assumption conflict that will then get submitted as a PR and require a human reviewer at the end to actually bring that assumption back into the system for future storing and future kind of research mechanisms. So it's a skill today that users use and they're kind of code in agents. And it works by grabbing assumptions from that central repository validated against the concepts that you provided and actually submitting PRs against that central repository to solve assumption conflicts or stale assumptions or add in assumptions that didn't exist in the first place. Thank you. It's such a unique approach. I just wanted to make sure that our audience is really clear on what it actually is. So thank you for that. So it's interesting. So the way this came up, I'll say is in using the framework, we get into these conversations around like, okay, how do we go ahead and assign a value to development speed? First, we have to decide how bit is our user base for that product initiative. And right away on one of the initiatives that I remember storing, I walked in with like, this is going to be obvious. We have the metrics. It's in our intelligence platform. We're using DX like we know exactly how many users feel that pain point. This is just going to be me showing up a dashboard and saying, hey, look, there's 350 users that match this pain. See where I'm leading into? It was not that easy. All of a sudden, the conversations you realize actually, I came in with an assumption that is very different than the other five engineers that all have different assumptions as well. And it kind of reminds me of this study. I'm not sure if you heard the the tapers and listeners experiments. It talks about the cognitive bias of the curse of knowledge. This idea that like if you have certain knowledge, you kind of over, you overestimate the amount of knowledge that others have in that same domain. So that this experiment was kind of an oldish experiment in the 90s. What they would do is they spent uses into two groups, tapers and listeners. And a taper was told to think of a tune, a well-known tune, like happy birthday song. And they had to go ahead and tap it on on the board. And it's in plain in their head. They know the melody. They're tapping it out. And they're told to estimate how likely the listener, not knowing the melody would get it right. Tapers estimated on average that, okay, got the melody in my head, I'm tapping out all the signals. The listeners will get a right 50% of the time. That's the cognitive bias. Inactuality, listeners were getting a right to 2.5% of the time. And I think of that story and I think exactly that's how we work in product. In product, I work with a specific user persona or user type for three months. I feel like I've sent enough signals out there. My engineers have been enough interviews. We've all kind of aligned at the same assumptions that we kind of think that everyone has aligned with the same assumption. And we are totally wrong often about how that assumption actually shared not just within the team, but I think more dramatically across other teams that have been a part of those same user research sessions or those same interviews. So kind of like walking into the framework and seeing this play out and seeing how assumptions are always often misaligned between product teams, we kind of hit this challenge with the framework. And the way we resolved it using the framework is these like 45 minute conversations that were super fun, super healthy, but also 45 minutes. The challenge of that is like we are getting to these assumptions, but we're not sharing it out to any of the other teams. Like a delivery team comes up with finally, hey, there's actually 300 users of this user type. But now how likely is that going to actually transfer to the foundation team or another team that is an in the same conversations? So that's where this idea kind of came from. I kind of realized that you know, you kind of like break assumptions down into three faulty states, I would say. There's assumptions that can be in conflict. And then there's assumptions are stale. And this happens often when you're working on a product area, maybe six months ago and you fall love with this product area and six months down the line, you pick it up again. And you're you're biasing towards stuff you learned six months ago. There's a lot of lots of change in six months. So that's when the assumptions become stale. And what we kind of realized is all three of those problems actually are solved in code. Right? You can actually solve conflicts in code. You can go ahead and codify the date of an assumption. You figure out how stale things are if we codify that assumption. And also again, by storing assumptions as code literally in a repository, you make it more visible. So in practical terms, what we did is we took kind of the the prioritization framework and we started adding our assumptions into a sensual repository. So look at all the user interviews. We've conducted as soon as they're finished. We go ahead and submit it as code into repository. All the data sources. So we have over six different data sources linked into this kind of like machine that pumps out different assumptions. But the key here I would argue is that it's all given in the loop enforced. Right? We were using this as a recall mechanism, not a judgment mechanism. So that's where we kind of store all of our assumptions in. And as we prioritize work, we have used or leveraged different like agent AI tools to use the recall system to help prioritize work and help kind of like also go through these assumption conflicts with each other. And I'll mention to so already what Mina built worked really well when we would be scoring for the prioritization framework and realize we had two folks who disagreed and they would talk and they would ideally align on something when there was already enough data to have that debate. One other thing I've really enjoyed about this framework and then Mina's solution was that sometimes we would debate and folks really could not agree. We had this assumptions conflict that we couldn't resolve. And so that to me as a product manager says go do a little bit of user research. Now obviously if it's a hugely important project that you're going to spend months and months and months on, I advise you to go do some user interviews. Like really maybe make it a little more formal, do your due diligence. If your assumptions are going to tank the project if you've got them wrong. But there's a lot between doing a bunch of user interviews and nothing. And so and we luckily had those tools that are disposal. Like I've already mentioned if you're doing kind of a bunch of user surveys, like for instance with DX snapshots, you have that data available. Another thing that we did sometimes was to send out a more targeted survey to a specific user group. Like for instance we had some observability as code and observability as code library. And we maybe thought it was being used. We didn't know how much. We were trying to decide if it was worth the effort to keep maintaining it or not. So actually using DX we were able to find the right user group in general to look at. We sent a targeted survey understanding how frequently folks were using this library because we figured if they're looking at it daily or if they're looking at the dashboards that it produces daily or weekly, probably pretty valuable. If they're looking at it monthly or not at all, like not that useful. And we were able to see that 50% of our respondents were using it at least weekly and that allowed us to feel good about keeping continuing to put in the effort to maintain this observability as code library. So all that's to say is that with the framework when we were disagreeing on values, we could go out and do a bit of user research. We could also sometimes we would just send out a few Slack messages to engineers with a couple of questions and within an hour or two we'd usually have at least a little bit of data to question or assert that our hypotheses were correct. And so we do this all this work previously but now once Mina put forth this idea of well let's actually codify the answers the data that we're hearing for these various assumptions, then we're able to take this research and now we have a place to put it and we could reuse it as opposed to having to read to a research study every time we think we might need that data. The as code aspect of this assumptions is code and I think what makes this so unique is that you took these things and you actually built like a coded framework that you could store in a repository that you could run some analysis on that most importantly you could then use this in a very transparent way to drive human discussion, human analysis. Really it's for a lot of conflict resolution around you know trying to achieve the same goal. It reminds me in some ways if you're familiar with the evaporating cloud model the thinking process in the theory of constraints. It sounds like this is like almost an evolved version of that where it's you know you have everyone trying to achieve the same goal. two people thinking they need two different things to achieve that goal. And how do we tear down the assumption and figure out, you know, how can we achieve what we need to achieve with everybody getting what they feel like they want. That's what it's reminding me of, but it seems like you've taken that and put a whole systems approach behind that. I find that really fascinating. I kind of like this fractal view in the sense that is platform engineering. We of course want to help development team standardize on tooling that we don't want, team A and team B to be building the same tool and duplicate. So for the prioritization framework and the assumptions as code system that mean is developed, same thing. We're kind of preventing different engine our platform engineering engineering teams from not duplicating effort around learning about users. You've got a lot of data. It's codified, you know, have you been able to find AI applications that have been able to assist with some of this analysis? I do believe if this is enabled by some some of the AI application for you. So this is something that we run any anytime someone uses an agente code in tool. As an example, or just a code generation tool. If you are the way that most of our users are using this today is they are, for example, building a product brief or a PRD, or they're creating the description of their story or right now through Epic. So if they're using the code in bots to go ahead and help this generation, it is actually checking this back end repository for those assumptions. It's informing the user, hey, these assumptions exist in a terminal in this case. And then it's asking the user to validate it right there within the terminal. And then it's actually generating the PRs from the interaction that that user developer or product manager has. The reason why I think this works really well with AI as well is data is expensive, right? Finding data, doing surveys is expensive and historically what we've done is we've done really good research for our biggest bets. But by having everything in this repository and by leveraging AI on kind of like product instantiation, we can actually store smaller initiatives as well. So your smaller stories also run through this prioritization analyst and recommendations are provided on maybe what tier of value this work gets put in. And because it's so cheap and easy to run, it's still useful to a degree for even the smallest initiative. So it's helped us scale, which was very hard to do without AI around this because we don't want to replace judgment human judgment. We're putting the human at the judgment level and at the assumption validation level. What we want to do is make sure the recall level is helped by or assisted by AI. So that's where AI is in the midst. It's the that like, okay, you've written a pretty interesting epic or pretty interesting product brief here. Just to let you know, here are some assumptions that already exist in the organization. Do you agree with them? And that's where AI can really help because otherwise, asking users to go ahead read through every data source, scrape every system on every initiative. Just probably does not pay out compared to the economic like cost of doing that research. So that's where we really see AI leveraging in the system before AI before this assumptions code framework. After we developed the original prioritization framework, which by the way, we really only used it for projects that we thought were about four developer weeks of effort or larger. Just because there was a time cost to prioritizing each project. I did work with another product manager on our team to try to do a prioritization of the backlog of it happened to be our software delivery team. And there was a lot of value in my mind to doing that. These are smaller backlog pass, but there were many of them. I think most teams usually have a significant backlog of a bit lower level or shorter tasks. So figuring out the highest value ones for our user users would have been great, but it was really laborious at that time to figure out the assumptions for each of these backlog tasks. And we abandoned that effort at the time because it was just two time consuming. And so I'm really excited for the way that we can use what Minas developed along with AI. We can do it a lot more quickly. I'll mention also Minas did something that I thought was great, very pragmatic. He added a very pragmatic feature. I'm going to pause for a second to say years ago, Min and I actually met at a place called pivotal labs. It doesn't exist anymore. And there are three values I loved. I'm going to quote them because it's relevant for now. The values where do the right thing, do what works, and then be kind. So I mentioned this because of course as a product manager, yes, if we've infinite time, let's please do the right thing. Let's do the right user research. Let's review each assumption for every prioritization effort we do like. Let's give it all of our focus because this is so important. But also we've got to do what works and realistically, we as product managers, but even more engineers and engineering managers who have many other responsibilities in their jobs, they're not going to have time to do a half an hour review of the assumptions perhaps. So I really like Minas built in different options, which I'll let him explain that really lets folks sometimes do the right thing and sometimes do what works, but either way bring more prioritization and assumption knowledge is in than we would have before. There really are different personas or different user types even using a tool like this. I think there's three modes, three to three game modes, but there is like a rapid prioritization. Then when you could just have the tool go ahead and recommend a store for you based on the rubric based on the OKRs of the quarter and based on the prior assumptions about the user base, et cetera. So it's like a 30 second plug in play. Hey, you just wrote a thin and gira. Here's what it actually, you know, it quays to, but then there is like standard mode, which is more conversational. You're actually working with an agenda conversational AI tool to kind of ask questions and where we've seen the most value because when engineer leads are going through that conversation standard mode, they're being asked by the AI tool. OK, looks like we are addressing this type of incident. I can make a guess that the incident has probably occurred at least two times a month. That's what the energy standard is to you agree. And then of course, I usually encourage the engineer manage to actually go and check their own data. So like I think that conversation standard mode is where we're seeing most of the value today. And then there is like a deep dive mode where, you know, if you are a product, if you're someone that's really interested in like the product thinking you want to guide maybe a new initiative. Maybe this is the first time we picked this user group. Maybe this is something that like we're pretty sure we don't have any internal assumptions about already. There's like a 15 minute mode that would really dig in and make sure that we are generating the correct assumptions and making sure that we are analyzing what assumptions already exist and work solving of us conflicts a little bit deeper than some of the other two modes. Because in a plot edge, we do have different different types of work that we would run through this and different type of users as well that have different needs from this tool. A lot of what we've talked about here is, you know, scale using technology and cultural process to, you know, allow more and more efficient use of the platform. And we spoke earlier about, you know, now we have this need and really the industry is seeing this shift moving from the term specifically software developer towards just builder. How has that distinction been important for both of you and what do you think the implications are in terms of scaling your platform? Well, I'd say it's something that's been on my mind quite a lot. As a product manager, by the way, I certainly work to empathize with my users no matter who they are. And as I've said, developers have always been one of the user types that interests me most. This builder persona is especially interesting because it's personal for me and for me in the sense that I am not a developer. I am a product manager, but I am what I like to call tech forward. I'm very comfortable with tech. I do have some experience coding. Other other examples that I've seen are, you know, we've got like a marketing strategy guy who has no coding experience, but it's just obsessed with AI and really loves diving in and figuring out how to build stuff. Or I've encountered another person who was one of the highest users in our chatbot. And when I went to talk to him, again, not a developer in any way, but he was just building this Python app. I asked him, "Chachy, pt, questions." This was before more of the agent, "I came out." So all that's to say is I really believe that in platform engineering, we have to cover these users. Platform engineering, it was very easy. It used to be people who build apps, our developers, our users. And now people who build apps are more than our developers. And we need to find ways to help them because they will proceed either way. Most people really want to have a golden path, follow security best practices, et cetera, et cetera. Let's give them that opportunity. I can add, like it's always validated. I think Platte Edge is an interesting place where we're kind of like to see what's going on the rest of the industry. And then we kind of replicate and build our worlds around it. Who are the people that are going to live in these little apartments that we have in our platforms? I think I read a stat recently around the GitHub. GitHub, a year over year, had a 25% increase of users just this last year. I find it hard to believe that we've built 25% more technical software engineer roles in organizations. So who are these users? Who is going into GitHub today and building these repos at a quarter of a higher rate than ever before? And I think what we are seeing is this democratization of building. And this has been covered kind of everywhere. And I think it would be-- kind of a shame to have this growth happen outside of organizations not replicate that same sort of like safe space within organizations. I think platform engineers are going to probably take a lot of lessons from like the open source world of what it means to allow for different types of contributors that are not you know traditionally the maintainer of that repository or that app or that idea. But what would it be like if we actually started allowing for different user types. And that's a fun but challenging space because let's not downplay the amount of effort and energy that we've gone through to make sure that software engineers like no security they they know InfoSect and you know PI rules that's a lot of work. So the answer is not to open the floodgates and let anyone push anything anywhere. But it is also I think at this point about enable in business value while also metadata and risk the same way that we've looked at everything else. We would be remiss if we didn't call out the challenges to platform engineering and mean as already mentioned the security privacy concerns. Of course also there's support ability suddenly you have all these folks who really don't know much about software development developing software. If we're not staffed in platform engineering to support these folks especially across an entire company. So we have to find better ways to support folks like this. And also there's concerns such as maintainability you know you someone builds a great app maybe they get a nice hundreds of people at the company are using it who knows and then what happens if they go to another job. You sometimes platform engineering might be asked to take on support maintain maintenance of this particular internal app like that's that's a little nerve racking too. We need to put the thought in to address all of these concerns that are coming out of platform engineering. But I think that they are all solvable. I'm really excited just for example on the support issue as we know these agentic coding assistants can read from things like skills like if we just do a good job of documenting security best practices that we want to be used at our company and put it in a GitHub repo that of course can be turned into a dox site. You can either have the human the not the builder the non developer builder they can read through the dox site and learn it or they can even point their coding assistant at that and that coding assistant will probably follow non deterministically but probably follow those security recommendations those privacy recommendations scalability recommendations whatever it may be so I'm really hoping that we can take all that knowledge that exists in many developers minds and try to get it out in front. And lastly, I'm very passionate on the subject. I'll point out that if we do have our more senior developers are InfoSec folks documenting best practices, it won't just help the non developer builders like there are even a junior developer and boy have I appreciated it when I have had access to documentation, especially customized to my company of what I should be doing. So I think that a rising tide lifts all ships if we make our platform easier to use for everyone smoother onboarding, better documented it'll support far more than just the non developer builders. So it sounds like although you know, scale has always been a challenge and you've worked on things and all of a sudden we've thrown accelerant, you've added the number of people now who need to be able to take advantage of this platform. You've also found ways where AI can help augment the work that you're doing and help solve some of these challenges as well. So hopefully we'll hit that nice balance and that'll help kind of navigate kind of this new shift. I love to hear you know, we talked about prioritization, we've talked about that framework for prioritization. What are you prioritizing for 2026 at serious in terms of your roadmap for platform and for culture? So it's funny. I thought a lot of the things we talked about today make it not obvious, but you're kind of see where we're going with the roadmap. So I think what we've seen is a lot more building going on. So I think the the bottleneck is no longer being able to build the tool. It's also discovering that it already exists. So you know, investments and internal developer platforms to ensure that there is an idea of what it's just already the ownership, et cetera, like those investments will continue to unfold at Series 7. And I imagine this to continue in other platform edge, but really increasing the knob of value that we get from discovering what already exists within a system and also align an ownership to individuals so that we could figure out who owns those things. I think as we kind of alluded to like I think that's become more and more important this year. So we have a lot of work in the books around kind of discovering those things. Also, if you think about it as well, like now that we've discovered, hey, there's a really awesome tool being built by this team, we're going to be building a lot of work to make it easier to contribute to each other's projects to break down these silos and make sure that there's actual tools that get added into the platforms make it easier to contribute some that already exists because the cost to build something on your own is actually quite low now. Any sort of barrier of friction to contribute to an existing project needs to be kind of removed. So we're looking at ways to remove that friction and kind of I think what's on everyone's mind right now. And this is an interesting one is like we've seen a high throughput this past quarter, like a high PR throughput, the edge has a great kind of metric around true throughput that we use as well. It's gone up. And that's great on one side, but I think what we're obviously looking at is that well, now you got a bunch of a lot more contributors and if our vision comes true around like different types of contributors, I think there's going to be efforts on making sure that the bottleneck does not stay closed on, okay, this is great code, but how do we review the code? So code reviewers, things to make code review easier, things to make code review faster, more automated and maybe possibly deal with some of the more basic ones and help prevent a lot of that PR bottleneck. I think the PR space is something that we're invested in this this time around as well because of all these, you know, part A like faster engineer throughput that we've seen as well as a higher volume of users that are able to contribute as well. So we implant for engineering definitely are thinking quite a bit about AI because we want to make sure that developers and non-developers if they want to use AI and feel like it's a valuable to them, we want to make it easy for them. We've had to, we've certainly done some work for supporting AI early adopters, but from a learning standpoint, they're great, right? Like the almost the definition I would say have an early adopter, especially on AI is they're so hungry and excited for AI that they just go figure it out and we have to unlock them with tokens, permissions, whatever. But one focus that I've especially been thinking about is well, what about the majority that we've got a lot of developers who just have so much on their plates and they want to learn more about AI or they want to get started at least, but they've just got a lot pulling on their time. So I think that we implant from engineering that think about how to enable these majority users, not the early adopters. So things like just really, really clear onboarding instructions, we really care about telemetry from them. So we've done work that instead of having to set various variables. So we get the right telemetry, let's let's find a very smooth onboarding process so we get telemetry no matter what so that they're set up to hit the ground running no matter what. Also, a few basic enablements sessions, people learn in lots of different ways. I was a high school physics teacher years ago and I certainly learned this then that some people just want to be handed a doc and you they get to read it on their own. Other people like to be shown even if it's simple, like, let me show you how to install this agentec coding tool. Let me walk you through the steps. Let me show you one or two examples and suddenly they feel unblocked. So we're looking for ways to help that majority and to say way into, of course, that topic that we were just covering that this AI enablement, of course, will help those non-developer builders as well. I would say the last focus very related to this is understanding more about who these citizen developers are, these non-developer builders. However we want to call it, we've got to standardize clearly on a term at some point in the industry. But as we talk in platform engineering about how to support these folks, a lot depends on what they want to build. If they want to build fully functionized productionized apps, that is a big ask that will require one type of guardrails and help whatever. Or, you know, some assumptions or hypotheses we have is maybe they don't care about internet ingress, maybe they only care about egress, maybe they don't actually need internet access at all. Another one would be which internal tools that have data do they need to connect to? Certain internal tools we got to lock those down others, maybe there's less of a risk if they want to access them. So we're doing some user research talking to these builders, seeing what they want to build or what they've already tried to build. And lastly, I'll say, figuring again, how much they need to scale it? Another assumption hypothesis, whatever is that maybe they only want to share with up to five folks on their team or build a POC that shown to 10 stakeholders and that's that. That is way easier from a scalability standpoint and a availability standpoint than something that will go to external users like our serious exam listeners. Or that will be scaled to a thousand internal users. I mean, they're just fundamentally different requirements. So we're really spending time understanding what these folks need. Thank you so much for for both of your perspectives. It sounds like, you know, you've done a lot of forward thinking about problems that I feel like a lot of platform engineers are actually just starting to run into today. So I think they're going to be some some really great takeaways for our audience, some really great perspectives on how. to deal with this new shift and these new challenges that are going to have to be dealt with. So, Mina, Eleanor, thank you so much for taking the time today and coming on the podcast. Thank you so much for having us. It was a pleasure. Thanks.