The State of Product Development 🔍 — with Doug Peete

believe in there being some level of pride and diligence put into the specs up front. I mean, we're seeing a real rise in spec-bred and development in organizations. So this is going to become even that much more clear when your specs suck. You know, there's going to be no hiding from it. When we talk about AI and how teams are using that, actually turns out that less than 10% of the teams are using AI for helping with product requirements and products specs. I mean, is that surprising to you or what do you think is the case? Yeah, I would have expected this statistic a year ago, two years ago. I shocked to see it popping up in the last two months since we rented the survey. It's crazy to me. I mean, I'm kind of speechless. I don't know what to say. Hey, Luca here. Welcome to a new episode of The Factor In Podcast, where every two weeks we interview a world-class tech leader. Today's guest is Dark Pete, Chief Product Officer at Atono. With whom over the last few months, we have developed a deep industry report about the state of product development. In this chat, we'll go through the main findings of the report, match them to our respective experience and explore ideas about how teams can do better with product development and AI. So let's dive right into the action. Welcome, Doug. And thank you so much for being here with us today. Thank you, Luca. Looking forward to this conversation. Yeah, me too. So the Geras CPU of Atono, which is an only one platform for product development. And we have known each other for some time now, collaborating on various articles in the newsletter. And most recently, we have run a big research together on the state of product development, where we surveyed more than 350 teams to find where are the gaps in the problems today, how teams are using AI and so on. And we published a report that was super popular and a lot of people replied to me via email, also with more questions and thoughts. And so I invited you on the show to comment on some of these results of the report and take stock of what we have basically. What do you think? Yeah, pretty interesting results. Please, so I definitely might talk about it. Hey, before we continue, I was reading the X-26 survey on how AI is impacting developer productivity. And the results were kind of surprising. One of the main callouts is that AI tools still lack critical context. This is how they describe it. An AI assistant can't reason over its lack thread from an archived channel or the mental model of the engineer who built it. So now that we are all coding with agents, it's clear that the gap is not a bigger context window or adding another MCP server. Until agents have the same tribal knowledge you have, the value you get from them is still limited. And this is exactly the type of problem our sponsor today, Unblocked, is solving for tens of thousands of developers. Unblocked is the context layer for modern engineering teams. It brings together your PRs, docs, conversations, tickets, and more, turning them into actionable context for both your team and their AI agents. Engineering teams that are getting ahead right now use a context engine like Unblocked to make better plans. Write higher quality code, use fewer tokens, and spend less time in review cycles. So if you're running Cloud Code, cursor, or any agentic workflow, give Unblocked a try. Get a free three week trial at www.gannemblock.com/refactoring. Excited. So the first part, I mean, just to give people more context, we try to analyze and survey people on the whole product development process. So not just the coding part, which is what gets the most attention these days with AI and everything going on, but also going back to the first steps, like the planning process and what happens mid-cycle, what happens with how the knowledge gets shared, and a lot of these things that usually get a little bit less of focus and attention. And so we started by looking at the planning process. And one of the first findings that we got is that it feels like planning is extremely fragile for most teams. That is, more than 60% of teams discover routinely missing tasks and dependencies mid-cycle. They need a lot of clarifying questions. And this is also true for companies of all sizes. It's not like startups are doing this better than large corporations or vice versa. So it feels like a inherent problem of how we plan product development work. So if you compare these with things you see from your own experience and your own customers, do you see this problem? What do you think? For sure. I might have a bit of a hot take on it though, in that I'm not ready to call it a problem. Yeah, I certainly was surprised by the number of 59% of engineering teams pointing this out. Yeah, I mean, it's high. But one thing that I'm always worried about are in the various organizations that I've worked in over the last three years. The companies that put too much effort in planning upfront oftentimes didn't have any more success in this discovery of missing tasks. I mean, one of the problems with planning is if you sit down at the beginning of whatever iteration and beginning the story, like the process you're using, you just think really, really, really hard about what it takes to get something done. That's fine. But most of the bumps that I see happening in the road happen because of the unknown. And so I'm actually okay with there being a process, mid workflow discoveries of missing stuff. It's going to happen. So that's the hot take. But one thing that I would say just coming from my side of the house products is that I've seen different disciplines for product managers, some product managers being far more on the business side, less particular in the stories or specifications or whatever it is that they're building out. And in my opinion, that's where we do have a problem when there's not that same level. I mean, think about what we expect our engineers to do with like peer review or paired programming or having multiple people contribute to code base. I believe that PM stories and specifications need that same level of diligence and collaboration early on in the process. And I've been having a great deal of luck success. I don't know, I don't want to be subulls, but certainly the feedback that I get when I interact with say, cloud, clouds actually my LLM of choice. Looking at my stories, peer reviewing my stories, the conversations that we have. I mean, their cloud doesn't always nail it in terms of what are the very specific things that need to get changed. I'd say it's hit raise maybe like 60, 70%. But the conversation is always super useful for me to discover the things that I didn't think of that would have emerged in Cycle if we hadn't had trusted out earlier. Yeah, that's very interesting. And so I want to go back to something that you said at the very beginning, which I think it's very important that is when it comes to discovering things mid cycle, for example, there is probably a healthy and unhealthy version of this, right? I mean, the healthy version is that we should retain that kind of agility and have the and accept that we move forward with things to do without having the full picture and then we adjust. But also there is probably the unhealthy version of these that is we found roadblocks that were totally foreseeable, you know, from the start. We've just been a little bit reckless in not creating good enough specs and we are now having rework and frustration and we cannot hide behind the fact that we are just being agile. Am I getting this right? I think you're nailing it. I believe in that agility in the planning process, but I also believe in there being, you know, some level of in diligence put into the specs up front. I mean, we're seeing a real rise in spec driven development in organizations. So this is going to become even that much more clear when your specs suck, you know. There's going to be no hiding from it. Yeah. Yeah. And I mean, it certainly rings to when you said that we historically we have put a lot of effort into a detailed and thorough like engineering work and even tech design, the sciences system design. But we've not put the same amount of discipline into product specs. I mean, for sure the exists, you know patterns and we know what user stories are acceptance criteria. But even if you survey teams about how widespread, you know, the formalisms are about this. This is not nearly as widespread as, you know, the equivalent for the tech side. What do you think it's the case? It's just that product has always been a little bit less discipline with respect to technology. What's your take? Yeah, that's a tricky question because it's purely from my point of view. I don't know the way phrase that is as product always been that way. But from my point of view, no, actually, I think the role of product management is kind of evolved over the years. And you know that at some point there was like the product manager is the CTO of that part of the organization or the CEO of that part of the organization. You get to run your business and then got really business oriented. And so we used to just have product managers, but then you saw the rise of like product owners. And so there was some differentiation between well, you know, the product managers more owning the business and then the product owner is getting in there and actually managing the backlogs. But you know, in smaller organizations, you don't have both and if you just have somebody with product management experience, then they often are not. I think interfacing with the engineering team in the right way. They just they are building these specifications. There are oftentimes leaving a lot of decisions or exploration to the team to just sort out. And I don't get me wrong. I'm a big believer in making sure that specifications aren't too strict in telling the team how to do stuff too early. I want to fire up the team in all of their collective experience to help solve the problem and talk about the how, but that still needs to then come back into the specs and get codified there so that it can play through into the code that gets written. And I just I think the role of product manager over time has gotten a bit away from that. It's more the product owner that is the one that has been making specs in the in the bigger companies. Yeah, and I think that that ties in nicely with kind of the next finding, which is exactly about specs requirements. And basically it ends up only one team out of four has for engineers in one team out of four both success criteria and actual acceptance criteria are clear from the tickets and the features specs that they they work on and they receive. And actually the surprising part of this is that the thing that feels the most unclear is not like the final goal of the success criteria, which is what we sometimes say that we need to keep engineers more in the loop on and so on, but more the actual details about acceptance criteria. So the more details specs about what needs to be done and what needs to look like, how needs to work as opposed to what success looks like. So did you expect this or what do you think? That that doesn't surprise me. That's that that sounds about right. I mean, again, I have seen like an erosion of story quality and the different organizations that I've worked with, which is typically our spec and it just seems like they've gotten higher level. And that that puts a burden on the team who is not not always as tuned into the end user needs as the product side in the UX side. So I do believe that stuff should be codified. I go back to when previous statements about the peer review and getting help and making sure that my stories are reviewed before I even get them in front of the team. But I'm still I don't know if this is old fashion or not, but I do believe in the core ceremonies for teams. Now I don't want to be agile. I believe they should be fast, but we do a weekly design review so that the entire team can see the different things that we're working on. Now it's optional folks don't have to show up, but it's their ability to be able to give feedback into the specs that were effectively building between PM and UX. It is in half hour meeting that we do every week, and it does allow the team to get in there, give the feedback early, get acquainted with the stories. And it happens in advance of we do our design review on, if it's Wednesdays, let me do a weekly sizing exercise on Mondays that gives the time for PM and UX to take any of the feedback incorporated and then let the teams know by the end of the day Thursday. Hey, we've got the latest updated version of what we believe will be the specs. Please take a look at it before we go into sizing. And then we do run a sizing meeting on Monday and you know, we can size any story within like two minutes. We use both the team doing their own poker session, but the the atono tool also has a AI tool that does sizing and looks at looks for similar stories in the backlog and then says, Hey, is this story bigger, smaller relative to that? It comes back and says based off what I've seen before, I think this is small meeting large, whatever, but we can turn through those stories in about two minutes. But those stories are getting in front of the developers multiple times before they even write code. And then once we get the code going, I believe in a shoulder surface soon as, you know, there's something that is demonstrable to be able to just make sure it's heading the right direction. So I don't know if that's old fashion or not, but as long as those ceremonies can be done fast and effectively, I'm so a big believer in them in terms of making sure they were building the right thing and that the developers know what the right thing is. Yeah, I absolutely love to hear this because I think you're right. And you know, implying that this is maybe a time in history where some of these ceremonies are in some circumstances, because like less fashionable anyway. And there is, you know, sometimes this push for we don't need out of these, you know, a child with capital A. But I think if you keep things lean and you focus on what needs to be done, having, you know, checkpoints that are recovering that create pacing and rhythm in how people work. So effective in both driving works forward and also creating the feedback loop, right? I mean, creating deep the points in which managers can act and understand that something can be done better otherwise. If you don't have cycles and ceremonies, in my experience at least everything becomes harder to act on because you don't have like the breathing and the pace of your of your team. Yeah, and we've we certainly again, this good sound old fashioned that it sounds like we're in some agreement though that we both like it. We were both just old fashioned, but you know, there are different techniques that we use to make sure that these things stay lean like a lot of the shoulder serves that I get now or recordings that are just placed in the slack. And yes, people can watch them at their leisure, but we still can capture the thread of comments of feedback that happen relative to what's going on in the shoulder serve. So there are ways to make sure that we're taking making good use of people's time. And also, you know, in this day and age were oftentimes dealing with very different work schedules, whether that's because it's just flex schedules or whether it's because they have people in very different time zones. You know, there's there's ways around this now that still allow people to collaborate and get stuff done. Yeah, absolutely. And I think technology has helped a lot with that. I mean, I remember I mean, 10 years ago, back in my days, my start, I wouldn't you have maybe big meetings in which just in case, you know, you would have everyone attending or many, many people attending because wise that they would maybe miss relevant details. Otherwise, I think what you're saying is that today you can have like people who are mandatory to join and then a long tail of people who may join or just get to recording with you know chapters and timestamps and ice armories and takeaways and action items. And so just consume one. Yeah, 1.5 x speed. I know I'm not the only one that's speaking of these recordings, but I listen to them. Yeah, absolutely. And so I'm curious, I mean going on a little bit on a tangent, but I'm curious you say you do like this design reviews and are they open to the whole team? So like plans, designers, engineers, I mean, how do they work? It's out of curiosity. Yeah. So I'm a big fan of Marty Kagan, not sure how familiar I was with his work, but he believes in cross functional product teams. I think the definition of what that is is certainly being challenged as our some of our product team is becoming a gentick, but I still think that it takes cross functional representation to get the specifications right, whether it's humans that are going to work on it or agents, either way, I think that that collaboration, that unique perspective that different people bring from their different shared background, it's all important. And it just, it makes for going back to the first point about planning, being fragile and stuff, being mischievous. The more unique viewpoints that you get, I think the better. You always have somebody on a team that has a good eye for the end user and their struggles. There's always somebody on the team that seems to do a good job of proxying these security requirements and needs. There's always somebody on the team that does a good job of proxying, like performance issues. So, bringing all of these folks together earlier on is important. I'll miss quote Marty, but he talks about something along the lines of if the first time the development team sees the story is when they're supposed to work on it and be failed. So yeah, our design reviews are open to everybody on the team, PM's, UX, engineers, quality. They're actually open to anybody in the company to come. What I do see is that we maintain design reviews that are specific to backlogs and therefore either team or if you have shared team sharing backlog. But the folks in the organization that really care about the products in different teams on different backlogs. Sometimes we'll want some visibility into what's going on just so that they can provide some of their points because there's a company grows. People that were working on something oftentimes get split off into another team. But it's still their baby and they still care about it. I love when people feel that way. That's been one of my challenges with PM's is to. So in some of the organizations I've worked, I've had a lot of PM's in my organization recording to me and the ones that would do best would really consider the product their baby and put that level of care into all of these decisions. And so I love it when I see that in PM's, other when I see the engineers, QA, UX, that's what makes it good. Yeah, and also, it makes me think that you're building a product for product people and developers. So I guess that should be one of the key factors also behind the fact that you can have a lot of dog fooding internally. Yeah. And is that kind of a sheet code to develop a product? Yeah, I was thinking sheet code. It's a little bit not fair. Because of that, but I don't know, you know, over my years, like I worked at ERP companies, I worked at a crisis management company. As a product manager, you do a good job of getting to know your customers, working with your customers. Yeah. And so, you have to represent them. You are their proxy. I do think it's possible like I've never been an emergency management director in my life. But I can play with really well on TV because you get that passion. And if you have that passion, you not only are as good as any one of them, but you actually have as a product manager, the unique ability to channel any of them. So yeah, I don't know. It's just it's a PM skill that I went when I'm looking at the folks on my team, the ones that I can see doing that, I'm like, they're going to be just fine. Yeah, love that. So to wrap up, let's say the segment about requirements and specs that we have been talking about. Yeah, basically we have said that in the report, we are sitting and looking at the fact that three teams out of four, at least engineers on these teams are saying that our requirements suck. Okay, they're confusing. We don't understand them. And you clearly have a great experience making your team participate in this. And so what's your take about what makes how do you make requirements not suck that you do you do you rely there are templates PRDs you can follow or the secret sources simply as you said, growth participation, having people involved early on if you had to point to point to one thing that you know, this is what enabled us to have great requirements and specs that people understand allow us to be a great product. Yeah, the latter. So like I said, I do believe like peer review with AI I think can really help up level stuff before you get it in front of other people. So that I mean, everybody's nightmare. I don't want to look like an idiot in a public space. But I want the engineers to be able to say that this spec sucks earlier. Right? Because if they say that really enough, we get it fixed before there's actual hands on keyboards or code being generated from whatever mechanism. So it is that collaboration piece early that I think cuts this off at the path. I mean, nobody's perfect. I'll expect you to be perfect book. So that the opportunity to find those gaps and problems early and get them fixed is important. Yeah, I love this take and I completely agree. So I want to to segue into something that it's not a finding from the report, but it's probably slightly correlated. So I want to get your thoughts about this that is about how knowledge gets shared across the teams. Because what we found is that it's not really shared a lot or at least most two teams out of three believe that critical knowledge mostly lives in key people's heads. And they really feel like relying on written down documentation and information about what exists, how we do things. Because this stuff either does not exist at all or is scattered across so many like incoherent or redundant sources that it's not to be trusted. Really, you have to get to someone who knows who knows better. Maybe that's related to the requirements problem also. What do you think is that your experience as well by working in many contexts? Yeah, for sure. I mean, context is a word that we're hearing so much these days and folks want to context engineer and capture knowledge. And I think engineers are acutely aware of this and have gotten out there. In order to get AI to do what we want it to do, there is a number of pieces of context that has to know. I mean, it has to know coding styles. It has to know how you build your code. It has to know what the different services you use. Like all the stuff that relates to the technical side of things. It has to know what the product currently does. And that's a piece that I don't think is as well captured these days. And then it has to know what it should build. And I think in that regard, that context is really, that's the spec, which is typically a story. But that gap around the product knowledge is something that I see quite a bit in organizations and it's something that I've taken a bit of a personal journey to try and figure out how do we capture that and make sure that it shares so that one thing I do find is slightly tangential but when I look around at any given organization very few organizations I think are able to have the same level of accept the success using AI technologies. If you're in the ferropic, they're talking about 90% of our code is age-adjointed but I don't see that being the reality in most of the companies that I talk to. What I do see is that I call them tremendously clever individuals TCI's maybe I don't know but they figured all out and they have all their different files, look I've got these marked down files to cover this and that and the other and I'm checking them in to get up and anybody can use them if they want but it's not really mandated in the organization and it's really only being built by you know a few individuals and they're really knowledgeable in their area but then it's just going to fall down the farther away that you get from it and so you know I think the real trick is how do we make everybody in the organization a tremendously clever individual and have them all singing from the same songbook and so I think they're in my world back to that personal journey there are a couple of things that I want to make sure happen. One is a product context around what the product currently does. I want to make sure that that is well established so that any decisions are being made in the early design planning type process they're informed by you know one of the ways that you build great applications is consistency. I don't want to go to different pages and have things work completely differently. I don't want my application to be like a children's discovery table where everything is being slightly differently so knowledge of how the product currently works but it's currently capable of is hugely important in building the plans properly. But then as you make decisions about how are we going to do it now there's discussions that the team has and there's discussions that I have with AI being able to capture those design decisions which you know a lot of it gets captured into the ACs but not all of it some of why is oftentimes not in the AC. Some of the things that I decided not to do is also not in the AC and so as you begin starting to actually implement the spec and you realize hey there might be something off here maybe I'll try this other way and go back to the PM and say hey look I did this way, is this work if it's already been identified that that doesn't work you know all of that sort of context I think is important to capture. So the solutions within our own product atono are designed around being able to one generate product context and then hand it to the different systems that need it. The other is this AI context and design decision context and putting this in their central place. Not I know I think that I'm a vendor and almost vendors come on a shows and they say I have the solution this is what you have to do. I don't have all the solutions the the way that all those design decisions and then like implementation plans like you know cloud plan mode type of notification on the technical side all of that information I'm you know putting a lot of effort into capturing and formatting because I believe that it will have powerful usage as software ages or as you continue to build on it you're going to need to go back to those core fundamentals of why the design is the way that it is and I'm trusting letting AI drive and capture all that information and then let it have access to that information and update it. So I don't have metrics like this is where I think it's like yeah how's going to solve everything and I can't prove that yet but I'm a big believer in instrumentation and you can't act off of what you don't capture so we're capturing all this information and then seeing how it's fitting into the different agenteic workflows that we're seeing rise within our customer base. Yeah I think it's pretty much an open problem because I think today there is this I think huge opportunity in capturing more things more information and more context because AI can make great use out of it that before maybe it was useful for humans but to certain extent or at least there was you know there were real trade-offs that you can you could evaluate about whether you know capturing and maintaining up to date more or less documentation and and I think what's still at least not clear to me is how me what type of information and documentation you you may capture and how because I mean you mentioned many mentioned different things for example for sure it's very important to capture what exists today like how the product behaves right but also what decisions we made and why so the why behind why the product behaves like it does today but even you know things that maybe have been superseded and are not true anymore maybe there is a history of why the thing that does this thing today behaves like this right and so how do you capture all of that I mean if you are a person you remember that history in your head and you know that's what we mean when we say tribal knowledge and so you can connect the dots and make the right decision but if you want to pass all of that to Claude maybe Claude would have the intelligence to make that decision if it had all this information but how do you capture it how do you keep things up to date I think that's I mean the jury is still out on that I don't know if anyone has a good solution that that works on you know for the long term yeah again you know I see vendors saying they have all of the answers but yeah it doesn't actually hold true when I start pulling it apart but I do think that AI has a a pretty solid ability to summarize this right like it we see it all the time summaries help out humans ears summary of the meeting type stuff but it's also for its own self preservation as it's trying to manage context windows and it has these gigantic conversations and then it's every so I was like I'm going to summarize this and so you know right right now again I'm I'm trying philosophically to let AI drive more and you know in my background was like enterprise class systems I've always tried to like force things into specific data structures and you know here's here's a decision record or something like that would have been the way that I would have tried to solve this you know five years ago yeah now I you know I'm I'm not I'm actually usually just trying to provide some storage for markdown contents to let AI populate what it needs to with a markdown get to it adjust it manage it over time and you know with all the powers of AI it has I believe a very solid ability to do what used to take ETL tools in the past so if at some point we decided I don't want this to mark down anymore I want this in a structured format yeah you can have it do that for you you know take take this large document and I want you to convert this into these structured records go yeah so I try not to worry too much about format because with all of the exploration that we're doing in terms of what what is the different contextual elements that actually help at a gentle flow operate autonomously I in that discovery I think we should be worried about going fast and so yeah they're through just not worrying as much about the structure letting AI kind of guide it and usually the answer ends up being markdown yeah that's true I mean that's it's it has an incredible capability in you know mangling different formats and just figuring out things even if you give it very little instructions you know about how what the format is how it works and it makes me think I mean we have been talking about AI and workflows and shared docs and artifacts actually one thing that came out the report is that maybe it's surprising maybe not but most teams actually don't have a lot of shared AI contacts documents even just you know clothe.md files the majority of teams don't even have that they have what you call like the tremendously clever individuals that patch together things for themselves and and they become like 10x or one underdecks engineers in Southern organization but they're either you know keeping their secrets guarded or I mean there is no structure to create this shared progress as a team So what are you doing to enable that instead? Or and why do you think it is the case that we are, I mean, on one end we're seeing 90 plus percent of individual AI adoption. I mean, every engineer, every PM is using AI, but we are still far behind when it comes to how we work on that as a team. - Yeah. So with more determination that I can answer from the product standpoint where the context is, product context and where the context is, the spec, the story, with less confidence and swagger, the answers for the engineering team and all of the concerns that they have is in something that I would say with any conviction. What I think the trick is for organizations and product managers, product owners is to give the team time to figure out like what I find is that stuff off the side of the desk then done off the side of the desk typically becomes that the main of the tremendously clever individual as opposed to figuring out how to do it in scale for the organization. So just like you would have SREs and architects and other folks making about what are the different practices that we have for engineering. And oftentimes you have people that focus on like all the build tools and the QA tools and all those other things. I think you need to provide the team time for individuals to perform those roles in building out the context required. As you point out, there's rules files and the cloth file and all the things that help define the syntax style and where everything lives and the things that are like above anything else, never do this sorts of directives that you get double exclamation points. Yes. But it is always funny to me when I do interact with AI and I don't get the result that I want from the prompt all off and ask it. But why did I have to correct you on this? How could I have raised it better in the original prompt? And it will come back with something where it's saying it's adding to the sentence that I had in my prompt to say this is important in front of that part of the prompt that I'm like, how does this work from like a rules agent standpoint? Isn't everything important? But neither here nor there. I just, I don't think that product managers are usually giving the team's time that they need to do this. The tool that we build a tono on the software project management side I think is the right place to be able to have things like product context and the stories are obviously going to come out of us. But developers, the vast majority that I talk to you is GitHub. You know, like I see it, I see it, it's about 98% of the customers that I talk to. And so I would expect those files to be well defined, shared amongst the team, whether the scooping appropriate, like you might need different rules files for this area of the code A, we have a repo for the front end versus the back end, you know, those will have different rules files. But that needs to be considered a first class part of that backlog. Or what I have seen done at some organizations is that there's a type. So would you look at the overall velocity of the team? You basically carve out some of it to say, look, PM, you don't get to say what happens in this little piece. We just carved out you get this for all of your, you know, a fun story building. But then this other piece here is where the team can do their tool sharpening. And so then the team has the autonomy to do the tool sharpening. That's the question I would ask you when you say time, what do you think this is, this should be implemented simply as, you know, at the most luck, you know, the to each individual task, you know, to embed some of this into each activity or some like intentional time every cycle for sharpening the salt, right? And that is just about that. I think it's the latter. I don't think PMs are the ones to be driving this kind of stuff. I want the PMs and product donors to be acutely aware of the user that the actual customer able to represent those things. And I want their time spent on building great specs, meeting with customers so that they can represent them. In those specs, I want their time on that. I believe that the craft of building the product is best defined by the engineering organization. I like to see team leads that take care in this and have good instrumentation around the team's velocity. And how is the like, I love to see teams that are able to scale linearly when they bring more resources on, you know, like pretty lame when you hire somebody and you know, do any sure turns? Yeah. So, you know, I really like to see team leads that have an eye for that. Someone who's looking at who gets promoted into a lead position. I like to look at the analytic capabilities of that person. You know, you're oftentimes are giving the best coder the team lead role, but that's not actually I think what makes a good team lead. You know, they're the ones that are good. They're good at figuring out the process problems. They're, you know, very data oriented to figure out where is any given bottleneck and then approaching the problem. Like you said, an each iteration was some dedicated time to say, all right, I have noticed that what's holding us back now is this. It's testing. Let's go ahead and put in some time now to see how we can improve that. Yeah. Yeah, I think one of the obstacles to do that is also being aware of what is holding you back. Right? I mean, if you are talking about AI, and I think it's a really great result if you develop that kind of awareness to be able to say, the thing that AI is getting wrong, the most of the time is this. Right? I mean, it's either testing or that interprets poorly, you know, specs and stuff like that. I mean, what's the part of the pipeline where we are the most behind it, where we need to, you know, human intervention to save the day? And I think it's still very much unclear how to measure how I perform some on all these very steps. And it's, I don't know if you've figured out something, but it's still pretty much obscure, you know, how to instrument your talk about instrumentation observability. I think it's very important, but still unclear how to do that. Right? I think. Yeah, I mean, a lot of metrics that we currently use are fairly gross in terms of like, you know, philosophy average is, right? Yeah. It's just an age old practice, but I still think it gives you top level views into what's happening and, you know, are you able to get that, you know, linear increase every time you're used to have somebody or if you bring in a new technology that's promising a velocity increase, do you actually really see it? A productivity increase, do you actually see it? Resulting in more velocity. So, yeah, certainly instrumenting that is, is there's probably going to be more metrics that we have to throw in as you go from co-piloting with AI to having pieces of their workflow being done agentically to having like end to end work get done agentically. I think there are going to be different metrics that arise that are important, but this is different, but I think that it's also just culturally important that yes, you can have reserved time to say, hey, look, I think that there is something broken in a major workflow level that needs to be addressed. Yes, that's good. But culturally, I do think that it's important that engineers be coached that when you don't get the right result from AI, it oftentimes is like, you know, it doesn't know how to write Java. It does fine with, you know, Python, it doesn't get Java and I'm like, that's not actually an okay answer. Like, is there something that you can do back to the various rule files and other things here where you can teach it to do this better next time, fix the inputs into it. Don't just say that it's broken and so I don't think all of this refinements around the various tools that the developers have happens in that carved out tie. And I think it's up to every single developer to figure out out how to maximize the results for their area of the code base and contribute to it. So those contacts files should have some level of peer review and other sorts of diligence because everybody should be contributing to them. Right? I mean, it should be pull requests. And when mergers are happening, they should be reviewed. Yeah, they become core part of your platform. I mean, that ensure the productivity of the whole team and they should be discussed. And I think, you know, when you find something that works, it gets shared across the team by default. I mean, it's part of the, yeah, the platform of the overall product. I wouldn't even know how to call it otherwise. And I pretty agree. And I think the last start that I wanted to share with you that it's from the report that it ties in nicely with this. Even if it's a little bit depressing is that when we talk about AI and how teams are using that actually turns out that less than 10% of the teams are using AI for helping with product requirements and products specs. I mean, the things that we've been talking for an hour, you know, that just one out of 10. Is that surprising to you or what do you think is the case? Yeah, 100%. I would have expected this statistic a year ago, two years ago. I shocked to see it, you know, popping up in the last two months since we ran the survey. It's crazy to me. Also, because I mean, if I need to reinforce and make it even more speechless, I think from the outside, sometimes, you look at the side guys and it feels like product managers are the most AI peeled, you know, and the AI ready people in the product development space instead. It feels they're not after all. It's yeah, it's shocking to me. I mean, I guess when I look at the role of a product manager, and why I've always gravitated to product management over the years. Again, I've been in product management for a long time. I've been in engineering sales, consulting, like other elements, but I always come back to product management because the role is to interact with so many different parts of the business and the that sort of, I don't know, the knowledge that you have is oftentimes like, hey, look, I have to represent UX. I have to represent marketing. I have to represent sales. And it's a hard thing to do. And when I'm interacting with AI, I find that it helps me get those perspectives because AI sometimes act as a junior UX resource. So while I'm relative to sales concerns, I will be able to represent marketing concerns, technical concerns, like a lot of the conversation that I have with an LLM while I'm building out my stories is just that discovery of other concerns that as hard as you try to represent them all, the AI acts as those other team members. So when you use it, hey, Kim is out there. If you want to feel like not just as an army of one, but like an army of many of them, those just you bring your AI minions along because it can really help you get stuff done. Yeah. And if you had to guess, I mean, by thinking about your own AI usage that is successful and trying to guess, instead, what is something that other product managers, maybe I'm not figuring out what that might be. I mean, do you think it's about creating good workflows or templates or, you know, default styles of interaction so that when you have to ask something about product specs and ideas to an AI, it's repeatable, it's easy to do and maybe there are other people for which this is hard to do because they haven't developed any kind of infrastructure and good defaults around that. What do you think is the bottleneck with the problem? I mean, you do have to get to know your tools well enough. Again, in sheep code, I'm using a tonal and building a tonal. So I'm really knowledgeable in the tools, but I've also done like with an MCP tool with the MCP server and and cloud desktop, I can bang out. I have no blank page issues. I can have a conversation saying, I pass in the product context saying, this is what our product does and then I say, hey, look, I want to add such and such. Let's talk about it. We have this conversation and then I say, okay, let's build some stories. Showing the stories we look like that. I get asked questions and this sort of interaction at the end, I'm usually able to say go and hit the story literally right in a tonal. Which is very early, you start very early, or like like a thinking partner in with the bouncing ideas off yet, absolutely finding your thoughts and so on. So there's that. I'm oftentimes able to say from that, say, show it to me. I want to see a mock up. It'll build these mockups. And I think one of the things that's hard for people when their building products is a lot of usability isn't in a screen. It's in the transitions between things. And so interactive mockups really help you envision like am I disclosing the right of that information at this level, clicking and then seeing more like I was just working on a feature to be able to take a story and split it into two stories. Interacting with hierarchical acceptance criteria, where you place that click or unclick, you know, what do you expect it to do relative to selecting all the children. Having interactive mockups really helped refine in my mind what needed to get done before I handed it to you. So that piece I think is greatly facilitated to say show it to me. But then the other piece that I see is just like product to you wise. Once you start getting going like if you're working on an epic where you know that features that I have a big feature that we're talking about, it could be eight stories. I can sure knows that lickety split, but in something like I had a session the other day and within like four hours I have built out 30 different stories because they were all interrelatives about MCP tools that I was building. And the ability to be able to go through and because MCP should be like an API in terms of their it should have a consistent style to the way that you do everything because I did it all within the same chat. It's just aware now of the design decisions that I've been making, the format that I wanted things in. The team I wanted stuff assigned to. So I could just bang stuff out. There's no way I could have written that many stories in that little time without that technology. So it's quite activity booster for me. If I had to guess you know for you know the different one of the key differences between how you use this and some other team that is not at this level. I think the fact that your cloud and AI has a lot the whole product context about how things should be done, what it says. That's really a killer factor because you can probably you know just in a couple of lines describe your product intent about something you want to build and cloud can figure out so much about what you mean by that. How should we build maybe create you say an interactive product for Mocha but maybe that's already kind of high fidelity right because it can build on the design that we're very exposed. I mean something that it's kind of faithful if not the real thing of course but. Here's some honesty on that. I have very few stories where the mock ups that I'm generating through AI are the only visual artifacts at this point the UX team is still putting stuff through Figma and so we're trying to figure out like what are the workflows to actually automate that. But I do just want to fully disclose again I think most vendors are like yeah I just work with that but yeah we're still we're still experimenting with that and you know what stories lend themselves and having to go through that but also like how can we just automate more of that let's generate the Figma artifact as opposed to what came out of cloud doing some by mocking. Yeah I think it's especially useful you know I saw direction I mean directionally you should be thinking that you give more more context so that you're you know you're prompt about what you want to build this smaller and smaller because the AI can figure out more things by itself. So Doc thank you so much for this great chat I love your comments about our findings on the report from your own experience and I guess we'll run this maybe one year from now and see if we have improved on some of these stats. Yeah. I'm trusting my fingers, let's add product teams, we can do better. We can do better. Thank you so much, Doug. Thank you. Thank you. Bye bye. Thank you so much for listening. If you found this chat valuable, you can subscribe to the show on YouTube, Apple Podcasts, Spotify, or your favorite podcast app. Also please consider giving us a rating or a living area view, as that really helps other listeners find the show. You can find all past episodes and learn more about the show at refactoring.fm. See you in the next episode.