The Most Misleading Plays in Product, Unmasked

Things aren't always as they seem, from bait and switch subscription pricing to the people who swear candy corn tastes good, and we have no shortings of reasons to be skeptical of everything, especially right now in product. The speed of change in this industry is moving so fast that trends can start looking like best practices, and the real best practices can start looking outdated, but when the goal is to build a product that will scale over time, it's on us to check ourselves from getting sucked into high-trained hypnosis. And I guess today is not grainy, CPU of SLEGO. Matt has spent over 20 years in B2B product and almost in 9 of them scaling the product team at SLEGO from 4 to over 45 people, and while AI has proven to be an extraordinary opportunity for the company, he's noticed a gap between expectations and reality when it comes to some of the shiny new tactics blowing up our LinkedIn feeds. From prototypes and masquerading his production ready code to too good to be true tools, he'll hear his take on where to tread with caution, and the time test who wisdom he's counting on to make it through this AI transformation alive. Let's jump in. Oh, by the way, we hold conversations like this every week, so if this sounds interesting to you, why not subscribe? Okay, now let's jump in. Welcome back to the product manager podcast. I am here today with Matt grainy, he's a CPU of SLEGO. Matt, thank you so much for making time in your schedule to talk to us. Thank you Hannah, great to talk to you today. So can you tell us a little bit about your background in your journey to becoming the CPU at SLEGO? Yeah, so I've been in B2B product for a long time now, about the last 20 years, SLEGO is an integration platform, and so I've been with SLEGO for eight and a half years joining just after series A funding, and another five and a half years before that, also in the integration space, it wasn't really by design, it's just how things worked out. Before that, I was involved in products around the software development lifecycle, including UML modeling tools for those of you who might remember what those were. Originally, I was a software engineer back in Australia working in telecom and defense, and first came to the US to work as a sales engineer actually for a product that I'd become a power user in while I was at Motorola. So kind of a diverse background and then took a tour of duty in product marketing and then finally made it into products. A tour of duty, I've never heard it describe that way. That's funny. So today we're going to be having a little fun with a very quasi Halloween themed episode. We're going to be focusing on the theme of bad product plays in disguise, and we're going to start by digging into vibe coding. So lots of opinions being tossed around in vibe coding in the product community, and I'll be the first admit, no code platforms are amazing tools. I use them all the time, but they can also lead to some very Scooby Doo-esque moments, mask off moments where we take a closer look, and it's, you know, not what it seems. So let's be in your experience, Matt, with vibe coding within product teams. Tell me the good, the bad, and the ugly. Yes, and we would have got to weigh with it if it wasn't for you pesky kids, right? So, yeah. I think it's democratization which is incredible, but we have to also think about the delusion. You know, with that power to make something look good, it doesn't mean it's necessarily built right, and I think there's a lot that has to happen under the covers. We shouldn't confuse perhaps quick prototypes with production-ready code. And so, okay, citizen creators, but you also need citizen architects, and in the context of the business wearing, for example, B2B, we're dealing with, you know, we're infrastructure software. I mean, we're talking about billions of transactions a month. There are only certain parts of the application, perhaps on the very front end, where we might be comfortable vibe coding anything. And so I think, you know, it's got to be about the right tool for the job just as it's always been. And, you know, while also making sure there's a culture of experimentation, making sure that we're encouraging the team to take risks, to try new tools, and stay calm with the latest development. So what is truly a groundbreaking time for the whole industry? Mm-hmm. So just to go a little bit deeper on that, when we think about, you know, the various levels of understanding of the technology, just all along the organization, you know, it can be very tempting for folks who are less experienced either with the technology or new founders themselves to kind of see what looks like functioning code and just kind of run with it and puts a lot of pressure on engineering teams to kind of match that speed or be able to develop at that level that quickly and make it look, you know, so shiny and new. So how do we help stakeholders understand the real constraints and considerations between the prototype that we're seeing from vibe coding tools and production ready software? Yeah, that's such a good point because, you know, speed to demo is not the same as speed to production. And especially, you know, we're talking about 5,000 customers, again, with B2B workloads. So we sometimes talk about what we do, infrastructure is the plumbing. So okay, AI might paint the house, it's not necessarily going to plum the house, right? You want to sure about some of these things. And there are some challenges because whether it's pressure on engineering teams or even pressure from the exacting, recently I've seen internally, you know, fairly senior members of sort of non-technical staff vibe coding proofs of concept to show new capabilities that are much needed by their customers. It's certainly fired the imagination, but it's also creating this unspoken pressure that maybe this is accessible to everyone that this is something that we can rush into production. And, you know, if we're talking about buildings, if I go back to painting houses or plumbing houses, I mean, these are not necessarily load bearing walls, right? These are the facade, it might look good. And again, that's not to say there is no place for it because the speed of POC is just incredible and we need to be embracing that at every turn, whether that's to help a product manager better explain requirements, whether helping a designer to show alternative workflows, whereas before they might have been going through designing many different screens in their favorite design tool, the power of working code is undeniable. And we need to be looking at embracing that at every possible turn while recognizing that it's not the same as production-ready code. Yeah, and something that's been on my mind as well is that these are tools that invariably they're going to get much more sophisticated. And we, as the user, will also become much more sophisticated at using them, which means of course we're training towards this cost of building, approaching zero. So for yourself as a product leader, what are the concerns that you have about that trend and what have you been doing to mitigate those concerns at SLEGO? Yeah, and I think that's a great mental model, a great thought experiment to run. Like, what happens if the cost of building goes towards zero? Okay, maybe it's fast approaching. The generations of these tools, as you say, the ability of users, skilled users, just as we see improvements in sort of the usual chat, GPT kind of experiences. So we can expect that a dramatic decline, so it becomes so cheap to build. I think that actually counter-intuitively puts even more pressure on product managers to make sure we're building the right thing. So it doesn't absolve us of all the right things we should be doing, looking at product analytics, quantitative and qualitative user research, customer interviews, product advisory councils, all those things, proofs of concept, A/B testing, working closely in a triad of product managers, designers and engineering, right? So none of that goes away. And I think we have to guard against that because to go to a Halloween theme, couldn't out with a bunch of zombies running around, like zombie projects, things that were so easy to build, maybe littering the product with all these ideas that never really quite made it. And I think, again, that some of the older disciplines of product management really come back to the fore, because we have even more choice now, I think the ability to make decisions to drive the right kinds of outcomes, I mean, all that has to remain. I agree. On the topic of maybe ill-conceived ideas, I have certainly been seeing an explosion of tools flooding the market that seem to offer, let's say, enchanting benefits while hiding what maybe even not even hiding, but just offering some very serious risks, like I've seen some fairly egregious concepts circulating that I could just so clearly see an opportunity for bad actors to just manipulate in a way that's not really what we want. So what are some of the things that you're seeing in this space that give you pause? And what role does product judgments still play that just can't be automated away? I think we're always been looking for maybe the magic 8-ball of product management, whether that's scoring methodologies like rice. I've seen them abused as well, because it still leaves a lot of latitude for product managers to have their thumb on the scale to influence the scores, and it doesn't take too many rounds of it to figure out exactly how to move the needle and tip the scale in your favor. So I think we'll see the same sorts of risks here as well. I think there are some tools that promise maybe they're going to vacuum up all the intel, in a product telemetry, every conversation there ever was. But I think we all know there's plenty of things that happen out of band observations that maybe, yeah, maybe it comes from a session replay tool, but it's not like written down in a form that an LLM is going to understand, right? So I think there's no substitute for sound product judgment. And while we are, maybe we have more data than ever, I think at the end of the day, it's incomplete data, and there still needs to be a vision, a strategy, and in product management, there are bets. And yes, we take bets knowing that we're going to be able to measure the outcomes. Hopefully if we do our jobs well, but it still has to be an iterative process. And I don't think there's any magic answer that I give us to suddenly produce an infallible roadmap. Put it that way. I tend to agree, I was speaking to a guest who has not yet been on the show, he was coming up quickly, but something we discussed was this concept that AI is kind of a jagged technology in which there are some things that it's very good at, you know, we can exploit those advantages, but then there are things that it's not so good at, that it looks like it's very good at. And so, yeah, it's a matter of the product sense, but also understanding the technology well enough to be able to kind of check yourself on like what are you really relying, overly relying on the LLM's to do for you. And I think even with basic chat interfaces, I think we've seen plenty of examples of some AI's being quite sick of Antic, right? And if you're not really awake to that, you might begin to think you always have great ideas. And so, personally, I like to spice it up a little bit and make sure that I have sort of an alternative view and ask for a hypothetical review of the ideas I have, because otherwise, I'm always sounding like a genius when I talk to my AI. You know, they love us, don't they? So let's kind of move past by putting a little bit and I'd like to talk about some other, let's say like attractive, but ultimately, unsustainable shortcuts that we're seeing product teams take right now. What are some of the bigger offenders that you've seen around? I think one of my favorites is OKRs. I think we've got sort of a troubled relationship with them, maybe at a company level not too bad, but I think in product, it can be a bit challenging. And I think, you know, it's what happens when you try to import something from a fang in this case, like one of the big name companies without necessarily having the rest of the culture to go with it. I think that's in general, you just can't graft on a limb. OK, now we're going to talk Frankenstein's monsters, right? If you don't have that sort of as part of the culture, these things are never really going to knit together properly, right? So that's one, I think there's also, we've all seen metrics theater, a bunch of vanity metrics that don't really tell us much or don't drive better decisions. I think that's a gotcha that's been around for a while. You know, maybe we talk about feature factories or feature farms. And maybe now we have the ability to farm about the acre, right? Because of the scale, again, our ability to produce is going up. How do we make sure we're producing the right things? So I think any of these sorts of things, essentially shortcuts, again, looking for magic solutions that apparently work somewhere because someone read them on X or on LinkedIn somewhere and without that sort of rigour behind them, it's just going to erode trust, I think. Yeah, I would agree. And on the topic of eroding trust, I think that one of the ones that comes to mind for me is how it's affected the UX research community. I know that the UX researchers, I think, have long suffered as being sort of the underappreciated aspect of the product process. But now LLM's, I think, we kind of get an even more muddled view of how to conduct that correctly and kind of the role of AI and assisting with that process. That's one, like I wish I could shut it from the rooftops and you just cannot substitute these research with LLM. And our head of obvious research recently was telling me the same thing that the AI transcripts are great, okay? You know, verbatim, this was what was actually said, but the insights often miss the subtleties certainly at the moment. And maybe that'll change, I think, as we say, there is continued generations of this technology. I think we can be optimistic about the future. But for now, I think it says it's most important to understand the limitations and guard against them through a fashion craft of good user research. So switching gears a little bit, let's talk a little bit about your experience scaling teams. So you've been doing it for a while. You've scaled your team at SLEO from 2 p.m.s to 10 times that size. You've got lots of experience in building processes. I'm sure that there have been missteps along the way. So can you tell us a little bit about what has been sort of your, or a few of your best takeaways in terms of scaling teams from small organizations through their maturity? What did you kind of learn that you think still holds true even today in this fast moving time of AI? So it has been a journey, as you say, Hannah, so I joined the company in Heritage to 2 p.m.s and 2 tech writers. So a team of four, and now we're, you know, about 45 close to 50, right? So it's been a lot, and it's a team of p.m.s, designers, researchers, technical docs and product operations. I think what I've learned is maybe the order in which to do things, right? So at the very beginning, life was simple, you know, by my side, on the more the platform, it's like working directly with our CTO, and three of us around a room prioritizing an entire backlog, right? I mean, it was, it was as simple as that, that that clearly doesn't scale in the long run, you know, dealing with offshore teams, both p.m.s and engineering being offshore, complicates things. And we gradually added process at the beginning design was just the best we could do with the tools we had. So it was p.m.s doing their best, you know, always thing. It was almost like stitching together screenshots like a ransom note. It's kind of how I think about it. It was so, so grungy and almost embarrassing. And then really, I think our first foray into, you know, professional designers, we didn't do that. Well, I felt like we used design more like an agency model, right? Make this look pretty instead of really thinking about it as user experience, as opposed to just design. Docs, we've always been fairly good, and that, that has continued to evolve. So we've added process as we needed it. I'm not saying we're perfect, but it's tended to work for us pretty well, and maybe haven't got all the right ceremonies in place at times. It feels like, but I think sort of directionally it's been correct. And it's being a case of just enough process and responding to the needs of the business and providing room to grow for all members of the team and so on. But it's been a journey. I've never run a team this large before, right? There are a lot of firsts here, and I have some battle scars and gray hair to prove it. Oh, really? Oh. Yeah. I'll blame my kids, maybe. Okay. Yeah. You can blame everything on your children. I do it all the time. So to expand on that a little bit and kind of tie it in with what we were talking about before, I'm curious about some of the tried and true product management practices that you think maybe are actually more important now than ever. Even if they seem a little old school, are there any concepts or frameworks or practices that you found yourself returning to more than ever or emphasizing with your teams and the age of AI? As I said, you know, as the cost to build maybe approaches zero, I think it actually puts more on us to prioritize. So I'm thinking more in terms of the tools for prioritization. So some of that has to be alignment around a vision and strategy, making sure that everyone on the team is clear enough to be able to make localized decisions. No substitute for first-hand contact with users, with customers, including, you know, I feel like I made my early days of my career with sort of hostage negotiation, like unhappy customers talking them off, ledges or whatever, right? I think there is no substitute for that because it really helps inform the full picture of what the product is about and gives the PM tools they need to better understand how they ought to be prioritizing. So I mean, there's no substitute for ruthless prioritization. At some level, you're going to run out of capacity. I sometimes look with MD at much larger companies, but I know somewhere they have exactly the same sorts of problems. I know all PMs do. You never have enough capacity. So it just comes down to prioritization and all the usual tools apply. I think as I said, where AI comes into play, okay, maybe to help understand a whole and digest a lot of information, indispensable when it comes to research. And then now as we talk about with vibe coding, when put in its place, you know, for rapid prototyping, I think what some people forget about prototyping, too, is the original ideas of prototypes is that they meant to be thrown away, right? They're not meant to be the basis for what goes into production. And if you can do all those things, I think then, you know, the tools are there to really accelerate the way we work and again to assist with prioritization. Yeah, well said. Well, to close on an optimistic note, where would you say are the most legitimate high-value opportunities for AI to enhance the work of product managers? And what advice would you give to product leaders who really want to embrace AI in a way that's thoughtful and without falling into any of these traps or potential mask off moments we've talked about? Yeah. I think really one of the big ones has got to be around research. I think the ability to do competitive research for PM these days, it's a tool I wish I had in the past. Sure, I think part of it, of course, is vendors tend to be a lot more public, like most docs for products that are available now, but you really need to be doing that. That is a huge one. I think obviously quickly putting together documentation. A lot of people talk about writing press releases first or FAQs first. I think this is a great opportunity. It might have seen the laborious to do that before, but it's a great way to get started today. And I think it's about using tools like this to help expand the imagination. What am I not thinking about, right? I think AI, when prompted in the right way, it can be really good at identifying some blind spots. Yeah, I might hallucinate sometimes, but even out of that, sometimes there can be insights or lateral thinking, perhaps, that hadn't come to mind. I think AI, though, can be a bit of a funhouse mirror, right? If it's messed up to begin with, if you don't have your discipline in place, then it's only going to make it worse. But when done right, it can provide that focus that product managers need. Yeah, I tend to agree. Well, this has been wonderful. Thank you for sharing all of your knowledge and for sense checking some of these things that we're seeing so much of in the space that really appreciate it, working folks follow your work online. Best to find me just on LinkedIn, I seem to be on there more than anywhere else. So look forward to catching up with people there. Awesome. Thank you so much. Next on the product manager podcast, if you thought this episode went hard on expectations versus reality, we are about to deep dive into LLM technology in an episode that will challenge everything you think you know about AI. While the potential of the tech is limitless, the current limitations are far more complex than we realize. And so are the impacts on us as both builders and users of AI products. This one is going to hit hard, so subscribe now to jump in with us next time.