How to Measure Your Way to Success by Sephora's Product Manager

So we measure lots of things. We can measure anything today. Anything. So it's important to know what you're trying to do. What are you trying to do without metric? And there's lots of different metrics. And with the ones you pick and how you do it, really matter. And sometimes you're trying to answer a question. Sometimes you're trying to make a point. Sometimes you're doing an analysis that's going to guide a real code change that's going to change an experience for your users. And knowing what you're going in with is really going to set the right tone for what you choose to measure. Oh, we'll fast on the trigger there. Okay. So when we pick those items to measure, when we do that, we're basically saying what's our focus and what's the outcome that we want. And how we do that is really going to change that focus in that outcome. And I'll give you some examples of what this means. These are all things that I've had to measure and that you'll have to measure. And some of you are already measuring or trying to measure or people are asking you what's it going to do for this? And how do I move that? It's a soup of data. So how do you figure out what really matters and how to organize your data in such a way that you can predict the outcomes that you can predict what changes are going to yield the outcome that you want to see in the business? So the first thing I think about is what am I trying to do? What's the action? What's the change I want to take? And when you're thinking about metrics, it's really about finding the ones that are actionable because it links a repeatable action, something that you can do or that your users will do or that your customers will do or whoever it is that you're is using your product. And that that action that you're going to change how they take is going to yield an outcome that you can measure. So you're linking. It's linking these things. And then that link should be through correlation and causality. And I know you guys probably all know, but I love correlation and causality. We talk about it all the time. So, oh, come on. I'm having a technical difficulty. Oh, here we go. Let's take a minute to talk about correlation and causality. So maybe this is a repeat, but it bears just refreshing that correlation is when your data moves together. So you see something, you make a change and an action happens and they are linked. Right? They're linked information. And so it's very easy to sort of see like, oh, those two things are together. They make sense together. But they're not always actually causal. So in causal, you can actually see and show that a causes b. And these do not always, there's not always obvious that this is the case. So I'm just going to take a, this is a real example from a real retail situation. So there was a seasonal dip every year. And in January, obviously, like almost every retail environment has a seasonal dip in January. In this particular case, the team tried to figure out how are we going to get over this dip? How are we going to really boost sales in January to do this? So they actually spent months, months coming up with some great marketing, an email campaign and a bunch of other marketing on the site that supported it. And they did a huge campaign. And the campaign was gorgeous. And the responses and like, you know, offline testing that we did were really, really good. They did the campaign in January at ran in the middle of January. And it was a complete failure. So the actual results were actually less even than the predictions for the lower dip in holiday sales that were expected. So the team was absolutely devastated because they were really confident in this creative, and it was just such a bottoms out failure. I mean, some of you are smiling. I'm sure you can guess what was actually happening. Anyone want to take a, just a wild guess at what was going on? It was holiday. There was a snow storm all over the East Coast. And literally people couldn't get to work. And the last thing that they wanted to do was go shopping. So literally this huge snow storm, a series of snowstorms, but basically it was just a weather event. And it happened to exactly correspond to the dates of this huge marketing campaign. So it took a while. It was maybe like, you know, multiple weeks of very, very sad and unhappy people looking at a lot of information to figure this out. But basically, they, what was really going on was that there was no causality in there. And my only point of this is to say, sometimes you think you know what's going on. It looks super, super obvious. But then if you can dig in and really figure out what's going on, you can find out that it's not just not really the case. And it happens in a lot of other examples that are less dramatic than this. We see it a ton when we're looking at data. And I'm sure you are and do. So the metric is actionable when it's a repeatable action that you're going to change the way people are taking it. And it's going to lead to an observable outcome through causation. So we're real tail tight. And we want to increase purchases. This is this is pretty standard. I'm just going to use this as an example to show you kind of how I think about structuring the data for particularly for stakeholders, which I feel like is one of the hardest parts about being a product manager. How do you talk about data in a really easy way without getting sort of lost in the stuff. So we want to increase purchases from from X to Y. This is obviously a little bit simplified, but we don't have much time. And so first we need to really think about like how does that happen? So what are those set of observable actions that lead to an outcome that you know are causal? So I just, this is a very basic kind of retail funnel, but I've worked in a retail at a number of different places and basically they look just like this. Like all my retail funnels look basically some variant of this maybe with a few more steps, but very simple. So you build your funnel and you know what comes in the top of the funnel and what comes through. And almost every business has this kind of people coming in action being taken an outcome at the bottom. When I'm looking at the data, I'm looking for what is the nugget, what is the that order of events that yields the outcome that I want in my business. And even at higher now where we have this really complex two-sided marketplace, I still have a funnel that's not much more complicated than this that really explains how people are coming through the business. So linking in here, it's a number of customers, they click, they visit the site, they engage, just engage could be seven steps, but it's a step that you can sort of you know pull up together to say like did they engage and then they make a purchase and that's your success. So build that basic funnel and then for me once I have this funnel, I use it all the time. Not some of the time, not most of the time, not when it's convenient, not when it's like I use it all the time. And the reason is that it makes it clear for everybody else to understand that context when things are changing or how we're making decisions about what we're changing on the site because or in the product if we're making changes, it's easy to link it back because I can link to where in the funnel, I want to see an outcome from the change in the product. So if we want to increase purchases from x to y, then we know we need to increase these other things at a predictable rate and if we link and link it back every time the same way, then as you're going through your metrics, you're you're always telling the same story, but every change that you make, you're able to link it back in this really structured way. And your funnel might look different and you may decide not to even have it be a funnel, but the idea is to be very clear about the business outcomes at a high level, at a business level. What affects the actual profit and loss of the company and link your funnel back to that? So when you have your funnel, there's really only two ways to move it. You're either putting more people through it or you're increasing the effectiveness of each step or of a step or multiple steps in that funnel. That's it. There's no other magic. None of the other stuff that you do is going to change your funnel metrics. And this is a place where as a product manager, and I'm sure that those of you who are product managers too, you get suggestions all the time for stuff that's great to do, fabulous ideas, but they're not going to change your funnel. They're not going to yield an outcome. And so being able to really clearly understand how certain changes are even able to affect your basic business funnel helps you do a very simple prioritization and to explain to people why that prioritization is important and why that idea while brilliant is maybe not going to yield what they think or is even bigger, even bigger opportunity than they think because it has a direct impact on either increasing the volume or improving the way that people are going through your funnel. So another part of—oh, okay. I feel like I should like not touch it. Okay, so another element I want to talk about is segmentation. So people talk a lot about segmentation and what does it mean? And for in this context when you're sort of building in your funnel metrics and a funnel can be any series of steps that leads to your success outcome, the segmentation is really looking at that same data in the same way but through a different lens, cutting up the way that people are going through it and really looking. The most common are by source where your users coming from, by demographics, are they men, are they women, are they from a different country, just anything that's demographic about them, it could be psychographic also, but being able to cut up that data and see how they move through the funnel differently. This is probably the most useful type of analysis is in being able to look at the same funnel, look at how different populations are moving through that funnel and see what the differences are between them. Because if one segment, I'll just give an example from our—I used to work at a hair color company and we had a particular segment of people who just didn't get through the funnel. They just didn't get through the funnel. They would come, they came in huge numbers into the top of the funnel and then they just kind of dwindled out. And there was something about our product that just didn't appeal to that segment but because we were able to separate them out, we were able to say, you know what, it actually is like much more successful than we thought for this other demographic segment, but for the one that it's not appealing, we can actually segment them out, treat them differently. We had to build some mechanisms to do this and then we were able to try to optimize for that segment because it was a pretty big segment. But we also would have been able to make the choice to just not work on that segment because they're just not yielding anything. So then we could also make the decision if we wanted to to say, you know what, those guys are never going to convert. That's just not our audience. They don't want this product. You know what, I mean, if there's certain people just don't want your stuff, like, that's fine. Don't waste time trying to convince them to do it. Work on this segment that has some opportunity. And if you're able to segment them out of your data, then you're not going to be confused by them all the time and you can really focus on it. So segmentation is really important. I had a great product manager who worked for me. One of the best product managers I've worked with and he was so rigorous about this. He would come through and show me the funnel and then he would show me the segmentation and what he would do and I recommend this to everybody is he would look for the best performing funnel and then use that as his benchmark to say, this is my best performing segment. How can I get all my other segments here? This is what I know is possible because I see it in this piece of my funnel and these customers I see how good it can be. I see what conversion rates are possible for me and I can prove it because I can show you when these customers come in in this source, they perform really, really well. So what's working for them and how do I figure out how to make that work for the other segments that I'm looking at. So segmentation is a very powerful way to cut your data and set some really clear targets for how to improve your product to get the outcomes that you want for those So we gained some insight. We did some tests. How do we show people? How do we show people what we did and one of the most sort of helpful things is to show people the outcome in exactly the way that you showed them that you made the decision to work on that segment. So I literally use exactly the same slides, exactly. So I say this is what we're going to do and this is why and then I bring the exact same set of data back and show the difference and that sort of closes the loop in a way that is really clear and takes away all the sort of confusion about whatever else might have been happening or specifics about the project. This sort of consistency of the way you look at the data in making the decision and then that same consistency when you're going back and saying what you learned after you made that change is really, really helpful to bring your stakeholders along with you and avoid confusion about the difference. You know sometimes you go into a project you think it's totally clear you come out you're like what did we learn if you can present it in exactly the same way then it's really really clear to people. So this is just a you know obviously slides are a little bit abbreviated but it's exactly the same and all I'm doing is I can call out and say before the change it was this and here's how much we moved it. Before the change it was this and this is how much we moved it and then the other thing I pull in is what's the business number. So here we were talking about purchases but of course when you report purchases you really want to talk about revenue sometimes revenue doesn't totally go up at the same rate that the purchase number goes up but you can show that relationship. So communicating consistently and tying it back both visually and in the language that you use really helps with stakeholder management and makes it easy for other people to understand why you made a decision to do it what the outcome was and exactly where that outcome was. So let's say you tried lots of things then you can't just do like pre-post you really need to think about how do we over time see the effect of loss of different changes. I don't know about exactly how all of you are managing your products. At higher rate we do continuous release. We might release five different things in a week they all have different metrics but in the end we have one final as a company and so the question is you want to measure the effect the local effects of each change that you make when you can do that but you also need to make sure that the sum total of all the changes that you're making are really paying off in the way that you want them to because some of your tests are going to be great and some are going to be terrible and you're going to do some stuff without testing and you're going to do some other stuff that's like really well researched but all that stuff is one product usually so how do you think about lots of stuff going on at the same time. So cohorts. I love cohorts. Cohorts are like the best thing that ever happened in data management for me in like this product management job of like managing change what do we decide to do how do we know if it's successful this is like the the bread and butter and cohorts are really just showing variations between groups of people at the same stage of the customer life cycle over time. I've spent a lot of time doing subscription businesses in subscription businesses every day or every week or every month however you want to cut it you've got fresh people like fresh new members of your subscription products and Netflix we lived and died by the new people that came in we were like how many new people you know who gets them who who gets to test on them and how many of them this was back in the day before there were resilience of people using it you had to like compete for what tests could go on at same time but cohorts let you do this and it doesn't have to be just time sometimes a cohort can be based on that segmentation that you did earlier but the important thing is that you're looking at like data exactly in the same way with a variable between them and by looking at them in this really organized way exactly the same funnel steps as you saw on the other one so when you're working with your stakeholders you're very clearly tying it back to the original funnel metric that you had or whatever your business model is and then by by really breaking it out by these cohorts you can learn a tremendous amount about where you're having impact and where you're not over time and this is net effects so in this particular example same retail site same thing with the in most retail sites have a huge bump in December and then this corresponding sort of trough in January so the question at any retail site is how are you going to sort of balance that out and how do you know you're getting better when your volume numbers are just all over the place depending on sales and whatever else is going on seasonality so the question is are you getting better is your product actually getting better and when you look at it by a cohort view you can very clearly see very quickly whether or not the net effect of the changes that you're making are having an effect so in this case you know november is looking pretty good here's your sort of a think of it as your baseline metrics you have a certain click rate you have a certain engagement rate you have a certain purchase rate and that yields your your end result how many how many sales did we make December is a complete blowout right you have a huge number of people everybody's buying in December it doesn't matter what it is they just need a gift so everybody's buying and so your click rate goes up and your engagement rate is out through the roof and your purchase rates crazy and you make a ton most people most retail sites make all their money basically during the holidays so many do not all so December is really really important but the question is when January comes and you've spent all this time improving your product you know did you improve the product how do you separate out these huge volume swings from how did you how's the product doing and by breaking it out in this way and really looking at those same metrics over time you can see in this case there was a very subtle but real improvement and click rate a very small but real engagement rate increase and purchase rate remained flat but the net in January because of those product improvements and directly attributable to those product improvements even though you have less traffic coming in you're able to show that you actually have a higher number of purchases because of these relatively subtle changes in your funnel so that optimization becomes super clear when you can cut up the data in a in a way that is is very very structured yeah so let's say number of purchases which are trying to increase the joy in an offer like 40% off it doubles down but then is it another product manager looking at the revenue or every product yeah yeah fighting with you for that or not fighting but um yeah I mean there's always a lot of complexity about promotions um and some companies are better than others about managing the level of promotion at any given time but yeah if this was real like if I was really doing like real metrics for like my boss you know or uh putting CO yeah you'd include in your funnel those rates about um basket size and purchase uh and and average uh item size in the order you working on both problems at the same time yeah and you'd break out your funnel in that case to show those very clearly because if it was the case that you just like you know you did a big discount so of course they bought more stuff that would play out when you you know if you had a much bigger slide you'd be able to show the final revenue numbers and see yeah you would just have it in there all the time I just didn't have that much room yeah okay cohort nothing's more fun than cohort okay local metrics so what we've been talking a lot about stuff that's connected so everything is sort of connected to these end outcomes and your funnel is like a logical set of of measurable steps that leads to outcomes but sometimes you're measuring stuff that actually doesn't ladder up it's you're just measuring it for its own sake to understand that part of your business and it doesn't necessarily like totally connect to revenue or to purchases or to whatever you know renewals and the situation or whatever it is that your funnel is about sometimes you just have to measure them so that you know the health of your business and I think of these like like a pilot who's flying a plane and he's got all those dials where she's got all those dials and you're watching each dial and each dial is telling you something a little bit different each one of them is not something that you're going to change the way that you fly but you might make some small correction based on that so I think of these as local metrics and I report on them very differently so these I just give some examples here these are things that are really not in and of themselves actionable and sometimes you find yourself at least I have in product management being being told or asked to like work on these because it's going to have a big impact on the business and the very first question is how is that going to have a big impact on the business like help me understand why this is important and what the outcome is not to say that they're not important but just being able to really clearly say like yeah we really need to improve these but are they going to have the outcome that you think that they're going to have and how do you sort of lay that out so I think of these as local metrics very important but measured a little bit differently and reported very differently basically if you see something go up or down you don't know what to do to fix it that's kind of the difference in for me between local and actionable if I see user satisfaction go up I'm like yay but you don't know why necessarily right you have to do more research to understand why it went up or to try to fix the problem of why it went down but just the metric itself isn't going to tell you whereas in our actionable metrics you know kind of what the outcome is going to be and you kind of know like where you need to go to fix it that's not always the case for these local metrics so particularly for a local metrics but really for any metric one of the biggest questions that come up and I just recently in the last few weeks had a very very long and contentious debate about with a coworker who is brilliant much smarter than me about whether or not we should use an absolute value or a ratio to define a metric and we spent I am not getting you more than two weeks talking about this and I was ready to like go home and have a glass of wine but we came in every day and we would talk about it some more so values and ratios this is a you know sometimes you just get this deep in the data and you just need to figure it out so I thought it'd be kind of fun to share this part of my life with you and bring you into the madness that is my world and talk about how we sort of made decisions about this because it was kind of fascinating so values are very specific to the the volume that you're dealing with and they show growth in absolute terms and perhaps obviously of course you want to measure this for sure but is it the way that you want to present your metrics when you're really understanding the health of the business this is where the debate came in ratios and percentages they give you the relationship between information bits but they don't necessarily tell you if it's like bigger or smaller in an absolute way so in the end after this very long and contentious discussion we decided that we needed to measure both but I will show you because it was fascinating what it was so so I work on matching which is people come on to hired and they post a job and then candidates come on to hired and they want a job and we match them and we put them in a list we rank them just like search we rank them and that ranking is really important because employers come in and they look at that list in order and in fact they're very structured about it they stop they start at the top and they look their way down so if I put somebody at the top they are by definition going to get more attention and time than the person I put at number five and number ten so it's really important to get that ranking right so what we are looking at here is the number of I put it in terms of retail because we've just been using that throughout but number of sales from the top five how many times when people bought did they find that item the top five and then the this one down here is the percent of sales from the top five so what percent of the time did they find that item in the top five same metric just a different way of presenting it and you can see from these graphics how wildly different they look so we it took like much data before we sort of tease out what's going on but if you looked at this you'd think there's a tremendous amount of variation in our product you'd also think that we had a really terrible week right here and if you look at this data you'd see a trough here but then you'd see it relatively flat amount here so the point is just is not to say that one is right and the other is wrong it's just to say that the way that you pull that data and the way that you present it and even if I had just changed like you know I changed the scale of the of the chart right I can make that metric look flat very easily but that all these things have a really big difference and the choice between percent and number ended up really changing the way that we approached the product because when we were just looking at this we would have had a little bit of a freak out session at the at when this data came in about why did we take that what went wrong that volume went down this much you know and you know same thing here and here we'd be congratulating ourselves but when we look at it this way we see actually we did have a problem and then we started to figure out how to fix it down here so I know it's not a silver bullet but it's just to say it's worth having that very painful two week long debate when necessary hopefully only 20 minutes for you right to figure out the right way to measure because once you start looking at that data in that consistent reliable way it's going to tell you a story and you need to know how to hear that story accurately and use that data in a really consistent and actionable way setting goals how are we on time actually where's my where's my Sam this is my time manager are we good on time okay so setting goals this is also a big part of every product manager's job I'm sure you've seen the smart as everyone seen smart goals before this is like very standard consulting speak but it's actually really really helpful so smart goals are specific measurable achievable realistic and time bound I like to add this meaningful one because sometimes that goal can be all of the other things but it just doesn't matter it's just not going to move your business and I and I'm sure many of you have spent lots of time on projects where you're halfway through and you're like why are you doing this like what is the purpose of this we just spent three weeks like doing this but I don't I'm sort of unclear on like what we expect to happen when we're done you're like oh we're going to move this number but like does it matter that you move that number so I have a great fun example for this so and this is real this is like telling you like the the horror stories of my of my career today so we were concerned about acquisition costs acquisition costs we're back to this is another retail example they're not all from the same company by the way so different retail companies but concerned about acquisition costs new customers costs a ton to acquire so how do you bring those costs down one of the best ways to decrease costs is to not pay for those customers that is like the cheapest way to get customers is when you don't have to pay for them so I'm sure anyone who's worked at a place where you're bringing in new customers talks about organic traffic all the time how are we going to increase organic because organic is free they're free I'm not free um but how are you going to do it so we actually made a cue on goal actually I don't remember what quarter it was to be honest but we made a goal to double our organic traffic that is a good goal that is solid that is cheapable it was in the quarter it was measurable it was so worked we had a great SEO person who did a really fabulous job he worked on it on a big content strategy we wrote articles we had beautiful pictures our design team worked overtime and like really made it great we had this wonderful content people love the content we have blog posts and interviews and videos and live Facebook events and all this stuff it was like a true content strategy it was awesome and we totally hit the goal we totally hit it it was awesome organic traffic and organic purchases doubled and it doesn't move the business even one tiny little bit nothing nothing basically nothing we got like a half a dozen you know I went with a dozen additional sales because organic traffic just doesn't convert it just doesn't convert and when we did all this we got lots of great people who love the content but they had no intention of buying and nothing that we did in this content was like by me now it was like you should really think about this this is really beautiful it's really gonna you know this is like a really fun thing to think about and you can take this quiz and there's this really beautiful video and just how you do it once you have it nobody bought it because that is not a purchase strategy that's a content strategy and if you're an advertising site it's awesome but when you're actually trying to sell people stuff unless you have a good conversion rate which we did not have then it's just not worth the effort to do that if your goal is safe now if your goal is just this that's not you know that's also like content seems to be great not I don't want to devalue the work they did which was awesome but if we put that same amount of time and effort into our product pages and really into like the the product purchasing button and explaining why that cost was worth what it costs and if we put that effort into even just literally the money that we spent paying the people for that time and developing all that content if we literally just put that into paid advertising we would have gotten more sales so being really clear about the outcome is awesome but knowing that that outcome is actually gonna drive your core metric is even more awesome so I encourage you to just sort of bring that with you next time you get a great content idea or something else where you're not clear like if you can't explain how the numbers work out then they probably don't right I mean you know maybe but it's worth spending a little time to say like let's say we blew it out of the water what happens because if we had just made this chart before we started we just wouldn't have spent that quarter doing it we just didn't take the time to do it okay stakeholder management I've talked a lot about it throughout here I personally find this to be the most challenging part about being a product manager is telling explaining people how decisions are going to be made and having them be part of that decision and then communicating back out what we're doing and when we're doing it and what happened when we did it and why we didn't do it or you know whatever stakeholder management so I just thought I would take a second to like reiterate what you probably hear every day and you're in your day jobs or when you come to things like this but stakeholder management 101 really being open but really being focused on the metrics so if you can always show your funnel and show your segmentation and show your code works and show how you're prioritizing your work based on the outcome that you think that those changes are going to yield all those conversations become a lot easier because you're saying that's a great idea show me the part where the numbers supposed to change help me see the vision that you see for why this is a great idea not just in a you know in a great idea it's kind of way but like where in the funnel where in the cohort are you going to see the metric move that really helps clarify all those conversations and then again like making those connections to the in the data really important showing the results in a consistent way I know I'm a broken record but especially with the product managers who I've reported to me I'm like that's a different chart why is it different you know show the same chart maybe boring but it's effective in people remember and learn over time you sort of training your stakeholders on how you think and then really giving frequent updates on how it's going and what you've learned and what you haven't learned people just like to know what's going on even if you can't point to an outcome yet you can at least say like just as a reminder this is why we're doing it but here's kind of how long it's taking okay I'm done and now I am happy to answer any questions that you have yeah um well there probably is like an ever ever a never ending level of detail that's possible I think that's really hard and usually it comes over time you realize just like the question that came up earlier about like what about revenue and you know what about like the cost of the products and all that you as you start to talk about it with stakeholders you'll see where they are sort of poking holes in your argument and you'll realize at that point you need more information if you also are noticing that over time you're not seeing movement in certain areas then probably you can you can push them together because the idea is to measure at the level that you're going to take action and so you know like engagement I showed like then there's engagement you know but that's actually probably for most products that engagement piece is sort of like a never ending series of steps and possible pathways that you do need to measure I think the idea is to try to condense it as much as you can and still get the insight that you need to know where you're going to focus to move that lever right and try to you know it's definitely more earthen science but like where does it show you like that part that's the part I need to move that all should help oh look at that magic okay how do you measure actions that happen off your platform for example how would REI measure if they're fulfilling their mission to help people opt outside so this is a great question there are a lot of off platform stuff and my question to you is like is that a product is that part of the product like as if you define your product and this is part of it is it part of your your funnel for success in your product and if it is that's great sometimes there are off funnel things that you're measuring and then know what and know whether it's a local metric this feels this is like me like I don't really know how REI does this but I think some sort of regular survey mechanism would give you that that outcome you'd be able to measure at least for some set of customers whether they're actually going outside but I would consider that to be a local metric because because it has it has its own thing it's not leading up to a bigger funnel what's your north star metric or the most important metric at hired and how often does it change oh well we just finished the planning the quarterly planning process which is a total joy in every possible way and this this metric this question comes up all the time so at hired the north star metric is hires like no doubt everything that we do is about yeah we want lots of people on coming on to hire people we want companies on their hiring people we want candidates on their finding jobs but if they cannot find each other then we are failing so they must get hired that's our that's our north star metric and everything ladders up to that in some way there's a question in the back I'm sorry can you say it one more time you mean like before you really know your funnel okay like a new business okay so the question is when you when your funnel is kind of unclear where do you start to measure is that accurate yeah how do you know what's important yeah that is such a great question and it's so context specific it's a little hard for me to answer I'm trying to think of a way to approach that that might be helpful for you I think the most important thing is to really come back to that outcome because if you can define your what's the outcome then you can ladder back to what are the steps that we would need to see happen to get that outcome and actually I'll give you a little story about hired that's kind of interesting so hired originally started as an auction so literally software developers engineers went on the site and they got auctioned off to the highest bidder they were like I do Ruby how much you going to pay for Ruby you know and people paid a lot so then they brought on more people and they were it was an auction site for a long time I mean in Silicon Valley long time means like a number of months you know but maybe a year I'm not sure it was before my time so you know at auction time it was probably like number of of auction you know number of candidates actually getting picked up at auction because I'm assuming that they had some kind of minimum offer you know like it's other bees like you have to pay a certain amount or you don't get them but so I'm really knowing what your outcome is what is that if you can't answer that then maybe it's a little too early to to to to really have metrics you know maybe what you're doing that is you really trying to figure out like what are I trying to do and that's a great question right that's a big product question in and of itself but it is from that that all the metrics will follow I would like to know your idea about comparison metrics generally in the slides there was a month over months but also we can do the year over a year yeah the amount of month is some months have holidays or a lot of dependencies like how to know it's like at the you pass a lot of things has changed so generally okay yes so the question is when you're when you're trying to figure out what sort of time period to use how do you normalize that data that is like such a complicated question I think that different companies do this different ways generally speaking you accept that there is a certain amount of variation and you just sort of notate it and so that it's clear when you're looking at the data or using the data that you understand what those are you don't necessarily have to normalize it out when in certain types of analysis and I'm so this is sort of starting to get into like statistical analysis you actually need to remove the data that's unusual to get statistical significance that's that's really measurable like when we did test a lot of AB testing we actually took parts of the population out because they were atypical for the note or we would we would normalize the amount of time such that the statistical significance could be exact but in most cases the most of the products that people work on you just accept that there's a little bit of variation and you need to know that like February is a short month and you know this month has more weekends than that month or whatever and just understand those it usually the effort of trying to normalize isn't worth the normalization value if that makes sense it really depends on on how statistically significant you need that data to be and for most of us it's probably not that green yeah all right I want to go back because I feel bad that I'm not counting the people who did the work of putting this into the the thing how do you attribute success to changes happening at the same time for example user fate user interface change and a campaign at the same time yeah this is such a great question so whenever there's those individual items ideally you're measuring both the local effects of the test that you're running or the campaign or whatever the interface change ideally you're measuring them separately so right now at these days it's like so easy if you have enough traffic to run an AB test when you do that you can you can look at data and see the specific outcome for each of those changes separately you may have a situation where the user interface change was positive you didn't AB test the the test population was positive you actually did a great thing and you could also have a situation where you had a great campaign running at the same time and then when you look at your cohort analysis for that month you actually might see a dip right that that can happen so being able to disambiguate the net effects from the individual campaigns it's important to really understand what you're looking at and in the case that something like that happens where you have some stuff going up and some stuff going down being able to like really sort of pull that apart that's kind of like the hard work of metrics is that like okay well that difference is you know 40% of that change is a trivial to campaign and maybe another 20% is attributable to the user interface change if you have enough granularity about the size of those populations you can pull those out it's a lot of work but it is it is possible how does the funnel work with returning customers ah yes I love returning customers so usually new and returning behave very differently so in any funnel that I've ever worked on we always did a segmentation between new and returning so that was just like table stakes like new and returning yeah I guess there's another part to which was say someone's bullet your product so my Netflix that signed up yeah do you care about them because the funnel was going to buy time up to buy time yeah yeah yeah oh yeah so um so in that specific example where you have a subscription product um you're looking at the data at a sort of cascade level so you're always looking new versus new like net new this cohort versus net new different cohort you know January of February versus March you're looking at the data in that cut and then you're also looking at exactly that same cut for returning and also by tenure so we would divide it by tenure too so like second month third month do you see what I'm saying so we would break it out to that level and we would just look at them all it just takes a long time but it's important to understand and this is where segmentation and cohorts work together so your core analysis is just like a fancy way of saying like how you look at your segments or how you look at your funnels over over time it's important to really understand where you see that that variation so for example and not just in Netflix with subscription where it's very clear because there's a start and then there's a subscription at the end of the month like success is like binary that's awesome that makes it super easy but when success isn't binary like in retail you're saying like this person bought they might buy again you know did they buy again this month well I don't know but they might buy again next month you know subscription you're either in or out or you make them back that's like another level but in a place like retail you're saying new is new but returning has five different definitions or 10 so what we would do is do some offline analysis like sort of deep deep dive analysis is what I would call it to understand those trends of returning customers how many months usually go by before the second purchase and then we would set our cohorts according to that normalized set of data so we we would normally say like if someone purchases at x time then they're they're most likely to purchase again within three months or never and then we would look at the data through that lens and look at the cohorts by that way but you'd sort of like use the data the guide you can't look at everything or you'll be you know you'll never make any decisions so you just have to figure out like okay what really matters in this in this repeat purchase flow and then look for those trigger points in your in your funnel okay what test are actions do you take when you move a metric like number of purchases upward but revenue remains stagnant or even decreases yeah so this actually this is this came at a question I think it was your question from before about like what are the other factors so there's our funnel and there's factors that we just didn't have in that funnel like what's the actual like number of items in a purchase and what's the the average cost of the items in the purchase and then the total basket size so in retail in particular it's it's usually that every business has their own set of metrics that make up these sort of other things that affect your revenue and the important thing is to know what those baselines are and be able to look at your funnels with those baselines in place so you just need to break out your funnel to the extent that you can see that level of detail and and that'll give you this action in this case you just want to figure out why like did they buy less items or did they just buy cheaper items and sometimes you can make changes to fix that you can promote different things or you can sort of change your mix sometimes that's possible not always yes I guess maybe it's specific to hired or something similar but after they are hired do you then ever track like the length of time they stay with the position or anything like yeah no I guess it would be kind of like return similar yeah we definitely track when people come back for another job yeah yeah I mean we don't always know if someone's like happy and state or whether they like moved to another state you know like ever like send out surveys like hey how do you like the new section you know actually I don't know that answer to that we do that's great thank you yeah I haven't been there for the very line and I'm glad someone else was able to answer so the answer is yes surveys okay at hired who leaves the entire consumers journey from awareness to retention product or product marketing oh we work so closely across all the teams it is like we're like this so actually we have a really strong senior leadership team and they actually do work really closely together like I'm working on a product right I'm not working on a initiative right now and my closest partner is the marketing guy who decides how much to spend on new customer acquisition so really we do try to do it together and basically we use the same funnel so we're all we're looking at the same metrics and so when he's talking about what he's doing it dovetails with anything I'm doing and we're just not in conflict because we know what the different things man he's got a budget he's gonna spend it but it's gonna affect the same funnel as my product changes so we do actually at hired work really closely together I know not every organization has that luxury but what we do I strongly suggest trying to get that alignment because it really helps okay in addition to measuring products and campaign success how do you measure your success as a PM and how do you measure success in your career oh boy okay so all the easy questions thank you so this is probably the most personal question so I don't I never in my entire life have ever asked myself how am I doing as a PM like am I a good PM you know I ask myself like are my stakeholders happy is my boss happy do I like this job do I want to do this anymore um but I don't actually like think about that I really think about it in terms of the stakeholder am I moving the metric am I getting the product where it needs to go are the things we're doing yielding results and whatever it takes to get there is like kind of what's on the table so I never really think about it that way in my life and this is like not to get super personal on you guys but in my life I think about is my job letting me live the life I want to live and I have a five year old who is so freaking cute who is in bed right now and I am not kissing him good night because I'm here with you guys who I love too um but that's really how I think about it so being a PM is like super stressful I work you know at one minute I've got like the new VP of marketing who's like and then the other minute I've got like the sales organization who's like we got to sell this thing how are we going to sell this thing and then um you know and but at the end it's like it's a great group of people I really enjoy working with them I'm able to communicate what we're doing and why we're doing it and I feel at the end of a week that I'm moving this thing forward and for me um moving this thing forward is like I'm helping people in my little way find great jobs and I feel really good about that helping people find jobs feels really really good and I got my job through hired and lots of other people have gotten their jobs through hired I feel really good about that I think there have been other jobs I've had where even when we had tremendous success I sometimes will come home at the end of the week or the quarter and be like yeah we hit our numbers but I didn't you know see my husband for like a week you know or um you know other considerations so I know this is maybe too personal for for this discussion but but when I think about success I really do think about it in like very human terms okay back to mattress how do you A/B test matching companies with job seekers so the sad thing is that we don't always have enough people to A/B test at a very granular level so um A/B testing requires volume and the smaller your conversion rate is the bigger your volume needs to be and there are days when I'm like oh gosh I wish I had the luxury that we had it and that flex record run a 12 cell test in a week you know but we just don't we just don't have that volume right we're matching people to jobs there aren't you know thousands happening at any given time so we reserve A/B testing for when we can test something that has like a pretty high conversion rate and a large volume um in other cases we just use different methodologies to try to get um actionable results so if it's low volume is it a fairly manual process to increase the accuracy? oh yeah accuracy is like very fuzzy right like if you're if I if you come from statistical A/B testing you're like whoa what what 12th up to number 12 um but yeah you just have to find other way that you're using a lot more of like directional data and data over time at at hired it's all about liquidity so we're we're literally trying to get enough companies looking for that particular type of candidate and those candidates enough of those candidates on the platform at the same time so our goal is like liquidity like you're looking they're looking can you find each other now can you find each other now at quality um and sometimes that you know it's a little more uh hand wave than I would like would that in mind uh religions like can you talk about quality approaches to moving the need of obviously when you when you can't A/B test yeah obviously you've got quantitative and that's what you really talks about a lot that can you tie together yeah sure so um we do a lot of user testing actually and we do a lot of talking to our clients and trying to understand things like candidate volume and candidate quality and the quality of the matching obviously for me is like a really big deal um so there are some places where I can use data quality of matching uh like I like I'm I'm partial maybe you could tell that I like the percent because knowing like what percent of people who found a candidate found them at the top tells me how hard the clients are working and I can see that in the data so if if 80% of our clients are finding a candidate that they want to talk to at the top of that list then I know I've made their job easier I've made it easier for them to find that candidate and that's real data even though it's not like statistically significant it's real data and it really tells me that 80% of the time they found the candidate in the top x number of results and I can see if that number goes up or down when we make changes to the logic um so it's really important to find that metric that is moving in a way that you can see and you get a you get a sense over time of how how real that is um percentages are really good for that because you can compare them over time you know last month we saw 80% and this month we saw 72% and then we can sort of unpack what what happened um in other places like um we serve a lot of different types of worker um in other in other places we're literally looking for trend lines so we might graph out the data and even though the actual absolute numbers are kind of you know not that high we can see variations in the shape in the shape of the curve and that helps us see sort of what's happening so it's it's numbers and it's metrics but it's just not the same kind of statistically significant um response that you get like when they these have it's you might be wrong basically you have a higher risk of being wrong um but if it's what you've got then it's kind of the best you you can do with that data and it's usually pretty good I mean if you think about how most decisions are made if your company is different you know god bless you and and you're very lucky but I think a lot of decisions get made by like that's a quite idea you know so any data is better than that I don't know if that helps yeah I was talking more about you know um using these research more as the drawing yeah oh yeah sure yeah I just hadn't seen much of a length of design and yeah it's a great question um so we do we actually use a lot we do a lot of interviewing um we've also like just this week some examples card sorts are great for information um to know whether or not what you're doing makes sense if there's a better way to do it um we also do a lot of surveys surveying you got one it sounds like I wasn't part of that one but we do a lot of surveys to say like you're hiring for this kind of candidate um you know what are you finding what are you not finding and that helps to drive product decisions as well and those are sort of non metric related data I also just spent a ton of time on the phone talking to customers on both sides of the marketplace you know like hey your DevOps like what do you do um and or like you're hiring for DevOps so like what are you looking for like can you walk me through like when you're looking at this person's resume like what are you looking for um and then we try to figure out like okay they're looking for this information are we asking them that question do we need to ask them that question that that informs a lot of our product decisions it's like what are the specifics around the hiring process and how can we flush out the date the information that either side needs to know to make a decision about whether to extend a job offer or accept a job offer and try to get that information front and center