How to 2x Your A/B Testing Output Without More Budget

Welcome to episode 117 of Marketing Operators. We've got a very special episode for you with Michael Ting. Michael is the GM of DTC at Jackson, which is a men's jewelry brand, pushing nine figures in revenue. Today we get into a lot. We get into why he believes there's no such thing as optimized ad spend, how he thinks about balancing today's contribution margin with long term growth, how he uses cost per email as a leading indicator of today and future revenue. So much more, what he's focused on going into 2026. This is a very tactical, very actionable episode. Michael's super in the weeds, very data driven and is just a really sharp DTC operator. So really hope you enjoy the episode today. Thank you to the sponsors, motion, rich panel, aftersell and house. Let's get into it. All right, we're back with another episode of the Marketing Operators podcast. We have Michael Ting on the show today. Michael, welcome to the marketing operators. Yeah, super excited. I'll mention it later, but really had to learn everything, not exaggerating everything from this show. So it's crazy to be on it, finally. Nice, excited to dig in and kind of figure out what was the most and the least valuable. Where are you calling in from today? I'm in Los Angeles, Westwood right next to UCLA. I commute down to Newport. So if you're from Los Angeles, you know what that's like. Yeah, I don't envy the hexcods like half remote, half in person. And I do not envy some of the trips that some of those folks are making every single day from like Orange County and like Malibu to the office in downtown LA and back every day. It's crazy. But like that's when I listened to the operator podcast, right? I got 90 minutes. I gave you the chill, just put it on to like speed. So it's baked into the, it's baked into the morning in the evening commute. All right, that's great. So Michael, I mean, you probably know this. We bit like to start every episode by talking about our new favorite protein infused products. And David protein is one that comes up a lot. They just launched their David protein ice cream sold out in 28 minutes. I was told that's what I saw from their founder, Peter. And I wanted to know Connor, did you eat your hands on any? No, dude. And it's so funny because I'm like not much of a consumer. I'm like not buying the trendy stuff. I'm not, you know, I'm never buying anything. And this is the first time I was like, damn it. I really want this ice cream. And I was like, I looked on Amazon. I looked on the site. I'm like, are all the flavors sold out? I'm like, this doesn't seem right. I was like, should I text Connor Connor? You met the like VP of performance marketing at David. I'm like, should I text Connor to ask his friend if I could get the ice cream? I was like, really going through it. But no, I haven't tracked it down. Hopefully soon. I don't know. I was like, maybe it'll be in grocery stores soon. I don't know. But I'm scheming. Yeah, I texted. I slapped Keegan. Or hold on Michael, did you get your hands on me? I haven't. But have you guys ever tried Hule? I think most underrated D to C brand out there, H-U-E-L. They literally exited the week after Groons. And you know, everyone's social media was all Groons. I think all the hours Twitter, everything. They exited for 1.2, 1.6 billion, pretty much the same size the week after. But no one in this space has heard of them. They're UK based. They do like plant based proteins. But they also make mac and cheese with like a ton of protein. Literally used to live off of it when I was like really grinding at Jackson. I just really have time to take care of myself. Like breakfast, lunch, and dinner. I would just like just mac and cheese. And honestly, it was a best shape. I was ever in my life. Did you ever go through a soy lint phase? Never a soy lint phase. Honestly, I'm not sure how I feel about the plant based approach. But I love the mac and cheese approach personally. Yeah, yeah, yeah. No, it's interesting. We're pivoting from David a little bit, but Hule is cool because they like they were like a fast follow to soy lint like 2015 or maybe it was like 2017 or something like way back when soil was like a hot D to C brand. I basically got in the D to C because of soil and I thought it was like the coolest brand. It was like this hat weird like dystopian branding. I'd met the founder at a conference back in 2016. It was really cool. And then Hule fast followed and then soil and I was like didn't pan out. They raised too much money. They got sold like for parts basically and then Hule has a billion dollar exit. So it just shows they pivoted away from like the dystopian all you need in a drink form to like normal things like mac and cheese if people actually want to consume. That's crazy. I never knew the history. I was just a huge fan of the brand. Probably top 1% customer and was always just shopped at like no one's talking about it because I think they executed really well with the interesting product. They're going it's interesting Hule's going more of the like a lot of these RTD protein drinks are like I like raw that's that's my go to and there a lot of them are in that like 170 calorie 20 grams of protein range. Hule's going they're like now this is like a full on meal replacement like 400 calories like you're gonna you're gonna be full after this. David to bring it back to David. I have never seen I mean this is obviously like this is their trend they're all about like the maximal maximum calorie to protein ratio you can get. There's an LB in full pint of ice cream is 210 calories and 30 grams of protein. It's it's it's it's insanity. It's insanity. You know it's been I've been following along on on Twitter and a lot of people are coming out of the woodwork on the ingredients and not that I mean David's never shy that David's not trying to be like a whole food natural ingredient based brand but like people are really coming out of the woodworks on this one because their flavor system just says natural and artificial flavor that's all it says for the flavor system and it was funny to see that. Well you know it's funny so actually going back to the the soil and point is like their billboards back in 2017 were like pro GMO that's what I like that's why I like them so much they like they completely own the artificial nature of it and at the time they were like natural food is obviously good and like there's there's no like qualms with that but at the same time when you think about like feeding tons and tons of people like I don't think over the over you know the next 100 years we're gonna you know exclusively produce natural food I think we're gonna have better and better natural ingredients that's how we end up feeding billions and billions of people over time. Yeah I just love how David's leaning into it and I also love the like I don't I think most people know this but maybe not that like David like Peter's previous company was the opposite end of the spectrum as what where David is you know Rx bar is all about whole food ingredients three egg whites you know 10 cashews whatever whatever's on the and then and then he has this which is all about maximizing calorie to protein ratio in order to do that he's using artificial ingredients but it's just I just think it's so badass that he's been able to like take to like you know same market to completely different ends of the spectrum on how he's positioning these products and like the ice cream is just such an exclamation point on that compared to what's out there right now so I loved it I think it's amazing and I'm gonna try to get my hands on some I slapped Keegan or what right when I saw this sold out and and asked these like hey I'll send you someone give me some listed give me a like no no I feel bad because I'm like I'm not asking you to send me some I'll totally buy it I literally just want to know when are you restocking so I can be ready to buy it but I'll get you on the list for the restock Michael I want to I want to dig into your background a little bit you have a pretty impressive involvement with Jackson's growth trajectory but I want to just before we get too deep into that can you just give me your your background like who are you in the context of digital marketing how'd you get into this world and like kind of what ultimately got you into your role which is GM of DTC at Jackson yeah it's been honestly a pretty loud unpredictable journey like I think most of us but I would say what define my career was essentially at any given point I'm trying to figure out who was making the decision and become that person and just wins and repeats so actually graduated chemical engineering so my first job was R&D I was trying to make memory forms and mattresses more comfortable completely different from what I do today I felt like the project manager was always telling me what to do so I was like okay how do I become a project manager when through the reiner kind of winning the product development and a lot of lean manufacturing stuff as a project manager program manager then I was like well now just the category of product manager keeps telling me what to do so how about become product manager so then jump to rugable back in 2021 and I was kind of category management there felt like same thing where analytics was the one actually finding the opportunity and I just felt like I was executing it so then I was like okay now I gotta go to analytics so for a little while as manager of analytics I then felt like well I'm finding opportunity but strategy gets defined where I'm actually looking for opportunities so jump to strategy at rugable and then finally I was like well I'm doing strategy at rug I'm just making power points for the you know exactly so how do I figure out how to become an operator so kind of made move to Jackson really just saw like we had a crazy brand present shout out Bayer who just absolutely crushes it with partnerships and building like these authentic relationships but then I saw the sales numbers and I was like oh there's a massive gap between the name recognition among like I'm probably target demographics of the name recognition between like my cohort and the actual sales dollars so I was a little little cocky and thought I could jump over and try to make the company what I thought it could be so it's really just been a story of trying to take it from low or I'm not low but like let's say eight figures and just a journey towards hitting like nine figures which I would say. where mostly successful one. I think the crazy part though is that like my background, analytics, project management, little bit of data science, little bit of strategy, never a hands-on performance marketing. About two and a half years ago right now, just organizing a show gap resulted in me and my buddy, Ricky, who leads our marketing effort, really having to make all the decisions, and I had no marketing experience. I had no idea. I didn't even know what CAC was. I didn't know what an MER was, and I was given a made-8 figure budget to try to grow the company. So like I mentioned earlier, I've just really had to rely on the operators' podcasts to try to crash course myself and then being able to execute things like advertising, budgeting was a main focus of mine, merchandising, just pricing strategy, or those aspects. Yeah, and honestly, it felt like learned a lot, learned from the mistakes, learned from the good parts, but it's been a while, right? I loved you to see it. Motion just dropped there, 2026 Creative Benchmarks Report, and it's been getting shared everywhere. Slack channels, LinkedIn, Twitter, sharing it in our private group chats. And it's great because everybody's been asking the same four questions forever. What is normal? How many ads should we actually be shipping? What is a healthy hit rate, and which formats really win? The report analyzes over 575,000 creatives from 6,000 advertisers in over a billion dollars in ad spend to answer these exact questions. And the report has some really interesting findings, like the fact that only four to eight percent of ads actually become liners and over half of ads actually lose. And for motion customers, this report is especially helpful. You can upload it into your motion dashboard with your run-with AI chat and compare it directly against your vertical benchmarks. Hit the link in the show notes. I promise you won't regret it. And as always, go to motionapp.com until the marketing operator sent you. That's interesting. I think that's a cool way to-- that's a very-- I don't know if I've heard someone position it that way before, but I think it's very-- it makes a lot of sense. You've let-- you want to be in the seat as the decision maker, and you've let that guide your career. And it sounds like that's led you to this point in every step along the way was, how do I become the decision maker? And that's what-- is that true? Would you say that's been like the lens? You've been viewing your career and like each step through? Yeah, definitely. And I feel like you guys can empathize. Where right now, I feel like I can make the decisions. I can make the big calls, but spreads over such a large surface area, just different areas. That I don't actually feel like I'm making the big calls anymore, where it's really just trying to build up the team to be able to make the decisions that I would inside their shoes. So now that I finally feel like I've made it, and I get to look at myself as an operator, I'm like, wow, the good old days, right? Yeah, well, and that's what I think that's what a good leader does is they empower their team to make decisions as well. So not everything ends up with them being the bottleneck. And I also love your background. I think that's the cool part about D to C performance marketing. If you put 10 performance marketers in a room, heads of growth, CMOs, what have you, they'll all come from different backgrounds. There's not like a clear path to getting into this e-con performance marketing world, which I think's really fun, because it just means there's such a diverse group of people that are in these performance marketing roles now, which is exciting. And it just creates a lot of different opinions and types of people. And I think it's good for our industry as a whole that there's so much diversity and background of the people that get into these roles. Of course, head to head if I can hire one person, I'm going to hire one with experience, but inside performance marketing. But it's just been interesting where some things that are abundantly obvious to me, like testing at the dollar, is how to run a robust testing program, how to do things with a lot of statistical rigor, how to get creative with numbers, which you set goals. They come naturally to me, but then some things that are just basic to others, like having an eye for creative or figuring out what a great partnership is, just completely foreign to me. So I think that a lot of the value I had where Jackson was a lot of people who are amazing, have brand amazing, have creative, just really able to go out there and go out there with our entrepreneurial spirit, essentially, just adding some level of quantitative rigor to it, and then it's just been kind of the best of both words, since I joined. What, I want to get into Jackson and what you're focused on there in a second, but before we do that, you started in product. You started, you started in like product as an engineer. How do you feel like that's helped you as you've moved into marketing roles? Like your origins, your base and product development and just understanding what goes into that? Like how is that helped you or not in your current role? Yeah, it's been interesting. I think that one big area of product teaches you how to do is, a lot of times, you're watching something just without a lot of data and trying to make a decision when you just don't have, you don't have clear numbers that we'll get a lot of the times. You just have to make a call and you have to kind of both think through, you really have to think through the problem or just trying to calculate an answer. I think that's an interesting part about product. I think the other part was just an appreciation for the challenges of supply chain, for project management, how everything is always there that you expect. For a while, Jackson was owning the demand planning aspect and talk about an absolute mess when I was running it. But you really just learn how to appreciate the other side and the challenges they're going through, which I feel like helps you integrate the team slightly better. I sometimes wish I had that background because I don't understand it. And I find myself at times in the past have found myself thinking, what do you mean we don't have, we don't have more knives, just order more. It's not that easy, of course. So yeah, that's cool that you have that perspective and you're able to bridge the gap between different parts of the org because of that. So I have written down here on our notes that UV Core thesis that there is no such thing as optimized ad spend. It's all about risk tolerance and time horizon. Optimizing for optimizing for results right now is a completely different exercise than optimizing for a 10-year timeline. So I definitely resonate with this note. We're a highly considered product at Hexclad and the ad spend that we're deploying today often doesn't really fully materialize until months or years down the road. So we're always trying to strike that balance of capturing demand and generating demand at the same time. So a few questions on this. Can you walk me through how you actually think about ad spend as a risk management exercise? What does that framework look like in your day to day? Yeah, so I think a lot of what's helped me find Success at Jackson is just these 90-minute commutes to OSE where I'm just stuck in my head and just running these like dollar experiments pretty much combined with listening to the experience of other people. So essentially the dollar experiment that had helped me with this one was the guy named Brad Jacobs and Fasting and Guy founded $8 billion companies, not $8 billion, but $8 separate billion companies. Crazy entrepreneur and he does this exercise where he tries to like accordion his mind back and forth to the really small picture and the really big picture and back to the really small picture. So then when we're talking about trying to optimize your advertising spend, what is the optimal advertising spend? Well, I think if you accordion to the really small picture and if you're trying to optimize for contribution margin the next hour of what we're probably all of us do, especially high of the one consideration process, we'd probably turn off all of our ad spend, hit SMS and drop our prices 50% and do a massive discount. That's probably what would optimize contribution margin for the next hour. If you go accordion back to the really, really big macro, big picture and you're like, okay, how would I optimize contribution margin for the next 50 years? Well, assuming that we compound growth for the next, let's say 50 years, this year is going to contribute a fraction of a fraction of a percent, where probably what I would do is I would focus on learning, I would focus on Jenny as many insides figure out whether we had the right product and aggressively testing and I wouldn't care at all about contribution margin for this year, I would just be learning exercise where I would spend as much as my cash flow could allow. So then if you go back and forth, that just means that essentially on one hand, one extreme you essentially have optimal ad spend is zero and the other extreme is optimal ad spend is as much as my cash flow allows, which just means the problem that we always discuss about trying to how much incrementality, whatever we try to achieve, it's less so like an optimization problem and just a problem definition where it sounds kind of hoody flutey, but it really turns into, it brings that aspect, an internal aspect of personal risk management, where there are probably two ways I can improve contribution margin and the expected value. And the number one way is that I just, get better at growth hacking, I just improve as an operator. Number two, a lot of times, it's just taking on more risk. And I think that like an intuitive example that we can all understand is that like, you know, when we're building a personal finance portfolio, you're buying stocks and bonds, so you keep, you know, and keep some money in the savings account. You're not fully optimized to maximize your expected value. You're not trying to fully optimize to maximize your return. There's an aspect of risk management where I don't think the money my savings account is compounding at the rate. I don't think it's getting the maximum value. But what I think is that given my risk tolerance, that's how much money I want to have on hand just in case everything hits the wall. There are like lotteries out there where for example, if I want to maximize my expected value, I put all my net worth into lottery when it gets to a certain point and my expected value is positive, but we don't do that. It's just risk management. So they just brains in the perspective of like essentially, I feel like we don't ask ourselves as operators often enough like, what are we trying to achieve? Are we really honestly trying to maximize our contribution margin for let's say a three year horizon, even if that would require us to essentially sack in a quarter, or is there a better balance where we are actually optimizing for let's say 80% for next quarter and maybe 20% for next year. Hopefully that resonates slightly, but I definitely feel like being high will be brand you guys can definitely understand that like feeling where I could just cut spend and make a little extra contribution margin today by no long term that would really bite me. When you think about balancing short term contribution margin cash flow and then where you want to invest that long term, it feels to me like if I think about how that gets captured at Ridge, it's like in our forecast, right? Our growth team is not allocating our ad dollars versus some form of catbacks or hiring or retail expansion, right? The marketing team is not making that decision that's coming from a much higher level. Where that should get captured is like next month, we want to do a million dollars at a 4xMER. That's what the marketing team knows and is optimizing towards. Then it is like the group of executives, whoever that may be, to then allocate, hey, based on these results, based on this forecast, we will generate this cash and we're going to invest in these ways. Is that how you would describe it? Then how do you decide, like what are some of the thought processes that you go through to then decide where are you allocating that capital so that you're maximizing the value over time? Yeah, so I think that it kind gets into the nitty-gritty methodology. I think on one thing where probably contrast is I hate budgets, I hate forecasts. I think that some, there are times where forecasts and budgets have been useful, but more often than not, what they caused me to do is they caused me to anchor to a result. One of the stories was back when last year Q2 tariffs hit. Obviously, there was a period of time where they were trying to lure up everything that tariffs we all went through it. A lot of our colleagues started increasing and then we knew we were probably going to miss a gross margin on a budget on a forecast, but what we saw was that everyone was pulling back spend so far that we were seeing record loadcats. So essentially, the equation was that, like, oh, well, we have a really low-cac that's way below budget or really high or lower gross margin, which is missing budget. I feel like that if we really fixated on a forecast, we wouldn't have done what we ultimately did, which is spend really aggressively into the low-cac and allow us to kind of take a lot of market share, which it was probably the best period we've ever had. I think that it just causes us to anchor in that every day, what we do at Jackson is pretty much telling the team, right now, given our risk tolerance, you need to be optimizing for contribution margin, or right now, you need to be optimizing for some of the leading anger dictators. And to talk about the leading indicators, I think we have a really interesting one, which is like a cost-per-any-house sign-up. And go do the whole dial process of that, but that's really been the major unlock is just kind of disconnecting, spend from your prior assumption from two months ago, from your Re4 task, or just from anchoring to some number that you're trying to hit at all costs, whether or not there's a better way around that number or a way. We talk about incrementality a lot, but how do you actually operationalize it to make your business better? That is one thing that I've been really leaning in with my team recently, and how since played a tremendous role, we use it for all of our experiments, all of our geo-lift testing, but we now use it for our MMM as well. I've been a design partner, I've been one of the early design partners on how to see MMM, see stands for causal, so it's one of the only MMMs, if not the only MMM that I've seen that's actually using your causal experiments to build the model, and so that allows me to just trust the data so much more. So it's not a black box, but actually informs our roadmap, and has been so crucial for allowing us to operationalize around incrementality. The house team is world class, I can't speak highly enough about them. They've also built a really amazing community with some of the best D to C growth operators out there. If a few exciting events coming up soon that they call the house growth lab, one is an LA and May 19th, and the other is in New York on May 21st, so I highly recommend checking it out if you're in the area. If you want to check it out, learn a little bit more about CMMM, go to house.io/operators to start making better data-driven decisions today. I think Conner, I agree with your point though. I think the forecast can, if it's set up the right way, be a guiding force on which moments in time you're leaning more or less into the ends of the spectrum that Michael just talked about. This is something that we had to figure out. A hex clad is, it was hard for us. It's like our mergals, let's just say it's a 5x for the year, but we might run the business out of 3x in some of the slower months in the interest of hitting our revenue target, and then coming in at a 5 or a 5.5x, and that's that balance of, in the dog days of summer, we're less efficient, we're obviously not optimizing for contribution margin as much, because our MIR is lower, but we have to keep spending, we have to keep driving traffic, we have to keep generating demand. If we're going to hit those bigger revenue targets, those higher efficiency, those larger contribution margin moments, but that is guided by our forecast. Whereas before we had that learning on the seasonality of our business, it felt like every month we were having these conversations on whether or not we should pull back or spend more or what have you. Now we have that baked in, but I think I agree with your note on letting the forecast drive that, because if you can set expectations on a more month-to-month level, then your marketing team just has a really unlocked to make good decisions. You're not having these conversations with finance every week on whether or not we should pull back or spend more. I think that's a very tough thing to figure out for your business. Michael, I want to ask you a question about some tactics on this. Do you have any examples? In reality, all brands are doing a little bit of both at any given time. We're running a summer sale right now. That is helping us optimize for contribution margin right now, but we're also not pulling, it's like, we're not spinning all the way to that end of the spectrum. If we were, we would put all of our budget into meta in search right now. We're still investing into YouTube and linear TV in podcasts. We're running an offer that's optimizing for contribution margin today, but we're also still investing in these upper funnel channels. Can I say one thing quickly? Just because I want to hit this point, because I like the point about forecast, I made that one. I'm super done to reforcaise all the time, which is maybe closer to the spectrum, Michael's on where it's like, hey, let's not be too anchored to any one specific number. I think that's extremely problematic too. But the forecast, there's all sorts of light constraints that a brand has to give themselves and the forecast is one, because Connor, you're saying it now, you guys are running a summer sale. If you were optimizing for contribution margin, frankly, you would probably turn off ad spend or a lot of it. And Michael made that point earlier. If you were purely in the month of June, we need to maximize contribution margin. We're just going to reduce a lot of ad spend, because I'm sure there are people that you're prospecting today who are not going to convert until October or whatever, right? You know, there's a ton of latent. It might be the consideration might be, period might be as short as it's going to be during the summer sale, but there's still a consideration period. And there's still people that are going to convert over like a very long tail. So one of the reasons you have to spend today is that your forecast says that you need to do like $400 million in December. That's going to do. So all of those things are just, I would even say, yeah, so I'll leave it at that, but I feel like that kind of, it's not even a marketing thing at that point. That's why I say it's like an org wide perspective of like, what does the business want to do from a contribution margin perspective, from a cash flow perspective, and like a future growth perspective? And all of those are, I think the forecast is a great way for it to get captured and then can provide constraints on like a day to day week to week basis so that nobody's running around being like, I'm going to run at a 10XMER right now, blast the SMS list like Michael mentioned and just generate a bunch of cash right now. And then, and then I'm going to worry about the September forecast when I get there because like ultimately that month will be smaller if you do that. I was just going to say I actually really like that because that is something we sometimes struggle to do is the tons forecast have been useful is when we say that, hey, we're going to do something that's not going to feel great in the moment, whether it's like service spend in October ahead of BFCM. That's when it's really been a cool, that's when it's been a useful tool. On one hand though, I feel like we've been talking about this but haven't shared our methodology which I feel like it's kind of missing piece of the puzzle because we're not just really running around choosing random numbers. I think the missing piece of the puzzle is we have a main KPI which is a cost per email sign up and just describing like why we see it is super valuable, shockingly valuable and I don't think talk about it enough. Is essentially a spectrum of metrics. You got impressions, clicks, add to cards, check out order, LTV and I would argue that the nuance of the spectrum is that one side it's really easy to attribute value. So if you tell me how. You got an order. I could tell you exactly how much revenue you got But it would be really hard for me. It would be really hard for me to tell you how much it costs to get that order accurately especially for a brand like ours On the other side if you tell me you got an impression I could tell you easily how much it cost it because it's pretty much immediate But the challenge is that it's really hard to attribute like what value did you get out that one impression wanting to Mentor question probably really hard and essentially goes so on so forth. So you got impressions then like I said You got clicked at the card checkout order out TV the one interesting metric though is if you run on a email sign up it kind of breaks the trend where It's easier to attribute revenue to an email sign up than a checkout or a ad to cart order even though it's earlier in the phone And the reason is I can just take my shop a list of emails I can take my email list and I can tell you exactly how much value you got out of every single email and the other advantage is that It's not immediate that I pay one dollar get an impression and get an email But it's much faster than the purchase cycle for a it's much faster than a purchase cycle for a let's say like Hi, a little V brand where it might pull forward the sales or the value like the point where you the marketing spend have the impact or by let's say like months So then if you essentially you view your business as these horrible businesses You view it as like a freemium model where my goal is to get a email subscriber and Based on a lot of analysis I can roughly estimate the lifetime value of the email subscriber Unlike CPG the timing of that Value is very different where if I get an email subscriber in October I'm probably going to realize most of the value late November Whereas if I'm getting a night email subscriber on bfcm I'm probably realizing most of the value on that day You can do those kind of calculations and then almost run these durable brands is that OTV over cack Where you just focus on the email subscribe as the name moment instead of an order It sounds there are a lot of like pitfalls that you can find and we had to trial and error essentially to figure out how to do this But the really nice part about it is that let's you find these pockets of moments. We always talk about it like I feel like every brand feels like They under spend and the lead up to bfcm And I feel like it's because they don't have this leading indicator that they can attribute Future sales to at an earlier moment But it's a lot of to find these random pockets where nest the buying intent might not be there But the marketing spend and the impressions is definitely there and you're getting high value leads is essentially the goal So it's just so essentially how our business runs is in a moment of like promotion We're really hyper focused on contribution margin, but let's say I'm March nothing's really going on for the business It's a slow season then we can hyper focus on getting efficient email leads that we can roughly estimate the value for an efficient cost Does that make sense? I love that so that's so that's an example of your optimizing more on the The the broader picture end of the spectrum versus the today's value So you're I just want to repeat this back to you and correct me if I'm if I'm Misswording how you guys have the setups. You're basically saying You are calculating like an average on average revenue per email sign up correct and not every not every email turns into that Some or more some or less, but on average you know that every email you capture is gonna is gonna lead to $75 a value or $150 a value whatever it is So you're basically using that you're basically you have a cost per email target that is related to That average revenue per email and that's what you're using to like Optimize your ad spend and and as long as you're like in line with what you know will As long as the cost per email is in line with what the average email will back out to revenue wise You you're kind of like keep continuing spending or pulling back spend. Is that how you operationalize around that? Yeah, and it's really just to aspect the first that helped me like kind of find this metric The first one is looking over our shoulders at our friends in CPG and just how nice their LTV over cac model Fellet watching it where they can spend unprofitably short term But they can have a high confidence in the long term benefit and for a high Decisions like the brand like most of ours That just didn't feel possible until we had this metric and we kind of shifted the focus to Finding when finding a moment other than the moment of order that we could attribute spend to But second one was actually listening to Ridge and you guys so when I took over our marketing budgeting We I would listen to the podcast and our go at the time Which is to win on contribution margin every day probably led to significant understand and I was listening to Connor Just talk about like how law of the MER they would run and delete up to a sale and How they would essentially break into contribution margin during their quarterly seasonal promotions and I was like Maybe there's something there, but that feels kind of ridiculous Why not win contribution margin every day when we started mapping out the lifetime value though of these like email signups We did see very very clearly that if you're able to get a lot of email signups prior to a promotion You're able to monetize them during the promotion so even though it might really really dent your contribution margin and let's say up October or for us another one is a cloud or say so in my day in April You can act with some confidence that you will eventually make back the value I think the challenge for us versus like a CPG brand is The timing of the value is really is just completely unpredictable for us So I can fairly confidently tell you how much a email subscriber will get us unless we kind of do something We are like dilated but I can tell you how much an email subscriber will get us over let's say a one-year period I cannot tell you how much it would be over 30 60 or 90 period because it's pretty unpredictable when exactly they purchase but the stability of the The stability of the metric over a longer term just enables you to make these decisions and have confidence that like Well, I'm not getting as much value within 30 days as I expected But I also understand that that's probably just noise and that overall big picture The stability I'll make it probably back in 60 days and 70 days or so on so forth. I have two questions one is if you are Like within meta are you optimizing for email signups or you optimizing for purchases and then monitoring cost per email and then two I'm curious how that's changed or if it's changed if you guys have diversified channels because I would assume that like an email captured from a snapchat is just going to be worth less than an email captured from like a super high intent meta campaign something like that so as As like the the email the quality of the email changes by channel or by person or whatever else How do you guess factor those things in yeah? I the good question. I feel like that's kind of thrown people off where Definitely would never optimize for anything but conversion unless you're doing some like reach campaigns Which is a completely separate topic so this is more just like carrying the coal mine We're not trying to get the canary to squawk. We're using it as a leading indicator of how our spend is doing Even though we're not necessarily optimizing for it. We do sometimes optimize created for it though, which is interesting The other part about channels also something I've talked about a lot and then tried to really dig into data wise The nice part is that email subscribe Is kind of an equalizer of intent So if I get a lot of really low quality traffic. They're probably not subscribing to the email as much It kind of filters through the really bad traffic Um, we can roughly estimate let's say um we get the number of emails and um in or theme from us or in channel compared to the revenue They generate and then do a gut check to see that tribute revenue over the number of emails kind of makes sense and checks out And we do see some fluctuation But by and large I would say that it does help equalize essentially the buying intent Um whether someone signs up because they saw Facebook ad or whether someone signs up because they saw a tick-tock ad kind of evens out So you're just blending it you're really looking at a blended cost per email sign up You're not getting too granular with like Here's our here's all of our email signups with a meta UTM here's Snapchat tick-tock and like you you have different models You're basically kind of going one level up and saying all right blended cost per email sign up blended average lifetime value of an email And that's that's kind of like what you're using for the model Yeah, and I think that it's not that like um we wouldn't be better off by getting very granular but Um, we're both a small team and I think that like when we get really into the day trading aspect of trying to like make every channel perfect Sometimes we sometimes we as a company definitely lose side of the bigger picture and really focus that focusing on the blended metric Really just keeps us on as keeps us simple where the additional granularity in theory would help but I feel like impractice ends up being kind of a distraction from the main goal Rich panel CEO just predicted his own SaaS products will lose 90% of its revenue and he's the one building what will kill it They are betting that AI agents will replace traditional support tools entirely no dashboards no seats no workflows So what did they ship an autonomous AI agent that handles customer support from end to end reading order data issuing refunds updating shipments without a human ever touching it It means you can run a faster leaner support team at a fraction of the cost while resolving tickets instantly 24/7 Rich panel is cannibalizing their own business to build the future of cx and support for brands Because if they don't someone else will go check out what they're building at rich panel calm and learn how you can automate your entire support operation today. Can you speak to how this is integrated into a weekly workflow? So on a weekly basis, are you meeting with your paid team? Do you guys have a cost per email sign up goal? Then on a weekly basis, are you monitoring where you're at relative to that goal? And then let's say your goal is $2 cost per email sign up. Are you then saying, "Hey, we're at a buck 75. Let's spend more because we're below target." We have some volume upside here. But can you just walk us through how you're actually in a day-to-day week-to-week basis? You're actually making decisions and interacting with the team. Just overall operationalizing around this KPI. Yeah, about to go totally off the rails and then bring it back. But I just love telling the story. I think it's applicable. So in Japan, they have these chicken gender. Oh, of course. Yeah, it's really out there. But I love telling this on any season tell. They have these chicken genders. And what's fascinating about them is that in order to form chickens, farm eggs, you need to be able to tell the gender of the little chicken hatchling. And they can do it with about 95 to 99% accuracy within one to three seconds. The crazy part about it though is if you ask them how they're able to tell. There's no physical sign. They just say like, "I know." And even crazier is even with all this crazy AI, all the technology, machine learning still can't do it as well as these people. And the reason they're able to do it so well is because they see probably millions of chickens a year. And just like Trow and Error, they pick up on the trends. And it's essentially building a feedback loop for themselves that they just hyper-optimized for. To a point that not even dispense the AI to it would do, I feel a lot about, that's kind of how I feel about performance, where not even a weekly basis, but I think my team's going to kill me, but I used to do it myself every single day. I would go through the numbers and write a couple bullet points about the trends I was seeing, what a contribution margin was, what cost per lead was, cost per add to cart, trying to put together, and then trying to form it into a co-consub narrative, where if let's say we see great cost per add to cart, great cost per lead contribution margin is lagging, but where ahead of a major season moment, the narrative would essentially be that like, hey, we're building funnels of this large moment, we're being efficient with our spend, even though we're not realizing the value. Whereas let's say it's our shipping cutoff and we see a huge sliding conversion rate cost per lead is super competitive. It's more like, okay, let's lay low with spend, we're realizing value. So I think that's kind of the nuance where I talk about it a lot with Austin from North and but being able to look at the number every day but not react to it, I feel like it's one of the hardest parts of the job because it's terrifying sometimes that like, when you look at the big picture for the year, it's very clear to say like, oh, I kind of over spend here, I understand here, but in the moment it is kind of nerve-wracking and it's really hard to see when you see your cost per lead jump down, it's like, do I spend into it? Well, maybe the second day you see it while you do spend to it. So I think that's kind of not even a weekly basis by think of us personally we like doing on a daily basis and then just to kind of step back from the day to day's noise, probably once a week, I have to put together a deck kind of on the performance and that kind of consolidate my thinking, let's me take a step back. So it's daily monitoring and then would you say, weekly making of moves, hey, it's like, hey, cost per lead's down a bunch today, but it's only a single day. We're not going to go in action this because it might shoot back up tomorrow and like there's so many daily fluctuations as we all know, but if you zoom out over the course of the week and it's still looking good, then that's when you're going to go and say, all right, we can go and spend more because over the course of a week our cost per lead's below, it's trending down or something like that, is that accurate? Yeah, I would say very slight twist where kind of at the point in the organization, thanks to just hiring some killer people inside our paid marketing team, no longer really giving them numbers on how much is spend, more so just giving them guidance of like this week, we need to focus on cost per lead or this week, we need to focus on contribution margin and just trying to build that feedback loop where they're making the decision. We have not really an ad spend, but to be honest, we have targets we're trying to hit and their goal is to allocate the ad spend to hit the target. We actually didn't have a financing for a very long time, which probably is why we had the permission to be able to do these kind of weird stuff, but their goal is essentially to hit some kind of target and then kind of what I see my goal in the organization is to tell them like what's target to focus on and when they make these decisions, I like separating the person who makes the decision versus the person who is like giving the feedback because I feel like too often when you're making the decision yourself, you can tunnel vision and do seeing the numbers one way because you really want something to work and it's just human. They've done the double blind testing to show that like it's just a statistical thing that if you're making the decision, you're going to be biased towards it. I think that is why like they're making the ad spend decisions independently of kind of like what I'm doing, which is kind of giving feedback and recording and posting every day on like what we're seeing, what the narrative is, whether what leading indicators are clean, which ones are messy. Sometimes everything just works out where you have a great cost for lead, great cost per check out, great cost per ad to car, bad contribution margin, but you're ahead of a seasonal moment. And those are the moments where we've done a really good job of order with as an organization of kick advantage of it, really raking it in those moments. More often than not though, it's nuanced where two out of three metrics might be great or one out three of the metrics might be great, but that one is amazing. And then just trying to just trying to be able to iteratively figure out whether you're correct, the other part about like just posting every day is that it gives you a lot of times up at that. Like I talked about the chicken jundering, a lot of it is just trying to build some kind of instinct. And I think the best way to build the instinct is to write down what you actually think and then be able to see if you were correct, like to make these kind of predictions. We have a public one where my team does it and I think it's been a great tool to see like, if we made a mistake, what were we saying in the lead up? That's been a really good retroactive exercise for us. There have been times where we understand, there have been times where we overspend. And it's kind of funny reading these old posts because you can just somewhat feel the delusion where I'm like, oh, like it's going to happen. We just really need to believe like we just need to run aggressively for one more week and it'll be fine. And then the week after it's like, well, that didn't work out. So let's learn from that. But I think that helps it. Sorry to interrupt you. You've mentioned this post a few times and I just want to clear up what that is. Are you saying like every week or every month or every two weeks, are you doing some sort of like, are you writing a memo? Like on the same day every week or every two weeks or something and that's like the post and that like is looking at the data and like you talked about, diving in the data, creating a narrative on like what's happening in the business. Is that what you're talking about? This post that you've mentioned a few times? Like what is that? Oh, it's a slack post. So we have a slack channel with all the leaders in the company and posting every single day actually. So we'll post the numbers on a daily basis week over week, year over year and describe what we think is happening, describe like how, whether it aligns with our plans. The thing is that it's not about trying to micromanage, just spend budgets every single day. It's more about just trying to, it's more about just making sure you're really rigorous and looking at the numbers every single day. So definitely should not be reacting. And I think that's really the hard part though is just sometimes posting the numbers, saying like, hey, there are some signs that this might be happening, but let's sit back and wait and get a little more evidence. That's really just been the challenge. But also I think it's been what's developed as performance marketers. And also, me myself, I had no experience in this before. I think that being able to have that retrospective look on when I succeeded, when I failed, what I was saying, when I succeeded, what I was saying when I failed has really just developed me and just my personal skillset and that respect. Can I ask a very, to jump back quickly to a very tactical question around optimizing for cost per email? What is the variance in the window of time in which you realize value from those emails? You kind of alluded to it earlier, but I'd love just like a ballpark of like, is it really, is it super short in November? And is it really long in September? Because I'm just curious like over what time frame, yeah, because that ultimately comes down to like that risk tolerance and budget allocation. It's like, how much are you willing to invest today and how long are you willing to wait to capture revenue from that email? So just like the band of oscillation would be interesting to hear about. Yeah, so you definitely get a spike on the first day. There are some impulse purchasers out there. And I would say for the first seven days, it's a pretty steep curve. Unlike kind of a lifetime value for like maybe a CPG brand though, it doesn't really level out where you can continue to realize value from emails from a year ago or two years ago. It's kind of crazy how how activated a lot of those cohorts are where I'm like, who signed up her email list and not interacting with the brand for two years and wrote suddenly decides to purchase. But I feel like as a marketer, I'm not going to be able to do that. It's hard for us to believe, but as consumers, like, I got a Hex cloud, I love it. Well, my purchase journey for that project was essentially, I heard about it on the podcast, actually. I saw a ton of ads and then after seeing all the ads, I got, I saw it everywhere, organically, where every single influencer chef was using a Hex cloud pen, that's kind of really what sold it on me. But then I waited probably two months and then I happened to get more than two months, probably nine months. And then someone gave me a Visa gift card, and I didn't know what to buy. It was like $300 and I was like, oh, this is perfect, I'll get a Hex cloud pen. The problem though, is that like, inside your Northeast model, inside your, inside your post-purchase attribution, no data point is really telling that story of like, how I bought where, you wouldn't be able to say, what was the CAQ to acquire me? It's just such a vague number. But then it's like one of those like, same that like every model is wrong, but some are useful. So then it just trying to find something that is a representation of reality that can take this whole story and distill it into something actionable, knowing that like, the metric isn't correct, but it has value because it tells you something like the law of averages where eventually you'll get the value from that. - In this case, I think we use the first touch attribution model. I think it's a podcast, I think it's a podcast purchase counter. - Yeah, I'm trying to remember what I put on it, but I don't think that you had podcasts as an option. Well, I think it was podcast generic, but not podcast operators. - Do we need marketing operators specifically in the post purchase survey? - We have a write-in, so I'm gonna go check out the write-ins after this and see how many are attributing to marketing operators. That's really, what's like the average? Do you know the average? Like, if you were to blend it out over the course of a year and you were to say, hey, I wanna know what's the average time from email sign up to order? Like, do you know what that number is? Or is it so variable that that's not even that? And is it so seasonal and variable that that's not even that helpful? - I would also bet, you know, like, Michael, you know, just from a cash flow perspective, there has to be some breaking point for the time in which you're willing to wait, right? Like, even if, even if you found out like every email you acquire today, 20% of revenue comes two and a half years in the future. It's like, you just can't, but that can't be your break even point. There's two and a half years in the future if you're gonna spend a dollar today. So, can it's a conscious point? Is there an average? And then, and then, how have you thought about the tolerance of time? - To the question of like, is there an average? It's hard to say because, like I said, the lifetime value of an email subscriber continues to appreciate over multiple years. So then, what is the lifetime value? Well, it's probably how long the person lives, right? So, I think that makes it a little bit hard to find the exact timing. I know that we realize a disproportionately large amount of value within, let's say, like, 60 to 90 days, potentially, and then past there, it's a very slow, but actually surprisingly steady gain. In terms of like the risk, or in terms of like the cash flow aspect, I think that loops it back to the first point, which is just, this is just a risk management exercise, right? Where, at some point, you need to draw a line in the sand, and it makes sense a lot where things happen that are unexpected. They're known unknowns and unknown unknowns and unknown unknown might be like, "Carous, unknown unknown might be a quality issue." There's always something that might disrupt your ability to get value from these email subscribers, so you need to hedge against it. I think you're just building a margin of safety where you say, you're building a large and not margin of safety where even if everything goes wrong over a reasonable time frame, you'd be able to monetize it. And let's say hypothetically, you make the call in January, and you realize that like, "Hey, we didn't, we weren't able to monetize the segment as expected, and maybe for a couple of days, if not like weeks, we spent unprofitably." Then you just bring in the feedback loop, and then you just really try to learn from that, and you better understand next time you're in those shoes, like this is what the reality is versus just the basic model, right? Because I think, yeah. - The other thing that I think is interesting in this conversation is, like as your business grows, your risk tolerance can obviously go up, or like the break-even point for which you want to capture value from a lead or a customer if you're doing like a more traditional LTV to CAC model, can go down, right? And it's like, you know, him's, I haven't seen this talked about in the timeline in a while, but him's runs at a 20% AMER. For every dollar they generate in a new customer revenue, it's been $5 on advertising. It's like the only reason they can do that is because they've been doing it now for nine years, and they have millions of customers. So they have a ton of returning customer revenue. So they can be way far out on the risk curve for acquiring that next customer, in a way that like no startup today would be able to, unless you just wanted to like, light money on fire forever. So as Jackson's gotten bigger, I would also just assume, if you want to continue growing and prioritize growth, you can get even more aggressive with your cost per email and the payback in which you expect to get a return on that, the payback period. - Yeah, and I think that's like, well, all brands are talking about the modes that their brand has. I feel like that experience one is, and surprisingly big mode, where honestly, I look at you guys at Ridge a lot for it, where some of the decisions you make, I'm like, I know it's more optimal. I know that we should be doing that, and honestly, I just, I don't have it in me at this point to do it, but it's something that, you know, I'm sure you guys have the same experience, where gradually, gradually, they're trying to error and seeing the successes, you're able to act more aggressively when you see an opportunity. I think that's something that like every brand kind of has to do within themselves, where we could describe all of our playbooks in exact detail, exact numbers, and someone else still wouldn't be able to copy it, just because they would probably blink when it got really, really tough. And I think that's a hard part, is that like these kind of decisions, when you take on risk, it gets tough. (laughs) And even right now in the organization, I think that's something that like, we're internally trying to do, is go from a mindset, we used to be trying to win every day, than every, we were at win every week. I feel like right now we've reached the level of we're trying to win every month, but I think that ultimately what's gonna grow the brand is being able to sacrifice entire months, being able to sacrifice October, being able to sacrifice April, being able to sacrifice, I don't know, July, in order to drive future value for the quarters, and then shifting the mindset to being able to optimize the recorder, maybe someday we're even thinking about all acting entire quarters to optimize the whole year, not sure we get to that point, but it's something that like we can all describe, but in the moment, it's just hard to execute, right? I wanna, you're talking, you're kinda dancing around a little bit, so I wanna ask you some questions on this. I understand you have a decision quality framework, a two by two of good and bad decisions, cross with good and bad outcomes, and that this is what guides a lot of the decisions that you're making on a day-to-day basis. So can you just like explain what this framework is first, and then I have a few of follow-up questions? - Yeah, I was going down a rabbit hole reading one about probability, and just two really good books, how I recommend. Number one is Thinking and Betts was written by a professional poker player. Number two is just Michael Malvusen. He writes a lot about, he's a big investor, writes a lot about the impact of luck. As a side, one of his really interesting points about like, you can tell whether something takes a lot of skill or a lot of luck by asking can you intentionally fail? Where I could not intentionally fail a coin toss. I could not, but I could intentionally fail, let's say like launching a coin, probably. They did a study where they took a bunch of hedge fund analysts, had them choose 10 stocks that they thought would intentionally fail in 73% of them still be the SMP. So it's just like a weird little aside about the impact of luck. But essentially, what the framework is, and it's just another interesting thought exercise, is that we all know that luck has an impact toward decisions. We can all talk about really easily, what are some good decisions we've made that led to good outcomes. We can all talk about what are some bad decisions we made that led to bad outcomes. I think the really hard one that I spent a lot of time thinking about personally is, can you guys list a good decision that led to a really bad outcome? But that if you were given the chance to make that decision again, you would make that kind of decision again. And can you do the opposite, which is, what is a bad decision you made that led to a good outcome? But if you could go back in time, you would never do it again. It's something that I kind of puzzle over, I'm surprised in a moment. What's an example of that? Like what's a bad decision that led to a good outcome and what's a good decision? Because I could see that the contrarian here would say, well, how could it be a bad decision if it drove a good outcome? Doesn't that by default mean it's a good decision? But I want to hear, what's an example of bad decision that led to a good outcome and good decision that led to a bad outcome per the framework you just mentioned? Yeah, so I think bad decision that led to a good outcome really hiring for me. So probably back in 2023, we were hiring a bunch of media buyers. And we found some great candidates, really, really loved one candidate. And as soon as I saw us moving in a different direction, within 24 hours, I wasn't hiring at the time. I just made up a role, got it approved, and had it in an offer. Had no idea what he would do, had no organizational framework, had no list of responsibilities. She's had a salary and an offer letter. And it turned out that worked out really, really great, where he's a great culture fit specifically for the organization. And within my team, shout out to Joe. But in hindsight, it was a completely irresponsible decision that could have blown up in my face, like hiring someone. without any idea what they would do, without any buy-in from the organization. Probably would not do that again. (air whooshing) Marketing operators, I wanna challenge how you think about post-purchase. If you zoom out, post-purchase isn't a tactic, it's a system. It's your cart, your checkout logic, your one-click upsells, how you increase AOV, revenue procession, profit procession, it's your confirmation page. It's all of that system together. The entire flow determines how much incremental revenue that you can make for order. That's why rock after sell isn't just an upsell tool. It's a design system for the full post-purchase experience. Rock after sell gives you one unified system, a smart cart, and checkout offers one-click post-purchase upsells, and thank you page monetization with rock banks. And here's where it becomes very strategic. Beyond the 30% revenue per visitor lift, rock after sell opens a monetization layer that most operators haven't fully priced into their unit economics with rock network products. Run the math on your own volume at 50K orders a month, that's 15 to 25K in pure profit with rock banks. At 100K, it's 30 to 50K. I'm not great at math, but I love those numbers. Straight to the bottom line as well. Every month, from a page, your customers already landing on, won't affect conversion rate. Just free money you can pocket at the end of the month. No inventory, no operational lift, no contracts to lock you in, and this is not just for Shopify, native brands anymore. Whatever platform you're on, rock after sell supports it. Operators, listeners can activate rock thanks and get the full after sell suite for a year, or grab an extended 60 day trial to test post-purchase performance. Go to after sell.com/operators, build assistments and let it compound. So you're calling that a bad decision because you weren't really prepared. Like you didn't have the roles and responsibilities outlined, you didn't have a 30, 60, 90 plan. You just in your head, where like I think we need media buyers, I'm just gonna like make this happen and hire this person, but you weren't really ready for it. So you're saying that's a bad decision, but they're saying over time, this person like figured out their role alongside you, and like they're providing a ton of value now. So like it led to a good outcome. Is that the, is that what you're saying? - Yeah, even simpler, I think that if I made this decision, if I made this decision 10 times in a row, it would probably worked out two times and that I was essentially saved by the bell where it happened the one time I made this decision, it worked out. The flip side is good decision, bad outcome. Like I said, I did a lot of man planning and we would make investments into like a new category, a new product, and we would make a bet on it where sometimes we're just trying to grow existing products that exist in categories that were a small percentage of revenue. We predicted we could 2X the product and we had the appropriate inventory buy for that and reality it like 3 to 5X, which sounds great, but it just meant that we sold out within like 30 days. It sounds like it's signing that bridge with all of your suitcases, but I wouldn't, I need to keep expressing to the team where that's a situation where it's a, it was a bad outcome, us not having a key new hero in stock, but on the flip side to go into demand, planning and say we can 5X any category is ultimately just gonna lead to like a disaster on your inventory list. So even though that was a bad outcome in that one instance, we shouldn't learn to suddenly aggressive, suit hyper aggressively forecast every product. That would just 2 out of 10 times will end up 5X in something and those 2 out of 10 times. So run out inventory, but by and large, if we have to keep making the decision, we should keep making that same decision. - I don't have a good example off the top of my head. I love both the ones that you just gave. What it does remind me of, and it's ironically 'cause I also read thinking in bets a couple years ago, but like Deek, I have had the conversation with my team where we need to de-couple results from the quality of the work that we're doing and trying to make that distinction a little bit more. There's a period in of time in 2024 where the business was doing, I felt like we were doing great work and the business was doing fine. And what I felt would have been the wrong decision to make at the time was to like, you know, all of a sudden change our processes, lose confidence in our internal strategy, our internal processes, our team, et cetera, and say, and instead say, "Hey, I'm gonna, I think I'm gonna de-couple the results from the work here a little bit. I think we were doing extremely quality work. I think we're making extremely high quality decisions. Let's continue down this path." And ultimately, it's a matter of luck to some degree where at some point, whether it's, you know, the meta algorithm or consumer confidence or whatever else these things that are out of our control will turn back in our favor and we will begin benefiting again from the quality of the work that we're putting in. So I do like that line of thinking. I think it's an important sort of concept for people to consider. - I think a lot of, it's like the decision versus the execution. Like you have to decouple that too 'cause I think a lot of times you actually, it's not the decision that was a bad decision. If the outcome wasn't great, it's often, the decision might have been good, but the execution didn't drive. So I think like, we've made a few influencer bets that I, like we did this Haley Bieber activation. I mean, this was a year ago. This is probably three years ago at this time and I just don't think, like, I don't, now I think in that situation, maybe it wasn't. I think it actually was a good decision. I don't think we activated it, executed the right way 'cause I think what we ended up doing was we basically sponsored this YouTube series that Haley was doing. We were integrated in the show. And what, so I think the decision to activate with her was a good one, but the way we executed in that integration was bad because we ended up just like integrating in a show that had a bunch of like, you know, 17 year old girls kind of gushing over Haley Bieber. And reality, what we should have done is led with like a paid deal and had the organic integration be part of the deal. So I think that's an example. I don't think the decision was necessarily bad. I just think the way that we decided to activate with her was probably not the right way for us to extract value. And we see this all the time. It's like, hey, we have a new offer idea on paper and we even have some data points that suggest it. It worked really well in some aspects, but we also have other data points suggesting that there was parts of the offer that were clunky and didn't work as well. So it's like, good decision, execution wasn't quite there. I think that's important because otherwise you end up like telling yourself you made the wrong decision when reality, it's like you might just need to iterate on the execution of that decision. And then that's the unlock to, 'cause it's like decision, execution, outcome. So like that middle point, though, I think in sometimes get glossed over, it's like, oh, you shouldn't have done that partnership. Oh, you should have done that offer. Oh, you shouldn't have done that ad. Well, it's like, no, not necessarily. Maybe we should have just done that partnership that offer that ad a little differently than we did. And that would have given us the outcome. So it's, I think it's interesting to hear this like decoupling of the two from you, Michael, 'cause I think it can very easily get connected to a bad decision when it might not have been. - So was that a variety of company sizes? Was that like five billion? Was that one hundred million? Was that, I don't know, almost half a billion? Just a variety of different company sizes. And I think as leaders in the company, part of the challenge is, as you scale, suddenly people start going for the 100% probability of a 15% return projects, where they're afraid of being Ron and they're looking essentially at their accuracy. But trying to shift them to focus on impact versus accuracy, where a 50% chance at a 3x return, if you keep making that that over and over again, will beat out 100% probability at 15% return longterm. But then trying to give your team, essentially the permission to fail. I tried to set the expectation that like, I told our team that like, hey, if you're not failing 20 to 30% of the time, you're not taking big enough risks, where you're expected to do things that do not work out, and you're not gonna be held accountable for it. If in the big picture, these things are having that benefit. So trying to have that fine line, of you still wanna hold people accountable to results, but whether you should hold people accountable to every decision, I feel like it's just gonna drive that like small, incremental bet constantly, versus trying to give none of the permission to take bigger risks, but also have an asymmetric outcome, essentially. That's been something that we've been working on a lot within the brand and just trying to encourage, these senior manager, direct-level people to make these big calls that might blow up in their face, but also have the comfort to do it, and also the expectation that they should be doing that. - Do you have any tips for teams that want to be testing at a higher velocity, like with this in mind? And this is a super important point. I've talked about it on the podcast a bunch. I heard it from someone at Elcaderton years ago, where they were like, we decided we have no ability to determine what's gonna work or what's not. So our main KPIs just volume of testing, and it sounds like you're maybe more in that camp, albeit like thinking a little bit about expected value. So for people who want to be testing at a higher velocity, what are some of the things that you guys have implemented that could be helpful? - Yeah, I love that question because it's one of my favorite topics, and I think it comes from background in R&D, where it was running essentially the testing program for these different memory forms. So I think that let's say like E-Com or holdout testing, go and focus a little more on E-Com AB testing. I think the standard is to run things to 95% confidence, and I always struck me as odd because as a performance marketer, how often do you actually make a decision with 95% confidence? Pretty rarely, we're pretty used to making decisions with 60, 70, 51% confidence, we just make the call. So then ultimately what happens though is that, just 'cause you have the ability to do something with 95% confidence on E-Com AP test, the cost is you just slow down your test velocity, where let's say you, go from 95% cough and to 80% confidence. That would literally two to three X your test output, ensure you would have more leakage, you'd have more false positive, more false negatives. But the resulting net benefit would be 1.9 X for your total testing program. There are certain conditions that you need to meet in order for this to work out, where essentially the quality of your test has to, the quality of the test and the backlog have to be the same as a current test you're running. So if I think I got five winners and 10 losers in my backlog, I should spend more time on the winners to make sure that I realize a full benefit from them. You need to have the development resources. But essentially it's like the idea of like a cost of quality, I think is eating into a lot of econ AB brand is eating into the testing program for a lot of econ AB tests. Because we're essentially just dating the velocity just so that we can have 100% confidence in something that like we definitely do not really fully need 100% confidence in. Holdout testing is another one of my favorite ones where I would really love if I sasted this. I did the math recently and let's say for like a $30 to $50 million brand, it probably need to be running like a nine plus month holdout test in order for statistical power, that's just how math works out. Even if you hit statistical significance before then, it's probably just noise. That makes you challenging. And then it's like, well, I'm like a $15 million brand. I probably should be doing some kind of holdout testing to get results like what can I do? Same idea of a spectrum of KPIs where you get essentially clicks at the card sessions, check out sessions, orders revenue, just shifting the holdout test and all of these in Shopify. And so looking at your incremental orders where you're never gonna hit stats so you're gonna be able to run a time. Why not look at the incremental check out sessions or the incremental ad to card sessions or the incremental clicks? That's really been an able for us for helping test these small channels where like, I look at the revenue and just complete noise. I look at the graph and one percent bought a solid gold chain and just completely made the test irrelevant. But for example, if you could tell me that Google is giving me a lot of incremental traffic, that's someone actionable. And then if you could tell me that the ad to cart rate for that traffic is comparable to the ad to cart rate for just normal traffic, then that's also actionable. And then essentially the benefit of this is that suddenly you're enabling yourself at regardless of scale, essentially, to get actionable findings out of these holdout tests. Of course, I would rather have the full like gross profit, contribution margin, impact with a lot of granularity. But sometimes just the time isn't worth it. The time to actually do the proper test is just ultimately not worth it. And by the time you actually finish it, the insight has probably shifted. So like I said, would really love if someone started making a SaaS platform to do holdout tests on different metrics, because right now I'm doing it manually. It's a great example of the art of triangulation, right? It's not perfect, but it's data driven, and it's going to allow you to make most of the right decisions and most of the time as long as you're confident in what those soft metrics are that you're comping to. But if you know Meta's like your number one revenue driver and you're confident in that and you have a solid cost per add to cart, well, yeah, then you go on Snapchat and you're getting a dollar better cost per add to cart. I'd be confident scaling Snapchat based on that, right? Yeah, and I think even something like YouTube, not a great click through channel. But if you can tell me that we're getting a lot of incremental traffic from YouTube, I can probably do the back, math in the back of my head of like, okay, what if the rep view per click is about the same as Meta, is that like a good channel for us to be on? It just unlocks a lot of different aspects, but then we don't want to be spending $100,000 per day on YouTube and you know, previous statistical significance within like 30 days. So then you just end up having to kind of find these round about ways to kind of like massage numbers into a way that you can form a narrative where the goal isn't to publish your results in our R&D paper. The goal is like, okay, I've seen enough that I'm willing to make this bet because I think that disproportionately it will pay off.