Season 7, Episode 6: Measuring the long-term effects of brand advertising (with Carl Mela and Ross Link)

Mobile game developers no longer need to pay up to 30% in major App Store fees. With XOla Web Shop, you can create a direct storefront, cut fees down to as low as 5% and keep players engaged with bundles, rewards, and analytics. Start today at xola.com. That's xsolla.com or use the link in the episode show notes. The problem is that the distinction needs to be drawn between the confidence of the economists and the correctness of their analysis. Hello and welcome to the mobile dev memo podcast. I'm your host Eric Soufer. I'm joined by two guests today, Carl Meela and Ross Link. Carl Ross, welcome to the podcast. Thanks. Good to be here. Yeah, thanks. I'm delighted to be here. I've listened to the podcast. Excellent. I'm honored to be asked. Well, I'm very grateful that you agreed to join me. So, Carl, you are actually the first guest that I brought on as the result of request. So someone requested that I bring you on. And, you know, obviously, if you're familiar with your work, you're a very prolific academic in the space. And I said, yeah, that's a great idea. Can you introduce us? And he said, I don't know him. I just think he'd be great to be on the podcast. So fortunately, we do have a number of mutual acquaintances. So Garrett Johnson, my friend, connected us. But also you worked with Julian Runch at Duke. He told me that you were the one who brought him out there. Is that correct? Yeah. Yeah. That's he's fantastic. It's been great interacting with him. I also loved your episode with John Lynch and JP DuBay on privacy. They're informed with you. Yeah. And obviously we had a question after you asked yourself those who recommended me. Was it a friend or a foe? Hopefully with a friend and someone in one tanker rating. Well, I didn't I didn't interrogate. I imagine just someone who as deep respect for your work. And then you recruited Ross to come along as more of like the practitioners voice. So I'm very, very happy to have you both maybe just to kick off the episode. You could both introduce yourselves to the audience. And maybe Carl will start with you. Yeah, sure. I'm a PhD marketing from Columbia. I've been a Duke professor since 1999. I served as the executive director of MSI and I'm on the board of the advertising research foundation. My early work was on measuring brand equity, which we're discussing today. And then I've shifted more and D commerce and digital advertising. And those two streams are coming together in commerce media. And I've covered that in your podcast. That's another interest. I should also add that I'm from Boston, but I met my wife in Seattle. So I'll let you figure out who I'm rooting for this weekend. Well, you all will keep that secret. And then Ross, how about yourself? Yes, sure. So I'm going to see you become a company called marketing attribution work measurement firm. We've been around for about seven years. I started out. I got my BSN O'R from Cornell and be a university scholar. I worked at booze out for a few years in their system strategy department. I worked at PNG and their media optimization group. And then I had a company called marketing analytics. I ran that for 20 years. One of the first marketing mixed modeling companies sold that to Nielsen in 2011 and then ran Nielsen's marketing mix business globally. And they're emerging MTA multi touch attribution business before I started marketing attribution in 2017. I guess it's around the two and eight years. I guess almost nine. Maybe just kind of before we get into the questions, what do you feel about the kind of renaissance of media mix modeling? Because it's become, you know, maybe tell me if I'm wrong about this observation too, but it seems like it's become more relevant again, especially with a lot of digital first advertisers who historically hadn't used it. What do you make of that? The kind of renewed interest? Well, I sold my first company, I'm marketing analytics to Nielsen in 2011 mainly because I thought marketing mix modeling was going to die. I thought the way we do the two measurements was using at the person level. It's definitely the way I would prefer to do it at the person household level. But in the middle of some tests we were doing at Nielsen, it was called digital media consortium. I recruited a bunch of companies, Coke, PNG, Pepsi, you know, last company's Chlorox and then you know other companies as well. We had Crocs was part of it before Salesforce bought them. But the idea was to test marketing mix modeling multi touch attribution, I expect some of mass panel tests and try and find out what kind of things worked best. And in the middle of the test, Google said, oh, because the Google is part of it and they were giving us very useful information including person level impressions and say, up, guess what? We can't share that anymore because we're very concerned about data leakage. Well, it turns out that data leakage really means revenue leakage because if you're giving out your person level impressions, your Google or your Facebook, you're giving out those, you're letting you know companies like us get those person level impressions. And you're releasing your targeting list. That's your targeting list that you just gave out. I gave those impressions, those people because they I study them for a while. I know they are in the market for luxury automobiles or something. So that is what they're really concerned about. So I build. So that's when they stopped giving out personal level impressions and that was kind of the beginning. It also was around the 2016 election came with Janolidica and all that stuff. But it really was. And what the thing I took away from it. What it was, it wasn't really privacy. It was self interest from the platforms that then kind of merged into privacy. And so that's why I don't believe we're going to get personal level impressions again from Google that we can then merge with Facebook. And so that's why I really believe that was kind of the beginning and we are coming back to my next morning, which I love. And it's my second best thing after personal level stuff, but it does work. And I think that that emergence is going to research. It's going to kind of stay. Yeah, fascinating. All right. So we we kind of workshopped these questions. I'm really happy with the way the questions came together. So what just kind of give the audience a sense of how we'll conduct this. So usually when I have two people on, I'll try to have like, I'll identify a main respondent for the question. Of course, the other guest is more than free to kind of also chime in, but I'll have kind of the person I've identified as the as the primary respondent. And then, you know, we'll get thoughts from the other person too. So the first question will start with Carl and it is what do most companies get wrong about brand measurement, both from an analytical, but also a conceptual standpoint. It's a great question. I have several points to make on this dimension probably the first line that should be no surprise brands like Coke and Apple. They weren't built overnight took years and years to build these brands. Why it is it takes along to build a brand. It's not just changing somebody's belief about the product attributes, but you actually have to change their preferences changing preferences is hard. It takes a lot of work. It takes time. It takes repeated exposure. Takes giving people compelling reasons for purchase, you know, discount isn't a compelling reason for purchase is an excuse. And it can't be measured with a small amount of data or an experiment. It's something that takes a very long period of time. I think the second point I want to make on this domain is all too often I see brand strategy follow from what can be measured instead of facing their mesh and run around their strategy and this leads to some bad strategies. So, you know, the canonical example of this in the 1980s was the release of store level scanner data prior to that. The sales were measured through biweekly inventory counts in the back room. So they literally had someone go in the back room and figure out how many cans of beans and sold in fact if you have offered to earn bean counter. That's literally where the term came from. And so if there was a discount for a given week, it would be sort of aggregated away over several weeks. And then scanner on lines or see me digital scanner data became immediately available so companies could easily see the sales lift from a discount. And so they started discounting a lot and advertising is advertising effects are hard to measure. We know that advertising the last season or low. But you know, advertising is intended in many instances to build brands. And that's not something that happens on a weekly cadence. And so what happened was company shifted budgets from advertising to discounting. And so that's when I started thinking about tools to measure long term effects because it's pretty clear that that brands were shooting themselves in a foot. But again, these things are hard to measure. More recent incarnation of this phenomena is the advances in measuring direct response advertisements very easy to a B tests. So ads are measured in terms of direct response. In fact, I think one of your past speakers is talking about one of the main KPIs as being clicks. The clicks and even purchase intent is not brand associations. It's not brand believes. I mean, I could drive clicks just by advertising a discount. So basically what's happening is because people can measure direct response so readily and so easily they have shifted out budgets to direct response. And I think that is that's an issue and that's again what I've worked on some of the long term measurement tools that Ross and I will talk about shortly. A third point, I think advertisers are very tied up for similar reason in targeting, which I think is wonderful. But in underestimates the value of reach. And so let me give you a very specific example. Imagine you have a carbonated beverage brand. And you know, you advertising that carbonated beverage brand is the best person to reach is the person who's trying a carbonated beverage for the first time on that day. Right. You want them to activate it around your brand when they make their purchase because the customer lifetime value becomes enormous. But you don't know who's just about to start purchase for the first time. I mean, there may be some signals by like age, but not everybody's young when they first drink a carbonated beverage. And so, you know, some of the exact same talk to an interested say, hey, no, a really sensible strategy is to advertise to everybody. And I know that if over lifetime, you know, if someone's roughly 300 days a year, what times 30 years looking at 9,000 days, 10,000 days, I know that on 9,999 days my ads are wasted and only on what day it's relevant. But more than oxets the law. that, again, this notion of reach versus direct response is something that's a concern. And one last point in this area, and that's obviously marketing mixed models and media mixed models. I think, you know, just, I want to clarify this before we get into it. Media is only a component of brand strategy. And, you know, I have a paper, I think it's journal marketing research in 2010, and it looked across 25 consumer patch good categories, about 709 brands over five years. And we found that this short term and long term sales of last, this is 1.37 for product and 0.74 for distribution. In contrast, it was 0.13 for advertising and 0.04 discounting. Now, ironically, like if you, if you walk the halls in many research companies, a lot of money is being spent on ad and price measurement, but not nearly as much as being spent on things like product and distribution in terms of developing brands. And again, I think that relates to part two to this measurement. So how does some companies solve this? They have one phase where they use their marketing mixed measurement. They use this to figure out the optimal media allocation. And then they use a media mixed measurement law. So summing up, brands aren't the lower night strategy should follow from what can be measured instead of the other way around. Reach is underappreciated and underweighted a lot of strategies. And we need to think about marketing mixed modeling instead of just medium mixed modeling. Ross, I don't know if you have anything to add on that one. Well, no, I agree with all that. We, I mean, you can't, I can't, I think most, most people can't do it. Actually, a medium mixed model that only has media. So I think we always, even if we're only focusing on the media, I believe everybody will put something in their about price and promotion. Or maybe they'll just take Nielsen's estimates as they're going to be, I mean, the, no, syndicated data vendors estimates as gospel. But yeah, you do have to control for those other things. But yeah, we do both types of models and marketing mixed models prefer to us. Definitely. Carl, so you talked about, you know, the kind of shifts to direct response just as a function of like, well, that's more measurable. And you kind of made a point early in your answer about the strategy is essentially as a mistaken approach, the strategy was downstream of what was measurable, right? So you had companies building strategies just on the basis of like, what they could measure. And then that, that would kind of, that, that would kind of result in this general shift towards like, well, we're going to spend a direct response because we can measure that. And so the measureability of that medium becomes a constraint on our media spend. Is that kind of, those are the same thing? Is that a phenomenon, is that a phenomenon of that, of that mistake? I think so. And let me give you another example again related to a recent episode of yours. You know, fundamental question a lot of companies are asking themselves that AI make ads as well as people. Right. And so you've had some ground break in some, it'll start to be some of the studies on that, on that very point, where the answer is being sort of the lawns people don't know it's created by AI. Yes, AI generally does better than people. But that, you know, again, the measures are things like clickability. But that's not brand attitudes and brand associations. So the jury's out like part of building a brand identity is relentless consistency about what you believe your brand to be like, AI is not tuned to that issue. It's not clear that it can generate executions in a manner over a long period of time that reinforces brand messaging. And so like the jury's out as to whether or not I could use AI over the long term to augment branding. Maybe I can, maybe I can't, but the point is like, we don't know because we've been focusing on actual current measures. Right. Well, that's that kind of fundamental attention. Did you see that there was that article that was circulating a few months ago, but it was a mental as I might be mispronouncing the name, but they have, they make all these different confections. So they have like chocolate. It's a model. That's right. And I'm forgetting all the relevant details of this article. But, but they had worked with like Deloitte or something to build like a model that could supplement the creative output, but it was trained on their own internal data. Right. So like instead of surrendering this functionality to Facebook or whatever to then, you know, we've seen some kind of scandalous results of that. There was a case where Facebook was circulating an ad. And they were a fighting an ad that they had built on behalf of a brand and they were using like an old woman as a subject of the creative when the brand is a men's t-shirt brand. So, but anyway, Montelei had built their own model for generating creative, which was trained on their own data. So it should be sort of more attuned to their brand sensibilities. Do you think that's a possible solution there? I mean, not everyone can do that, but if you can, is that a possible solution? I think it's a step in my right direction for sure. Right. We need to think about brand guard rails when we're developing an AI strategy towards creative. Yeah, absolutely. Got it. It's a rise. Well, kind of next questions for you. One of the most common internal political issues that you see arise with marketing measurement within organizations. Who are the less obvious stakeholders for marketing measurement? And how do needs sometimes get misaligned with measurement? Well, those questions are linked up with things Karl does too quite a bit. But so I think the biggest political issue in measurement is short term incentives of brand managers. Their incentive system is set up to drive volume for the next one or two years. And the owners of the company want the brand to be successful for decades. So that's a misalignment that causes them to focus on volumes for the next couple of years and not necessarily building long-term brand loyalty. That's why I've worked with Karl before. I'm doing long-term effects miles just to try and try and change that. So that's the biggest one. That's one's been around for a long time. Another one that we have is that we work, but I will say that there's less political conflict in what we do now than years ago though. I mean, there were many times when we first started doing this. I mean, I've been doing this since like 91. So at the beginning, there were many people that just question the whole value of quantity of marketing measurement. So that was the problem. There would be marketing managers. You just don't believe it. It's a soft science. You can't really measure this stuff truthfully. When I was in business school, that was I believed. You can't measure, you're not going to be able to measure advertising, maybe promotion. But you can do it. So that one's kind of gone. But we do have some, so the agencies, the agencies, they also, they have an incentive mainly to drive the top line. Because, you know, saying with the brand managers, they're not really responsible for all the costs for the cost of goods. So they can't control all that stuff. So you can't necessarily compensate them on profit. So they are mostly instead of to drive the top line. And measurement firms, if you work through it, if there's somebody tells you, I want to optimize profit. It's very clear how to do that. Some sort of quantitative optimization. You know, you consider all the domestic returns. You have all the response girders and you tell them to keep pushing, you know, spending on advertising until the next dollar doesn't drive enough revenue subtracted cost of goods sold to justify the cost of the media. It just, it's, it's a very clear where how to optimize profit for us as a measurement firm. How to optimize revenue? That's really, it isn't really literally what people really want to do. But they'll say that I've just focused on revenue. You focus on revenue, you spend up a zillion dollars, all you got in the borrow more and spend that too. So you really need, you need some sort of waiting between the two and then judgment comes in. So as an measurement firm, it's much more clear how to follow as a optimized profit. We tend to focus on that. And so therefore the agencies generally want the client to spend more money than we tell them to. So that's, so that's the conflict that we run in on that one. Another one that is kind of near and dear to my heart is like, who does the measurement? Okay, we're a measurement firm. And then everybody's got all the different people that do measurement will have their own incentives and biases and so on. You know, we measurement firms, their typical biases, not me. We don't do this. We focus on straight shooting, just to help people truth. That's a stick around and keep doing this for 30 years. But there are a lot of firms that if you tell your incentive system is to tell the client the ISD ROI, to ring out the highest ROI you can get from that data, give that to the client because the people that are paying you, generally that's what they want to show to either brand management. They want to show their managers or it's the agency or it's the platform pretty much almost everybody that's our stakeholders that maybe the CFO wants us to show a unreasonably high ROI. So that's the measurement firms platforms. They all also want to show a high ROI and they'll do that using a couple methods that tend to show high ROI. I don't think they're all, you know, being evil about it. They're using methods that some people tell them is good, but some of these methods do tend to give you high ROI. One is match panels. If you just just look at a very simple, here's people that are exposed, here's people that were not exposed. A reasonable person would think, well, that's a good way to do it. Problem is with all the high sophisticated targeting mechanisms that they're out today, people that were exposed were not exposed for some random reason. They were exposed, but they're likely to buy. So then you compare people exposed to people who are not exposed. You know, oh, wow, here I looked at a whole bunch of people that saw a loan ad recently because I mean they watched a loan television program or something, because that I hit them with an ad. Well, geez, that's why, you know, that's why they took out more loans. They were already more likely that is a big problem. And that is in most platforms measurement system. It's in there. It's not in some of the best ones like say Google's, for example. I think they do a very good job of something called ghost ads. It's a complicated way. It's in an auction. Again, in their defense, I don't think everybody's evil out there, but auction trying to do a random I control trial in an auction system is quite complicated. I think Carl knows about that really is. And the it's like, when do you substitute in the control product before the auction or after the auction? You have to do it after the auction. And so Google does that. Most people do it before it is so complicated. So that's and then and then. So it's between the match panels, the sort of buy-a-stocks, and measurement platforms tend to give buy-assessments. So that's another thing about who's doing the measurement and what is there in the center system. And then you ask about less obvious stakeholders. We have a lot of stakeholders. So the beginning, you know, I've been doing this for a long time. So things have changed beginning. It was not everybody. The CFO didn't really care when we first doing this stuff. But now everybody cares. And pretty much everybody supports quantitative marketing measurement. And they support marketing mixed modeling too. For or whatever reason, either because they understand this thing I talked about about giving away your data, it's giving away your target list. And for whatever reason, they don't trust the platform. So, marketing business is very accepted. So, our stakeholders includes key one is the insights of the analytics team, them, the brand management, going up to the CMO, media buyers inside the company, finance, and the CFO. And we do work with the CFO, CFO, CMO. I know the CEO at some of our biggest, like $10 billion plus companies are aware of the stuff that we do, trade promotion management. Because we do marketing mix modeling. I've been doing that for a long time. We try to model everything, price and promotion, and have it all be reportable. So, therefore, we work with trade promotion people at some of our clients, revenue management, pricing, we do pricing models along with it, and the ad agency, the platforms. And the platforms themselves also, they pay us directly. We do a fair amount of studies for all the major, not all of them, but like maybe five, six of the biggest platforms to do best practice studies for them, for the benefit of their client using credit money or something like that. But then they have their another stakeholder. And finally, our favorite stakeholder, procurement, just love them. But with this many stakeholders, if there's any less obvious ones, I don't want to know about them. Basically what I'd say about that. And I think you had one more question with his bad alignment about misalignment or something like that. Yeah, you can get misaligned. When you're doing it, when your measurement firm like us, marketing mix modeling, when you're going from one client, then you go another another brand. It's kind of the same thing. OK, guess what? Today, it's the same thing as yesterday. We get all your marketing. We put it in a month, best model we can do, with the most granular data we can do. And we report on the ROI of it and recommend what you do next year. So that's kind of the same thing every time. But it is important still at the beginning to ask the client, what is their needs? What do they care about? Oh, we're moving into a new sales channel. We have a new product introduction. Oh, we're concerned about this competitor. Things like that, you do want to know, what is your copy size and normal stuff? What is it? And then this all you do that, then generally, your measurement is aligned with the needs. Call it, does there anything to add? I just three. So breathe for a minute. First, I want to give a shout out to Garrett Chonson, because Ross mentioned, ghost ads. I know Garrett's been a guest on your show. And I think that's a really nice thing that every marketer should be familiar with. That's doing digital advertising. I want to score his point about the challenges of an independent analyst giving an honest answer to firms who have invested in interest in spent. Because you're the number of employees that you have. And your power in the company is correlated with your annual budget or spend. And so if your budget gets cut, your power gets cut. So there's a tremendous incentive for a lot of these companies, not to want to listen. In fact, I co-chirred the American Marketing Association School Market Research for a couple of years. And we always did a top 10 list where we surveyed the attendees on what they saw as the biggest problem in market research. Number one, and not my small amount, every year was their managers telling the design studies to show that whatever initiative they had worked. And that's a real danger. Finally, I also want to add to this point about brand managers, unless you have a long-term measurement system that we're going to get into shortly in place. I tell my students who are brand managers, I say, the best thing you can do is ruin your brand. Just cut ads, discount the hell out of it, move atomic cases. Yeah, so what if people now think it's a cheap brand? You've harvested. Leave that mess to the next person. Because if you go in there, and you invest in the brand and it's brand equity and do a lot of brand building, that's a cost. It takes time to build brands so you get the cost, but you don't immediately get the benefit. Or you're competing against the other brand manager. Look, that's a really big problem. The one-year tenure brand managers, unless you have a system in place to deal with this. And again, we'll give some examples later on how to manage this. Yeah, I feel like marketing, especially when you go to big firms, like I mean, I imagine the kind of firms that Ross works with, like the marketing dynamic is probably so ripe for issues of principal agent problems. Because it's like to your point, Carl, it's cool to spend a lot of money. It's really fun to be invited to Super Bowl, you know, sweets and get Taylor Swift concerts and to have your TV ad being talked about by your family and friends. It's really cool. It's a really nice thing. And so, you know, whereas shareholders maybe would have a different opinion if they were more attuned to the kind of underlying performance, right? And so like there is, I guess there's a little bit of an alignment just naturally, inherently in that. But like, then it's a question of like, who wants the bad news, right? Because if the measurement's going to give you bad news, the external, you know, unbiased measurement company that you're working with is going to give you bad news. And the marketing team is the one paying them. Then you might run into misalignment just naturally, right? So I've seen a lot of cases where the CFO steps in and says, "I don't trust that. I'm going to be the one interfacing with the measurement company. I'm going to pick the measurement company. I'm going to audit their performance. And I'm going to be the one that decides, you know, whether you're spending the appropriate amount of money." But I just one quick question I want to follow Ross. So you talked about agencies and kind of managers kind of having the same, like a little bit of the same sort of bias to just driving that top line revenue. Because with agencies, I think it's pretty easy to understand because that's how they get paid, right? And then with marketers, it's more just that what I just talked about is cool to spend a lot of money. Could you maybe just expand on the idea with the agencies a little bit more? Because I think that's something that seems obvious, but it kind of sneaks in more nuanced ways than maybe some people understand. Well, I mean, the way that's coming to my world in the last few weeks with two different platforms, actually. It made me the last couple months with two different platforms, is that we did the best practice studies for them and then identified, you know, different types of ads that have the highest ROI or, you know, highest ROI. There's different measures of ROI, whether that include margin and our cost goods, or not depends, sometimes that's confidential. But, you know, we focused on ROI and then the, well, this was actually the platform. So, but the agency does the same thing, which is that they want to drive revenue. And therefore, they even will question whether they should be focusing on ROI or not. I've seen that. They say, well, why, why did you tell us which ads were most effective? And we'll do those. Effective, there's, there's, it's kind of getting the weeds here, but effectiveness in our world is generally defined as the incremental revenue per impression or thousand impressions. Regardless of how much those impressions cost. Then you add in the cost of the impressions. And then it's incremental revenue divided by media spend. Then you might, you might multiply the revenue by the margin to get the actual incremental profit. But, but the agencies and the platforms are saying, I was asking us, why don't you focus on effectiveness and forget about, you know, not ROI, because you're going to stop, you're going to stop spending way too soon if you focus on ROI. But you don't really maximize ROI. And we're doing, you know, we do this market mix models. Then we put them in an optimization, like a nonlinear optimization. Big ones that we work with, if I said it's Stanford, I'll let it tell on a next curve optimization algorithm, especially, especially as far as curves. But what I'm saying is that you don't actually maximize ROI after you're done with your marketing smile. You maximize profit. And so, or revenue. So anyway, I guess what I'm saying, you were asking to get a little bit more into the details by the agency. It ends up, you know, having to convince them that you even do need to look at costs. You know, you can't just maximize effectiveness forget about the costs. Somebody gives you a billion dollars of expense. You know, I mean, clients do spend some and do spend a billion dollars on advertising globally. Even billion dollars to spend, you don't just, you can't forget about costs. What you have to do is you have to maximize the, if you're all your focus on revenue, fine. Focus on your comment only. Forget about margin. You still have to get the maximum incremental revenue per dollar. So you get the most revenue for your billion dollars. But anyway, it's, it's, it gets in the weeds about that kind of stuff with the agency. Yeah, see a most love ROI because they can walk to the CFO and say, what these numbers are eye popping, even if the marginal profit is negligible. Yeah. Let's get real. Half the time, you're not sure if your campaign's work. You just kind of hope they do. Attribution numbers look good, but what if those conversions would have happened anyway? Incremental helps you find out for real. No tracking people, no guessing games. Just clear answers on which campaigns actually move the needle. If you want to stop guessing and start knowing, go to incremental. Because good marketing isn't about getting clicks, it's about getting results. Check them out at incremental.com. That's i-n-c-r-m-n-t-a-l.com. Or click the link in the show description. Carl, so you wrote kind of a canonical piece. If brands are built over years, why are they managed over quarters? And in that piece, you argue that a disproportionate focus on short-term data supports a constant cycle of price discounts that manned up eroding brand health and pricing power by increasing consumers' price elasticity, which could be economically destructive in a long term. You propose two measurements of brand performance that firms should incorporate into their strategy, quantity and price premium. Can you explain what those are? And how would you suggest that firms adapt these metrics in their broader measurement portfolio? So let me start with the last question. Because it's easiest how she firms adapt these metrics to be blunt, call Ross, or sound-of-like Ross. The reality is the concept's actually good easy. But when you talk to someone with 20 years of experience, they know the benchmarks, they know that errors, they know the norms. There's a lot of trial and error over the years. And so much money is spent on advertising. You really need to be with firm that does things correctly. And to Ross is earlier a point that you could trust. They'll tell you if something isn't working. In terms of the example and what this all about, let me give you a very specific anecdote that got me thinking about this because when I was an MBA. I remember learning about, you know, how do I evaluate the profitability of a discount? So I was taught you just look at the lift. If the lift is really big, taupeway sort of lost in margin, then the promotion looks great. Go ahead and do the promotion. But let's think of this through. See, I sell a lot more. That's smaller margins overall, my profits are up. But let's suppose I am discounting. What happens if I do it next month? Next month consumers have learned to lie and wait for a discount. So the baseline sales where the sales and things are off promotion are very low. And they have cremmental responses even bigger. So if I go through and I do the things I was taught when I was UCLA MBA on evaluating profitability promotion, then cremmental lift is even higher. And so the promotion looks more profitable. Next thing you know, you're in a death spiral. And so what you really need to do is take a look at these baseline sales and these promotional lifts and the promotional lifts are basically price elasticity. Higher price elastity, higher promotional lift. So the idea is let's go run a marketing mix model. And in the marketing mix model, it gives you very, very useful pieces of information. It gives you the baseline sales. The baseline sales is typically something like an intercept in the model. It tells me if the marketing, all the marketing tactics and the price are the same, which brand sells more. The idea of brand selling more is a measure of brand power. And then I could look at elasticity, which again, how sales change with price. And the lower elasticity, the higher you can raise your price. And therefore the higher margins you have, you'd think a lunch or it goods. So if a brand does really well, hopefully it has to see high baselines and the price elastisties. And you can track these over time. So in one paper I wrote, I remember doing these buy plots, where we looked at the path of brands over five, six years where we just plow these two things. And we saw real movements and interestingly enough, this will talk shortly, those movements can be tracked back to your marketing strategies. So one of the things, again, I've found repeatable in my research and I was very relieved when others found the same thing. It rosses more cases than I do. Not surprisingly congruent with the example I gave you a moment ago, is that a lot of discounting tends to show up in subsequent periods as greater lift and lower baselines. Now advertising tends to lead to higher baselines. So linking this back to the product manager example, if one of my students took my advice and trashed their brand, what would you see by the time that persons reviews being written up, you would see baseline sales go down and price elasties go up. And so even though they stole a lot of cases, the manager is gonna say, well, you trash my brand. And this is created a very powerful incentive to get companies back on track to managing brand health over the long term. Ross do you have anything you want to add on this? 'Cause I know you've done so many more of these studies than I am. - Well, I mean, this is a classic paper. I mean, one of our clients was in it, Steve Gary from Chorox. No, it's a classic paper. I mean, Carlos, they can't have long term effects honestly. And so no, I don't think there was this paper that the algorithm is about the modeling the base volume and the price less. The other thing he came out of there, I think it was a little subsequent paper. But no, that is classic. And then we built our algorithm on another paper of Carlos later on. - Well, so what I loved about that, so in the paper you talk about, you give a example of the cost, right? And so it's interesting because I've always known the cost as that kind of premium tennis brand. I didn't know that they were, it was a quiet, so the story was it was this French brand that was known as kind of like I guess a luxury tennis apparel brand. And it got acquired. And the acquired just caught the price and tried to just increase the distribution. And then as a result, it became worth a lot less because they were selling more but far lower prices. And then the original owners bought it back and kind of returned it to the luxury status it had. But then you also talked about Nike. And so with Nike, with the footlocker distribution, Nike had said, hey look, here's the, these are the terms. If you want to sell our shoes, you have to position them as high quality goods. And so the paper was from 2007. But Nike had the more recent experience, I guess in the last two years, they kind of shifted into D to C. And then they started relying more on just like direct response advertising and then pushing promotions to make that more effective. And that I think diluted the brand, right? Made a lot more accessible. And then they ran into trouble with that. And I think they brought in a new CMI. I think they brought in a new CEO if I remember correctly. But they had to kind of revert strategies there. Just maybe if you could just kind of expand upon those. I just thought those two examples were really interesting, especially the Nike one with the more recent experience of having kind of taken the opposite approach is what they did as what you depicted in a paper. - So Nike's challenges have evolved somewhat. But let me just back up briefly and why did I bring up those specific examples? There's always some conflict with champ. In the case of Nike and the paper, the channel wanted very liberal terms on credit. From Nike and Nike didn't want to give liberal terms a credit. So these conflicts are all about, how do I allocate sort of the value I created in the brands I built? So create the value and how do I allocate? And to me, this is all a function of who owns the customer. So if the customer is loyal to the retailer, or is the customer loyal to the brand? And in fact, one of the very reasons we see Shopify it gains so much market share in E-Com relative to Amazon is because Shopify sits on the backend, Amazon sits on the front, Amazon controls the customer experience, doesn't give a company a chance to do any branding or brand building. Unlike if I use the backend tools at Shopify. So Nike was extremely successful in owning the customer. Fast forward to now, it's a different world. The barriers to entry of DTC are pretty low. And so you have all sorts of fragmentation and ankle biting. So Nike's a really good brand. It has a great reputation, but maybe it isn't the single best brand for bike shoes. And so the fact that I've got more DTC and ankle bitters means it's like you have to fight a little bit harder and think about how to own the customer and how that affects their advertising. I think that also the poster child recently for all of this is the acquisition of crap tights by private equity. Ross, I don't know if you've been part of the measurement in that space, but they went big on controlling costs and part of that was cutting advertising. And it's been a really rough time for craft tights. So you can't cost cut your way to excellence, to be honest. You have to invest in build brands. Oh, that hits the cross-synatic long-winded. - No, no, that's great. And I guess I feel like the private equity would be especially bad at that at operating a brand centric company because the whole goal of private equity is to buy an undervalued asset, kind of integrate some efficiencies and then flip it in a few years, right? Well, if you look at ad spend and even if you've got these kind of provable results of like, look, when we support these brand initiatives, they drive real value. Yeah, fine, they drive real long-term value. We're not gonna be the owners in the long term, right? - Yeah, by the way, that's one reason they'd handle us. Some analysts, I don't know if you've worked in any of us. Some analysts have started adopting these measures too as they evaluate companies. - Yeah, I did work with some people at Vanderbilt that were helping out Wall Street firms evaluate mergers and acquisitions and using cross price elasticity. See what happened if these companies came together? Would they be able to raise prices more? I did read that in that. - Ross, I wanted to switch gears into measurement. What are the differences in approaches to marketing measurement that you see across verticals for successful companies? So how does effective marketing measurement differ for instance e-commerce retailers versus app developers versus large CPG brands? Or does it? - Okay, well, we haven't worked with every single vertical, but we have worked, I mean, between my first company, marketing analytics, then when I was at Nielsen and then I'll hear market innovation, we have covered a lot of verticals, including gaming. Like you mentioned, there we've done that. There's not many verticals that we haven't done. I think some B2B things we haven't done probably because that's a little bit, that's kind of different. That's what like CRM, which is not really our specialty. But I would say, so how do they differ? Well, first of all, the KPIs. So, we're model sales and some verticals, we're model downloads for some loan applications. Sometimes we also, we made model different portions of the funnel, of the purchase funnel. So we may, so for a bank, we might, you know, model loan applications and then loans given out or something like that, there's different levels of the whole funnel we can measure. The data differs e-commerce, retail, they generally have better data and they have sales at the consumer level. If you have, if you know who your consumers are, you can do a lot better stuff with modeling and yes, because you know, you know, sales by consumer and you don't always, not necessarily, like e-commerce, oh yeah, they would know everybody they sell to, they don't necessarily know a lot about them. They may sell through some sort of affiliate and then not know everything about the who it went to. But so the data differs and then the methodology differs. So if you do have consumer level information, like randomized control trials are the gold standard in measurement of pretty much everything in medicine and in marketing and all kinds of stuff, they are, they are the gold standard and you can make sure you don't have any biases with them. But it's a lot easier to do if you do sell directly to consumers. So that's easy to do if you're e-commerce or you're a retailer. We've done some RCT, random X-Pro trials that were, I thought were super interesting. This was in CPG. We had a 10 million household panel basically that we could send ads to and see how they behave. So 10 million households, something like 30 million people, you know, 100 million devices or something, it was, it was a rather, large experimental lab that we had. It was super awesome. We couldn't measure as many campaigns at the same time as we can do with marketing mix. We can only do like maybe six at a time. But it was so expensive because we had to pay an identity management firm. If you're a retailer or you're a e-commerce, you might not have to do that. So that's where practical in those verticals, B2B, like I say, generally doesn't have enough volume. You can't just sell like one or two giant industrial machines a year and do some kind of model on that. That's kind of hard to do. It needs some kind of volume. B is kind of a load. And then long term effects, you know, this podcast is kind of focused on long term effects. That also, your ability to do that difference by vertical quite a bit. And well, even even just any kind of measurement for, let's say, for automobiles. That has to, you can't automobiles have such a large long-purchase cycle that it's really difficult to model sales with marketing mix model for cars. So we generally model something like an attitude like purchase intent or some kind of thing like that. There are some products that have a very long purchase cycle like mattresses that we are able to model. I don't know if it's because they tend to have really strong holidays. So like, you know, it's going to be a few periods where people are going to buy a bunch of mattresses. They're president's day, like right now, Labor Day, those are big seasons for that. So you can imagine you can see the effect of re-avortizing in some sort of short period of time. So I'd say that those are differences across there. And then as far as like, you know, really successful, you know, so many successful marketers that we work with, I think an e-commerce company that we're working with is like, really, I think they're one of our more successful clients. And I'll say, here's some things about them that I think help, well, to quantify what they are, what they're doing, they went from about 250 million sales to about 550 million over the time that we've worked with them, more than double sales. Their CMO is a super awesome guy, crack the saw up. And he is always pushing us to push the measurement envelope forward. He had us doing our first share of search models that were popularized by a guy named Les Benets. I think he's in the former UK, but share of search is an interesting thing. It's a potential long time effects model, although I think it has some pros and cons for that. They also pushed us to do our first affiliate marketing. That's an important thing for e-commerce where you sell through affiliates. You give them like a rather potentially large commission, like 20%, 30%, or 2%, depending on what kind of affiliate it is. So, but so modeling that and like how much extra value would I have sold without that affiliate referring it to me, you know, you prefer not to give them 20%. And then working tightly with their media guy, we turn things around really fast for them. I just love the way these guys are pushing everything. So for them, after each of their major holidays, we will get the model turn around in five weeks. So five weeks after the holiday ends, and the Sunday of the holiday or whatever, the Monday of the holiday, five weeks after that, we're done. We've collected all their data, we're going to meet a guy, we've run the model, we've given the results, put it in optimization, recommend it what they do. And we're done, and we're ready for the next holiday. So I think guys like that that really push, you know, try new things, push the envelope, always, you know, I think those guys get good results. It's good to be good at measurement. If you're swimming in dashboards, but still arguing about what actually drove installs, this is for you. Branch is an AI-powered MMP built for growth marketers who care about signal quality and outcomes, not just reports. You can quickly answer questions like, how is my TikTok spend really performing? Or which partners are driving net new users and even launch campaigns that move users from offline to app without breaking attribution? Branch's AI proactively surfaces what's working, flags issues early, and takes care of the busy work, like link creation and tagging, so you can move faster and spend smarter. Learn more at branch.io. That's branch.io. So next question is for you both, maybe I'll let Carl start and then Ross you can follow up. So let me outline the ways to do it and then the pros and cons. I think Ross will probably take a dive into some applications, but three basic approaches are using surveys to measure brand attitudes, brand associations, sales-based measures, like baseline drivers, as we discussed earlier. And then you know, some people use stock returns, Tobin's Q. Specifically, Tobin's Q is a measure of market value relative to assets. The idea is the market value is higher than the assets, some of that is related to brain strength. Surveys are easy to measure, but they're often a weak link to purchase. So for example, purchasing tent, R squares on purchases are not high, and you have serious response bias, because you know, who has time to fill in surveys. I don't know about you, but I haven't done many surveys in a long time. Sales? If the marketing makes type things, that Ross is talking about, easy is state of that or typically part of the standard tech stack, mart tech stack of a company, it's behavioral, it's field-based. And so it's a really nice tool to get a read on how a brand is trending. The downside of course is it's difficult to link directly to profits, which is sort of the key financial KPI. I mean, it can be done. The stock price, it does go into firm value, which is again, which you need to raise capital and sort of its fan game. But it's a bit of a challenge to spend to stock price for a couple reasons. First, stock prices are reported to the branch. So I can't figure stuff going out in the brand level. The second thing is costs, many costs are under the marketer's control, like cost of goods. And you know, what is the result you're looking for? Ironically, if you have a null effect, then you're at optimal spend, because you're spending too little, you should spend more raised profits. If you're spending too much, you spend less to raise profits. So if I'm looking at a direct link between spend and profits, the only way you know you're doing it properly is a null result. In, of course, null results have all sorts of problems, not the least of which being you can't really test the null result because of power issues. So those are sort of three, and the pros and cons, Ross? Yeah, no, Carl outlined them. I think there were attempts at measuring long-term effects that go way back. I came across a paper just in the last few days when I was talking to my team about getting updates from them about what they thought about long-term effects models of work that didn't work. They told me about one way back from 1978. It was called assessing the long-term value of advertising. I'm not sure if you ever heard of that one Carl. From some guy in Nairimum, Dala from J. U. Walter Thompson, from 1978, they told me about this one. And it's been going back. I don't really think that one's super promising. So the different methods we've tried, brand awareness, we tried using that. So let's go survey method. It had a nice, we worked with Millward Brown together on that, on that part of Cantar. We got a base awareness that was served as a long-term effect and it was big. And so that was good. It was good about it. But it is survey based. It's a relatively small sample. People also get confused with awareness data. People get confused about brands that they've seen. So you'll ask them, you wear this brand or something or you wear this brand's advertising. And they will say they've seen ads that weren't even running at that time. But then what we find out it is, it's another brand that's kind of similar to it that they got confused about. So things like that make it survey data, as Carl said. It's like, there's biases in it, selection biases and other things. And there's people just don't remember stuff exactly. And so that's when kind of that one. We've had other survey data. There's a thing called brand asset valueator that's interesting data. It goes back a long time. It's got a lot of brands in it. We use that data, it's perception data. We thought it got decent results. Senior management wasn't convinced that the survey data is going to let us double all the ROI. I mean, the general rule of thumb is what you measure in a market makes models like the effect on sales in the first year maybe. And then you should double it to get the total effect, something like that. And so the two acts multiple, whatever we got at that study, the senior management didn't take much action as my recollection. We've also tried other behavioral metrics like share of search. That was the one that was popularized by Les Penais. His first stuff was on using durables, I think it was phones and something else. And it looked really cool. And I do think that they would, it's a cool model to use. But it's not clear to us that branded what we're talking about is branded organic search. You're searching specifically for some sort of keyword that has the brand's name in it. You would like to, you're advertising the agenda to drive people to search for your brand, learn about stuff like that. So that's why branded organic search is a good thing to drive. But anyway, it's not clear that that's a good indication of long term, brand loyalty or anything and things like CPG or soap or toothpaste or something like that. But I believe it in durables. Nielsen used a states-based model for a while that was based on new and repeat buyers. That sounds quite promising to me. If you contract somebody, like if you're a phone company, a telecom company and you've got somebody that's a customer of several years, I think that could be promising if you watched somebody over a few years. I haven't personally done that one with a phone company. But there's been several attempts at using that, like tracking buyers over time. See how many are new, how many are repeat and trying to have a model, trying to predict why, how do I get new customers that then stick around for a long time. Those are very promising. Another one was done by NCS, that's Nielsen. That's an alliance. Nielsen have with a company called Catalina Solutions. It doesn't do this anymore. But a woman named Leslie Wood, she's done a lot of cool stuff. She did a household panel, long-term effects model that had new and repeat buyers. Also, the problem with that is in CPG and you have a panel. There's a lot of churn in these panels. Like after a year, so many of the panelists have churned over that she could really not measure a long-term effect for longer than one year. That's not really what we call love effects. We generally call them long-term effects after that. We've used a lot of them and I will say that none of them are perfect. It's a hard thing to measure. It should affect on your sales. Again, some of these things I thought at first when I heard them is like, I don't think it can be done. Trying to measure your effect on your sales two years out or three years out. But you can't get decent estimates. The best model that we've done, and let you say, no, no, perfect. But the best one, in terms of getting intuitive results consistently, they're accepted by the client, and that the client comes back and buys from us again. I like to read pre-projects as much as the next guy, is Carlos, two-part model, that one that I guess he did outline the kind of basics of an HBR article. But then later on, he develops some methods to try and estimate it. So we do two parts of it. We estimate the impact of your advertising on base core sales. Like, on a quarterly, we do quarterly data. So we wash out the spikes, and we just want to know total volume, especially core volume that you're driving, and long terms, we model that, and then we model your base elasticity, and see what happens. How does your media, how do your promotions affect those things? The core volume, do you have any giant promotional spikes that drive down the volume between the promotions that you're selling at full price? Is that what you're doing? How you affecting price less is you. Nobody wants a high price elasticity. That helps you cut price. People want a low price less, so you can raise it. So raise price. So that's a really good model. We've done that model for 16 years when I first started working with Carl was a long time ago. I guess it was 2010. I brought it into Nielsen when Nielsen acquired us. They're still using it to this day to my knowledge. No other long-term effects model that we've ever used. We've used a bunch of them. I listed some, we've used more than that, too, I think. None of them lasted with more than two years with us, anyway. Maybe with somebody else, but both us, we went away. So this is the only long-term effects model that I've had a long-term relationship with. So I thank Carl for that. And then I think I'll say that one thing I'll say about Carl's model is not all verticals support the price elasticity part of it. Sometimes we'll be in a vertical, but you just can't get good price elasticity. So we just do the core base volume part of it, and that's good enough. And that is the key. It's a modeling sale. So there's a lot of long-term effects models for your modeling. So I'm kind of attitude or something else that management doesn't really believe that that's connected to sales. Model sales, model base core sales. Everybody can agree that that's a good thing to do. And I think that's part of what works really well. Yeah, I remember that. I remember Ross coming down to Duke 2010. Remember the time we were spending on the whiteboard? [LAUGHS] Oh, yeah, that's some great ideas. Ross mentioned Les Bene for those who are not familiar with them. He probably should look them up. He said, "I con in the advertising industry is a stalwart on protecting Brandon in the advertising mix. And I know he's done a lot of work in the space of brand management based on sales." Now just saying Les Bene is the way he talks when he did this thing about share of search. It was during COVID. And I remember watching it just singing off. So relaxing and nice. It's just a great speaker, nice guy. [LAUGHS] Yeah, so Ross, that was a great segue to the next question. I know we're just at time, but maybe I'll try to squeeze this last question in. So Carl, we're kind of talking in the context of like a lot of brands that are sort of established. They've got long histories, maybe some like Coca-Cola going back one of those centuries. What tools does the CMO have for like a nascent brand? So imagine like a brand, a D to C brand that's just starting up, right? What tools does the CMO have with a nascent brand to make the case for investing in this? To investing in brand advertising to build these kind of like long-term brand effects? How would they make the case to like the CEO or like their investors that this is something that should be invested into? So I had one thought on that. Ross had one thought on that. I'll share my thought on that and jump to Ross. But analogs are really important here. The reality is you can't make up data. Remember, it takes five years or more of data to really truly understand one from in fact you could think of this in terms of data variation. How many long-term cycles of spend can you really have over a very short period of quarter? You can't. So if this stops from being measured in quarterly variation baselines, I mean to have any statistical power you need enough quarters really to link long-term strategy to quarterly baselines. But there are analogous brands and channels. So one of the really interesting discussions that you've probably filled it on your show or you may soon is with ads woven into generative AI, what's their performance going to be relevant to search engine marketing? Right. And then you start thinking, well, are these analogous? You know, and they should be in any way. Like the SEO tools are same. I mean, I still have to have my website be discoverable. All those tools we did to get higher SEO and better SEM tend to apply in this market. So it might be reasonable to say, as an initial guess, I can expect an analog. The same would be true for new social channels with existing social channels adjusted for demographic and those sorts of things. But I want to call it shit. You can't think of a brand new channel as brand. You can't prove a brand new channels brand building until you have an updated approved. That is brand building. You might have good faith or good reason analogs might be part of that. And Ross, like I think you have some experience again with Lotus's old narrative rules around this. Yeah. I mean, that the original two X thing came from, I meant to mention this to an earlier discussion. But there was that 1978 paper that I only heard about until recently. But there's a classic paper that was done in '95 by Lynn Lotus, G. Day Abraham, Bruce Richardson, a couple other people from IRI and Warden. It was a summary of 55 in market experimental SMS along to effective TV advertising. That was experiments like using behavior scan markets. Those were IRI now called to kind of a way back when they had small cities, peory or something. Little kind of cities that were kind of isolated from other media markets. And then they would run some ads in one city and then do something else that they are within the same city. I think that's split cable. That's right. Within the same city, they had some people saw one ad, some people saw another ad or didn't see an ad. And they did that and then watched for like two or two years out. They watched for two years out, I think. So that was a very interesting thing. That's where they got the 2x multiply. First thing I said, the total effect on that is generally twice the measurement you get in the short term. That rule of thumb, all kinds of long term effects, we all go back and make sure we're somewhere near to. Otherwise, we're probably doing it wrong. But yes, I agree with Carl. That for a brand new brand, you probably should just use the 2x thing. There have been some attempts to do like Nielsen after we brought Carl's model to Nielsen. Some other guy that came in with my team to Nielsen. He took that model after I left and then tried to make it so it could work. Because like with Carl's model, we generally say we have to be at the client for three years because what we get there and then we have two years of back data. And then once you've been there for three years, we have five years of data. Then we can use them. Carl's model works pretty good with five years of data. And so that's what we'll say. But you know, that's like kind of hard to sell, especially if it's a new client coming in. So I can understand why Nielsen tried hard to make it, tried to work for a new brand. And what they did was they took ad stock. It just kind of extended ad stock way out. That's been done by some ad stock is for anybody. Probably most people listen to this probably know what ad stock is. But ad stock is you take GRPs or impressions and then you kind of do exponential smoothing. I mean, just kind of smooth them out into the future and then you use that as your variable. And then through that, you get an effect going on in the future. You try different decay rates. It's pretty much the same thing as exponential smoothing except it has some kind of sad, different people, different companies have different kinds of proprietary or maybe they admit it, different kind of saturation estimates. They lay on top of the on top of the smoothing thing. But trying to get a new brand and just doing ad stock to go out in the future, it doesn't work too well because it makes all the, you spend it out so far in the future. It just looks like a pancake and then it's highly correlated with the intercept itself. It doesn't work good. The one thing that we did do that was for for a new brand carl and I did find this one out just a couple days ago was we actually knew about it, but I didn't really think how that is pretty good. It was for in and pharma are actually we do and we're doing pharma or OTC over in Europe. And over there, they have a mixture of the things that we buy as OTC over there. Some of its pharma and some of its OTC, but they have a measure that we could be getting over there from some market research firms called the Caverage Weekly Recommendations. So we advertise to the healthcare professional and then we get data on how much they're recommending that product to their patients. And so I say even some OTC things like nicotine replacement therapy, that is you go to a doctor over there to get a recommendation. And so we'll model AWRs and then we'll model the effect of AWRs on sales so we get kind of two kind of long things. Effect of marketing on AWRs, that has a big lag up the lag in it. The effect of AWRs on sales, that has another lag. We get out about a year maybe in terms of the effect, but like I said, it's about the cut off of where you might call it long term. So bottom line, I agree with Charles, I go with Carl that benchmarks are probably the best thing. There's a couple of things that are decent. Yeah, Russ, I like your example. That's actually called the surrogate variable technique. You know, buy a bunch of academics. The problem of your, your office is fantastic. The problem is the assumption in these models is that this intermediate measure of the surrogate, fully mediates the effect of advertising on future outcomes, which just is insensible in most applications, but not yours. It's actually really an application. Carl Russ, this was a fantastic episode. I learned a lot. I imagine everyone listening did too. How can people follow you? How can they consume your content? How can they find you on the internet? I'm at www.marketingattribution.com. Yeah. And thanks for having us. It's great to inject long term back in this. Again, I'd love that it's top of people's minds. In terms of my case, I like to say that if you're going to have a kid, think about how to name them and that people can search them and they're the only result that shows up. Like to my knowledge, there's maybe five Carl Mewis in the entire world. So to find these pretty easy in a Google search, just you know, enter and find name and it'll pop. And it brands are built over years, why they manage up a quarter is just to a simple search on that will pop up at HBR. I think that is a really good starting place for a lot of people. If they want to take a read, what's going on here? So thanks again. Yeah, thank you. Eric. No, you're just just pointing out to people that that is a HBR paywall gated article, but the PDF can be found through Google search. So they can find it that way. I really appreciate both your time. Thank you very much. And enjoy your weekend. Thank you. (upbeat music) (upbeat music)