Season 7, Episode 26: Demystifying open source MMM (with Michael Kaminsky)

AI is changing everything, but it's only as good as the signals behind it. Branch connects customer interactions across paid, organic, offline, email, web, and app touchpoints, and turns them into the trusted context you need, with links and attribution that capture the full user journey. Learn more at branch.io. And while you're there, check out Branch's AI Search and Discovery report, covering insights from more than 300 enterprise marketing, growth, and digital leaders to understand how the industry is responding to the rise of AI search. That's branch.io. The problem is that the distinction needs to be drawn between the components of the economists and the correctness of their analysis. Welcome to the mobile dev memo podcast. I'm your host, Eric Sufert, and I'm joined today by Michael Kaminsky. Michael, welcome back to the podcast. Eric, thanks for having me. Happy to be here. So the occasion for us speaking, although I've been intending to bring you back on, the occasion for us speaking was an article that I shared in the mobile dev memo Slack recently that was kind of questioning whether open source MMMs and in particular Robin and Rideon were allowing meta and Google to grade their own homework. And so this kind of, I don't know, catalyzed a conversation. I just thought it'd be interesting to get you on to get your take on that, but also to talk all things. Let's call it open measurement. Before we get it to that, could you please reintroduce yourself to the audience? Yeah, absolutely. So I'm co-founder of CoCO, this company, ReCAST. We do marketing measurement, planning, analysis, including media mix modeling. I have a background in statistics, causal inference, econometrics, but I've been really doing marketing science, marketing measurement for the last 10 or 15 years. Okay, got it. That was very efficient. Okay, so let's talk about Robin and Meridian because that was the source of this article. And I'll link to it in the show notes so people can go read it if they want to get more context before listening. But what does it mean for an MMM to be open source? Why do you think some people view Robin or Meridian as an attempt by meta or Google to grade their own homework? And why is that viewed flawed? I mean, that was the point I made in the mobile dev memo chat just to kind of not bury the lead here. That's a flawed view in my mind. It doesn't allow these companies to grade their own homework. There's no baked in preference for meta or Google and there's no way for these open source libraries to artificially inflate purposefully, deliberately artificially inflate the performance metrics of meta and Google. But why do you think that view persists? And what does it mean for an MMM to be open source? Yeah, I think the question's one at a time. So why does the view persist? I think it's reasonable and justified for lots of people to be skeptical when these sorts of ad platforms propose measurement methods. Right? They are obviously sort of deeply and broadly incentivized to try to make their own platforms look good. So that is very reasonable to me. I don't think it applies in this case. I agree with you. But if you log into meta's ad platform, you should always be asking yourself, like looking at their reporting, how is this biased? Why is it biased? Meta is obviously very incentivized to make their own ads look good. Google is as well. That should be baked into how you're thinking about looking at the reports coming out of any of these platforms. And so it's reasonable for a lot of people to be generally skeptical. And those who don't understand how open source software works or how media mix models, MMMs work, they probably just apply that same skepticism without thinking deeply about what's actually going on under the hood here. So what does it mean for an MMM to be open sourced? What does it mean for any software to be open sourced? What it means is that all of the underlying code is available for anyone to look at, read, and in the case of both Meridian and Robin, download, change, use, and even resell based on the open source license that they have. So that is to say that I can download all of the underlying source code for both Robin and Meridian. I can read every single line of it. I can change that source code. I can run it on my own data and I can see how the results come out. Being open source means that anyone in the world can do that. And the reason why I think Meta and Google wanted to open source these tools is because they wanted to give that extra layer of transparency to the world to be able to say, look, we are not biasing this towards our own channel. You can look at the code under the hood and confirm that for yourself. That is not something that you are able to do with their default ad platform reporting based on lookbait windows or even based on their quote unquote incrementality driven conversion optimization. We can't see any of that underlying source code, but for Robin and Meridian, we can't. Yeah, that is what strikes me as so odd in making this particular accusation because it is entirely verifiable. By nature of being open source, you can look through everything. The actual platform based attribution tools are not. If you are going to make that claim, would you make it against the opaque platform attribution tools? Why would you make that claim against the open source library where all the code is just available to you to bet? I fully agree. I mean, again, it gets eyeballs. It gets people talking. This just feels like a thing that someone wanted to publish to get attention, which works pretty well in the attention economy of today. I want to kind of just hover on two aspects of this. So the code is visible to you. There are no starting weights here. It's not like sharing. It's not like an open weights model. An open weights model, the model is trained. This is just the model framework. You train it on your own data. There are no starting weights that you might apply to your own analysis out of the box. This is a model that you run on your data. It's meant to be trained on your data to start with. So there's no way for them to bias or inflate the kind of performance baseline of their own channels, because that's just not how the models work. They're like a blank slate. It's just a code. That's exactly right. You're right. There's no starting point. There's no priors here. It's meant to be run on your data. There's nothing to it until you run on your data. But you could just clone the repo, tell codex, "Hey, could you just look through this repo and see if there's anything in here that would bias meta performance?" That would be a very sensible thing to do. You should do that with any repo, to be honest. Look for vulnerabilities or anything like that. That would take five minutes. This is not some big mystery here. There's nothing that we need to uncover. We're not necessarily arguing in the abstract. You could just do that. I would, again, recommend that people do that. There's other very simple tests that you can run. One that I often suggest is run the model with Facebook, Activity Labelled, and Google Activity Labelled, Run it Once, and then run it again and just switch the names. You'll see that it's no different. It's not keyed off of the names or off of anything special about any of the different marketing channels. You can split the marketing channels any way you want. You can give them all different names. The results will not change. It's very unclear to me how you would imagine that there could be some inherent bias in these models when you can do these very easy tests to demonstrate that that's not true. I just said that we're biased, but I've been trying to not say that we're biased because there's two interpretations of that. You could think of bias as being some inherent prejudice against something that would be more like a cognitive flaw. Or you could think of a bias in the context of machine learning. The bias variance you trade off. Like bias meaning the risk of underfitting and variance meaning the risk of overfitting. There, I think you could make the case actually. Here's where I'll kind of contract myself a little bit. It's not about meta or Google per se, but it is a reason I do think why they did release these as open source because I do think that absence, this kind of framework, absence of probabilistic attribution solution, you might actually underfund a meta or a Google. They might get less credit than they otherwise would. You might be over crediting things like linear TV or radio or out of home. If you are primarily a digital property and you're doing those legacy, you're investing in those legacy channels, you might actually be under attributing these primary channels that are driving a lot of top of the funnel awareness, even if they're not necessarily capturing every click. I do think that was their motivation here. It was like, "Well, we think you're undercounting. We think you're undervaluing our contribution." If we give you this tool, it'll give you the ability to see that actually meta or Google or whatever is generating more value for you than what your measurement apparatus is telling you at the moment. Is that a reasonable assertion here or is that a weird thing to say? I think it's a little bit more complex than that. Let me talk through what I think is actually going on here. I think Meta and Google are great advertising products, probably the best ever. They are very, very good. I think Meta and Google, what they found was that a lot of their advertisers, the brands that advertise with them, were using other types of media mix models, either ones that they were managing in-house or ones that were run by other big consulting firms or even their agency. A lot of those other media mix models are, in fact, or have been historically structurally biased against digital channels. would do things to put their thumb on the the scale for the model to make specifically channels like TV look good. There are a variety of reasons for that. A lot of agencies historically made a lot of their money, specifically from TV advertising. That was a cash cow for them. They wanted to make it look good. A lot of CMOs and even higher ups in marketing at the brands themselves felt strongly that they wanted to do a lot of brand advertising because it's fun and it's more interesting than a lot of the conversion-based advertising. I think Facebook and Google felt like, look, these MMM results keep coming back, saying that Facebook and Google are really bad. We feel quite strongly that they're good. We think that it's because the MMMs are biased against us. They wanted to release these open-source MMM products as a counterweight to what they were seeing coming out of these other MMMs that in fact were structurally a biased against digital channels because the analyst or whoever was managing them was putting their thumb on the scale for the channels that they wanted to look good for a bunch of different reasons not actually related to performance. When I think about why are they doing this, it's more about that reason combating other MMMs, which are actually of even lower quality, versus necessarily trying to give a better way of measuring versus a digital tracking type measurement. Right. Yes, I think that I would be much more inclined to trust one of these tools than to trust the output of an agency's tool because the agency really is incentivized to have you think certain ways because they're taking a cut of spend. Does there fee? Yeah, I think that's exactly right. I mean, again, there's lots of problems with these open-source MMMs and I'm happy to talk about what those problems are, but versus some random regression that some analyst at a random consulting firm or a random agency is running, the open-source MMMs are going to be way, way, way better than that. And again, more inspectable, more interpretable, especially in the world of AI and LLM's, have codecs, have clawed, look at it, have it pressure tested, that's just going to be a much better experience for most brands. Well, so let's walk me through. Like, what do you think the flaws are? What do you think the shortcomings are of these open-source MMMs? So I think MMMs, in general, like the default assumption should be that they're wrong. The problem that an MMM is trying to solve of looking at aggregate data and sort of variations in that data and then trying to attribute causality to that is just an like almost impossibly difficult problem. It is so, so, so difficult. There are so many ways for it to go wrong. Easy ways to demonstrate this to yourself are to simulate data that you think is realistic based on how you believe marketing works in the real world. Run that through any MMM and then see if you get the correct results back that matched your simulation. When you make sort of very obvious assumptions and the way that you simulate the data, for example, that marketing performance can change over time as creative changes or as the market changes that, you know, there's a lot of correlation between your different marketing channels, that there's seasonality. You run the data that you simulated so you know what the truth is. You know, truly how effective every marketing channel in the mix is, you run it through the MMMs and you get results back that don't match that at all. And so it's just like obviously on its face, the MMM isn't working, right? It is not able to find the true causal signal in that data. And that is the fundamental problems with MMMs. And I think people, you know, there was a, over the last couple of years, people got really excited about marketing mixed models, especially because these open source tools were released. And I think as people are using them now today, the shine is starting to wear off because people run their data through the model and they run it one week and then they run it the next week and the results totally change or the results just like don't make any sense. They're clearly contradicted by well run incrementality experiments, either geo-lift or user level ones. And so it's just like, well, these tools just clearly aren't delivering on the promise that people think that they're making, right? This idea that we can connect or generate a causal relationship or identify a causal relationship between the marketing activity on one side and the business outcome on the other. Right. And that's really the fundamental issue. It's just, you know, people misinterpreting what these tools are supposed to be able to tell them or what they do tell them and assuming it's some kind of replacement for attribution when it's not. I think that's exactly right. And I think to hear this, some of Vinder in the space, like we sell a marketing mixed model, but most businesses do not need an MMM and in fact an MMM will be net negative for them because it will actively mislead them. Whereas, look, last touch attribution has lots of problems. I'm very familiar with them, but for a lot of businesses, it's way better than anything else that they have. And an MMM would actually be worse for them than just following the last touch attribution. Oh, wow. That's a spicy take. We got to let that hang in the air for a second. That's a spicy one. I mean, look, you can generate data that shows this. Again, if you imagine that, you know, most MMMs are wrong and will actively mislead you because the model is just incorrect. It's just learning the wrong things from the data. Then like, it is a net negative. Yeah. And I think last touch attribution, again, has a ton of flaws, but especially for smaller businesses that are really performance marketing driven, it works really well. And there's been a couple of studies that have been released recently. I'll see if I can track down the link to share with you so you can put it in the show notes. That showed that last touch attribution was like 85% is good as like only doing incrementality experiments. Like, that's great. Like that's awesome. That's an amazing tool to have at your disposal. And I've seen this over and over again. Like I work with, you know, lots of very small businesses, just like personally, that it's very clear that they spend money on Facebook, they spend money on Google, they get a lot more conversions. And you can sort of see it just by eyeballing the data. And that lines up really well with what you see in last touch attribution. That makes a lot of sense. Last touch attribution starts to fall down when you have more complex mixes, when you have much larger media budgets. But again, an average MMM is actually going to be net negative. And so it's like, it's worse to have the MMM than not to have it. Mobile game developers no longer need to pay up to 30% in major App Store fees. With XOla Web Shop, you can create a direct storefront, cut fees down to as low as 5% and keep players engaged with bundles, rewards, and analytics. And we're today at XOla.com. That's xsolla.com or use the link in the episode show notes. Do you remember that fear that erupted on LinkedIn a while back about last touch? And I had written something, I can't remember what the title was, there's something like, you know, the need for common sense and digital advertising. And my point was like, yeah, okay, fine, last touch is not perfect. But if you're spending your first dollar, are you really concerned about the causality of the conversion? Like if you're launching a product and spending your first dollar on digital advertising, are you really worried about misattrafuting that like, is that really a concern? Is that really something that you should be investing time and money into in terms of doing a holdout or just some sort of like deeper incrementality? And I was like, is that really a pressing concern for you? And then, okay, well, if it's not and you say like categorically it's not. And then like, okay, then there's a, now we start talking about this second dimension here of just common sense of realistically, how likely is it that I'm misattributing the effects, right? And my dollar spent is not truly incremental. And that gets like to your point, that gets more realistic as you scale spend and you've got word of mouth and you develop a brand and you develop loyal customers and you start diversifying your mix and you start, you start having multiple digital channels being run at the same time. But even if you're spending, let's say you're spending a million a month only on Facebook, do you really need to do some disruptive incrementality study where you just shut off and, you know, you do, and like you might say, yeah, you do need to do that. A million is a number that necessitates that and okay, fine, we could, we could find, we could probably find a number that we both agree on, but the broader point is like, there's that second dimension of just like, well, I've got to use some common sense here, like how likely is it that there are competing claims for these effects? At the first dollar of spend, maybe we both agree they're none and I think most people would agree they're none. And then it's like, well, then what's the number? What's the dollar number? Yeah, I think that's right. And I think, you know, every business needs to sit down and think about this, but I think for spending a million dollars a month mostly on Facebook, it's like probably your whole business is driven by Facebook. That's probably 100% incremental and there's like not very much baseline. So that sort of thing can push people towards doing some sort of incrementality experiment, you know, again, shut off or spend up additional dollars in some number of geographies to try to get some signal. Like how far off are we? Are we off by 5% or 10% or more than 20% and I give it's more than 20%? Okay, yeah, let's like dig in and try to figure out how much and if there's, you know, what the size of the pie here is, but if it's like 5%, does it matter? Like is it worth spending a bunch of time and energy on this? Probably not. And again, like MMM is even worse because it's so hard to validate if an MMM is correct or not. You can always run an MMM and always get some number out on the other side. But how do you know if it's right? That's the hard problem that is like actually very difficult to answer and especially it's very difficult for like a marketer to be able to answer. And so is that actually adding any signal? Is that actually helping you make the next decision? In most cases, no. And so that's why I look at people who are again, these like fairly small companies running these MMMs and they're getting what effectively is just a random number generator out on the other side and they're frustrated by that and they're like, this doesn't make any sense. And I'm like, of course, it doesn't make any sense. You haven't thought about how to evaluate if the results of this model are even usable or not. Right. Right. And I want to make something clear. I'm not claiming that no one should ever run an incrementality test. I want to make sure that it's very clear. Income mentality, yes, it's a gold standard. That's the word I'm supposed to use. It's the gold standard. You know, you should be doing a commentality testing. It's important. My point is more about scale and when that becomes mission critical. For some companies, of course, it's mission critical. You have to be doing this necessity. But my point is more about the scale question and relating back to like when and when and then actually might be net negative in terms of measurement value. Okay. So that raises a good point. So there's these flaws in the open source approach and the flaws in the open source call it the interpretation or the utilization of these open source models. How should a marketing team evaluate an open source element? Like what are they looking for? They all roughly, you know, are they just fungable? Are they interchangeable commodities? So the answer is no. And the way that you know that the answer is no is again, you take your data, you run it through two different MMM. So whether they're open source MMM's or even what like meridian with two different settings, right? You can change some of the settings and you can run it. And you will see that you will get two very different sets of results generally. Run one will tell you Facebook's your best channel. You spend a ton of money on Facebook. Run two will say, take talk is your best channel and Facebook is terrible. So immediately you sort of know that like, okay, this isn't really reliable. Like we made one small Arborshire change to the model configuration or we switched between two different libraries that supposedly do the same thing and we got totally completely different results. So the first thing that any team should be thinking about is they're thinking about whether or not they should use an MMM or which one they should use. And this is true of whether it's an open source approach or working with some other vendor is how are we going to evaluate if the model is corrected? Everything else doesn't matter if the model isn't correct. If the model is going to mislead us and tell us that a channel is good when it's actually bad and vice versa, then the shiny reports, the dashboards don't matter, nothing else matters because it's wrong. So it can only be valuable if we think that the model is actually right. And so every brand needs to think about how are we actually going to evaluate that before they start on any MMM project at all. I think that there's a couple of ways to think about evaluating an MMM. The first I mentioned earlier, which is parameter recovery, we're going to simulate data where we know what's true based on what we believe about how our business works and we're going to see if the MMM can get the right answer. That's a good way to check if this could even work at all. So that's check one. Check two is being able to predict the future. Can we train this model up to two months ago and then have it predict what's going to happen over the next two months? That's a second way to evaluate, okay, could this plausibly work? That is a better check if you're actually changing your marketing budget over the last two months, right? Because what you want to know is can this model predict what is going to happen as we make changes to our marketing budget? As we spend up on Facebook or spend up on TV or whatever, we want to know that the model can predict what's actually going to happen. That's what it means to have a causal understanding of the data. And then the last one is verifying the model with lift tests and experiments, right? Can we say, hey, this model says that TV is really good if we go run a lift test, does it actually line up with what the model is saying? Or historically, we have a bunch of lift tests, does the model get that right? Those are the ways to actually think about evaluating an MMM and those are what you need to be doing before you embark on this journey. Again, whether it's open source or whether it's working with a partner. Right. And so how should a company approach on boarding the tool? So again, I mean, I would say like make the checklist and start running these checks, start by simulating data. How do you think your business actually works? How do you think marketing actually works in your business? How long are the time shifts? How much does marketing performance change within a channel over time? Simulate the data, do the parameter recovery exercise. If you're onboarding a vendor or working with your internal team, have them do the holdout predictive accuracy check. Only send them data up to two months ago, ask them to forecast the next two months, and then check and see, like, is this right or is it terrible? If it's terrible, then you shouldn't trust it for anything else. So just go through and actually do the checks. Unfortunately, it's a lot of work. Like this is hard stuff. It requires writing a lot of code. There aren't great open source tools for this. But this is where the actual valuable work lies is in setting up the framework for how are we evaluating? It's very similar with machine learning, right? Like running you know, XG boost is really easy. Setting up the framework for how are we validating that there's no leakage between our training and our test set that when we actually deploy this production, it's going to work the way that we expect that we're actually getting the estimated benefit from deploying this model in production. That's where all of the really hard work lies. And if the same thing with an MMM, but even worse, because the feedback loop is up in much slower than it is with like traditional machine learning applications. What are you seeing as the, like a reliable feedback loop there? I mean, are you seeing companies that can do this with like like weekly feedback? Or is it just it has to be a monthorid, has to be two months or a quarter or whatever? So you can start to get feedback after a week. The problem with something like an MMM is in general, the forecast that it's making are generally fairly uncertain. And so within seven days, it can be very difficult, like you have to make very large changes to a marketing program, to be able to invalidate a forecast that an MMM is making over the course of a week. And so what that means is that in general, often you just have to wait longer than that in order to be able to see, hey, did these changes that we make? Are they consistent or inconsistent with the forecast that we made in number of days ago? Over only a seven day time period, unless there's a huge change in the marketing activity over that time. And specifically huge changes in marketing activity that we believe only has short term effects or has most of its effect in the very short term. That's the only way that you can actually invalidate a forecast of an MMM over such a short time period. But if you imagine that you have some sort of awareness type channel that you believe have fairly long periods of effect, 30 days, 60 days, 90 days, well, then it's very difficult to invalidate an MMM's read over a time period that's much shorter than that. You're right. And then not to mention other effects, right? So like spillover or whatever. But like, which that's another whole another kind of worms, I guess. But if you're running a fully digital regime, I imagine you can get and I mean, like all your ads are digital. And you're selling digital stuff, right? So like, you're set your nap, right? And you're not fulfilling rideshare. It's your game or something or fitness app. It could be a shorter feedback loop, right? I mean, because it's not you could you could make those adjustments faster and you could see the effects fast. I mean, like you probably see the reactions faster, right? I mean, you could have a shorter feedback loop than a quarter, probably even shorter than a month, right? Yeah, that's absolutely true, especially for like very short term purchases, like games or fitness apps, right? I work with, you know, a whole range of businesses that range from like very expensive luxury products that are not bought on that schedule. But like, you're right. If you have an all digital business with like a very cheap first purchase price or even a free first purchase price, then you're going to expect much shorter time shifts and you can get that faster feedback. But you do still need fairly large changes in your marketing budget in order to see it in the MMM because of the uncertainty bounds in the estimates around a channel performance, right? These econometric models inherently they have a lot of uncertainty. They're not very precise measurement tools, right? You might have an estimate that the cost per acquisition for Facebook is between 50 and 75 dollars, right? That's a pretty wide range. And so you need to see a fairly large, you know, movement in Facebook activity to be able to see the result to be able to say like, hey, is this right or not? Does that make sense what I'm describing? Like the way that the uncertainty in the parameter estimates makes it so that it's difficult to invalidate just because the uncertainty is so wide. Of course, right? And so like, you would need a dramatic, remember that? But that's I think that's not really that different from just using a platform's own tools, right? I mean, if you reduce the budget by 5% or something and you've got these massive confidence intervals, like, okay, well, you're not going to get any information there. You don't have to like cut the budget in half or drop it to zero or something. But I do think like if you're running a fully digital regime, you're in a position to do that. It's not like you've got IOs, you send out six months in advance with like a TV campaign or something. Totally. If you're willing to do it, right? A lot of organizations do not are scared to make such large changes in their marketing budget. Again, this is just based on I think a lot of like, oh, Facebook has always been our best channel. We're very nervous to make changes larger than 5 or 10% because what if that makes it so that we don't actually hit our target for the quarter or whatever? So you're totally right that is like, theoretically possible. I think in a lot of organizations, practically, it's very difficult to do to make these like big swing changes. Obviously, we see it happen that there are some organizations that are very willing to do it, but a lot of organizations, especially as they get bigger and they're at the level where like an MMM actually makes sense, you get a lot of very natural conservatism that starts to kick in and people are very unwilling to make those sort of changes because they're very risk averse. Why did the MMM space become so crowded? I feel like it's just I see an overwhelming amount of MMM content on LinkedIn and I have no idea when that happened because it seemed to be recent. It seemed like there was just some kind of moment where everyone runs some sort of MMM shop now. Yeah, well, so I think, you know, I'm obviously part of this, but I think a couple of things going on. So one is technology made it possible to like do MMM's much faster than what had traditionally been available. And so again, if you imagine that like sort of legacy MMB vendors, they did everything by hand. It was a lot of analysts, time, it was a lot of like putting together decks. It was like, oh, we don't actually have to do that. We can run these models. We can do it in the cloud. We're able to take the work out of the analyst hands and put it into the computer's hands. And so that opened up the ability to do this in a really interesting way. Second, open source tools come out. And then third, you know, a lot of people started to get worried about digital tracking methods, attribution, last touch attribution, and started looking around for other tools that could potentially solve their measurement problem. So all of those things come together. There's like a big appetite for a new way of doing things. People see the new ability to use MMMs to do it. And so they start talking it up. And so I think that is what drove like this explosion in MMM vendors. And again, some of them are sort of only or MMM first. And some of them are other like attribution tools that tack on an MMM. Because they're like, well, why not just like run meridian in the background. Alongside our, you know, MTA solution. And we can say that we have an MMM as well. I think the big problem in the industry is that there is no good way for marketers to evaluate the MMMs quality in terms of what matters. Right. This is the thing that we just talked about that what actually matters is is the MMM right. Is this, you know, this MTA tool that is running meridian or raven or their own homeworld thing behind the scenes. Is it generating like actual useful output or is it just a random number generator. And because it's very difficult slash impossible for marketers to evaluate that. Then they don't evaluate these MMM vendors on what actually matters. They evaluate them on a bunch of like the shiny dashboard type features that are easy to see and easy to evaluate. But aren't actually the thing that matters for actually using the tool to drive profit into the future. And so that's a big problem that we have in the industry right now is it's really easy to run an MMM. And it's very difficult for buyers to evaluate the different MMMs on the access that actually matters. Which is is it actually learning true causal relationships in the data or is it just spitting out random numbers in a fancy dashboard. Well, yeah, I mean, that's that's the the challenge here with MMMs. I mean, you've got to deliberately instrument the experiments. I mean, you have to be able to do that. And if you if you don't do that, you're not going to. It's an MMM. It's going to operate across like this historical data set, but it's it's it's not like. Prescriptive in that way. And so. You're always going to be like it was just weird to me when people complain about. Not MMMs in particular, but like when there was this transition happening on the advertiser side from, you know, just pure play. Last click, especially in the app space to, you know, the probabilistic methods or well, how do I know what my role as is? You don't. It's the point. You have to test that. Right. You have to test you have to either MMMs measurement tool. You have to test it. You have to create the experiment that gives you the two things that you compare. That's the whole point. And I was just like that. Okay. Well, that's new operating model, right? For a lot of people. And it's it doesn't fit with how they function. And I think I mean, that's totally right. It doesn't fit without a function, but it also doesn't unfortunately a lot of vendors in this space and like hype people on LinkedIn. They have pitched MMM is like, it's a crystal ball. You don't need attribution anymore. Just run an MMM and it'll tell you the true incremental value of every single channel and campaign you have. Like that is just not true at all. It totally oversells the capability of this technology and it sets people up for failure because CMOs or VPs of marketing they hear that and they're like, great, I'll just buy whatever shiny MMM. It's going to tell me the truth and all my problems are going to be solved. And like that's just totally unrealistic. But again, for marketing leaders who are not steeped in the history of econometrics and the history of MMM, it's easy to see how they have gotten misled by people, you know, snake oil salesman effectively really dramatically over claiming what this technology can do. Yeah, right. And just selling it as something it isn't exactly. So I get it's attribution selling it as forward facing attribution versus just measurement, which is very different. Let's get real. Half the time, you're not sure if your campaigns work. You just kind of hope they do. Attribution numbers look good, but what if those conversions would have happened anyway? Incremental helps you find out for real. No tracking people, no guessing games. Just clear answers on which campaigns actually move the needle. If you want to stop guessing and start knowing, go to incremental because good marketing isn't about getting clicks. It's about getting results. Check them out at incremental dot com. That's I and see RM and TAL dot com or click the link in the show description. Okay, so we last spoke. I forgot to look this up, but it must have been like two two and a half years now. It's been more than one year for sure. Yeah, it's been a while. So let's say it was three. I think it must have been roughly three because I was in Austin when we spoke. How is the measurement landscape changed since we last spoke since since from three years ago? How how it takes change in that time? That's a good question. So I think, you know, I already alluded to this at the top of the call, but I think we've seen we've already seen peak MMM and I think we are now on the downward slope again. I think a lot of people got really excited about marketing and exmodeling. They overpromised what it could do. They brought into their organization. It failed fairly dramatically. And now they are sort of either writing it off or starting to realize like, oh, actually, this is a, this is not a magic, you know, a crystal ball. This is not actually going to solve all of our measurement problems. And they're starting to really rethink what the role is in their organization. So that's sort of one big macro change. Obviously, the other big change is like AI happened in the last three years, which has put a lot of power into a lot of people's hands to be able to run their own MMMs, set up their own measurement stack and be able to really interrogate the numbers in a way that there really wasn't possible before. And I think that's really interesting and exciting in that it allows some companies, at least those that are willing to put the work in to do it, to really start to evaluate these tools on the dimensions that matter and be able to see up close and personal, hey, if I run this MMM with two slightly different data sets or two slightly different assumptions, I'm getting different results. And that means that it can't be a crystal ball. And we need to think about a different way to use this tool and to use this paradigm. And maybe the last sort of major trend is the rise of incrementality testing. Again, a couple of like new vendors have popped up and are having a lot of success, which I think is great that people are starting to think about, okay, we're just like deliberate experimentation fit into the way that we strategically want to operate a marketing program. And so what are we doing to do with the marketing and marketing and marketing and marketing? And then we're just going to talk about the real estate, the real estate. And I think that's a great, really interesting thing. I think about the AI analysis piece because that's really interesting. I do think that you've got a lot more people empowered with the ability to do to be like a marketing data scientist than then could have done that in 2023. And so that to me does feel like a material changes. Is that something you're seeing? Yeah, absolutely. And I think, you know, again, you have all of like the problems that people at least like the influencers on LinkedIn constantly talk about like there are cases where data analysis can be challenging, it can be confusing and elements get it wrong. And so you have to like pay attention, but broadly, I see huge amounts of people who historically weren't even analysts like, you know, VPs of marketing directors of marketing sitting down and really diving into the data. And I think that it's really positive to the extent and it's like, this is what I look for and what gets me really excited is they start to find flaws in their own assumptions. And they can do that much faster than they were able to do in the past. So in the past, you might imagine that like a lot of VPs of marketing, the data that they get comes through in PowerPoint slides, they see it once every couple of weeks, they aren't ever really getting into the weeds. And so they aren't ever really forced to confront the potential misconceptions that they might have or the inconsistencies in their own beliefs. But as they're able to actually get into the data, they can see that much more clearly where they're able to put the pieces together to say like, wait, if Facebook is actually that performance and we just increase spend on Facebook by 30% why didn't our sales go up? And they can start to really dig in and like dive into those types of questions, which I think historically they like maybe vaguely are aware of, but don't have the time or the ability to actually ever dive in and check that out. And now they're able to see it like face to face with the data via clot or codex or whatever. And I think that's a really like that is a really powerful new thing that is happening that is getting people to ask themselves and their organization more of the really hard questions that they need to be asking. And there's always risks there, right? I mean, you get someone are diving into the data who doesn't have like the fluency with, you know, just even like basic statistics that could go or I, right? But I think in general is very good thing. And I've seen that. I mean, you're seeing that across the board and not just marketing, but in marketing, there's a very acute application, which is just like, can I can I understand like what the impact is, I understand the incremental impact. And then I think that it is the ability to do that, I think, made incrementality much more real to a lot of teams. I think maybe they viewed it as like this kind of like abstract, nebulous concept that didn't really apply to them, but. But now that the fact that you can interrogate that question very easily, right, with a couple of prompts and maybe seems like there's more of a like for some marketing teams that where it didn't exist before there's. there's more of this nexus of scrutiny, which is nice and that's good. That I think does probably at least nudge people away from last click where they might have been totally relying on it before. >> Yeah, I think that's exactly right. I've seen things even where marketers are like, "Oh, we haven't been able to get our data to run an MM and that was the big blocker." Then they're like, "I just asked Cloud to do it and Cloud pulled all of the data." Now it's there. It went from being a thing that felt totally impossible to them to like 30 minutes later, it's done. It's hard to understand how much that opens up the ability to get to the next interesting strategic step. I think that that is we're only starting to see the impacts of that, but I think it's going to be huge. >> Yeah, right. >> How are advertisers adapting their MMM's to CTV? Because I feel like that's the big growth channel for a lot of direct response marketers. They're just seeing CTV as the scale channel. That necessitates maybe the use of an MMM for the first time, or it necessitates, like you said, you onboard a new channel where the numbers are going to change, but they're going to change more slowly. You've got to be prepared to invest while the MMM adapts over the course of a month or something. How are they adapting to CTV while using an MMM? >> Yeah, that's a great question. Broadly, one of the very nice things about MMM's, at least in theory, is that they are channel-agnostic. If you can get the data about the channel activity, whether that's Spender Impressions by day, you can just slide that into an MMM and it will just work. Whether it's CTV or some other future new channel that hasn't even been invented yet, you can always just slide it into the MM and everything will just work. That's a very nice feature of MMM's. You're totally right that as companies are moving into doing more CTV measurement, that often spurs them to consider, like should we be using an MMM? I think, again, the answer is maybe, depending on the business, depending on how big the investment is. I think for most businesses, one of the nice things about CTV, connected TV in particular versus something like traditional linear television, is that it is possible to run Geographic Level experiments with it, which is what I would always recommend doing before starting to do an MMM. Linear TV is very difficult to experiment with because of the way that it's bought. Like, National TV has a very different buying mechanism than local TV. And so it's very unclear even if a Geographic Test on Linear TV bought locally, extrapolates to National. It has all of these other crazy problems. But like, CTV doesn't. You're able to naturally target at the geographic level. And so that makes it very easy to test and learn into in a way that's going to give you valid incrementality results. And so what I recommend to most businesses is, it's like, look, first, just launch it and see if there's any signal at all. Look at your, you know, look at your vendor's measurement panel, always take it with a great result. But like, take a look at it. Look at your post checkout survey results. Is anybody saying that they've heard about you from TV? If there's a sign of life there, start to think about scaling it up. And then start to think about, okay, how do we measure the incremental impact via something like a Geographic Lift Test. And then only at the point where it's like, okay, we can't run tests fast enough to be able to, you know, continuously measure this channel. Or we need to be able to, you know, incorporate all of our different results into something that we can use for forecasting and planning into the future. Only at that point should you really be thinking about, okay, yeah, let's go do an MMM and let's use it for all of these different things that aren't already solved by our touch-based attribution system, post checkout survey, and some amount of like incrementality testing. Right. Yeah, I mean, that's the, you know, the nice thing about CTV is you stop and start it, right? You can scale it up. It's great. You can do it on a dime. You can do it in different geographies. Like, it's a, it's so much easier to work with and to get started with than like traditional linear televiter. Is it harder to parse out the effects? Like, because I mean, you know, you've got the second screen phenomenon, right? I mean, is that, is that, like I said, I mean, you really can only do measurement here with a probabilistic model. How are you seeing companies deal with that? Because you might end, you might just end up over-tributing to meta, right? I mean, because if you're someone's seeing the ad and then they're scrolling meta and they see another ad, that's it, right? That's what it looks like. It looks like that drove the convert, like how are you seeing marketers be able to parse that apart? Or like, what, you know, because I mean, I understand the, I understand how to do it. But like, what, what challenges are you seeing marketers face when doing that? Yeah. I mean, it's a challenge and it's like, the bad news is that like, it's not, it's not really solvable to the degree that people would like it to be. Like, there will be some amount of leakage with an MMM or with an incrementality experiment. You're going to have really wide uncertainty bounds around what your estimate for the impact of that is and you're never going to be able to get it very precise. So like, the answer is just like, you're just never going to know. And what you have to do is you have to sort of cobble together signals that are all going to be imperfect as best you can and then try to use that to make a decision. But like, you're never going to know, oh, is the true cost per acquisition from CTV $64 or $68? Like, you're just never going to be able to answer that with very much precision. You're going to get ranges of like, well, it's probably between 50 and 75, but our post checkout survey responses around TV has been trending up. So we're going to have to just like, jump and say like, okay, you know, we hope that it's actually on the, you know, the low range of that CPA. So it's more profitable for us. And we're going to keep doubling down into it because strategically we believe it's important. And we want to diversify away from Facebook. Those are the sorts of decisions that you just practically have to make. You're never going to get this sort of like, really fine grain precision around the estimates of the performance that most marketers would like and that would make the decision really easy. Well, I mean, I would argue that they were overly confident in those metrics beforehand anyway, right? Absolutely. You know, that row as number was never some discrete number, right? And you know, that's that's part of the issue. It's with like the last collect attribution tools in particular. They just acclimated customers to seeing a single number and having a false sense of certainty around that. And it just that was never justified. Yeah, that's totally true. And again, I think I think a lot of people at least now intellectually recognize it, even if they don't like it in their heart of hearts. But what I point people back to is that you should be thinking about doing forecasting. And if you're using these numbers, the like last touch number and you're plugging into your forecast and you keep missing your forecast, that means something is wrong and those numbers are wrong. And you need to like go back to the drawing board and think about like, just make your beliefs consistent. Your beliefs in terms of marketing performance should be consistent with the top line results that you see at the end of the month. And if they aren't consistent today, you need to like go back and think about what has to change to make those consistent. And that's really where if you put your focus there, you put some amount of attention there, it starts to alleviate a lot of the problems that I think we've seen in the industry. If people overly focused on last touch attribution, when it isn't actually delivering, you know, the results that people think that it's delivering for them. Yeah. Well, amen. Michael, this was great. Thank you so much for coming on on short notice. How can people interact with you? How can they consume your wisdom elsewhere? Yeah, follow me on LinkedIn. Michael Kaminsky, that's a great place. We also have a great YouTube channel that we've been, you know, putting out a lot of great content on the recast YouTube channel. So check me out on LinkedIn and YouTube. Michael, appreciate your time. Eric, thanks so much for having me. Great conversation as always.