Marketing Mix Modeling: Utilizing Open Source Technologies to Improve Advertising ROI

Foreign. Welcome everyone to the first episode of a new sub series to Meta's Business Innovation and technology podcast. You're Better Off Knowing by Meta Marketing Science My name is Thorsten Muller Klockmann and I'm working for Meta out of her Hamburg office in Germany, leading marketing science in the Dach region. Marketing Signs that Matter means working with advertising clients to see what works well for them on our platforms and what doesn't. We are ultimately here to help advertisers get the most out of our platforms for their money. And your better of knowing is actually a statement from the core of our philosophy. We strongly believe in permanent testing and learning and evaluating the impact of your advertising efforts. The most successful companies in the world are usually successful because they always challenge the status quo and and strive to optimize their activities. So they need to evaluate what works even better than their potentially already good business as usual strategy. And maybe even more important, they are also not afraid of identifying strategies that fail because you're better off knowing if you really know what you should stop doing. And with this sub series in general, we want to build a community of marketing scientists and media planners. We want to share interesting and valid insights to help make decisions grounded in data and science. We want to talk about new or existing measurement tools and solutions and their pros and cons. And we want to share interesting and relevant study findings. And also very important, this is not supposed to be a promotion podcast for Meta. While the majority of our data and studies are of course coming from Meta, our goal is to share inspiration that can be applied broadly and beyond just our channels. So I'm very excited to kick this off and hopefully build a community with this. This first episode will be about open source measurement tools in general and solutions for marketing mix models in particular. And I'm super happy to introduce the guests to this episode and we actually have three of them. Let's start with Karen Flores from Deloitte. Karen, I'm really happy to have you with us. Can you share a bit of an introduction about yourself please? Hello. Hello Tarsten. Hello everyone who is listening to us. I'm really happy to be here. I I am Kareem Flores. I work at Deloitte in Spain in the artificial intelligence and data area and we do a lot of measurement and mainly marketing measurement using advanced algorithms and mathematical models. Basically, we let the data speak. Pleasure to have you with us. And next we have Joshua Pearce from the HUD Group, home of brands like Look Fantastic or MyProTeam. So Josh, great that you are Here as well. Please introduce yourself. Yeah, of course. Hi, Torsten. Hi everyone. So my name is Josh Pearce. I'm the head of Brand performance at the Hook Group based up in Manchester. My role really is a relatively new role, so I've been in it now about 18 months. In the brand function, we focus on everything, brand and customer at thg. So across all of our brands, we're looking at how do we move away to a really sustainable marketing approach, how do we absolutely obsess about our customer and learn the best ways to talk to them, the best places to reach them. MMM came into my radar around two years ago now and we've been on a really exciting journey along, working with Meta to scale up our approach and using Robin. So, yeah, excited to be here, looking. Forward to your learnings. But last but not least, I'd like to introduce our amazing META colleague, Igor Skokarn, or Iggy as we usually call him. Iggy. Please share some insights about you. So, hey, thank you, Torsten. Hello everyone. So I am Igor or you know, IGI Den. I am also part of the global marketing science team here at Meta. I am based in London. Before Meta, I used to work at Omnicom agencies, which, you know, I joined Meta about eight years ago now and in my work and research, really focused on the intersection of contemporary mmms, you know, holistic business, attribution experiments and open source. And you know, being one of the co authors of Robin along with Fong, Leo and Antonio, you know, could be more excited to join the podcast today and you know, be here with Karen, Josh and you to discuss how, how it all works in practice. Yeah, great, great to have you all. Let's dig into the topic. Let's get cracking on open source technologies. So Iggy, Meta and also other companies are providing open source technology packages that help measure the impact of advertising. Why are we doing this and can you share a couple of examples for such packages? Yeah, yeah. So like open source and open sourcing in general, really it's an important part of our Meta culture and an important part of our operating model. So if you, for example, listen to any of our, say, last two earnings call, you might have heard, you know, Mark, Mark Zuckerberg reference our open source efforts and you know, he actually noted our long history in this space and you know, that we believe that making our code and technology actually freely available to the industry and to and to the data scientists and marketers, we can, you know, can help us define industry standards and drive innovation. And actually there is like so many Examples. So one of the first things we open source was actually the design and specifications of a data center. I mean the entire architectural design of a data center in Oregon. Since then we made over 700 code repositories available and those include packages such as React or Pytorch that are hugely popular. And today over 1 million people are engaging with our code packages on GitHub. And actually don't forget llama, which is our open source language model. Yeah, right. So there is just so much and on our measurement and our marketing science open source packages are really in line with this spirit and we want to drive innovation. We want to do it in a collective, public, collaborative way. Really what we are trying to do is to harness the power and intelligence of the global data science community and try to solve collectively together some of the hardest problems in our industry. So the marketing science team at Meta have built and open sourced now several code packages to solve some of these challenges. So for today's discussion, we will mainly focus on Robin, which is our automated MMM marketing mix modeling package. But you know, some of you may have heard about GeoLift, our experimental geo Geo experimental package that is suitable in certain situations where, you know, people based conversations if this is not possible. Or simulator with Triple M which helps users generate simulated or synthetic data. And then you can use MMMs to model how these responds to various scenarios, but also others that we have contributed on an industry level that we haven't built, but we have contributed. Cool, thank you. Now let's look into MMM and the open source solution we have built for that called Robin. We have Deloitte and we have the HAT group here to talk about their experiences and advice for advertisers. But first of all, for those who may not be so familiar with marketing based modeling, Karin, can you briefly explain what MMM actually is? Okay, yeah, for sure. I think I first will give the book definition of an mmm. An MMM is a mathematical model that helps you to decompose defects of the variables that are having in a certain other variable that is the dependent one. For example, if I am selling books, that will be my dependent variable and I will have other variables that are affecting the sales of those books. Also it takes into account not only the direct actions that I do in marketing, but other things like seasonalities, the economy and other exogenous variables that are affecting that. That will be like by the book definition, but also if we translate that into business. I think this is the main point of doing a market mix model and it is a tool that gives you and gives you an idea on how to optimize your media mix and how to maximize your return or your volume. And this is the strategical view that we have to take into account when we are doing an mmm. Thanks so much. And yeah, I think to me, the beauty of MMM is also that these mathematical models are built on time series of either impressions or general spent levels. So two benefits from that. A, you don't need individual people or user level data, so no limitations that arise from privacy regulations. And B, you can really measure media like for, like, because all input from all channels have the same data background granularity. Right, so that's very good. Now, MMM has been around for decades already, but there have been a lot of new developments over the past year. So, Iggy, what do you think are the most exciting ones? Yes, so MMM has, as you said, has like predates the Internet. Right. So like, it actually does not inherit some of the challenges that have arise since. But anyway, it has sort of undergone significant evolution over the last 18 to 24 months. Massive, massive comeback. But it is not the traditional MMM that, you know, many brands have known since the first wave. But it's like the modern machine learning enabled MMM with more granular, faster, you know, more automated, with, you know, with a greater emphasis of transparency. It's really, really, you know, more advanced than ever before. And we see this, like the technique, you know, decades old technique sort of transformed itself in the hands of like the modern data scientists. And now AI is driving like further evolution even within that. So with Robin, we've put a stake in the ground of what we believe is best in class. Mmm. We truly believe in like opening the black box, transparency, you know, customizable models that can democratize access to very good MMM and, you know, solve for some of the challenges around analyst bias or subjectivity. So that's, you know, for me, these sort of like, aims to reduce human bias. The evolutionary AI algorithms in MMM ground truth calibration with experiments to account for causality, these sorts of things along with like, you know, having it faster, more automated, more granular. Really, this is what the new MMM is about. That's really impressive. So let's dive into practice now. Josh, you have utilized Robin at the AD group. Can you share some business context and your reasons to look into? Mmm. Yeah, absolutely. So the main objective that really kicked this off was us starting to look at our brands as we're reaching a point of maturity now. So the Hook group in relative scale is quite a new thing and we have relatively new Brands So it's somewhere between 15 and 20 years old depending on when we acquired certain brands. But really our objective was now we've hit a point of maturity across these brands and look fantastic. My protein and cult beauty are very much market leaders in their space. How do we start to transition away from predominantly online marketing and become very real and actually work our way towards sustainable growth? And we knew that a lot of that involved investing back into our brands, growing our brand presence and our brand awareness and really elevating ourself in the mind of the customer. And something that wasn't going to allow us to do that was being reliant on last click attribution. So we started to explore actually how do we value marketing spending outside of one measurement platform that we know is going to be overly biased towards a type of marketing that is very direct and very click based. So that was really where the journey began for us in a sense of, and to be honest that that spanned everybody regardless whether you were a brand or a retailer. It was all of the same problems and we, we started to explore different solutions and we, we, we thought about MTA for a while but ultimately for me, and this is me being semi selfish in a, in a slightly brand role, I don't, I want us to be able to value the importance of impression. So therefore it was really, really important for me to find a measurement platform that could acknowledge that. And as we started to go through it, I think touching back on what Iggy just said, the, the open source solutions to this were the biggest unlock because they had full transparency around what we were doing. I'd say the Hook Group is an incredibly open minded company when it comes to change. And as much as we've relied on last click for quite a while now, when we started to look at actually how did we embed MMM within the group, the transparency of it being open source won over so many stakeholders because they could see exactly what we were doing with the data. We weren't necessarily hiding anything behind the curtain and just coming out with a graph that said oh actually billboards or paid social is worth way more than we thought. So it's been massive in helping us get it off the ground. Cool. And how did you approach this? I mean did you have in house resources that could start with Robin right away? So when we initially started this we used MMM and it's I guess its most basic form which wasn't mmm and we looked at some very simple linear regression and we quite quickly realized that a lot of our businesses had Far too many moving parts to make sense of the data. So our journey really started when we made the decision to externalize the resource initially. And we actually have worked now with a contractor over in the States who I've got to give a shout out to because he's been absolutely instrumental, him and his business, in getting us off the ground. So it's causal analytics and Jim Jiang Olio. So how we, how we arrived at that and it kind of ties back to the open source package. My goal was how do we get up to industry class very quickly and our options really were do we try and write our own code? And being realistic, we don't have much experience writing mmm code. Do we go to a kind of a full externalized party where they'd use a proprietary model and we probably wouldn't have full sight of the code, just the output or there was open source packages where we could actually partner with somebody who was very experienced in that and then leverage the kind of best in class industry code for us to get there. So it's been a very joint effort, but we're currently in a really exciting position actually where we're now looking to internalize that resource. There's some really exciting roles across the Hook group at the moment where we're bringing this back in house so that we can scale it fully across the group. Exciting. And how long did it take you until you had a working model and what is the status now? Yeah, so I'd say the first model from it, from it being an idea and finding Jim on actually a Facebook page to having a kind of first piece of paper to look at was about three months. And that honestly a lot of the time there and I'm sure we'll touch on it was how do we get the data in the right place to enable us to do this properly in terms of actually when we had that done, Robin is in a way so simple to use that that was the easiest part. But I'm very, very proud to say now we're at a point where our biggest brands across the group have all got models. We're looking at internationalizing those across multiple locales. But yeah, it was, I wouldn't lie, it was a slog to get there. But now we're there and we're set up from a data perspective and we've got the relevant automation, et cetera, in place. It feels like we can fly with it. So cool. And what were the biggest wins for you so far by utilizing Mmm, yeah. I'm going to sound overly biased to matter and Paid Social now. But I'm incredibly, incredibly passionate about, as I said before, how we can really educate customers about our brands and our products. And we operate in two industries where product education is so important and that's nutrition and beauty. And actually a lot of the channels that we could communicate proper education through and kind of product benefits were not truly acknowledged through Last click. Something that MMM's helped us to do is understand things like Paid Social, for example. So we, we broadly see across the group that Paid Social is up to eight times undervalued on Last Click. This is a massive win for us being able to unlock the true value of Paid Social because all of the marketeers that were not too handcuffed by Last Click data really understood the true value of this. But it was really hard to put a numerical figure on actually what do we think it's impacting the customer. Whereas now we're in a place where MMM has helped us do that. The other biggest thing that we have kind of taken from this as well. So aside from the channel effectiveness views that we get, Robin has really helped us with the budget optimizers, in a sense of how we start to look at ad stock and saturation curves that come out of it. We get a true view of the delayed effect of some channels versus others and what's immediate, what's not, and then where could we actually move inefficient spend into more efficient spend? So full budget optimization, et cetera. Cool. And what were the biggest challenges you faced on your way? Absolutely the data collection. And thankfully, Jim reassures me every time that we go through this that it's not just us, it's a massive thing. I think that probably a lot of businesses don't appreciate is that the data is the fundamental of this and your models can only be as good as the data that you have. When we started this journey, we predominantly have mostly digital and online marketing channels, so it was maybe slightly simpler than someone who's kind of full omni, but it was very manual and there was a lot of Excel spreadsheets flying about with three years worth of daily data for every channel and every objective that had to come out of every ad platform and, and kind of God knows where. And I mean, even just down to the simplicity of. Because we were literally copying and pasting things into Excel, formatting errors started coming in and some people had put a pound sign and some people had missed out commas or we had pasted text as numbers and all of the kind of fun stuff you'd appreciate with that. That was definitely the biggest challenge, in a sense of we had to appreciate that to reach a point of scale, we had to automate that process and we had to do two things. We had to take the human error out of it, but then we had to take something as well that didn't take up the business's time, because one thing that everybody is so short of is time. So we managed to leverage an API aggregator partner called Adverity that we work with. And that has completely transformed many things for our business, including reporting, but especially how we use our MMM pipelines. So we've gone from a place of being kind of handcuffed by Excel and when teams can get around to extracting data from all of the various platforms, etc. To a place where Robin is just straight away picking out of our bq. We don't really need to do too much at that point. So that's been our biggest challenge, but also our biggest win. Yeah. Sounds also like a pretty rapid learning curve, so well done. Any challenges you haven't resolved yet? We. We've got a few, actually, and I think some of these come down to the. The nuances of various businesses. So, as I said before, like, we're building the same model in context for all of the brands, but every single model is so different, it's unbelievable. And even if we look at the beauty space, Court Beauty and look fantastic on paper, have relatively very similar businesses in the sense of their very large beauty retailers, we're doing it within the uk. The proposition is broadly similar, therefore the MMM should be broadly similar. But actually something that we very quickly realized, and this is unique to most digital businesses, is what's important for us to understand from this, is the impact of influencer as a channel. And we have the same issue here with affiliates. MMM is obviously, in a very simple term, and you'll love me for saying this, it's pattern matching and we need the data to be in the right pattern as such. And what we found with influencers actually is our influencer spend or impression data isn't actually linked to when influencers are posting, it's linked to when we pay them. So we almost have an inverse correlation there because the model is trying to pick up something that it didn't actually happen on that day. And this is something that we're very much working through. We've tried flat phasing through the month and we almost come to a point, point of no correlation because there's no change. We've. We're Trying do we map influencer spend to code attribution? So when you convert with an influencer code on any of our sites they they tend to have unique codes so that we can credit commission back to the different influencers and things like that. So it's very much a working challenge and kind of taking it back to cult beauty. Cult beauty is a really interesting business model. It has by name a very cult following. It's one of the I think the coolest propositions within beauty at the moment and it has very niche brands. It has all of the brands coming from the states that everybody wants and the influencer proposition for cult beauty is very important. But what's actually happening with influencers on cult beauty is the, the influencers that we use are so strong in their niche that they don't need to post for the code attributions to be flying because the kind of. I almost think the fans of cult beauty know that these codes are always there and they don't need these various influencers to post. So tying back to the point I said before around how we've tried to map code attribution to maybe infer when we think we had posts it still doesn't match. So it's a constant kind of working idea as such to go look, this channel is massive and we know it drives so much revenue but what is the true value that we think it actually drives? Because every time we try a different spend attribution method we get a completely different answer. Very interesting. Iggy, Karen, any thoughts that could help Josh here? Actually I would have to give the easiest answer to that and I think it will be experiments for sure. Attribution and MMM are not as strong as when you calibrate them with experiments. So yeah, I do believe that Measurement360 and this triangulation idea is the way to go because the incrementality actually give us the causal impact with these experiments. So yes, that's mainly like the gold standard of everything is incrementality and I always use this saying that I have there that an MMM with calibration with experiments it's like an MMM with steroids because it's stronger and it's bigger and it's better. So yeah, I think the way to go there it would be experiments and maybe you can. I think there is challenges in order to do these experiments but you have other solutions like GEO experiments that also there is an open source code developed by Meta there. So yeah we do have more tools in order to solve those questions. Great, thank you and thanks for sharing your fabulous insights, Josh. Taking this to a level of general guidance. Karen, earlier this year you have published a white paper called Marketing Mixed the Opportunities behind Open source Techniques which provides actual guidance on how advertisers can utilize those open source techniques to build their models. So from what you've just heard from Josh, does it sound like a typical journey? Is it similar to others and where is it maybe different? I think I would say it's not the typical journey. But now everything that we have seen, probably it's atypical. I think I could divide the advertisers into two and I think Josh has this amazing business where you have a combination of both. I could select these VR strong doctor advertisers and I think they go with these. Like Josh was telling this journey of trying to develop it in house. Can I do it? Can I develop it, the code from scratch? And this is when I use the open source. And there are these other type of advertisers that maybe they are like more traditional CPG retail advertisers banking that they go straight to the third party in order to have these optimization of the media mix with the mmms in this outsourced world. But what I like about Josh was telling that he tried to, well, his team tried to develop it from scratch then using these open source tools and then he also used a third party in order to have this put in place. So I think Josh has like this super beauty, super beautiful story about using these three parts, the outsourced, the in house and the hybrid world. So I really enjoy listening to Josh. Me too. So yeah. Karen, what was your work exactly about the white paper? Can you talk about that a little bit? Yes. In the white paper we actually try to avoid these kind of challenges that Josh was mentioning. He was mentioning that they were focusing on the code, but then the challenge was the data. So in the paper we tried to give some advice to advertisers that are starting with the journey in order to have these questions. First, don't go to this challenge that Josh was mentioning about the data. We put into the paper, these key points to tackle before you start the MMM process. Because you need to have the data science team, you need to have the data, you need to have the right stakeholders in order to develop this good process and obviously the collaboration between areas. I don't think we mentioned this before, but you can have the best model in every metric. But if it doesn't resonate with the business KPIs or the business questions that want to be answered, then your model is not going to be working at its best. It's going to be saving a draw and that's it. So that's not what we want out of these beautiful models. Yeah, yeah, right. And in general, what, what benefits from implementing mmm? Did you see? Yes, we have a really good number there in the, in the paper. We have a benchmark, a Deloitte benchmark, and we have seen that advertisers, 44% of advertisers have enhanced their ROI by at least 10% when they use MMM. So if you put that into a revenue of a big company or a small company, improving 10% of your ROI, when you translate that into a revenue, that's a lot. So yeah, we have seen that a lot of advertisers get a lot of gains out of doing an mmm. Yeah. All right. So in general, if a company thinks about in housing mmm, what guidance would you give? Well, first of all, I think that the most important thing when you have to develop an in house MMM is to have the data science team. In order to have an in house process of mmm, I would say somebody has to run this code. So it has to be the data science team. I think that's the main priority to have also the primary input to that. It's the data. To have organized data, to have the data. I believe that I heard this from one of our clients, but they used to delete the data every year. So we were crazy, like going crazy when we heard that they delete the data every year. So you are not going to have a use of that data. That was crazy. So yeah, the data would be my second point. In order to have a good something that is super needed in order to have this MMM in house. And also the communication between teams. Because as I mentioned before, you can have the best data science team and you can have the data, but if you don't have the input of the marketing team, that's not going to be so realistic as it should be. And also something that Josh was mentioning in understanding the paid social media, the caveats that comes from the marketing execution, I think that this is also key to have into account when you develop an mmm. All right, so that's requirements. What would be your guidance on steps a business should take when they start to model? Actually, yes, I think something that it's useful and we use this a lot in consultancy is to have a framework in order to have any question, not just mmm, you should have this framework of first of all, set the. The clear Objectives. What answers do you want to get out of this mmm? As Josh was mentioning, they wanted to have this. How are influencers, for example, performing? So you have to first of all set the questions that you want to answer with this model. Then the modeling strategy. What are you going to use? For example, online channel, offline channel by brand, by category? Setting this modeling strategy, it's also really important in order to have a good framework. The third one, it's the modeling, the act per se of doing the models that I think we focus a lot on the modeling. But there is a lot of things going on besides the modeling. As Joshua was mentioning, the data, all of these things that go beyond just the modeling, I think they are really important. And the fourth point would be the scaling and the maintenance. For this, we recommend also to have visualization tools to be democratized within the company. The results. Having a good sponsor, an executive sponsor also helps because the data science team is going to execute models, but then the marketing team is going to be the one making the decisions out of these models. So I think these four steps are the key ones. In order to have a good process in the mmm. And talking about sponsors, I mean you definitely need the marketing team. Do you also need C level involvement? And if so, who's even more important? Is it the CFO or the cmo? Well, I will say that with Deloitte we have observed this tremendous shift because before we used to only talk to the CMO when we were talking about marketing effectiveness modeling. But now the CFOs and the CEOs are getting into the table to talk about these results. Marketing mix modeling has become not just about marketing, but about everything that affects the sales. For example, we have a client where we not only do media mix modeling, we also include distribution pricing. I remember that one of our clients was telling us that the manager of the branch branch had a big input into the sales of that branch. So this is something that goes beyond marketing. We need to take into account all of the direct actions that the company is doing in order to drive sales. I would say that it keeps being the cmo, the one that is making the calls with the mmm, but it has also being highly adopted. Sorry. By the CFO and the CEO. Yeah. Josh, what's your experience with getting C suite buy in on your journey? Yeah, very similar actually. But we have another key stakeholder who is actually the kind of driving force behind this, which is our cbo, which I think is a really interesting dynamic. So it's our Chief Brand Officer, Hannah is The main executive sponsor for the MMM project within the business and she works very closely with myself. Looking at how we kind of leverage top line brand activations and how do we create a kind of full funnel marketing approach. So CMOs are very much involved and completely instrumental obviously to kind of budget planning, etc. But we have a really interesting dynamic as well where CBO has a very much an equal seat at the table in terms of measurement as well as kind of aesthetic and messaging and everything a CBO normally would care about from a brand perspective. But it's absolutely right what Karen said. CEO buying is also so crucial. The divisional CEOs that we have across Nutrition and Beauty have been incredibly supportive as a project. And kind of going back to what I said before, the reason for that is the full transparency around what we're doing. But I completely agree, without the C level buy in, we could do the models where I don't think we would get the traction is actually being able to implement change. Yeah, that makes sense. Thank you. And Karen, you have seen various open source solutions for mmm. How would you compare them actually? Oh, this is a tough topic. So I would say that online we have a lot of qualitative comparisons of the features of each open source MMM and they have like some, I don't know, classifications of good, regular and bad. But that doesn't really help you make the decision on which one to choose. And I'm going to quote Iggy on this one because we have a conversation about this one day and it is not about the differences but about the commonalities. So I would say that beforehand what I could suggest someone that is trying to do it in house and they want to choose an open source tool, probably you should recreate some dummy data that adapts the most into your business and then you can run each one of these tools or these libraries and you can measure the map, the error, the fit and you can have more like a quantitative number in order to decide. I would say this, but also I will do a spoiler. We are planning on developing a white paper to give more guidance on how to choose and it would not be a qualitative, it will be a little bit more into the, into detail. So yeah, yeah, we will have that soon. Looking forward to that. Yeah. So super interesting and hopefully helpful guidance for the audience if they think about implementing mmm. Let's try and look into the future. So Karen, maybe you go first. What do you think the future will hold for MMM and open source? I obviously want to see everyone taking decisions Every company taking decisions with MMM that would be like the ideal world, obviously calibrated MMM with experiments and using these three methodologies, attribution, MMM and experiments. But to be super honest, we are moving really fast in terms of the methodology, but we are not moving as fast as we should in terms of the data collection. As Josh was mentioning, the data keeps being the most painful part. So in the ideal world, everything will be connected and we don't have to spend weeks doing the data collection. I would say I would like that in the future. Do you see potential solutions for that to ease the pain on data collection? Well, first of all, awareness of the importance of having the data structured and in one place. I will say that, yeah, mainly that. And I think we have a lot of tools in the world to have data and connected data. It's just that we don't have the awareness of the importance of having it. It's just that many people underestimate that. Right. It's like, yeah, we have all the data somewhere, so it can't be that difficult. Yeah, it actually is. Yeah. Yeah. Okay, thank you. Josh, what are your next steps and plans for utilizing open source and mmm? Yeah, I think we're. Well, we actually have two things on the horizon at the moment that I think are really exciting for us as a business. So the first one is our MMM journey has got to a point now across a lot of our brands where we are starting to do testing and actually kind of mirroring what Karen said, we are actually using the GEO experiment packages by Meta. So some of the initial tests that we are doing, we've taken the UK for some of our brands, we've split them into test and control groups based on some of the channels that we can activate by location. And we're starting to do real world tests with not just the MMM outputs. But actually what's been really important to us is how do we find kind of a triangulation of truth between multiple measurement sources? Because I think one thing actually that is really important to understand with MMM is you go from last click or MTA to mmm and in the best way, probably none of them are right. And it's somewhere in the middle and that's kind of how cautious we've been with testing. But we're at a really exciting point now where we're making real world change with these models and actually we're getting proper data back on how does the difference it's made to our marketing mix actually come into effect? So, yeah, I think that's a really Exciting one and then another one that we're starting to focus on which will benefit my team greatly. So I'm very biased is we're actually working very closely with Meta on a kind of a module as such add on to Robin where we could incorporate some of our brand tracking data within to the Robin base code. So what that would allow us to do is not just have variables of spending impressions that you mentioned before, Torsten, but brand awareness or brand equity. Brand consideration as almost. Whether it be a context variable or you'd almost treat it potentially like a channel. But what we're trying to look at there is as we develop our understanding of the kind of intercept so that the, the outstanding brand value, how does brand awareness contribute towards that? And that will help us get to a really important point where I see a point in the future where we can say that a 1% increase in brand value equates to X amount of pounds. So we're on the journey to get there because as soon as we can unlock that, the absolute focus as well. How much do you want to grow brand awareness by and what do we need to do it? So yeah, two really exciting things that we've got coming. Yeah, that's a great one. Actually on the agenda for further episodes we have brand building and how mmm, like you just mentioned could be used for that. So really glad to hear that you're planning that as well. Iggy, what are Meta's plans for the future of open source in general and Robin in particular? Yeah, I mean this is all very like super fascinating discussion here and like we do believe there are like several areas in measurement but like, you know, mmm, really, really crucial for advancement of Robin. But they also like hold key to, you know, these sorts of pressing challenges that we see, you know, in the measurement and marketing industry. And so like one, we're thinking about how to further integrate the online and offline sales and, and enable certain functionality for like more omnichannel or concurrent modeling, you know, dynamic effects. There's a lot of things happening in the world, right? So how do we actually incorporate some more dynamicity into the models? We're thinking about how to bring some unified modeling, you know, the short term, long term, the brand models that we just discussed in a minute and there has to be like further advance advances and we're thinking about this, about model validation and using of causal inference. As. Well as I guess looking into the future within, within the model like forecasting, scenario planning and how do we enable better ways to run, you know, scenarios. But we are particularly, particularly interested in finding ways and further ways to harness the, you know, the community and the collective intelligence and collaboratively build this kind of future. And because we do believe and we see that like working together as a data science community, we can overcome these challenges much faster and you can continue to innovate in the industry in an open and transparent way. It's actually also a very interesting point. So how many people outside of Meta have contributed to Robin actually? Yeah, I mean, that's super cool actually. Like we have, I think maybe 15 or so from outside of Meta. So this is not Meta code, right? And people said, oh, this is meta score. It's not, it's maintained by us. It's, you know, other people are. Obviously we're driving majority of it, but we're actively looking to increase this number. And you know, say last year the big community contribution was the Python API. I mean, it's pretty wild actually. And this year we have some, some other ideas. I'll keep them for myself for the moment because I don't want to overshare, but we're finalizing it and it's exciting how we can keep the community involved and keep continuing this journey of innovation and transparency and then like looking forward to working with community ever more so and with great partners like the Hat Group, Deloitte and others. Right, thanks. So we're coming to the end of this episode. Thanks so much, Karen, Josh, Iggy, for your contribution and your absolute top notch insights. I found this was really interesting and have learned a lot. But of course, more important, I hope the audience and our listeners feel educated and inspired as well. Thanks so much everyone for tuning in. If you have questions, feel free to leave comments or to reach out to the guests or to myself. And of course we would love you to subscribe to the Meta Business Innovation and Technology podcast and tune into our next episodes. We have already planned a lot for the youe're Better of Knowing subseries, for example, on brand building, as just mentioned on automation and various other topics. So thanks again for listening. I'm already looking forward to the next one. Bye. Sam.