Meta’s Product Director Explains How Its Ad Algorithm Works

All right, welcome back to Marketing Operators. I'm Connor McDonald. I'm here with Cody Ploffker, Connor Rolane, and we've got a very special guest, Sue Keshe, Miriola, who's a director, PM at Metta, specifically for the core delivery product. So we go super deep on Indromeda, Gem, Ladis. We talk about value expression, and what that means for marketers today. Thanks for joining. We'll jump right into it. We are often accused of going really deep to a nerdy degree talking about ad delivery. But there is a man that exists that would allow us to go even deeper and even earlier as it relates to ad delivery. So we've got Sue Keshe, Miriola here, the PM director at Metta leading core delivery products. Thanks for being on. Excited to be here. 100%. So I thought we'd start super high level. We have a bunch of performance marketers and advertisers listening. Why don't we just start very high level how the current delivery system works? Everybody's heard bid estimated action rate, ad quality. Maybe you could just give us a broad walkthrough of how the bidding system and delivery system at Metta has worked as advertisers know it over the last few years. I think the ad delivery system consists of, first we are trying to optimize for what outcomes that people want. So if you really think about it, there's a bid, but that bid consists of a set of value. And what the value is consisting of is what outcomes as you want and what's the probability when we think those outcomes are gonna have. And this is the output, if you think about it. And then there's another big component of this and that, 'cause it's of like how much we think is the advertiser, an individual advertiser's value. And then there's another piece of it, which is what we think is the user value of it. So fundamentally, I think we wanna maximize the entire ecosystem, right? So the ecosystem, 'cause it's of what advertisers want and how much value we think we'll get from showing a particular ad impression to a user. But there's also a piece of like, where's the ecosystem? And that is a user quality bid and we could like talk about that. And both of those together combine like, what did you talk about the bid? And so that is like the outcome, but how does that happen? So one is, first you gotta tell me, the way I think about it is, how do we calculate this? First, and most importantly, you gotta tell me who you wanna reach. And that's like if you think about it, the broadest audience classification, people in a certain geography, people who made purchases, you can kinda have a semantic or a language around which audience. Most of them, you wanna kind of focus on which region or what are the, that's targeting. Then after you tell me who you wanna reach and you gotta tell me what outcomes do you want. And those are like optimization goals and we could go deep, you guys are nerds about that. So we could like talk about it. But it's really around like, what is the, how do you value, what do you want? And I think we could talk about like conversions, you wanna wanna get a sale or you could talk about value of a sale, like the margin. But then you know, those are like on the DTC side, there's also different folks that want leads, different folks that want you to like install their app or get value of an app or there's a whole slew of things that we kind of wanna optimize for. And then, and then if you give me those two, then you gotta tell me how much you're gonna pay for that, like how much you're willing to value that action, that comes through your bid. And then, relatedly is the budget piece, right? Once you tell me the bid, you gotta tell me how much because obviously like the more you want something, like you're gonna get, there's a limited set of audiences and we're gonna generate, we have to keep going down there. And then the last piece I would say is that quality factors are telling you which we would calculate internally to make sure that we understand like, how valuable. That actually is that. - 100%. So before we peel back the layers any further, I wanna kick it to Cody quickly. 'Cause Cody Connor and I are, we're on the brand side. So we are not nearly as familiar with the sort of delivery mechanisms. Like I mentioned, we've heard all these terms. But Cody, what's your understanding and with all of your performance marketing background and all the growth that you helped lead at John's Road, like what was your thought process around acquiring customers and how it relates to the delivery system? - I feel like it's changed over the years and I know it's changed like the last few years. I'm excited to get into it and hear more. I'm like, you know, there's been so much chatter lately in the last like two years about. And Dramada has changed everything. You've probably seen all the tweets and podcasts about Dramada. And then there's obviously Jem and Lannis as well. So I'm excited we're just like armchair quarterbacks, we're podcasters, we're trying to put things together and read the documentation, but excited to have you break it down. But yeah, I think my understanding is like, creative has just become so much more important. Obviously, we've talked about a ton that like manual targeting is way less important. And then they're, you know, so creative is more important. I think the delivery changes, especially as, you know, social platforms have changed and follower-based feeds are less important and just like the types of people, of content people are interacting with, like has gotten way more important. You need way more creative than before. And then obviously these systems behind the scenes are actually allowing that to happen like in Dramada or things like that. So I just think the role of what we've had to do from the brand side is obviously shifted from, let me go in and select who the target and make all of these manual changes. And yeah, let me kind of be a little bit more hands-off, but focus on what I can control to actually guide delivery, which is not, oh, I want to target this person, but it's actually, hey, like, I'm going to make this creative, I'm going to make these lane pages, is that right, unfair? Yeah, I think you got a lot of it. And I would like really try to like break it down and maybe like structure it, maybe slightly differently. But I think that's a nice way of saying you were wrong. (laughing) No, no, no, you were right. I think, you know, I think the way I would think about it is at the broadest level. We really invested in like, if you think about an ad product, and you could think about it as a product space, and then like, what happens behind the hood? Like, how do we deliver the outcome? And so one of the ways we've been really investing over the last, you know, several years, is the value expression piece. And that's like, how do you talk about the language of what you value? And I think ultimately the closer, I think, in general, we get towards the exact value on how, you know, an individual advertiser thinks about outcomes from meta, like a sale, or then the closer we can get to being like, hey, you don't have to like, press all the knobs. 'Cause we know what you want, we're aligned. And so you could give us more freedom across like our different optimization approaches to be able to deliver that value. But in a world where, you know, we don't have that value expression, then you need to go back and like set very manual targeting and it really kind of break that down. I'm not just targeting, but across the kind of spectrum of control. So the way you think about it is like value expression and control. The more richer your value expression is, the more it's aligned towards how you actually, like, ascertain value, the less control, you know, I think you need, because then we can then optimize across all of our systems to be able to deliver that. But the more the, obviously, the less, you know, we're aligned, you have to then do this mapping where you have to look at, hey, this is what meta says, this is what I say, there's like a gap here now how do I kind of tweak the knobs to get there. So that's like one big area we can talk about that across optimization, like attribution, there's some pretty, really cool launches. We've, products we built over many years and large investment there, like incremental attributes. >> Yeah, that's tough, huh, try about that. And then behind the scene, and then we're talking about, you know, Andromeda Jam and Latta, so we can go deeper into there. Those are like ways, the once we kind of align on like, hey, we're talking the same language, we're pretty close. Then we can invest, and that's a lot of the advances in AI, to be able to like, deliver that over a broad audience. Like, ultimately, like, if we have, say, a very large audience, with a lot of different ads that can compete for it, we need to do much more sophisticated job on ranking, real time, to be able to like, do better predictions, and across a larger potential sets of opportunities. And that's where a lot of our ranking systems have been really advanced as well too. So they really go hand in hand. But I would say that, at least that's the framing that's like helpful to break those down between the two. >> That's perfect, why don't we go deeper on this idea of value expression? From the advertiser side, for a long period of time, we didn't have a lot of control over what we were ultimately optimizing for. And it's really interesting to see meta, and you guys were really clear at the meta performance marketing summit over the summer on you are developing the tools to best align an advertiser's interest with what platforms delivering. So maybe you could give us a couple examples of what that looks like now that listeners might recognize. >> Yeah, I mean, the biggest one, I would say, is value optimization. So if I can like, take a little bit longer, I'll tell you, like, originally, that was one of the first product I built. Like, you know, we're done launching a long time ago. And so originally, we would get, like, sales conversions. Like, it would just sell conversions, but then we realized, hey, conversions are great, but like, you know, somebody buy something for a dollar or something somebody was for a hundred dollars, you value the one that's a hundred dollars more, even though they get you more of the dollar ones. And so we built value optimization, you know, that's been an evolution where we really built that up. But then we are building, like, value itself isn't like, it's just revenue, but there's other things like margin. It's also how you attribute that. And so there's a lot of, so value optimization is a big product. Value rules, like where you could look at an audience and say, hey, look, you know, I value this kind of audience better. So those are all different functions of, I would say, value expression. We were looking on. Everyone is talking about AI, but few operators are actually using it to speed up their marketing and make better decisions. At Ridge, we use Runeth. It's an AI brain for the whole marketing team. It watches every creative you've ever run, knows what's inside of each one, and connected to it actually drove revenue. You can plug it into your clay video shop. Northpeem, your whole Google Drive, it reads all of it. So instead of asking what work last quarter, anyone on your team can ask for 10 evergreen concepts, worth testing before Q4 and get them back within minutes. We use it, Hexclid uses it, Jones Road uses it, it's the best AI brain for e-commerce on the planet. If you don't use it, you're going to fall behind. Go to runeth.com and book a strategy session. Mention marketing operator sent you and you'll get $1,000 in free credits. 100% and would you include some of the custom attribution that you guys are supporting now? We run a lot of our business using first party MTA data. And yeah, we've often found it beneficial to just allow meta to understand the results that we're really measuring and looking for. Absolutely, I mean, we have incremental attributes here, which is really tilted towards like delivering conversions or sales that you're crediting us for. You're like, hey, this wouldn't happen without meta according to your systems. And then custom attribution is even that step further where it's like, hey, we have a very specific way that we think about attributing sales from meta. And then we're like, tell that to us so then we can learn it and deliver that. Yeah. So Connor, what has your guys' approach been between incremental attribution, custom attribution, value rules were brought up? Like, how is Hexclid leveraging some of these tools today? Yeah. Why are you big hug after this? 'Cause we've put so much spend into value. Yeah, I think I'm value and it's like, we are huge on value and it makes sense for our business. If you were to go look in our CDP and you were to look at our LTV, it's 100% correlated, the most value we can extract on the first time order, backs into the best 12, 24 lifetime value over all time. So for our business where our hero product is a big set, we have people in our funnel for a very long time that we're educating on this. We've put a ton of add dollars into value optimized. I was looking last night and like 50% of our add account right now is optimized against value and it's just so aligned with our business goals and I think that's what's exciting about, I heard all the stuff you just said. It's like the custom attribution which we use, the incremental attribution which we use, lowest cost we use, value we use. Like we have this hero funnel that we use a lot of value optimized around but our business is becoming more and more nuanced in robust and now we have funnels that it doesn't make sense to optimize for value. So we're able to just like align these different campaign objectives with the goal of that funnel and now we have a very nuanced robust add account where certain things are using value, certain things lowest cost and for us value has been huge. Like we've gotten so much scale out of it we're acquiring exactly the type of customer that we want because of it. And then we're able to point and shoot, you know when we roll out a new product, like all right maybe value is not what we want for that. And now we're able to balance things like, hey sure the value optimized stuff isn't going to be best for like first order velocity but it is going to extract exactly the type of customer you want for that product. And then we have these other funnels where it's all about volume, all about lowest cost and not as worried about you know extracting as much value up front. So it's just like way like if I were to look at the add account three years ago, way less you know campaign optimized optimization objectives live and it's exciting to see that we're able to like point and shoot our business where it makes sense with different things and you know, the values definitely been the leader there with just like what we build our acquisition funnels around. >> That makes my heart happy. >> Great, awesome, awesome. >> One of the things Connor touched on, which I'd be curious your perspective on, is not only just something like value optimization driving the type of order that you're looking for, but using them in conjunction with one another. And a discussion we've often had is like can optimizing for lowest cost as well as value, as well as layering in some of these other sort of value expressions allows to reach new customers. Is that something you guys, yeah, what's your perspective here? >> No, I mean, I think like, no, that's great. I think there's a great question. I think something you said is something I will say, which is ultimately like every business is nuanced and it's complicated. And I think the greatest thing is to be able to, you know, you are the expert at Hexport, other than I have the Hexport, it's great, great pan. But you are, you are the expert in how you value the, how you see the whole thing, 'cause ultimately like I met a, we see like a part of the picture, there's a much bigger part of the picture you see. And so it's really about, you know, making sure like how our products fit with how you value in which map and finding that mapping is the key. >> My question was like how, what Connor was describing was using different value rules in order to reach new customer? >> Oh, I think got it. >> Which is highly related to meta's prioritization of like explicitly trying to read, there's a great segue into that. >> But more like, is there more incremental reach instead of just being completely on Vio, but like having like Vio and CO side by side because you might be reaching different audiences. Like does meta know like hey Conner's, Conner's a high value customer who's kind of by the 12 piece set and Cody's only got to buy one thing. >> Oh, I mean, I think the new customer, like existing customer it's the optimizations isn't necessarily like I can make a definitive statement that like CO plus Vio is going to let you reach a new set of audiences versus just Vio. Because it depends, it depends on like audience, depends on how well your audience is educated about your product. And so ultimately the way we think about it is you know, we have a prediction system and that prediction system is focused on trying to find like whatever you tell it to say. You're really pointing at it in a very like myopic way in a way where you know, if you tell us like, we don't really care about new customer, new customer all I'm trying to predict is the cheapest way or the most valuable way to get your next sale. I think we don't have a concept as much about they like once you give me an audience to be like, oh, there's a new customer, we're gonna prioritize like a new customer versus an existing customer. It's really dependent on each impression. What's the next best one? And it just happens to be someone who's purchased before or not that you know is irrelevant. It's really just dependent on what's the next best. Does that make sense? - In Cody, you've obviously followed closely like the new releases around optimization events and things like that. What are some of the ways that you've leveraged them and maybe what are the options that you're most excited about? - Of specifically like new customer stuff or anything. - New customer stuff, value optimization, incremental attribution. I might be leading the witness here a little bit but I know some care deeply about. - Yeah, so we were part of the like new customer investigation where we met with like meta product team. - Oh great. - Yeah, I mean, just like share stuff. Cause like the challenge that we found is probably like a year and a half ago was like, you know, we were just, we were spending more and not as a reaching new customers anymore. We felt and had data like our exclusions weren't working very well. You know, we started using like a third party tool and it helped a little bit but like we would try to exclude and we would still reach the same people and there's probably a lot of like creative stuff and other things we could do. But we, you know, really tried to push meta and meta seem to take it very seriously. I know it's a big focus of how can, if brands care about new customers, how can we provide stuff for them? So yeah, we've been like, I don't know, my toxic trait is like I always like to test things early so like Jones wrote part of disruptors. So like any alpha, any beta, it's like, yes, we'll try it so there's obviously like, then you can tell us about the customer lifecycle strategy and some exclusions thing. But for us that's just like really what we care about. It's not that we don't want any spend on existing customers but we had times where it was much higher than we wanted. And I think maybe that's a, you know, maybe a flaw in our business or whatever it was but like for us to grow and hit our targets, we really need, you know, new customers. Incremental attribution was great for us. Like really both on that. Obviously we talked a ton about incrementality and we care a lot about that. So I think that was a big one. We were starting as I was leaving you on to test like on the channel optimization as well 'cause yeah, we have our own stores and we can pass that through. So there's a lot. I think it's, you know, it used to be very simple. There used to be the power five and like, this is all you have to do. And now it's like, there's no one playbook it seems like but it's like, you have to figure out what works for your business and what works for you guys and you guys is gonna be like very different than what works for us. All right, this ad is technically a job posting for NeonPixel but stay with me here. NeonPixel is hiring performance marketers. People who actually get incrementality, they understand holdouts, they understand triangulation and really that's the entire pitch because when you sign with NeonPixel, these are the marketers that are running or CTV strategy. We've been with NeonPixel at Hexclad for three years now and they really do work like an extension of our team. Live on real living room TV and days, helping with creative strategy, same day end card swaps and measurement read in our stack not just some black box dashboard that we don't really understand or have insights into. And that's ultimately what a managed service looks like when it's staffed by real operators and real marketers. So whether you want to work on the sharpest CTV accounts in DTC or you want that team running yours, go to neonPixel.co/careers to apply. - So I'm curious from meta side, it does feel like there's this shift in perspective to Cody's point, power five, relatively simple, talking about consolidation, there weren't a lot of optimization targets. Was it two or three years ago where you guys realized, hey, we don't have maybe the full context or we can't perfectly align every business's goals with meta ads in the way that we do now. So we are going to roll out these new optimizations. I'm just curious when that shift in perspective change because it feels like all of a sudden to Cody's point on the channel optimization, new customer optimization, value optimization, like now all of a sudden we have infinitely more tools. And I'm just curious, at what point did that shift and now that we're like reaping the benefits of it? - We've always thought about like, hey, we need to align the customer with where we are. I think what's expanded is we've invested a lot over the last couple of years. So anytime you see like a new product, we're likely working, [BLANK_AUDIO] I think I had the first meeting with a very senior engineer on incremental attribution was called something else internally, maybe two and a half years before we released. It just takes an incredibly long time to be able to build these new models, understand how they work, get the signal. And so it's something we're there. I think we just got the, you know, in 23, we just released a bunch of stuff. If you imagine there's been some changes in 21 that we were really working towards for a couple years as well, like with ATT and kind of rebuilding our systems post-ATT, which is a challenging time. And so I think it's just a function of like, you know, we spent a lot of time working on that. And then we had like space and bandwidth to continue like our kind of forward progress. Yeah, so a lot of brands here are really focused on lifetime value, or at least the brands that are that are listening to this show. I think there's some brands like Hexcladware. We get most of that value in the first order. I think Ridge is very similar. You guys are very first order driven. You have to be first order profit. We're in that same boat. We also, it's a little bit easier because we know that the higher the first order A/OV, the longer the lifetime value, we have tons of data that supports this. Now some brands are like very, very different. They're getting, you know, two, three, five X, LTV, like five X growth on that first order A/OV in a year or in two years. And like they need to be optimizing for a longer window of value that's outside that first order. What's like the, what's going on behind the scenes because we've talked about this with some folks at Meta that you guys are building LTV products and like, what does that look like at Meta and like, what do you expect to roll out with like optimizing for LTV? No, I mean LTV has been like a north star for a very long time. It's a very difficult, like, problem to get at. We are in the process of, you know, Alpha and Bay are testing LTV products and we're hopefully going to be able to like ship something that I think is useful for folks. Take a step back. I think LTV, like always been the north star, but the difficulty in getting to LTV is that the depth, first of all, it's like a harder prediction that the further you get from like, you know, something that's very quick and like, you know, say, click on super easy to predict relatively to an LTV is that there's a lot of space between them. And also the LTV definition for different advertisers is different. I remember like, you know, many years ago I was in a meeting with the CMO of a very big company and they're like, we want LTV and I was like, well, go out there and write the formula. Yeah. And it was difficult because it's like, oh, well, I'm not sure exactly. And so that's where it becomes like a challenge, but yeah, when you say that, you're saying like some people are saying like when they say LTV, they mean lifetime revenue and then some people actually mean LTV is like taking out costs in like an actual like lifetime margin. Let me give you a bunch three months and be with me in 12, exactly. What do I mean 12 months? What do I mean three months? Yeah. Like, you know, some advertisers are like, well, you know, after 30 days, all the extra value is actually from us because we're doing a great job upselling or we're a great job like remarketing. And so really that definition is very nuanced. And so like, and then the ultimate for a machine learning system is you need more data. You need to be able to have like a standard definition. And so that's like the hard difficult, the difficulty of balancing like the customization versus the one I've heard as well, especially for like a consumer brand is like, there's no good expression of value and an event to go back. It's like, like, it's one thing for like, uh, for an app or something like that where it's like, hey, this person logged in this many times or whatever they, they made it this page. Like, you can pass it a signal back, but it's like we're obviously, you know, selling something offline in the physical world, like how, like, what is the attribute that's able to then correlate and they're like, Oh, this person did this in the unboxing and they took the product this many times. Like, there's no way to optimize for that as well. Exactly. And you guys have to also, you know, realize is that, you know, whenever we release a product, we always are like comparing it to the next best alternative that we have. So we're constantly also battling our own cells. And so like when we improve value optimization for seven days, our, say, LTV product has to generate more LTV than our seven day value, which is already fairly good at generate LTV. So it's also like we need to make that benchmark higher to be able to build a product. But we are excited. We made, you know, we have some great products in the hopper and you know, it takes years, but we're excited about launching it. And just in terms of like a little bit to Cody's point, how a brand will ultimately express what they think of as LTV. Is that something that you guys will ultimately define and then like sort of extract from the brand's data? Or is it something that we will be able to explicitly state? Look, we haven't like finalized the product, but I can, you know, I'll just share my intuition. The intuition is that it will be a lot better for the outdozer to tell us how they value LTV than for us to intuit it. Because first of all, it's the calculation. You guys need to align with it. It goes back to value expression. Like if I think LTV is X and you know, you think it's Y, you're right, you're the customer. But then I need to know, right? And so ultimately it's better for that. And by the way, just to like back up your point about this being North scarf for a while when Jones Road first joined disruptors, probably like four years ago, like a lunch, Jake Bailey who leads the disruptors, like that was one of his main things, he was excited talking about LTV and stuff. Yeah. So it was four years ago. So it seems like it's been in the works. Yeah. We're excited about it. It's our North Star. But you know, you got, you were taking steps to get there. Most brands treat customer support as a cost, but with AI, every support conversation is now a chance to increase retention, recover revenue and grow your LTV. Which panel doesn't just use AI to answer tickets, they allow operators to run revenue experiments inside your support. You can test different winback offers on refund requests, subscription save offers during cancellations and product recommendations that turn chats into sales. Plus they guarantee you're going to save money. 50% of your support volume automated in 30 days or your money back, that's a pretty good offer if you ask me. So at richpanel.com/demo, they'll build an AI support team live for your business and show you how it can drive incremental revenue for you. I think we've covered value expression really thoroughly. One thing we mentioned earlier, and this was like the buzzword of 2026 in my opinion is in dramatta. But you can throw in gem. You can throw in lattice. You just talked about ad ranking and sort of like the predictive mechanics of that. Can you talk briefly about like what the new system is and maybe how it has improved over, you know, how ad ranking worked three, four, five years ago? Yeah. Let me get it. So we talk about value expression, so now, you know, imagine a world where all advertisers are, the value is aligned, like the way you define values, the way we think about value. That now it becomes a problem of a big machine learning system. How do we predict that value as quickly and effectively as possible? And in the past, I mean, I mean, I mean, I joined ads a long time ago because I thought this was like one of the greatest applications of AI. And we really seen that come to pass. The way I'll break it down one by one that we can get into it. So Andromeda is the first step of time determined, which ad we show is retrieval. It's like the first stage where we're saying all the universe of ads, which are the ads that we could show for this particular ad impression. And that's a complicated problem because at any given time, if we are building a system that is value aligned, there's more opportunities for us to look at. And so we need to retrieve more ads. And so the previous version of like our retrieval, you know, we're also time limited. If you think about it, we have limited amount of compute and we have limited amount of times because, you know, when people are scrolling down their feed, we have to predict like which ad to show. And so given those constraints, we would do, you know, we would retrieve a certain number of ads to say, hey, this is where we think we should look. And then there's these later stages of ranking where then you can like refine that prediction. So it's not like one system where you look at it and you predict it. You say first system looks at, you know, what are the universe of ads we could look at? Some of it is easy. Hey, are you in this country? Are you not in this country? But with automation, you would have a lot of ads. And so what Andromeda does is is a step change in ability for us to look at that initial step. I think the, you know, it's about 10,000 more ads can be ranked within Andromeda because of advances in AI than it did previously. Can I pause for questions? Sure. Yeah. So way more ads available to be shown for that impression, right? Yeah. What is the impact on the advertiser? Is there more competition? Like, do they ads have to be better because now you're competing with way more? Do you need more ads to try to get them there? Like, what's a practical takeaway to like a D to C founder who's watching this? I think the practical takeaway is that you're competing, you're in, and it's a good way because you're able to like, say in the previous world, right? We're like, you know, we, there is a concept called truncation. Like we'll like look at the top 100 at, yeah, I'm making up a number. And then we'll like the, the rest of the 900, we'll just not even look at because we have a limited time. Kind of, you know, kind of look at, we don't have time to look at the whole universe. With Indromeda, we're like, okay, now we can like look at the entire universe. And so for a founder, I think you're competing in, in for more audiences. So it's a good thing because we're able to find a better match. It's all about personalization. And so I think that's a positive thing. It's all happening under the hood. So there's no, there's no like, I would say, impacted like, accrues to you. If you have, you know, a value line campaign that's liquid, that we can look across a big audience for, you're going to actually, your ad is going to be considered for everyone. So I think that the main founder, the main concept is if you're going to like really restrict your ad account on the targeting side and the audience that so we can't reach everyone, you know, there's others that what the same Product with the same thing. Well, if you have a more value line in liquid campaign There can be like competing in a lot more auctions Yeah, it's almost like the bar is higher like you're able to show the the user like a better ad for them or more relevant ad Yeah, absolutely right. So imagine like I was trying to pick like you know I'm looking to stick a simple example if I had like you want to pick a pair you want to pick a new shirt to wear If you had two shirts like it might be pretty good it might not be they might be the two best shirts Yeah, but if you have a thousand shirts that you could choose from the probability that like the 998 shirts like you might find one that's like better than the first two that are obvious You know, it's higher and so you you you can offer better personalization for the user Which is great because they're getting more relevant ads and then for the advertiser It's great because you know, you're also you know like able to like be considered for a lot more folks Amazing, okay, so we talked in drameda Do you feel like if we could unpack a little bit more like gem and lattices approach and that I think it just comes down to that refining process. Yeah, we can talk a little bit more about how that works Yeah, so you know the first stage is I have to like you know think about a retrieval being able to retrieve all the ads For each user and not each user each given time right that's basically the kind of the next step is now you have You know 10,000 more ads to rank from how do you rank that better and so what gem does is it's a foundation model that gives us more information so the concepts of And so what we can do at the later point is that we can understand ads better and look through this universe And again generate more personalize retrieval so knowing you have 10,000 ads if you don't evolve your later-stage ranking system You're not going to be able to better serve so they really work in conjunction with each other so we can The first step allows us to look at more ads and the second step allows us to better understand them And that's what gem does is the big founder smells so we could better understand Not just like a new single ad impression, but also there's a bunch of sequences like so we can start predicting You know, not just a single what's going to happen in this particular instance, but like the sequence of actions that might happen And so it's really a cool way to You know again better personalize will we deliver can you say more on the sequences and sequence when my understanding of it is like The system is better able to understand where somebody is in their buyer journey and show them not only the shirt But actually like the ad with the shirt that's going to match where they are in their browsing journey Yeah, and so I think the the concept of Sequence learning is that like you know You want to simplify the problem in general like when you do a big ranking system You want to be as simple as possible, so then you can like scale and what The gem model does is allows us to when you do sequence learning not just predict like a single action But like try to understand like what are the sets of actions that may happen that will deliver and that allows us to deliver more You know valuable coverage. So that's what it does is like better understand the user journey to be able to then predict You know, which ad will do better and just a clear farther point further because I've always thought of Sequencing and again, this is us more or less learning we always joke we do the podcast for free consulting and this is extremely bad People are the most important part of your business and finding real a players is hard and incredibly time consuming That's why I want to tell you about propel propel is honestly the best recruitment agency that I have personally worked with They are marketing and operations specialist say source talent from us Latin and beyond for top e-commerce brands Like Jones Road like many others that you know and love at Jones Road We've hired over 12 people through them. We've hired video editors graphic designers. We have multiple retention managers Creative strategists from them what I love is not only they easy to work with they're quick But they work on performance. So you only pay a one-off fee only once you hire somebody you don't have to pay You know any retainer know ongoing fees plus every hire comes with a three-month guarantee So I highly recommend them they've worked really great for us. We have a really great team that they've helped us source I had to link in the show notes and submit the form to get 10% off your first hire But the idea of sequencer like like I'm loading up. I've hundreds of ads live in my add account And meta now has a better ability than ever to not only know like who's in market for a Ridge wallet But the fact that they've maybe already seen one ad they understand what it is And now it's actually more about a more bottom of funnel Maybe we're talking about social validation or the lifetime warranty Is that something that's happening under the hood a little bit in terms of best sequencing out the ads in the messaging that will lead to the conversion I'm looking for. I think it's trying to understand You know where you are in the Not just like in that particular ad campaign, but across all of your ad campaigns Uh, and so it's really trying to like frame up like where you are across the whole journey versus you know I think like you know you hear this concept like with full funnel like I maybe I'm misunderstanding so you guys don't tell me But like you know where you're like at the brand stage of an ad where you're like not really like trying to get some outcome So it's not at that level where it's you know, like trying to understand like a very long user legacy But it it is trying to understand like where you want the journey across You know all your campaigns and the stuff that you've done 100 percent. Okay, so before we started recording You said that you spend quite a bit of time on X Twitter, which I'm not sure like I'm sorry for our behavior Um and threads. Yeah, of course What um what so this is where we're we're always talking about the latest in the the meta algorithm and ad delivery Yeah, what would you say is something that marketers um get wrong most often what is for you also said you listen to us So what do we get wrong most off? Oh, yeah, yeah, you are That I think you guys are pretty good. Uh, I think there's an over attribution of the volatility Um, you know, I think fundamentally when you make you know harder predictions Um, you're you're gonna be you're gonna be wrong more often, but over time. It's the right thing to do I think uh, that's one two I also think there's a lot of like fluctuations in ad demand like an any given time like different advertisers come on There's lots of seasonality and I think people take those changes and then they attribute them to things that they've done And so there's this like link on causality like hey, I did this thing and Then this happened and therefore those are related and a reality could be But if there's you know, there's just like a lot of volatility um across the board the seasonality And I think the biggest thing I would say that people kind of get wrong is Is over attribution of something that happens to you know some action that they took you know I learned something earlier um, I always thought that cpm worked on like a dial I thought that you guys had to get dialed that you would just turn up on cpm and they give us turns actually a lever If there is something back there with the rich ad account just kind of no, I'm kidding But um, we we talked about this one often yeah, we just talked about it recently Cody's a part of a new brand called links and brand new ad account and I went through this experience earlier at rich We launched a new brand as well um and when we first launched ads I felt we had like an almost an artificially high cpm Yeah, and this has been an ongoing conversation on twitter recently maybe you could shed some light on whether we're just making it up Or if there's something about the ad delivery system and the auction that just Hey, it kind of has to calibrate who you're serving ads to and therefore we started this higher cpm and it normalizes over time We don't have uh, you know cpm lover. Let me anything That is a dial Um, that's what I do full time Hey, just don't like what Connor said on the podcast No, um, maybe I could do miss if I like how the prices are set I think that might be a better because um, you know, we're talking about conversions right we're talking about value optimization We're talking about very deep uh, you know um, you know Optimizations, but then you kind of narrow back to like prices right because it's cpm is a reflection of prices You know, we run an ad auction and there's uh, I'm gonna go back to eca one and one and everything as it took that but there's a supply And there's a supply of ads that we have Um, you know, based on users based on where they are how how much people use our apps Uh, and that supply is determined by humans and I think you know our supply is growing like we have, you know People are very engaged and you can see the results like people like our apps And so our supply, you know, we have a healthy amount and it's growing Um, but that's controlled by how many people kind of use our apps and how engaged they are and partially by like the quality of the ads And then we have demand and demand is like all the advertisers there And so uh, we talked about like you know, we have different DTC brands But we also have a lot of app advertisers we have lead gen advertisers, you know, people who want leads We have folks that are like movie trailers and like brand advertisers, you know coldgate png and so we have Uh, a healthy broad base of demand and you know my goal is to generate uh, you know to build products that work for everyone Right and so you know DTC is a huge um, we know we really love our our our advertisers there But we also have you know omnichannel we have retail advertisers and we want to build great products for everybody And and that ultimately is the demand that where we have and the way our auction works is we're predicting like how much value We're generating for each advertiser and and ultimately if I'm doing my my job right we're able to build great products for every advertiser And that sits in an auction and and and the more demand we have in an auction the more liquid that demand is The more people are competing for the supply of ads that we have and the way our you know the way our auction works is we want to allocate the the ad impression, to the place that we think is going to generate the most value, and I talked about it. Recently, value is consists of advertiser value, like how much we'll get, but also the ecosystem value, the use value, and we didn't talk about that too much, but I think that's a really big component. And so we're trying to do maximize value, and the pricing is set by like, we always want to set by how much competition there is for that ad impression. And so ultimately, that's how prices are set, and then it gets filtered back down to like a CPM, you could see that as like a CPM. But we're, and so ultimately, you know, we're not really, you know, the thing I focus on, and I think most focus, you know, on my team, are generating the most amount of value for everyone. And the prices are just like an outcome of this market mechanism, of how much supply we have, and how much demand is in our system at any given time. And together, you know, you can, you don't have to remember, you can't 101, like, you know, where supply is, where demand is, that is the market price. And that's kind of, and it's more, when we're complicated about that. That's how CPMs are set. So at any given time, like, I definitely think like, our predictions get better, and then, to more directly answer your question, when you have a new brand, like we have predictions, and you know, the more data we have, especially with the sequence learning, we can learn across a broader range of data. I think we get better at predicting. The newer brands, like, you know, we're learning, and so we have less data. So that might be a function, but, you know, we've seen, you know, I've been advertised with like new brands that also do well, so it's really just a function of that market mechanism, versus like, hey, there's like a, you know, it's a new advertiser, let me, you know. - We talk a lot about CAC on this show. You can't control what metacharges you, but you can control what happens after someone lands on your site. That's where our sponsor aftersell comes in, helping you make every visitor worth more, from cart offers, and one click post purchase upsells, to subscription upgrades. And here's the best part, rock thanks turns your confirmation page into another revenue stream with cross brand upsells, with earn and save, those earnings also offset your aftersell subscription dollar for dollar, and you get to keep your entire payout, head to aftersell.com/earn-save to see what aftersell could unlock for your business, and how you could get the entire software for zero dollars a month. And, I mean, just to the point of a new advertiser code, you just went through this experience, but it's not as if the CPM's remain cost prohibitive. It did feel as if it were learning phase over time that then normalize. - Yeah, and it feels like to me, without like knowing what's behind the scenes, it's like, as we introduce more positive signal into the account, that we think is gonna convert better, be like a strong signal to meta, it helps that prediction. For example, a lot of people say, whitelist thing or partnership ads can help, obviously there's signal that it uses from the creator, and, you know, those have really helped us. You know, we've verified our business manager and not brought him down quite a bit. We haven't done it a ton, but like, I've heard people like, they've had their founders go on podcasts or stuff like that, just like organic traffic, right? Like, people tagging us on social. Like, I have no idea if any of these are actually important, but it's like, my philosophy so far has been like, let's just introduce more positive signal into our account, into our business, and it's not gonna be a negative thing, and we have seen CPM's come down with that. - Yeah, if I can add, yes. - Yeah, I think, the thing I would say is, obviously the more data we have, the better prediction, so we're not kind of, if you think about it, we're not showing too many false positives, we were like, oh, with this person might like it, they don't, and then, you know, our predictions get tighter. So that's absolutely, so signal setup, sending data is really valuable. Using somewhere our best products is also good. On the flip side, I also think, you know, over time, there's also this organic conversion, right? So like, what we deliver is we can deliver a user to, you know, for a DTC website, to a website. Ultimately, if you're doing a great job of having, you know, a great ecosystem where people trust your product already, people look at creators, and they're like, oh, I trust this person, that conversion happens, that's the credit for the, you know, that's the business's credit. And what we do ultimately is like, if there's a lot of like, say like, intent that's already sitting in the ecosystem, that's great. That means, you know, we're generating value that's like users like it, it's easier to sell. And so we have to, you know, go back to conversions, we can just, you know, give you more conversions for the same number of dollars. And so there's that other piece of it too that people sometimes undervalue, which is like, if you have a great product and people are buying it, it's easy for us to like, buying people to go buy it. - Yeah, and we've seen even like, when we launched TBI, so the first time, like meta-performance significantly increased, we care about like, we look at like new visit percentage rate and like that improved. Like I think it was just like very positive signal. So we've seen that. And even from like brands I've talked to, like the only few brands I talked to have that have not had the CPM measure, like they were taking organic social, like pretty seriously, like kind of started there. So that's what we've seen. - Yeah, and people are familiar with the brand. And I think about like, when you're schooling like a feeder, you're like, you may be kind of interested, but you recognize the brand, or you've heard good things, like you're gonna click on the ad more likely. And if you click on the ad more likely, and it's a good great product, you're gonna like purchase it more like. - And I think it isn't the correct name wrong. I think it isn't the documentation of maybe gem or something like that, where organic is actually like a signal in the algorithm. I've seen that like in the delivery system. - Yeah, I mean, like when you look at user-journaly, we look at all the signals available to us to be able to predict that, right? I mean, everything that's, you know, we're able to access within a policy. - So in that example, that Cody gave, he just went through like a laundry list of things that he's been doing, and he's saying, I'm doing acts that I'm seeing my CPM come down. And that situation is he being like, let's say they unlock, they roll out a new white listing at it, it has a great CPA way better than the account wide. Is he now getting awarded better CPMs to the same audience? Because you got your system saying, oh, Winx is now providing more value to this audience, because they're providing more value, they're gonna need a lower CPM, or are they reaching a different audience? And that audience is just a cheaper audience to reach than maybe the one that he was paying for a month prior. Like what's happening in that situation? - It's hard to determine, like again, going back to it, it's really like, there's not a system here, like oh, we're gonna find a more expensive audience, and we're gonna find cheaper audience. It's a more simple function, right? Like what's happening is we're able to find more sets of folks that we make predictions. So if you remember what I was saying, we were trying to find, based on the optimization, the next cheapest of your value optimization, we're trying to find the cheapest roll-ass, which is the most revenue divided by the cheapest price, and the cheapest customer we want to do is be able to find the most amount of value, right? And so what Cody is doing with all the work, I think creates, I think more signal for us to predict, but also more, the audience, the users, more demand, more intent for that. And so that's what's happening. And so it's less about like steering the system towards like this audience, which is that audience, I think there's like a misconception there. It's more about, there is just generally more positive signals. And so we can predict better for a broader set of audience, and there's more, you know, and it's really a credit to the businesses that you're building a great product that people wanna buy. And I think the way our CPS work is, we're not like, again, not a CPS job, but it's like, hey, going back to the market mechanism, I wanted to say, I wanna generate more value. I think it's easier, I think we get better predictions. We're like, oh, more people want this product, so we think we can generate more value. And it's cheaper to do that because we're right. And then once we're right, we charge whatever again, based on like the demand is over that system. Feedback cycle. - Yeah, exactly, and so it's great. Like I love it when people are like, hey, I only, you know, where a few people tell me this, and they're like, I, you know, it only cost me, you know, X, and I get 10X the revenue. That's great. I have no desire at the point to like, generate, you know, like, you know, the, I think that's a great place to be. I wanna do that for everyone, right? What I wanna do is ultimately, if I'm doing a great job, then there'll be more and more advertisers that would see that big value. Is that makes sense? - Yeah, absolutely. - Yeah, I think I just wanna be clarified that it's not like, there's a CPM. The CPM is an outcome of the other pieces that mark a supply and demand and demand being an outcome of like how great you're at it and how good the user value is versus like, you know, what the CPM is. - You guys are hard to hear, very clear, no CPM dial. - Why we're here in person investigating. But if you, so if you have like, let's say you're an advertiser and you have 10 ads live versus 500 ads live, if you have 500 ads live, then theoretically your chances of reaching more audiences are way higher than if you only have 10 ads. And if you're reaching more audiences, then chances are that you're gonna be reaching audiences that are maybe less expensive to reach than the audience you're hitting with 10 ads. Like is that like, going back to like Andromeda and Creative Diversity, like would you say that that does contribute to getting better CPMs? If you have more ads, speaking to more people, and if you have, if that's the case, then inherently you're reaching different sub-pockets of audiences in Meta and like, would that help drive your CPM down? - You know the difference between hitting your numbers and missing them? Clear signal on what's actually driving growth. It can get really, really noisy. There's so much noise, you got platform data, you got blended data, MMM, all the acronyms, MTA, experiments, all of it, all pointing in different directions. The more you're spending, you move, the more the bad signal can cost you. That's why we use house and we've been using it for years. That's why the other marketing operators do as well. They're the best tool on plan earth for measuring what we call incrementality, which we talk about a lot on the podcast. What is the true impact of your advertising dollars on your business? We have causal MMM for channel level budget calls, causal attribution down to the ad level, and architect their AI agent tells you exactly where your next dollar should go. And the results speak for themselves. StockX saw a 41% lift in IRO using house and you're not stuck with a help desk. You get an embedded measurement strategist who actually helps your team make better decisions. Their whole team is great. We've worked with a lot of them. They are world class air. Go to house.io/operators. H-A-U-S.io/operators and start backing your budget calls with real causal data. I think creative is a big lever. I think having a diverse creative. Again, let's go back to what we were saying. When you're value-optimized, you can then be more liquid, but creative is always something the advertiser will own. And so having more creative, more diverse creative is generally a more effective way for us to be able to deliver. And so the way you should think about it is if you have more effective creative, say everything is the same, but your creative, more creatives to choose from. Our system would be able to better find the right creative for the right user at the right time. Yeah. One of the things that stood out to me from the Meta Performance Marketing Summit was how Andromeda and the new ad delivery system had led to an increase in relevancy scores. And is that just another way of saying like we're creating a better user experience, user value, and therefore theoretically, you know, the advertisers have more diverse creative. And I don't want this to focus on creative, but like is that relevancy score sort of a metric that we could be thinking of as user value from the way that you're describing it? No, I would think about it as advertiser value, right? And so ultimately, like, you know, for any given, you know, the best possible outcome is I show one ad impression and I make the sale, right? But, you know, that rarely happens. Like we generally have to, you know, show multiple ads, the same person, but also adds to different people. You know, advertisers are still learning this concept, though, that it takes more than a single ad and a single campaign to drive budget. So yeah, I mean, yeah, good good point. Yeah, I would love to one shot it. Yeah, but we're not we're not we're not there yet. And so ultimately, what, you know, all these ranking systems do is allow us to better predict, right? And so the relevancy score is we're able to do those predictions more accurately. So we can drive more clicks for the same ad impression. We can get more clicks out of it. And for, you know, every click with generate more conversions out of it. So really think about it as, you know, we can basically use less ad impressions to generate more value. Yeah. And I do think there's a the secondary effect of generating more, you know, user value because they're seeing more relevant and ads. But I think it's primarily like a positive to the to the business side, to the advertiser side, where they can get, you know, more relevant ad delivery. You can get the same outcomes with less impressions. Talked about value expression. We've talked about the new delivery system with Andromeda, gem and lattice. And, you know, for a long time, and we've spoke about this at length on the the podcast, for a long time marketers would sort of like toil away within the ad account, making all these optimizations. And frankly, you could create a lot of value that way. I'd love for you to speak through your perspective and meta perspective on what the marketers role is today with these new tools. You know, I think the markers will is huge, but it's shifted. I think in the past, it was like you said, manually tuning a lot of the knobs, making sure the output is matching how you see value. I think in this new world, it's really about understanding your business, like what is really driving with incremental like revenue, the incremental value to business, making sure that is aligned with how meta is pointed. So you got to steer a system towards that alignment through incremental vision, custom attributes, your value optimization, value rules, LTV, and just really mapping that together, making sure that signal is strong, making sure there's all aligned. And then there's also a huge piece on the creative side as well, making sure there's now that you have a huge audience to go after, making sure the creative is relevant, creative is fresh. And so it's really around like moving away from a manual tuning of knobs to like telling us the broad picture, making sure we have the right assets and making sure that we're delivering what we want, what you want. 100%. And I think this is a really exciting future. Connor, I'd love to hear you speak about with this idea, which I think we've all been aligned around for a few months now, like where are you and marketers at Hexclad spending your time if you're moving a little bit out of the ad account and the day-to-day tuning to something that best aligns, you know, the Hexclad business with the metadata platform. Well, I think what's cool about what everything we're talking about is that like all the same levers from five and probably longer or 10 years ago are still the same levers today that really move the needle. It's great creative, it's great destinations, and it's great offers. And I think that's still where marketers need to be focusing their time is like what's the next big creative swing that you're taking? I think that at least what I've seen that's still in the in the human's hands of like creating those those big swings, those net new productions, and I think the creative gen stuff is more relevant for like iterating on stuff that, you know, a human power team has made. So I mean, that's where we're focused on is like what are the products we want to be building funnels around? What are the best ways to explain these products in creative? What's the best destination to send someone to? And like what's the offer on that? Like those are still the same levers that we're focused on. And that's what's allowed us to scale outside of just like the hero funnel that we had, you know, two, three years ago. Now I still think there are some things that we're doing to make sure that we are aligning with what Meta wants from a signal perspective. You know, we obviously have like a capy set up, you know, we're using event match tools. So we're giving it getting like the best event match in Meta, like, you know, of our type tool. And then as far as how we organize our ad account, I do still think there's a lot of importance there. Like I'm not going to go and, you know, we've talked about this last week or Japanese product, right? These are 130 dollar products. I'm not going to go and drop those ads under my value, optimize 12 piece set that has a million dollars put through because I just don't think that that's really setting that up to go find that new customer for this new product funnel. The best, you know, we're that that ad account or that that campaigns trained on a $700 set buyer. So I don't really want to drop ads promoting a $130 product in there. So I still think there's some like alpha in that with how you decide to set up the the account and the campaigns to make sure that you're passing the right signal back and not confusing buyers. But I still think that like 90% of of where our energy is focused is on the ads, the landers, the offers. And that's the same thing we were focused on in 2020. That's the same thing we're focused on in 2015. It's just that the technology's better. So now we're getting even more, you know, you talked about like jam and lattice and all these behind the scenes things like that's nothing that the the marketer has to do, right? We just need to be giving it really good inputs. And now the the ad tax even better. So that's how we're thinking about things. Again, I think I think there is a lot of importance with how you structure. I mean, we've spent a lot of time. Do you do like separate campaigns, separate ad accounts, separate pixels? I do think there's a lot of ways you can go about that. And that's important. But yeah, those are those are the things we're focused on. And you know, it's working because to catch is a happy customer. Exactly. Right. He's got that hex code lever going. All right. Before we jump your code to anything you want to add on this this last point, that's a great one. Fantastic. We'll end there. So catch thanks again for coming on. It's a pleasure.