Season 6, Episode 6: What is signal engineering? (with Itai Kafri)

So you measured incrementality. Great. Now what? You know what worked? What didn't? Maybe even what was cannibalizing your organic. But turning that insight into actual decisions? That's where most marketers get stuck. Incremental doesn't just measure incrementality. It helps you operationalize it. Shift budgets. Pause wasteful campaigns. Double down where it actually matters. No tracking. No experiments. Just always on automated truth. So if you're done measuring for the sake of measuring and ready to act, visit Incremental. That's INC-R-M-N-T-A-L dot com because knowing is only half the battle. The problem is that the distinction needs to be drawn between the competence of the economists and the correctness of their analysis. Hello and welcome to the mobile def memo podcast. I'm your host, Eric Sufert. And I'm joined today by ETAI Coffee. ETAI. Welcome to the podcast. Thank you. Is that it would be here? Well, I'm excited to have you. So I've been talking a lot about signal engineering lately. And I'm always remiss when I talk about it to not point out the fact that I sort of cribbed that term from voyantis. So that's a term that I learned or adopted from voyantis, which is where you work. And I'm really excited to talk about it today because I do think it's sort of an aspect of marketing, of product marketing, of growth, you know, whatever discipline you want to highlight that is being under invested into and is generally not recognized as being as important as it is. So that's what we'll talk about today. But before we get to that, why don't you introduce yourself, introduce voyantis. You're the second person from voyantis to be on the podcast, but feel free to introduce the company and give your sort of background in space. Oh, perfect. So yeah, my name is ETAI. I lead the product growth at voyantis. This is an industry leading solution for LTV optimization and we do this through signal engineering and you know, we gained the trust of investors like Intel capital and others and raised over $60 million so far. And is really to help advertisers drive a significant double digit uplift if possible in their performance by optimizing the signals that are sent to the ad platforms. And by that, we basically retrain the ad platforms delivery algorithm to target better users rewind a little bit and give a little bit background on myself. So I'm going to add a little bit of context into why I'm such a strong believer in signal engineering. I joined at tech somewhere around 2013 started following you back then. And interestingly enough, my career path follows the evolution of the industry very closely. So I spent about five years back then in a platform that did automated bidding. And it's a thing right back in 2013. And it's because meta and other you know, and Google and so on. They offered CPM CPC CPI. Those didn't really deliver our pitch back then was well will translate your row as goals your CPA goals deeper funnel into the CPM bid for you to be able to get better performance. And you know, the campaigns were then very small audiences. They had a ton of ad units and it couldn't manage us manually. So it all made sense. And then fast forward in 2014 met the release of CPM. And that changed everything because that was really the beginning of a full automation for the auction. So O CPM is basically met us saying, we will, you know, bidding is is CPM bidding. But you can now tell us within the audience who you want us target. And you do that by optimizing towards something that you want the user to do. And we will do the bidding on your behalf. And so basically that changed the way the industry works because now we're starting to go deeper and deeper into the funnel in order to drive better users. So it started with O CPM for installs on on the app side and then it drill down into in happy event optimization and so on and so on and row as and so on. The intuitive explanation of O CPM is meta is asking who described a good user for us. And we will try to find that user within the audience pool. That worked great. Right. Now when that happened, the advertisers are now looking at their list of campaigns, you know, what's the next big move if the auction automated and creative became the king. And so, you know, I worked for the next five years in a company that did creative intelligence very sophisticated, but then fast forward again, you know, with uac advantage plus he max dump all your creatives into this one big bucket alongside with creative, you know, with a gen AI. The focus then change and shifted over time from creative intelligence to creative automation. So again, we're in the space of automation. And then I moved to Tik Tok and this is really where my signal kind of career begins because in Tik Tok, I live a product strategy for app signals. And that's where I really learned how the back end works, how they collect all these signals, build the signal tower user tower signals app tower signals, put it all together. And that powers the ad delivery. So now circling back to my current position, if you consider this evolution process to the point there it is today where you campaigns, very big audiences, everything is automated by the channels, all their decisions on who to deliver the add to and at what costs are all based on the signals that they collect. And the more black box their automation becomes, the more important it becomes to feed the machine with better signals. And basically we're trying to help their automation help us. Right. And that's the new industry called signal engineering. Maybe we could just hover there for a second. So thank you. That's a great sort of summary of the developments to date. Right. So okay. So if you think about the standard configuration here, the relationship, the touch points between the advertising platform and the advertiser. Right. Let's start with the app use case, I think because that's more straightforward, but we could also move to the web use case to, but just starting from the app use case, so I've got an SDK integrated or, you know, I'm just doing copy, right. Let's assume I've got an SDK integrated. Now I've got a number of events that are instrumented right that are natively instrumented and I'm matching those to the standard library on Facebook. That's not signal engineering. That's just using the standard library. Talk to me about what signal engineering is like say some of first principal standpoint. I want to actually engineer the signal. I don't want to just send the stuff that they're expecting to receive. I want to actually engineer signals. Talk to me. It's just in the abstract as a concept. What does that mean for an app advertiser? And we could also talk about web advertisers too, but what does that mean for an advertiser to actually engineer the signal versus just instrument the standard library. Great question. I'll answer the single answer for both web and app because it's really not different when trying to answer what is signal engineering signal engineering refers to using the ad platforms existing signaling APIs, which are designed for deterministic events by definition they're designed for deterministic events. So we're using those to send probabilistic predicted outside of the box information signal engineering as we define it and voyantist is a twofold process one side. It's predicting the user's true value for the organization and that can mean a lot of things and we'll talk about that later probably. And on the other side, it's how do we take that prediction and translate it into an impactful signaling strategy. So in order to put more context into this, I'll ask the question of why it's even needed right why do we need to go into this sing signal engineering space. So because it was a twofold solution, the why is also a twofold answer. I'll start with a prediction when you select your optimization event and you can use those standard events that you mentioned or custom event doesn't really matter. You select the event that you want to optimize towards. As I said earlier, you're basically describing what a good user looks like. And that's exactly what they're asking you to do and then they'll try to chase down those users within the pool of audience, but it has a caveat. The description has to be based only on what the user did within their initial journey within what's called the conversion window, right, which is typically seven days from click. So anyway that you want to describe a good user, it has to be based on what the user did within their first seven days after they clicked on the app. So that leaves way too much variance and user value and let's give some examples right is a good user one who started a trial because many apps that have a trial period optimized or start trial is that a good user. Well, it's definitely better than one that doesn't, but no, not all users who start a trial will convert is a good user one that does convert that subscribes. Well, not necessarily because many of those customers will turn within the first month is a good user one that makes a first purchase, which is another very common optimization event. And then some will never come back for the second, so it leaves a lot of room for variance and what we're looking for is what is a good user not necessarily what they did in their first seven days. So signal engineering that first portion, the prediction is basically saying we need to fit the description of a good user within the conversion window typically again seven days. So in order to do that, we try to predict which user or what is the probability of a user to buy to stay or to spend. And we. basically take the entire user journey and condense it into that conversion window. So that's the first answer to why we need to signal engineer. The second piece is once we have that prediction in place, we then need to translate it into a signaling strategy. And you may ask yourself, well, why do we need to translate? You have a prediction. A user is going to be worth $100. Let's take a purchase event, place $100 value in it, and send it to Google or to Meta. Well, that doesn't really work, unfortunately. I wish it did, but it doesn't really work. The ad platforms evolved over the last 15 years or so to operate based on deterministic conversions. And probably, when you introduce probabilistic values, you need to carefully package them. It's not the same. There are no best practices. There are no playbooks. There is no documentation in their API docs or no official recommendations on how to do it. Why is it different? Well, deterministic events have a real time. There's a timestamp of when that happened. They are 100% accurate. If someone came into my website, bought a TV, worked $500, I know exactly when he bought the TV, and I know exactly the price. And I can send that signal as close as possible to real time, curl requested by the channel's API documentation. But with a probabilistic event or with a predicted event, first off, there's no real time that doesn't exist. And it's never accurate. It's as close as we can get it, but it's never 100% accurate. And now comes the question of, if they want it as close as possible to real time, the subtext is as early as possible. We need to make a decision here. The prediction, if I take a single user and predict their value one time, and then again, and then again, every day or every hour, the more I wait, the more engagement the user will have my platform, the more accurate the prediction will be. So I need to decide, because there is no real time, on how soon can I send that signal, where it's confident enough to drive the right impact that I want it to drive. And then there's more, right? If a user is predicted to be worth $100 and 30 days, that value will accrue over time. In my signaling strategy, should I reflect that by sending three or four signals of, you know, five signals of $20 value each, or should I send it all in one signal of $100? So basically what I'm trying to say is that the question of how do you send a signal for something that never happened is a big question to answer. And there's no best practice exactly on how to do it. So it requires a lot of experimentation, which we've done over the past, you know, four or five years, thousands of experiments to get it right. And we're still continuing to do those experiments as the channels evolve and as more use cases come up. That's great. Maybe we can hover for a second on why it's a bad idea to just insert the predicted LTV into a purchase event right away. That tends to not work. I think there's some waiting that's hidden to us around the way that the platforms perceive that, probably based on just other conversion events where they can recognize that it's not possible for someone to do like a $100 purchase within five minutes of having clicked an ad. Why does that not work? Because I know a lot of companies do that or have tried that. Let's just predict the LTV and push it into a purchase event. And then it looks like they just purchased $100 and they're $100 in that purchase, whatever the case may be. And then that'll be how Facebook or Google or Snap or whoever values them. And so they'll start, they'll just disproportionately target people like that. Well, I'll start by saying it sounds like a should. It really sounds like a should work. Before my time in Voyantus, you know our CEO and founder told me that that's how Voyantus started. They had a very strong prediction engine and they just sent the predictions as is. In some cases it worked and some it didn't. And when it didn't, I'm not, I'm not referring to it, brought a smaller uplift. It sometimes actually brought worse performance than the BAU. So it really didn't work in some cases and it didn't some hit or miss. And that caused Voyantus to go into this a whirlpool of understanding why. So I have some answers now, right? But before I even go to the examples, it starts from what people really tried. So a few months ago, I was invited as a speaker to a meta event in Warsaw, LTVision workshop. Now, LTVision is Meta's open source tool to see if you have correlation. Like to basically see if there's an opportunity for predicted LTV. They asked the audience how many people in the audience have somewhat of a prediction already in place within their organization. I didn't count the fingers, but it was about half of the room raised their hands. This is a workshop for LTVision. So they're analysts in the room, half of the room raised their hand. And then the next question they asked is how many of you successfully implemented a PLTV optimization strategy and zero hands were in the air? Absolutely zero. So the reality is it doesn't work. And now we can assume or guess why. The first guess I have is that when you build a prediction model, in many cases that prediction model takes time to mature and advertisers send that value a little bit too late. It doesn't have the right impact. Another consideration is the exact opposite. They try to predict it too soon and it's just not accurate. Prediction is just not the right prediction. But then the third one is, you know, I bring this from the world of bidding because I've been there in the past, when you want to bid on ads and back in the day, you know, CP, MCPI, BIDS, CPC BIDS, if you would go to bettine, ask them, what bid should I place? What bid value should I place? They would tell you place the bid that actually represents the value of the event that you're converting. But everyone knew that's not the way to do it. You had to play around with those bids. You had to increase them, decrease them based on performance. You had to really optimize. And I think that signal engineering is the same idea on the signal. We are taking a value that is a predicted value. We're taking a value that we're basically inventing and we're saying, how do I send this in the best possible way to drive the highest impact in a world that has no documentation? And that's where the example that I gave earlier, right? If that user is predicted to be $100, I can send it as $100 value. But what happens if I break it down to $100 signals of $1 each? Obviously, you know, that's extreme. I won't do it. But just for the sake of the example, the behavior of the platform is going to be different. And so it's really a matter of the opportunities that I have into building a strategy of signals that works. By the way, there is a world where you can actually not send a value at all. You can use volume of signals to represent a value of a user. So a good user, let's trigger five events, five conversions, a bad user, let's trigger one conversion. And that's also a world that works as well. So in some cases, we actually lean into that. So building that strategy is essential if you want to ensure success. And I imagine that the strategy is different across platforms and across products. I mean, different advertisers would benefit from different strategies. And there's like a matrix, right? Probably by Ecom or even type of Ecom product and then platform. Is that correct? 100%. I don't know necessarily if it's the vertical, more about their business model. And I can elaborate more about that in a moment. But the channels are very, very specific as well. So Google rewards early signals even more than Meta does. On Meta, on the other hand, if there's an update to the prediction, you can update that prediction only upwards. So you need to be very careful with how you send the first value for that user because you can't decrease it afterwards. So these nuances are also extremely important. In Meta, there is an option to backdate an event if you want to send it. And in ad manager, you'll see it backdated, but the delivery algorithm doesn't consider the backdate. The delivery algorithm only considers the time that the event was received. And so these nuances are important to know per platform. And that's why signal engineering is different per platform and different per use case that the client wants. Most apps react to historic in-app behavior. But the context platform helps you respond to what users are doing in a specific moment. By using on-device AI and detecting real-world signals like motion, device state, and time of day, the context platform helps apps understand user intent and trigger offers when users are most likely to act. One client saw an 80% boost in conversions just by getting the timing right. Learn more at contextsdk.com. That's contextsdk.com. Let's talk about the amplified importance of signal engineering in light of the growth of the total automation platforms. I'm talking about Advantage Plus, ASC, PMACS. Why has this become so much more of an imperative as advertisers shift more budget to these platforms? That's a great question. The background that I gave on myself kind of already gives a hint into the teaser for this question. The more automation the platforms take on themselves, and that's a great thing, by the way, because we've seen how performance improved over the years with these automations. But the more automations they add, the less levers an advertiser has to control. In other words, if you want to optimize your campaigns. And you don't have a lot of buttons on top of the auction, then you need to start considering how you feed the auction. If everything is automated and it's all based on the signals, then you really need to invest more in how you signal to the platform. >> With these platforms, when you consider the breadth of data that they have, right, and the sort of rapid experiment. So like, to my mind, a lot of the value from these platforms is not the underlying models per se, because I think the models probably haven't changed too much. I think the value is that they just experiment much faster than the marketing team could. Right? It's just real-time, total exhaustive experimentation with everything, right? And so that allows for more creative to be tested, that allows for more placements to be tested. And it also just ignores or doesn't even consider in the first place, while any of the preconceived notions that a marketing team would have about who their audience is, right? That's the real limiting factor in a lot of cases. It's just like, here's the audience we're going after. And what a system like ASC or PMX does is we don't care about groups. We go after individuals, right? I mean, go after is a weird way to phrase it, but we target individuals based on behaviors. We don't target groups based on groups, averages, or anything like that. And so what I'm doing with an ASC or PMX is I'm targeting the user through their apparatus that is most likely to be very relevant for my product or my product must likely be relevant for them. In that context, right? It's really about the individual person and their preferences and their proclivities and their sort of taste, right? And in that sense, what we're doing with signal engineering and tell me if I'm wrong here, is just getting sort of like a more distilled piece of feedback, right? Then we were when we were just targeting broad groups of people based on their average behavior. Because within that group, we didn't really know who the individuals are that were generating or the system didn't really know or couldn't care that much if we're just targeting the groups, who the individuals are that were providing the most value. But when we actually are just targeting individuals based on preferences, then that really matters. And so those differences matter, right? Is that right? It kind of, but I want to crystallize that a little bit if you don't mind. So when you build a that PMACS campaign, you're selecting a very, very broad audience and you're telling meta or PMACS is Google or telling Google, within this audience, I want to optimize towards a certain event, towards a certain conversion or a certain value. So you're telling them, if I take that value to be unoptimized towards a first purchase and FDD, something in that nature, then what they're doing is they're looking at the entire audience and they can do a user level bidding, right? Which no one else can do, like no advertising can do. Only they can do that. But they're looking at the entire audience and they're basically, you know, simplifying this a little bit, but they're basically clicking this little sort button to sort all the users within that audience to who is more likely to submit an FDD and they are delivering ads to those who are more likely to deliver that FDD. What we are doing with signal engineering is we're saying not all FDDs are equal and value when it comes to the user value. Some will deposit a small amount, but will eventually be worth a lot for you and some will deposit a high amount and will never come back again. So what we're trying to do is saying, instead of optimizing towards that FDD, let's signal something different. Let's signal a the true value of a user. Let's look at this user. Let's look at this user's behavior within their first hour and their second hour and compare that to your entire user base and everything they do and every click they do within your digital assets and look at what those good users, those that retained for a long time and spend a lot of money, what did they do in their first hour and their second hour and their third hour. And now when a new user comes in, I can predict their future value. And instead of telling Google optimize towards the first purchase, I'm telling Google optimize towards high value users. And when I say value here, I'm trying to figure out what the value of a user is in the same way that the board will do it and your CFO will do it. So it's not a marketing nuance to say, okay, a user that for FDD is a higher value than the start trial. It's not about that. It's what is the true value of that user for your organization? And let's predict that for you and then let's send that as a signal to meta and then we align their bidding algorithm to your business goals. Yeah, and you made a very important qualification there that we should highlight. It's what's the predicted value of this user to your business, right? Because any given user might have different value propositions to different businesses and so that's really what's important here. So again, because someone could be a high value user, quote unquote, because they spent a lot of money on e-commerce, but that doesn't mean they're likely to buy your shoes, right? So that's another piece of this expected value decisionings, expected value calculus that happens in the auction is that is this person likely to connect with your product and that's becomes much more of a consideration when you're actually targeting towards individual people versus just targeting broad groups, interest groups or demographic groups. Yeah, and it becomes even more complicated because even in the same product, you know, different vendors like I'll try to give a real example here. We work with multiple different food delivery apps. Okay, so I'll use that as an example because they view the world differently. One of the major food delivery apps here in the US, their goal is to optimize towards LTV over a 90 day period. So they want users who submit orders in high value within their first 90 days. That's what they're looking for. It can be a single order very, very high value or it can be a magnitude of different orders that all together are very high value, but they're looking for high value 90 days on the contrast to them. RAPI, which is, I don't know if the biggest or one of the biggest food delivery apps in that America, they view the world differently. A good user for them is one that makes multiple orders within their first 30 days. That's how they define a good user. As an organization, they want repeat users. That's their focus. And so they're looking for users that will make at least three orders within their first 30 days. And so it's not only about is this user going to communicate or connect to your product, it's how do you in your organization define a good user? And you'd be surprised how many times I asked marketing teams, you know, so what does a good user look like for you? And they don't really know. They don't really know to give an answer. So we need to dig into the data a little bit, you know, ask them what's your churn curve look like and dig into it in order to get a clear answer. And then we find what can we predict that we'll move the needle in the best possible way to drive the best success. So your board and your CFO and your management is happy. Yeah, I think that's really clarifying. Thanks. Thanks for for, for taking that in that. So that kind of is a good segue to the next question because you talk about again asking the marketing teams, you know, what's the value of these users? So who within the organization is best positioned to manage the signal engineering process? Is that a marketing job? Is that a product job? Is that something to see a foe should be doing? Is this a cross functional thing? Who should be managing this? It definitely is a cross functional, but the driver within the organization is the growth marketing in almost all the cases, but it does take a cross functional team to drive success. So the resources that are needed in order to implement this include data science or at the very least a senior analyst, a real senior analyst to build a model. And then you have to have a product manager that is, you know, has the specialty of understanding the APIs nuances very, very closely, very intimately. And obviously you need R&D to put it all together into action. So it's a team effort. In some cases, besides the R&D, the marketing team will have, you know, some of these resources, the engineering portion of the signal. So, you know, I'm separating between the prediction and the signaling. The signaling is where most of the organization falls short. They think that a prediction is good enough. And we've had situation where they come back after they tried to do this on their own because they missed the engineering portion of the signal. And so how do you, like, on your side, right? So as the platform that enables this, like, how do you guide a company from zero, right? Like, what is your sort of recommendation to get started here? Do you do the heavy lifting? Or I mean, I guess, how does a company go from hearing this podcast to implementing this? Great question. You know, the easy answer for me to be contact voyantist and we do the heavy lifting. All you need to do is give a sex to data and to add accounts. And we will do the analysis. We'll come with recommendations on what we can predict. You know, we'll do a lot of consultation there and build a prediction model and then do the AB test and prove that it actually drives value. And then you can decide if you want to work with us or not. But I want to also give the audience here an answer that they can at least think of how to approach this as an entire project, right? And really, there are a few steps that you need to take. The first one is you need to understand the gap between what you're doing today and what you're optimizing towards today and the true value to figure out if there is an opportunity to use BLTV, but decide how to go into the project itself. First figure out what is the opportunity, right? And so that's where you want to look at the correlation analysis between what are the signals that we're sending to the channel today and how correlated are they to my LTV? One small example. It's very common in gaming that D7 revenue, which is an available signal. You know, you need to engineer it. You don't need to do anything. D7 revenue is highly correlated to LTV. So a user that drives a lot of revenue on day seven is likely going to have high LTV and a user that drives a low value in their seven days is going to likely have a low LTV. If that is the case, then a prediction as much as it sounds interesting won't really change the way they deliver ads. So the first thing you want to do is understand that gap. And there are a few analysis you can do. You can mean versus media and LTV. What is the spread you have of LTV and then the correlation between D7 and DX, whatever it will be 90, 180, whatever you choose there. So that's kind of the first analysis. And that will give you some kind of indication on the value that you can gain from a predicted strategy. You can also use LTV vision, which is an open source by Meta. So that will help as well. And it visualizes the data very very nicely. The second step would be define what a good user looks like. Right. If you define a good user by repeat purchases or by high value purchases, it will rank the users differently. And you know, this is a ranking gain. We want to rank the users based on what's important for the organization. So again, first step was to identify the gap between what you're doing today and what you can do with a prediction. The second is to identify what is a good user really to define what is a good user. Crystalize it. And then just be sure you have the technical readiness. So you want to make sure that you're a database. You know, you're not overriding data. The data is not mutable. You have access to all the match keys. Everything is in place to be able to build a good prediction model. And once you're done with that, the remaining portion is really the data science piece. Build a model and start signaling. You ever run a holdout test? Wait between two and five weeks for the results. And then realize you just wasted a big chunk of budget? Yeah. By the time the data comes in, the campaign's over, the money's gone, and the insights basically post mortem. That's why incremental the always on measurement solution exists. They give you always on incrementality measurement. No experiments, no holdouts, no delays. You get answers while the campaign is live. So you can actually make changes, not just analyze the wreckage. With incremental, you don't just learn what worked. You do something about it. Check them out at imcrmntal.com. So I mean, you mentioned the food delivery companies where you expect that a good user retains for years and years, right? I mean, it's maybe something they would you know, interact with once a week or more in my house anyway. But it's talked to me about like e-commerce. So like a lot of e-commerce advertisers, they're looking to recover the entire of the CAC in that first purchase, right? And so they're really just trying to drive first purchase AOV to be, you know, to surpass the CAC. How do they implement signal engineering? What's the sort of objective here? Well, that's a fair question. Many e-commerce advertisers really focus on CAC recovery and the first purchase. And if that's really the case, I don't know if predictive strategy would be their top priority. However, there are many e-commerce businesses that do have a cohort of repeat buyers, whether transactional or an e-commerce subscription model. And then they want users to retain with that subscription. So for these, the cohort of repeat buyers or retained users, if that's a priority for the organization, then there is definitely room for predictive uplift. And again, there's a quick analysis you can do on, you know, the ratio of one timer is the repeat buyers. And what is the LTV of a second timer? What is the LTV of a third timer? Just to kind of understand that. And that will help you understand what you want to predict and how much of an uplift you can gain. Within all that said, it will shift from the lowest CAC to LTV to CAC ratio if you want to go that route. Because the CAC may go up. If you're extremely sensitive to CAC, then again, potentially predictive strategies are not necessarily the highest priority for you. So if you want to increase the LTV, you are taking a risk that the CAC may go up. If the LTV goes up by 30% and your CAC only goes up by 10%, it's a win. So the LTV, the CAC ratio becomes more important than the CAC itself. So I think sometimes companies that kind of failed to grasp that are, it's maybe counterintuitive, but the LTV going up is the point and you'd expect the CAC to go up. It's just what's the magnitude of increase for both. So talk to me about that. Because if I'm getting better users, they should have higher LTVs. That's what the better user is. And I probably have to pay more for them because they're probably, it's just more competitive to reach them. And so this is really like this, it's a process. It's not just a one-time thing because you implement the signal that gives the platform more information about who to target. They target those people to a greater degree. And now I'm getting more of them in. And so my overall TV is going up, I'm paying more for them. So my overall CAC is going up. So talk to me about like this is not a one-and-done thing. I mean, this is like a process that a company would pursue and they try to find like some sort of new equilibrium point over time, right? Yeah. So again, I like to give examples as much as I can. I worked with a client that is an e-comp subscription that sells all kinds of help supplements. And you know, they have a pretty big catalog and they sell a lot. And what we found in their data is that their average LTV was about $106. And users that do come in for the third time, when you look at their LTV, it was 212. So it was more than double the LTV. So now the question is, if I optimize towards those three timers, users who what is the probability of a user to make three orders within a certain period of time? And I bring an uplift of you know, 20% in those kind of users. It's fairly easy to calculate what is your overall revenue now? What is your overall a new average LTV? And then from that you can derive, okay, what's the maximum CAC I'm willing to pay for this? Now, when we go into a project like this and obviously when we go into a POC, we invest a lot. We want to win that POC. We run an AB test. And if we see an uplift at the end of the AB test, and when we look at uplift, we look at LTV, the CAC ratio or or row as we turn on ad spend, right? Which basically includes it. So if we win or if that campaign strategy, if the predictive strategy wins, then there you have it. It may have a cost on your CAC, obviously. I will say as a side note that we have seen situations where the CAC actually went down, not many cases, but we have seen a few of them. But in most cases, yeah, you would see the CAC go up a little bit. The win is if the LTV goes up by a higher ratio, then the overall performance is much better. So then what does the effort as a do then? I mean, I guess the point is like, is this a continuous processor? Are you segmenting users and determining different signal based on like some profile that they might fit or like some tier of predictive LTV? Is the average is coming in, finding a signal, engineering it, and then saying, okay, that's it. Or are they pursuing this in an optimization process that's similar to just running user acquisition? So it's a little bit of both. I'll start with the beginning. To start off, we don't build necessarily. I won't recommend building a single prediction model for all the users. I would recommend in some cases, segmenting the users to build different models. In Honeybook, for example, we have a separate model for users that come with a personal email because there be to be personally email versus the corporate email. We actually built a different model and in a company that is in the dating or matchmaking industry, we have a separate model for males and for females. So to begin with, we don't look at all the users as just one big bucket. And I don't recommend that anyone does that. I recommend if you have distinct audiences within your entire audience group, address it. And then as it goes into optimization, this is a process because when you experiment with it, you experiment on a single campaign in a single area. If it's non-brand search campaigns, then that's the area you're in. And then when you want to expand the brand, it behaves very differently. The audience pool is smaller, the inventory is smaller there, and it behaves differently. And the goal there is different. The goal there is to reduce costs and maintain LTV. So it is a process of optimization. We're bringing a new tool to optimize. And as an advertiser, when I look at a list of campaigns, I ask myself, I would ask myself, what is my next big move? What is my next big thing? So this is a new tool that allows you to really hone in on your highest value users and will give you the room to experiment and to drive more value and to really go back to optimizing, which we lost over the past few years. Talk to you about the different approaches that different types of advertisers take that you've seen be successful. But so I mean, you've got, I imagine that there are different approaches. I mean, I don't know how dramatically different they are, but I'd imagine that. Well, certainly the way that they think about what you said before, like good users is different, right? But talk to me about mobile games versus E-Com, versus subscription apps versus like E-Com subscriptions. How do these companies approach this task differently? Although the vertical does have some kind of indication on the value it will provide for some advertiser, I wouldn't put the weight on the vertical itself. I put the weight on the business the business model and the conversion funnel. So any mobile or web app that has a free trial, Regardless of which vertical they're in, followed by a monthly subscription, should definitely, definitely prioritize signal engineering, because their current optimization, BAU, is probably towards the free trial start. And that's not a good indication of a good user. Instead of relying on trials, they can engineer the signal, you know, now you have a crystal ball. What do you want to optimize towards? So you can engineer it to long-term subscribers. Any service that has a premium version should also prioritize signal engineering, because it's likely that they have a longer conversion funnel due to the premium option, and they would benefit a lot from a prediction. And then any service that has challenges with retention and experience high variance in their LTV across their users should also consider signal engineering. So instead of breaking it down by vertical, I'm kind of breaking it down more by their business model and their conversion funnel. Does that make sense? Yeah, totally. So maybe, because I mean, mobile gaming, I think, is a very unique model, right? So maybe we could just kind of isolate the model for a second. So with mobile gaming, I mean, how does that approach differ from like a subscription app, right? Because mobile gaming is just, it's pretty play, right? And then almost all subscription apps have the free trial, usually seven days, and then you would subscribe up front, but then the actual purchase happens after the free trial. And so maybe talking about the differences in those two different models. Yeah, so when we talk about mobile gaming, I want to differentiate there are different sub-articles there. So I want to differentiate the real money gaming, which is a very, very different from casual gaming. In casual gaming, it's a bit more complicated. And I gave that example earlier, and I'll repeat it again, because I think it's one of those examples where the vertical does actually matter. I think that the behavior of the consumer, where it's based on micro purchases, a lot of small tiny micro purchases, that behavior is recognizable within the first few days of a user's journey. And so because of that, within a seven day period, which is still within the conversion window, you can identify a good user. The correlation between what the user is spending within their first seven days is very highly correlated to their long-term LTV. The implication means that even if you do build a prediction model, and that's not the challenge, it's not about building the prediction model in this case, the challenges will drive uplift when you send that signal to the channel. So let's say that you build a super accurate prediction model that the new signal that you're sending to the channel doesn't re-rank the users. So the channel's now receiving new data that they've never seen before, but the impact on the way that they deliver as is minimal. And so for that genre of games, they won't benefit a ton from a predictive strategy. And on the other side, when you talk about subscription, with a trial, without a trial, it doesn't really matter if it's a subscription that is a monthly subscription or a weekly subscription, and then there is churn. And if there is churn, there's variance in LTV, which you don't know upfront. When a user converts and subscribes, you still don't know if they will remain subscribed or not. And there's no indication within their first seven days in the form of an event that you can use. So with subscription apps, it's very obvious, with a trial without a trial, with a trial it's even more obvious. But anyway, you look at it, you want to optimize towards users who subscribe and retain. And in order to do that, you need to predict their value, their future value, or their future propensity to remain subscribed for a longer period of time. And I can give an example here as well. One of our clients is a streaming app, they have a seven day trial. Their BAU before Voyantis was that trial period. With Voyantis, when we ask them what does a good user look like, we have to dig into their data a little bit. What we found in an ideal world, you'd want to say a good user is a user that converts and remains subscribed forever. That is not something that you can predict. So you can't do a forever in a prediction model. So what we're looking for is what is the shortest horizon, the shortest prediction, we can generate that will drive the highest value or would be the closest we can to that forever that you're looking for. And what we found is that their turn curve was very steep at the beginning. And users who remain subscribed the third month were 96% likely to remain subscribed for the end of the year. So all we need to do is optimize towards users that subscribe for three straight months in a row. And so now we build a prediction model. What is the propensity of a user to remain subscribed for three months? And then one user will get a score of 96%. One user will get a score of 22%. One user will get a score of 46%. And you can send those scores as values and run a two row S campaign to try to optimize towards highest value. In other words, you're training Google's model to drive users that are more likely to remain subscribed for three months. That's very interesting. What about hybrid casual games? I mean, where you're talking about a mix of ad and IAP revenue that mostly skews towards ads. Is that something you guys have encountered? And how does that change? I mean, I really hear-- I guess you're just talking about the differences in the LTV modeling, right? It's not. Again, we can do an LTV model prediction very easily for gaming. The challenge is, does that drive value? And that we do through understanding the correlation. And that's exactly why Meta released LTV vision, exactly for that reason. So with IAA, it would be kind of the same. The value of a user can be recognized by their behavior within the first few days. If they are retained in your app, if their average time usage in your app is I within the first seven days, they are a good user. You don't need to predict it. You know it. So I don't know for those kind of clients, I don't know if a predicted strategy is going to drive on a value for them. It will probably drive some uplift, but it's not going to be a double digit uplift. Like we see with subscription or with the premium or long conversion funnels, like FinTech and things like that. It's not going to be the same. So we've been talking about the advertising use case for signal engineering, which I think is obviously very valid. And it's probably the most obvious use case. So talk to me about have you seen any of your clients use these signals for product personalization? So let's say a user comes in, you're deriving some sort of like user level predicted LTV very quickly. And then that is fed back to the ad platform, but maybe potentially it's even used in some way by the product itself with some sort of dynamic personalization mechanism that says look, the content that we expose to this user should be different if they are predicted based on like these early interactions to be really high value. Then if they're predicted to not be, because actually if they're predicted to not be, maybe there are things that we can do to nudge them in the direction of seeing the value of our product. Have you seen any of your clients utilize these signals in that way, like internally with the product focus? There are definitely additional use cases that you can use a prediction and drive value for your organization. One of them is personalizing content, as you mentioned. Another one is, we have a client that connected all of these predictions into their braze and they're offering treatments for users who are likely to turn. So they're using that same prediction, that exact same prediction of number of orders a user will do in the next 30 days. And if a user is predicted to be to submit zero orders in the next 30 days, they can now match the treatment and offer them a coupon or whatever it is. That's another use case. Another interesting use case, but this one is more of a niche, is in a world where it's more common in telehealth where the funnel ends in a book meeting with a doctor. It can also be in that same funnel like automotive where book a test drive, things like that. There are a lot of no shows. And so predicting the user's likelihood to show up is also very valuable. That doctor is sitting on the bench and waiting for calls and it costs you money. So if you double book, because a user is only 25% likely to show up, then you basically have a more coverage and save a lot of costs there. So there are different use cases. You mentioned one of them being personalizing the content. I give two additional examples of life cycle marketing and that niche case of show no show or bookings. So these are just a few additional examples of what you can do with a prediction. Super interesting. I can imagine that that show no show use case could be applied to buying like airline flights, right? So you strategically double book or over book the flight based on these probabilities. I guess they do that because I've been double booked. Yeah, that's right. It's I appreciate your time today. And I appreciate you walking us through a pretty complex topic. How can people learn more about wait until? Well, before I say how can people learn more about wait until I'll just try to summarize this whole thing, right? The main objective is if you don't have a lot of controls on top of the auction, you need to feed the auction in order for it to do a better job. The auction is automated and it drives value based on signals that are sent. So just make sure that you're sending the best possible signals you can to the network in order to gain higher lift and performance and then why I just, you know, you can look us up in our website. We also have some articles and blog posts specifically on Signal engineering and the five steps you need to take in order to start this project and I recommend that you go into our website and that search for it Great and I neglected to mention this at the outset, but I'm also an investor in Voyantus a very happy investor I might add so I'm very impressed by what the company's achieved in just a few short years since I invested at least I think it was the precede so it's really impressive growth and it's obviously very relevant for this operating environment that well Thank you for investing you made it happen. Yeah, I don't know about that I kick some money in anyway. I eat so I thanks so much for your time. Thank you