E17: Regressions, Gin Rummy, and a VERY Special Guest (w/David Nelson)

If you could talk to anyone, who would you want to talk to? And I was like, anyone? And he was like, anyone? And I was like, I seem to leave. Number one, I seem to leave without question. Anti-fragile black swan. I seem to leave. And then I was like, I'm right named like a tot. Steve Lavitt. And then I was like, Obama, go find Obama. And this poor HR guy, the thing sucked as he worked at London and London, you do what you're told. And so he'd been told to go ask me. And then I gave him this list. And he was like, oh, oh, oh, oh, oh, oh, oh. And for me, I just like go and see what happens. Anyway, the dude reported back once a month for three months. He was like, still nobody's answered. And then still nobody's answered. And then the third month he was like, nobody's answered me. And I was like, okay, idiot. And then I just wrote to levitt. And I was like, yo, wanna do something? And he went back like, yeah, let's do it. And I was like, see, it's not hard. And that's how that thing got stuck. - Let's start with utility. I don't understand what it even means. (upbeat music) ♪ Everybody has some kind of utils in their head ♪ ♪ That they're calibrating ♪ ♪ There's hardly anything that hasn't been used for money ♪ - In fact, it may be a fundamental problem in modeling the one on model. - Game of Codum's cast, episode 17. And the fire did not stop last week. It is still burning somehow. There's enough wood in the furnace. We have a guest with us today. I am happy to introduce Mr. David Nelson, the CEO of Magnetic, which has some sort of description on your website that was generated by chat GDP. - Yes. - You mentioned that yourself. Former VP of experimentation at King. David, how are you? - Doing great. Thanks to him, you're on. This is, I've got my game face on. I've been looking at Excel all day so that I'm ready. - Just gotta agree. - Now you have enough of that soup. - So we put on his game day glasses for us too. - I did. I did. I just put numbers in a sheet and then I used the sum thing 'cause the sum always feels Greek and cool. So I just summed them and then I was like, I'm ready to talk to these guys. If you want to look like you're doing a cool math, you put a little sigma there. - Oh, that's a baby. Summation. - It's been a running joke for a long time with some people that I work with that are smart, but I still don't really understand what a regression is. - We talk about p-values. - When progress goes backwards, like you had a bigger house, but then there was a recession and you had a downsize. That's regressing. - Okay, but that's regressing, right? That's regressing. I know a lot about that. I've 40 years old. My body is regressing like nobody's business. But I've had dead serious people that have listed that they are economists in somewhere in their title sphere, whether it's Twitter, LinkedIn or MySpace or wherever, that said to me, I'll just put together a regression. And I was like, oh, you, oh no. When I get it, will everything just be better? That's what I'm really excited to finally meet this legendary regression. - Speaking of regression, I'm happy to introduce Chris. Chris, how are you? - I don't like that. - Well done. - You should introduce Derek first. If you're talking about like, oh, LS regression, I'm doing very well. Couldn't be happier. Couldn't be better. New products coming out, the left and right out of our ears at Star Atlas. - Star Atlas taking up tweets these days. Getting like, just getting retweeted about the CEO, Chris. - Yeah. - Oh, me, yeah. No, I'm, I try to stay under the radar, but I just can't help but interact with community. It's like an addiction. It's not good at all. But no, we've been doing really well, seeing revenue going up and really strong ARP dials, stuff like this is really exciting to see as a company. And the web three industry that's just not a trend that you're used to seeing. So feeling good? Excited for this conversation today. David is a, he's a titan. So even though he doesn't know what a regression is, we'll forgive him now. - And of course, Eric is also here, head of economy design at SuperLayer. - Doing all right. - Yeah, but thinking about this, we launched a mobile app in Germany. I think a week ago, so I've been, that's all in my head, that it's all just sign up issues at this point. It's all like, you're going through and you're like, oh, like, but why do all these users disappear? Oh, there's a bug. Okay, we gotta fix that one. But yeah, it's all the app launch kind of crazy. - Is it like an experimental launcher? Is this a full launch, just like a German app? - It's like the small one where we're not blowing a ton of money on user acquisition yet, but I'm trying to iron out all the kinks. But yeah, I'll be talking about that a bit today. I think it's an interesting model, I think, especially from an economics perspective. - Speaking of things we're going to be talking about today, we have two wonderful topics. Eric, what are you going to be talking about? - I'm talking about trophy. It's this mobile game kind of advertising platform, I guess. They call themselves loyalty, but really what it is taking money from game advertisers and trying to pass it through to players, like mobile games spend tons of money on advertising, can you get more players by passing some of that revenue through? And what are some of the adverse selection effects that happen? - And for our second topic, we'll of course be speaking to David and man, we have so many things to talk about. From Steve Levitt's paper to the origin of the experimentation group to what is a regression. Let's see how far we can get. Before I get to either those, let's talk about what we've been playing. - Got a selection of good things on sale, stranger. - Eric, what have you been playing? - I'm been playing a ton of F099. F09 is this old racing game. It's set in the future. It's a play on Formula One. It's like Formula Zero, it's even faster. But as an old game, the franchise died, they took all their racing games and just put it all into Mario Kart. But F099 is a battle royale version of this old Super Nintendo racing game. And it's a lot of fun, it's fast, it's hectic, it's very high stakes, and if you crash, your car goes bouncing all over the place and you explode and you die, which is great fun in a battle royale format. One thing I think is particularly interesting though is that this game has no monetization at all, right? And it seems like they made it with the plan that it would be short-lived. They did something similar with Mario 35, which was like a Mario battle royale. And these battle royale games, they tend to die up pretty quickly. If you're gonna sell in-app stuff, like, 'cause there's an expectation you maintain it, and support it for a long time. And if you go under, if it's a small indie studio going under, people accept it. But if Nintendo takes away all the skins you bought or whatever, that's a bigger deal. So yeah, they made the interesting choice to make it free in the online subscription. And it's, I'm pretty sure they're gonna shut it down in three to six months. But yeah, but have a lot of fun while it lasts. - Are there endgame micro transactions? - No, none. - That's okay. - So we were talking about the Scent Disc Court. What do you do with this game? Like, how do you monetize the love in this sort of model? If you're not gonna have IEPs, like, how do you get that depth of engagement? That's to me's always been the frustration. How do you align, or at least, how do you correlate the marginal benefit games that can provide thousands of hundreds of thousands of hours of gameplay provide with some sort of cost profile. And MTX to me was the solution to that. Or even coin arcades, putting in a coin into the arcade was the answer to that. That's how you correlate those two things. Can you even do that in this description? It doesn't have a subscription. It's flat pricing, right? You can't correlate it. It's naturally gonna be unaccorrelated. But you could go with coins though, right? Like you could do the, if you think about all the free to play models, if you make it, when you make a purchase, your subscription could give you 50 tokens to play the games. And then if you really love it, you could put in gets more tokens. - Literally go back to the arcade machine, model. Yeah. - How different is the arcade, if you think you're playing Royal Match, you're getting gold coins all the time. So you put the coins in when you want extra moves. And then sometimes you run out and you go to the cashier and you get some more and they're seeding you the whole time. It steps between the arcade and the casino. It's some middle ground with no cashier. - I heard rumors that Game Pass Xbox was thinking about something like that as well. - Oh, how would that work? So basically you would play a flat fee and get what, like a couple tokens a month. - Yeah, and each game might have a token price to play. - I don't think they're committed to this. It was just an idea that was, I heard, thrown around. - Oh, interesting. - Is that really that bad? An idea? I mean, like consider the Netflix stuff also. You could do the same thing. Your subscription gets you like this much. - What about the spending a version issue? The same reason Disneyland doesn't charge prices for their rides is 'cause they don't want people to think, oh, I only got 10 tickets left. Maybe I don't want to go on this ride and that decision analysis affects your enjoyment of the experience. - But see, the thing is that some park experiences do that. So when the car needs come to town, you are back to a ticketing machine. And that seems to make sense for economists. You have a ride, it's more popular. You need to ration what the way times are. It has a certain amount of capacity. Okay, you can raise the amount of tickets that it costs to go on the machine. So it's not just, I want to maximize as much revenue that the ride gets. It's also like the functioning mechanic of hey, something costs more people are going to do less of it. The man curves slope downward. But Disney doesn't do this. - It doesn't count as though the booths and the rides are competing with each other. Whereas in Disneyland, it's all owned by one thing and they're in different world ride you go on. - When you say the booths and rides are competing, what do you mean by that? - There's literally separate companies. There's separate revenue streams. - I think so. I'm not sure I've never worked at a car, but I assume so. You know the CFO, I think it's still the same CFO. It spends Newman for Netflix. He was the CFO for Parks and Rides or whatever for Disney. He was at Activision for a while and then went on. - Maybe that's why he likes the bundle models so much. - I saw Netflix stores are opening up soon. I don't know if that was like a fake news article that I saw, but maybe that's his brain child. - No, that's true regression right there. I think that's what we would do that was talking about. - We've identified the regression. So to me though, this token model is, it's a totally different monetization strategy. Like it's in that capacity, it is like pay per player, pay per viewer. It's, to me it's almost a premium model. Like I know I'm gonna have this fixed amount. Presumably there's some way to spend more. So my marginal, all of a sudden my marginal spending is now a part of my decision again. To me that's just premium or free to play or whatever free to play, but with that subscription. Like basically pay to play premium games with microtransactions, which we know players don't really like that much, especially the types of be signing up for Xbox Live, these types of services. So I wonder how that would change the, to me it's more this would impact user retention, user acquisition, you might see higher spend paying player, but maybe you see aggregate across your entire player base, you maybe you see a reduction in players or something like that. - I never think we got the shareware model. There was supposed to be middle ground, I think transition period between a full box price product and like a true free to play or MTX model. And Blizzard did a little bit of this, right? You could get to level 10 or the Warcraft and that's before the subscription would kick in. I think they started to experiment with this on almost all their products. It was like a very mild shareware and no one really figured out how to make it work in the long run. But I wonder if there's something there for the type of games you're talking about Eric, like F zero 99. You play for free and then at some point you pay money. But how do you deal with the expiration date of the game? You know this game is going to be dead in a year, right? I think you could cap. I think you could at least, that's just a retention problem though, just for the entire product. But I still think the shareware gets you the on ramp and at least can maintain some sort of user base. See, Rainbow Six Siege did something like this too where they slow down progression for players on a particular version. Now there wasn't entry fee to still get the version that was slow is about $5. And then what they did is that if you bought the full box price product that you would progress at the normal speed. So I thought there might be something that model too. So there been real small experiments, but just because it failed once, I wish someone else would take it swing at it easy for me to say, but I just I wonder if there's something maybe you can get to level 25 and F zero 25 and F zero 99. And then you have to pay tokens. Maybe there's something there. There's got to be something that like you're looking for something that you can layer onto any game for these types of subscription services, right? You don't want anything that's specific to the game contents like boosters and candy crash or are extra moves or something of that nature. At the end of the day, this is like pretty antithetical to not to Nintendo micro transactions. They want to provide a pretty fun little product at a very low marginal cost for their paying players to try and keep them on board. That's the way I see this. This type of game like it. I don't think Nintendo has to put as much. I could be wrong. I don't know how much how much a game like this costs to produce, but I don't think that it takes as much to put this game out as it does to put something out that like the Xbox crowd would be interested in playing tunic. Oh, too. Mecha. David, you've been playing anything. We'd like to give you the opportunity here. What would you like to big eye us with? What do you been up to? What are your gaming habits? We're going to like really go in the other direction. I was a professional poker player for a couple years. Then when I was eight years old, I learned how to play gin rummy for money. My grandmother. So she'd tell my dad to go borrow 20 bucks from my dad. I'd borrow 20 bucks. She'd beat me for all of it. Then we'd go out and she'd buy me like $5 toy and some ice cream and pocket the rest. So she was early on. She sat down and was like, look, I'm going to show you why this is just math. Then it's not that hard. Then I played gin rummy for money through college. I played some poker, but not that much. When the poker boom happened, I was working in a factory in Sweden and I started playing online poker. I played online poker for two years before I started working in the poker industry. I've always enjoyed card games. I think they're really cool. I was talking to one of the founders of King and they told me about a solitary cash or whatever. They were like, dude, they have real money. Apple pay inside of the app. There was like no way they have Apple pay inside of the app. He showed me and I was like, this is stunning. So I downloaded it. But it wouldn't process any of the payments. I tried to make it a challenge. I've written a customer service. I don't know. It doesn't work. I really wanted to find out. But then the problem is they only seed you with six bucks or something like that. I want to get into one of the actual tournaments with Apple six crowns. It's like a solitary competition. This real money solitary you versus another player actually, it's like a six person tournament top three get paid or whatever. You break it very hard. Half of it disappears and break it. I wanted to see if I could free roll my way all the way up to the bigger cash ones. So I just as couple days we grind in gems when I have a free moment in a player to player solter. So up from 300 gems to 1,180. Then I just need to get to 1200 and I can free roll. And what is this called? Salad cash. I think I've seen ads for this. I was like, oh, this is a scam. I'm not going to click on that. Is this where that cat game that King made came from? There was some sort of cat running everyone thought it was super interesting. And then it died like it wasn't super. You know, it's so mad about that. That was so mad about that. They call it shuffle cats. And it was a gin rummy game. And I'm like, you know what? Gin rummy people want to do. They want to play gin rummy. You don't need a cat. You just need a really good gin rummy game. Damn it. This is one example of being not being able to just paste a picture of a cat over top of something and get more money for it. It doesn't work. It doesn't work. Especially with gin rummy players. Just interview a few of them. Not of them want cats in the game. They just want cards. I've been playing that and then I still I'm way deep into Marvel's map. I also card game. But I love the complexity of it. I've got an old gray Thanos deck. Are you bored? There isn't a lot of variation in the core gameplay, though. It's like your lanes and then your decks fairly stable, though, right? No, I don't know. My son and I have been playing it. So we've he made this Thanos deck for me for a while. I was like, there's no way it's going to work and it worked really well for while. And then I switched over to this Mr. Negative and Jane Foster Thor. And I love that combo. And I was like crushing people got up to infinite and was like, got pretty high with that one. You got to put in the hour. Like you got to put in some time on it. But it was super fun. I always get fun when I can sometimes feel the vibe. You're like, I'm definitely planning to 12 year old. And then you're like, loser. I don't know a lot about the poker industry or the kind of gambling, like online gambling industry. I'm assuming there's a bunch of regulatory stuff around letting people win prizes. And I'm curious, why doesn't a game like Marvel Snap, but for money exist? Why doesn't a gambling version of Marvel Snap exist? Is there a reason for that? Is it just the wrong? There's no overlap in the customer. If there's going to be any payout, you have to legislate. There's legislation. So most countries, you need to get a license. I don't know what the, I knew it really well in 10 or whatever. I don't know it anymore, but there used to be like here in Europe. There's multiple to Brawl Tour. And there's a few other places to get a license. You could also get these sketchy international licenses and like Kurosau. And there was I love man and all these different places. One reason poker was a bit weird was because a lot of people that were doing poker were like, this is a skill game. It is definitely a skill game like good players win. This is in gambling. And then there is a whole longer story for some other wonderful podcasts where we can poke about the poker stars and to poker, more of these big companies. And they were, they had lawyers that were sure that told them that we're all good. This is not, you're not going to go down. You're not going to be in the wire after everything to be fine. And then the FBI ceased. I was working for full to poker in front of my call down a Friday. And was like, Hey, you need to go to the website, man. FBI just took it. And I was like, what? And you go and you log on to go on to fulltipoker.com. And there was just the stamp of FBI. Just domain has been seized by the FBI. So that's really interesting. I guess my question was like, why doesn't a very gamey, exciting version of log of legends exist with a kind of gambling or poker or payout back end? And it's just like that's the reason. And it's not only that, I would say probably these are rough numbers, but like something like 5% to people are willing to wage a real money to win real money. And you can get more people to do it if they're in Vegas for some reason or another, but like on a regular basis, the general population's uninterested. So it was very deep. And then I think like in general, also, I never wanted to work in the casino side of it. Like poker cards and I enjoyed it. But like the casino side for me just gets dark. It's like trying to convince somebody that like, you're like, no, it's okay. Two, two minus four is like, what about one? I don't know. It doesn't feel okay. I would love to have an episode about casinos and just like the whole entire industry. We will bring on Matt, who is also a game economist who works at W Games in Toronto. He works on social casinos. Just like a really misunderstood genre that I don't think a lot of people understand that they generally say it's just a bunch of generates, which really is in the case on mobile social either because there's going to be no payout. It's all trapped within the system. I think it's just things people haven't understood about how these games work. And I think Matt will be able to illuminate us look forward to that in a couple of sets. Just a real quick. Another one of these like inside on the inside. We're sitting working in online poker and Zingapoka comes out and we're all like, there's no pads. Nobody's ever going to play this. This is silly. It's going to go away. And then it was like, Dows going up, Revin is going up and everyone's not just going to blow up. Everyone will stop soon. But you actually, I think it's an interesting question. It's the opposite of your legal language questions. The game of poker has a certain, it's fun in its way. And as long as you believe that having a big pile of chips is good, it doesn't need to be cash for that to be a fun way to spend your time. I think the same thing happened with the social casinos. People are like, no way. And they're incredibly viable businesses. Let's dive into articles. Eric, do you want to set us up? So I mentioned where we launched this app called trophy. It's a, we call it a mobile games loyalty platform. Really what it is play advertised games and earn some money to do it. And so some backstory here, David Phil, feel free to chime in because you guys have been in mobile games longer than me. But user acquisition is really important for mobile games for a couple of reasons. One is that the tracking and attribution is way better. And therefore you can see your return on ad spend and optimize that much more effectively. You got your devices, everything from click, watching a video ad to clicking on it to installing is all tracked, right? Whereas maybe in the traditional PC box game, you have run a big advertising billboard and you have no idea what the impact of that was right back in the day. The other is that there's big discovery issues with mobile. The app stores are notoriously difficult to search. And so you have to get direct to users somehow. And yeah, mobile game is spent a ton on user acquisition. A lot of this is in video ads. If you've played any mobile game, you've seen video heads for other mobile games. And another technique that's often used is called what's called in like an offer wall or there's these affiliate marketing networks where it's like rather than just paying to put a candy crush video in front of somebody and giving them 10 cents every time someone clicks on it, you might say, hey, for every user you get whatever platform you are a third party to get to play candy crush to level 100, we will give you a dollar. And so these often will create offer walls where it's like, hey, play candy crush level 100. They'll try to attract users who are willing to play that far in order to get that money. And sometimes these affiliate marketers pass that revenue back to player. So they might earn a dollar for the user who gets to level 100 and they might have an app that says we'll give you 50 cents if you get to install this game and get to level 100. That's affiliate marketing and a nutshell. A lot of these offer walls struggle from fraud because first off, if you're paying people new stuff, they'll figure out ways to bought it and cheat it. And so there's this big adverse selection effect where like the people you're getting from the offer walls probably are not as good as the people who clicked on a video ad of their own volition because they opted into the experience. Traditionally, you think of when you sell a product, your first users, your first customers are the ones with the highest willingness to pay the highest utility. Sometimes it's called the golden cohort, you know, the first month of users you get a stick around way longer than anyone else. Whereas when you start paying people, you get to get the opposite effect. And this adverse selection issue, it's you see an insurance, but it's this tricky thing where if you pay too little, then you're only going to get shitty users. But if you start paying more than you're like you're losing more money. So this is like a very tricky pricing question to balance. But yeah. So anyway, so what trophy does is there's the offer walls that might say, play Candy Crush to level 100, get a dollar. And what we're trying to do is try to gamify that process a little. So an offer might, for example, might require a user to play for 10 hours. But the user sees that and they say, I don't know if I'm going to play this game for 10 hours. I don't know if I'm going to get all the way to level 100. And so what trophies trying to do is try to gamify that it's almost like amortizing a loan or offering insurance where we're saying, we'll pay you 50 cents when you get to level 100 and we'll also pay you maybe five cents an hour along the way. Try to breadcrumb it a little bit. And then we take some haircut off the top to make it the offer more attractive for someone who might play for an hour and be like, okay, I'm out of here or so or they might get an hour in and say, Oh, actually, this is fun. And I do want to play all the way to that level 100 goal. And yeah, so trophies doing this. There's a couple of misplay is the, I think the biggest platform that's doing this currently. And it's all time-based rewards where it's just play this game. And for every hour, you earn a certain amount of dollars. But yeah, it's an interesting, it's a big pricing exercise because you say, okay, the candy crush will pay us a dollar if they get to this point. If we pay five cents an hour along the way, what does our retention curve look like? If we pay slightly more or slightly less, it'll affect the dropoff rate, but it also affects how much money we're making. And there's a ton of opportunity for personalization here as well, where if you can tell someone plays a game for an hour and quits, then you might try to offer them a lot more of these short or like front loaded games. Whereas if you know someone will find an RPG and play it for a super long time, like how can you target them? And how can you offer the right amount of payout where it's high enough to get them to the goal, but not more than you need to offer to get them there. But yeah, anyway, that's a failure marketing trophy in a nutshell. It's an app then, right? So I download the app and then I go in and there's some offers for me. So you got a bunch of first party data. In other words, if I'm picking this one and this one, you do it all for me, this one and this one, and you can start tailoring what you offer me. That's right. It looks like a mini app store where it's like a game store where there's only like 10 games on there and they're all rewarded. How is the equilibrium here not to pay the players so that the payoff for the game is just epsilon? If you've got, and correctly, if I'm misunderstanding this, but so the affiliate advertising, you've got some game and they are paying a third party that's going to say, we're going to bring in players who are going to play your game and they're going to they're going to be exposed to a bunch of advertisements. Those advertisements are going to pay you, let's say Candy Crush. They're going to pay a king a bunch of money through advertising because they got to level 100. So king has made a hundred dollars to advertising. They're willing to pay you guys X in order to for those customers. How is X not 100 minus epsilon? How is it not like this tiny amount because anything any additional users through this platform is good. So I guess like, what's the equilibrium? What does the equilibrium look like for these wages for these players? If they're producing, let's say they're producing 10 cents an hour, are they getting like nine, nine cents an hour? To be honest, I don't actually we don't actually know how much the game in this case Candy Crush is making from these users. What we have seen is that for games with very reliable revenue, for example, ad funded mobile games, they pay a lot more or they're willing to pay because they know, okay, in order to get to level 100, the user has to watch 100 ads, which will earn us X dollars and we can give them X minus epsilon dollars. But I'm sure that market power effects here as well, where even like they don't have to pay X minus epsilon, they could pay X minus a much larger amount because they know we'll still feed them users regardless. I guess that's like my question is once a competitor comes in, what's stopping them from like driving that wage? Oh, I see. Like for competing offer walls. Yeah, I guess the equilibrium would be where the players are actually getting paid epsilon, which is just this tiny amount. There's a really complicated problem once you have competition in this market. Yeah. So there's a bunch of these offer wall companies and since they all use the same affiliate networks, they tend to have the same things like all of them will have Candy Crush and all of them will say we pay you on level 100. I think where we're trying to differentiate and they are very commoditized and the users will shop around and see, okay, I see the same offers, but this one pays me more. We're trying to differentiate is what I mentioned with the breadcrumbing of trying to pull the rewards forward and potentially eat some loss on it in order to make the experience more attractive for a user who might not get all the way to the end. But yeah, I think in a and there's some platform lock and effects. If you can get better games or platform exclusive games, higher quality games, then you might attract people. But yeah, there's definitely is competition on that front. I'm just struck by how brilliant this idea is. Of course, this makes sense, right? User acquisition costs have skyrocketed. What's the natural evolution of free-of-play? Pay-of-play? Not until it takes something out of Web3's playbook here. Like this is much more straightforward, much more controllable than a lot of Web3 things. But I guess what I'm interested in is it becomes it becomes inbound acquisition rather than outbound. Like normally acquisition happens on these platforms when you create a really valuable service and then you just slap ads on it. Like Facebook, Facebook, there's nothing natural about ads on Facebook. It's a social media network and they just slap ads on it because they buy balls and they also have data which lets them have targeted and personalized ads and it fits naturally into the feed. So I guess maybe perhaps there is something natural there. But set out to be a social network and ads just happen to make sense. But there is something to like, hey, this is clearly an ad platform and if you're interested in finding games, I guess you'd want to go there. There's a conglomeration effect. Like I'd want to run to this app. I guess the personalization piece is more interesting because you don't know who the winners or losers are going to be when you have an ad on Facebook. You don't know who's actually going to be the person who monetizes. Like when we say cost for install, it's actually average cost per install and some of those users are going to subsidize the other ones, right? Some users are going to be worth $10,000 and that's what subsidizes all the zeros. The average here is doing a lot of work. Like the median is almost always zero. But if you had a platform in which you knew that you could serve an ad to someone who was verifiably someone who had paid beforehand, you could give them pretty significant payout, right? You could give them a much better rebate on their in-game spend. You know that they're valuable for and this is the whole thing with Facebook post-backs, right? Is that Facebook started to know who the spenders and who the spenders weren't? It's funny that we talked about gambling beforehand. There's two things that this reminds me of in online poker. There was something called a "Rakeback". Do you guys know about "Rakeback"? Is this like a common. Basically a couple companies built this poker platform and they had a brand on it and then they realized, "Hey, I can like white label a bunch of brands so other people can use this same platform." Because having a big base of players mattered. Each little brand couldn't set it up its own liquidity matter. You had to poker players, right? So basically people started using it and also they had to eat brands using the same poker network in plain. So what happened is the first ones are doing marketing to bring people, but then the smaller ones start and they just start going into forums and offering "Rakeback". So "Rake" is how much you take out of the pot each time. It's the revenue that you make. And what they would do is they would say, "Look, if you play a hand and "Rake" gets pulled out, some of that "Rake" gets assigned to you as a player. And then if you came from brand X, maybe brand X is offering. I'll give you 50% "Rakeback" or 10% "Rakeback" or whatever, and you get some cash put back into your account. So like, this is similar to that where you maybe as a player you start thinking, "I'm playing and putting a bunch of inherent spending on it and I'm going to get some kickback." So wouldn't surprise me if especially games that are high-spend oriented if it went that way. I think it's harder to work with things like Candy Crush because the relationship to spam, I think people are even aware of that much about how much they spend or they don't think about it that way. I'm sure there's a few people, but for the most part I'd say, "If you want to look to Clash of Clans, there's probably a bunch of people that like, "I spend money, I love this game, I spend a thousand bucks a month, I think it's going to be really hard to find those players in smaller or casual games." So probably the actual balance is what is acquisition costs. So if you know that there's a person who's like super active on a game, playing mid-core games, and spends a lot of money, then maybe Clash of Clans is super-salesable and say, "I'll put this much in to get that player." Although I can't see some personal doing it because it's going to be harder with you. Because it's going to be against the UA cost that you're really bouncing, not how much the player spends. It's the market that decides how much it costs you. We're all trying to tweak all the payout rates and seeing how retention changes. How do people respond to changes in wages in games? How elastic? Yeah, so generally paying more causes higher attention. Whatever the highest paying game is, it tracks all the farmers. So there's a bunch of people who just idle in the games to abuse it. So we always have a honeypot game, which is the highest paying and all the fake users go there. Because when the games actually look at the stats, they're like, "Hey, these players have higher retaining, but they're not doing anything in the game." So we don't want these users. So we've got a honeypot game, but there's also that selection effect where the lower paying games have higher, when you increase a game's payout, sometimes or feature it. Sometimes the retention goes down because the players who opted in at a low wage rate are the players who like playing the game the most. And when you increase the wage rate, you attract players who don't like, say, card games as much. And so they turn faster. It's actually been not obvious to me how to optimize these things. - And I guess that really strikes at the point about personalization you were saying is if you know the bots have zero spend on the platform and you're able to track that, to assign them zero wage rate and let the people who are valuable in the platform pay them higher wages. They're more valuable. You can price discrimination here. I know we say personalization. That's like the nice word for price discrimination these days. - Yeah, for sure. - They're paying for those users, for those views. Because those views convert into downloads and those downloads convert into spend. If you've got all the sudden, you've got like this in group of players. If you add in K bots, that's the same size as N. All the sudden, and their average spend is zero. Let's say their average spend is zero, like just zero across the board. All the sudden that lowers the value of the entire pool of players, which should lower the payout of that the advertisers willing to spend in the game. So there's this like big giant IO problem going on where if you let these contaminants or these bots get in and contaminate the sample or the population like destroys everything, I'm almost like, that's right. - Okay, sorry. - Yeah, no, that's totally right. And these affiliate marketing networks, like I mentioned, they're open to a bunch of third parties and there is a ton of fraud, like exactly like you said. And they turn up blind eye 'cause like they're just feeding installs to these game companies. But part of what our goal is, and these affiliate marketing networks, first off, the games don't pay as much 'cause they know there's fraud. And also the network itself takes a pretty big, I wanna say 40% cut. And so part of our goal is to make direct deals with these games by showing, hey, we actually have high quality users that aren't fraud, you can defrauding you. And if we can establish that, then the games will be willing to pay us directly and we get a much bigger cut because of that. But yeah, the whole system operates on low trust and everyone knows everyone's frauding. And so that's why all the wages are lower. - So do you have any, I know it's probably the answer is no because of like GDPR and stuff, but do you have any idea what the users look like? Are they from very low income areas? Are they just bot farms? Are they like, I'm picturing the guy with 1000 phones on his bike running around with Pokemon GO going? - We definitely run into multi, there is a dude who made a hundred accounts on one device. There's people who have like multi devices. - Yeah, so there's definitely a lot of that stuff happening. - Anyway, it's funny to me that you guys, nobody reacted this, the fact that you tell somebody how much they get paid for it. And then the lower number gets higher retention, the more you pay the retention drops off. It's such a fascinating discussion for like, how do you motivate people for things? And how we relate to money and where is our motivation really? - But I don't think that's what Eric was suggesting in the way that you're talking about it. It's not that those same users, but maybe. - It's a trick of the numbers. You just in low users that drop the average. Higher numbers and you get a bunch of, yeah, exactly. But I wonder if it's not also related to it. If you show me a game that I would be intrinsically interested in, but it costs like right now I'm gonna pay 10 bucks for playing it. I would have ever changed just my relationship to it. I wonder if there isn't some weird, - No, I get that like shit outta here. - Hey, hey, I'll bring up a paper I did in my undergrad on zero prices. And so this is a horrible topic to bring, or a horrible paper to bring up because it's by Arielie. So I, exactly. - Exactly. - And Arielie is dating. - I know about the fraud. - And I know all sorts of wonderful, it's great. - Okay, so I didn't commit fraud. Arielie committed fraud, probably on this paper too. But basically paper that shows that, so we all know the demand slopes downward. So as something becomes cheaper, we consume more of it because of our budget constraints. And then he did this little cute experiment where he was offering people, I think Candies at the end of a checkout line at a cafeteria, like an experiment that would just not, like wouldn't pass the sniff test today. But he's handing out these treats for 10 cents a pop and people are buying them. The second he offers them for free for zero, all the sudden the quantity demand drops. And so his big thing was like, oh, there's a moral kind of cost associated with purchasing, actually my argument in the paper that I wrote in undergrad was, there's a moral cost to getting something for free. Now this is like super behavioral, Phil's gonna throw up if I keep, if I talk too much longer. So I did this and I then explored negative prices 'cause it's like, okay, if zero cost has the moral cost to it, negative must have an even bigger moral cost. So what is if we give them 15 cents? This is where psych fucking fails, Chris, thank you for this. They don't take it to the next level. You got zero, why not go negative? That's what I did, baby. Yes, yes. I still have my sign that says chocolate experiment and I'm like this, like, pudgy faced undergrad. I think I was a sophomore. Anyways, and demand did, quantity demand did keep going down for those negative prices. But I think, David, I could be completely misinterpreting what you're saying. But that's what I'm taking out, you're hearing is like, do my, at least the result of my utility function or my budget constraint. Does something get funny when I'm getting paid to play this game? That's supposed to have him to play it. There's a completely different social contract. You're saying people took less chocolate when you paid them to take it? Yeah, the more I paid them the less it was. Yeah, but it's weird, right? That's good. That's exactly what I'm saying. Okay, Phil, yo, I'm going out with some friends of mine. You want to come with, it's going to be fun. We're going to get dinner at this place. Like, that's probably a 20 bucks. It's going to cost you 20 bucks for dinner. We're going to come up. And you're like, I don't know, maybe it'll be a sheep people. And then I say to you, Phil, you got to come and I go to my friends, don't worry. I'll cover dinner, dude. I'll give you 50 bucks. It's going to be like really great. They don't have the same effect as we do. I'll be sure. I will happily can see that there are social norms and that there are heuristics we associate with certain activities like paying. So I'm going to do something. Can change the guys of the activity. I will totally can see all those things. I would just say that those results, which tend to be on the fringes of the 1% of human activity and human life, usually get blown up by psychologists and use to destroy the entire theory, the entire model. And they never end up being able to do that. They just end up describing a very fringe part of the human experience, which still deserves to be described. But what they think it proves is much smaller than what it actually proves. The man curves do sound lord. Yeah, yeah. But we could just go back down to this app now with the affiliate thing and ask ourselves, is this going to be the reason I'm asking this is because I look at like retention numbers for UA. And when we're running ads for people, a lot of times the retention for when you've when you've gotten people through UA through ads in Facebook and stuff, their retention is better often than organic installs. People that have searched and found the game. So why there's a bunch of different things going on, but it's valued differently, right? The other one, somebody's something's said, this is a thing, and I value it higher than a thing that I stumbled on. And I don't really know if this is a serious thing or not. I try not to. There is how you're introduced to something matters. And I just wonder what happens when you hike the prices. Maybe it does affect actually not only that, but like the relationship. I'll argue the fill thing here and that I think that's just a selection bias. I think that's just the types of people who are coming from those. I haven't necessarily, there's no counterfactual there, right? You can't know whether did Chris's brain change when he got a chocolate for free. No, it's not worth running an experiment on it really. It was just interesting. I just wondered if they'd be able to see it in the natural data. Like when we change this because you're tracking people, right? So you could say it turns out the same players are being paid one to play this game, nine to play this game. Their attention's actually better to lower the prices, no matter which player you should be able to see that. Interesting. It wasn't site. Speaking of interesting, David, you're here with us. You have more citations than I think all of us combined. You've outside of us, I think. Hopefully we won't under Google Scholar. Your profile is more filled out than any of us. You published a paper quantity discounting, quantity massive, only discounts on a virtual good. The results of a massive pricing experiment at King Digital Entertainment. It is published with Steve Levit of Freakinomics fame. John List of Field Experiments fame. God, pray that man gets a Nobel Prize in the next 10 years. That's what Mr. Levit said to me, by the way. Mr. Levit said to me when we did it, he said, "Look, it's fun that you'll be on paper. "We'll be on a paper together, but nobody will care about this." But when John List gets his Nobel Prize, you're going to be so happy that you're on a paper with him. And I was like, "Thanks, Steve. "You've got to go to a meeting, bye." I really hope he does. He needs probably just 10 more years. There's like an age limit or age floor. And you, David, and also you're also on this paper. We've referenced it multiple times on the podcast here. It's how we met. I remember this getting published. This is published in July 2016. You ran a massive discount with Steve Levit and Candy Crush on the number of gold bars that you could get in a particular scheme. How did this come together? What's the origin story of working with Levit, with Kang? What can he tell us? And your origin story as well, David. Yeah, let's get it all. Yeah, in a little town in Rhode Island a long time ago. Actually, I think my parents were actually in Martha's Vineyard. Anyway, nobody on. I joined King in 2013. And at the time, the company was like blowing up. And when I joined, I started working with a guy named Loshjornov, who was known as a sort of Maverick and the company. He dragged the company onto Facebook and then dragged the company onto mobile. And then it started working on gross topics. So the first thing I did was actually go to start working with getting Candy Crush on the Kakao Talk, which was a big social media app in Korea. And the next thing I started working on in September 2013 was I actually went and met Twitter. I went to Twitter. And I had this theory-- I don't know if you guys-- do you know how to solve Candy Crush approximately? There was a saga map. And there used to be these things called like collaboration locks, where you'd play 15-- at the end of an episode, you had to get three friends to send-- you would send out on Facebook help me. And three people would have to respond to unlock the next levels. Or you could pay a buck. Oh, you hardgated it. Holy shit. That's insane. So it was like those notifications-- That is bold. Super fucking annoying. Yeah, yeah, it was awesome because this is one of the some of the most amazing conversations This is like a serious conversation inside King at the time about cutting down on the amount of notifications that are going out And we've got people in marketing saying we have to cut these down. They're pissing people off And we've got a backend developer very senior but backend developer going Maybe it's the best marketing we have and I just lose so much fun with like this made no sense It should be the other way around anyway, but so I got this idea that what if we did this weird hack with Twitter where what happened Was you if you logged in using Twitter into the Facebook we checked how many followers you had and then we hardgated some Content based on how many followers you had so if you've got I don't know five followers on Twitter You can just one person or maybe even for a longer period of time so we deal but if you're Tom Cruise You need 10,000 people to help you and I just hit just be such it be mayhem But you so much fun and like these there's a bunch of celebrities on Twitter They don't have anything to talk about anyway. Nobody really wants to talk to them about anything now they've got something to talk I'm stuck on level 570 forward and I remember the King leadership team was like go ask him So I flew there and met with the Twitter people and it was I just met with some engineers and some product guy and the guy was like hacking our graph I was like I wouldn't do that But and then in the middle like basically at that time King had a data science unit that was wrong on its own that kind of Sad as they just the same with the teams but they're really their own functional unit and then there were like product managers and that producers and people running creation and Somebody again in Marcus Jacobs had an idea to put these two things together with the data people and Product managers really put them into one one group and work closely with the game teams and so in the middle of me coming up with These crazy ideas then somebody was like hey you want to be involved in the start of this and I was like yeah sure Let's do it. So I went down to Malmo Which is the southern part of Sweden and started working with Petraski, which was the second biggest second biggest game at the time And then I did that for a while until 2014 I thought we weren't taking it up risks and I'd started working doing more innovation stuff Testing more things. I was harassing the leadership team all the time I'll give you this is what we can do because the game economist of this is really fun. So we've translated candy crush into Japanese and we've got a request to translate pet rescue into Japanese and I sent to the leadership team I'm just gonna say random number so it's not the I don't remember what the number was I said what are you gonna do with that $537 man and the CEO of the company said David How do you know it's first and I said ratios Japan revenue over global revenue X over global revenue Anyway, is that how you can summarize all economics ratio ratio baby? Why run a regression analysis when you can use a ratio exactly? I know regression is a ratio. Yeah, I'm not even joking No, it literally is if you really get into it It's just what is it RSS over TSS or something shit like that? It's just giant ratio. So then we were also putting games on kind of stuff And I was like look any why are we doing this stuff? This is peanuts And then somebody said what should we do doing then David and I said literally anything would be better than this and I got a hold of Steve Levitt and the Instinct Tom and run a project with TGG so it was Steve Levitt and Daniel Conneman's company the greatest good and that I sent over for consultants and Steve also came and and went around on tour with the place and talked about our results and stuff and he posed a bunch of Experiments he went and looked through all the data and like with the consultants with the data And stuff trying to figure out like what we can do and this was one experiment that really like that we got out that was big enough And at the time you really were working inside of King each game team. It's a really important thing to King that each game team decided for Themselves what they were going to do. Of course you're working in a big company So the COO tells you to do something you're probably going to do you're probably not going to say not but in theory each backlog was owned by the Game team itself but it meant that the way that things were really working is like Candy Crush would do Something and then it would work and because it was big if they would just tell everyone else what to do and everyone else to just do what Candy Crush was going to tell them. So at the end of this Steve Levitt Thing and he'd done this paper we'd go around I started trying to convince people that we should do is we should do all the crazy stuff in the small games Take all the risks there and if something works in the small games Then we can take it up to Candy Crush where you know have a huge multiplier And I remember having like serious conversations with NBA types and they were like they're never going to Convert from the small games to the big one and I was like yeah, but I was like wait they convert From the big games to the small and they're like hmm not the same thing pretty sure it's exact and then like those same people like only I don't know six months later once we'd had a couple big hits We're just like you know why this works David because you're and then they're like this lecture on optimizing your Opportunity cost for the big kick and then okay put it in a PowerPoint Send it around sounds good But then we just started running these experiments and the original short we had like the requirements I had to the game teams was it needs to be like as impactful as possible Because your games are too small so if you do something it's a 1% we're not gonna see it So you need to be like 10% or over is the only place so like I had this one arrow this way It needs to be 10% or over and then it needs to be as generalizable as possible So if you're working on like a clicker or pyramids or goes in a solitary game If you've sold something in solitary yay, but like we can't do much with it Mate if you give me something at the time I use this example if you give me something that if Twitter found out about this They'd be like oh shit. That's great. So over 10% and extremely Generalizable anything that you can max out on those two is on the table do some damage and that's how we kicked off the Permutation group and so you run this paper you run this experiment calling discounting you're holding the price of gold bars Constant you're increasing the amount of gold bars you get at each of this priced skews and the results come back And I remember I met Steve levy I talked about this paper I went out to Chicago to talk to levy about this paper and he considers it one of his few failures That he likes to open his when he has people outside of academia He likes to open hey if you come to me I can make you more money people don't believe in economists I've done it look at my tracker Except for one and he talks about this paper and how he failed and you were not able to increase revenue Even 1% by doing massive quantity discounting and not only that the the result that always blew my mind That gets very little conversation in the paper is that there was no conversion of non-pares to payers In each of the experiments So you would change the amount of gold bars you would get at let's say $5 by as much as 60% you would get that much more value And you couldn't even convert 1% of people who are non-pares to payers on coin discounting alone That to me was a mind-blowing result. He considers it a failure. Do you think of it as a failure? Was that a failed paper a failed experiment? No it opened up for at least for me and I think probably for all the people who worked with 10 Failures only crappy if you didn't if you already knew what the outcome was If you failed because you're trying something and it's way too marginal Like it's not going to have a big enough effect. That is a proper failure. You should like punish yourself And you and then because then you knew that you weren't really having any serious I think as long as you actually doing something that can you think is going to have a huge impact Then the failure is just more data input to the next iteration. So I don't consider it a failure I think one of the fascinating things with this is at the time we thought this would be interesting And now later now you just look at piggy bank which and and battle pass and you realize Because this is fascinating right when we put in piggy bank in the game conversions like through the roof just oh We're like holy shit tell us what a piggy bank is piggy bank is like many many battle pass right you're like playing and you earn Gold coins that go into the piggy bank and then you pay five bucks and you get them. That's what the Mechanic is it was originally in a game. Oh shoot. What was it called like an egg game Egg insinator or something like that an article came out on what of the game Websites and everyone read it and we had this system at the time Where you could put in Everybody in the company could put in requests into the into the we called it the hot list It was like a list of what we what might do and that article came out literally I think five people put in that we should do this test To test piggy bank and so the fascinating the really interesting thing It's easier to look at it from battle pass if you want to get conversions You just mess with prices the people that think it's would okay to pay in the game You mess with prices you might get more you might get less It's the same as we were talking about with with the app like it might change a little bit how much I pay or whatever But it's not going to open new payers, but all of a sudden if I feel like I've earned stuff It changes the way that I relate to this particular package. It's no longer like just cheating It's this other thing. I've earned these things and these earning mechanics They work without question and they change they change the entire like this relationship for paying for a game There's stakes they feel invested. Yeah, I feel like I've already 90% of the way there David I wanted to ask a question We had Julian Roonge on the podcast last episode It's incredible that we have you here because we talked about this paper and that episode And I wanted to provide his critique. I don't know if it's a critique But at least the reason why he says his paper is different and it's it comes down to we're talking about conversion We're talking about people on the margin and his big argument is the The effect or the sample that you guys were actually that I guess the intent to treat or whatever if you were in a labor economics class You would talk about the treatment effect on the treat it the treatment effect on the if there are any labor economists I'm sorry, I like it's been a long time since I've looked at all those terms, but basically the idea is You have the whole population or the sample who might be getting the treatment. There's only a select few of those individuals For whom the treatment is actually relevant. These are typically the people on the margin And the argument is that your skew that you didn't even start testing this the lower skews the skews that would perhaps be the entry point vice-dead So So, yeah, so I think you started a hundred gold or something like that, but the lowest queue isn't actually, there's no discounting there. And I think from a mathematical point of view, that was probably for, that was probably for methodological reasons, like you needed a solid control or something like that. No, it's just technical. I don't think we had, you couldn't lower the price because then I think the lowest price point was an actual price of five extra moves or whatever. So there's no, I guess you couldn't give them two or something for it, but it was like at the time it was like we were like, that's the bottom, that's the bottom one. I think there's been a technical reason we couldn't do it or whatever, but I also met, he bought me or we went out to drinks here in Malama. And so I've heard this critique. Oh, nice. This is how waves hit me. Yes. Yeah. And I've got, so 10% of the purchases that happen are for that greater than 99 gold bars. And so that was where you were testing was that it, what one could argue and Julian argues, like that atop 10% tiles, what about all these other people who aren't being impacted and how big of a difference do you think that would have had in your results? If you had been able to somehow access that bottom 90% percent. I'm pretty sure. Completely agree. I feel like this is such a bizarre critique from by him because we're talking about what is the relevant group to be treated. And the people who are making those extremely small purchases compose such a small percentage of total revenue. The people that you really want this experiment to get some juice out of is the people who spend a lot of money and those people tend to spend far more on average on a particular skew. Like they're buying on a regular basis, $25 skew or $50. They're buying more price point, they're trying higher price points. So to me, it's far more important that we understand what's happening to those users rather than the people spending a dollar or $2. That's not interesting and free to play context in my view. Phil, you're talking about intensive margin versus extensive. And I think that's like, I think to criticize this paper and say, well, it's no good because it's not exploring the extensive margin. We don't know who's converting. That's not what the paper is about. The paper is about, like you're saying, the intensive margin for these people who are spending. Can we get them to spend more? And it looks like no. It's, I think, we know way more about this now. So we've done a lot more tests and this was nowhere near as nuanced. It was pretty. This test must have happened in time. There's some of that. Or maybe we, I think we did it then. I don't remember the exact date of it. Another just side story. One of my favorite conversations with one of the leadership people also, they said, David, you called me up and they were like, this paper that you're writing. Are you sure it's okay to publish it? I'm like, I'm sure. It doesn't give away any very secret sauce. And I'm like, if secret sauce is fairly miserably, yes. Otherwise, don't worry, dude. No one's going to, this is not, this is academically interesting. It is not commercially interesting. Don't worry about it. It's only commercial interest for the gaps that it leaves. And the reality is that the big opportunity was still at that point. A small amount of people paid. We know that actually their pain habits, they're not improved by discounts. It's not driven by discounts. Most of the pain habits are driven by your experience. So your short term, pit term, long term goals of the game and how you relate to them and how you can intrinsically value it. And those things can be affected a lot more by the way the game structure than the pricing stuff. And at the time, the game was underpriced. I can tell you that without question. So there's another, we can get into the actual experiments. I don't want to do that. It's not over here. So what? So follow your heart. What is the learning that you take away from this in terms of external validity? Is it just that, hey, we can't do quantity discounting or hey, we don't want to fuck with prices? How much are you willing to? What is the scope of learning that you take away from some of these experiments? You actually, I think it was on Deconstruct or Fundright. It wasn't on this podcast that you were talking about the door one door two and door three. So just to set that up for listeners again, Fortnite made some changes in its pricing structure. What they chose to do was hold the amount of hard currency you would get constant and instead increase the USD price of the SKUs. So it's the exact opposite of what you did in this experiment where in this experiment, you held the USD price constant and you increased the amount of gold bars. There's actually one more scenario you can consider to change price, which is to hold the SKUs prices in real world currency constant as well as the hard currency that you would get at those SKUs constant and instead change the virtual currency prices. So there's three different levers, all with different costs and benefits that you could pull to fuck with prices in a digital game. Okay, wait, don't say anything else. This is really exciting. We've got some fresh blood here, Chris and Eric. Okay, so you've got those doors to walk through. Which one do you want to walk through? I'm 100% in door one, the one or night shows. It changed the dollar to V bucks, increase the dollar amount, leave the V bucks quantity sizes the same because you've tuned your skin bundle sizes accordingly. And very importantly, this allows for flexibility across countries. If you want to Brazil seeing hyperinflation or whatever, you can adjust the price there and not anywhere else. You don't have to worry about weird discrete effects with the price sizes. And it lets you are in game pricing. The people who price V bucks to content lets them be basically independent of what your decisions are making on the pricing side. That's fair. I think like for me, it would be typically when I'm doing an exercise like this, I always think about like where is the, where's in the player's mind, the final sale? Is it when they purchase the SKU or is it when they go to buy battle pass for that season? And I think, especially in the case of Fortnite, I'd be more upset if I found out that the cost of battle pass had gone up. So that's door three, correct Phil, where the actual costs in the game are shifting around. I think I'd be most frustrated by that because I have a huge, I have already a mental model of how many V bucks do I get each season if I play through the battle pass and how much is it going to cost me next season? But there's two different, there's two different purchases. There's two different points of sale here where I'm buying the SKU and then I'm also buying the battle pass. So I don't have a super strong, I think definitely the second door sounds like the worst to me. I agree with you Chris. I think I'm going to walk through door number one with Eric. I guess I'd also add that I think the effects between all these different experiments, if you actually got all of the equivalents right. So it was a truly clean experiment between all the different variants. I would argue that probably the differences between all of them would be, again, that's some pretty vague terminology. I would say I'm very skeptical of doing anything that creases the friction of a player having to make a real world purchase or having to go back and make a real world purchase again and again. If you have to do that multiple times over a given period, that to me is points of friction. Your credit card could be out of date. I'm going through the first party SKU process is rather apporant. It's not as easy as it is spending hard currency in a game. To me, when you have door number one, when you're raising the price of a SKU, but you're holding all those other things constant, the schedule with which you purchase SKUs can also be the same as it was previously because you're getting the amount of hard currencies the same. I gave this example and deconstruct your fun, but let's say you buy one cosmetic a month. If I'm just raising the price of a SKU, but holding prices in hard currency and the amount of hard currency constant, then the amount of times I'm going to refuel. The second thing I want to do is fill on hard currency is going to be the same. Whereas if you fuck with door number two or door number three, I might have to go back to the first party SKU again and make another purchase. In door number three, if the hard currency prices increase, that means I have to go back to first party SKU potentially two times to be able to buy that cosmetic or potentially what I need to do if the hard currency is less in door number two, then I also need to go back to the first party SKUs. This introduces pain. The only reason I'm skeptical of door number one is because I think people are more sensitive to prices that are in real world money than prices in hard currency. I think they have more heuristics around them and more comparative pieces of information, but I would still probably narrowly give it to door number one for the pattern. In addition to that final point you're making, perhaps, in favor of door number three, I would add this final piece to door number three. It's that people are terrible at discounting into the future. So I think I probably have to make sure that I give it to the big girls for that one. I get it to the big girl. No, no, no, no, not hyperbolic discounting. This is pure risk aversion, the inter-temporal discounting into the future. I just, there's some ambiguity with the future, I discount future value at some interest rate basically, and it's maybe it's three percent or something like that. So I see that future virtual purchase that I have to make. It's less impactful to me than money now. So money now is more valuable than money later, and that's why we have, that's why people require interest when they've lent people money. So to me it's pure like, pure theory, you don't have to go behavioral for that. I think they're interesting. So for me, it's much more in the game world. If you change the price of a thing like if a gun or five extra moves cost this arbitrary fake thing, they cost ten of these fake things for. And now they cost twelve. I feel like there is something, it's like I've been cheated. There's something that doesn't, it's like a, this is actually like, people get mad when the price of milk goes up and gets discussed like the price of it. I think that the, there is more of an emotional bond of the thing that I'm buying and that I think the actual, my relationship to purchasing the soft goods or transfer, whatever it is outside, what I'm using real money. I think that is much more of a budget question. What's okay for me to spend? And I think this is in like grand numbers. This might be more casual than it is mid-core, but I've seen if you look at payer behaviors over huge numbers, there's a lot of people that just pay five bucks, five bucks. Five bucks, five bucks, five bucks. And that's what they pay. And I think that you'd think, I've looked at this data and been like just by the 50 bag. Wow, but I think if you interview them, you'd hear people saying, I only want to spend five bucks on this game. And nobody says, yes, but how many times and how often that's not the ice spent five bucks. So I think that there are some sort of like methods going on in like how I think of my money. And if you change what's in the bundle, the best thing to do is make the bundle still look good, even if the actual thing that's like the currency or whatever that you get for it is less, but there's a lot of money. other bonus things that you don't have, you have less utility or whatever that will work. But I think that there's a difference in emotional relationships of these two things in how I see it. Also coming from online poker, I knew a lot of, so there's a while where I could see the data about a bunch of players. I switched from one company to another one and I knew a lot of people that played on the platform. And so I knew what their outcomes were. Like I knew whether they'd want to lose money and how many deposits they'd made. And I asked, you guys up or down, how's it going? And I could tell you every single one of them had themselves higher, closer to positive or positive, even though they were negative. Nobody hit the estimate right and certainly nobody underestimated. And I think it's a similar thing. You purchase a few times. If you ask somebody how much of you spent on this game, they'll be like a hundred bucks. So let's adopt this. Let's play in David's world for a little bit. Let's call it the budget hypothesis. If we assume the budget hypothesis is true, I think there's some testable implications or some things that should be true empirically. One of which should be that there's little difference in spend velocity for a given spender. So if we're to someone in a query database and they were look at the rate at which they make purchases, they should be fairly steady. It's not like they're going to drop a hundred dollars or an increase even. I might actually throw a little bit of cold water. I think there's some information. I might doubt that. So I think there's going to be some information gathering people are doing as they're playing games. I think they're favorite. They're dating. They don't know if they want to get married yet. And I think that relationship changes and at some point, like they do decide to get married. There's some information gathering they're doing very early on. So I might throw that, but it's testable. It's a testable hypothesis. What do you mean? We have ways to spend like a little bit of money to and then see if I like the game before I start moving off and spend more. Yeah, I don't think they understand their time horizons yet. And so when you get into a game, if we think about some things that you would purchase in some of the virtual worlds, the costs or excuse me, the benefit of that item that you could purchase is amortized over your lifetime of playing the game. And so the longer I play a game for some items, the more the value of that item increases, like a battle pass, for instance, that thing increases in value the more I play. And so I think players might be uncertain about their time horizons when they download a free to play game. And I think they figure out very quickly because we determine who's to spend or a non-spender usually within the first two weeks. You figure out like 95%. So I think they figure out the time horizons very quickly, but I don't know if it happens on day zero. But I still think there's probably something to your point. But if that were the case, if it's really a budget decision, if budget is almost exogenous, the decision to spend a given amount of money is almost determined outside of the game. So there are casual players and their mid-core players and they have different budgets. Then really the goal of free to play games for us is to maximize retention because we can't really fuck with anything else. We just need to fuck with retention and ultimately extend those budgets over a longer period of time rather than trying to get people to change them. That is our easiest lever to pull. And also if you buy an package and has less gold bars in it, at least in one of these games or whatever, and product costs has been the same. So my package usually when I spend ten bucks, I'm getting less stuff for it. I run out of money quicker. So I might actually spend more often now to spend my go up. I just think that people don't have very good. They're not keeping track of how often that spend goes up or goes down. If you ask some people, just go and ask 100 people how often do by candy bars. You think there how many of them you think nailed the number or like underestimate or maybe they underestimate. They go like a few times or whatever. Would you stand behind your budget theory for both cosmetic based economies and more mid-core games like 4x? Or do you think this is specific to something like casual? I don't know. I wouldn't say budget. I don't mean budget. I'm reasonable of reasonable amount. Do you understand? Like I'm saying if you like there were a range of. It's more like that. If you said to me like how much is reasonable to spend on this? I might be like five bucks. This is no time to spend. I'm not going to spend the $20 one. But I might buy the $5 one a couple times as I hit pressure and walk the thing to move forward. Whereas somebody else might be like I'm happy spending $2000 bucks in this game because I love. So I think that there's like a whole much I'm willing to put into the machine without feeling shame. I think it's more like that. Not budget necessarily. Tell me about the experimentation group. I love what you guys are doing and I'm not politically very active. But one thing I really care about is I wish games and tech companies would share more data for the sake of science and advancing our understanding. I think we as an industry can make better games if we collaborate and share more data and teach each other things. So I love what you guys are doing with the experimentation group. What's your address? Where do I need to spend the check, Eric? What a fucking layup, man. It was through the joint as head as brand marketing. Wow. Yes. That's my contribution to the movement. I think there's a whole bunch of stuff that's interesting and I think that you guys I'd love to have minds like yours looking at things that are going on in games and actually get more ideas because I think there's way more left to do in mobile games. There's way more interesting things to test. There's way more to try. A lot of people are just in the same like lane doing the same stuff, but they've always done just borrowing from some other things and plugging them in. And mobile games is the quickest place in the world to go from like an idea to a thing in front of like a million people. Why aren't we just doing a bunch of interesting things and trying to find out what happens? No platform ever has been able to deliver something to end customers at this size at this speed with data ever. It's insane. So let's start actually figuring out what people give a shit about and really discover what really matters to our players. Yes. That's our tagline. We need to get that in David. Yeah. That's what it is. Yeah. Yeah. I go. I was going to say, does this lead into what you're doing now with? So for listeners who don't know, by the time this podcast drops David, myself and Tom store, who is the product lead over at King with David will be launching the experimentation group. Very similar to what David had at King also called the experimentation group. You'll find a link to the experimentation dot group in the show notes. We are trying to get companies to run experiments with us and we will do the experiments with the company for free. We will help grow your game. We think that we can use the science and the experimentation playbook to grow games and ultimately make products more profitable and we will be doing it for free. That is right for free. But the thing that we are going to do, the twist here that I think everyone listening to the game economist cast is going to love is that we want to do it out in public. We want to do it out in the open as long as you're willing to publicly share some of the experiments that we set up when we run them. The results. We hope to get a public conversation going about what we're doing and how we're doing it. And so I know all you game economist cast listeners. I know all you game economist discord members. You're going to be in those comments telling us we're wrong. And honest to God, hope to do. Honestly, God, hope you do because that's how science happens. And every time I hear Supercell doing something in Finland, it's like they're sharing their dashboards. I'm pretty sure if you have a finished ID card, you get access to Supercell's dashboards. They're extremely public and open about what they're doing. And I feel like they're so little of that in the game industry. And it's time to show everyone not only how you do science, but also how you have a public conversation about games. And we've been lacking that for so long. And I'm really excited to get this started with David and Tom. And if you know someone who has a casual game and has over 10k DAU and is able to run an experiment, let's get in contact. Let's talk about it. I'm sure they're going to have a million questions. What does it mean? Things have to be public. What are you talking about? How much has to be public? Do we have to release the Excel file? There's a lot to talk about. And we're here to talk about it. So reach out. We'll have a calendar link as well if you want to book some time with us. All right. Sorry. Go ahead, David. Now I got the picture. No, I think it's a really interesting reason for companies to do this. For game teams to do this is an opportunity to run a test, get together with get together with us. We'll propose a test. The results can be public. And as soon as the results are public, amazing people like Eric and Chris, you guys are going to be able to look at them and be like, wait, why didn't you think of this very? Or if that's true, what about this? So it means that the single experiment you ran that you want to keep in your little house and not telling anyone about as soon as you let it out, you have a world of smart people that are going to start talking about what they what else you could do and other ways to think about it. So I think it's going to be a talent magnet for teams that want to try this. And I think the other thing we were talking about earlier today is that their knowledge spillovers in games, even if they're not public, employees go from place to place and they take knowledge with them and then they spread that knowledge at those other firms and then the people at those firms spread that knowledge elsewhere. That's how you end up having all these agglomation effects, different parts of the world when you make games. And I know David a lot of the learnings that came out of the experimentation group at King are already in all of these companies. And a lot of King employees have not gone to these companies yet somehow they figured out about them. It's those knowledge spillovers. So people are concerned about doing these things in public. I guarantee you it's already happening. It's already happening. It just hasn't been accessible to people and it's time to make that accessible for all of us. This is a rising tide that will lift all boats. I'm going to I'm going to be the greedy monetization person here. Are you guys offsetting? Are you pulling an Uber and offsetting the costs right now for like an inevitable price hike in the future? Is this just pure like what? Yes, we're on an economy. Is this no money? What's the money about science? Be surprised. No, it's not about science. It's never about science. This is an account that we cannot possibly. Okay. Interesting. I mean, I can say this Chris that I really like disruption and I have sacrificed in my life cash for the opportunity to disrupt things on a regular basis. And it's not a problem for me. And I think I got a track I can show you pay slips if you'd like to prove it and missed promotions and whatnot. And I know actually that Phil, this is like indeed from the first moment of contact with Phil. He was okay. This is a conversation with Phil. I don't know if this is how I remember it Phil. Our first phone call. You're like, so wait, how are you doing these experiments? I'm like, we just make the change in the release experiment. Okay, okay. But how big are these like these treatment groups? Is it like you got 5% that's in the treatment or whatever and not 50 50. What? At 50 50. Otherwise, you don't really get to see the signal. Wait, you just put 50 people 50% in the treatment. How big of these games are they like 10,000 or something like millions of dowels and feel just like this. Oh, and it just melts it. It was a glass ice. water to someone in hell. It was not running a B tests. Nothing. They had nothing. It wasn't Zingat Scopli. There was so few A B testing around. In fact, I don't even think we ran a single A B test in the time I was at Scopli. And to hear that someone was doing this was incredible. I'm just as a pitch for Phil really cares about the science in this. I think that you're running into zealots here. These are not normal economic actors. Sorry. Chris, trust me. I have a lot of ways I waste my time. I can tell you this podcast ain't making money. That's definitely true. If it is, we don't know any of that. I think this is really interesting. This radical movement. I definitely like it. One thing I want to touch on since we've in this conversation, we've mentioned John List's name. He recently released a book called The Bulltage Effect. And it's all about how to make sure that the ideas are scaling properly. So the idea that, hey, this looks like a really good idea. Even our A B test show that this is going to be great. But once we implemented all of a sudden, it actually shifts the prior, it shifts the entire distribution in a way that in the long run isn't necessarily good. So I think about if you were to run a promotion, that you'd get a result that said, hey, the promotion brings in more money. But you can't run the promotion forever. It doesn't, that's not how it works. Promotions aren't scalable. Most promotions aren't scalable. How are we worried about that? Do we care about that? Are we just throwing caution to the wind? Are we saying fuck the Bulltage Effect? And just look at me double that. David, have you ever replicated an experiment? Have you ever run a replication test? What do you mean? Do you mean like we ran a test and then ran it again in other places? You mean in other games? Yeah, and all the time. Everything that we did. I mean, we were passing it. We were passing it. No, we were passing it between games, of course. So everybody did it. And we have extreme arguments on the different impacts. But yes, replication was constant. The battles were constant. The, how was it done? What's implementation like? What is the ranking of it? How often is it shown? They took a lot of our stuff and made it live-ops to save it always on. And then there was a battle about why. God, it's the same replication is triggering. So, hold on. I was rude. I was rude about Chris' question. He is right about the Bulltage Effect question. I don't know if you've read Johnless's book. I actually-- I haven't. It's quite an insightful piece. You should actually check it out. First 50% is really interesting. I love the first half. Yeah. Not going to lie to you. It's like me. This is an comedy podcast. Oh no, I wasn't joking. But I didn't see the joke. OK. [BEEP] The thing you said about game designers wanting all players to have the same experience, I think, is a really-- I don't want to call it like a moral debate. And what counts as the same experience and all that. Yeah, that's what I think a great thing is. How to convince your designers to let you run a BTS? And not only that, I mean, David-- and one of the things David had mentioned to me while back is that game designers make the best experimenters. And I found that to be true once I've started to talk about experimentation. You have to flip them to your point, David. You also mentioned this, too. But when I've gotten them to really sit down and think about these type of models, they understand the player experience in a way that no one else does. And I think they get at the heart very quickly about, what if I change this? They're extremely intuitive about this. If you guys are joined anytime, and you can even just have me on for 15 minutes, and then cut me off, so you can talk about intelligence, stuff, or whatever. But just really interesting, we had these-- whenever new game design people would join the company, they would come, and they would-- so we had this process, I think, to cover this one of the time also. We had a hot list. So the idea was, you needed to keep game teams autonomy, not just because of the political game team you need to choose themselves, but a game team that wants to do something does it way better than a game team that's forced to do something? So basically what we had is we had like a peak. We have 10 game teams, just like that, working from Seattle, Berlin, London, Barcelona, all over the place. And so we'd have this list, we called the hot list. And it was like 10 ideas or something. Sometimes it was 20. And it was like these ideas we think are interesting. We had another process, which was like how to get onto the hot list. Anybody in the company could put in ideas. And we would look at it. Sometimes it was someone who'd already tested or it wasn't part of our strategic priority, priorities at the time or whatever. So we'd have an open meeting, people would come, we'd tell them how we chose this over. The amazing thing is every time creative people showed up and it was like it got to the point where it just pained me. They'd be like, you know what I think we should do? If you take pricey and it was like, price, Jesus! Like you're a game designer. Where's the creative? And they're like, yeah, pricey. If we can get them to converge. And then you'd get these people from the NBA would show up. These people that have been to Boston Consulting Group and they'd show up and they'd be like, what if game board pieces had faces? And you're like, why are you pitching these creative ideas? It was just the strangest thing. It was just such a bizarre, it was like they just crossed wires. But just like you're saying, Phil, every time you got one of the game designers to want to find something out about what really matters to your players, do you want to find something out? Then go ahead and pitch something drastic. What if all their friends on the map were gone? Would they care? And we're like, I don't know. Would they? We don't know. And then we tested stuff like that. And it was just fascinating. So every time you flip one of them, they start to ask tough questions themselves. - Let's wrap. - I wish we had another hour. - I think we'll definitely have you back. - I love you guys. It's been really cool. - It's too great. - I think you guys do great. I love your podcast. Thanks for doing it guys. - Yeah, in the rolling part of your view, guys, listen to that too. You had a great episode on MonopoliGo. And you've covered social networks and app stuff, which no one else is talking about. I got in to be real because of you guys. - No. - And it's failure and it's an ability to grow. And I would actually be real before we came on on the podcast and I'm like, "Oh, wow, I really get what they were talking about." And I would have never been exposed to that if I hadn't listened to the rolling product. All right, cool beans. - We should teach this to our children. - Economics is major, major. - Everyone has to major in economics. Number one for personal survival. - It's like an economic, it's amazing.