Why Your Free Users Are Your Real Growth Engine – Cem Kansu, Duolingo CPO

Welcome to the subclub podcast, a show dedicated to the best practices for building and growing app businesses. We sit down with the entrepreneurs, investors, and builders behind the most successful apps in the world to learn from their successes and failures. Subclub is brought to you by RevenueCat, thousands of the world's best apps trust RevenueCat to power in app purchases, managed customers, and grow revenue across iOS, Android, and the web. You can learn more at RevenueCat.com. Let's get into the show. Hello, I'm your host David Bernard. My guest today is Gem Kansu, chief product officer at Duolingo. On the podcast I talk with Gem about the premium trap many apps fall into. Why free trials work, even for premium products, and how try for zero dollars and zero cents actually outperforms try for free. Hey Gem thanks so much for joining me on the podcast today. Happy to be here, excited to chat. So I wanted to talk through Duolingo's monetization story. I mean, few apps out there are going to have tens of millions of users and being the position Duolingo was in when you started monetizing, but I think there's so much to learn from the lessons you all learned along the way. So let's talk about that. Duolingo was a free app famously, completely free. And then the first step was layering on ads. It was actually was a website. It was not a mobile app. And this website's idea was actually to be completely free. So there would be actually no monetization for the users that are using Duolingo to learn a language. However, as users did exercises to learn a language, which includes a lot of translation. So if you're learning Spanish, a lot of your exercises, you're translating Spanish to English or English to Spanish, users would do these exercises to learn, but there would be a crowdsource model to collect those translations and eventually turn that into a translation service. We actually did this like this was built. So I heard the story, but I don't know you actually this was built this was this went live and actually generated some revenue, even though very small. So we would do think we would have customers that are like at the time, like the big news outlets as well, like CNN buzzfeed. They would feed an article they wanted to translate from English to Spanish. And then our users in bits and pieces would translate that article and we would crowdsource a translated article and then provide a a translated article back to CNN. It was actually a genius model. If you want to do something crowdsource, it's actually you're kind of serving both sides a positive value. The challenge with that model was though is twofold. One of them is translation. Also, this is pre-element. Like translation now has changed a lot post-element, but this is you have to remember this is 2013 as well. Even then translation was a race to the bottom business model. So if we were providing a service for, I don't know, how are three cents, ten cents, three dollars, whatever it is, there was always somebody else trying to do it for a little cheaper by having someone in the Philippines translated for cheaper. So there's always a race to the bottom. It is hard to scale for that reason. And the number two is more fundamental, which is the translation quality of a language learner is actually not great. So you end up putting practices in place to uplevel the quality of that translation, which then you turn into a basically a translation business and then really start shifting your resources towards basically translation and not teaching. So we realized that had these these problems. And around the time I joined Dueling Go, which was 2016, we had pivoted away from that model and was trying to figure out what the next model should be. Actually, our very first experiment with anything monetization was the streak repair as an in-app purchase. And around the same time the ads experiment started, but I think the first thing that user saw at Dueling Go as like I think they could buy was repairing your streak for I think a dollar or three dollars. That was our first entry. It was honestly it was a good experiment that we actually learned a lot. We learned that users really cared about their streak. And some a good percentage of our users are willing to pay a dollar or two to repair their broken streak. But then I think then we get into we got into this experiment mode of what really works with our audience. And how do we make sure that we keep the product highly attentive and a great user experience? Coming back to your question, that is the moment where we started trying ads. So that was 2016. It's funny if you launched, if you go watch and I think these talks are still out there, our co founder CEO Luis gave this talks for Dueling Go's launch and he said we will never have ads never in-app purchases and never subscriptions. And those are the exact three things that I built in that order basically. And the reason he was so adamant and saying those was we Dueling Go's mission is basically centered around providing access to education. And at the time every education business model, whatever they did, they would lock the education value behind the pay wall, get someone to pay. Think about there was at a stone model, for example, that was the very stem model at the time where you're selling content. We never want it to be that. But I think we realized there's actually a way to build do that, which is provide access to education, but also monetize the bells and whistles. That's why we set on this premium journey ads fits into that really nicely. If done well, users don't have to pay you anything, but you can actually generate some revenue. So ads was that test. Honestly, testing ads wasn't rocket science. It was fairly easy to test our way into, of course, making sure the user experiences great, but also how do you maximize revenue from ads while making sure your retention remains awesome. The hardest part I would say was the cultural shift of going from a pre-monetization app to a post monetization app, because until then, so until the time I joined, there was like 60-ish employees. I think a lot of folks joined the company thinking it was maybe even like a nonprofit-like mission, because the mission is so grand and so awesome. And the company never quite bothered to think about revenue and monetization very much. It was a mindset to the company to be, hey, there's things we have to do. So we can generate revenue. And if we generate revenue, we can actually fund our mission so that we can be more impactful. Was probably the hardest part of turning on ads on Duolingo. I guess you weren't there at this time, but I did hear a story out there around venture capitalists pushing this direction. What was the story there? This story, I think, is when capital G invested in Duolingo. Leila, who was also on our board after that investment for a while, she told Luis at the time, this is probably the last investor you will have in this company before you figure out monetization. So in a rounded way to say, you have to figure out monetization. As a growing company, you at some point have to do that. But I think also the common wisdom at the time for consumer companies like Facebook is like just grow audience. You'll figure out monetization later. So I think Duolingo followed that. But until a point where I think capital G and Leila had this big push to be, okay, it's time. And I guess that was around 2016. So when I joined, the mindset shift has had moved to Luis's, like, first thing he told me was like, we hired you. Your sole job is to figure out how to monetize Duolingo. And that's that's what I tried to figure out. That's awesome. What a great story. So ads, they're tricky. How did you introduce ads? One in a way that didn't totally piss off users and tank retention. And I mean, I'm sure there was user pushback and bad reviews in the app store. This is part of the story. I think a lot of folks can learn from because any monetization change you make will spur some level of like bad reviews or negative feedback or churn. So how did you balance that as you started to roll ads into the product? This is a very good question. I think that there's the, I'll give the obvious answers. And maybe the hardest navigate parts. First, for a consumer product, you should obviously dog food your product heavily. So so that whatever you're doing to the experience, you have firsthand experience on how it feels meaning you can easily say, oh, we'll put an app open ad. And if you're just watching metrics, I'm sure that might look fine. But when you test it, you're like, what the hell is this? Then you should know what you've introduced. And obviously, I've worried the things that you wouldn't want to see as a user. Hence, our first test and honestly, kind of even today, that's our only ad placement is after you've competed a lesson, we show you an ad. Now, not during, not before, but always after because we believed that doesn't interrupt your learning. So one is the qualitative. It should be a good user experience. And you should know that by testing the app yourself every day. The second one is quantitative, like measure everything you can, like his retention fine is app closes fine. What's happening to your reviews? You might have to make some trade offs. Ideally, you wouldn't, but realistically, if you go from zero ads to some amount of ads, there are some risk that some curves might come down. But I think you should know what that cost is. And anyway, that's like through a B testing. That's how we measure it to be. Honestly, our retention was maybe a slight hit, but mostly neutral for most of our ad tests over time. That's one. The third one is there is, of course, you go from a free product to selling something, users won't always say, thank you for doing that. Like this, imagine if Uber started completely free. And one day they said, okay, now we're going to charge for our Uber's user would hate that too. And that was kind of what we did. I mean, in a way, we provided a free service and we kept a lot of the elements free, but we said, hey, actually, you can buy a street repair now. And you might have to see an ad at the end of your session. We got some complaints. The thing that helped us have clarity was one retention retention was fine. Like retention. was neutral on most of these experiments, and when you looked at our community's complaint volume, if you quantify that, for example, for per a million DAUs, how many ad complaints are there, or ad negative reviews, what a however you want to count that, versus other topics. Like, for example, I realized this is, I did this quantitative analysis in 2016 because we were at this pivot moment of like, do we launch ads, or do we not launch ads, and we looked at, for example, the number of complaints about why we didn't have the finished language on Duolingo was three extra number of people that are complaining about Duolingo having ads. So that starts putting it into perspective. Like, it's not zero complaints, but actually in the grand scheme of things, this is not going to break the app, and our community is not going to run away tomorrow. So I think understanding the volume of what you're doing to the community, as much as you can, gives you perspective. Otherwise, every negative comment as a product builder hits home really hard. Like, everything looks like the world is falling apart. That's why it helps to like, look at the broad picture of like, all of our issues as a product, and all products have many issues. That's exactly what I was going to say is that it's so hard as a builder. It's really hard to get those few negative reviews. And I think there's a really helpful insight to look at that bigger picture and understand, like, I got a bunch of negative reviews from my weather app for the audacity of charging $40 a year for a weather app. And I did get probably over focused on those negative reviews. But then when you look at all the people who were we're subscribing, we're staying subscribed. Actually, I have pretty good retention. It's like, you got to like zoom out and not let those like that small subset of users who are going to complain tank what's ultimately going to help drive the business forward. How did you work on ad quality? I believe you implemented ad mob first time. And again, in my weather app, I had ad mob. Eventually, I ended up pulling it because it was like gambling ads with like the 777 and animations and like all this crap. How did you work with ad mob and how do you think about ad quality as part of the experience and how it can actually be detrimental to the experience? Yeah, I mean, ad quality we, I would say we are like incredibly strict on. But with programmatic, you'll of course have some amount of control, not full control. But with anything that is honestly, it could be even slightly, I see we blocked all those categories. Like if it's gambling, even if it's like, games that look kind of sketchy, pick your category, any of those that we didn't feel was also young audience safe and or just brand safe, we blocked. So the starting point is we used ad mob first and then we integrated other sources like Facebook ads or at the time flurry. And and now he's also I think multiple networks. We use unity. The that was to mainly broaden demand. But for category and what kind of ads we allow, we tried our best to block off any categories that are dicey, even though it might be a revenue loss. We're like, we don't care. The most more important thing is good user experience and ad quality. So we did, we did a lot of category block category blocking. And if we needed to do custom blocking, we went and tried to work with ad networks to be like, this type of ad has like three X buttons to exit. We don't want this in our app. Please don't give us this thing. So we also did some of that back and forth as well. That's our stance this year also to improve ad quality. We're also doing our own direct ads more and more because then we control quality entirely. Like it's an advertiser that we know like we shook hands with and the format is what we've created. So it's beautiful. And it's a great advertiser. So one now new lever for us is to reduce programmatic volume and increase direct volume because it's just a better user experience. I love what you're doing. I mean, it's hard for smaller apps. You don't have scale to do this kind of thing. But tell me more about how these partnerships work to improve that ad quality. So that it does improve the user experience because you're going to be incorporating dolingo characters into the ads with the branch your partner with. Yeah, you've seen them. It's very cool. But I read about them. It's very cool. It's currently only in the US. So we're focusing on growing these in the US because obviously we create a great format. But to be able to scale this, we also need to be working with companies. And right now we're just focused on companies that would want to advertise with us in the US. We created multiple formats. But the one I love the most and that seemed to be working really well is one of our characters. So we have eight dolingo characters on dolingo that you see is introduces the ad. For example, pick a brand. Let's say it's a detergent. I'm making this up. It's not like we had exactly this. But one of our characters lily lily would make jokes about like, Oh, like I need to wash my clothes. I wish I could do that. And we kind of write a custom script to introduce that ad. That's very short. And then the ad itself plays so that at least we are building a little bit of our character world further. And it's not jarring as like you ended a lesson right now. That's what our ads are really a little bit where like you see an ad. It feels like you're easing into the ad. We're trying this right now. It's when you see it as a user. It's it's actually I could as an ad. It's a very delightful format. Our challenge I would say is scaling it. I think we have, since it's a custom format, well, you got to go up to advertise and explain how this works. So it's it's a bit of a slow adoption process. But we're going to, I mean, our goal is to basically increase the volume of these ads and reduce the programmatic by a good amount in 26. One of the ads you do often show is for the subscription itself. How do you balance from an LTV perspective and monetization perspective? How do you balance when to show those ads for the subscription? And then when to show programmatic or these new brand ads? So we just to set the stage, if you do a dualingo lesson, and this includes any of our lessons, so you could do language, math, music and chess, at the end of that lesson, you see an ad. And we decide, like you said, if we should show you a programmatic ad versus a one of our internal ads, which could be an ad for a subscription, or it could also be an ad for our other products, like the duolingo English test, but we call those basically internal ads that we decide we've produced. And it is a dualingo product. Majority of them are going to be either super duolingo ads or dualingo max ads. The decision logic is mostly now controlled by ML actually, but the ML uses various signals that is trying to figure out LTV maximization at its core. And it is looking at every signal that we feed it that could be an indicator for someone that is more likely to buy super. Or it also balances out, of course, like if, for example, it's shown you the same, it doesn't really show you the same super ad three times, but it's shown you super at three times that clearly you're not interested in. It backs off and shows you other ads. So it is basically built to optimize LTV with a likelihood to purchase being the objective function. A lot of apps that have ads in them don't do this kind of thing, but like as cheap as the CPMs are these days, I think all developers should be experimenting with putting ads in for the premium product. It's a perfect way to kind of balance that monetization of if you have a premium product and you're not making that much on ads, which you just don't and actually follow up with a question there. But, um, but it's a great opportunity to like remind people, hey, there is a paid price. There are features you can get and highlighting the benefits. I've seen quite a few of the ads early in the journey when I was first using Deolingo. And they're really good. Like they've really highlight well, like, hey, you're doing great, but you can do even better and they really highlight the product features. And so I think that's a great opportunity for premium developers who do have ads to be exploring that more to kind of push that upgrade to the subscription. 100% and I think the other reason I like our ads is they're just better than programmatic ads. Like a user, better user experience. It's just a better user experience. I mean, I think if I had a magic wand, I'd rather have no programmatic ads within my app. And only internally produces because they look better. And we're also building kind of lore for our world characters through even though they might be talking about super, they're also cracking jokes about their own. For example, we have this kind of controversial super ad. I don't know if you've seen this one where junior one of our characters isn't a bathroom stall. And he like peaks and like makes like a roll joke because there's a toilet roll that he's like peaking under. It's kind of weird, but it's like we're kind of also entertaining a little bit with the ads we produce because we have produced them with that intent. It's way better than showing programmatic ads. How does ad monetization do per user compared to subscription? And I would assume part of the idea with these partnerships and other things you're working on is to continue pushing that monetization better and better for those free users. So for us, if you look at Duolingo statistics, subscription revenue is far and above the biggest chunk of our revenue. So I think it's above above 80% and adds as a much smaller percentage. But if you look at payers about among our DAU base, actually, MAU base, it's about 10%. So 10% of our users subscribe, however, subscription is more than 80% of our total revenue. So we focus naturally when it comes to what we think about for monetization, mostly on the subscription side of things because we believe it has one much, it's already a much larger business, but also much higher potential because we still feel like with any of these actually, but certainly still even with the subscription that has now become a sizable business, there's just so much more to do that is making the subscription better. And also with the Femium app, you realize over time, then this is a lot of experimentation we've done. One is like building the product to be better. So making our subscription features more valuable, but there's a lot of work to be done and also making sure you can communicate those well. We touched on videos, videos certainly help because that gives you a format where you can actually say, hey, this other thing that you could buy has all these features because there's no a lot of other easy places to tell that story. On top of that, there's like how you design your purchase pages, make a difference, for example, how users enter that full, makes a difference. So anyway, that's in my head how you're communicating your subscription. So we're putting more and more of our monetization energy into those pieces, meaning make the subscription better and more useful and make sure we can communicate it well. That tends to have higher return for us in general as a business than ads. So most of our efforts is on subscriptions. Yeah, I mean, you know, Facebook's numbers are just insane. You know, I think that's all like 250 per user per year in the US, which is about what to do a plus subscription cost, but the time and app, the way they're able to do their bid structures and things like that. But I imagine there's still a lot of room to grow, but it's interesting that at Duolingo scale, the subscription opportunity is still so much better than the then the ad monetization. I think so. And I think with social media apps, the reality is it is fairly easier, I think, as a product problem to integrate ads into a feed. And in a product where users are there not for a particular utility, but they are there a lot of times to to pass time or they don't even know why they're there. So I think it's way easier to increase ad load in a way that is not disruptive to how the user feels. When you are there on an app for a particular utility and for us, a lot of users are there to learn a topic. We're not a lot. All of our users are there to learn something. I think increasing ad load is not particularly that tolerated. I guess. So I think that that creates a challenge for a challenge and opportunity. Honestly, for us, subscription just fits as a better business model than increasing ad load the way Facebook might be doing. Yeah. One last question on ads and then we'll move on to the subscription. But one of your the product managers and your team came on the podcast. I think 18 months ago and he talked about holdouts that you'll do long-term holdouts. Did y'all do a holdout on ads? Are there people today? Is there like 0.001% of subscribers that still don't have ads that you can measure against the people who do have ads? Not anymore. But we did run I think almost like a year or two year at long holdout for ads. We do run long holdouts. Although I don't recommend year-long holdouts because what also eventually happens is you forget that you have a holdout and then eventually you introduce something. You're not testing it every day. You introduce a bug to it. So like even though you have a year-long holdout, you realize at some point you actually polluted it because you're like introduced bugs there. Anyway, so very long holdouts. We are anti just because of a practical perspective. But we do weigh more like three months, six months holdouts on things that we know have different short till or we assume that we'll have short till long-term effects that might be different. One of those for example, this is not exactly monetization but anything that has to do with our leaderboards. We try to run long period experiments or long holdouts because on the dual-ingo leaderboard set, if you start from the lowest league, it takes you 10 weeks to get to the highest one. If we change the dynamics of how the leaderboard works, we might not see the actual result in two weeks. So we kind of let these run for longer. Another one is push notification experiments where every, of course, new push notification you introduce has amazing returns because it has amazing novelty. But then if you wait out six weeks, sometimes it doesn't just come back to neutral. It actually turns negative because you're sending too many push notifications that turns people off. So some stuff that we assume, not just monetization, might have longer term adverse effects, we run long holdouts or just run a long experiment. The way dual-ingo test things this fact, we could talk hours just on that. But I do want to get back to the monetization journey. So at what point did it become clear that you needed to introduce a subscription? And this was pretty early in the Apple famously opened up subscriptions to all apps in 2016. I believe you launched a subscription pretty soon after that. 17. So what was the story from, okay, ads are starting to work. We haven't pissed off all the users. But now we're going to like actually start charging. So the journey we went through was ads was the first experiment. And then after ads was introduced as a good PM, I was reading every review that was written on dual-ingo that included the app store or Reddit or our own forums. And there was a significant amount of people that were saying, can I please pay you guys? I don't want to see ads and I love dual-ingo. I get so much value out of it. It wasn't like a hateful I hate ads. It was more like, I would love to give you guys a few dollars just to remove my ads. And we thought that was a good idea. And subscriptions were quite early for mobile apps at the time. You're right. The majority of the app wisdom was generally you want to sell IEPs. And I think games had obviously like kind of popularized that. And that was pretty common. Subscription apps were earlier. There was the big like streaming and entertainment. But outside of that, there wasn't a lot of digital subscription behavior. So we took that idea to heart. But we said we're not sure if the subscription thing is for us. Our mind was still like, we probably going to have an IEP model. So we tested an IEP, which was honestly like looking back. It's so janky and it's not clear why people bought this. But we said you can buy an ad free month of dual-ingo. So for 30 days we'll turn off the ads. But it's a one-time purchase. And we tested different price points. We tested three dollars, five dollars, and 10 dollars. And everyone thought this kind of wouldn't work. Like it's like why would people pay this one off bizarre exchange? Honestly, the uptake was kind of good. And way more than what we expected. So we were like, clearly there is some demand for turning off ads. Even though we created the most bizarre package to sell, people want to buy this. Let's make it a subscription. But to make it a subscription, just no ads felt slim as a subscription service. So then we built offline capabilities. So we said you can use dual-ingo fully offline and turn off your ads. And then when we designed the purchase page, we're like, this looks really weird with two features. We need a third feature. But we couldn't actually at the time figure out the third feature should be. So we kind of wrote a line there that was the thing that users were telling us, which is support dual-ingo in its mission of free education. I think we just said support dual-ingo's mission for short. And that was our first entry into subscription. That was dual-ingo plus. And honestly, what I think we saw happen was probably majority of users early on bought it for the fact that they enjoyed dual-ingo so much and wanted to contribute back. The other half, or maybe the half of the demand was from no ads, then minimally for offline. And that was that kind of start us giving us a run rate. Obviously, I mean, I'm happy to talk about it more. But from then to today, the subscription has evolved a lot. But that was the very first version. Yeah, that's fascinating. And I do want to go through that journey. So that is pretty light. I mean, again, this was 2017. So best practices and everything else hadn't fully evolved. And this was just an experiment. But how did you start evolving that over time? What were some of the, and maybe just kind of give us the highlights of the last almost decade now eight years? The kind of biggest unlocks. When we added this to the paywall, when we added this new tier, what were some of those big unlocks over the past eight years that really kind of moved the business forward? There was a few. And let me try to think 10 years is a lot of time. There's a few levers that have looking back have really either caused step function increases immediately or over time, it really compounded. So one of them, I guess now, if you work in subscriptions, now this is like kind of a no brainer. But at the time, it really wasn't having free trials is one. I actually, as a PM, I resisted free trials for a long time. I assumed it would like completely flop. Product was already free. Yes. Exactly. It's like, why would people want to try free trials for the exact same product they've used for years? The product is free. Like if we were a fully premium product free trial, and I had made a lot of sense, but for premium, like you said, it didn't, I thought it didn't make sense. Turns out I was wrong. Free trials are really helpful. For the obvious skeptic user that might not want to pay up front, but they are interested in trying it out. It just makes it so easy to just jump in and try. It's very intuitive. Now as a user, I love free trials because I'm like, well, I don't want to commit to pay. Let me give it a go. Were there specific things that were then behind the paywall that people needed to experience? Not really. Really. Offline. You can say offline, but it's offline is the same as online. It just works when you're in airplane mode. So if you imagine this subscription, which is no ads and offline, you didn't really need to experience it, but I think it just removes the payment hesitation. So it lowers the barrier to entry. So you're not committing payment. You're like, I'll give it a go, and I think that's the real benefit of free trials. So at this point, we have, we're very pro free trials, and I think our purchase flows are now designed to highlight how our free trial works as a mechanic. We even have design pages which like communicate, hey, there's a 7-day free trial on day five. We're going to send you a notification to see if you still want your subscription. Because I, and this has performed well in AB tests, because I think users have a subscription aversion, and I think in a lot of countries, maybe not as much the US, there's also subscription scam aversion where you click on something you somehow like can't get out. I think if you highlight how what a free trial is, one is like, you don't have to pay anything really, and out of what the mechanics of it, it relieves this subscription aversion. Well, well, we're on the topic. I know you recently did an experiment with a try for free button. We've been experimenting with a lot on everything our purchase page is, and maybe it's worth talking about. You mentioned AB testing, maybe I'll take on, take us on a tangent for 10 seconds. So, The dual-lingo concurrently runs somewhere on 400 AB tests, so the dual-lingo app itself is heavily evolving real estate and a good chunk of those, not majority, but a good chunk of them also happen on our purchase flows. So if you look at any element, whether that's like duo's flight path on the super page, or what the copy we write, how it's highlighted, how it's underlined, what the button color is, what the button copy is, all of those elements, we have teams trying to figure out how to make them better. And of course, for the record, we're talking a lot about monetization. That's not the only place in the app where we're doing this, like our onboarding flow is the same. Our session end is the same. Our lessons are the same. So we take kind of like maximal care into improving all these flows, but monetization is no exception, but that button copy we've experimented a lot, and that is one learning that I honestly didn't have or didn't really expect. What you say in the purchase button makes a massive difference on your revenue curves. We have tested many things, there's a couple magic strings that we're like, you look at the graph and you're like, there must be a measurement mistake. Like why are the numbers so green? The two specifically were, we went from, I think, start my free trial. So I think we said something like, subscribe, that was, that's your baseline. And then we tried many things, but one of the big up list was start my free trial. That was very good. And then we tried, what did we say, try for free, that was better. And then we tried try for zero dot zero zero, weirdly, that is the string that I think hacks your brain to really grok that that button is entirely free zero dot zero, like two zero decimals. Now I think that's what's live in the app if I'm not mistaken. And that performed really well. And just changing, try for free to try for dollar sign zero dot zero zero. You saw a meaningful left in conversion. Correct. Yes. I guess the start trial conversion, but then also the trial conversion to pay, like both numbers went out. Correct. Wow. Yeah. Yeah. It was unexpected. Honestly, it's like you wouldn't think this makes that much of the difference. I think one other thing I've learned with freemium. And honestly, any subscription and paywall behavior is a lot of, when you go and design these flows, you are designing with logic, right? You are saying, let me communicate my features, so whoever is going to buy should read them and understand what they're buying. So they can do the logical exchange or if I part with this many dollars, I will get this many features. That is how you go and design a flow. But the reality is humans do not experience mobile flows that way at all, actually. It's way more based on emotion and reaction than, oh, let me read and think about the value exchange of what is happening here. It's way more emotional. So I think these these things that are kind of sound like, you know, they're saying the same thing, right? Zero to zero and free are the same thing, but they create different emotions, make a difference. That's fascinating. Well, we've dove down multiple levels of rabbit holes, but to surface us back up in thinking through the past 10 years, were there any other kind of key unlocks specific features that you put behind the paywall that really moved the needle? Definitely. So we talked about free trial. Another thing for us that was quite a meaningful turn in our evolution was honestly putting more useful features behind our subscription. So I started with saying, okay, we had ads and offline. We started adding more, we added two features that were basically transformational to the subscription. One of them was Doolingo had this pacing mechanic called hearts. Now it's called energy, but hearts was not integrated with our subscription at all. The way it works is you can go through Doolingo and do as much as you please, but if you make a certain number of mistakes, you start running out of hearts. And then before you can do new content, you have to regenerate hearts and you can do that by doing practice lessons or you can do that by watching ads or at the time we said you can renew your hearts with an in-app purchase. We decided to integrate the hearts mechanic with our subscription and said, hey, if you subscribe, you actually get unlimited hearts. That was a big turning point because it turns out a lot of users don't want to practice old content. They just want to do new content. Even though practice is good for your learning, we said, okay, if you would like, you can choose your optionality if you subscribe. That was a good turning point because it made the feature set of plus Doolingo plus at the time way more integrated and way more useful. Another feature we created was again around practice, custom practice. So we created a feature that is a separate tab called practice hub. We made that part of super at the time. Actually, it's now became fully free. We now gave it all to free users because we realized it should be free. So we did that. But at the time we said, if you would like to do very custom practice on the things you've learned on Doolingo, here's an extra place where that's a serve to you as an inventory. So you can go back and practice anything you'd like in whatever order you'd like rather than the order we prescribe. Anyway, this is a very long-winded way to say feature improvements that expanded the feature set of Doolingo plus was naturally a win on two fronts, both conversion and retention of the subscription, which is great. We love that. And that gave us a lot of scale. Other than that, honestly, it is rare to have put like one silver bullet that improves things. For us, a lot of this has been continuous A/B testing improvement. I think we talked about the copy version as like one thing, but imagine that happening across every bitten piece of our purchase flow. And I think that's why Doolingo's purchase flows get copied a lot because I think people have heard us say this, I say this a lot, which is like, we did A/B testing many things. So if you want to copy something, probably you're copying something that has been tested at least for Doolingo, it probably won't work for most apps, but you can look at us because we've tested stuff. That's fascinating. Before we move on, I did want to talk one more angle of monetization. I know by the time this podcast airs, you'll have taken something that was previously paid in moving it into the free tier Y. Why do that? Let me talk about actually two features that we, one of them, we just did, and we are about to do. So practice what I talked about. We have realized that number one as a goal, we would like to provide more value to our free users. There's the missionary reason why we want to teach more people. But there's the business reason which is free users are our marketing engine. Doolingo grows as an entire user base and as a business through a word of mouth. Word of mouth is much stronger if your free product is stronger because you might or might not pay irrelevant. If you're using Doolingo, you tell your friends about Doolingo and say, this is awesome. I'm learning Spanish. They try it. They do the same. So that's how Doolingo really grows. The majority of our growth engine and that growth engine, of course, gets stronger if we can provide more value to free users. So it's a delicate balance if you're a free product to decide how much value to put on each side, right? And we certainly want to put value as much as possible to our free user base. So that's the over arching reason on why. But the two features that we did was practice sub and the second one is explain my answer. These are really, I mean, I love these features because they are really additive to your Doolingo experience. So practice sub we talked about. I'm not going to repeat it. That is now actually already live with free users and explain my answer is moving from being a Doolingo max feature, which is our highest tier to free users because we have realized the revenue value is not that high. So we want to meaning it doesn't make that much revenue. How did you determine that? We tested it. We tested giving it for free and compared it to a B condition or a control condition where it is part of max. The impact what there was impact what we decided to kind of reduce our revenue, but rather provide value for our free users because we believe that's the right thing to do in the long term. And there's one more reason, which is when we first introduced this feature, it was financially impossible for us to make it free because it uses LLM calls to explain basically what it does for the record is you give an answer, you answered something, and we basically use LLM to describe to you what you got wrong for your specific answer. So it's very like tailored, that's why there's an LLM that is used. LLM calls for really high. So we had to put a part of a subscription just to provide the service, but costs have gone down significantly and we've improved how we cash things. So since costs came down, our only decision was revenue loss and revenue loss we decided to take to provide this, honest, this service for yeah, it seems like that's a real theme is being very strategic about what's free and what's paid because of that balance because you want to create that amazing free experience, but you still monetize and so that tension must be difficult as a product team. Yes. And you know, honestly, the hardest part is for me, it's really easy. I think for like CEO, it's very easy, we always want to do the right thing for the user. It's not hard for us to think about, okay, like how do we improve the free experience? There's some obvious things. The hardest part is taking a loss in an experiment is not intuitive. So the teams that are working on these are finding it hard to be like, I understand all this good stuff you said, but we're going to drop our metric. It's like, it doesn't feel intuitive. So I think convincing the organ, convincing the teams and explaining why this is the right thing to do a long term is the hard part because you can imagine the positive side of what I'm saying doesn't really have a metric, but the cost of it has a very real metric. So I think that like they run an Indian hand, but I think you have to realize, especially if you're a freemium business, is if you only always go by positive revenue metrics, you will burn yourself to the ground. around. I don't want to call out specific companies, but you can see a lot of subscription businesses that have effectively nickel and dime their users if they're freemium and then burn their growth engine even though their revenue kept growing. But that only works for some amount of years, but eventually your growth stops. I think for us what's most important is our growth engine, our user growth engine keeps growing. That's why you have to make these hard calls. Sometimes we're going to take a loss. It's okay because we're helping our growth engine is the convincing part of the hard part. How do those calls get made when an AB test shows a benefit? Imagine this happens in more than just these monetization features. How do you balance the kind of art of product management against the science of product management when you run 400 simultaneous AB tests? This is a great question. I think this happens a lot for us, I would say, because we have this other side of our equation as a business. We talked about user growth and monetization. That's the same for every app, but Dueling Grows is one more unique element which is learning and learning efficacy. We run AB tests to try to improve our learning efficacy. The problem with that is it doesn't have a single metric you can put on a graph. Even if you could, I mean, we use proxy metrics. There's no perfect metric for it, but even if you have proxy metrics that are going up, let's say your learning goes up, but your revenue comes down. Should you exchange that? Should you not exchange that? There's these trade-offs you have to make, and I think that's where you move away from mostly purely science to, like you said, the art of product management. How do these decisions get made? If they are really consequential, it's like, all right, the revenue loss might be large or there's a DAU hit. Whatever it is, that the clear business hit, we have to bring in most senior people to decide if as a company we should take this trade-off. That means our CEO would be certainly involved in saying, okay, committed, let's do this. If it's at smaller scale, I think most of our teams understand these trade-offs fairly well. Meaning, if it's an exchange of, for example, clearly the thing we're doing helps you improve your conversation skills, but in turn, maybe it makes our content a little harder. So maybe it hurts some of our business metrics, but if the amount is small, teams can independently make those calls. If the magnitude is large, but it's a strategic decision we should make, then I think we bring in senior people to sign off. It's kind of how we operate. This is why I do a link that's such a well-loved app. I mean, hearing you explain that and understanding how many product decisions have been made over the years to provide that better experience at the expense of DAU or monetization or other things. It's why people love it and that's like you said, it's hard to put a metric on that love and social sharing and the feeling I use for kids when they open the app, but it sounds like that's something you protect very carefully as a company, which then pays dividends over the much longer term, even if you can't always AB test your way to that. That's very well said. We have this operating principle that we I love and we repeat a lot is take the long view. Meaning you can make business decisions very differently if your goal is to be a very profitable or a very fast growing company for two years versus if you want to build a company for that will be around in a hundred years. I think there's like very different decisions and to be specific, you could monetize the dual-ingo feature set very heavily if you want it. And for two years, you'll make a lot of money. But one, that's not our goal and two, that would be terrible for free users. But three, we will burn ourselves to the ground if we do that. We cannot build a hundred year business by taking more and more from that's what a lot of premium apps fall into this premium trap where like the easiest win you can always do is like, oh yeah, let me take some free. If you just make it paid and immediately all your metrics are green and that does kind of work for six months or maybe a year, but eventually it stops working. The other thing it does is it creates opportunity for someone else to do it for free. The dual-ingo story, the very early one that we didn't really talk about is why dual-ingo grew so much early on. At the time, there were tools teaching languages. There were the Rosetta stones of the world, but they were paid and expensive. So being free and fun, created a massive defenciator, if we were to tomorrow wake up and say, all right, we're going to charge for all of our great features. Very likely in two years, another great app will do it for free. So I think preserving the free value is just a moat that we have to keep and we want to keep for missionary reasons. But for business reasons, it also makes a lot of sense. Yeah, that's such a great answer. I did want to move on to the new products that dual-ingo has been introducing and becoming a multi-product company. For a lot of apps, it's a big leap. The most famous example is calm finding sleep stories and that secondary product market fit being this new lever of growth that really moved the needle for them. So tell me about why you decided to start adding new courses and we can talk through how that went and what you added and everything. Definitely, it's honestly, it's been a really fun journey. It's been now, it's been four years since we introduced our new subject, maybe three to four. But now we have math, music and chest on top of languages. The reason we chose those specific subjects, they're all varied and they have their own thesis. But at its core, they have a few properties that carry across. One is they could be taught on mobile in a fun way. Not every topic qualifies. For example, coding, we considered, but it's really hard to teach on mobile. So we decided not to invest there. That doesn't mean we won't ever do it, but today it felt too hard to do. Another one is they all have large markets of people that are already learning these subjects. So we're not trying to create the market necessarily, but we are, there's already existing demand. That's two. And three, and I think this is connected to mobile, our expertise in language, which is at its core, if you think about it, a bite-sized fun experience, can carry you over fairly well to these subjects. The anti-example of this would be stuff that is done offline, like running could have something we decided to do, but that is an offline experience that you tie with an online experience. That is not exactly how we're used to doing bite-sized exercises. So we wanted to make sure our competencies played well in these subjects. So these topics really well, we actually looked around quite a bit before we invested in each of them. The maybe the fourth internal reason these topics is each of them had one or more people inside the company that were quite tenures, like tenured builders that really wanted to build these. So they became kind of the seed for these projects. So I did want to get into the staging of this because you first launched math as a separate app, right? So what was the thesis behind doing that? And then what didn't work out that you now brought it back into the app? Yes, our thesis changed actually over time. So when we launched math, we assumed our future would be separate apps. Like our strategy, because at the time, we also actually had, we still do, but we had another app called Duolingo ABC that is meant to teach literacy to younger children. So teaching their first language, rather than second language that Duolingo teaches. But our strategy was headed in a direction where all these new topics would be separate apps. We thought at the time, this is partly true still, but our thinking certainly evolved, that different audiences would seek different apps, rather than be in one mega app for multiple topic learning. So if you want to learn math, we thought the overlap between you and the language learning would be slim. And that's why you would probably want a separate app for doing one job. And I don't know if this was common with them at the time, but it felt really good as a strategy. We have, we did realize one thing as time went on, which is actually two realizations happened. One turns out a lot of people want to learn multiple things at once. At least Duolingo learners want to learn multiple things at once. So it became a inconvenience actually to switch between apps. And for two apps, maybe you could survive, but if it's three, four apps, you're just not going to do it. And there's this is inherent human demand that we realize exist within Duolingo, which isn't just like I'm here to learn Spanish. There's certainly a lot of users that are there for one thing, but also a lot of users are there for self-improvement in general. So it was a natural behavior that we just saw happening, which is I'll learn a little bit of math. I'll learn a little bit of chess. I am there to spend my good screen time on Duolingo to improve myself. It is less relevant exactly what that topic is that might change over time. As long as I'm improving myself in something, that lend like kind of let us towards well, let's make it easy for these folks to do that. So that was one. The other one is just the business realities of growing a separate app is extremely expensive while we were like, why are we playing hard mode when we can play easy mode because in the same app, you have a user base. It's way easier to get a user that is in the main app to try something new, then get them to download an entirely new app. It's so crazy to hear that from you. I mean, of all the companies with 10 to millions of users and beloved and everything else that the separate app was hard mode. It's hard mode for sure. And I think a lot of companies have come to this conclusion. I think there is successful and unsuccessful examples of both of these actually, but the famous examples are I think Uber and Facebook have done both separate and combined apps in multiple forms. And that's why I don't think there's a right answer for every company, but for us, if you think about user behavior, which is generally, there's a lot of users that are around self improvement and let me learn something and that will change over time, it just made sense for them to be in one app. How did you solve the product problem? I think one of the challenges of mobile is that you have such tiny real estate to communicate everything to shoehorn a feature into this tiny space that's already crazy. brought it by other things. How did you solve the product problem of it? - A lot of work is the short answer. And before we launched anything, these projects all took probably from somewhere from six to 12 months to go from the first idea to full launch. So that means in the meantime, we probably designed, tested, prototyped, redesigned, prototyped, tested again on ourselves and some beta users maybe many, many times until we got to the state where it's like, okay, within the confines of this mobile experience, this feels fun enough. It's actually teaching something useful and it's good enough to go in the app and be on par with the quality of where the language courses are because language courses have been improving for 13 years whereas this new thing is starting from scratch. So anyway, a lot of dog footing and prototyping, I think is answer number one. But the other one is, I think you can eventually teach a lot of things on mobile. There's some courses that are easy, there's some courses that are harder. Music is really hard, I would say. Chess is fairly easier because you have one visual element that is the chess board. As long as you fit that into the mobile experience, you can build a lot of stuff on it, but really it's now contained. Music is way harder because we're, for example, teaching you piano skills on a screen and we have to think a lot more creatively on how to do that. And you can imagine the amount of like stuff we've tried and still are trying to improve, which is you can't fit an entire keyboard on a piano, for example, and you also can't fit it when the phone is vertical, you have to turn it horizontal. To fit more key space, we actually tilted the keyboard a little a few degrees, so it gives you a bit more angle for touch space. So all that stuff is like innovation that you end up having to figure out. And I would say your music is the hardest. Chess and math are easier because math follows a very similar visual style to in exercises language. - How do you surface it to users in the app in a way that doesn't get in the way of the people who just want to learn language or are just there for music, or the crowding the screen real estate with these multiple things? How do you tell that problem? - We've made it fairly integrated so it doesn't feel like you are crossing into a different app, but it feels like you are just switching what you're learning. So we had already built a system basically to switch courses. So if you were learning Spanish, you could switch to learn French or Italian, any other language. In that system, we now also offer with some different hierarchy of course, unlike if you could switch to French, which is the second language you learn the most or there's your other topics now, which are math, music and chess. We integrated there and honestly, it's worked fairly well because we do already have a lot of users that learn multiple languages per day and they started discovering it. We also did some light promos just to raise awareness on the topics. That said, this remains a bit of an unsolved problem if I'm being honest, which is a lot of our users have no idea that we also teach math, music, and chess. So next year, we're still gonna try to figure out how to make sure that at least our user base that's coming to the app every day is aware that these topics exist. - Yeah, it's a tough problem. How do you think about competition with Duolingo? Do you think of YouTube and Netflix and other screen time as competition or who competes with Duolingo? - I think any screen time is probably our realistic competition. If you think about the natural idea of competition, you would probably think, well, education, any education app, but realistically, one, we are already way bigger than any education app. So at least user-based-wise. So we don't particularly see behavior of, oh, I stopped using Duolingo. Now I'm using an education app. Most of any turn or any user that we lose goes to spend time on Instagram or YouTube or anything that is fun in their phone. So for us, I think real competition is good screen time or I guess good or bad, I think that's subjective. But anything that is spent in screen time is Duolingo's competition. And that's inherently why we spend so much time on making Duolingo fun. It is also, I would say, it's headwinds to try to take something that is brain intensive. Learning a language is not an easy activity. It's not as passive as scrolling on your feet. It's actually like burning your brain a little bit if you do it for a long time. And we try to make it as easy as fun because otherwise, well, you'll go to Instagram. You'll go to TikTok. There's many engaging things that are just pulling you. So that's how, if I'm being honest, we don't really think about competition because it's so weird to think about screen time. But if you were to define it, it's really screen time. - I think that's honestly the case for most apps these days unless it's a specific utility where you need to get in, get out, solve a very specific problem. If it's any kind of, even habit forming out, I mean, almost any app, when you launch your home screen, are you pulled to Instagram or are you pulled to do a link or to do your lessons? And it's tough. How do you think of AI as competition? And will AI just subsume everything? Nobody learns a language 'cause they've got their AirPods automatically translating. I imagine that's a hot topic right now. - Well, I get this question a lot. And I actually really love it because I think we have a pretty clear thesis with two parts of this question. So one part is, will AI remove the need for learning languages? And in that case, we'll do allingo, well, we'll do allingo do if that happens. The other one is I think will AI disrupt do allingo because AI has language teaching capabilities. I'll start with the first one. I think the reality is being able to teach a language is not particularly a new part of a new ability for a tool to have. What I mean by that is textbooks also can teach you a language. Now AI can, which is for a technology perspective, it's incredibly impressive, it can do that. But texts, it is not enough, unfortunately, to remove the need to learn languages. People we've learned, and I'm only talking about languages and we can talk about other subjects, that the reason people learn languages incredibly varied. So you might be learning to connect with your heritage. AI doesn't really solve that at the end. You still will want to learn languages. This is similar to when Calculators came out. We didn't stop teaching math. It's quite similar. And the need to learn languages does not disappear, even if you have massively good simultaneous translation. And AI can do that really well. So that's, I think, we don't see language learning demand going anywhere, and we certainly don't see it post-Elevens either. The other question is there are now other tools that are non-dualingo, like the chat GPs of the world, that can actually teach you language. If you ask it to teach me Spanish, it will do something. And I think that's very fair. But I think that my textbook analogy still applies there, where it can teach you a language, but is it exciting? Is it fun enough? And until that is the case, a thing being able to teach you a language is not a threat to our business. I think we don't see that much, and we continue investing in making things fun for that reason. And maybe the third thing I will add is, we see AI only as an accelerant to dualingo, because we use AI to do mainly two things that make dualingo better. One, produce content at scale, at higher quality and higher volume. The other one is introduce features that we couldn't do before that solve some of the teaching problems that we really care about solving like conversation practice. So in a way, AI has helped us a lot, and I think it will continue to. And we will continue building better products with AI. So we're happy in everything that's happening. Well, let's move on to a few of your contrarian takes. So don't build for a persona, build for the general population. Why? I mean, there's classic PM thing. Build a persona, figure out who you're talking to, all that kind of stuff. Yeah, this is any PM fear Merck would say, don't build for a general audience, always pick who you're building for. I think when building consumer products, at least this is certainly dualingo, I don't want to generalize too much, but we are like very anti-persona. So if someone comes in from the outstart world, joins our team, most of the time their instinct is like, oh yeah, I'm going to build some persona maps. I'm going to call this person the wander, the other one the explorer, and which one am I going to build for? And we're immediately like, please don't do that, like stop. Because I think if you want, it really depends on what kind of product you want to build. If you're trying to get to a billion users, which we have kind of the ambition to, you have to eventually build for a general population. That's like, I think, my theory number one. Theory number two is with language learning, if you pick a persona, you're actually really cutting off the rest of your audience. And language learning is such a long tail of reasons for learning. Someone could be learning to get a job. Someone to be learning to move countries. Someone could be learning to connect with their heritage. These are extremely varied. So if you pick one and say, oh, this is really important. Yeah, like the common PM framework would say, that's how you succeed. I think for us, that's how you fail. Because then you've ignored basically the entire long tail of why people are learning languages, and you've built for a very specific one. So if the use case of a product is varied, which a lot of consumer products have that property actually, you have to figure out how to build for the general population. That's great. Next one, people don't read. Okay, this is another favorite one. This kind of connects to something I hinted at. Most early career, or honestly, even people who join, do all you go with experience, product designers, or PMs fall into this trap of what they write on the screen as copy and text. Think is read by the user. The reality is on mobile apps. People's attention span is already shrunk. It's like you might be like cooking with one hand and doing your app with the other hand. You might be on the toilet. There's all kinds of situations where now your attention span is like shrunk. And the other reason is, You are kind of going through an experience by not kind of paying attention to every detail but the core, maybe like the 30% of a screen. So we always repeat this. Of course, it's an exaggeration. Of course, people will read some things on mobile. But really in a graphical interface, they read very little. But to simplify my extreme hot take is people don't read, so don't rely on text, rely on great design to guide the user. And maybe the better said version of this is, if you need to write a tutorial on what to do, you've already lost it. So it needs to be designed that leads the user, not the text and the labels that you wrote. All right. Well, let's close out with the three questions I ask every guest. What was the most impactful experiment change or something you implemented this year? Your biggest win. It's not exactly this year. It's been two years, but this is worth calling is the introduction of video call on Duolingo. I've been using Duolingo to improve my Spanish for nine years now. And this is the one feature I can confidently say. It's helped me speak Spanish way more than before. And I mean, we're very excited for what it can do. It still feels very early, but already if you use it, you can just speak Spanish better. And I feel great about that. That's such a great example of building a truly great product experience with AI. A lot of people are slapping it on and trying to put a chat bot in their app where it doesn't make sense. At the video call, it's such a great example of taking this new technology, actually creating a great customer experience. That's cool that you brought that up as such a win. What about the biggest fail? Biggest fail is this is not, I don't think this went very public, but actually what a lot of our community doesn't remember or don't know is before video call, our first attempt at like a big speaking feature was a human based tutoring product. So it was a higher tier subscription because it was quite expensive for us to logistically provide. But you would press a button. I think we called it, we call it tutors, but it was a higher tier subscription, but you would press a button. You would connect with a Spanish tutor on demand, no booking required. Honestly, it was pretty awesome. Like for speaking practice, it was really awesome. It had this one massive problem, which is no one wanted to do it. It was really scary. With dualing it, imagine this super fun experience and we would tell you, hey, you are going to connect with a human, you're going to practice Spanish, people to press it. They would dial in. We would see their faces come online and they were like, oh shit, get me out of here because one, it's really scary to speak in Spanish like on the language that you're learning. And two, there's another real human on the other side that was extremely jarring. We iterated a lot to ease you into it, but eventually we gave up because humans just did not want to do that. And actually, this brings me to another hot take that I do want to mention. Do not take user feedback at face value. For years, this is what people asked us to build. Like we're like, please build me a system where I can talk to a human tutor. We built it. No one used it. Such a great example of a great feature, but just it doesn't work. As product people, we still get into our heads, take feedback and build something that's truly great, but it doesn't work. That's such a great one. All right. Last question. Growth would be easier if I think growth would be easier if two things were true. One is pixel perfect design didn't matter in consumer because I think consumer app design bar has become so high that if you want to make progress and lead product led growth, you have to get pixel perfect design that will go out to users. It can't be just a wireframe looking button that you put in your app or your illustration is like not really cared for. It, this increases the time to ship products. So that's one pixel perfect design does matter and it growth would be way easier. Maybe this was the case when apps first came out, but it matters and it makes a difference. And I think two, if AI really contributed to productivity in both prototyping and coding. And what I mean by that is I think we are seeing today in the ranges of 10 to 20% productivity improvements. That's really impressive. But like when LLMs first came out, my hope was that they would be kind of 10X Improvers. And I think if you've got the Silicon Valley hype today, that's what the messaging is. But really I'm looking at what we're doing and what every company is doing. Really the speed up of the product pipeline is in the 10 to 20% range. Growth would be easier if we got what was promised. And there was 200% improvement, but I don't think that's the case. All right. Well, as we wrap up, anything else you wanted to share any jobs you wanted to shout out or anything? I will definitely do that. We are currently hiring for, honestly, we're always hiring. But currently we're hiring for quite a bit of product manager positions. I think the audience of this show probably is interested in monetization since we also talked about it. And we're also hiring for monetization product managers. They're all in our creator's web page. Nice. I think that would be a great job for this audience, so if you're really into subclub, you're probably really into monetization, so apply. All right, Jamie. It was so much fun talking to you today. Thank you so much for joining me. Thank you so much for having me. This is awesome. Thanks so much for listening. If you have a minute, please leave a review in your favorite podcast player. You can also stop by chat.subclub.com to join our private community. (upbeat music)