Why Duolingo Worked (with Luis von Ahn, CEO)

Hello acquired listeners. We have an excellent ACQ 2 episode for you today. We are here with Luis Von On, the co-founder and CEO of Duolingo. Luis was a math professor at Carnegie Mellon in Pittsburgh from 2005 to 2012. He previously founded Recapsha, which I'm sure many of you have filled out hundreds or thousands of times that sold to Google in 2009. He is Guatemalan and immigrated to the United States studying mathematics at Duke and later earning his PhD in computer science from Carnegie Mellon. Welcome to the show Luis. Thank you. Thank you for having me. We're very excited to do this together. I think Duolingo is a completely fascinating company. And rather than me saying, Duolingo is a, why don't we ask you, what is Duolingo? Well, we're most well known for an app that teaches languages. It is the most popular waiter learned languages in the world. It is the most popular education app in the world. We have about 100 million monthly active users. We're also known for unhinged yet wholesome green owl that does weird stuff on TikTok. It's the mascot of the company. You know, that's basically it. We're doing what we can to teach people languages. It started because we wanted to teach people English. It turns out English is transformational. In the US, I guess two of you are not English in the US. You really should learn English, but in non-English speaking countries, English is pretty transformational. People who learn English can just make a lot more money. So we started with this very mission driven idea that we could teach people English all over the world. But if we were going to teach English, we'll teach all the languages. We started teaching all the languages. And at this point, we're the most popular waiter learned languages in the world. It's amazing. So my first question that I've been listening to you on other podcasts and doing research on the company and preparing for this, why has Duolingo worked to such an extreme scale? You're a $9 billion publicly traded company. Why has it worked when so many companies in the language category in the past have been these small, fragmented sort of sub-scale companies? There's a number of reasons. But if you ask me for one single reason is that we understood something early on, the inside is the hardest thing about learning something by yourself is stay motivated by a margin. And so what we've done with Duolingo is we really have made it so that you want to learn. And I think that's what has made us very popular. I mean, most education apps that try to teach you something concentrate mostly on the learning outcomes. They try to actually teach you the thing. And that makes sense. However, that's not the hard part. The reality is you can learn anything with a book. You can learn quantum physics. You can do so with a book. It's just nobody's doing it because it requires reading a quantum physics book. But if we can turn it into something and people actually want to do, that can be really powerful. And I think that's the biggest insight. I mean, there's other things that we've done. We became really good at product. We have a really good design. There's a lot of things. But ultimately, it's just that we understand that we need to get people to want to do it. All right. So what are some of your key levers in making people want to engage? We started doing a link, which was like 2011 or so. We started working on it. We were building a program that taught you languages. And it was with my co-founder. My co-founder is a native Swiss German speaker. And I'm a native Spanish speaker. And what we decided we were going to do is I was going to make the first Spanish course. Even though I don't know how to teach Spanish, but I do know Spanish. And he was going to make the first German course. That was really stupid. I thought German and Swiss German were the same language. It turns out they're not. I thought they were like British English and American English. They're not. It's like a different language. Anyways, he does happen to also know German, like actual hawk German. So anyways, he made the German course. I made the Spanish course. The goal was that we were going to learn each other's language. But we ran into this problem that we would go home. And then the next day we would come to the office at 9 a.m. and I would ask him, you know, did you do your Spanish lesson? And he would say, no, man, so boring. And I had the same problem. It's because when we started, we were making these lessons that were like 30 minutes long, first of all. And there was no gamification, et cetera. And we couldn't even get ourselves to do it. So this is when we started really trying to add all the mechanics to get people to do it. Was this in any way connected to your research? Or was this a complete side project? It was. Well, it kind of was. It was my PhD student. That was a professor. And we were trying to find a PhD thesis topic for him. That's what we were doing. We didn't even know this was going to be a company. I mean, we kind of in the back of our mind thought, maybe this could turn into a company, but we weren't super sure. That was the idea. And it was kind of connected to my research, but ultimately was just try to teach you something with a computer. Your research was around crowdsourcing, right? My research was around crowdsourcing. Yes. And this was just more about trying to teach you something with a computer. So we were doing that. But this is when we learned that we needed to make it more fun. And so we started doing a number of things. I mean, the first thing is 30 minute lessons. No good. You want to do three minute lessons. So that's the first thing with it. We made it in three minutes. That's very important. And even if ultimately people do spend 30 minutes on your app, making each chunk short matters a lot because that means you can do them while you're waiting for the bus. Oh, while you're literally going to the bathroom, 30 minutes, it's just too long. So you have to be able to chunk them. So we started chunking them. That's important. So we made the lesson experience. Be something that was quite palatable. Three minutes, there's a progress bar that you get to fill. And then we started adding all kinds of little dopamine hits. Like when you get something right, there's like a little explosion. That's kind of one thing. Then over time, we discovered a number of other really important levers. One super important lever has been the streak. So we have a streak. We've had a streak for like 12 years, basically. And the idea is, you know, the standard with a streak that if you use it for seven days in a row, you have a streak of length seven. And if you don't use it on the eighth day, it goes down to zero. So it's just a counter of how many days you've done it in a row. This is super powerful. At this point, we have, I think I'm going to get this number right. I think we have eight million people who are daily active users who have a streak longer than 365. So we have eight million daily active users that haven't missed the day in the last year or more. So more than 365. This is very powerful. Strict is very powerful. That's around 8% of your users. 8% of our active users have not missed the day in at least a year. That's very powerful. It really is. Another thing that has been real powerful is notifications. You know, many apps abuse notifications where they just notify you like crazy. It turns out that if you're teaching something to somebody, there are a lot more okay with you sending them notifications because people think, well, this is good for me. So we've taken advantage of that. While we're not spamming, we do send notifications probably at a higher rate than we could if we were not teaching you something. We have a very sophisticated system that tries to figure out, it's an AI system that we've trained over the years that basically tries to figure out when to send you a notification and what to say in it to get you to come back that has been really successful. And then the other thing that has been pretty successful that we've gotten just a lot better over the last few years is a social component where we have leaderboards. Leaderboards, by the way, there's a whole science of how to do them well. We got them very wrong at the beginning, but we're much better now. So, you know, we have leaderboards. You can have streaks with friends now. You can have quests with friends, et cetera, so that has helped out a lot. And we should say this notion of gamification. I mean, this was like a real hot word 20, call it 12 to 15. Yeah. And you started the company in 2011. So you were kind of blazing the trail on a lot of these things, the idea that you could bring game mechanics to non-game experiences. It's pretty interesting reflecting back on that time, how that was the buzz word in every VC pitch that we were seeing that you really did build a public company on that concept. Yeah. For some stuff, gamification is not all that useful, particularly things that you anyways have to do. So, for example, a smaller part of our business for Duelinggo, we have this thing called the Duelinggo English Test, which is about 10% of our business. It's a standardised English test, like tofu. It's like tofu. It's a competitor to tofu. It's a test that people have to take to get into a university, for example. That thing, you can gamify it, you can not gamify it. It doesn't matter. The reality is people have to take it and they'll go through it. For something like education, I think gamification ends up really mattering a lot because it's all about motivation actually keeping you there. So I think this was just a very good use of gamification in the end. Do you have a sense of if you've expanded the market for people learning a language who wouldn't have sought to learn one otherwise? Massively. It depends on the country, but in the United States, for example, 80% of our uses were not learning a language before Duelinggo. You're a typical American. It was not learning a language. Yes, but they are now with Duelinggo. It's a similar in the UK. It's a similar number in the UK. There are other countries, particularly countries where people are learning English. There, we probably haven't expanded the market as much, but in English speaking countries, like the US, UK, Australia, we've expanded the market by a lot. Did you encounter when you were fundraising a lot of pushback that, "Hey, this market isn't that big?" I imagine it's probably hard to sell the thesis. Our product is so good that we will massively expand the market. Yeah, we did. Our first fundraise was 2011. Union Square Ventures. At the time, Union Square Ventures was very hot. They had just funded Twitter. They had just funded. It was like a big deal. Anyways, we went there and we were so fortunate that they funded us. Brad Burnham was our partner there. He got into our board. We became good friends. It's been a many years later. He told me, "Ah, we didn't think you were going to succeed at that. We just liked you." We thought you were going to pivot into something else and maybe you'll do something, but we actually didn't think you were going to succeed because we didn't have a product. That was the first round. That was the Series A. By the time we reached the Series B and we raised that kind of the resource book for themselves. We weren't making any money, but we had a lot of users, especially if we had only launched like maybe two years before that or something. We're ready. really already had millions of users. So it became a lot easier when we had, I don't know how much product market fit was, but at least we had some amount of product market fit. - I'm curious what your thought process and discussions were as you were starting the company, but even before then, and you're looking for a PhD thesis, were you always planning to be mobile first, mobile native? 'Cause it was right around that generation when it switched from like, you're exactly right. So in 2011, almost no company was doing mobile first. Everybody was like website first because most of the traffic was on the web, like a big laptop or even a desktop. When we started, we thought this was gonna be a website, but relatively early on, really, maybe in the first few months, we realized almost everything is going mobile, and we had this insight, which was, we're very fortunate that we had this insight that at the time most companies, they were website first and they had a companion app. Like banks, you could go to the main website, but in the companion app, you could check your balance, but you couldn't do anything else or whatever. It's just like a companion app. We had this insight that, you know, we could actually put all the functionality of the website in the app. And so it was not a companion app or anything, and as soon as we first launched the iPhone app, it overtook the website traffic within weeks, we had more traffic on our iPhone app than on the web app. And so right then, is when we decided we were gonna be mobile first, and at this point, my god, I don't know, it's 95% of our traffic is on mobile and 5% is on web probably. - We're y'all watching Instagram, as this was happening. - Yes. - 'Cause the analogies are so close here. We were, and by the way, we were very fortunate with Instagram, and I also think we gained a lot from this because, you know, we were not the first language learning app. There were others, but I think we were the first good one, and it's because before us, what people were doing with apps is they were basically making websites and making them tiny. It's like a tiny little website, but they just kind of shrunk everything into a mobile screen. And when we came out with a mobile app, by the time we came out, we were watching things like Instagram. There was another app called Path. - Oh, yeah. - That was a girl. - You were saying that was amazing. - Yes, they more. - So we were watching all these apps, and we were like, this is how you design an app. Not make a website and shrink it. You actually have to change all these things. So I think we benefited a lot from the thinking of, let's make an actual native mobile thing, as opposed to a miniature version of a website. We were very fortunate with that. - I'm obsessed with this misperception of market size question. I'm sure you get pitched a lot of ideas now. Many people probably come to you with the idea, hey, just like Duolingo, this doesn't seem like a big market, but actually we're gonna build a big business in it. What's a good heuristic for when that is possible versus not? - I don't know what a good heuristic really is, but I can tell you this. I think language learning was massively underappreciated in the United States because in the US, people don't have to learn a language. The reality is if you already speak English, English is the monster language in the world. That's the big one. I grew up in a country where you had to learn English if you wanted to be mildly successful. This was true in my case. Had I not learned English, I wouldn't have come to the US, and I would have definitely not succeeded the way that I did. And my co-founder was also not born in an English speaking country. And I think when you grow up in a country that is not English speaking country, I think you realize, yeah, you may think that the market is not huge, but pretty much everybody is either learning a language in particular English or wants to learn English. And everybody pretty much understands, so at some point you just realize, well, boy, billions of people want to learn English. So it just has to be a big market. There has to be a big need. So I think that was the case for language learning. No, I don't know for the other ones. What we do inside Duolingo when we're looking at opportunities is we definitely look at just how many people are either doing something every day or close to every day. So this is why we decided when we're trying to expand to other subjects, like we're doing math and music. We went for those because those are things that literally billions of people are either doing or want to do. And we didn't go for something like, you could go for chemistry or something. Love chemistry is just, there's probably not a billion people learning chemistry or wanting to learn chemistry. What's your early signal on Duolingo for X? You mentioned math, there's music. Did you start with the very best use case for the type of product you build or do you feel like, oh, actually, that was a local maxima. There may be even better products that are more well suited. There's a lot of companies out there. I mean, we get the ads all the time on my Instagram or something, Twitter or whatever. That says, the Duolingo for whatever. But there's the curious one. The Duolingo for anger. And I don't know if they mean that you, like, it teaches how to get angry or not. But anyways, I get a lot of ads. It's the Duolingo for anger. Anyways, my sense is that there are probably other things that are pretty good, but it's hard to compete with language learning. I think math and music are going to be big. I don't know if they're going to be as big as language learning, but they're going to be big. But the reason I think it's hard to compete with language learning is that not only are there a lot of people wanting to learn English in particular, but also language learning has a few things that are good. One is that it's one of the few things that people want to learn both in school and outside of school. So most things, people just kind of have to learn like in school, math is an example. So not many people want to learn math outside of school. But language learning, if you ask the average person walking on the street, "Yo, would you want to get better at Spanish?" Most people are like, "Yeah, why not?" So I think that's a big thing. Another big thing that happens, this is in contrast with music. Music has one big problem, which is you need an instrument. And that's a big deal. I mean, unfortunately an instrument is $2, $300. With language learning, you don't need anything. It's just that. There's a lot of things that I think work really well for language learning. I don't know if it's the best use case, but it's a pretty good one. - Yeah. So thinking about the language learning business, you mentioned earlier, English is the number one language that people want to learn. So most of your usage, please correct me if I'm wrong, is outside the US, people wanting to learn English. But most of your monetization happens in the US. I was wondering if you could just tell us a little bit about sort of how that evolved and how you think about it. - Yeah, if you look at the numbers, about 45% of our active users are learning English. English alone accounts for 45% of our users. And that's mostly outside of the US. In terms of users in the US, so about 20% of our users, our active users are in the US, and the other 80% are outside. The terms of revenue, behalf comes from the US. And the reason for that is because, you know, the US is pretty large and wealthy country. But the other thing that's important to understand is people in the US are very accustomed to paying for digital subscriptions. But what we're finding is that in certain countries, the US is ahead of most everybody. In certain countries, people are accustomed to digital subscriptions in others, people are not, or they're really against them. And these are countries where even a Spotify or a Netflix are just not as penetrated as they would be in a place like the US. So I think that's a big thing. We just have a lot more propensity to pay in the US. And wealthier countries, the other thing that happens with us is, you know, we have a really good free version. In fact, if you look at our monthly active users, only about 9% to pay. The other 91% do not pay us. And it's because our free version is so good and what we're finding is that in poor countries, even if you bring the price down a lot, people have this mentality that is that they won't pay unless they have to. So even if it's like 50 cents, like super cheap, unless they have to, they won't pay. Whereas in the US, people are a lot more willing to pay for convenience for things like, "Well, you know what? I don't want to watch ads, so whatever, I'll pay." We're not seeing that type of behavior in a country like Vietnam or Brazil or something like that. I mean, some people pay in those countries, but it's just as much less than the US. And you do have an ad supported tier, right? Yeah, it's the free tier. It has ads at the end of a lesson. But the ad load is pretty light. I mean, basically you get one skippable ad at the end of a lesson. So five seconds of ad after about three to four minutes of thing, it's pretty light. Yeah. And the other thing that I think is important as listeners sort of understand this part of your business, when did your product launch and when did you charge the first dollar of revenue? Oh yeah. We launched in 2012. But when we launched, we made no money. We didn't have ads and you didn't pay us. I mean, simply, we made no money because there was just no way for us to make money. We just didn't put it there. And that was true up until about 2017. We started thinking about monetizing in 2016. What happened was we had just raised funding from, at the time it was called Google Capital, but it's now called Capital G. We had just raised funding from them. And I don't remember the valuation. It's about a half a billion dollars valuation, but with no revenue whatsoever. And the partner from there, Layla's 30, flew to Pittsburgh where we were. And she got us drunk and she said, you got to start making money. Like tomorrow. And of course, as a founder, you clearly knew the secret that the only thing better than making a lot of money is making no money. Everything in between is not good. Yeah, exactly. No, that is the secret. And so went to the office the next day. And it was pretty interesting. Because for the first several years, we made no money. And we weren't even really trying to make money. And everybody that we were hiring was very mission driven. We were hiring all these people that loved the idea of giving free education. So when I came to the office and I told everybody, hey, we have to figure out how to make money. People were pretty shocked. And they were like, why? But I'd said, well, you can't have to pay your salaries. And they said, you've been paying our salaries this whole time. Yeah, right. Has it been a problem so far? Why can't you just continue doing whatever the hell you you've been doing and I had to explain you can't raise funding forever. Here we're going to do a Duelingo session on late-stage capitalism. It took about six months to convince the employees of the company that it was, in fact, not evil to make money. Eventually, people got convinced, and I think the main way in which people got convinced is I think everybody understood that. If we made money, we could invest more in the mission. I think that everybody was like, "Oh, okay, so you mean we can still put most people to work on actually making a better product as opposed to making money?" I said, "Sure, sure, that's what we're going to do." Today's day, it remains true. The fraction of people that are working on making money at the company is relatively small compared to the rest. It took about six months to convince people that we should do that. The other thing that was complex is at the time we weren't sure how to make money in a way that respected the fact that we wanted to still be the case that most people could learn for free. You know, at the time Spotify was kind of growing, etc., but it was not super clear that this freemium model worked. This freemium model, which is like, "You can use it for free, but you may have to see some ads, and if you don't want to see ads, you have to pay." It wasn't clear that we could make that work. It turns out you can make it work. It worked really well. The fact that that worked was really good for us because it's a very mission-aligned. Basically, you can learn entirely for free. You can learn without ever having to pay us, but if you want to turn off the ads, you can pay us. That's a very mission-aligned for us. But we're fortunate that that worked. It really was not clear back then. It is so funny. That's just such a basic assumption now, but even 2017-2018, YouTube, you couldn't pay them. You could only watch ads. Actually, David, I think Google contributor did exist. I don't even know what that was. It was the predecessor to YouTube freemium, and it worked across all their properties where you could basically pay the amount that Google would make on you from all the ads that you would see across all the ads. Wait, did it strip out ad words? There's no way you would do that. It might have been except for ad words. I don't know. I should get my facts straight on this. I was a paying Google contributor, a customer. It says a lot that it was called contributor. That's funny. In any case, for us, it was not super clear that this was going to work. Was it strategic at all? Was there a network effect component? Is it good to not put any inhibitors on your growth rate because more users begets more users or something like that? I am a huge believer that the reason that we have been able to grow so much is because we didn't monetize early on. Now, I cannot recommend that to every single company. I mean, for some companies, you probably should monetize early on. But in our particular case, what happened was this. We launched. Because we weren't making any money, we also couldn't spend money on marketing. We couldn't do performance marketing because performance marketing is this thing where you can pay Google 30 bucks. And then they send you a user. But to me, that seemed really dumb to spend 30 bucks to get a user when we were not going to make any money off of that user. You want to be making more than $30 from that user. Yes, or whatever it is you want to just do more than that. And so we couldn't spend money on marketing. We weren't making it any money. And we had a bunch of engineers. So what were we going to work on? The only thing we decided to work on was teaching better and increasing user retention. That's it because that's the only things we could work on. So we spent from 2012 to 2017, basically optimizing retention rate for users. And we optimized it a lot. I mean, for example, we went day one retention. So that's the retention of people when they show up what fraction of them come back the second day. When we first launched that number was 13% as in you lose 87% of your users the day that they well, they may come back another day. 87% happened to not come back the day right after it could be that they come back later. But that's what we saw. We ran who knows how many AB tests over this period of time. But by 2017 that number was probably about 50%. And maybe this remembering exactly the numbers, but we were able to drive all of our retention numbers by a lot because that's the only thing we could work on. Had we not done that we probably would have spent all our time optimizing our ad pipeline and optimizing all kinds of weird stuff that ends up making you money. But I don't think helps you grow that much. And I think we were in retrospect. This was a very smart move. I'm at the time. I don't think we were thinking how smart this was. We just kind of did it. It gave you a durable asset. I mean the fact that you have a product that ensures that you have fully maximized the potential for a user to stick around. It means the user base you've built and the machine by which you eventually do make money is far more durable. That's right. And I believe that now of course I believe all of this in retrospect at the time. None of this. We were just like, I don't know, make a thing that people like always also weren't you are raising money at a half a billion dollar valuation. So this was when money was free in some sense. It's just like here. Have more money, but it worked. Okay. So you joke that you weren't doing any marketing in some sense. That's true. In some sense it's not. You have the green owl. I have had a great brand forever. How did a couple PhDs from Carnegie Mellon of all places come up with a cutly green owl and build a. I don't think of CMU as wonderful as an institution as it is as like the top designed and marketing place in the world. They happen to have a really good design program, but look this evolved over time. We did know one thing. We did want a mascot early on. We were like, look a mascot. Somehow this will help us. We didn't know exactly how, but we did want a mascot. Eventually we chose an owl because of in Western cultures, at least a symbol of knowledge. They ended up being green because my co-founder hates the color green and I thought it would be funny if it was green. So that's why it was green. And now your whole office look behind you. The first version of the owl was not very cutly or I mean it was kind of not great looking honestly. Over time it evolved. We hired some amazing illustrators. We hired this guy who still work for us. Greg Hartman who's in our head of art now. He just made an amazingly beautiful owl over time that evolved. And so that's one thing. We just made a really good asset that looks good. And then the other thing that happened is it started getting a personality. But the personality was a little bit accidental. The first way which it gets the personalities. The owl was actually not in the app at all at first. The only place where people saw it was the app icon, which is kind of a very close up version of the owl's face. And then also when you get a notification, it has the app icon so it looks like it comes from the owl. So the first place where the personality for the owl came up was basically what we said in the notifications. And we started saying some weird stuff in the notifications. And I think people started giving it a little bit of personality because of that. Then another thing happened with the notifications. At some point we decided that we can't be super spammy. So we're going to stop sending notifications after five days of inactivity. So if you don't use the app for five days, we'll stop. Then it occurred to us, well, we're going to stop sending a notification. We may as well tell you. Right. Get the credit for it. Yeah. So people know we're good guys. This is what we thought. So that last notification on the fifth day, we wrote it. It's just said, these notifications don't seem to be working. We're going to stop sending them for now. That's what it said. Well, we didn't know this. We didn't expect to see turns out this got people to come back a lot. Because they thought that Duelingo was giving up on them. It's like your mother. It's kind of like guilt tripping you. Because of that though, it started giving rise to all kinds of memes on the internet about how the owl was obsessed with getting you to come back. And he would kill you if you didn't come back. But that was the community. We didn't do that. After that, we realized that we could push on those memes. And so we started making fun of ourselves with those memes. And that has worked really well. And at some point, we got the idea of getting an owl suit. The suit that somebody could wear. At first, we thought it was good. It was going to be good for recruiting events, like going to try to hire people at universities and the owl suit would show up. Turned out, you could start putting somebody in an owl suit to do weird stuff in TikTok videos. And that gets a lot of people to watch it. And so, but that was a little bit accidentally discovered. And in fact, the first person who started doing this was a very young woman, Zarya, who when she proposed that I thought this is such a dumb idea, I was obviously completely wrong. It turned out this has given us a lot of prominence in the world. Have you ever tried to quantify? I mean, this is like a core pillar of dual and go business strategy at this point. Have you ever tried to quantify the business impact of having all this earned media? Yeah, I mean, we have our belief is that about 15% of our users come from that. On any given day, roughly 15% of our new users come from the owl for lack of a better word. Some sort of owl thing. The majority of them come from people telling their friends about it, but about 15% come from this weird stuff. And just to multiply some other numbers together that you've talked about, if it really is 15% and you have 100 million users, what's the value of 15 million users? Well, if it's actually at 30 bucks a pop, would be the paid cost to acquire them. That's what $450 million. Well, you're only taking the active ones. So it's $30 on Google to get a user period a user that may not stick around. So it's more than that. This is probably when I don't know the exact number, but it's hundreds of millions of dollars of fun. It's the most valuable Alan history. I don't know that's true, but it's been really good for us. It's fascinating. All right, listeners. Now is a great time to thank a new friend of the show that we are very excited about. Sierra. Yes, we are thrilled to be working with Brett, Clay, and the entire team over there. So why are we excited about Sierra? Well, one of the things that we've learned from making acquired over the years is that a great company is often defined by its customer experience. Yep, but being great is hard. Talking to customers is expensive, and while websites and apps are great, they're also kind of slow and clunky, and your customers have to learn them. They don't learn you. Sierra changes all that. They build customer-facing AI agents that can do an insane range of things like finding the perfect home or picking TV shows or originating mortgages, shipping us so far, returning shoes, authenticating patients for healthcare, ordering credit cards, saving subscribers from cancelling and on and on. In just two years since founding, they've become the leading conversational AI platform with hundreds of incredible companies like ADT, Clear, minted, ramp redfin, rocket mortgage, SafeLite, SiriusXM, and Wayfair, all trusting Sierra for their customer experiences. Sierra was built to be powerful enough for fortune 500 companies, including heavily regulated industries like healthcare and financial services, but it really works great for any business, including yours. With Sierra, you can build your AI agent once and deploy it everywhere within weeks on the phone, and chat, SMS, WhatsApp, email, all in over 30 languages. You can even publish it to chat GPT. And with their unique and insanely aligned outcomes-based pricing model, you only pay for the value that Sierra delivers, increase customer satisfaction and resolution rates, lower costs, and higher revenue. Sierra enables the great companies of the world to show up at their best consistently every minute of every day, and in fact, we think so highly of Sierra that day, then, I even invested in the company. To find out how you can build better, more human customer experiences with AI, visit Sierra.ai/acquired, and tell them that Ben and David sent you. Okay, I'm going to keep coming back to my, like, what's a heuristic type of questions. Startup comes to you with the idea that they're going to run a similar strategy. Any advice for when it could work or not, or is this pure lightning in a bottle, I have no idea when it will work? Some stuff, I think, you can reuse. There's a few pillars in our strategy. One thing that we haven't talked about that I think accounts for probably half the value of the company is our obsession with AB testing. We have AB tested our way to where we are now, and we have probably run, I'm going to make this up, probably about 2000 AB tests per year, of all kinds of things. And so we had just built the company around our entire product organization. That's not just product managers, engineers, designers. The whole system is built around shipping AB tests as fast as possible. That has worked really well for us, and I would say, you should do that everywhere. If you have a consumer product, I mean, you shouldn't be AB testing stuff. If you have like a B2B, Sasting, whatever, do something different. But for a consumer product that has a lot of users, you can really AB test your way to improve almost any metric you can. That you should do. The other thing that I would give advice to is hiring really good people that you should continue doing. In terms of the specific strategy of having a premium thing that you don't monetize for a while, I'll tell you one of the reasons why that may not work anymore. We have found ourselves, but we have seen it all over the place. We were fortunate that we put out an app in 2012. The app market was not yet set in stone. It's pretty close to set in stone these days. It's not 100%, but if you look at what are the top 50 apps globally for the year this year versus last year, and you compare, they're pretty similar. Maybe one or two changed, but they're pretty similar. It's similar to the year before and similar to the year before. At this point, it's pretty hard to enter, and I don't think that the strategy that we employed in 2012 would work today because the app market is pretty set in stone. It's just kind of is. What about the strategy of something clever earning you hundreds of millions of dollars of sort of free earned media? It's great if you can do it. There's of course a lot of cleverness in this that many employees at Duelinggo did, but I think we were lucky in a few ways. I think the fact that we're an education company gave us license to do some stuff that many of the companies may not have been able to do because it's kind of hard to say that language learning is bad for you. That means that we can do risk-aestuff on TikTok, but if a company that is trying to do like, I don't know, like some sort of fintech company or something, I think it's much easier to accuse them of stuff. I think that helped with us that just come on, everybody, we're the good guys. I think we were able to get away with a lot of stuff because it's hard to say bad things about language learning. Right. If I'm coming at you with you could argue it's a fair interest rate or you could argue it's a predatory interest rate. If I have a mascot that's sitting there with a knife saying, "Use our app more. It's not going to go well." Exactly. I think we benefited a lot from the fact that ultimately we're doing it for education so people are okay to take some of this. Right. Fascinating. By the way, you know, another place when we talk to other companies, every company wants to talk to us, not because they learn from us, but because they're like, "Oh, you guys, you guys are kind of fun. This is good." Nobody thinks of us as competitive. Well, that's great for us. And it's because almost every time we talk to whoever, fancy person at random companies, my daughter loves you. You know, I think we've benefited a lot from that. Which is great. How many other 10 billion dollar companies could that be said for? If you're a 10 billion dollar company, almost you'd think like de facto you're going to have people who are your mortal enemies. Again, because everybody's like, "Ah, green owl." Yeah. Yeah. So it's an interesting comment on competitors. At this point, you probably don't have a whole lot of meetings about legacy language learning companies and saying, "How can we acquire users versus them? Who do you think about as competitors for your users?" I mean, the way we see it is we compete for time. And so our competitors, our Instagram, TikTok, etc. That's we compete for time. And by the way, we're losing. We have much less time spent on our app than TikTok has or Instagram, etc. But when we look at our users and we see that our users maybe stop using the world or something, we ask them, "You know, why'd you stop?" The most common answer is like, "Yeah, you know, what I'm spending more time on, pick your random social media site. I'm spending more time on Instagram." That's it. The competitors, whatever takes their time. All right. Well, you planted a seed of something that we talked about for like one minute in this podcast. But every person I talked to, who's ever interacted with Uolingo, former employees and other podcasts you've been on is all about experimentation, AB tests, being data driven. There has been lots of inks build on this for anyone who wants to read excellent blog posts about the, you know, the AI behind your push notifications and how all those systems work. One high level question for you, how do you as a leader balance a cultural tenant being a data and experimentation driven versus instinct driven? We're actually weirdly both. I'll tell you where this comes from. We realized early on that pure experimentation can lead you to really bad places. If you kind of just don't think about it at all, you're just going to optimize some metric. I can lead you to really bad places. One of the best stories that I heard about that, and we repeat this a lot of Doolingo is something that we learned from the founder of Groupon, who happens to be from Pittsburgh. I mean, we're located in Pittsburgh. He happens to be from Pittsburgh. Yes. We're huge Descript users. Yeah. Yeah. It's a great guy. He told us the story, which I'm going to tell it. At this point, it probably has changed a little bit because I probably remember now what is in the law of Doolingo as opposed to exactly what story he told us. But he told us a story about emails that Groupon set per day. Groupon, basically, you've got a deal and the idea at the beginning, they could only send one email per day with one deal. He was strict on it. It's like only send one email per day because more than one email is spamming. At some point, some enterprising product manager shows up and says, what about if we send two and Ramesh and says, no way. No, we're only going to send one. Then this product manager gave this amazing argument against it. We're not trying to launch two. We just want to know, are you against knowledge? And of course, nobody's against knowledge. So he said, I'm using this. Fine. Fine. Fine. And so they test the two. Guess what happened when they test the two? Two is way better than one. So they start sending two. And then like, wow, if two is better, test three turns out three is better than two and four is better than three, et cetera. And they A B tested their way to like, I don't know what number seven eight or something until the channel stopped working because people marked it a spam and then it died. The channel died. But so this is the thing about A B testing area. Like you have to, you have to really do the common sense thing above the A B test. And so what we do at Duelingo is we have, we have this two step process. I mean, the first one is before you even A B test something. And by the way, here we are against knowledge. There's certain things we just don't want to know. And it's because it's too tempting to do it. So the first step is, before we do an A B test, we have this process called product review, which I assume happens in many other companies. But in this case, every single change that goes to the app gets product review. And product review is basically a set of leaders in the company that includes me. There's about five people and we look at every change and we have to agree for it to go out. We stop a lot of things before going out to prevent this type of stuff. Here we use intuition and just common sense. We're like, no, nobody wants to receive 32,000 notifications. That's just dumb. Even if the A/B test says so. I think that has helped us quite a bit. Do you have an example of a way in which you've been anti-knowledge in the past, something you decided you didn't want to know? Oh, yeah. I mean, pretty recently, somebody wanted to know what happens if you put an ad that takes over the whole screen when somebody opens the app. Like a product manager wanted to know that. And we were like, yep, but we don't. Particularly with monetization, it's so tempting because what happens with monetization is you run the experiment. And then it says, it's just something like, if you launch this experiment, it will make you $50 million a year. And you look at it and you're like, that's nice. And then the finance team figures out that this happened. Like somebody from the finance team looks at this experiment. Suddenly, the CFO is in my conference room or something. And he's like, yo, 50 million bucks a year, you got to launch that. We try not to know the answer to many of these monetization experiments. When you wear your CEO or your head of product hat, these things are sort of obvious. And what's underneath it is this implicit assumption that there is a future dollar value associated with treating the user with respect and having a feel good association with your product. It's basically like all of the potential future dollars we could earn are jeopardized by doing this. It's we're pulling forward revenue. But it's very hard to quantify, but it's very hard to quantify complicated expected value calculation on it. Yeah. Have you ever tried to quantify that brand value or future value to we haven't know we haven't what we just have is a group of I'm saying five, but it's on the order of five leaders that have a very good gut feeling about no, this is this too much or no, this is not good. And I think their gut feeling is quite good. I would say we run into this problem. Sometimes, you know, we do run an experiment. And where we have to make a trade off like, for example, we're in some experiment because we know if we do this, we lose 10,000 users, but make this much more money or whatever. We've never known how to make these these tradeoffs. So generally what we do is we go for users as in if something decreases the number of active users that we have, unless it's going to make us 500 million dollars, we're really not going to launch it. So just take a laugh. I need a bit to be out using that line on me. This applies to any business like for us, if you told Ben and me, okay, go objective function maximize the value of acquired, we could come up in 10 minutes with a whole list of things that we know exactly what we would do. And it would make an obscene amount of welcome to this daily episode of acquired. Yes, it would, but for the next few months, and not for the next few years. I'm not even sure that that like that might be the value maximizing thing to do, but we would never do it because it would just kill our hearts, you know, like, yeah, that that also matters a lot. Yeah, it's funny though, your heart, you can boil a frog quite well. And product managers at Duelinggo have gotten really good at boiling my heart slowly. Things that I think I was not okay with five years ago, I think I'm a lot more okay with because they just kind of push the envelope every slippery slopes, they get you. But do you end up feeling like, okay, you know what, they had a point. Like, I no longer. Yeah, sometimes I've definitely been wrong about stuff. I mean, that's for sure. But I still claim in the long term, it really is better to do what's best for the user. And I don't mean what the user's asking for because the users ask for weird shit. I mean, what you think is better for the user. I think I think that in the long term, I'm pretty sure that's that's value maximizing. I think this is a pretty specific thing to founder led companies, which is we will do what I personally feel in my gut is best for the users. And like that, just carrying the day every time, that does not happen in a professional management setting. It may not what I'm happy about is that I would say five years ago, it was solely my gut. And I think at this point, there's a group of leaders, Jim or had a product, semi-archief design officer that who's gut somehow is like identical to mine at this point. Maybe one percent different, but I feel pretty good about the fact that if I were hit by a bus, these these people's guts is pretty good. So we've talked a little bit about this idea that you have a bottom of funnel governor on what goes out. And we've talked about that there's a mid funnel thing that's the AB tests. And I think you've said in the past, the latest number is 50/50. 50/50, okay. It is eerily close to 50/50. As in like, it really is 50/50. It's not 48/52. I can't explain one of those where I'm just like, really 50/50 come on people. It has to be something different, but now it's pretty close to 50/50. So that gets to this question, why is it that over time, your collective gut has gotten better about what experiments will succeed? The way it has gotten better is because we get the results of the experiments. Our gut is kind of a learning machine. We're learning, we're like, yeah, that type of stuff. At this point, we have years of experience that people that have been doing product review have been doing it for years now. We have years of experience knowing this type of change does that type of thing. And we're pretty good at it. We're not 100% but we're pretty good at it. Not only do you know which direction things are going to go, but you also know which ones have more impacts of, for example. We know that the session ends screen. So that is the screen that you get after doing a lesson. Matters a lot in terms of getting you to come back the next day. But we know that other screens, like a screen in the middle of a lesson almost matters not at all in terms of getting you to come back the next day. So we've just kind of learned what what screens matter and what parts of the app matter for certain metrics. We just have a really good gut on that. Do you have to treat big experiments differently than little experiments where your gut may not have as much training data in it? Yeah. And there's some experiments that are a lot more complicated than others. Some stuff takes us a long time. Well, first of all, now we also have pretty sophisticated data science that tells us for how long to run certain experiments. We also have a pretty good gut feeling about this particular experiment should run for a pretty long time. Most experiments were run by the way we're running for two weeks. We have enough data in two weeks to know where to go. But there are some experiments that we run for about six months because we know that you're going to see a very big difference at the beginning than later because people get used to the thing. There's a very heavy statistics component to the core competency of your company. Where did that come from? I mean, the first many employees were basically computer nurse from Carnegie Mellon. I think that's where that came from. We understood things like statistical significance coming in. Yeah, it makes sense. Does it feel related to your crowdsourcing research or does it feel like you brought to what on who was a super key hire and then made that a pillar of the company? I don't know if it's exactly my crowdsourcing research, but it's generally the first few people that worked at the company. We had a pretty good grasp of statistics. What I will tell you though is that it's funny. This is true for most of the things and I don't think this is unique to do a lingo for most things. That's good. But the level of sophistication that we do analysis today compared to that is insane. I mean, we now have people with PhDs and statistics who actually honest to God know what they're doing. As opposed to we had quote unquote a pretty good grasp of statistics. Early on when we were figuring out how to teach things, we kind of had an idea of how to teach things. By now, we've hired dozens of people with PhDs in second language acquisition and the level of sophistication that they have in terms of learning outcomes and stuff like that compared to what we were doing at the beginning is night and day. So we had some expertise for a lot of things, but today we just we've become a lot more sophisticated about most things including the statistical analysis of our experiments. Knowing what I know today, I'm pretty sure that experiments from eight years ago, many of them we analyze wrong. What do we know? We're like, yeah, we look at averages, we look at statistical significance. We were not looking at the stuff that people look at today. In fact, I don't even understand what they're doing. So just very smart people come and tell me do this. I'm like, sounds good. So all this to me and I bet to many listeners sounds like the perfect TF friendly AI. You are a perfectly suited company to AI maybe just at the highest level. What's going on? What's changing? How is it impacting everything we just talked about? You are right. We are very well suited for AI. I mean, we've been the whole time. AI of course means something a little different today than it did 10 years ago, but AI has been around for a while. But since 10 years ago, AI meant everything we were just talking about. Yes, yes, statistics. Since we launched, we were very clear that what we wanted to do is we wanted to have a computer teach you a language, not another human teach you a language. Another human can teach you a language very well. That works. No problem. But having another human teacher language is expensive. We wanted to have a computer teach you a language. So because of that, we're very well suited. We need to make a computer that is as good as a human teacher. And so we've been working on doing that as much as we can. Over the last couple of years, what has changed is these large language models. And again, I emphasize that the second L is language. We are a language learning company. Large language models are particularly good with language. And because of that, we've been using them a lot. I mean, so I'll tell you the two big things where we where we're seeing large language models be used. One is it used to be the case that a lot of the data that we created for courses was half handmade. So what that I meant, we made tools, but But usually a lot of the stuff was humans doing things and then maybe the computers would fix it up or whatever, but it's kind of half-hand made. We've gotten to the point where most, it's not 100%, but most of the stuff that we're creating data for language learning is made by an AI. There's a couple of good things about that. First of all, it saves us money. It's a lot cheaper than doing an humans. But the most interesting thing that has happened is we can now create data so much faster. And it's not that we can go fast. That's nice. That allows you to do things that you weren't going to do before. So for example, about five years ago, somebody pitched a feature on Duolingo, which was a type of lesson where it's kind of like a mini, call it like a two-minute podcast where you basically learn how to listen in the language that you're learning. It looks like a little animated thing. We call it dual radio now. But it was like a thing where you could watch two minutes of something and listen to it and you're learning how to listen. And I was told that creating the data was going to take five years. And as soon as I was told that, I was like, no, no. This feature, red light here, do not proceed. I do not want anybody working on five years of things. Something that I don't even know is going to work. A very similar feature now, we can create the data for this in a couple months. And so now, I'm like, yeah, sure. Why not? What is it about today that causes that incredible thing? Well, all of that data is done by a computer as opposed to having it done by hand. I mean, for it to make a difference in the dual-ingle language learning app, we needed to make hours of content, like many, many hours of content. And specifically in this case, you're talking about animated graphics. Not just animated graphics, the audio of it. Like actually, you know, what you're going to say, you have to make it interesting. So you have to make something that people had to listen that was interesting, et cetera. So you have to do all that. It was super slow to make hours of content. And it was going to take five years to make content. Not only that, you have to make it in all the languages that we teach, we teach 40 languages. And we have to make hours of content. So that was going to take an inordinate amount of time. But now, you know, with an LLM, you can do it pretty fast. And what we do is we do it. Then we kind of spot check it. And if we don't like it, do it again. We're very happy with that that you can basically do things that before we're just, it were not impossible to do, but practically impossible to like, why would you do that? So that's an important place where we say, I, the other big one, is just conversational practice. We're working on this feature that some of our users already have. We haven't fully really launched it to everybody. But it is basically, but you can have a conversation with one of the dual and go animated characters. And it looks like you're having a video call with an animated character. And it's a really good way to practice a language and actual conversation. That's something we just couldn't do before. Yeah, I've heard Mark Zuckerberg say about LLM use cases a few times. Or maybe it's about meta AI use cases that it's for people to practice conversations that they don't want to have yet in real life, but they need to sort of feel like they're having the conversation and get some authentic responses. Well, so that in the case of language learning, what's amazing about it is, look, you want to learn a language you have to be able to practice conversation. You just have to do that. And people don't want to do it. In fact, we've done insane amounts of user studies where people, they may tell you they want to practice a language with another person, but when it comes down to it, they don't want to do it. There's some small fraction of extreme extroverts who like it. But the majority of people would just rather not talk to another person in a language that they're not very good at. But the beautiful thing here is that you can just talk to an animated character and you really don't care because you don't think they're judging you. Even though technically they're really judging you, because that's mainly what the thing is doing, but you just kind of don't think about that. Well, hey, that's just absolutely incredible. And AI data also opens up the other direction for you too. You have an incredible amount of data within Duolingo that can be used and fed back into LLM's your own, I would assume, and others that would just be incredibly, incredibly valuable. Does that open up a whole other new business dimension for you all? Maybe at the moment, that's not something we're doing. We're using the data for ourselves. More than one billion exercises are solved by our users every single day. So we have the data, and basically we're watching, I don't know, tens of millions of people learn a language every single day. We're using it to teach better. Today we know that Duolingo is about as good as a classroom in terms of learning a language, but not as good as a good human tutor, like a one-on-one, good human tutor. The problem is those are like $50 an hour, like a good one. We're not as good as that. It's hard to predict the future, but I'm assuming that if you give us three years, we'll be about as good as a human tutor. Well, and it requires making an hour. I mean, if you're doing three minutes at a time, that's a structural advantage where you're just able to address people that will never, ever make the hour. Yes. And you can't have a paid human tutor that is there for three minutes and then another three minutes later. And then this. They'd have to be sitting in the stall next to you to. Just waiting around. Well, it's cool. I mean, it's funny. You say three years. I wonder if it's fast because you've got two tracks of progress happening that are accelerating. You've got the foundation models of LLMs themselves, which are getting better. And then you've got your data, which you're getting, you know, what did you say, 10 million solved exercises a day or two? No, a billion per day. Sorry, a billion per day, right? So the quality of both of those are accelerating. So I'm really bad at predicting the future. How long things are going to take? Well, humans are and you're a human. So. Yeah, just this very hard. It's a matter of a small number of years to get as good as a human tutor. You're in this education space. Are there any education companies that you look up to as shiny examples of people who built great companies or do you mostly look outside your sector for inspiration? It's mostly outside. Education has been hard because you want to start an education company. You're into a lot of problems. First of all, for some reason most education companies have decided to monetize by charging the school systems, which is fine, but we just can't look up to those companies because we're so different. We're a consumer company. We've having the past look at those, but this is very little to learn from somebody who has spent a lot of time making a sales force for K through 12 schools. I'm glad they're doing it. Good for them. It may work. I don't know, but it's not much for us to learn. So ultimately for us, where we learn because we are a direct consumer, we learn more from games, companies like Spotify, Netflix, any of the meta apps. That's where we learn the most, but it's because we are a direct consumer. There's just not very many direct consumer scaled education companies. Well there are, but they're all nonprofits with brands like CMU or Harvard or Stanford. Yes, they are there, but they're not apps and they're not touching 100 million people. They're touching a few thousand people. But yes. It's kind of interesting to think about. They started as every institution starts as this niche thing using the latest and greatest technology available. Then over time you build a durable brand and you try not to jeopardize that brand in any way. And those words that I just said could apply to do a lingo and you're just like 75 years earlier on your journey than these other institutions. Is the goal to become a accreditation, bearing institution where people feel like when you've graduated from it, it means something? I would really like that. We are doing that for language. We're trying to do that for language. We have a standardized English test. We're doing it for English mainly because that's where it matters. Of this test called the dual in going English test. And the idea is that this is a test that you can go and take it. You can take it online and it gives you a score and it tells you how good your English is. And that is actually valuable. For example, it is now accepted by most universities in the US. If you're a foreign student that wants to come to the US, you have to take an English test. 98 out of the top 100 US universities according to US News and World Report, except our test. We haven't quite connected the test and the app, but we want to. For some languages, it's already there. We're giving you an estimated test score. So if you just use the app, we will be like, oh, my test score is this much. And then you can go and certify the score. You know, what I would love to do is to be able to own the score for languages. So right now, if you ask somebody how much French do you know, the answer is usually something to the effect of I took four years of high school French or intermediate, but they doesn't really tell you what we would love to get to the point is if when you ask somebody how much French they know, they'll say, oh, I'm a dual English 65. We're not there yet. Right now people tell you they're using dual English when they may tell you they're dual English streak, but that's something that we want to get to. And I think that'll be really valuable. You're right. I mean, maybe in 75 years, we'll just have a really stodgy stone buildings and a 30 billion dollar endowment. Well, it's funny. You took, I mean, like, what would the market cap of Stanford be if it were a publicly traded corporation? Like, it would be quite large. It would. And you know, we have this board member, Bing Gordon, who's who's amazing. And he'll happily tell you. He'll say market cap of a university is twice its endowment. Oh, wow. That feels like a lot multiple to me. I don't know where he pulled that from, but this is why he'll tell you. I don't know. I don't know where he pulled that from. So Stanford's probably, you know, I don't know what they're endowment is 30 billion or something like that. 40 billion to 50. I guess. Yeah. Okay. So it's a $80 billion company. This is such an interesting thought experiment because the mission of education is so unbelievably important, but it has historically, you know, last 20, 30 years, been so hard to invest in education technology companies because they so rarely are using modern business models. The technology that all the other consumer companies are using the distribution methods that all the other consumer companies are using. They're instead sort of trying to snap. to how do I sell into the education supply chain? And you just haven't. You might be the one example of this education technology company that said, I am not playing in that existing system. And that is how you invent something new. It varies on the day. But we're certainly the most valuable at tech company. We may be the most valuable education company. Some days Pearson is more valuable than us. Some days we're more valuable than them. But and what's interesting is we're doing it with just one app. And if you look at something like Pearson, they sell gazillion books and stuff like that. I mean, we feel pretty good about that. That we really are actually able to teach better. But you know, we feel pretty good about that. All right. I want to close with a question on headwinds and tailwinds. What's the most insane tailwind that just makes things easy for you running dual-lingo? And then other side of it, what is a headwind that you think you are just always going to have to deal with? And it's a thorn in your side. But that's all right. AI is certainly a big tailwind for us right now. I think there is a growing proportion of people who want to do useful things online. And I think we get tailwind from that. I think if you use tooling over 30 minutes, you don't feel the same as if you use Instagram for 30 minutes, where you're like, my God, I just lost 30 minutes of my life. I do think that's a real tailwind for us. In terms of headwinds, this is more from me as a person. Employee problems are tough. We hire the top of the talent pool. I'll use the word entitlement is there. That's tough for me. I grew up in a country where literally for two, three hours a day, there was no electricity. There are some people here who are like, why do we only have two flavors of coconut water? That's always hard. It's funny to connect that with something you were saying earlier. Part of why that exists is the XYZ elite educational institutions of these days and their brands and their brand caption one. But like if they were just a duo, Lingoscore of like, how good a product manager you are, that would level the playing field better. Yeah, exactly. We could get this people that weren't fed with a silver spoon as much. To go back to your comment of you grew up in an area where, there were multiple hours of the day where you didn't have electricity. I think you went to one of the sort of rare private English speaking schools to have the opportunity to learn English. How much of that formative experience for you contributed to you founding to Lingoscore? It's a lot. First of all, in a place like Guatemala, language learning is life changing. That was a big deal. But I think the other thing is I was fortunate. I received a rich person's education. Even though I didn't grow up rich, I was fortunate that my mother spent basically all her money on sending me to this fancy school. So I went to school with very rich kids. Yeah, it's not like we were poor. We were not. But we were kind of middle class in Guatemala, which probably for US standards is poor. But in Guatemala, we were middle class. I really saw the difference because where I lived in the neighborhood where I lived, the kids didn't go to the school that I went to. And I could see how much more I was learning. I mean, it was pretty clear that I was learning, I don't know, twice as much as they were. That was a big thing when starting Duolingo. We wanted to do something that would give and you know, access to education to everybody. So it's a big thing. Me growing up in Guatemala had a lot to do with starting Duolingo. Fascinating. Well, that's a great place to leave it. Luis, thank you so much for joining us. Ben David, thank you. Great questions. And listeners, we'll see you next time. We'll see you next time.