Season 7, Episode 35: OpenAI's advertising ambitions (with Asad Awan)

AI is changing everything, but it's only as good as the signals behind it. Branch connects customer interactions across paid, organic, offline, email, web, and app touchpoints, and turns them into the trusted context you need, with links and attribution that capture the full user journey. Learn more at branch.io. And while you're there, check out Branch's AI Search and Discovery Report, covering insights from more than 300 enterprise marketing, growth, and digital leaders to understand how the industry is responding to the rise of AI search. That's branch.io. The problem is that the distinction needs to be drawn between the components of the economists and the correctness of their analysis. Hello, and welcome to the mobile dev memo podcast. I'm your host, Eric Sufert, and I'm joined today by Assad Avon. Assad. Welcome to the podcast. Really glad to be here. Thanks for having me. Thank you for coming. I am unusually excited to have this conversation. I'm always excited to have these podcast conversations, but I am unusually excited to have this one because I have written extensively about OpenAI's ads business, and it'll be great to discuss that at length with you. Before we dive in, could you please introduce yourself to the audience? Awesome. So I am a solo one. I'm a monetization product and engineering at OpenAI. Before this, I was building a A16Z back startup on creators and creator monetization. And before that, I spent 10 years at Meta, where I started as an engineer working on ads, and then when I left, I was leading ads, SMB, creators, creator monetization, and the marketplace products. So a lot of the public content and business products over there on the Facebook, blue app. I think maybe the through line for a lot of the work I've done is like, how do we democratize access to economic opportunity? How do we democratize voice and make that available for everyone? So at OpenAI, as you know, and in fact, we started with a careful ads pilot, and we've been expanding the product as we learn. And I think in the intro, it's useful to start with the five principles that we started our ads product. So the goal for ads is how do we democratize access to AI, so ads solves that mission. I think like the next most important thing is user trust. So how can answers always be independent of ads? So the model that answers the user is not influenced by ads as they're always separate. And I think we'll talk a lot more about this today. And then, of course, the privacy of conversations, the private conversations remain private. And users have control over the data that is used. And of course, we have paid plans if you want to opt out of ads. And finally, the goal is how do we optimize for user value across the product? Empty carriages, not time spent, and we explicitly codified that into our principles as well. Since then, of course, we have grown at a fast cliff, hitting a billion ARR in less than 200 days since we started and now expanded globally to more than 40 countries. Right, thank you. That's how I think it's a good idea to set the table with the principles, just to remind people of those as the kind of operating mantra of the ads business. Maybe just before we dive into the business side, I just on a personal note, when this idea was pitched to you to come in and do this, I mean, this must have been like a daunting proposition, right? I mean, like Greenfield's opportunity, but there's a lot to build. Just talk to me about how you viewed that job opportunity, like what was going through your mind? And when you spoke to this with kind of, I guess, mentors trusted confidence, like, what was the kind of feedback? Because I imagine this seemed exciting, but also like a lot of work. Yeah, so that's folk a lot to Fiji and Vijay. And I think like every, every decade or so, I feel like you have a consumer product that changes everything and chat GPT, certainly that product for this decade. I think both in terms of the user experience, but in terms of the intelligence that we want to bring to everyone, like our mission is truly like AGI that benefits all of humanity, not few people, not people who pays, not like a some small segment, but everyone. So I think maybe it's a combo of kind of a new paradigm in consumer products, like in your lifetime, you don't get a chance to work on two of these. Of course, I was a part of that at Facebook as well. And then the chat GPT, I think, both jumping on to a great consumer product, which is growing. And then it is a product where I think people are making decisions. They're coming to fit their goals and certainly very, very suitable for an ads business to truly add delight and value, not just like, you know, a good business model. It grows with the consumers, but actually add delight and value to both users, build economy, shankton, like, you know, small businesses, I think all of that kind of came together and fits with my, like, that overall mission, like, how do we democratize access to economic opportunity for everyone? Okay. I'm going to kickstart the conversation. I usually like to instantiate the conversation with just a really big broad question. How do you conceptualize the opportunity for ads with chat GPT? Yeah. So I think it goes back to the mission is to ensure AGI benefits all of humanity. A big part of this is how do we bring broad access to the most powerful AI to consumers? And then on top of that, of course, people trust chat GPT to have them think clearly, make decisions, achieve their goals. And people are coming here to work through a decision often, even before they know the product that they need. So I think ads kind of fit with each one, each part of this, like, how do we take this to all consumers? And then if people are using this in a very specific native way, how do we build on top of that? So the opportunity for us is to build a business where we enable SMB's businesses of all sizes to reach people while protecting user trust. And I think if done well, it can truly enrich users by connecting them with services and products that they would love. And then for businesses to reach like specific niche users, I think like that's what a lot of the economy is built on. And then, of course, like, introducing an ad at the right moment for the user and help them make a next step. I think this is a much broader opportunity than matching, let's say, a keyword to a link. The native chat GPT experience is quite deep and ads very naturally fit into this. And I think, finally, this is like a global opportunity, both our product, consumer product is growing globally, a lot of businesses are growing globally. So I think it fits with that mission as well. Well, that's a good segue. So, you know, a point I made, I wrote a piece a couple weeks ago called the privileged position of chatbot ads. And you know, I kind of conceptualized chatbot ads as being this kind of like separate category of advertising, certainly placement, but I think even like fundamental call it to value opportunity. But the point I made in that piece, but I made it elsewhere, is that search queries is kind of this compressed expression of intent. You're not given a lot of space, right? You're not given a lot of surface area to express what that intent is, whereas a conversation can reveal preferences, constraints, my budget, you know, just personal trade offs I have with different product characteristics over a number of turns. So the chatbot can ask me follow-ups, like, and you know, provide more information. How does that richer context change what an advertising system can do beyond just improving the relevance of the added selects? Yeah, I think like, first of all, you know, like spot on, like the users, when they come to chat, they're not just like trying to do a transactional search, but it is like they bring their goals, they bring their constraints, they bring their preferences, and make decisions as they converse, and they iterate through that. I think like the one, one further insight before I answer the question is that users are engaged in multiple user journeys at a given time across conversations. And then the memories product that we have helped bridge those preferences and journeys across states. So as a result, you could continue a previous conversation and a completely new thread, or on a completely unseemingly unrelated topic, the preferences that are learned through your memories can make that experience richer. And I think like so in some sense, the user has multiple journeys going on the app, and it's not just even a single conversation. Of course, within a conversation, you see people iterate through the product, and I think the memories is a pretty big part of that, and that does enable us to serve really, really good ads. And of course, I think, as you said, first of all, this enables us to show ads that were truly delighted user, both, I think, intentful and prospecting, I think like that comes together, that we can help you discover a new product that you never thought about, or even deliver a product that you truly want. And then, of course, I'm the right sage of the journey. But to go beyond ad selection, which is your question, I think this goes to what a truly native ad experience for ChatGPT could look like. And I'll give three specific ideas just to make it concrete. But I think each one of them is a class of things that we can continue to improve on. So first is dynamic creative. So I think selecting the ad is the part of the job, but the key part is helping the user understand why we selected advice connected to what they need. So if I'm a nurse who's on their feet for long hours, and I'm showing you a new sneaker, I want to communicate that, you know, that this is the shoe is very comfortable. It has extra cushioning. And you could have created infinite number of creatives, or you could do that dynamically through an explainer text, or how you actually have the creative show up to the users, so that they connected to their problem. So this is even before, of course, the ad has been selected. Now we're communicating to the user why this matters to them in a way that actually fits. I think the second one is variant selection. You could say, again, it's a specific thing. But let's say if I'm looking at a shoe, what's the size of the shoe, what's the color of the shoe? There are so many things which, you know, in product feeds, we actually get that information and you can do much better in terms of that variant selection, not just the product selection. And I do think that's a very, very hard thing to do in many other platforms unless you have a very rich understanding of the user. But the final one, which is probably the most interesting one is like, you know, more powerful post click experiences. And I think like the one which we are very excited about is the sponsored agent. So a person can choose to enter a clearly labeled conversation with a business after an [BLANK_AUDIO] click and their original chat GPT conversation remains separate. And since sponsored agent is a native experience, we get the ability to guide further product or service, lead qualification or any other thing that can really truly help the user, but being very clear that you're engaging with the business. So I think maybe that of course can be built on top of the rich information that we have, but still in a very, very commercial context so that the user is very clear, who are the parties which are interacting. But I think like each of these I do think are three different classes of things which are like how do you express the ad better to the user after selecting it? Once you have selected that ad, is there something that you could do that the next step in this journey, which is like click to the website, let's say is better because you know through whatever is the API between us and the advertiser which is privacy safe in this case product feed and variance can help communicate a lot of those preferences in a very, very privacy safe way in some sense and then of course native experiences on both. Yeah, I think that's a good set up for next question, which is that the sponsored agent part of the answer is a good set up for next question, which is that you know there's this recurring argument that I've encountered and you know I've called it a persistent superstition, which is that chatbot advertising is non starter, it's non viable because it by definition undermined trust in the core products because the user wouldn't be able to believe that it's best interest or being served by the chatbot versus an advertiser. How do you navigate that? And my sense is sponsored agents is one way right because well we've made the handoff now, you're not interacting with us, the chatbot, you're interacting with the company that that showed the ad that you just clicked on. I think that's one way but just just walk me through that because I first of all I reject that premise but just walk me through how you think about that. Yeah absolutely, I think it's certainly like user trust is something that I think are coordinated product depends on like you know I'll take a photo of my kids' bruise and ask chat GPT a question about it and I need to trust the platform even more than like you know it doing the right thing, I think I need to trust that it will keep that information safe because I'm using it for many many use cases. So I think like from the get go from our principles user trust was a cool thing and then automatically as we did our internal round table we discussed with external partners, we came up with the principles, the five principles that I mentioned at the beginning and stated upfront which is like answer independence both in terms of the ads don't influence the organic model and then when the ads come out they are separated in the user experience that showed up in those principles and I think like what the implication of of course we published those principles that keeps us accountable I think nobody almost has ever published ad principles in the way that we did and I do think it drives more accountability but I think like what it does internally is even more important is like our internal rubric for any ad decision is user trust is more important than user value, user value is more important than advertiser value advertiser value is more important than a revenue like revenue is a side effect of us doing all the other things correctly but the long term value really depends on ensure user trust and then deliver user value and advertiser value which shows up like you know when we design an ads core metric user value advertiser value is a core part of that so this means like every day as a UI designer is making the decision or an animal engineer making the decision they are thinking about this rubric in the decision some of it is very deeply measurable and some of it is I think principles I think like one implication of this rubric and the principle was also that user trust is not something just a guardrail which is how I think it's it could be treated in many cases but we want to optimize for it and improve and sponsored agent I think product design is a great example I think you very very correctly identified that the clear context of where the user is and why is this but helping me purchase a product is very clear because you are interacting with that business so I think it gives that space to bring that intelligence that we have plus the information about the products and user preferences in a place where it's very much in the control of the user but also very clear that you are interacting with that business but I think it connects back to the design principle and then I think like hopefully we'll iterate through products which for the kind of built around like how do we increase user trust not just use it as a guardrail yeah I think so I am you know the high response to that what I've called that persistence to precision is just been that you know it's a design question I mean if you sort of clearly demarcate an ad I think you avoid that that mistrust and you know there's another piece to this which I haven't spent as much time articulating but so I kind of think there's a really interesting natural experiment here if you look at OpenAI's advertising initiative versus Meta's because there's this question now in the EU especially but it's like people saying like well you know when Meta offered subscriptions we can't really understand in the EU because they're forced to we can't really understand what the natural demand for that would be because you're taking away something free but had they offered subscriptions or ads as a choice at the outset people people this is an argument that I hear made more people would have chosen subscriptions because because there's just this this natural hostility towards ads among consumers and then what but we can just never know because Facebook was free and then they had to offer subscriptions and you can't just take away something free or impose a cost on something that was previously free and understand what the natural demand for that would be and now I say well but with OpenAI you can because we can make that comparison because it was subscription-based and there was the free tier of course at birth alongside the subscription but it didn't have ads and what you allowed people to do with the introduction of ads was to access more compute exactly so right out of the gate you got something in exchange for that and consumers I think understood that well hey wait a second in staying with the free tier that has ads which I can also turn them off I have the opportunity to do that I'm getting more I'm getting more I'm getting something in return that's just a very concrete value exchange and it's it's been a really interesting thing to witness because I do think it doesn't validate that that argument about well you know we don't people just just like ads people dislike ads and they would have chosen subscriptions out of the gate but it wasn't an option out of the gate and now they're not going to pay for something that they've previously got for free because you can just point to OpenAI absolutely I think that's a very good observation and certainly we believe that by adding ads you are creating a really clear value exchange like you know better rate limits and being able to use the product fully without any constraints that's our goal which is like again and it goes back to the mission as well as like yeah for everyone without constraints which basically means higher rate limits and you don't going to get blocked because you created three images today so I think that's very interesting and of course I think like you know there is always an opportunity to delight the user with very relevant stuff help them discover something I think like that's why people like some of these the ads which help them find something that they are never thought about and that's the bar that you have to kind of hit I think the question we might have to answer is like you know how can people who pay get access to ads because that sometimes comes up but yeah well yeah baby it's probably it's probably a good problem to have but I think it's also a benefit of the conversational interface right because some commercial intent is latent people might not know that they would like to have some product right they might not know that some product could add value to them in a conversation could surface that where someone is just ignorant of a product category right like if I don't know that this product exists right we get into rumsfeldian unknown unknown territory here but like you could surface that because you know the chat bot knows yeah absolutely so I think this goes back to like you know what the native user organic user experience looks like and that is very very different with conversations that we were discussing which is like people are continuing on these user journeys and they having this kind of relationship with the product where they're trying to make these decisions and goals and as I said also I think like memory is a key part of that which is like it connects the various parts of the user experience so already users know that when I say that like hey I'm going to get hiking shoes the fact that I was discussing that I have to go to Alaska it knows that I need something warm as well right it's not the first time that the system is learning about it and I think it's less transactional than just a keyword but I think even so first of all is understanding that these users are in these user journeys these user journeys are already impacting organic conversations because we are making connections that that the user might not have thought about even this like the fact that I have a five-year-old and then I'm going to book an event I need to know that the kid can go with me I think the system actually does that in the organic product already and memories makes that happen and then I think like same thing allows us to do the do exactly the same stuff for ads and I think I would actually break this down to get more concrete break this down into three different parts so one is within contextual ads we can do much better with personalization right so which is if the system knows that I'm into hiking then a conversation about planning a trip to your summit is not restricted to showing me a hotel over there or a flight to your summit which is what would usually happen because it's like but you could actually show hiking here and it would feel quite relevant and then that explainer text that I was saying which is like hey you show that above the ad this is like hey you're going to your summit and you might be hiking and then an ad for hiking so I think it makes contextually relevant you could say this is still contextual relevance but personalized based on a little bit longer history you could say it's one step removed in the contextual it's not like say your summit is equal to location is equal to hotel right which is how okay what system would work I think the second one is that there are clearly parts of our user experience which are very amenable to a different ad format which could be truly purely personalized and I think a good example is imagined is like Okay, I'm waiting for the image to get generated. That's a really good place for our strengths. And say if I showed you the best continuation of a user journey with an ad that you were talking about, like, again, let's say the hiking shoe, this could be a great place for me to show some more variance of that because in some sense, I'm not in the contextual mode anyway, I'm using the product like a tool not to answer a specific thing and I think like, of course, I imagine is one place where you can make it more visual and very interesting as well, but I do think this this applies to other places which are tools style usage or the product is like right an email for me. The contextual ad might be an email writing tool, but not really that's not very interesting for the user, you could actually say, hey, let's continue this journey. And then I think finally we do want contextual and personalized to compete in some sense is like a really good thing that was on top of my mind just because I'm doing something else right now. And even if this contextual conversation could have a really good ad, I think we need to make them compete and it but it does go back to we have to do a lot of form at work where the personalization connection can be communicated very clearly. So I think maybe that's the work that has to be done, but we truly believe that there is an opportunity to show the best ad to the user and not be constrained by this and the organic product. I think one thing, which I do want to add that like not every conversation is inventory, like you don't want to turn a personal topic into a commercial opportunity like you know my kids, Bruce photo, the ad has to fit in the right moment in the right journey, which is like if I booked a hotel, don't show me hotels again, we can do that much better now than ever before because we understand the user journey. So I think we are learning through that and the bar is like to through this feedback and careful experimentation, we can get to this. Right and you know going but just going back to the design principles also the data that you draw from some of it's just off limits and the context has to be appropriate exactly. You know those channels your colleagues keep bragging about the ones getting all the credit. Yeah, they might be doing squat attribution makes every channel look like a hero even when it's a zero incremental tells you who's actually doing the work. It's like a lie detector for your marketing budget start using incremental today get your demo at incremental dot com that's I and C R M N T L dot com mention that you came through the mobile dev memo podcast for a special 15% discount for the first six months. I wrote this piece the privileged position of chat about advertising and I mean it was a good you know is a popular piece. I got piece pushback though I got a piece pushback that I thought was actually it was valid it was awful. So let me just summarize the argument and I'll talk about the pushback so I argued that a chatbot can monetize conversations that contain no commercial intent. Just by showing ads that are relevant to the person rather than the topic that's what you just were sort of articulating. And the pushback I got and I said search can't do that searches can't do that it's keyword based right and the pushback that I got was that well it could do that search could do that Google could show an ad for a car or something next to a search for dog food they could do that they just don't because presumably it doesn't work. They tried it and it doesn't work let's just presume that they tried and it doesn't work my response that was well okay but it's it's a different environment right and also. The search use case predates display advertising there was no paradigm of an ad that is utterly unrelated to what I see on the page or in the product when Google introduced search it just that just wasn't a thing let's certainly met a wasn't the company that introduced it but let's say that met a. Popularized it feed ad the display feed ad that has nothing to do with what you're seeing or doesn't necessarily have anything to do with what you're seeing on the page in the feed in that moment totally utterly detached from the context let's say met a popularize that if they didn't introduce it and so. Google is just anchored to that paradigm for historical reasons but is there all is there another argument to me you know i've got the benefit of having you on the podcast now to talk about this with do conversational interfaces just also create more latitude for that kind of discovery advertising. Yeah I think it certainly does and as I said like you know because the user is engaging in these journeys and and the organic product itself is bridging the gaps between journeys and connecting the personalization piece I think it's it's very natural but like I think like a search experience is certainly I think there is two parts of it like how the traditional buying was done for a very long time which is. Keyword targeting because the relevance was determined by keeping targeting but also just the user experience is an navigational user experience which is which is like I asked for something and then I get the next link to click versus here the people are iterating through decisions and I think like composition happens to be the. To the container of that stuff I think they're iterating to that journey with the app and clearly memories show that that works really really well when we are able to connect pieces even when you actually like you know remind people of a conversation that was happening so I think it the organic product certainly gives us that latitude I think because users are not here to navigate they are here to kind of I trade in some sense. Got it that should be the ads have evolved at like this blistering pace I mean it's really been fascinating to watch you all just speed run the you know feature launch list but you launch capy launch pixel so talk to me about the marginal value of advertisers applied data so what's what's the value of that data that the advertiser gives to you in this environment where you know users are kind of telling you directly what they want. Yeah I think it's still quite important and I think there's two types of data and and both of them are important in slightly different ways so one is the information about the products being advertised which is like you know really good feeds the even but I think if you want to elevate that like what is the product philosophy like there is a ramen noodle that I discovered which was a vegan ramen noodle like what is the product philosophy why did they created what's the brand voice and why should the user care about it I think that information is either often on the landing page but it could be hidden as well I think us understanding why that product exists why that product would be valuable to match to the intent of the user is interesting so I think in matching the intent of the product creator. To the intent of the user that's the thing that we're doing some of it is in the landing page but I think there is other stuff which for example we allow context hints which lets you tell us like hey this is our brand wise this is the kind of people who are interested in this is that's about targeting. But much more about telling us why should somebody care about this which then helps us make that decision and I think like you know internally. I want to pronounce really think deeply about the products and they can add that some of it isn't landing pages of course if it's in the landing page we can understand from that. I think the second is conversion data which is of course I think like really really valuable to calibrate and ground the first party signals right like of course like I'm saying I want this does that actually convert into an action is there a stated and reveal preference mismatch is there inflection at which I start taking those actions so I think that the conversion data still helps us ground what we see on our platform because the user is talking about xyz do they actually finish that action at what point does that inflection happen or like you know what is the reveal preference like hey I want really amazing you know high quality products but I need a lower price point how do we understand those things I think like that could that show up in the conversion data I mean over overtime how does this evolve I still believe that like the intent of the product and intent of the user need to be matched a lot of that can be in the landing page but it could also come from creator content and other types of content sources the conversion data is I think is always still valuable to ground everything because in the end that is the combination of the user behavior and it's a way to calibrate what the first party signals me yeah it's the label it's just but it would be less data hungry for sure I connect that I do believe right but it ultimately it's the it's the source of truth it's the label and it's the source of truth exactly it's kind of like a corollary to that question so I mean you know you work that matter for a long time and you know you're building ads for a chat bot which is a different fundamentally different product experience in a social media feed but you're also building a whole ad system from the ground up. And so you get the opportunity to build it with the benefit of hindsight and knowing you know what mistakes were made in building you know meta's infrastructure how do these different features earn their place. So like you know I imagine coming in it's like how do we envision a modern ad tech stack is it put aside the fact that we're building for a chat bot let's say that we were just building for a new social media platform but we were building this thing from the ground up like what earns its place and what doesn't what's antiquated how do you make those decisions as you're as you're building the infrastructure behind this. Yeah I think this is a really great question so I mean of course like we were in this privileged position where chat GPD has more than like one billion weekly active users of course it's grown through the year to this point and as a result I think we could scale our business in such a small time so I do think it starts from like a really strong consumer product that was growing and then ads naturally fits into into that experience. But of course I think like just being large with a lot of consumers doesn't it earns you a conversation but it doesn't earn you the budgets and the technical integrations and and the fact that like advertisers still have to evaluate whether we're delivering performance and that's what we work a lot on. I think going back to your question about like how does it change how we build the system I think like some parts of it is that how can we simplify and automate more of this I think over time we've learned that every time a constraint was added which made sense at some at some point in early stages. At some point in feed, you could only see businesses ads from businesses, which were one hop connected to you, like friends who are fans of a page. You could see that only from that till it got to unconnected. So I think similarly, I do think how do we start from max automation from day zero? How do we ground it in conversions and signals that truly matter to the business? How do we simplify creative optimization because that matters so much? Because eventually everybody starts with like, hey, that a very specific creative should be created. And it gets to, hey, actually, just needs to be optimized for that user because every user is so different. So I think we want to kind of start with knowing that what AI can do to select a really good ad, what AI can do to make it presentable to the user. How do we maximize the ranking forces? Let's say narrow targeting. How do we simplify bidding so that it's not like 50 types of bidding systems? Of course, we have a max bid, but we have an automated now as well, which is doing really, really well, like really finds you cheap opportunities. So I think like kind of maxing out towards automation because over time, every system kind of learns that automation is the best way to deliver actual advertiser value. But I do think we continue to have to prove that value meeting the advertisers where they are in some cases. Like, I think, for example, PTC and integration with Shopify and HubSpot, that's a really good way to like, can we just integrate into the products that they use? And they press a single button, it's like, hey, I want to spend 100 dollars today and get like these conversions go, right? How do we make that really easy? How do you write a prompt and just create an ad? We just launched that as a plug-in to chat, and then I think sponsor agent, which is a very native thing. So maybe those are examples of things which we kind of skipping a few steps because we know how the Jardin ads and not making the same mistakes as before. But still, very importantly, I do think that's the reactating thing which like has to be grounded in advertiser value with these things. It's not because we want to do X, it's because we know we can deliver better value through it. So you also launched the prompt-based optimization and analytics tool, right? I mean, that's like kind of a thing that you would launch for a modern ad platform. There's nothing to do with you being a chatbot product. It's everything to do with, this is how an advertiser should interact with their data. That effect. It's not a million clicks, which is annoying as someone who managed teams and did that. Yeah, absolutely. I just simply want to say like, what's my best campaign and what should I do differently? It should give a pretty thoughtful answer. Right. So with search, you know, advertisers historically express what they want to buy through keywords, and then, you know, there's, you know, kind of expanded mash, but still it's all anchored to the keyword. That's the input from the advertiser. And in social, it's more anchored to creative and then conversion objectives and automated targeting. So first of all, like, what's that atomic unit for chatbot ads? And then what do you kind of need to get from the advertiser to maximize the opportunity for them? Yeah, yeah. I think the most helpful advertiser input is the value that that business can provide to the user, which is like the products. Of course, the landing pages, the actual philosophy behind what their product is. And what are the results that the business values? Like, am I looking to sell more things with this much budget? And mostly like budgets and bid is a good way to express that. Or like, you know, I want to sell these specific types of products more, which is you could say narrowing within the range of things. So I think like, what is what is the business offering? Make it clear through all the different means. Landing pages, product feeds, context sense. What are the things that the business values, which is, hey, I have a budget of X dollars or I have Max bid or like, I want to achieve this kind of ROI. So that's kind of the basis of it. But familiar inputs are still important. So I think we need to kind of cater to some of that. But our goal is like, you know, with run driven ad creation, you could just translate a website or a brief about a campaign, some optional text and landing page, some structured data and create ads. So I think at some point in the future, CMS is like, you know, what kind of codex did for coding, which is like, you explain the clear what you want, you explain what you want and how it's done becomes irrelevant. Is campaign an important thing? Yes, it's there for some control, but just express that and under the hood, it could get automatically created because I do think having that that ability to not have to manage these very low-level units is maybe the final future for us. Were there any ideas that you had that you felt like were clear improvements to the ad-buying media buying process, but that were just too off the wall or like, that it would be too fundamentally new or unrecognizable to advertisers to implement or it was it was it was pretty clear what you needed to build or where the things you wanted to build that you felt like, this is just too far away from how people buy media we can't introduce this. Yeah, I think that's what I was saying, which is like not some familiar input still matter. I think if you look at our ad manager, it does look very similar to a traditional ad manager has the campaign structure and everything else. One of our teammates started the site project, which was just created a plugin for chat GPT work, and when I first used it, I used one of my favorite brands, Solostov, and I like create an ad for Solostov, this is my website, and that's it, and it actually came up with really good ideas. I think in the future, I think that would probably be the key way, but you have to start with some of the familiar inputs, and that's not what we started, the chat experience is not what we started with, but it's growing really, really far. It's like people just go connect their account, and actually it just creates your account. If you say I want to create ads, and you can just start creating ads through chat duty work, but we started with the ad manager, which is slightly more familiar. I do think that of course, there are different entities in the system, like you you have larger advertisers, you have bigger teams, you have agencies, so I think they have different use cases, but if I think about the SMB, the final frontier would be that you could just write the plan, and then daily gives you a summary of what's happening, and based on that, you're just like, as if you were talking to your CMO or your whole marketing team in English, I could do that. I think that's what the chat GPT work plugin does, our ad manager is slightly more traditional. We do want to convert these things. What can you share about the roadmap? Because I follow this in the weeds, I post, I try to stay on top of everything, your team releases. Maybe we could just give us a lay of the land. What's the current state of the chat GPT ads platform for the people that are not following it as closely as I do? Then what can you share about the roadmap? What's coming up? We are now available more than 40 countries, tens of thousands of advertisers, really, really strong self-serve, advertiser-based business can buy through partners, so we have API partners, or they can buy through ads manager. We have a few bidding approaches, of course, CPM, CPC, but the most important being outcome-optimized bidding, and then measure downstream action through pixel-cappy. Kind of like very standard from that perspective. I think maybe the new things, which is like the AI tools for campaign creation, the sponsored agents, I think those are the new areas that we're testing, and then of course, the integrations which Shopify have spot, which then takes us to where the business is already operate, and we launch that recently as well. I think in terms of the roadmap, maybe still like three big aspects. One is expanding access, so growing globally, I think we'll be launching two even more countries as we go. I think measurement is still very important because going back to your small platform syndrome, I think it's a low measurement syndrome is the thing as well. We select the tools that help people measure and understand values, not build for a new system, so we need to adapt to their existing tools. I think there'll be a lot of work to make that happen, and of course that measurement then helps us optimize, so I think that's a conversion optimization loop that we need to continue to improve, and a bunch of work in that. And then the last is, I think format optimization. I still feel like the iteration towards what a native experience looked like is going to be interesting. I think how the sponsored agent work for Legion ads, how the sponsored agent work for e-commerce, different articles, I think that's very interesting, how the experience shows up, whether it's contextual personalized across the chat, and how to be conveyed to the user. I think that remains a big part of our work. Well, the measurement piece certainly resonates. I think one of the issues is that SMB measurement is very different from the perspective of an advertiser here. SMB measurement is very different from big co-measurement, right? And the big co-measurement is like, yeah, of course, like an onboard a new channel, it's truly easy. I have measurement that is spans everything, and gives me insight into new channel performance, and SMBs just don't, the reliant on the platform level measurement, and that really is the hurdle to clear. I do think that's changing. I mean, you have companies like incremental, like house, they do great work with incrementality measurement, and they support SMBs full disclosure of an investor in incremental, but then they're a sponsor of the podcast. But, you know, it's still difficult. And so, I mean, I think that's the biggest challenge. That is the crux of the small platform syndrome problem. I want to close on going back to the principles. So I wrote the series Google's gambit. I talked about search as this distribution mechanism. You go to search, the whole goal is to go somewhere else. You don't want to stick around search, and actually search does a bad job if you have to scroll, or if you have to click to the next page of search results, that means the search failed, right? So you go to search, you want to click somewhere else as fast as possible, that's their motivation. Chapats are a destination. They retain the attention within the conversation across conversations to your point about memory. Successfully helping someone might also mean ending the conversation quickly, right? So how do you define going back to that waterfall? You talked about the value waterfall, like that. I don't remember the placements, but maybe you could remind but how did you define success for the ads product when more engagement, more impressions, more advertising opportunities, and better user outcomes might not all coincide. You might not get all of those things. What are you optimizing for at the end of the day? Yeah, I think the rubric which I mentioned goes like user trust more important than user value, more important than advertiser value, more important than revenue. And I think that even differentiating between user trust and user values is an important thing because trust is a very different thing which is like I might love an ad but I could still think that I don't trust the platform, right? So we have to solve for that. And then within the principle, the fifth principle that we had and we specifically put that was like we're not going to ever optimize chat GPT for empty calories. This is like time spent and you know just random clicks. And of course this goes back to the independence part also because of the organic model is completely independent, completely separated out. So I think if somebody decides quickly that is success, I do think that if we do that job really, really well people will come for more decisions to make trust us with more things, be a part of more of those user journeys. So I think it's about deepening that relationship versus like you know, extracting from a single relation like a single kind of a user journey that's happening. I think building a really good habit for users where truly they find value in it is very important. So I think if someone decides quickly that is still success. And then I think as long as within the ad system, of course, like we optimize for ad score, which is a user value plus advertiser value kind of a matrix. So I think that gives us a very clear direction of ads need to become more relevant and even every impression needs to come. So I think the nature of the product is that the strong understanding of the user and strong relevance can be capitalized for delivering highly the element ads with much fewer impressions. It's not like you're scrolling through a lot of stuff and then everything has to catch attention and the more you scroll through the better. I think if you have really identified a really good deep user need and you self that through the organic product and combined with the ads product, I think you can deliver much more value to the user by with much fewer impressions. So I think like again, focusing on the long term value for users so that they keep deepening their experience coming back to the product to solve their problems. I think as long as that continues the business with time. I saw this was fantastic. I appreciate you coming on the podcast. I appreciate the pace at which you are deploying. How can people stay on top of what's happening with ChatGPT ads? Yeah, so I think like we like I think we have a very good clip of like updates which are coming every week or two. We post about it on Twitter as well. Often I do that and then on ChatGPT ads block. Great. I saw it. Thank you so much for your time video.