Ep 59: OpenAI Product & Eng Leads Nikunj Handa and Steve Coffey on OpenAI’s New Agent Development Tools

I'm Jacob Afron, and today on Unsupervised Learning, we had a really wide-ranging discussion. We talked about how developers should think about where these agents do and don't work, as well as computer use models, and how those are being used. We talked about how enterprises should be building for this agentec feature, as well as what will differentiate application builders who are building on top of these models. And we hit on AI infrastructure, what the needs are still for developers, and where they're still room for startups to compete. This was a ton of fun to do this right after a really compelling release from OpenAI. I think folks will really enjoy this without further ado, here's our opposite. Well, thank you both so much for coming on the podcast. Really appreciate it. Yeah, good to be here. Awesome to be here. Yeah, I mean, congratulations. Never a dull moment of OpenAI, but I feel like the last month has been like an even crazy by your standards, the amount you've shipped. Yeah, yeah, it's been quite a journey, hasn't it? Yeah. I can certainly imagine, well, I feel like well, there's a ton of things we'll want to dig into in all the stuff you've released lately. But maybe the starchest at the highest level, I'd love to hear kind of your long-term vision for how we as consumers will interact with agents in the next five, 10 years. Yeah, I mean, right now we see it all happening in services like chat GPD. You've got deep research, you've got operator, people are like specifically going to the spot. I think the most exciting thing about releasing models and APIs that are underlying these agentic products is that we're going to see them in more and more products across the web. So computer use coming to a browser that you like to use are operator automating like a task that you do day-to-day at work. And doing all the clicking and filling out forms and all the research for you. I think it's just going to become more and more deeply embedded into products that you use today, day-to-day. And that's what we're most excited about, at least, in the API platform is to just disperse this thing and have it be everywhere. Yeah, I think one of the cool things about working on the API platform is you actually don't know what people are going to want to build. It's very vertical, it's right? So on chat GPD, first party, we kind of have an idea of what people want to do. But in API, people know their domains way better than we ever could. And so it'll be really interesting to see how these products, and these model capabilities, make their way into verticals across the-- Is there a particular agent that you're waiting for? Like, God, I just can't wait until I have. I've run always like the travel agent for some reason. There's one that's top of mind for you guys. My top one is an API designing agent. I'm off time. Stephen, I'm back and forth. Yeah, we're just going through every single parameter name that we can think of. Should be Bram config or config parameter. Yeah. Yeah, that would be amazing. We have some deep research thing that looks at the best API design things and it's really-- Yeah, we could just be fine tuning it on all the APIs that we really like. That's actually a really good idea. Yeah, I hope someone takes your API and then gives you a product back. That's great. That's great. That's great, right? Yeah, exactly. I guess a question a lot of folks are asking is we're obviously in the very early earnings of these agents and they're kind of accessing the web and communicating with each other in ways that have been built for this previous paradigm. There's all sorts of futuristic ways that folks think about how these agents may access the web and kind of communicate with each other. There's even that viral YC demo where agent realizes talking to an agent and they switch to something that's easier to exchange information. Like, how do you guys think about how this evolves? And obviously, I'm sure the developers will take even all sorts of directions, but any early inquiries of how this might work. Yeah, for sure. I think on agents communicating or getting information from the web, we've already seen a big change. We've gone from this world where an agent would do a single turn, decide whether it wants to search the web or not, get information from the web and synthesize a response. That was like what I think 2024 was about. 2025 was already about products like deep research where the model is like getting information from the web, thinking about what it got, reconsidering its stance, getting something else from the web, opening multiple web pages in parallel to try to save time. And this whole chain of thought tool calling tools in the reasoning process is like a significant shift in terms of how agents access information from the web. And you can totally imagine some of these web page extraction details being replaced by other agents in the near future, where I don't even know if this agent needs to know that it's talking to an AI agent on the other end. It's just like an endpoint that it calls. And it's like, oh, it got some very useful information that it uses to make its decision or backtrack or do something completely different. So yeah, I think it's all going to be pretty seamlessly embedded in this chain of thought process where tool calling is just happening between both the internet and your private data and your private agents. So that's where I see it going like pretty much in the coming months. Is this something that you think companies should be-- because obviously one version of the world is they can just wait for agents to start accessing their sites. Another version is they should be building actively toward this and create the agent themselves that makes it easier for a consumer agent to hit it. How should folks that are running these-- running products at some of these companies be thinking about this? I think the developers are already doing this. So we put out the agent's SDK for this very reason, because people are creating these multi-agent swarms of multiple agents to solve these business problems. So if you look at a customer support automation problem, you have one agent that's looking after your refunds and other that's looking after billing and shipping information, something else that makes a decision on pulling the FAQ or escalating to a human. And so we already see this multi-agent architecture be very popular. And we want to make it much easier for developers to build on it. And that's why we build the agent's SDK. Now, when do you start exposing these agents to the public internet and how that becomes useful? It's going to be very interesting. I don't think we've seen too much of that. But it makes so much sense that that will happen at some point. And my advice to companies and products would be just like build these AI agents internally to solve real problems that your company is facing today. And whenever it becomes apparent that exposing this to the internet for someone else to communicate with you makes sense that will just happen. And I don't think we're too far from it. But I think it'll just happen in the coming months. Yeah, totally. I think what's really interesting, too, is that before we've seen most of the data that a model is seeing is either your own data, chat history, file search. And I think what's really interesting, especially with these tools that are much more connected to the web is that we'll see a lot more data going into the model that's actually from around the web, and not just data that you're providing. Which is really interesting. That's all for sort of thinking about incorporating and using these APIs. What he rustics do you guys use for where agents do and don't work today? And how would you advise folks? Let's take a step, a little bit of a step back. So, 2020 for what most agentic products looked like was a very clearly defined workflow with about less than 10 tools. So about a dozen tools at most. And this very well orchestrated go from here to there to there to there. And that's how a lot of companies like built a bunch of really cool coding agents, built a bunch of really cool customer support automation projects, DP search projects, et cetera. In 2025, we've gone to this model where everything is happening in this chain of thought. It's like the model is in this reasoning process is clearly smart enough to figure out how it should call multiple tools. And then also figure out that it's going down the wrong path, take a U-turn, and then try something else. And I think you've gone away from the whole deterministic workflow building process. You know, Pneas has been working on tools like reinforcement fine tuning, et cetera, to make this something that developers can use themselves. I think the next step after this is going to be, how can you get rid of that 10 to 15 tool constrained? That you have, how could you just expose this thing to hundreds of tools? Have it figure out which is the right one to call and then make use of those tools? I think that's really the next-hand lock. And then this thing becomes like, it's like, it has all the super power it's needs. It has the compute. It has the way of reasoning about different tool trajectories and it has access to a lot of tools. So that's sort of really excited about in the coming months is removing the number of tools constrained. But yeah, it's kind of hard to make that work with today's models, but I think that's going to change. Yeah. I think also just increasing the available runtime that these models have to go off and do what they need to do. If you're a human, you can go off and work on something for a day and use as many tools as you need to get the job done. And I think now we've seen runtimes that for models, especially deep research that are in the minutes. But being able to get these things to go into the hours and into the days is going to yield some really powerful results. Last year, you had to put such specific guard rails and chain things so closely together because you couldn't let things go off the rails. It seems like now you're even more flexible in what you can allow. And then obviously the dream is just like, yeah, go off here. It's like the hundreds of tools that you could use across every task. You can go figure it out. Yeah, exactly. I think it's not like-- let's see how the next generation of models generalize to all of the use cases that developers are going to have. There's also this reinforcement fine tuning technique where you're creating these tasks and graders. And if developers can create their own tasks and graders and get the model to find the right path, the right tool calling path to solving a particular problem that's very unique to that developer's domain, that would be amazing. So I'm really excited about the next series of models that are going to come out. And our early results from reinforcement fine tuning, all that comes together. that makes agents that are actually very useful in her body. - Yeah, the really cool thing about that is you're really steering the model in its chain of thought and you're kind of teaching it how to think about your domain, which is just like a really powerful mental model when you think about it, you're like, okay, how do I think about, like how do you basically train a model to be like a legal scholar essentially or trained it to be like a medical doctor or something like this, really like training the way that it thinks in the same way that four years of university would train you to think in a specific way. So I think the reinforcement fine-tuning thing is a great example of where you're gonna see like a really interesting verticalization for these models. - And for that, I mean, how have you thought about, I feel like one of the classic problems or people talk about there is you can provide, I'm sure folks want something off the shelf that makes it easy to do the kind of grading and evaluation. And at the same time, like some of these domains are so hyper-specific in their own problems. How have you thought about like the infrastructure level, the right level of tooling to provide to folks that are doing that kind of fine-tuning in like a domain like legal or healthcare? - Yeah, I'd say it's like still a work in progress. We're like, I think right now the things they're exposing is basically we're allowed, we're kind of giving developers a way to build their own graders. So for example, if you have an e-bell that you show does 50% on a medical task, right? You can build these graders, let's say that can cross-reference a model of this chain of thought or something else that it's out putting based on like some sort of like known ground truth, like a medical textbook or something like this, right? And so over the course of fine-tuning, you can sort of like steer the model in the direction to build the produce like better and better outputs and just kind of be able to just steer it in that way. So we're kind of providing the basic building blocks, really mostly just like these really flexible graders that allow you to like take a model output and then grade it against some sort of ground truth or execute some sort of code to prove like, oh yeah, this is mathematically correct. We're not just checking that this string equals this string, right? It's actually some like mathematical correctness to it. - Yeah, I mean it feels like the biggest question across the board in so many aspects of AI right now is like what actually can be graded. I mean, it feel like it's the big question and like test on compute and what you can scale. Like, and obviously I think if you take healthcare and law, for example, one critique of some of these evals is like, well cool, like being a lawyer is not passing the bar, like being a doctor is not like passing these medical exams. Anything like you've seen folks on the ground doing that you feel like is a creative way to actually like best use this type of approach? - Honestly, like after we're having talk to the folks who've built things that are on operator and the research internally, like it's pretty challenging right now to do this stuff and takes a lot of iteration. I don't think I've seen anything out there that's like productized grading and task generation in a way that just like nails it for your domain. I think this is like the biggest problem to be solved this year and if not like it might even go into like next year. The technique is gonna come out, but how are you going to actually build really good tasks and graders is something that's gonna be pretty challenging. Yeah, I know it's possible now, these products exist. So you like know that like it's possible to build something like deep research, it's been some replications of that around the internet as well. So you like have enough proof over here. It's just about how do you productize it so that like almost anyone can make use of it that's gonna be hard. - What about computer use? Like how do you classify for developers today? Like, you know, how they should think about using that where it works, where it doesn't. - Computers when like a surprisingly a lot of cool use cases. You know, initially we thought that a lot of computer use use cases will be around legacy applications that don't have APIs and people have been trying to like automate this thing for ages and they haven't been able to and that's definitely the case. Like we definitely have had a couple of customers try it out in sort of the medical domain where these like super manual tasks that people are just like clicking through across three or four different applications to do things. And that works really well. But you've also seen, we've also seen examples of companies that are using it to do like research on Google Maps. So I think UnifyGTM is one of the companies that had used it earlier during our alpha phase and they basically, they would have like climate tech startups ask questions like has this company expanded its charging network and so what the agent would do is like open up Google Maps turn on Street View and like go to places and see whether they're like more chargers or not. And like Google Maps does have an API. Actually, don't know if Street View has an API, but it's probably really hard to like figure out which exact location in which direction to look at maybe. And so the all of these like, you can pretty much automate anything. It's kind of cool. So you could start there and then you could maybe think about API approach after that. I mean, there's a whole like many, many domains like don't map to JSON, right? Like you can't serve them over the web in plain text. So like these kind of use cases where you need some sort of accommodation between like vision and text ingestion, like I think are really, really well suited for for. Yeah, that's a really interesting example. Yeah. UnifyGTM is fascinating. That's so cool. I was struck by obviously you really, you had a bunch of alpha testers and whatnot. So you released this. And then the next day, like I feel like every big company was like, this is the awesome thing we built. Like the same. Yeah, right. Any particular, you know, even just in the, in the weaker system spin out, like any particular favorit's that like you didn't expect or kind of cool ways people would use these. Ooh, that's a good one. Post alpha, let's think. Well, the computer use ones are the coolest. I think you have, I really am excited about the platform players on computer use as well. Like if you think about the other tools that we have. So we have web search, we have file search, and we have computer use. Web search, you have a bunch of companies that provide APIs for people to be able to get data from the web, put it into the models context. File search is like pretty mature. Honestly, you have the vector database industry. And computer use, like, I think things are super early. The main thing people want to do or businesses want to do is like take these, like Docker containers or these VMs in the cloud and then like put their software in it, put their authentication into it so that they can go and automate things. And there are a couple of really cool ones. This browser base that provides this service, this like a YC startup called ScrappyBara that has I think one of the better developer experiences around making computer use models work really well with hosted virtual machines. And I'm like a developer platform person. So for me, it's like looking at those platform plays and like, all right, like what's the thing that people are going to build on top of that is very exciting. And so yeah, I'd say like those were my talk to browser base and ScrappyBara. I'm pretty excited to see what they do. Yeah, I thought to arc was doing some pretty cool stuff. They were basically building a tool where they were going to have, basically, you just like open a tab and give it an instruction and then it kind of goes and does something in the background. I think it's very much an operator like use case, but it's really baked into the product. I mean, it's just a web browser. You're using it, right? It's not like necessarily like baked into a tab in your web browser. It's really just like part of the browser itself. That was that sort of native integration was really cool. Yeah, I think they're calling it DL or something. OK, right? That was super cool. Yeah, that's awesome. Has anything you've kind of noticed so far that like maybe some of the most sophisticated users are doing with the APIs? Like, no, I wish we could disseminate this more broadly. If only we were on a podcast and we could tell the world-- This is a good way to use some of these things. But I think you've noticed patterns that some of those sophisticated folks are using. Yeah, for the tools like it, it still feels like pretty early. I think during the alpha phase, we definitely found folks who are-- they try to get the model and the tool to do the thing that they're trying to get it to do. And if that doesn't work, they try a bunch of prompt engineering. And then that doesn't work. They make this a step in the workflow. And I think by going through those steps, they typically get what they want. It's like, hey, web search, the tool is not giving me exactly what I need. But can I make it part of my workflow, where this is just one of the steps that gets information from the web and then like a boss it onto something else, either deterministic or another LLM step. On net, I'd say it's pretty early right now. And we're going to discover a lot of this in the coming weeks. Yeah. I think one to invert the question a little bit. One thing that I'm really glad that we were able to ship is in the agent's SDK, this idea that we're going to split the concerns of what your job is or what your task is across many different agents. It's very much analogous to the single processor computer versus the multi-processor computer. You just allow each agent to focus on one task and you give it all the context. And then your efficacy on those tasks goes way up, right? Because you're not trying to prompt engineer one agent to do 100 different things, right? You're kind of just like spreading that across. So I was really glad to see us sort of like, I'm not sure if we invented that paradigm or not. I'm assuming we didn't, but just to like ship that as a really first class pattern, I thought that was really cool. Yeah. No, it's not interesting. I feel like you alluded to the fact that like, hey, if it's not working, you can kind of just add it as a step. And I feel like one interesting, you know, quantity that we have on the investing side is like it feels like, you know, a lot of people, whatever the current capabilities of the model are, they kind of build whatever scaffolding they need to make them work. And sometimes you're like, well, that gets you the product in the market now and gives you a product that is valuable. At the same time, if you went to a beach and waited three, six months for the models to get better, they may just be able to do it, right? With your 100 tools to one thing versus like, chaining the steps together. And so, you know, I'm curious like how you think about, like, you know, the kind of steps of people are building around the models, like, does that all get obviated over time or is like some of that useful? I think that this is the most like agent, or agent and tool orchestration is like, the most important thing right now, because my opinion is that the models are much further than where most AI applications are like, making use of things. And, There's so much value to be extracted from these models that building things around models to make them work really well is an extremely important thing that AI startups should be doing and AI products should be doing. Yeah, it's like time and time again, where even on customer support automation, which has been a thing that's been around as a concept for a while, we had a couple of companies like really crack it in late 2023 and early 2024. And the adoption has been like kind of slow. You don't see that many companies move as fast as the first 10, 15, 20 companies moved. And it just shows how important it is to be good at orchestrating, to be meticulous about looking at your traces, figuring out how to prompt engineer, having an evil said so that your problem doesn't degrade something else. This is so hard today, it's crazy how hard it is. And so I would tell people that's the exact thing to be focusing on is how to make these models work really well. Yeah, 100%. And I think too, just the idea of splitting up your task on many different agents is just makes debugging the whole workflow way easier. Because if you have a really capable model and it has 100 instructions and you change a few tokens, it might drastically change the outcome if you've got it. But if you just have one handoff agent, you have one triage agent, you have one this. Like tweaking each one of those becomes a lot more isolated where you're not the blast radius is much smaller as you're sort of like hill climbing on your e-vow. I think when you were on late in space, you mentioned that you like over time want to add more knobs to make things more customizable for developers. What do you think that looks like over time? And how did you think about this kind of tension of providing something that's like relatively easy to use out of the box for the ultimate amount of customizability? Yeah, totally. I mean, this idea of APIs is louders is really something that we took from first principles when you were designing the responses API. And I think it really comes down to a couple things. You want to give a lot of power out of the box. You want to make doing the simple thing really easy. And then you want people to be able to get a little bit more reward for every effort, every amount of effort that they put in. And so for us, this looks like a great example, I think, actually file search where it's actually really easy to use just out of the box. You upload some documents, you don't even have to do it in the API. You can just do it in the other website. You pop in your vector store ID and it just works. And now let's say, OK, well, this actually isn't quite working for my use case. Well, OK, now I actually have knobs to go in and tweak the chunk size. The default is 400. Maybe I want it to be 200. Maybe I want it to be 1,000. So I have those knobs. They have sensible defaults. And so I can go in a little bit deeper and get a little bit more reward for everything I'm putting in. And it goes way deeper than that in the file search example. You have metadata filtering. You have the ability to customize the re-ranker. But this stuff isn't-- we don't force you to set all this things right up front. We kind of give those things to you and expose them. They're in the docs. You can find them. But if you're just kicking the tires with the API, you don't want to think about-- you're like, what the heck, re-ranker, right? So yeah, that's going to be what we think about it. It's like, make it as simple as possible. I think we actually spend a pretty long time trying to get the quick start for calling the API down to four lines of code of curl. And we were really obsessing over that. It should be this simple. But then there's also 50 more prams that you can set if you want to. And they'll have reasonable defaults. Over time, what other knobs might you want to add? That's a good question. Oh, yeah. I mean, for tools like for web search, you want to basically add site filtering. That's been a big ask. Right now, you just have to search the whole internet. Or you can prompt your re-in to it. Specific location to on web search. Where now you can set the city, you can set the country. But like actually setting down to the block or the even court. Which is super important for weather, for events, type of queries. Especially in SF microclimates. Yes, seriously. Yeah. Actually, one of the things we're really excited about doing with the responses API is building all the features into it that we had in the assistance API, but not forcing users into it. So we released the assistance API, I think, November 23. It had this full concept of storing your conversations, storing your model configurations in a assistant object, et cetera. And you found that the hill to climb to get started was a lot. But responses were taking the other approach where you're starting off with a single API call in a single endpoint in one concept you have to learn. And then maybe you want to store your conversation with us. So you can opt into using the equivalent of the threads object. And maybe you want to store your model configuration with us. So you opt into an assistant type of object. And those things you just plug in. It's just one parameter you configure. And that's a knob that you have to have OpenAI host the thing for you. So I think that's another set of knobs we've in the short term we really want to get to. Exactly. Reflecting back on some of the previous APIs, you've released obviously these ones are meant largely to supplant those. Like any learnings or things, you're like, hey, we got that really right. Or actually, we kind of missed the mark on that. And we've kind of fixed it in this current iteration. Yeah, totally. I mean, I think the thing that we really got right with the assistant's API, especially, is tool use. That's where we really figured out. We saw a ton of usage, especially with the file search tool. That's where the API really found market fit, right? Was people wanting to bring their own data to the API and have the models search over it. But what we got wrong is a lot of the things that could just said really it was just too hard to use. You didn't have-- you know way to opt out of the context storage. A lot of people didn't like the context storage. They wanted more of a chat completions interface where they were able to provide their own context on each turn of the model. But also the chat completions interface is quite limiting, right? The API can only output one thing. And the model does many things, right? And so you want it to be able to do a bunch of stuff in the background and then kind of give you the results of all of its thinking and all of its doing. And so we really tried to take the best parts of assistant's API, sort of the tool use and the multiple outputs and all of that stuff and the easy views of chat completions and bring those things together. Makes sense. How should developers think about this kind of suite of developer tools now and the MCP landscape? Yeah, I think they're like-- they're probably solving different problems, right? So the response to this API is focused on making these multi-turn interactions with models really good. So a model should be able-- we're providing a foundation for the model to be able to call itself multiple times. So have multiple model turns and call tools multiple times. So have multiple tool turns to get to a final answer. So that's like-- we've set the building block, which is the response of the API. MCP is sort of like how you use tools and bring tools to models. And I think these things are honestly pretty complimentary in some sense. And we have to figure out what we do on the tools registry and the tools ecosystem side. But I'm super cool. And that's something we have to figure out in terms of how we bring that to our ecosystem as well. One thing I'm struck by is obviously I feel like in the first years post-chat GBT, there was a lot of AI infrastructure companies that popped up that were trying to do aspects of what you're-- you've really snatched Asian orchestration and vector data bases. How do you think about the opportunity for standalone AI infrastructure companies and where it makes sense for those to exist on top of what you guys are building and where it might not make as much sense? Yeah, I think on our side, we're working with our users and listening to what their asks are. And they want a one-stop shop for the things that they want the LLIMMS to do. They want it to be able to search their data and search the internet. And so we've taken a step in that direction. That being said, I feel like the AI infrastructure companies are building low-level, very powerful APIs that are infinitely flexible. And there's always going to be a big market for that kind of stuff. And I think we just got to build the thing that our users are asking for, which are these more out-of-the-box tools. And we're taking a different approach to this whole space. But there'll be vertical specific AI infrastructure companies. I think there's certain companies that build like VMs just for the coding startups. Yeah, coding AI startups out there. So that they can test their code and spin down the VMs quickly as possible. I think they call it run loop or something I've heard of them. Yeah, totally. So we think verticalized AI and for our-- which seems like it makes a lot of sense to keep doing that. Totally. Yeah, it's stuff that we're not always going to want to be in the business of doing. I think, too, there's a whole class of LLIMOPS companies that are doing some really interesting things, like helping you manage your prompts and helping you manage your billing and understanding where your usage is going. I think that that sort of stuff is really cool, too. It's not necessarily low-level infrastructure, but it's still stuff that developers care about. In a multi-model fashion, multi-border, and all of that stuff. Yeah, like open-router things like that. Yeah. Yeah. And obviously, you guys spend most of your days talking to developers and getting their wishlist on tree. It sounds like you got a lot of it into this current generation APIs, but I'm sure there's always more to do. How do you think about-- you were talking about eVals earlier as the problem. But how do you think about the stack-range problems that are still working with these models painful today for developers and what some of the most important things to be solved are? Yeah, I think tools is definitely a very big thing for us to figure out. We have the foundational billing block. We need to build the tools ecosystem on top of it. There's obviously great work on the MCB side over here. And that is top of mind for us to figure out what we do on that front. We also have-- the computer use VM space is pretty early. and I think that's it. that's another big one. Like how do you get enterprises to securely and reliably deploy these virtual machines in their own infrastructure and observe them and all the things that the computer use models are doing on top of that? I feel like these computer use models are going to get so good so quickly, because we're just at the GPT one or two of that paradigm. And this thing is going to be incredibly useful. So I'm very curious to see how the infra on that front takes off. I mean, I think that one of the things that was really interesting to me during the alpha period was all the different environments that people wanted to try out the computer use tool in. We saw folks, the model works best in a browser environment. It's kind of what it was trained on. But people were trying to use it with iPhone, screen shots, and Android. And I was like, wow, that's so interesting. Having me thought about doing that. And so I think that the sky is going to be the limit on what people will want. Is there going to be a company that just does iPhone VMs or sort of like a-- there was a company that used to do just testing frameworks for iOS, things like that. But just now it's for AI models. Stuff that's really interesting. Because different flavors of Ubuntu, all of that stuff, it's really just a huge amount of fragmentation. And so it's going to be really interesting to see how the community kind of steps up to fill the gaps there. I also think people are doing-- I think this started trying to do cybersecurity work. So trying to find vulnerabilities in other sites and surfaces using computer use. Look around for a third of minutes. Yeah, which is pretty cool. It's pretty cool. It's really cool. Yeah. That's really interesting. I guess obviously one of the fun parts of your job must be your-- obviously probably really tightly integrated with the research team, see the models as they come through. Like, anything that you're looking at for on the model side, like I'm sure you get the next computer use model or the next models that are used for agents. Like any milestone is your capabilities that you're like, god, when we can do X-- any time I get the new model, I try X. And if we could do that, that would be so game-changing for our developers. Yeah, that's an interesting one. I actually have a bunch of prompts that I've gotten from a bunch of YC startups. And they always like, this thing never works. And I actually have them saved as what we call presets or prompts in the Open-Aid dashboard. And each time something new comes, I try three or four of them. They're all pretty much focused on like, agentic tool use. And there's like six or seven different tools that are pretty straightforward. And I'm just like looking for these like reliable executions of them from turn to turn. And I'm pretty optimistic like with our next series of models, but there's certain of them that like, it just doesn't get right. I'm also like really keen on finding much smaller and much faster models like faster than for many for sure that are like pretty good at these tool use things. So if you think about the workhorse models or the supporting models that sit around the O1s of the world, that can do these really quick classifications and guardrailing and all, I think that there's a lot of room for improvement on those type of things. And yeah, just like the fastest, smallest classifier would be really, really cool to work on. - Well, especially because they're so fine-tunable. - Yes. - And you can just like really tailor those things like to your heart's content on like a specific needs case. So yeah, that would be really cool. I'm pretty pleased of those. For me, it's Diff's. I just want the model to be able to spit out a diff that can apply cleanly to my code and it'll just work and I don't have to budget to get it to, that's gonna be huge. That's gonna be really, really huge. The models don't really like to understand. - And we don't really understand line numbers that well. - What was your reaction? Obviously there was some really impressive agent work out of China recently and I think that like, you know, it kind of always seemed that the most cutting edge agents would go alongside the most cutting edge models. But obviously, and I mean, I think they're using Anthropoc models and whatnot. But I feel it might have challenged that paradigm a little bit. And so, curious kind of your reaction to some of those demos. - My reaction was like, this is what we've been saying internally is that like the capabilities are there in the models, but like, so few people are able to make use out of it. I think it's crazy that like it's still like this. We need to make it easier for developers and everyone to be able to build more powerful things with the models without like being, you know, like in, yeah, like exceptional AI and ML people. And so, I just feel like it validates the fact that give people the right tools, give people the right models, help them put them together with things like the agents SDK, make these things observable so that more and more people can build things like what we saw come out of China. Yeah, that's my take on it. - I think just making the flywheel spin way faster from, you know, e-vows to production to fine tuning it back again. Like that is such a powerful loop that we just need to make way. Where's simpler? - Yeah, what do you like to think are the key things to make that simpler? - That's my question. - That's the biggest thing to figure out, honestly. Like, if we have a good answer. (laughing) I mean, the research team does it at OpenAI all the time. Like the model is getting better at chat, it's getting better at like doing all the deep research things. The next operator model is gonna be so much more powerful that doing computer use things. How do you productize that is like the thing that we need to figure out. Obviously like with a lot of toil and like really closely observing your traces and like creating the right e-vows and graders, like it works for sure. We just have to productize this and we need to figure out how to make this easy. - It needs to be about 10 times easier than it is. It's definitely doable. You can create an e-vow, but it's hard to work to create an e-vow. So I think that's the biggest thing for me. It's just like how do we make that process of e-vowing your task, your workflow a lot easier? - No, I mean, it's funny. I am struck by it feels like we have a new model and people spend like six, nine months trying to discover the use cases. They probably discover what one percent of what these models can actually do. And then it's like onto the next one. And so it's pretty wild. I mean, obviously, I think we all kind of feel like we're on the precipice of this super large change and it feels like we're going, especially as you make these tools easier. Agents are going to be increasingly ubiquitous. If I'm just like a normal enterprise or consumer CEO today, I haven't really thought about this so much. What would you be doing in those people's shoes? If you're running a company that probably in this agentic feature has some way of interacting with these models? - It's got a build. It's going to start with, start exploring these frontier models, start exploring the computer use models, take a couple of workflows internally and try to get a feel from building these multi-agent architectures to automate things end to end. I feel like that's the most actionable and actual thing that you can do right now. On the tools side, figure out which of your manual workflows need a tool interface and start doing that. I feel like the whole digital transformation and automation thing that it had, it's like a thing during the cloud days is coming back right now. - Totally, yeah. - And so sometimes I talk to users who are like, we want to automate this whole thing, but 90% of the work to be done is to figure out how to get programmatic access to certain tools that you're using. And the LLM portion is just like tiny in the middle and I'm like, this is a very different problem for us and like yeah, you can like solve it with computer use right now and try to like get it to direction, but really just like finding ways to like automate your applications, trying out the frontier models is probably like the main thing I'd recommend. - Yeah, I think it's really interesting being a developer in this era because for a long time, like we have as developers been constantly automating away the bottom 20% of our job, whether it's through better frameworks or better programming languages or what have you. And so I think that like for me, if I were running a company, I would just be asking my employees, like what's your least favorite thing that you do on a day-to-day basis and like, let's try to figure out ways to automate that. That's gonna make like everybody happy. It's gonna increase productivity of course. And so yeah, that's how I would think about it. - Yeah, have you guys done that? - No, I'm not. - I'm so busy. - I have to go to my head, great. (laughing) - Yo, I love that. - I'd be looking at a fascinating conversation. We always like to end with kind of a quick fire around where we stuff a bunch of overly broad questions in the last five minutes. And so maybe to start, I'd love your take on like, one thing that's overhyped and one thing that's underhyped in the AI world today. - No. - My answer, yeah, my answer is like, agents are both overhyped and underhyped. We've been talking about agents for like a couple of years. - It's not through like two full hype cycles. - Yeah, exactly. - At the same time, like underhyped because like, hey, the companies that actually figured it out and build deep research like things are fully automated, like some really manual task are able to just do so much. So yeah, that's my take on it. - Yeah. - I mean, obviously you guys are so close to the cutting edge here. I'm curious like, what's one thing you've changed your mind on in the AI world in the last year? - I think it's the, for me, it's definitely the power of these reasoning models has been, like I don't know, they were always, we were always like aware of this reasoning thing coming and I did not appreciate how that combined with two use is gonna create things like operator and deep research and just seeing that it's possible to like move away from this workflow, like set up that every company was doing to this completely agentic product that figures out two use in its chain of thought and actually it lives like really, really powerful results. That's been like the biggest shift for me. and then seeing early results of our reinforcement fine tuning alpha, those are, you know, that's been the biggest shift for me. And it was like, how it's possible to do this? >> Yeah, for me, it's just fine tuning broadly. You know, I just like the power of being able to, kind of thought that all the knowledge that you could put in the model is kind of baked in when it comes off of the GPUs, but being able to like really add a bunch of your own custom information and seeing how much that moves the needle for a specific task is pretty impressive. >> What do you think will be like the biggest differentiator of application builders long term? >> It was like the question and venture. Is it like, you know, kind of deep knowledge of the models and like how to, you know, how to really build these agents? Is it just like knowing a domain super well so you know what to build? Like, what do you guys think of that? >> I think it's kind of a combination. And then there's this idea of like, if you have whatever special sauce it takes to be able to really bring the AGI out of the models that like we think is in there, I don't know what they do. It's prompt engineering or workflow orchestration or something else, right? Like, I think that is going to be like a huge differentiator. >> Yeah. >> Yeah. >> For me, it's like being really good at orchestrating. Like, I feel like that's going to be the biggest. >> Yeah. >> What do you mean exactly better? >> Like, bringing together your tools and data with a bunch of model calls, with a bunch of models, either in the fashion of it being reinforcement fine tuning and like calling these tools in the chain of thought, or in terms of like, chaining together multiple elements and being really good at like doing that quickly, evaluating and improving it. I think that's like the biggest skill that would move people forward, like in the next year or two. >> Awesome. >> Yeah. >> What do you think are some of the most like, under explored applications of these models today? >> Well, you see, I'm in like, seeing anything crazy on the scientific research site. I think that would be really like when the O series model started, like, that was the main hope and expectation was that there'll be a step change in how quickly scientific research happens. And I think we've seen some like early reports around that, but very curious to see how that changes. >> I think that like so much criticism about like the AI industry as a whole has been that the interfaces are not quite right yet. And I think like, especially for a space like academia, you know, where everything is kind of the same way it's been for a long time. I think like finding the right interface for that is going to be really key and really drive a lot of adoption there. >> Yeah. >> About it, too, maybe. >> Yeah, that's like, it's probably time for something big to happen. >> Yeah, the origins of opening up. >> Yeah. >> Good overbe scoop. Do you think model progress will be more the less or the same as last year of this year? >> Oh, it's going to be more. >> I think it's got to be more. Yeah. >> Especially as like, I mean, it's a feedback with it, especially as like, we, the models are time teaching us how to like improve them with better data and things like that. It's like something we do a lot on the research side. >> Which AI startup are like categories are you most excited about right now, like outside of OpenAI? >> I came from a travel background. I was doing a travel company right before I joined OpenAI. So I'm just like really excited to see sort of like somebody really crack that. I think the travel industry is like super entrenched and just only handful of like big players. And so I'm really excited to see who builds the actual AI travel agent. >> I think it's a favorite demo for you. >> Exactly. Yeah, but like nobody, it's not like, there's not a product that people are using there. So I'm really excited to see. >> Yeah, why doesn't it work yet? >> I don't know. I'm going to go figure that out afraid of this. >> I use granola a lot. Have you heard of that? >> Yeah, of course. That's my favorite AI tool these days. And every meeting, I'm in like very meeting heavy rolls. So it helps a lot. >> Yeah, great product. We'll look. I mean, I think there's a ton of interesting threads for folks to pull on. Obviously, a ton of great stuff that you guys recently shipped. I want to leave the last word to you. Where can our listeners go to learn more about the APIs, about really any place you want to point them. The floor is yours. >> Yep. Our docs, platform.openad.com/docs. And also like the OpenAd devs channel on Twitter, or the account on Twitter. >> And the community forum is always a great place to check out. >> I don't know the domain of that one. >> Is it forum.openad.com? >> Just Google OpenAd.com. >> Yeah. >> Or ask Chad to be T4. >> Yeah. >> Awesome. Well, thank you both so much. This is a bit of a tough one. >> Awesome. Thank you so much. >> Thank you. >> Thank you. >> Thank you. Thank you.