Ep 95: Modal CEO on the Future of Inference, the Chaos of the GPU Market & Origins of the Sandbox Bet

How much of an organized marketplaces can keep these days versus like the Wild West? I went to GTC last year. It was like an open air drug market, but like GPUs. Walking around, like talking to people, trying to get GPUs and meeting people in hotel lobbies, exchanging numbers. It's very far from an efficient market. I mean, some people are working on this. Compute comes in many different shapes or forms. I also feel like structurally it will be a little bit like sort of unwieldy for a long time. Like it's just not something you can like sort of buy and sell in an exchange because it's not like a fungible contract. Welcome to Unsuit Advise Learning. As you can probably tell from my voice, I am not Jacob. I am Patrick Chase, GP at Red Point. So awesome to have Eric on Unsuit Advise Learning for the second time. He is the founder and CEO at modal. I have learned so much from working with him over the last three years. He is one of my favorite people to jam on AI ideas with. He has an incredible ability to move seamlessly from the lowest altitude of developer shipping code to the highest altitude of where the market is going and building what is missing. We talk about the modal journey over the last three years, the crazy technology they have built to make it all work, inference, sandboxes, and what's coming next. Hope you enjoy the episode. For you guys, there is obviously a ton of AI compute demand and lots of people going out and trying to reserve GPUs and CPUs. What's it like for you guys finding compute in the market today? It's pretty rough. I mean, not rough in a sense that we can actually find compute. I think there's a little bit of a misconception that we are certainly scarce in terms of compute in certain ways. But there is capacity you can get. You just have to look a little bit further, accept higher prices, accept longer terms, et cetera. And we do that a lot. I mean, we probably do like two compute deals every week at this point, if not three. And so we keep on stacking up more and more compute. And I really think that's the beauty of modal is we can aggregate all this demand and manage the capacity and aggregate much more efficiently. And the valid problem of modals has always been like, don't think of my capacity, like we had a lot of that annoying stuff to sort of the day-to-day knife fighting of getting capacity of working. We work with 25 different classes at this point and adding more all the time. How much of an organized marketplaces compute these days versus like the Wild West, like random people that somehow have capacity and late night text messages? It's a lot of late night WhatsApp groups and like it's pretty unorganized. Like, I mean, I went to GTC last year, was there earlier this year in San Jose, the NVIDIA, a big conference we're doing. And it was like an open air drug market, but like GPUs, like just like walking around, like talking to people, trying to get GPUs and, you know, meeting people in hotel lobbies and, you know, exchanging numbers and, you know, I mean, at some point we, I think we got connected with someone who had GPUs. Because I think our VP of sales cousin was golfing with someone and then we got connected, there's many such cases, like it's very far from an efficient market. I mean, some people are working on this and I think over time I've got a little bit more efficient, but it's very heterogeneous. Like compute comes in many sort of different shapes or forms. So I also feel like structurally, like it will not, like it will be a little bit like sort of unwieldy for a long time. Like it's just not something you can like sort of buy and sell in an exchange, because it's not like a fungible contract. Yeah, the key differences being like I guess the kind of actual powered shell or size of the cluster and like the chips themselves or, you know, to your point, there's a lot of people who have been trying to build like, hey, you know, compute as this fungible single asset that you could build a marketplace around. What are some of the kind of the key differences you observe? I think a lot of it's actually boils down to reliability. I mean, first of all, it's like, you know, differences and like disk and networking and of course CPU and memory and things like, like all these things vary and region, like location, like they're all over the world, like we have GPUs and the US, of course, like all over the US, but we also have GPUs and like Indonesia, like all over Europe, etc. So location is one of those big factors and then, but also reliability. And that's another thing we do exceptionally well is like we aggregate all this capacity and then when neoclass die or like when they have failures, which they often have, because some of these are like former crypto miners haven't really thought about, you know, running a 24/7 operation, we can just take it out of rotation or like when GPUs fall off the bus or whatever, like we, in many cases, we just file a ticket automatically. Someone goes and looks at it. Have you found the data providers? The companies that provide pricing for GPU and that sort of things. Have you found them to be helpful? I think moderately helpful. I think the pricing less so than just their ability to connect with a lot of providers. I don't fully trust the pricing today. But I think these, some of these providers are actually very good at like, you know, knowing all the neoclass and we work with them sometimes to get connections with cloud providers. The pricing I think over time will get more accurate like our internal pricing does not match what we see publicly when we buy capacity. But I think over time, hopefully, like we will have more efficient like external weaknesses. I think that would actually be very useful. And then on this like rocket ship journey these past years and so lots of things we'd love to dig into for our listeners on the business side. Maybe starting, you know, I think one of the interesting things about the history of modal is, you know, for a while it felt like you, you know, publicly said, I think on your first unsupervised learning episode that you were less focused, you know, on these LLM inference calls specifically like a simple API, you know, call for a model. And, you know, you kind of recently released a product there, maybe talk a little bit about the journey and kind of that initial reticence and what made you decide that like now is the right time to really, you know, double down this. Yeah, so we started building this in late 2021 basically or early summer on that time. And the initial product was this serverless function. And we thought it would be really useful for a wide range of stuff around like data AI machine learning broadly speaking, kind of like looked at a lot of different use cases. And but then like chat to PT came late 2022 and stable fusion also came around the same time. It turned out like we were like super well positioned to host a number of stable diffusion applications. So that provided that was like the first year of growth was like all stable fusion. And then we started like expanding into like adjacent use cases like audio and video and stuff like that. And it was like early teams that, you know, they were deploying January in friends. They didn't want to deal with the other language infrastructure. They wanted to move fast like we, we started agreeing a lot of different GPUs across a lot of different clouds even back then. We always knew we wanted to be multi product, but we started with the serverless like this sort of compute privitics. We felt like it was so general purpose. And we focused a lot on like kind of what I think was like high code use cases like focus on like machine learning engineers and researchers who want to write code and focus a lot on like being kind of low in the lower in the stock, right. And pretty early we built the also storage layer file system to store data and in particular container images. Then we launched a sandbox product in 2023 and and the reason why we launched it back then we started seeing people like messing around with LM to generate code. And we figured actually like a lot of stuff we built on the infrastructure sites like really well suited to run that code and they're going to need a place to run that in an isolated way. It didn't immediately wasn't like an overnight success, but like basically 2025 kind of blew up because I think models got really good at writing code back then. So about a year ago, so that was sort of a venture into like kind of becoming like a multi product vendor like starting about 12 months ago like sandboxes has been in a tier and grown. Like I don't know thousand x almost or something like that is about a third of our revenue right now standalone is the 300 million run rate business sandboxes. So that's been incredible in terms of you know becoming a multi product vendor we just launched also a training product. So sort of you know building trying to lay out like all the different building blocks and you know what is that suite of tool looks like and what are the tool boxes. In this like new world of like AI applications like they're all going to need a bunch of new primitives and like we went off for all those primitives and what is that what does that look like. So LM inference of something we were a little bit on the sidelines for a while until actually I would say the deep seek moment so that was about a year and a half ago like early 2025 went when deep seek came out which was like the first Chinese open source model that was actually really good. And there was like fuck it like we got to get in here and like we started building up a team to do a lot of inference optimization models has always been really good at LM inference like but we didn't necessarily have a product for it. In fact we didn't even have an inference product until very recently we were just like really good at hosting inference including LM inference but it was like kind of custom and a lot of the teams that want to run LM inference a year or two ago like they have to deploy themselves or like we would work with them. But we started seeing a lot of demand with the deep seek moment over the last 18 months started seeing a lot of growth LM inference is not about a third of a revenue to so it's we've seen incredible growth with that. And a few months ago we started we we ship the number of LLM specific features because we think you know we kind of laid this like foundational layers like the storage to compute the serverless functions the file system etc. I think there's a lot of opportunity to move up in the stacks we started moving up in the stack now and and launched dedicated inference products so you can get like a single tenant models like all this data the R Chinese models or a bunch of us wants to like one single. Click or see a lot of command you can just deploy on modal as well as a much of shared end points like you can just call models off the shelf without having to deploy fully token based pricing. And so I think those are just you know the first out of like many more products to come like slightly higher up in the in the layer we think there's a lot of opportunity because we're sitting on this like very solid foundation. and the compute and the storage. Maybe like you guys will start at the bottom of the stack and work our way up. But going back in time, you touched on it, but you guys invested in some really hardcore technology in the early days you built your own file system, container run time, those types of things. How does that show up in the product today when the experience developers can have? Yeah, I think a massive source of alpha in this market is not being afraid to do hard things. So we went all in and realized in order to develop, in order to deliver on the developer experience we wanted to have in the very early days, we were like kind of to build our own file system to do container images really fast. So we started building that in the very early days. And that become like over the next few years still is like a very massive source of alpha. It's been a longer journey for sure. Like we've had to go much longer route like building our own file system, building our own container run time, our own schedule. Like we don't use Kubernetes and Docker. Like we use like very little off the shelf software. We use device, which is a fantastic Google product for isolation. Recent started using also KVM for certain tasks. But like decided pretty early on that like a lot of the stuff we're going to have to build ourselves. And now let's just do some really cool stuff. I think you know one of the best examples of it where that shows up is we can do call starts extremely fast. Turn out like the initial developer experience that we wanted like very fast feedback loop when you're launching stuff in the cloud. That is also very helpful when you do building auto scaling applications that need to scale up and down very fast. So first of all, the loading of containers very fast. But over time we also launched a couple of really cool features around CPU snapshotting and GPU snapshotting. So we can like very fast load data from the disk or from network straight into the GPU and fire up like even these like large models in a few seconds, which is really cool. And over the last six months we built this incredibly complex system that basically uses RDMA over peer-to-peer between the nodes to copy data very fast. So I think a lot of this stuff is like kind of accumulating and really starting to pay off. Like we built some crazy stuff. And now that's like we're really kind of reaping the dividend dividends of those sort of investments that we made for many, many years was a longer route. But now I think we have some incredible unique technology that's very hard to replicate. And then kind of moving up to inference, what do you view as you guys have made tons of investments there as well and optimizations? What do you view as the unsolved problems in the inference world or in the inference stack? I think we're still pretty early with inference. I think a lot of stuff is like one of those very empirical sciences, like I think people are trying different things and seeing what works. I mean, over time people have kind of figured out a lot of stuff. Take something like, decide you get a pre-filled, right? Like it's a very non-obvious idea. Like you sort of do the pre-fill on a set of nodes and then you copy the data over RDMA and do the decode and kind of swap. There's some very non-obvious things. But like we've invested a lot in those types of things, the KVCaching, there's a lot of complexity around KVCaching. Of course, like running more efficient kernels. That's something people do a lot. There's a lot of these things that we have a team of, I think, 20, 30 people right now who are extremely focused on just like LM performance optimization. And it takes a lot of work. You have to stay up with research. You have to read papers. You have to look at all the software coming out all the time and try and see what works with this and work. It's still kind of an empirical science. Over time I think, you know, patterns will emerge. I think also the models themselves will change very quickly. Like now we have like mixture of experts and more complex things. Over time, like who knows? Like maybe we'll swap out the transformers for something else. And you know, and that would change how like KVCaching works. Like there's all these things like that could, you know, change overnight. If someone figures out like a very different paradigm for all these things. And maybe they won't, maybe they won't. I don't know. There's also obviously a lot of stuff around quantization and other things. So yeah, it's a wild world out there. But yeah, we're playing around with all these things. >> But it feels that there's so many folks that are working on, you know, different parts of improving inference. And obviously what's, I think, been really ashamed the last two, three years is a lot of those optimizations and improvements have been in open source, right? And like, you know, for the broader community, maybe shared via research. And it's like a CEO, how do you think about like the level of what the kinds of problems you want your team to go solve versus, you know, it's like, hey, we're going to benefit from just the entire community like pushing this space forward. >> Hi, I'm Ann Maris, and I'm the producer of unsupervised learning. Thank you so much for listening to the show. And if you've enjoyed the conversation, please subscribe. Subscribing helps us grow, which helps us continue to lock down some amazing guests. Thank you so much for your listenership and see you next episode. >> It's like a CEO, how do you think about like the level of what the kinds of problems you want your team to go solve versus, you know, it's like, hey, we're going to benefit from just the entire community like pushing this space forward. >> We actually love open source, because I think a lot of our alpha is in the proprietary side, in the infrastructure layer, like all this like cool stuff around like auto scaling, for instance. So I think on like other inference providers, we've actually leaned very heavily into open source. We work very closely with both the VLM team and the SGLM team. We contribute very actively to both those repos and do a lot of optimizations with them. And so I think this may be a little bit of a different approach than maybe other inference providers. Of course, there's going to be, you know, certain things that, you know, we keep internal, but almost everything we push to open source. And I think it's, you know, it's because like, you have so much alpha in the infrastructure anyway. Like we might as well just, you know, work with the amazing open source engines that are out there that maybe were, you know, they weren't stated the art like, you know, a few years ago, but now they are and that's an incredible thing that like would love to leverage frankly. >> You know, you were talking about heterogeneous compute and the different places where you're getting compute from and also how the, you know, kind of inference stack is changing, how important are, you know, the long tail of chips to you guys, the non-in-video chips and things like that as you think about your supply or kind of what the, what your serving stack looks like over time. >> I mean, I think Nvidia is still like the king here like they have the best chips. There's way more demand for Nvidia versus anything else. Like, you know, we're, you know, curious about some of the other chips like, you know, we made some experiments like taking a look at it. But it's been a video definitely has the best chips. But, you know, like looking at AMD, Tranium, TPUs, et cetera, like starting to be, like pretty good, so we're always curious about that. And then, and of course, there's a lot of like older chips too. And I think the challenge has been like, for a lot of the LMs, like you kind of constantly pushing, like, you know, how many parameters can you fit? So, so typically the latest and greatest LMs, like they will always require, you know, a full node, maybe even the future even multiple nodes who knows. So like eight, B 300s, like they will require the latest chip and like eight of them. 'Cause that's how many you can fit on a node. But there's plenty of models outside of that. Like older LMs, smaller ones that have, you know, they're optimized for performance might be fewer parameters. And then outside of language, there's a lot of stuff around like voice and, you know, audio transcription or speech synthesis or like, you know, pharma, like bio stuff. Like a lot of that stuff actually, they don't require like the latest and greatest GPUs. Like they actually get more cost-efficient servering on hoppers or even emperors or like, you know, different types of like L40X or RTX Pro 6,000 stuff like that. So we actually, I mean, we are ramping up like very, very aggressively on black walls, primarily GBs now. But we see, I actually like plenty of demand also in the older chips. Which I think is quite exciting. There's just so much stuff going on. People forget about the hoppers, but like they, they are our most common type right now, still at Modal, like H100s. We run some very like large production use cases based on hoppers. - How efficient like is model to chip matching today? Like, you know, in the industry at large, do you feel like most people are using the like, optimal, you know, chip or, you know, from price perspective for what they're working or if there's still a lot of room for improvement there. - There's a little bit of room to improve. - I mean, one of the things that we want to do over time is like kind of automate that for people. I think in practice, it's like less of a problem than it might seem like, 'cause often like, you're kind of memory constrained. So like knowing how much memory you need for a model, like you basically pick the cheapest GPU that fits the model. That's usually like the heuristic that works in almost all cases. So if you have, you know, if your model is 80 gigs, like you can either fit it on a hopper or an older A100 80 gig. And I guess in that case, it depends a little bit on the flops. But in many cases, it's like whatever GPU, like it's the smallest that fits your model. I think over time, like there's a bunch of stuff we've been thinking about that we're like basically, you just deploy something on a model and then we manage like different ships and maybe automatically give you gist counts because like we can, you know, the more options we can have to like run it on a lot of different ships like that. Let's just aggregate capacity in a better way and run things at higher utilization. 'Cause then we can switch between different GPU types as, you know, there's like underutilization on those. So that's something actually we're quite interested in doing over the next six months. It's like giving people an option to deploy things in a way where like we automatically switch types underneath in a way that's still consistent with your dollar per request or dollar per token or whatever. I think there's been incredible cool stuff you can do there. - Super interesting. And I mean, one thing I imagine is so interesting in your role too is like you're obviously predicting for like what your compute fleet should be. obviously you see what the workloads are today and you probably have some sense of you know what models from wall and what chips, then you're also trying to predict, like, hey, a year or two from now, what kind of workloads will we have? I think there's lots of different debates about whether most tokens running will be really large models, or actually, we saturate in most tasks, and these models will get smaller and smaller. As you think about the evolution of your fleet in the next year or two, what's your perspective on where you'll see relatively more different kinds of workloads and less? I think there's been basically a trend of larger and larger models over the last three, four years. You're better than that. I definitely feel like that trend is going to continue. We're going to need, like, if B300s had more memory, I'm sure a Kimi K3 would be bigger. People are kind of like training the models in a way that they fit on these chips. I think on the LM side, in a way, kind of a crude type of world model, you're trying to encode as much information as you can in these LM's. You need very big, almost look-up tables, of facts, in a way, it's kind of a conception, I think about it, and that sort of pushes the memory requirements a lot. At some point, it might taper off, who knows, but for now, people will make use of every possible memory, the more memory, they'll just train more parameters, for sure. You obviously sit in, in some ways, I'm like the eye of the hurricane of a very exciting large and competitive market. I'm curious as you think about what ultimately will differentiate players in this space and win out. I think there's so many true believers in the size of the overall inference market, and I think lots of folks are asking what actually wins out in the end of the day, and I know you have a good thesis on this, so I would love it for you to share with our listeners. I mean, I think a lot of the inference market will be quite rough, right? I think to sort of straight down the middle, serving tokens, you already see sort of open router basically commoditizing the underlying providers. I tend to think the real alpha or the real point of differentiation, there will be a lot of differentiation just in more efficient serving, but it is a little bit of a race to the bottom, and I think there's going to be a lot of opportunities to sort of stay ahead of competition. You're going to have to put a lot of efforts into that, but I think a more durable differentiation will definitely be in providing the entire platform of a lot of different things, which is I think something we are quite well positioned to do. We have the sandboxes, we have reinforcement learning, a lot of customers doing also normal fine tuning, a lot of customers obviously deploying other types of models. We have batch processing, I think my view in general is customers, they don't want to buy like single trick ponies, they want to buy like a multi-product set of tools, so that to me is like probably the biggest source of differentiation, it's like how do you provide that entire set of tools? How do you provide this like very general-purpose thing that lets engineers iterate very fast, deliver value, and then to sort of day to day, we're still going to have to go and do the knife-fighting of staying ahead of your competitors in terms of performance and use the latest kernels and the much more efficient KVCation, there's going to be a lot of that stuff for a long time, this is just so much money in it, but I'm kind of pointing out, I think that's sort of, that's like less durable as an advantage, I think, because over time, these tricks will sort of spread and people figured out how to copy them. The more durable thing I think is being that portfolio of a lot of stuff around the inference, not just the inference in itself, having all the models and a lot of stuff like reinforcement learning, maybe, or other things, the biggest model customers are on multiple products. Maybe can you touch on a few of the more interesting ones? Yeah, we have some incredibly cool customers, like Cognition is using us at large scale for both for inference and sandboxes to do a lot of reinforcement learning on model, these sandboxes for all the rollouts, we have customers like Ramp using us for background agents for the internal coding agent that they actually launched as a standalone product today, that they're selling, we have customers like Suno and Runway that use this regenerative video and audio, Dekigon is another customer that uses us for a lot of customer support stuff, voice and text, DoorDash uses us for a bunch of internal applications, mostly alarms, but all kinds of other stuff, we actually have a lot of customers in computational biotech using us for various types of drug discovery, using molecular simulations and sequence alignment, all kinds of stuff, a bit of a bunch of customers with robotics, 1X and a few other ones using us to train robot models, like doing a lot of training, but also batch processing of video and things like that. A lot of customers in the data generation space, so customers who are building environments and running benchmarks in the model, off of using sandboxes. One thing you kind of mentioned is portfolio products, and I think that's like one of the most interesting parts of the business you're building, and I imagine like honestly one of the hardest parts must be just prioritization of all the different things you could build. There's probably so many new things that come up of really exciting tools you can build for developers, and I guess I'm curious, obviously you kind of with sandboxes, you have incredible products around coding agents around reinforcement learning, how do you think about kind of building down that vector, there's also kind of like the fine tuning or all of the service stuff that some of your competitors seem like they prioritize, like how do you think about I guess, where to prioritize within that portfolio? I think it's an incredibly hard problem, I mean most startups, but especially multi-product companies, the sort of balance of like first of all when do you go multi-product and then where do you put your bets, we're 150 engineers to something like that today, and I struggle with it every day, I wish we could do that and whatever. But I kind of know that even if we had 1,000 engineers or 10,000 engineers, there's a lot to build ours, I wish we could build a rocket and go to the moon and whatever, it just never goes the way, so you kind of have to be ruthless with prioritization and really think about what's the highest ROI, and where do you draw the line, and you have to be sometimes pretty aggressive also saying like actually I know we started on this project, I know we're really excited about it, but this other thing kind of came up, so we're gonna have to pause this for a second, we'll get back to it later, we've had to do that many times too, it sucks when it happens, but I really think that sort of prioritization and how do you allocate resources, I mean resource allocation is basically like number one thing that I think a CEO has to really be ruthless about it, the company, and so much value is being created, like allocating the work to the right resources. The modal roadmap in each board meeting is extensive and very daunting, and I'm always impressed by the amount of software that the team can ship and hardcore software while keeping the system running. How much is, you guys were building the company pre-chat GPT and then post-chat GPT moment, how much has that changed, maybe on the product side, what you guys were doing, but then also when you were thinking about building and the velocity that you can ship software at? Yeah, I mean, we were always benefited from being quite well positioned at a time where AI kind of blew up and that's been incredible to see, and we're partnering so many different AI applications today that I had no idea it would exist, like four years ago, five years ago, and we started models, so that's been kind of incredible. Like you said, software in itself is also how we build software has changed a lot. People write, I think probably at this point, 95% of the lines of code that gets merged as model is probably written by agents to some extent. Infrastructure sell, I feel like realize a lot on like agents are like kind of good at it. I've actually been on a tier like the last few days, like building or the last few months actually, like writing like an insane amount of code using agents, and like, it's kind of incredible like what they can build, but like you kind of still need you to like supervise them quite a lot. I'm like, you know, infrastructure has a lot of tail risks around the reliability and making sure that things scale and like stuff like that. So at least for now, I think software engineers are pretty safe, and I think we still benefit from having incredibly good engineers that can supervise these agents, and I think probably for the next few years. Another thing I think has been kind of interesting to see is like actually like think the adoption pattern is actually the opposite of what I would have expected, like I thought it would be the junior people that would adopt agents the most, I've actually seen almost the other way around. It's like the very senior people, because for them it's like giving them like 10 interns, like they've been kind of constrained by like typing speeds for a long time, but they have all these ideas in their heads, and I think using agents, now they can get it like they can get it out like much faster. They're really good at building a test suite, like you know, give the agents like here's an extensive test suite. It implements all of you know, the post-extended for file system, now go write a new file system, and it just goes on like this writes a file system and takes like, you know, three days to do it. So we've seen, you know, that some of the very senior engineers like go like complete agent maxing, and I mean, it's kind of incredible like what you can do with agents, you can move very fast with them. At the end of the day, it seems like humans are still safe as software engineers for the next few years. I don't know. We'll see. How have you found the velocity of shipping software has changed what you guys need to do with code review, or CI/CD, or deployment, have those pieces of your development lifecycle changed at all? I think a little bit. I think we're coming up against some of those constraints, like we've had to spend a lot more time on CI/CD, like I always like, you know, even longer for agents. I always like, you know, I have to see through for many years, like I was always like obsessed with CI/CD, as like, you know, very, very important to figure out. So like, in that sense, like, I don't know if it's like changed like my view as like, you know, like, you really need to have like automated deployments, automated tasks like all these things. Now, by the way, it's are pretty good at writing tests too that weren't like a year ago. So I don't know if it like we've necessarily like yet like run up against the foundational limits but I mean anthropic like you know a couple days ago they announced they're moving off get I believe or it is GitHub. We might we might run up against those limits at some point when we have you know 10x the amount of commits every day. What is interesting to think about kind of the you know how different the world is with these really powerful code agents and obviously I think you know part of of what I believe your your co-founders talked about is is kind of the shift even in who the consumer of modal is over time right from a from a developer to you know to agents themselves right and I'm curious like how you think about what that means for modal as a business and some of the decisions you might make differently as you know more and more of folks interacting with modal are either doing a VM agent or purely end to end an agent. It's actually changed like a lot less than than you then I think I would have expected it turns out agents I mean like in the early days like we built modal in order to give a engineer as a better developer experience and the ability to move faster. It turns out like agents kind of want the same thing like they want fast feedback loop they want like good error messages good documentation a lot of the stuff like that that engineers want to. So so most of the stuff is actually quite similar. I do think it's like certain things a little bit harder like one of the things like we found for instance like deprecations is much more annoying because agents are trained off in like a lot of old code that exists online. So removing features is much harder for us like we can't just rely on like you know remove it from the documentation and wait three months because like it's in the training data people will use you know lm's to generate modal code and that will be like you know using deprecated features and that's like quite hard to change so we've had to change our deprecation we know quite for quite a lot. Another thing we we've started doing quite a lot is also sending up a lot of automated pipelines to basically have like lm's continuously write modal code and making sure it compiles and or not compiles but it builds and runs trying to just like catch like things that agents struggle with and then maybe another thing we spent a lot of time on is more like kind of like agent optimization stuff which is actually more like similar like searching and optimization like if you you you can go and ask you know cloud code like something like hey like I want to do something with inference I want to deploy this model and not say anything about modal and just like see like which tool does it recommend and and and that obviously we wanted to recommend modal is as much as possible so that's something we track internally make sure that you know over time like and actually I think that's like something that that will change software engineering a little bit maybe almost like to do worse like I like I feel like in a way like because of the training data and the training loops like there's more like you know like intimacy power in a way like there's more staying power in software today like if you get inside the training data and you get the models to recommend you know it's a certain framework I've heard people say like oh I'm just going to use React for everything because agents are really good at it I'm not going to try these like new fancy frontend frameworks right like I think there's like something similar with like a lot of the frameworks today is like that staying power is actually very very valuable now so so we obviously want to make sure that agents recommend modal as much as possible because that creates a little bit of flywheel I mean one thing I'm struck by is you obviously had this like very prescient bed in 2023 on on sandboxes right it sounds like you guys had kind of this instinct of like hey this is going to be something that's required for for coding agents and then you know it took a few years for to really take off and then you know it became this really important primitive I'm curious is there anything else that you guys are working on today that you feel like you're a little early on but like you know you're you're equally as excited and how much of your product portfolio is like those kind of bets versus like okay we know this is a hair on fire like here and now problem Jacob are you are you fishing for the next thing we should be investing in 100% 100% I'm nothing if not shameless I I feel like sandboxes they did create a little bit of like a high bar for us like everyone's like what's the next thing then you know what's the secret sauce you're working on and like what's cooking in the lab you know and and I wish I had like an easy answer like yeah like a hundred percent confidence you know this thing would you have answered sandboxes in 23 or is it just like part of a bunch of things you were doing no because what sandboxes represented back then was like a little bit of like an asymmetric bat like sandboxes was not like a very hard thing for us to build back then like to getting out like a prototype of sandboxes took like up shut like a few days and then I spent a little bit of time you know like improving SDK and some other people like started building and then we started seeing some customers we wrote it and then in 2025 when it started exploding then we have to rewrite the system like five times just to handle like more and more scale and make it faster and and and more reliable and things like that so but but the initial investment was actually pretty pretty basic so in that sense like it was a little bit like a sort of asymmetric bat like a kind of a low downside high upside bat I don't know if there's like anything similar today that we're working on I am I mean like just just kind of called out like a couple of like you know things that I'm quite bullish on and these probably represent like way larger investments like you know larger risks and you know in terms of like that size a little latency like I think you know there's going to be a lot of voice to voice a lot of real-time video type stuff that requires us to rethink a lot of how we manage to compute like globally how do we get down the number of hops like how do we scale up and down in different regions like almost like building a CDN frame friends I'm pretty excited about that I think training is actually kind of an underrated thing I think there's a lot of opportunity and training like mid-sized models and reinforcement learning that's been a big bet for the next for the last couple of years like getting all the RMA infinity band type stuff working but I'll say I think a lot of VM like stuff is quite interesting like I think you know supporting full-blown operating systems like Mac and Windows and Android and iPhone could be kind of cool I think there's a lot of upside there so I have a lot of those ideas in my head and we're like building a lot of those things I don't know if there's anything that represents the same like sort of explosive potentials like sandboxes I mean maybe you like if we have this podcast again in like three years like we'll kind of look back and we're gonna make it recurring at this point so we'll look forward to revisiting that you know I guess like obviously you know sandboxes have become this fundamental primitive for you know coding agents and it feels like that that brings you into this world where there's like you know I imagine you look at the current state of coding agents you know like God there's so many more tools that would be helpful for these agents to make them more effective for people that use them how do you think about like you know whether you further build into that world or like so this has the things that are interesting for you there we do think about it but we I think we think about it more from an infrastructure perspective I think it's very easy to kind of get lost in like you know whatever is like the latest today when I look at like the architecture of like agent harnesses and like how we're building agents and stuff like that I still kind of feel like we're so early and these architectures like you're evolving so fast and like what our job to some extent is like to try to figure out is like what are like to sort of core building blocks that we can put out today that like well last you know like people we need them for the next five years and that to me is like less obvious like we are interested in like sort of for agent orchestration as a category I think there's a lot of interesting stuff there we're interested in like you know e-vals and model gateways and all kinds of other stuff around like you know like the whole like you know life cycle of learning from agents and and iterating on out agents but but I think it's like less clear exactly like how the architecture my views like it will be more cloud-centric and it will be probably more you know like like durable objects like kind of workflow engine type driven and we have some ideas of like you know stuff to build in that space but but I find it like harder to like you know lay out exactly what agentic development will look like in five years partly because like I also think like to some extent like all software will like basically be agentic like I kind of hate and hate the word agentic in certain ways because I feel like it's like okay so it's using AI to make decisions like well shouldn't all software do that so like I almost like feel like if you look at it on a five year horizon like I think like software like all software will become agentic and that what does software look like then like I don't know but I you know I I hope we can build a lot of the building blocks support all kinds of aspect of like you know in general just like how software uses AI yeah I mean it's still like one of the hardest parts of billion in for a company right of you know across the board for many people you know building AI and for companies of you know what what actually persists right versus is a pattern you know is it's like rag from two years ago or like you know whatever it is rag is a good example exactly like if we had made a bet on like rag like two years ago like it would have been kind of a bad thing to do so yeah but I would also so it's hard but I would also say like one of the benefits of building infrastructure is like you can take a longer view and you don't have to like listen to all the noise yeah I mean you you have to listen to the noise but like you kind of have to like ignore a lot of it and trying to figure out okay what are the patterns here you kind of alluded to some of these other you know other workloads that like low latency workloads like voice to voice in real-time video I think there's a lot of chatter in the air about like multimodal inference robotics inference these other categories I believe that you're just riff a little bit on like to what extent are those problems you know similar to the set of problems the inference markets had before to what extent are they like different and new I think they're all quite similar but all I mean they're like snowflakes unique in their own way robotics is going to be a lot of it on on device of course so that's a big difference and and you know I don't sort of envision like we're going to support you know most of what robotics looks like because a lot of is going to be on the device system cells but I think we're going to be able to support a lot of the testing and development and training of these of robotics and a lot of the simulation environments so that's something I'm very excited about on the multimodal stuff I mean there's audio and music and image and stuff like that like people don't talk about as much but that keeps growing like crazy too I feel like L-Lamps have kind of dominated the airspace in the last few years but I'm just incredible. You know excited about all the different applications like why are like voice assistance so bad in 2026 like I don't get it Like they're just Horrible like I have a Google home and like every question. It just bungled. It doesn't understand what I'm talking like you know So I just go to chat GPT like why can't you just connect all of it like N2M them train like a system So that that can you know do like speech-to-speech I I think there's a lot of we're gonna see a lot of sort of Incredibly cool applications in that space. I always think about you know you look back like you know four years ago three years ago Whatever, you know, could you be a guest that like things like instinct or sudo or like ignition or like none of those things like are very easy to guess Like you know like three years ago So I kind of feel like also like if you look down the road like three years like Most people don't have an idea of like what the world is gonna look like in three years It's so hard to know but like what I do know is like kind of judging by the past three years It's gonna be like this can be some really cool applications. I just don't know what it's going to be but they're gonna be very cool And hopefully they'll use modal. I was looking back at an email you sent me With the modal series a pitch deck and this is what you said This was after we had signed a term sheet By the way, we never actually reviewed a modal deck But here's something we made earlier that I just updated with the latest numbers could be good review at some point When you sent this you are roughly two million in run rate You now, you know, you've crossed a billion run rate, which is insane and in three years What's been the biggest kind of change to supporting that kind of scale in just you know a couple of years? So first of all, I actually realized like the other day I said on another podcast that I've never made a pitch deck and then I was actually thinking about it I was like maybe I did make a pitch deck After this year's a because we had a couple of angels that came in and they were asking about it So I was like I'll find I guess I'll put something together So I I realized I did lie like I did make a pitch deck, but it was not but you never showed it to us So I don't know that it counts. Yeah, so I don't know if it counts like I never like sent a pitch deck to an investor I sent it to fiat angels actually But that and that's the only pitch deck I've ever made actually But in terms of the scale. Yeah, I mean, we're grown, you know, 500x I guess it turns to revenue something like that since then over the last three years Um, so a lot of girls um, I think I think you don't weigh like I don't know. Maybe I'm like just you know my my experience is mostly been at higher-pest scale startups like I was a spot at five for seven years Uh and some other places so so for me like this like fast growth is kind of all I've seen like you know that there's like You know every every six months you have to rewrite all the systems You have to kind of rethink everything you're doing you have to kind of change all the processes like you always feel like you're behind Like I don't know I kind of like the mess. I guess we would love to you know since since we have your your mind on the podcast But love to kind of zoom out and maybe talk a little bit about the broader and for market and some things that have been in the news of late um You know maybe maybe most top of mind these last few weeks if there's been tons of discourse about neo clouds Right, they like you know how the debt's trading they traded down They're kind of back up a little bit now and I'm curious like What do you think happens to this landscape over time? I think the neo clouds are like Fine business models like I think people have been like confusing it like a little bit. They're like Wow, there's so much demand like revenue is going to grow like so fast and then you know There's a little bit of sort of a almost like a VC mindset of like oh like thinking like revenue multiples Uh, but I think all of these like these neo clouds like growth is actually quite expensive for them Because they have to go and raise a lot of debt And so I don't know if necessarily like you know just because like they grow a lot like you know Like it's very like capital if inefficient like these companies I at at their core like you know Financing vehicles like they're like they're doing leasing like of hard assets And that's you know, and that's like an inherently you know harder to scale the capital now that that market is kind of saying Well, we've deployed a lot of capital into neo clouds like maybe Maybe hold on for a second like you know, we don't have that much more capital to lend and so the interest is starting to go up um But also by the way like these companies have incredibly high leverage so of course they're going to have more volatility Like the the residual equity values going to go up and down much faster on the stock market with GPU prices and stuff like that Um, but they're like fine business models like you know, they kind of locked in customer contracts Uh and raised that against it and like you know now they're paying off that debt and and in many cases turns out the residual value has been hired and people expected Lifetime of GPU has been longer than people expected Uh, so they can resell once they paid it off like at a higher price than people thought so like they're gonna make a bunch of money from that Like it takes a few years to get there Uh, but I think in that sense like no class will do well. They're not particularly differentiated So I you know long term. I think they're gonna start Yeah, and you work with a lot of them like is is there any differentiation you see It's like a little bit, but like at its core. It's like you know like relatively on differentiated space Uh, so I I mean my view is like, you know, I mean Elon was like tweeting the other day like all of your clothes are gonna die like I I think that's kind of bullshit. I do think Elon had a point when like basically like when he implicitly was saying like SpaceX has a much lower cost of capital I don't think there's a coincidence that if you look at the like the the three successful hyperscalers Amazon azure Or AWS azure and gcp that they're all attached to companies that had very solid cash flow and the world public companies like that have very low cost of capital Uh, because that adds core is is makes the life a lot easier if you want to build something as a very capital a fish capital intensive So I think in general like that is something that you know class will struggle with like they have you know They're gonna have a high cost of capital and and it's gonna be hard for them to grow fast because of that um So so we'll see about the market, but they're like decent but this is models like I think there might be an opportunity to buy the dip I don't know you had an interesting tweet that was a lot of the the neo labs Could become neo clouds, you know, if their their model doesn't work out, but they locked in these contracts Um, I definitely could see the the world going that way uh, if you know if you have valuable compute that continues to appreciate because there's so much demand But yeah, I mean mistral kind of did that right and you know like it seems to be fine I think a lot of people are making these like very large buying decisions and they kind of rationalize it like they're like Well, if you don't need the GPUs, we can just resell them right like that's almost like an informant insurance And then you have to you know assume with like all these neo lab, you know There's so many neo labs out there that have each race one billion that they always write race exactly one billion dollars for some reason I don't know why uh, I guess they like that number uh, and then buying GPUs very precise financial modeling that points. Yeah Yeah, yeah Yeah, so some of them I think we'll probably work out really well and you know But I imagine in the grants, you know like some of them will not work out and if they don't work out Well, now they're sitting on a ton of GPUs like you know Let's resell those GPUs right like especially if there's ever like a scenario where like VC funding pulls back Because I think what's what's kind of a scary thing about some of these commitments is like, you know Effectively like what what happens today when you make a GPU commitment is you only have to pay about 30% up front Sometimes a little bit more sometimes with it less But what that means is like you can raise a billion dollars You can go out and and get GPUs for three billion dollars and only pay a billion dollars up front and maybe not exact math But the problem is like, you know, in a year now you're on the hook for another few hundred million and like every month Now you have to pay So so I I think that creates like a little bit of like implicit leverage in a lot of these companies and and and it and a dependence on subsequent fund raising And you know, that works as long as they can go and raise VC dollars to fund the GPU bills But at some point it might not work and in that case, I think you're going to see it a little bit more of a Secondary market where some of these neolabs might you know pivot to doing a little bit more sort of neocloud style We have a bunch of GPUs or or just resell it back to the cloud or I'm very bullish on AI as a whole, but you know, I always like look for these like sort of structural like risks So overall, I think there's going to be like incredible application incredible demand But like you know is 100% of the neolabs going to succeed and you know be you know It's really a dollar company like probably not you know I feel like on the a I emphasize one big category that this got a lot of buzzes like this Arles a service market and kind of a bunch of folks, you know, almost forward deploying and like helping companies build their own models And you know, you have had folks like Satya saying hey every company is going to have their own model Like do you subscribe to that and like what do you kind of think of this of that overall market now? I think some companies will have their own models I think that like weighted by dollar revenue Like I think most models will probably be post-trained right like But but like weighted by company like in terms of like numbers of companies like most companies will not have That the need to like post do their own post-training right but like the large ones but like cognition is training their own code models and And and cursor date and and all these companies are and like though those are the ones also served a lot of tokens So I think when you look at like reinforcement learning and post-training like certain types of you know parts of it is very attractive Right like you can work with companies and and and serve you know better models and and do it at lower cost and higher throughput and all of this stuff Some of it looks like it's kind of like just a service business So I think a lot of the sort of tension over the next few years when you look at post-training is going to be Can you turn this into product why can you sell that product to these companies? This is something we think a lot about is like how do we create this like amazing product? You can just kind of sell And not turn it into just like a service business And then I you know I guess last one before I move to the quickfire round would just be like you know I think there's this this question about you know How many tools live under like a single infer provider in the eye world versus you know our separate categories I'm always struck when we talk to you about like just how much you want to go build and and all these kind of building blocks I'm wondering if there's categories of AI and for you know that you think will be unique and distinct from the stuff of your building that you're optimistic on. And then we'll kind of like live as still large standalone opportunities. - Yeah, I mean, we're not gonna go into model training, I think like, I think Type Safe with Jav is gonna be, this can probably gonna be a lot of these like custom models. I'm sure you know the verticals to, you know, other modalities like audio and video and music and like obviously on the application side there's gonna be a lot of incredible companies. And then there's probably gonna be a lot of companies like focusing more on like softer development and stuff like that. And, you know, I don't know, like my general view on these things is like there's always like more boxes on the market map than you expect. But I also do think that overall every company that is successful in the end to some extent has to become a multi product company too. So I don't know if that's attention or not. Like, I think, you know, we obviously wanna find like a cluster of like Ray and Jason things and do all of it in a way that like, you know, maximizes the revenue and the shareholder value. But I, you know, by no means, you know, are we gonna try to expand into everything? I think people kind of make this like false assumption also like, you know, you look at like a topic or open AI and they're like, why don't they move up to stock and launch drone law firm and launch drone, whatever. Like, they're very good at like serving these models and doing that at scale. That's how they achieve scale, right? Like, and they're very good at doing that. Like, I don't think it makes sense for every company to do everything. And so it's similar to for us, like, we're not gonna do everything, but we're gonna do a very large cluster of adjacent things and do it exceptionally well. And I think there's gonna be a lot of room in the market to create, you know, a bunch of, you know, very large, you know, a hundred billion or trillion dollar opportunities in the market. - Well, and on our quick fire round where we stuff, probably a similar set of questions what we asked you three years ago, maybe a little bit of an evolution. To start, maybe I'd love just one thing you've changed your mind on an AI in the last year. - Well, the opportunity size of sandboxes has obviously been kind of crazy, right? Like, I didn't think it would be anywhere close to, you know, I mean-- - I thought you knew that in '23 and that's why you made the big bet. - Yeah, so that's been kind of incredible. - Coding agents have changed a lot of what it means to be an engineer. How, if at all, has it changed how you think about hiring? - Not much. I think we still get to need humans for a long time. I think the sort of demand for humans is just going up the more productive they get. I think, you know, we've seen this for the last 30, 40, 50 years of software engineering, you know, whether it's compilers or high-level programming languages or CICD, like every time you make engineers more productive, like the demand just goes up. And so I think we're going to need a lot of different, a lot more engineers. I do think maybe on this short term, it's going to be like a little harder for like new grads, like, you know, except for that the very high side of the spectrum, but in the long run, I'm very bullish on, you know, computer science. Actually, ironically, I didn't study computer science, but I'm very bullish on studying computer science in general, like if my kids wanted to do it, like I would definitely encourage them. - What's the biggest mistake you've made in building mobile so far? - Ooh, I don't know. I think one mistake I made in the past is that, I think sometimes like, you kind of want to sequentially do things, you're like, well, first when you do do this, this is the blocker. We'll get to this other thing later, 'cause this other thing, you know, is blocked on this. But like in a world where like everything is moving fast, like one of the things I've learned is like, sometimes you kind of need to do both at the same time. Like you need to assume that like, over doing this thing to unblock it, you might as well do this other thing at the same time, just in the interest of moving fasts. That's something I, I'm not sure like, not mega aggressive, which is kind of ironic, 'cause like we were on 15X in the last 12 months. I sort of had to retrain my brain to like be a little bit more like aggressive and saying, like this is a massive market opportunity. We need to be even more in like hiring, deploying capital, like whatever it is. Like massive is like paralyzing everything we can't. Sweden has a ton of interesting startups that have emerged. What are your thoughts on why that is? And has the Swedish house mafia been replaced by the Swedish tech mafia? - Do you know that like one of the Swedish house mafia members was my classmate? That's a fun two of you. - Really? - I could not know that. - Yeah, back in the early 90s, grown up in Sweden. When I was eight, he told me my bike was ugly. I was very upset. Yeah, I mean, we've had a lot of tech in Sweden, Spotify, Clarina, you know, a bunch of gaming studios. More recently obviously lovable and the Gora, stuff like that. I think my thing about like a lot of this stuff is like, success kind of just creates success. Like, you know, a lot of like what makes people going to tech is like, they know about some cousin's friend who made a lot of money in tech or something like that. So a lot of it is I think it's just like a little bit of a flywheel effect. I do think that Sweden has a European countries that has always been pretty entrepreneurial and commercial. So I do think it's a little bit of that. Like we've always been a little sort of maybe going back to the Vikings, like kind of let's go conquer the world. In the past, maybe it's been through music and furniture and stuff like that, but now increasingly with tech. So, you know, there's that. Like I think there's sort of mentality in Sweden. Like you can actually, Stockholm is like funny 'cause like everyone in Stockholm thinks they're like, there's like mega global city. That's like part of the world. In a way where they actually think they aren't, but like I think the fact that they think they're so global is like kind of good. Like they have this like vision that like everyone cares about Stockholm, but actually kind of like that. Because it means like they're out there like building, you know, things and staying from day one. Like we're going to conquer the world. Like we're going to do the startup is going to be awesome. And I like that. Like they sort of make a laminia. - And how this vision of like your elementary school reunion just what a wild, a wild group that would be. That's quite the cast and what I imagine was not a particularly large class. - Yeah, yeah, it's got a random effect. - I guess a shameless question. Do you think GP rental prices 12 months from now will be higher or lower? - I think higher. You know, it's funny. Like I actually changed my mind on this. Like in the last 48 hours. (laughs) And I'll tell you why. I read the Anthropic S1 and I actually had an agent summarize it for me. But it was kind of incredible how profitable that company is. And to the degree that I hadn't anticipated. I thought that like in my mind, like 48 hours ago, my argument was like, oh, Anthropic app. Like they have really good gross margins, but they're spending so much money on training that like as soon as they go public, they're going to have to like slow down and how they grow training and post training and they're going to, you know, maybe grow inference as much, but not the other stuff as much. But then like I realized like they're basically profitable companies. So like they have like zero incentive to slow down. Like capacity, capacity acquisition. And I imagine there's a similarly dynamic going on in OpenAI. And so an OpenAI and Anthropic has obviously been like the biggest two buyers of capacity out there. And so the fact that like, I don't think they're going to slow down buying, you know, because of, you know, clearly like financially they're doing very well. I think it will mean that like there's going to be a lot more demand. And I think for that reason, prices will probably go up in the next 12 months. - I've heard you say that you've always thought modal, you've thought of modal as your 20 year project, your five years in. What do years 10 through 20 look like? - Wow. I mean, hopefully I'll hire a bunch of people that can do a lot of those sort of, you know, I'm two in the weeds of like everything right now. So I, but I don't know, like I just like love this stuff. Like I love to keep working on this. Like I think, you know, you're 10 to 15. We're going to look more like a conglomerate. We're going to have a lot of different products. We're going to have probably some like crazy bets going to moon and Mars or whatever. But I still hope to be quite involved in this sort of the high level like resource allocation. And what are the things we're betting on? And I'll probably be less involved in sort of day-to-day, like, you know, let's get a contract from this GPU vendor like whatever. But like, I think that like sort of like high level like resource allocation and stuff like that, to me is, that's the most fun and hope to stay involved in that for another 20 years, hopefully maybe longer. - Yeah, I mean, if we keep current pacing at least five more on supervised learning episodes in that time frame, right? - Yeah, exactly. Well, Eric, thank you so much for joining us. We really appreciate it. We'll give you kind of the last call to action for the audience. If there's where can they go to learn more about modal or anything that you would direct them to? - Well, it should obviously go to modal.com. You should also run pip installed modal in Terminal and play around with it. You know, there's a lot of people maybe listening to the show who are building startups or thinking about building startups and, you know, we'd love to see all the cool stuff they're gonna build and, you know, hopefully add modal. We can help you with those startups. So let me know, I feel really shout by the way. Like, I always love talking to customers. But I feel free to play around with modal and looking forward to see what everyone's gonna build. - I'm Jacob Efron and this has been unsupervised learning. A podcast where I get to talk to the smartest people in AI and ask them tons of questions about what's happening with models and what it means for businesses in the world. As I hope is clear, I have a ton of fun doing this. It's a night some weekends project in addition to my day job as an investor at Red Point. But our ability to get these incredible guests on really comes from folks like you subscribing to the podcast, sharing it with friends. It's really what ultimately makes this whole thing work. And so please consider doing that and thank you so much for your support and listening. We'll see you next episode.