Why AI Compute Comes in Rectangles — Erik Bernhardsson, Modal CEO

[MUSIC] >> Thanks for coming on, Eric Fernhardtson from Model. Could you explain the business a little bit and maybe how you got here? >> Yeah, so we build, the easiest way to put it is we build AI infrastructure. So we offer a computer platform and a storage platform that enables a lot of customers like Decagon, and Suno, and Runway, and Cognition, and many others to essentially build AI applications. So what that means in practice is a lot of inference, a lot of sandboxes, sometimes training and fine tuning, sometimes batch shops, and other things. So we have basically built a software SDK that enables customers to move really fast and build this. We manage all the capacity underneath, make it easier to deploy and scale and run these things in production. >> So you were on late in space like two years ago, a lot has happened since then. Like how brick were you back then, like whatever numbers you can share, and comparatively, yeah, how big are you now? >> I actually don't remember where we were, but it's probably at least like 50X. It's sort of like revenue and users and users. >> Yeah, yeah. >> I use it actually for like a side project that I have where-- >> Nice. >> Yeah, it's like, yeah, it turned like 2D images into 3D models. So yeah, just hosting like-- >> That's awesome. >> Models here. >> What do you think, any feedback? >> I mean, I love it. Nice. >> Good. >> I mean, yeah, that's why I use it over like any competitors or anything. >> Good. >> So before you started, modal, you were at Spotify and also like better.com, I think. >> That's right, yeah. >> Yeah. What were you doing at Spotify? >> I actually do a lot of different things at Spotify, but my main contribution was I built a music recommendation system, the first version. I joined after school in 2008, 2009 maybe, and spent seven years at Spotify, and most of that time was building basically from scratch a music recommendation system and built up a small team and we launched a bunch of features like related artists and discovered weekly and a bunch of other stuff. So a lot of metrics factorization, a lot of compute-intensive stuff, and arguably like a lot of what sort of later became the genesis of modal was just building AI applications and realizing the infrastructure, it's fucking sucks, it's got to be a better way to do that. So a lot of the rationale for modal was kind of building a tool I was wanted half throughout my years at Spotify. So modal was more of an offshoot of the stuff you learned at Spotify, not much from better.com. >> Better.com was a little bit of a side tour, it was kind of a roller coaster, a long story about the company, talk about it separately, but I was a CTO for five years of better and grew to tech team from I think, you know, two to three, four hundred. So I learned a lot about management and building teams and hiring and also learned a lot about developer productivity, thought a lot about tools and things like that, did a lot more like full stack stuff, but yeah, it kind of came back to my, was close to my heart which is infrastructure and AI and machine learning after that. >> Well, it was better.com doing again. >> Mortgageous. >> Okay. >> Yeah. >> How did you get into that? I mean, we don't have to spend too much time on that. >> Yeah, no, it's fine. Like this guy, Zach Frankel, who I didn't know at that time ended up recruiting me to better as a CTO and the company was 10 people and, you know, for whatever reason, you know, the founder of each all, very interesting guy, very, very smart, a little crazy in mostly positive ways. We got very well along. So we, yeah, we started building it in 2015 and I spent five years growing that company and then, you know, eventually turned out that doing work is pretty hard and, you know, interest rates went up very quickly after the pandemic and that caused a lot of issues at the company. But by then, I had already left. >> That was like 2020, 2021 when you left? >> Yeah. >> So, okay, you left before you had the idea for model or you left and then you started it. >> It's hard to say. I had a lot of random ideas when I left better and I actually wasn't sure if I wanted to start a company. I was kind of just, you know, during the pandemic, I sort of almost took like a sabbatical. I had two kids, I mean, I still have two kids, but I mean, at that time there were, I think, two and four and so I decided to hang out and took some time off and was kind of working like part-time, like hacking on certain things and wasn't sure if I should do some consulting and I started building stuff and I just got like super pumped about this idea that eventually became model. But actually, both, you know, model was like, I think the 30. I built that. I got, you know, so there were like a few attempts before then that there wasn't nowhere. But my model was like, the first idea was like, this is kind of cool. Like I should do this. This could be a really cool platform. >> Were the other ideas like also in a similar vein like developer tools? >> The first thing was like some sort of web scraping thing which went nowhere. But actually, you know, in hindsight, maybe it could have gotten somewhere. >> And was this before the chat GBT launch? >> Yeah, yeah, 2021. >> Oh, okay. >> So chat GBT was late, 2022, right? Then I started building, so maybe actually to rewind a little bit of context, like whenever I worked at Spotify, I released an open source framework, Call of Duty, which is like workflow orchestration. So it was the same category as Airflow and Prefect and Daxter, sort of dated categories. Does that make sense? Like people don't talk about it as much, but for a while it was like, you know, it got a lot of attention. Anyway, I never turned it into business, but a couple of years later when in 2021, when I was thinking about business ideas or things to work on, I kind of came back to work for orchestration. And I started hacking on a bunch of ideas of, you know, running code and, you know, like basically orchestrating code running in the cloud. And what I realized was like, you know, at the end of the day, like core puzzle pieces, not just like orchestrating the compute. It's like executing the compute. And I was like, that's kind of a bigger problem actually to solve. So in a way, actually like, you know, left it on the shelf and then said, like, actually, I'm going to like go to this other thing over here and I'll go back, I'll get back to the workflow scheduling a little bit, but like, you know, I never got back to long story short and ended up building model, which is like the focus is more like, let's just figure out a way to make it easy to run code in the, in the cloud. And the focus was initially like fairly general purpose. And actually, in a way, like it's always, we've kind of gone full circle more recently, but we focused on so many different applications. It was kind of the idea was like, let's just make it super easy to run code in the, in the cloud, could be GPU, code, AI machine learning, it could also be like data stuff. And focused a lot on, you know, how do we scale it up? How do we make the developer experience awesome? Realize super early that like in order to do that, we'd have to go very deep in a stack and built on file system, built on container on time, and did a lot of, you know, solved a lot of hard engineering challenges. And then you mentioned chat to beauty, that was actually not quite the killer app. But like, if you remember, like that fall, another thing came out around the same time, which was stable diffusion, stable diffusion turned out to be a killer app for our, for what we're building at that time. We were only four or five people at that time. So we started seeing a lot of people coming to us to wanting to run a stable diffusion in the model. And then we were real, that, that made us realize like maybe Jenny, AI inference is actually a really good use case for us. How are we finding here? I'm this like compulsive tweeter. I was even more like now, like I'm too distracted by building a company, but I always like tweeter a lot. I always blogged a lot. So I think I had a little bit of a following. And so, you know, in the early days, like a lot of what drove the growth is just like me just compulsive, just like tweeting about the model and like what I was building and getting people excited and writing long blog posts about where the cloud is going and all these things. And I called it a bit like founder let marketing. I was like feel like you kind of have to lean into like what you're good at and like what I'm good at is like I guess like just shit posting and being terminally online. And so, you know, like work, it's it's still like, you know, I think a lot of our growth is we're just kind of good at social media and just like being, you know, online a lot. Nice. How long until you had your first customer to go out also did you raise like early on? We didn't raise for a while. I actually self-funded it for a while, even with up until the first few employees. Which in a way was like kind of nice because it may be really like frugal about money and like wanting to see like ROI. I wasn't taking out a salary for a while for a long time actually at some point it became a problem because I needed the money. But we didn't raise for a while and until early 2022 raised a seed round. So that one had all your been working on it for about a year. I actually joined us to CTO and co-founder in about about five years ago, about now August maybe 2021. But then it took until like basically like late 2022 for us to see any meaningful revenue. I guess that's why I'm glad you came out. Yeah. And it's able to fusion then. Yeah. In our case, what was the main thing? So we went from like, you know, $100 a month and like, I think like November 2022 and the next month we did like $2,000 a month. And then like January 2023 we did like $30,000 all of a sudden. And you know, so finally I felt this like massive, just like anxiety thing off my chest. I'm like, maybe this is actually like, we're on to something, it feels like we're on to something. And then 2023 we grew very fast and I guess we've been growing very fast since then basically. That 30K a month moment was that just like a couple big customers or was it like very like diverse? It's just one and I don't know if I ever talked about like who that was and if they even know it. But I think they suspect it. In New York called Mage, they're doing like a stable diffusion company. So at some point, you know, in the first, I think first quarter of 2023, I think there were like 89% of our revenue. But then after that, you know, it started diversifying a little bit and late 2023 we got soon as a big customer. They were a big chunk of our revenue for a while too. But then, you know, it's over time like we actually have really good revenue diversification now. Yeah. So, stable diffusion stuff, I remember using it for like come for UI, I'm assuming a lot of people were using come for UI, like running come for UI workflows on it. Some people were. But I think the people that were using model, they tended to be more like low level stuff, like they did like complicated stuff. What they like with models, like it's always been pretty high. hackable. So they deploy their own custom workflows with adapters and custom inference and a lot of optimizations. And all the stuff we had done in order to get the developer experience to be really good, turn out to be really good for also running production workloads because auto scaling turns out to be actually a very similar problem like you want to scale up containers up and down very fast. So we just happen to be like very well positioned to capture a lot of those workloads in the early days. Did tools for auto scaling not exist? Why did you have to build those yourself? In general, my thesis is every new era of technology runs up against limitations of tools, whether that's, you look back, like the web or the cloud or mobile or PCs or AI or whatever. So I think it's often the case that tools exist, but they're often not particularly good. And often you kind of need to rethink a lot of those tools in order to reap the benefits of you know, and really, you know, turbocharge and start building for this new world. And you saw this with cloud, you know, we have to invent things like Kubernetes and Docker and things like that. And in a similar way, you know, like with with Jenny, I, you could build these things on the sort of prior generation of tools. Like you can build these things on top of Kubernetes and Docker and in these things. But those technologies are built for a very different type of workload. They're built for a very sort of steady work, steady back-in workload. They have an even, you know, steady number of requests. Like running a pretty low utilization because CPUs are normally pretty cheap. And so people don't really know where it's much about cost. And but now you're entering this new world and you have, you know, very expensive GPUs. You want to manage capacity and utilization. You have very large models. They need a load over network into CPU into GPU. And it turns out like a lot of these tools, like, you know, like they were just not built for that. So we felt that, you know, we had to rebuild a lot of that, you know, building our own, I mentioned the file system, but we also spent a lot of time over the last few years like building, you know, some really cool stuff from like CPU snapshoting, GPU snapshoting and audit. We built it on autoscaler and scheduler and and and all kinds of capacity management tools. And we kind of feel like you have to do that because like you can't build, you know, for this new generation of software on top up, you can't build like a wrapper on top of the old stuff in order to do the new stuff. You kind of need to start over a little bit from scratch. So I guess like the auto scaling stuff. Like did you, did you see any like weird behaviors on the platform? Like was there anything that broke while you guys were scaling? Make capacity is always from time to time been been a challenge. Like how do you get like, you know, hundreds and thousands and tens of thousands of GPUs? So that's, that's what was challenging. We had to go multi-region very early. So I think that was like kind of a fateful decision in hindsight, like doing that and kind of spiting the bullet and building it. Like we always thought of like our capacity management is like one big pull. Like it's like one big pull of it over the entire world. Like we don't think of it as like separate clusters running in each region. And so that that was like one of the like very consequential decisions we made early to kind of think about it that way. So I don't know if there's like been like, you know, specific issues. I mean, we've obviously had bugs and, you know, outages. And I hate, you know, as much as I, you know, would love for a modal to be a hundred percent uptime, you know, and I think recently we've basically hit four or five nines. But throughout the history of moral, of course, we've had, you know, run up against scaling issues. Yeah. The GPU market. And well, actually before that. So you guys use AWS, like GCP, like all these cloud providers underneath. Like do you also buy any infrastructure or no? Let's talk a bit about like cloud capacity. Because I think it's actually changed a lot. Like in the first few years, we did, we got almost all the capacity pretty much from from the hyperscalers, spot and on demand capacity, mostly spot capacity, running in all the different regions, you know, using initially AWS, but also GCP and overtime measure. We started renting some capacity from Oracle pretty early because we wanted eight tens and eight hundred and we got really good rates on those. And over time, we added a couple more clouds like doing reservations. But like still up until like, actually end of last year, like most of our capacity was either spot or one year reservations. Spot means you're just like renting it right there. Exactly. Yeah. And it's also preemptible, which we had to deal with. But it sort of works in aggregate because you can sort of just, you know, aggregate capacity from many, you know, hundreds of regions, basically. So that that model worked really well until we started running up again. I mean, first of all, like modal started growing like very fast as we needed more capacity. But fundamentally, also like the cloud market started changing a lot. So in the last year, basically now in order to get the capacity we need, we have to work with a much larger set of clouds. We actually work with almost 25 clouds at this point. And in many cases, in order to get the capacity we want, like, we have to go and do, you know, three plus year deals and you have to pay a meaningful amount of money up front. And you're locking in pricing on a three year term, right? So, you know, like, and so we have to spend the entire, you know, last six to nine months, like, you know, kind of rethinking how we get capacity. We have to build a whole procurement, like desk, like, we have like four guys working full time basically. All they're doing is like procuring capacity, calling the clouds, you know, doing deals, vetting the capacity. We've had to build a whole engineering team. Basically, 10 people just like managing like bare metal capacity from all over the world and dealing with network issues and bad drives and GPUs falling off the bus and in all these things. We don't own capacity. We rent all the capacity so far, you know, on typically this three year deals. And that's like a big decision. Like, should we buy? Should we, you know, or should we keep renting? I'm actually like not like mega-digmatic about it. I think renting is nice because there's a certain amount of capital efficiency to it. If you're buying, you know, like, I mean, first of all, there's obviously like operational complexity of it. Like, you have to, you know, deal with vendors and suppliers and, you know, power and cables and all kinds of stuff. But there's also, you know, financial complexity to it. Like, how do you fund it? Like, should you raise equity? You know, and you can raise a billion dollars and go buy hardware for it. That's a very diluted way to do it. You can raise debt. But, you know, if you raise debt, you know, you typically want to have, like, locked in long-term revenue to pay the interest rates. So that's also not trivial. But, you know, like, we look at the economics, we look at the operational complexity, we look at the risks. It's a complex trade-off, and we're open to buying. Well, we're sort of, there's other constructs too. Like, we're, it's to get the capacity we need for 2027. It's going to take a lot of work on, you know, going a lot of different avenues to lock it up. So, do you have to hire a lot of like finance people to do all this stuff for, not really? You know, I've had to spend a lot of time on this, and we've had to spend a lot of time on this. And I do think over time, I mean, there is a lot of, you know, these are very expensive assets, the hardware. And, you know, I think, I mean, it's kind of staggering. You're starting to see, like, AI investment started to become like several percentage points of GDP of the US, and, you know, starting to see, like, interest rates in America shooting up in the last few weeks. And, I mean, I, like, at this stage, like, you know, and you go and look at the neoclats. Like, these are very, very sophisticated financial engineering companies. And so, yeah, we, we look at that and, you know, we look at all the options and it's possible. We might have to do similar things ourselves. I wouldn't want to, like, I would rather focus on being, you know, like a very lean software company, like doing the things we're good at. I don't think it's good in general for software companies to own hard assets, you know, in the same way as, like, should you own your office? Like, I don't think that's necessarily the right solution for most companies. But, of course, like, in order to get the capacity, you want, like, you know, everything's on the table. Yeah. And, I guess, the supply is going to be constrained for a while. So, you kind of have to do that. Like, do, do you agree with that? Or do you think this is just like a moment in time? I don't necessarily agree with that. I mean, like, Nvidia is delivering, you know, several hundred thousand black walls, you know, a month, right? And I don't want to say, like, the market is like 100 percent efficient. But, like, you know, in general, like, if you're willing to pay a high price, you can get cheap capacity, right? Like, if you're willing to pay more than everyone else, and, of course, inthropic and open AI is buying up as much as they can. But fundamentally, like, there's constantly supply coming on the market. And so, I don't necessarily think it's like a fully cornered resource. Prices are going up. And I do think it's worth kind of reflecting on that is like, the normal state of hardware is like, it's like, I don't like gravity. Like, eventually, hardware prices always go down. So, we're a little bit in like a weird set of period of time where like, hardware prices are going up. And, you know, it's hard to say, you know, how long is this going to last, you know, could be, you know, the six months, could be another 36 months. Like, it's really harder to know how long and, but, but I fundamentally don't know if like, it's true to say, like, you know, you need to buy capacity in order to me. That's a little bit to sort of almost like the great toilet paper shortage of, you know, the pandemic or whatever, like, you know, at least this, like, you know, run on the bank type situations. And, but that being said, like, you know, capacity is going to be tight for a while. And, and, you know, prices are going up. And, you know, luckily, I think, you know, it's not even necessarily bad for us if prices go up, because, because, you know, scarcity also means like we can charge more from our customers. And, you know, and the fact that we have a lot of reservations locked in at a rate means like, to some extent, we're effectively like long GPU prices already. So, um, I, I don't necessarily worth too much about prices going up. I do worry more about actually prices collapsing. I don't think that's going to happen anytime soon. Uh, but I think that would be a quite bad thing for a lot of the players in the market. Why would that be bad? I think it would be bad for if, for anyone who's locked in a lot of supply. - Yeah, 'cause they paid way too much. - They paid too much, you know. So there's a risk of buying at the top, right? And again, like, I don't worry about that that much. I don't think it's gonna be like, you know, it's gonna happen anytime soon, but like, you know, like I said, like hardware prices in the very long run, they tend to go down. And so it's worth, you know, reflecting on that, right? Like, right now, I think your capacity is constrained and that leads to sort of, you know, in economic terms, there's a scarcity of rent, right? Like, who captured a scarcity of rent? I think the primarily goes to two places, mostly in video, but of course, but also to some extent, neo-class. Like, neo-class, in general, like, you know, anyone owning capacity has better economics. Like, that's undeniably true today with capacity being constrained. Long-term, you know, in the very long run, I don't think that's necessarily entirely true, right? Like, there's always gonna be a little bit of a premium you can get from owning GPUs. But is it worth the risk, you know, is it worth the risk of, like, holding this, like, very expensive assets on your balance sheet? I'm not so sure, but, you know, we're gonna see. Like, it's gonna take a few years to get through this and then the long run, you know, we might own some, we might rent some, maybe a combination, maybe do some other crazy stuff, joint ventures, who knows? Maybe some complex SPDs or whatever. Everything's on the table. - How do you compete with OpenAI and Anthropic in buying GPUs and capacity? I'm assuming they wanna, like, buy up literally everything that's available. They are and I think what you're seeing is, you know, a lot of the prices going up is the reflection of that. That being said, you know, Nvidia's shipping a lot of stuff. I think also, like, to some extent, like, Nvidia's, like, kind of, prefer a little bit of a diversification, like, they don't want, like, you know, small amount of large buyers. In much in the same way as you see, you know, hyperscalers getting a little bit of a constrained allocation. I think, you know, also Nvidia prefers to see, you know, a lot of NeoClouds and a lot of alternative clouds, like getting more allocation. And I think a little bit of what we see in the market is also, this is more speculation. But I think we see, you know, smaller clouds that are a little bit more sort of risky to work with, being more of an opportunity for us, like, where, you know, maybe Nvidia, sorry, Anthropic and OpenAI, my feeling is, like, they prefer to buy from, like, to try to true, part of it is also, because they do a lot of training runs. They need to have, like, very large blocks. But there's plenty of capacity out there, you know, you can get 32 nodes in Armenia, you can get 128 nodes in Indonesia, like, you know, like, we love that kind of shit, like, you know, so we can go out there and, like, get a lot of, you know, smaller capacity all over the world, pretty good prices. - Is it pretty much just, like, if someone offers you, like, a certain number of GPUs, like, that's all you care about, is there nothing, like, oh, you care about, like, how reliable it is, you know, is it just some, like, janky warehouse that someone has these in? - I mean, sometimes it's kind of a janky warehouse that feels like, you know. But, like, we do have like a spec sheet, basically. Like, when we talk to vendors, the first thing we do is we send them a spec sheet. We say, we need this number of IOPS. We need, you know, like, the CPU needs to be at least, at least, you know, have this many, you know, whatever the pass mark scores or whatever it is. We need to see, like, you know, minimum, you know, north, north, south bandwidth, egress, and ingress. So, we do have a list of requirements. Often, we try to do, like, an acceptance testing period where we get, like, one node and run some tests. Over time, you know, like, to some extent, it also involves a little bit of a leap of faith. You know, sometimes we sign very big commitments, you know, tens or hundreds of millions of dollars, and we start working with the vendor that we still don't fully know. And, you know, over time, like, you know, we'll see. You know, and I think over, like, in a little bit of, like, there's a little bit of sort of, just, you know, our own sort of, like, you know, how, like, semi-analysis has, like, a cluster max. I kind of feel like we almost, like, have our own, like, modal max score internally. Like, I know which vendors I like more than others, you know, like, and we sort of prefer to do additional deals with the vendors that we've seen, you know, very good reliability. Not just that, also, like, it's not just reliability. It's also, are they very responsive when we have, you know, support issues? Do the fixed issues we ask for the turnaround time? If things go down, you know, how quickly do they fix it? Like, things like that. Do you agree that the demand for intelligence is basically infinite? And if so, like, why wouldn't you just buy up as much capacity as possible? And then, like, if you over-buy, you can just, like, sell it to somebody else. - I think it's very easy to say when there's, like, a scarce resource, like, let's just buy as much as you can. But I think, you know, first of all, like, if we bought it today, we couldn't utilize all of it. Like, we could utilize a lot of it, right? But, like-- - But you can sell it off to somebody else who wants to use it, right? - I think that challenges, like, if there was, like, something in the market that happens, there are a lot of correlations that happen at the same time. If, you know, price is going down, and utilization going down, and the market for reselling capacity goes vanishes at the kind of the same time. I'm sitting here and, like, I'm extremely aggressive, like, buying capacity every day and, like, raising money to buy, you know? So, overall, I'm very bullish. But, like, I do want to point out and, like, you know, like, there are terrorists here. And when those terrorists happen, like, it could be, get very bad. Part of it is, like, maybe, I'm, you know, I'm just old and I've seen this in the past. Like, I actually had my first job in 1999, so I saw the dot-com crisis, and I've seen the great financial crisis, and things like that. And, you know, like, I want to build a company that is around for, like, 20 to 30, 50 more years, right? Like, and so, you know, I think we can certainly go out and, like, buying a sane amount of capacity right now and trying to just corner the market. But, you know, do you know about the Hunt Brothers? It's like a legendary story in trading. They try to corner the silver market. It feels spectacularly, you know? So, it's cornering the market, I think, in general, like, it can be not, not a good idea. But we are, in general, very, very aggressive. But we're ramping up, you know, like, at an aggressive plan that basically matches our demand growth and matches where we want to try to get to, at the end of next year, and trust a runway, a run rate, man. - Are you only buying up Nvidia stuff or also, like, you know, maybe a capacity from other types of companies, like-- - I mean, we're training them, and-- - We're looking at it. I think the alternative accelerators are starting to get very interesting. It's, you know, ironically, like, every year and the last five, maybe even 10 years, I've been saying, you know, like, next year, it's gonna be the year of, like, non-in-video accelerators. And then I'm always wrong. But I do think 2027 might be the year of non-in-video accelerators, finally. So we'll see if I'm wrong this year, too. But, so I, you know, AMD's starting to get, like, really good scores and certain things. I think, you know, TPUs are obviously used very, you know, widely internally at Google. It seemed to work really well. I don't know much about training. I don't feel like I see quite as-- But, you know, the team is very good. I know they're working on a lot of the software stuff. You know, hardware is hard in a certain way. Like, software is, like, easier. And that makes me feel like, you know, if we just fix the sort of software layer for these accelerators, like, I think there's gonna be a lot in the man for them. And I think fundamentally, you know, with them video being such a bottleneck, like, we'll probably see, you know, more alternative accelerators. That being said, you know, there's bottlenecks for it down the pipe. I mean, you know, there's, like, TSMC and A's of L, maybe, like, a deeper bottleneck to all of this anyway. But, you know, long-term, I think monopolies don't always last. And so I think you'd be healthy for the market to have more alternative. As much as I love Nvidia, like, to make that clear, like, I mean, they're obviously an amazing company and a within-amazing product. I do expect to see a little bit more diversity in the market, as well, also things, you know, like, cerebrus and grok in these things. - Yeah, what are your thoughts on, like, some of those providers? - I think they're pretty exciting. And, you know, and you see them doing really well on certain workloads. And, you know, there's some interesting, you know, people doing these, like, hybrid, you know, disaggregated inference where they run, you know, Nvidia for the pre-fill and cerebrus for the decode and things like that. - I've heard of that, actually. - Yeah, and, you know, grok, too. Like, 'cause a lot of these things, you know, certain things are compute-bound, certain things are memory-bound. So you can sort of mix and match. And I think that's quite exciting. It starts to get very complex when you're deploying these things in practice. But I do think there's a lot of, there's a lot of, you know, opportunity and these sort of, you know, more complex hardware setups. - Are there any learnings that you get from some of these NeoClouds? Like, I don't know, like, Nebias or whatever. Like, the way that they're doing things versus, like, what you're doing? - I think one of the things that, you know, is very interesting with NeoClouds is like, how can they expand so quickly? I think, actually, the most crazy example is, like, end scale, which went from, like, nowhere to, like, you know, having, like, you know, whatever, hundreds of billions of dollars in the pipeline, despite barely having any capacity. How can NeoClouds grow so fast? I mean, there's, like, you know, and basically, the answer is, like, you know, and this might be obvious. But like, what they do is, like, they go out to, like, you know, anthropic or, or, or, or opening eyes and some big buyer, or whatever, and basically say, like, hey, do you want to buy, like, five years of chips? You know, sign this contract, we give you a good rate. And then they take that contract, and then they go to, you know, Wall Street and say, look, we got this, like, you know, big buyer, you know, they, they committed to $10 billion worth of chips. We just need the money right now. And then they go on to raise the money. And then they, like, you know, go to, like, super micro and dollar, whatever, in the video, and, like, get the hardware, and then they install it. And then, you know, now they have, like, you know, deliver it to the customer, right? So you have this, like, very nice sort of financial construct where you can basically sort of presale things that you haven't even bought. And that means, you know, you can scale up extremely fast. And so that's what we've seen with these Neo Clouds is, is they can scale very, very well. I think Modal, because we've almost had a little bit more of a PLG focus and, and not the sort of, you know, ability to lock in long-term contracts in the same way, it's a little bit more challenging from a financial perspective to scale that fast. even though I think, you know, in the long run, like we'll get to the same point. But I think that's been kind of a fascinating thing to see with these NeoClouds, and especially like CoreWeb and those big ones, very, very savvy with, you know, how they raise money and how to, you know, think about GPU as collateral for loans and all these things. - What's the relevance of, like, I forget who it was, but like some like financial company was like, oh, we're gonna make, oh, I think it was Nvidia actually, they were like, we are going to turn GPUs into an asset class. Is that just 'cause like, they need to raise money and they're like trying to figure out where to get the money from? Like, what's the relevance of that? - I mean, there's a bunch of startups, I don't know if you're thinking about that. There's like Silicon Data and Computer Exchange and like, Orn and a few other ones that are like, basically like trying to get to like, you know, creating price indices and indexes and trading derivatives and things like that. I think it's interesting. I think with compute, like, it does, there are a number of challenges with compute as an asset class, like, it's definitely an honest homogenous as other asset classes. You have different network and different locations and different disks and CPUs and all kinds of stuff and reliability. I mean, that being said, like, you can say the same thing about oil and like, we kind of created like, you know, relatively commoditized, you know, tradable assets out of oil. It remains to be seen. Like, I do think, you know, at this scale where we are with this sort of AI, like data center, GPU deployments, like, we do need to be a little bit smarter with like risk hedging and how we share risks and, you know, having more like sort of insurance and things like that. So I'm very happy that people are thinking a lot about this. That being said, like, I'm not entirely sure if we're going to see, you know, GPUs becoming sort of, you know, whatever it is, sort of Henry Hub, like, you know, Brent Crude type, like, you know, super commoditized, like, tradable assets. Yeah, I'm going to shift the conversation to modal sandboxes. Yeah, I guess like before we do that, what are the different products that modal has? It's, I guess like, the main product, obviously, is like the GPUs. Then you have modal sandboxes. Is there anything else? I think of the like, the two big products that we have is basically number one is modal functions. Number two is modal sandboxes. So with modal functions, we launched that, you know, back in 2021, there's the idea of like serverless functions that are very easy to deploy, very easy to scale, and we manage all the capacity. That turned out to be really, really good for inference, and it turned out to be really good for managing GPUs and running on things like that. But we actually never had like, you know, dedicated like inference product or like a GPU product per se. Like we never thought of modal as like a neoclod, and I think there's like a risk of like getting sucked in that direction and becoming a neoclod, like just selling, you know, sort of what I think was like a rectangular, like here's like a three-year, you get this mini GPUs. Models always been very good, like you deploy something, you can kind of scale it up. You have this like very sickly color, like bursty stuff, like we manage all the capacity, and the beautiful thing is like 'cause we have many, many thousands of customers, we can sort of aggregate that capacity and get very smooth workloads, you know, very smooth aggregate demand, which we can manage very, very efficiently. So modal functions was the core. We launched that in 2021, like I mentioned, we started seeing product market fit in 2023, and now, you know, it's about two thirds of our revenue and empowering a lot of, you know, in practice it's mostly inference, but you can use it for all kinds of other stuff. Models sandboxes, we launched that in 2023, and it's actually kind of, you know, took a while to see product market fit, similar to functions. Why did we launch it? We realized a lot of people we talked to, we're using LLM's to output code. This was like me and Akshat talking over launch one day, we're like maybe they need to place the run that code, and we realized like actually we kind of have like, a lot of the infrastructure already to just take code and run it because we've already built the primitives for like managing containers and isolation. We already had, you know, GVisers and isolation primitive and things like that. So we got kind of excited about that idea and Akshat wrote a prototype and launched it in basically one day or two days or something like that. Zero, like we saw like very little like growth. I don't want to like trivialize that, we did see some growth. Like there were like some sort of, you know, it was 2023, so yeah, it was 2023 and like, you know, like some people picked it up. Like we had some early users who kind of enjoyed it for certain things, but like overall, like it was like a pretty insignificant source of revenue into going into the summer of 2023. So two years later, we even discussed like, do we even need this product? Like should we shut it down? Then it just started exploding. So 2023, and I think what really happened 2023 was like, LLM's got really good at generating code. - In 2023. - 2023, yeah. I mean, it was like around that time. Like I feel like that was like when they went from like being able to like refactor functions and like write snippets like generating like full apps, right? And so we started seeing a lot of growth coming from like vibe coding, you know, companies like Loveable and other ones. And then, you know, companies like ramps started using model for background agents. And then was the start seeing tremendous growth coming from a number of neo-labs doing things like reinforcement learning and model sandboxes. So sandboxes is now about a third of our revenue. You know, we considered like basically like our second product. And to me, it's like really exciting. 'Cause like even back in the early days of model, I always knew that model, you know, is about like creating a whole toolkit of products. Like we always wanted to be almost like a cloud, you know, you have the storage and the compute and a bunch of different ways to work with it. We always wanted to be a multi-product vendor. But that wasn't necessarily true in practice until like, now we're like, I think now it's like kind of clear. We have functions, we have sandboxes. We're working a bunch of other stuff too. We're working on training, we're working on batch. We're working a bunch of other stuff. So to me, that sort of evolution into becoming like a multi-product company, I think it's been amazing. 'Cause that was, that was always so core to the vision. Even in the very early decades, I had a bunch of slides on this. But it wasn't necessarily true to the external world until recently. - Do you have anything in the works that, you know, you think is gonna be like that next product? - Any predictions, maybe about the market, anything like that? - I mean, I'm very bullish on training. I think training is like kind of an underserved market. Obviously, there's like a lot of focus on like a very large, you know, training runs. But we actually see a lot of demand, the sort of smaller and mid-sized training segment. So I'll be talking a little bit about that later this week. We're organizing a conference. Batch is I think like a really interesting market. I think we see a lot of demand for sort of, you know, customers that need a, you know, we had a customer the other day that needed to transcribe, you know, 5,000 years of audio. They had like, you know, millions and millions of audio files. And we had a lot of capacity. So we let them do that, you know, and took a few days. I think there's some really interesting use cases there. Very bullish on like low latency stuff. I think there's gonna be a lot of, you know, once we sort of get the audio, like the speech-to-speech models working, like the voice models, I sort of expect that to be very interesting market. Then of course, like all this like a gentick stuff, right? In a way, like I have like mixed feelings about like the word "agentick" because I feel like in certain ways, like, I think all software will become agentick. So it's almost like to me, like, what does it mean to be agentick? Like I think all software in a certain way like will become like AI powered over time. But like clearly there's like a set of like building blocks of like, you know, doing more sort of LM powered software that seems to be sort of, you know, common across all the different use cases. - We're like, you know, the way I think about models, like we're into business of building building blocks. You know, we wanna give you like, amazing generalizable battle tested, you know, general purpose, composable building blocks. And I think there's some really interesting stuff also, like, you know, with the agent use cases. - That's interesting, you bring up training 'cause I think in another podcast maybe like a year ago, maybe like two years ago, you said like training, you were not focused on like specifically, like what changed in the market? - We've been building a training product for multiple years so you know, on and off so. So we, I wouldn't say we're like, we're never been focused on it, but it's like, it's like one of those things we're like, I mean, now we're like 150 engineers, but you know, a year or two ago, we were much smaller number of engineers. And so, but just virtue of, you know, being smaller, we had to focus a little bit more. And so inference became like our sole focus and was our sole focus for many years, but we're bigger now, like we do have opportunities to also place bets in other markets. I'm also really interested in, you know, kind of continuing the list of things. Post training, reinforcement learning, I think there's some really exciting stuff around that. I think there's also many other like, stuff around like storage and building data warehousing and feature stores and real-time streaming pipelines and all kinds of like, I don't know. I think the long-term view of model is like basically like, you want to build some cool AI shit? Like, you know, we got you covered. Like, you know, we got all the building blocks. Like, comptosts and like, we'll help you, you know, take all these building blocks and compose them. And that's the beauty of softwares. Like, you can, you know, build almost anything if you have the right building blocks. So that's very much the sort of company and the product that want to build. - When you say training, are you focused on more of like the bigger enterprises doing huge training runs? - No, okay. - No, they're going to go and like, you know, do these like, you know, big, Harper scale or rather neo-cloud mega deals and get, you know, 100,000 GPUs or whatever. But, but there is like a little bit of an underserved market of like, in my opinion, you know, like researchers iterating or like people training smaller models. There's, you know, Texas beach and speech text models. Competition biotech is actually a market that we see a lot of demand from. We have a lot of customers in that space. Training relatively, you know, modest models. You can train on 64 GPUs or whatever. Recommendation system. A lot of sort of traditional machine learning. There's a lot of things like, you don't need it, you know, 100,000 GPUs. That market I think is like, we're not going to like, focus on that for a while. But I think there's a lot of stuff in the, you know, slightly smaller. markets that are relatively underserved. It's very hard to set up training clusters. - Yeah. Have you noticed, like, since, I don't know, like, GLM 5.2, or, like, Kemi-K, maybe, like, Kemi 2.5, that, you know, a lot of people are using those and, like, fine-tuning them? - If anything, I've actually seen the other direction. I feel like fine-tuning the value prop a couple of years ago was like, yeah, you know, open-source models, they're not quite as good, but, you know, if you take, you know, small model and you fine-tune it on your specific domain, you can actually reach the frontier in terms of, you know, quality and cost. Today, I would say, like, the open-source models are so good that, like, you can basically just, like, swap them out and just run, you know, instead of, you know, the frontier ones, you can run, you know, Kemi-K3 or whatever, deep-seek 4.1 or GLM 5.3, and they're, like, basically, almost as good as, like, the frontier model, at a much lower cost. So, we still see a lot of demand for post-training, you know, I do think it's a really interesting space, and we're working on some products in that space, you know, both sort of SFT and also reinforcement learning and things like that. I would also say, you know, fine-tuning is no longer the thing that you kind of have to do, like, it's very optional. Like, you can open-source models, like, work really well just off the shelf, too. - Back to Sandboxes, actually. I forgot to ask you. So, like, what did you have to update about the Sandboxes that you guys were offering? That, like, existing Sandboxes did not allow for? - Not a whole lot. I think the thing we've always been very, very good at. I mean, we're in for geeks, right? Like, we're very good at, like, you know, scaling and reliability and doing a lot of stuff, and I think that's why that explains to a large extent, like, while we have, you know, in my opinion, I don't have numbers, but, like, I think we have, by far, the largest market share in the space. Like, we're just very good at running, you know, hundreds of thousands, or even millions of Sandboxes. - Yeah. - I will say, we'll have to, we've had to rewrite our systems, you know, a bunch of times in order to handle that scale. So, but we were always, like, very good at it. There are other things, like, over time. I mean, the Sandbox space in general, like, you know, like, it's, like, not the most, like, differentiated space. But, you know, we're working on some really cool stuff, like, over time, like, we've had to add support for VMs, like, virtual machines. So, we previously bought our Sandbox product on Gvisor, but now we offer full virtualized machines as well, turns out a lot of, for certain use cases, you kind of need full Linux compatibility. There are other things we've had to build, you know, better primitives for, like, more interactive, you know, piping data, back and forth between Sandboxes, working with file system. We've had to spend a lot of time on snap-shawning. I think a lot of the things that, over in the long run, will differentiate, will be, like, you know, really cool things, like, you know, forking Sandboxes, snap-shawning, resuming snaps, like, live migrations. In general, like, I think Sandboxes, the novelty of Sandboxes was always, you know, this sort of, a familiarity, like, you can spin up Sandboxes for a few seconds. And that's something I, like, that's a string I want to pull more in. I think there's a lot of really cool stuff you can do, you know, kind of decoupling the agent and the harness decoupling it from Sandboxes, having agents control many Sandboxes, forking Sandboxes, you know, snap-shawning things, like, you know, like, running basically, like, very shortly, maybe one per command, like, things like that. So I, I think there's a lot of cool stuff there. We've already spent a lot of time doing that, you know, around snapshots, but I think there's a lot of cool stuff to be built there as well. - You brought up snapshotting a couple of times, like, what do you see as, like, the use case for a snapshotting? Like, what would that give us that we don't get right now? - I think the, the big thing for me is, like, today, a lot of Sandbox workloads are, kind of, like, lift and shift. Like, people think a lot about Sandboxes, almost like, you know, I have my dev box, now I'm going to shove it in the cloud, I'm going to have an agent control it, maybe I'll run the harness inside the Sandbox. I don't know if, like, long-term, that's, like, the right way to think about, you know, on, like, compute for agents. I think a much better model would be, you're going to have these, like, very long-running agents, which, almost like, to me, are, like, kind of, durable execution workflows. Like, they're going to run, you know, 24/7. You know, every once in a while, they're going to run commands, and, you know, having the ability to, like, spin up a Sandbox in, you know, 100 millisecond, run a few commands, and then, like, shut it down and, like, snapshot it. I think the faster you can get that, like, you know, wake up, you know, go to sleep cycle, the more you can sort of create Sandboxes that only run for, like, a very brief period. And that will make compute much more efficient, and it will also enable you to do some really cool things, you know, like, forking and moving things around, and, and, you know, having, like, imagine, you know, like, clawed code instead of, you know, having, sort of, a very linear flow, like, sequential flow, like, kind of, branching and, you know, running hundreds of agents at the same time. I think a lot of, like, CI is, like, another, like, illustrative example. Like, for most of the, the history of CI, you know, talk about, like, GitHub actions, and things like that. Like, you had a very sequential flow, like, you know, you slowly install a bunch of stuff, then you run the test suite, and then, you know, generates and reports, or whatever. And it's like, you can break it up into shards, but it's kind of annoying. Like, what I would love to do is, like, take your entire test suite, and just run, every single test in its own sandbox, right? Like, and just paralyze the shit out of that. Around, like, a thousand sandboxes. Why can't that be done? I mean, the problem is, like, there's too much, there's too much overhead around, like, you know, starting and stopping sandboxes. So I think the more, like, you create, like, you know, frictionless, like, starting and stopping, the more you can enter a world where, like, you can do this, like, extremely bursty, like, large scale, like, run as much as you want in parallel. So, you know, I'm an infrigue, like I said. Like, I get very excited about these things. You know, once we figured out all these primitives and the right building blocks, I think it's gonna unlock a lot of really cool stuff that, you know, we previously could not do with software. Why hasn't that already been built? I would assume even previously people would want to parallelize as much of their CI as possible. You can't, but the overhead, it's like, Amdol's Law, whatever, right? The overhead of parallelization right now is very high 'cause, like, right now, like, you can get up actions, for instance, like, we parallelize in the model the test suite, right? But, like, what that basically means is, like, now, if you take your test suite and you split it up in two, like, now, each of those shards, both need to do all the installs and the test setup. And that takes a long time, like, it can take minutes. But let's say you can do the test set of ones, and then you snapshot the container, and now you clone it, and now you can run the test suites independently. I think a lot of the benefits cannot be, you can't materialize those benefits until you get the friction and the overhead down, the transaction cost. - I think you guys have a partnership with Anthropic, and they're using Sandboxes for something, like Model Sandboxes. - Yeah, I mean, the managed agents with Anthropic, there's a sort of bring your own sandbox integration. So you can use Model for those, yeah. - Oh, cool. How did that come out, like, come maybe? - I think Anthropic is like trying to create, you know, like, a thriving ecosystem and community, and so I think, you know, they make money from inference, right? - Okay. - Like, so, I think it's very much in there, and just to promote, you know, like, for everything else, you know, as much of, you know, if an ecosystem is possible, and they want to have the largest possible surface area, as long as you use Anthropic for inference. So I think it makes, like, very rational sense for them to partner with us as a Sandbox provider. - Also, I feel like we have to bring up the hugging face incident. Are you okay talking about that? - Yeah, yeah, yeah, it's all there. - Yeah, yeah, yeah. So, yeah, could you just briefly describe, like, what happened? And I think they were using the Model Sandbox, but it wasn't your, you know, it wasn't like a fault of the Sandbox. - Yeah, exactly. I don't forget all the details of the exploit, but I mean, you know, I think what you're referring to is the fact that, like, Model was involved. - Yeah. - And so, you know, just to comment on that, like, basically, the agents running inside, you know, that exploited a hugging face, they discovered a vulnerability in an app that was deployed on Model. And they used that to sort of, it was almost like a springboard for getting access, getting full internet access is my understanding. I don't remember the full details. - To be clear, this is an app running a Model, that was hacked, like, it was not a Model, that was hacked. So it's like AWS, a lot of people run stuff on AWS, location of those app get hacked. Does that mean AWS got hacked? Like, no, of course not, right? And so there's a lot of people running apps on Model, and, you know, one of those apps got hacked. And I think, but like, there are some lessons we can learn. And I think one of the things we've been reflecting on is like, could we have more sensible, you know, security restrictions by default? I think to use the AWS analogy, and just as another analogy, you know, S3 used to be this like, you know, it used to be very easy to accidentally make an S3 bucket public, and AWS realized like, you know, it doesn't necessarily cause a, you know, poses security problem for AWS per se. But a lot of their customers accidentally would, you know, like leak data. And so S3, it's much harder now to make S3 buckets public. And so in a similar way, it's like, we're exploring like, you know, can we make security a little bit more restricted by default without necessarily, you know, destroying the magic that makes Model good? And I think there's also, you know, it's kind of a separate side track is like, agents are getting really good at cybersecurity. So we've actually also, as a sort of, you know, separate sort of learning from this incident, like we started using agents internally quite a lot to try to find vulnerabilities. So far hasn't found anything, but like, they're very good at like probing into like very deep parts of the system and exploiting, trying to exploit things and trying to escalate and get out of sandboxes. So it's something we're exploring very actively right now. - How exactly do you get the agents to like run through the code base? You're just like, hey, like find security vulnerabilities? - Well, no, you're just like, you're just putting it in. put them in a sandbox and tell them like, can you try to explain? >> Just run through that. >> Yeah, can you just try to escape from the sandbox? It's like, escape the room, but for agents. >> Cool. When did you start doing that? >> We started doing it a few months ago. And it's pretty exciting. >> Like when people, I guess, came out? >> Yeah, roughly around the same time. >> Okay. >> I mean, my view on cyber security is that, you know, actually I think they're very good at finding vulnerabilities. But I think you can also use them defensively, very well. >> So I think, you know, companies like Motel and many other companies right now, like I think, you know, if you go to talk to like pretty much every season, I'm sure they're all thinking about the same thing. Like how do we use agents defensively to fix all our vulnerabilities? >> So over time, actually, in a way, like, I'm not that nervous. I think if anything, like agents and cyber security, all the breakthroughs and cyber security will make the world safer if anything. >> Do you think that like all of the cyber security, the big cyber security companies are just cooked? >> I don't know if that's true. Because I mean, you're asking something that's like very far outside of my, like core expertise. But when I think about like cyber security, a lot of what you're selling is effectively fear. And that's almost like a branding and like marketing strategy. As much as, you know, obviously the core products. But I think, you know, brand and things like that are actually like, arguably like more valuable in the era of AI. So I wouldn't necessarily jump to the conclusion that they're cooked. I think they have, you know, deep relationships with the enterprise customers. And they have, you know, big sales teams are very good at telling, you know, spooky stories and things like that. And I think they're going to continue to be very good at that. >> I just ask because like, yeah, you're doing it yourself. You're not going to want to like the Palo Alto Network or whatever. But I guess like it's only like giant enterprises that care about that. Also, they can blame somebody if something goes wrong. >> That is true, yeah, yeah. You kind of want to hire in many cases, just like an established vendor in order to blame them. I mean, to some extent, it does reduce a little bit the need for us to do, to bring in pan testers, like human pan testers, which most startups like do on a regular cycle. So maybe previously, you know, we would do it. I actually don't know what the cycle was. But I think we used to do it like once every six months roughly. Maybe now we only need to do it once every year, I don't know. But like, I think with all of these things, you know, humans can also use them to get more efficiently. So like, you know, I don't know. I'm kind of a believer also that like humans with AI is even more powerful now. So like, maybe there's an argument that like we can bring in pan testers who are really good at using agents and, you know, find vulnerabilities even faster. I don't know, I'm not that, you know, I'm an optimist. I started a company. So like, you know, I'm generally like optimistic about everything. But like, I certainly don't think humans are cooked or companies are cooked. Like, I think in the era of AI, a lot of companies are going to figure out, you know, how to use AI to be even more efficient. I want to transition a little bit to like developer experience. Because like early on, I mean, even now, like you guys focus a lot on that. Young, you've talked a lot about that. Does developer experience even matter anymore? I think it still matters. I know, I think you're, you're sort of, you know, like the context. I think you're fish or like the thing you're talking about is like now with agents, right? Like, does the developer experience? Like the docs are already there. The agents are going to look at the docs, like all that stuff. To that point, though, like almost all the things that make developer experience good also makes the agent experience good is kind of what we've seen in practice, which is like things like having good docs, having descriptive error messages, having fast feedback loops, right? Like, I do think there's certain things have changed. The other day, I was like setting something up with Cloudflare. And it's like, I find Cloudflare like notoriously annoying to work with sometimes. And I threw an agent at it. And it was able to just configure things in Pulumi for me. And so I will say agents, they do make it a little bit easier to work with things that are harder to work within the past. Like this sort of human toll of dealing with Yamal has arguably gone down a little bit because you can just tell your agent to deal with it. But for the most part, I think the things that make developer experience good also make agent experience good. I think to some extent also like infrastructure over time anyway, it needs to be more programmatic. I think that's always been the thesis about Sandboxes is that, you know, like things that you previously have to configure using Yamal, that should just be an SDK, that should be more like imperative, like launch a sandbox, like do stuff with it, tear the sandbox down, right? Like rather than sort of declarative using using, you know, some sort of DSL, it also pushes like a little bit of a different interface that maybe previously didn't exist. I think in either way, like SDKs are as relevant as they always were, if not more. Is there anything you changed in your product? Just because like, you know, most people are just running agents on it? We do. I mean, we also, we actually do run a lot of internal tests where we have agents, we tell agents to like build a model app and run it. And I mean, a very concrete thing that is kind of funny is that we've actually seen that like because of agent training data, it's very hard to deprecate things now. We see that like agents often like love using, you know, old versions of models syntax and it's kind of hard to nudge them away from that. So we've actually had to be a little bit more conservative going for it. Like, you know, over the last year or so with, with how we deprecate and remove features, it takes, we have to be very careful to update every single example that's out there online. Yeah. And, you know, yeah, there's like stuff like that. But, but like, there's also an insane amount of like, modal code that's now like online, too, that goes into the training data. I've seen like that to be like kind of an, you know, a little bit of like an interesting example where like, another thing, by the way, too, is like, there's obviously like, to sort of eat similar to like search engine optimization. There's like, agent engine, you know, whatever people call it, agent optimization. We often like run internal benchmarks where we're like, ask an agent like, hey, like, can you deploy a stable efficient for us, use whatever tool you want? And we see if it recommends model or not. And those things are very useful for us to sort of track over time. Or like, hey, spin up some sandboxes. Or like, hey, like rank all the sandbox vendors. So we spend a lot of time like doing those things to make sure. I mean, what do you do about that? Yeah, you say, hey, like, you know, build a stable diffusion thing. Does it recommend modal or not? Like, yeah, what do you do about that? Yeah, I think that's harder. I think, you know, it's like search engine optimization. I think it's primarily through content. It's like you just have to put out a lot of content. Interesting. You can also, to some extent, like sort of see the perception. I think a little bit more. Like, you can ask an agent like, you know, how does, you know, rank all the sandbox vendors by reliability? Or rank all the sandbox vendors by price and sort of see. And it does feel like agents are a little bit more prone to like, mis-understanding pricing and, you know, things like that. So we also spend a lot of time like just making content more clear and more like, you know, understandable for agents. Because we know they're all the indexing. Like, everything that's out there and that's talking about modal. Have you thought about just like making a bunch of GitHub repos and having the agents like write a bunch of modal code in there? Have you tried that? Slow canon. I think the truth is, like, that is probably something you can do. My guess is like LM training data today is probably quite insensitive to like, what's like AI output in the first place. And you can probably like create a lot of slop. Like, we thought about it. Like, I don't love that idea. But like in theory, like, more like, you know, SEO in the early days and still to some extent, like auto-generated content is like a very effective strategy. I think it's possible. Like, you know, you can do those things too. Yeah, I want to switch to FTEs. Apparently you guys hire like forward-to-point engineers. Like what do you do? We're in a new era of like building applications and, you know, things that we couldn't build, you know, just a few years ago. And we see a lot of companies that, you know, they need a little bit of help, right? Like, you know, and so we started investing in FTEs a couple of years ago. But it's really been this year. It was kind of scheduled it up a lot. I think we started this year having like four FTEs. Now I think we have like 24 or something like that. So we've grown a lot. And they do it, you know, it's more, it's not exactly the pound-turn model. But they do a lot of like heavy, you know, sort of customer engagement. Like they work with a lot of customers on performance tuning, especially with LMS. We see that there's so much benefit from like, you know, sitting down with customers and training draft models and tweaking, you know, parameters and DLM and SGLANG. They often, you know, fix bugs or not fix bugs. But like, you know, when customers run into issues, like they often work with customers trying to figure out like, how do we work around that? Or another thing we've used FTEs a lot for is like, you know, kind of proactively, you know, if we see certain customers, you know, over-provisioning with model. Like customers requesting too much CPU or too much memory, for instance. Like, that's kind of expensive. And like, we try to catch those things. And we try to sit down with those customers. Like, yo, like, you're spending a lot of money on model right now. We can actually help you save money. And then of course, like with, with sort of active deals, you know, we helped in migrate code from other platforms, things like that. I think, you know, we never have the intention of becoming like a service-heavy business. I think that the goal for FTEs is like, you know, A is to be a part of, you know, that this is sales process and like closing deals. B is to serve customers and do some like, you know, customer support. But like, I actually think that like the most important thing is like C is like bridging gaps in the product that we later want to turn into like real features. So I think to a large extent, you know, anything an FTE does, you know, my question is always like long term. How do we turn this into products so we don't need an FTE for this? Yeah. Do you guys hire PMs? product managers. No. What would you say these FDs are kind of like pseudo-ish pms? It's a little bit of a different function like I think certainly on the more like you know customer facing side they are but there's like you know on the more like inward you know internal side like this is sort of separate part of like product management which is more like what are the next building blocks models should invest in and like you know and then there's like something in in between like you know looking at like maybe 10 20 different customers they're all asking for like slightly different things but maybe what are the patterns like how do we turn this into product we do have people doing product with us and it's sort of evolved you know we've had people basically like become product managers after previously being either dev rail or engineers and the showing strong aptitude on doing thinking very strategically about what we have to build so I'm actually not like anti product management I do think that a lot of companies in the past hired product manager and product managers too eagerly and I think there's a risk that you kind of I can sort of see it in a lot of products like certain products like you can tell like there's just like eager product managers that'll just thought of like kind of like witty things and just ship them I think doing product management at an infra company really well is very very hard because I think it takes like a very deep you know level of strategic thinking thinking about like what are the foundational building blocks where do we want to be in one to two years because the truth is building infrastructure is very hard and how do we make sure we don't end up with like 35 features that are like kind of similar looking but all do like a little bit slightly different things that sort of level of thinking is something that you know is very rare to find and for that reason we haven't aggressively tried to build up a product management team yet yeah that makes sense how would you say your teams have been evolving maybe like since generally because I would assume yeah I mean you obviously worked out like Spotify you had like a certain team structure um have your team structures like evolved a lot now that you know we have coding agents in a funny way like yes I do think team structures have changed a lot but I'm actually not convinced it was like Jenny I driving that I think in particular like one thing I've seen a lot is that there's been sort of a focus like a swing back towards being in IC that I actually think is very very positive for the space as a whole I think you know if you look back at the space like 10 years ago 20 years ago everyone wanted to be in management because that was the way you you would get more money and more you know more recognition and more prestige and I think that shift actually started even before Jenny I but I do think it's like heavily accelerated with AI like you know we we we make it very clear to everyone at model that like manager RC it's like there's no difference in prestige like everyone is just a member of technical staff you know it's about ultimately your contributions and we see a lot of ex-managers you know becoming ICs and and because they want to go back and build code and so and you know and there's been sort of shift away from like you know non-technical managers everyone pretty including me I still write a lot of code at model and an actual to and and pretty much everyone in the tech team and a lot of people who are not in the tech team still write code so that to me is like you know a recent phenomenon but I actually don't know if it's like necessarily cost by Jenny I but I think Jenny I has like greatly accelerated it because it made it even lower the cost of like writing code is even easier so if you have like non-technical people like committing code like how do you how do you review that like how do you guys do code reviews we do code reviews but it's we think of it as optional I mean for most my my view is like code reviews in large part is like people in many cases trying to just escape the the accountability of like you know merging things into production and if you have that like you know if you have that culture like then to me that's a sign that you need to invest way more in you know cd and and c i and and and things like that is c i in particular but also cd contains integration because it's deployment we do code reviews like I mean like when I write code like occasionally like I'm I'm asking you know authors like is this in line with how you think about things I hope I'm not changing the architecture fundamentally but we also make it clear internally that like code reviews are quite optional and you know if you're just changing docs if you're just doing you know some basic stuff like don't wait for a human to look at especially in the air of AI why do you need a human to look at every piece of code you don't have to but that does require you to have you know very good controls you know internally around you know testing things yeah exactly like you know for companies like we we need one more nine than the average startup right in terms of uptime but like you know these are investments you just have to make like you have to build systems that are very resilient to like you know you have to have rate limits everywhere you have to decouple systems you have to make sure that like nothing is a single point of failure they have to have extremely good test coverage agents are actually really good at writing test these days which was like you know a few years ago I thought like one of the major things missing in agents now they're actually quite good at like because I think it takes a little bit of like sort of like need to step out of what you're trying to do for a second and be almost a little bit like devil's advocate to write good tests but now they're actually quite good at writing tests too so I think all these things help do you guys use like frontier models for coding or do you guys have you guys experimented with like token source once we experiment with everything but yeah I mean you know initially it was a lot of cursor then you know of course clawed and codex and now a lot of people are using you know night shift and all these open source stuff for other stacks so I don't necessarily have like a strong point of view like I want people to experiment and try different things like we you know use frontier models use open source models we're trying to do a little bit more dog floating internally now that we're actually running in friends for a lot of the open source models like a very large scale I think it's also helpful to like you start internal endpoints for a lot of agentic development so we try to promote that a little bit something I read maybe like I don't know a year ago maybe like two years ago you were saying that you didn't think that there was going to be like a big on-prem business for you guys have you maybe like changed your opinion on that no I am not changed by opinion I think model to me I'm a cloud maximalist and maybe to take a step back like when the cloud came in 2008 2009 or whatever my first reaction was like that's crazy like why would I want to take my code and run it on someone else's computer that's like insane and like six months later I was doing it and I think you know like took me six months to change my mind it took probably enterprises like another 15 to 20 years but like you look at you know vendors like Snowflake or I mean AWS and like I think there's many examples of data dog et cetera like the cloud is clearly winning right like and the multi-tenant systems are clearly winning and you know to me on-prem looks increasingly like sort of a legacy business model and and I actually think you know BYOC you know sort of VPC peering like all of these things you know like to me they reflect sort of an outdated security model of like we need to protect everything inside our own network right like using the network as a mode and that I don't think is true anything anymore right like we've sort of moved on to more like actually robust security paradigms like you know zero trust and things like that and so to me BYOC and those things like they look increasingly as like outdated at least from a security perspective from our perspective also like the fact that we have full control over our entire infrastructure stack we rent all the hardware we deploy we manage all the infrastructure we deploy in like manage health and observe you know all the metrics and all the hosts it means we can iterate very fast and build a greater product and I think that's very very valuable versus you know customers in the past who had enterprise additions that they had to deploy on customers environment they had to be very you know they have to think very carefully about release management and and having depend like like we don't care like if we want to introduce a new database like we can introduce a new database because we have control over it like we don't need to think about what customers have access to so to me multi tenancy is is there are so many different benefits I mean that being said like you know if we were to work with like you know some very large fortune 500 customer in the future and you know they ask for BYOC you know we've done this in the past we'll give them a number like you know that number is very high and no one said yes to it so far but like who knows like you know at the right price I was still build it and and I think there's like maybe separately like another thing which I think is interesting which is like in theory BYOC might also be a way to work with and get capacity in certain ways like we sometimes work with Neil Labs and those Neil Labs have already have very large capacity reservations and you know the question is like you know how do we how do we get them to run modal when they already have capacity in BYOC might be a solution we can still keep the control playing in our environment and the other flip side of that is like we could maybe even run like other people's stuff on that capacity so there's some like kind of fun stuff you could do in theory where like you can sort of you know like kind of aggregate all that capacity into one big pool so we're excited about some of those ideas but in general like I feel like on-prem I don't know if we're ever going to do but BYOC like maybe bring your own cloud I can say BYOC I should explain it. Let's say hypothetically you are going to raise I should spend more money like soon what would you use that money for? I mean like I mentioned like in in this capacity situation a lot of the money a lot of the cash flow I should say that's required is capacity prepayments right and so that to me represents the best use of cash today we're ramping up capacity very aggressively maybe we move into you know other types of deployment models in the future ownership but whatever but I think there's always going to be a cash component of that, like some sort of down payment or prepayment of some sort of that. So that, to me, is the greatest use of cash right now. There's, of course, other things like, you know, salaries and office spaces and stuff like that. But that actually, in the ground scheme of like our, our financials, like it's basically a rounding error, right? - Yeah, interesting. - Do you think that's gonna change? - No, I think, you know, the sort of operating leverage that we have in this business is kind of insane. Like, you know, we have 200 employees, you know, they cost a lot of money, but, you know, the GPUs cost a lot more money. - Yeah. If you had to say, like, what are the top, like, I don't know, like, three things that are your biggest focus or the business's biggest focus right now, like, what would you say those are? - Capacity and the sort of financial complexity and, you know, in general, like, how do we, how do we think about that space? How do we think about pricing and packaging? How do we think about the sort of tension between ownership and renting? Should we go in a more neoclass direction? Should we go in the opposite direction? The sort of counter-position? Like, that's probably the biggest thing that I think a lot about. The second biggest thing I think a lot about is probably around, like, more like product strategy. Like, I think a lot about, like, what is the next building blocks, like, if we, you know, do more like, agentic stuff, like, what are the things we need? And, you know, are there certain storage primitives or compute primitives or, in general, like, what is the thing that the world is gonna wake up and realize they need over the next few years? And, and ideally, we have that at that point. We've been pretty good in general at, like, you know, anticipating demand and building products in advance of it with the serverless functions in the sound boxes. So what's the next thing? Then I would say, like, actually, the third thing I think a lot about is, like, more like culture and company building and stuff like that. I mean, we recently crossed 200 people. We're growing extremely fast. We'll probably hit 250 by the end of the year. Like, when you're growing very fast, you have 250 people that are all trying to figure out how they can add the most value to modal and, you know, making sure that they have the right context that they have the right motivation, that they're excited to come into work every day 'cause they get to build something incredibly cool and they know, you know, what makes modal successful, what they can do, aligning the incentives and making sure that, you know, we, we, people have, like, a very positive culture where they have this sort of implicit trust to do a lot of stuff, but also the accountability to, like, you know, deliver the best things and, you know, and be held accountable to, like, if they're not doing it. To me, those are, like, you know, super critical things around this scale, like, when he's, like, 10, 20, 30 people, you can all just, like, kind of sit in a room and, like, it's all kind of implicit and you kind of just, like, feel it. People can sort of read me, like, they can tell me when I get a little bit annoyed. They can tell when I'm happy. Like, now we're 250 people and, like, people don't necessarily see it and, you know, of course, there's, like, a lot of new people that are, like, maybe a little bit nervous to, like, talk to CEO and, like, all this stuff. So, like, you know, how do I remain approachable and how do I, like, it's, it's, there's, like, the Dunbar number, which I think is, like, more sort of, you know, faked than anything, like, at least, like, the story behind it. But, like, it is actually kind of real still. Like, there's something around, like, 150, 200 where, like, laying the foundation of the culture is very, very important. - How many people are you guys trying to hire over the next year? - We've grown about four acts on an analyzed basis in the last 12 months. I think that's, like, an unsustainable growth rate on a relative basis. I've seen that, you know, you can double the team every year and that's not super hard to do. But four acts, what you start to see is, you know, cultural drift, you start to see islands of different working styles. You start to see people doing, like, very speculative things. Like, that's something I always watch out for in the engineering team. You go and talk to the engineer and I'm like, what are you working on? And it turns out they're, like, building something, like, kind of, like, fringe. And like, is this really what you should spend time on? So, growing very fast has that cost and, you know, it has the cost of ramping people up. So, I don't know, I think by the end of the year, maybe 250, by the end of the next year, you know, but like, you know, we're gonna keep growing, like, maybe two and a half, three acts, so it's very likely we'll end up, like, next year, like, 700 people, something like that. - And I'm gonna assume most of those are engineers. - Most of the engineers. - Like, we're helping them sales a lot too. - Okay. - What kinds of engineers? - It's primarily systems engineers. I mean, we've always been, like, very deep into, you know, Linux and file systems and scalable systems and networking and cloud and stuff like that. So, that's the core of what we're looking for. Like, people are really good at those things. We're also looking a lot at, like, you know, more like, 'cause we moved up in the stack a little bit. Like, we're doing a lot of work in the LM, in front space. So, now we're dealing with a separate set of concerns around, you know, CUDA, like GPUs and MFU, you know, like, maxing out the tokens that you can get per GPU and a lot of that things. And, you know, even though I actually, like, we'll say a lot of the work in that space, in my opinion, turns out it's like more systems driven still, 'cause it turns out like KVCaching and routing, it's actually like, it's still like very, like, systems like, like, these are very, like, systems have problems. But yeah, a little bit more, like, sort of, GPU performance, that kind of stuff. We're starting to build up a research team. We're very excited about, you know, moving into stuff around, like, you know, post-training and distillation and better methods for quantization. - For your customers or for internally? - Both. I think it's like a little bit like with the FDEs. Like, I think we initially will do a lot of stuff with our customers to optimize their models. But over time, figure out how do we turn those into products? I think we'll sort of arguably fail in a certain way if every time a customer needs to optimize a model, we have to go and work with them and, you know, long-term, it has to be turned into products. And so, with the FDEs and with the research team, initially, you're gonna have to do a lot of work with the customers when we are at it. But over time, we're also building a lot of internal tooling and a lot of, over time, internal products that are fully self-serve. So we're working a lot of stuff right now in the post-training space, but also in the GPU performance space. So a lot of cool stuff, a lot of cool stuff. - Back to the product strategy stuff that you were talking about? - Yeah. - Yeah, like, what are your thoughts, like, what do we need over the next year, two years? - I think it's incredibly hard to say. And if I said, like, I know exactly what we need to build, like, I think I would be, like, delusional. I think what we've seen so far, you know, and what I kind of believe in on a more general level is like, we need better building blocks. Like, this is a new era of software. And we're unlocking a whole new type of applications that could not be built a few years ago. You know, doing, you know, all the Jenny-I stuff, code that, you know, LMS that output code that, you know, and needs to run, like in an isolated way. Like, there's so many, like, new primitives that are needed to work with GPUs and work with LMS. And, you know, we built some building blocks so far. And, you know, that's unlocked, you know, a bunch of applications that the world, you know, has seen over the last few years. Not just stuff with us, of course. But I think over the next few years, like, just going to be so many more cool things. And so I mentioned some stuff. Like, I'm very bullish on training. I'm very bullish on fast jobs. I'm very bullish on, you know, certain things are on, like, agents and low latency and things like that. But, like, in the end, we consider ourselves like an infrastructure shop, right? Like, we're not trying to build, you know, very narrow things. Like, we try to build, like, very general purpose building blocks and put them in front of customers. And then those customers will combine those building blocks and build some incredible applications that we could have never anticipated. >> I want to go through some of your tweets. This will be okay. >> That's part of it. >> Okay. >> And we can just, like, maybe if you want to double click on anything, a lot of long running persistence sandbox stuff right now. Today feels lift and shift. I suspect sandboxes will all from dev box in the cloud to something more. A femoral one environment for every command and now I'm shutting in between to can pause, resume, clune. Yeah, I mean, we talked about this a lot. >> Yeah, we talked a bit about it, right? Like, I'm fully, I think the median sandbox in modal today is like eight seconds long or something like that. Like, we just kind of incredible when you think about it. And over time, I think it might get even shorter. We want to support long running sandboxes too. Like, I think they're awesome. Like, very valid use cases around long running sandboxes. But like, my vision is like, you know, the world needs also, we need to figure out this. Like, how can we just massively paralyze things and reducing the overhead of starting and stopping sandboxes? >> I don't know if I ran an IPO ready, startup engaged in cat fax arms race for the confederate, it would be nice to coordinate and paste arms race a bit. >> No comment. >> The biggest downside of GPU scarcity is not that it's hard to get GPUs, but that almost all capacity is sold as three years, as a three year rectangle is starting three to six months in the future that's not what demand looks like for most users. I think we talked about that a little bit as well. >> Yeah, and you know, maybe just go a little bit deeper now. Like, one thing I think more is always been very, very good at is fully on demand capacity. Like, if a customer comes to us and need 100 GPUs, like we can typically give them that like immediately. You don't even have to talk to salesperson. And a lot of workloads in real life, like they don't look like these like flat, you know, rectangular workloads. Like, they scale up and down throughout the 24-hour cycle. They maybe grow over time. They maybe have certain burst periods. And to me, that is a challenge with a lot of customers today. Like, and I do think it's a legit problem for the space. It's like the only, the prime way to get capacities to buy these rectangular blocks. It's not great. I think the world needs to, and that's fundamentally, I mean, I'm very biased, but like that's why I think modal can play a very large role. And that's sort of, you know, whatever you want to call it, sort of duration transformation of offering more flexible capacity. And we can do that because we can aggregate many, many thousands of customers. And the man gets very smooth and aggregate. So that's a lot of the value prop of modal is exactly that. - Would you say that's a differentiator? Like, do you-- - I think so. - Oh, really? Okay. - Yeah, massive differentiator. Actually, in fact, like, I think modal's like, if you want to be like very sort of reductive, like you can break down modal's value prop into two things. What is a developer experience? The ability to like iterate and like build things and not have to think about it. for structure, the other thing is the capacity stuff. The more and more I think about model, and where we're going in the long run, is like we're aggregating the world's compute capacity, and we're very good at slicing it out like very granularly, 'cause we've built all this work around, scaling up and down extremely fast, all this stuff around GPU snap-shawning, we're working some ways to make it much, much faster. And that model works at very large scale, like really, really well, 'cause you can aggregate lots of customers. - Every month I keep saying our sandbox revenue might apply to it, but it keeps going up 30% every month. - It's kind of nuts, like I don't understand, like every month I'm like, well, you know, sandboxes grew a lot, but like clearly the kind of plateau, and then they keep growing even more. So I don't know. And a lot of VCs, like they're like, how did you know sandboxes was gonna be big? Like, and there's like, I didn't. Like, I thought it was gonna be bigger than it was in 2023, for sure, but like the fact that sandboxes is a third of our revenue today, it's insane. And you know, we're obviously very happy about it. We have like a very large team investing in sandboxes. You know, I don't wanna like trivialize it. Like we obviously, I do wanna give ourselves like a lot of credit for seeing sandboxes, for what it is today, a little bit, back in even 2023. But I think, you know, I probably would have never, you know, taken a bet that revenue would be this large from sandboxes, and the market demand would be so much. - There's an interesting structural reason why inference providers should exist separately from model builders. The best model will keep changing a lot. And inference providers keep, can easily reallocate resources between models and overall demand is quite smooth. - So I mean, if you think about all the Chinese labs and you know, in the future, hopefully more US labs too, every time a new lab releases, you know, the latest and greatest open source models. Like you see big shifts in demand, you know, for these models. And so I think it challenged with, you know, if all these labs independently like held capacity, they would see when they launched a new model, demand would go up a lot. And then another lab launches a better model and then the demand goes down, you know, but of course, like in the background, they're working on the next version. So like, I think, you know, there's like a structural reason, like I said, like why, you're gonna start, you're seeing demand kind of shifting to like the latest and greatest Chinese lab. I mean, not like, it's not like, you know, this is like everything just moves overnight 'cause a lot of models and prompts and stuff like, like a lot of things that get sort of, there's like a certain stickiness to it. But I do think, you know, there's an argument that it's very easy for model to like, you know, right? Today we're running, you know, X number of thousand GPUs for one model, tomorrow we need to shift like, you know, 50% of that to another model. Sure, that's just like, that's what the system was built to have. - Yeah. - No one even needs to tell us to do that. It just magically does that. - So you agree that basically the LOMs are commodities. And if you agree, then where do you think the business for the big labs is going to be? - I'm not sure if that's necessarily true, right? Like, is Taylor Swift the commodities? Like, she releases a big album and goes in a tour and makes a lot of money. And then, you know, then there's like a down year when she goes and records, I don't know, 'cause it's the best analogy. And then she releases a new album, you know. So I don't know, like she still has like a lot of brand and there's a lot of power and like, makes a lot of margin. So like, as long as the labs like, repeatedly can launch new models. And as long as the margin that they have in those models are such that they can sort of pay off, hammer ties like the cost of training of those models, it could still be like a decent business, I think. - So you don't think that the intelligence is like plateauing? - No, I don't think so. I mean, I think it remains to be seen, right? Like, models are today are like very good at like, doing things that can be trained very efficiently, you know, using reinforcement learning. So we've seen that initially with, you know, coding and writing simple snippets, over time, it's kind of really good at also writing production level code, it's kind of really good at math, it's kind of really good at cyber security. Like these are all like very quite verifiable things. I still think it remains to be seen, you know, our model is going to be really good at whatever, cooking food or, I don't know, like things that I have like more like diffused targets and maybe longer feedback loops or whatever. I think humans probably like, my feeling is like, will be quite good and quite competitive with models for a long time in those domains. But, you know, certainly, you know, AI models, they're getting very, very good as I'm sure everyone can agree with. - Yeah, so obviously like the models are getting really good at coding, they're already really good at coding. What other applications do you think, maybe that aren't as big right now are going to become huge over the next few years? - Pretty low limbs. - I don't know, but I think anything that has like, kind of a verifiable. - A computational bio, I think you mentioned it. - Yeah, a computational bio, as I think those are like, not exactly verifiable, but there's like, certain ways you can sort of run, seem like, you can use an element to output predictions, then you can do like computational like methods to like simulate in molecular dynamics and get like kind of feedback loops. I mean, like, I think like material sciences, like maybe another area where like, you have a little bit of that loop where you can sort of, you know, synthetically manufacture certain materials and measure the properties and use kind of put an agent in the loop and like, you know, do things. So, you know, I have a physics background, I'm pretty excited about that. So, drugs, materials, what else, and drawn a blank, but I'm sure there's many other things. - Cool. I mean, those are all the questions that I had. Is there anything you wanted to talk about that I have in? - I'm good. - Cool. - This is great. This is a lot of fun. - Awesome, thank you. (upbeat music)