The Race to Production-Grade Diffusion LLMs with Stefano Ermon

A big thanks to Blitzy for supporting the podcast and sponsoring this episode. Want to accelerate software development velocity by 5x? You need Blitzy, which brings autonomous software development to your enterprise code base. Your engineers declare intent and Blitzy agents map your code base and generate an agent action plan. Once approved, Blitzy gets to work, autonomously generating hundreds of thousands of lines of validated end-to-end tested code, more than 80% of the work completed in a single run. Blitzy is not just generating code, it's developing software at the speed of compute. Experience Blitzy firsthand at Blitzy.com/twiml. That's B-L-I-T-Z-Y. com/twiml. If you need to scale up these models and they are actually getting into production, the price per token or what needed per token becomes the key metric that you care about. And so what we are seeing with diffusion language models is that they scale better than auto regressive models at the inference time. They are cheaper to serve, they are faster, you get more tokens per GPU, which means that the price is actually lower. And so that's why we felt like this is the time to do it, and in fact that's what we're seeing. All right everyone, welcome to another episode of the Twemble AI podcast, I am your host Sam Charrington. Today I'm joined by Stefano Irman. Stefano is a socioprofesser at Stanford University and the CEO of Inception. Before we get going, be sure to take a moment to hit that subscribe button wherever you're listening to today's show. Stefano, welcome back to the podcast. It has been a while. Yeah, thank you for hosting me again. Yeah, it's been a very long time since we last chat. Yeah, think about eight years or so. Certainly lots has changed and we'll get into some of that in particular what you've been doing with diffusion models. But to get us started, why don't you tell us a little bit about what you've been up to for the last eight years maybe. Yeah, so I've been working still in the same space. So I've been working in generative models. I guess my whole career, my whole life. Now what does change is that they feel really too cough. I guess now it's called generative AI and everybody is paying attention to it and you know it's it's become the thing that everybody is looking at and everybody is trying to get into. So yeah, it's been it's been exciting to see the growth of the of the field and the capabilities of these models when I started back in. 2014 or so, you know, we were barely able to model and these images and it was all like very blurry and that was already like a big result. And now yeah, the bar I shifted a little bit in terms of like what these models can do. So yeah, it's been it's been exciting and you know more specifically. My lab at Stanford has always been kind of like at the forefront of what these models can do and kind of like always been innovating at the model level, at the architecture level, the emulsies level. So I did early work on the fusion models back in 2019 when you know everybody was using generative adversarial networks if you still remember. So we kind of like him up with this alternative approach that is what's now called the fusion model, which is now used pretty much every generative solution for images, video music. And yeah, since back then I've been trying to get this models to work on text and code and DNA like discrete objects and finally been able to get some really really good results with this approach and that's what I've been doing at the section. And currently the CEO and one of the founders of startup called inception where we are developing a new kind of alarm that is based on the fusion and these new alarms are way faster, more efficient, higher quality. So we just launched our newest model mercury to a couple of days ago. And so, you know, if you want to play with a different kind of alarm, something that it's fundamentally different in the way generate text and code give it a try is really really fast. It's a great solution, especially for thinking about latency sensitive applications of our lambs, very tight latency budgets. These models are really, really quick and they get really high quality answers. So a lot of developers are already building a bunch of real time AI applications on top of them. And so that's what I'm most excited today. That's what I've been spending a lot of my time kind of like figuring out who get this models to work even better. Take us back to the creation of diffusion models. Like where do the inspiration come from? Yeah, so back then the field was dominated by GANS generator adversarial networks and you know that's the that old approach where there is to neural network that is one that generates images and there's one that is trying to discriminate and figure out the image or real or fake and then you train them one against each other. And it's a very kind of unstable and challenging kind of optimization problem because they are, you know, there's this gain theoretical kind of aspect to it where you know they need to outcompete each other to neural networks and it was very, very unstable, very, very difficult to get it to work well. A lot of tricks were needed and so we were trying in my lab to experiment with alternatives and one alternative was the usual outdoor progressive approach where you can generate the image. Let's say one pixel at a time and that's never worked particularly well for work for images and video and it still doesn't. It's just very slow and not very accurate and so we came up with this alternative approach which is now called the diffusion model where essentially you generate an image by starting from noise and then iteratively refining it until you get a crisp kind of like nice image that you know is consistent with the prompt at the end. And the key benefit is that the training objective is very stable. The neural network is trying to just denoise an image. You take images you had noise and you train the neural network to remove noise which is a fairly standard relatively easy kind of like optimization problem that you can use to train neural networks over large data sets and it works reasonably well. And essentially you can then use this neural network study inference time to generate images because the networks have been trained to remove noise to improve the samples correct mistakes. And so you can turns out you can just start with pure noise and then you apply this denoising network a bunch of times and at the end you get a really nice image. And the key benefit back then people were not thinking about test time inference and both got this but it has or the one of the reasons we really wanted to get this to work was that it had the flavor of a kind of neural network where you have a very deep kind of inference path because you are training together many many evaluations or of this neural network at inference time. So you have a very deep kind of like computation graph that can do very very powerful things but it's still very scalable during training because you don't have to enroll all of this computation during training during training you just train the model to remove noise. So you just basically need a single neural network evaluation during training. So this idea of having some theories chip to train yet very powerful at inference time has always been something that was on the back of my mind and trying to think about ways to do this sort of computation sufficiently which is now kind of like showing up in a different form in the context of a lab where people are very excited. You know, change of thought and being able to kind of like adjust the amount of compute at inference time. I feel like it's a similar idea although implemented on top of a very different kind of generative model talk a little bit about the path to getting from diffusion models for images to diffusion models for text. Yeah, so that it took a while. So immediately after you know getting the good results on image generation where you know initially we showed that you know these models were better than guns and then very quickly like the field switched to diffusion models and stable diffusion came out. And mid journey and then quickly basically took over the whole field. And since basically back then I started thinking about how do we get this kind of ideas to work for text and code. And we know but we wanted the model to somehow generate discrete objects. You mentioned discrete a couple of times as opposed to continuous can you talk about why that presents a challenge for the fusion models. Yeah, of course. So if you think about you know like an image or just even a single pixel you know it's a you can think of it as a bunch of colors and the interesting thing is that if you change the colors a little bit you know the meaning doesn't change. So in particular you can kind of think about two possible colors for a pixel and all the kind of things in between them still make sense and they don't change the meaning of the image in any dramatic way right. But if you think about text and you take two words then it's not clear what's in between the meaning of two different words right. And so there is no real geometry to the space of possible tokens or possible words. And so that makes the idea of denoising much more challenging because there is it's not clear what it means to perturb the noise to text is not clear how you build. the whole geometry does not exist. And so a lot of the concepts that were defined that were invented to get diffusion models to work on images and video, they were relying very heavily on the fact that there is some kind of continuum of possible images. And you can kind of like interpolate between them. And it makes sense to get the model to kind of like smoothly move from one image to another in the context of text and code. Everything is very discrete. And so it's not obvious how you get the mathematics that were developed for continuous spaces do not translate immediately to discrete spaces. When you talk about the idea of words between points and words in the neighborhood, it calls to mind and bedding spaces and the like, to what degree? I imagine that's been tried or an end or maybe part of the ultimate solution by getting it to work for text. So that can be the there are approaches that essentially try to build the diffusion models for language generation, kind of like in the embedding space. So first you embed everything and you build the diffusion model. And then the problem is that essentially, you have to eventually decode back to text, right? Eventually, you cannot give embedding to your users or your customers. And so that's always the problem. That essentially, at the end of the day, the diffusion model will make some small mistakes. And it might not end up exactly in a point to the corresponds to one of the existing words in the dictionary. And so it's actually pretty challenging to get these models to diffusion models to work well in latent spaces. There's been a number of papers, including from academia, industrial labs. But it's not been very successful, but it is one of the approaches that people have taken. So what has been demonstrated to work for text with diffusion? So the initial results were still sort of like in the academic setting. It was actually, again, from my lab, where we had a paper a couple of years ago, essentially showing that for the first time, it was possible to train a transformer-based model. So you basically took a GPT-2 size model, and then you train it out aggressively the usual way. You train it to predict the next token, the way everybody else is training a lot of labs. And you can train the same neural network as a diffusion model. And in that paper, we showed that for the first time, we were able to match the quality. So in terms of perplexity, in terms of the quality of the text, that these two models are able to generate, it was about the same. But the diffusion model was significantly faster. But you could generate the same quality of text in about 10X less. So 10 times less, sort of like a number of neural network evaluations. So the diffusion models were significantly more efficient at the GPT-2 scale. And so just so I understand the setup there. Are you saying you said train it all aggressively and train it the diffusion? Are you saying is that two different models that you're comparing or are you sequentially training it all aggressively and then with diffusion? So it's really just like almost like an eB test, where it's not very much a fair comparison. In the sense that you take the same exact neural network architecture with the same number of parameters you train it on the same amount of data. You just train it on the one hand as a typical autoregressive model where you just predict the next token. That's how you use it at an infinite time. And then on the other hand, you can train it as a diffusion model. And so at that point, the difference in performance is entirely due to the different modeling paradigm. Diffusion versus autoregressive model. How did you overcome the discrete challenge in training at model? Yeah. So that was the main idea in that paper. Like there were some new mathematics, some new methods of basically figuring out what it means to do diffusion in the context of discrete text-like objects. And then it was demonstrated to actually work well in practice up to the GPT-2 scale. The next step was that we've had inception. And I was very excited about those results. And so I started a company called Inception, where we've been scaling that up. And so we've trained commercial scale diffusion language models, so much larger models, some more data. And now the results are extremely good. Like the latest model that we announced this week, Mercury 2, is actually matching in quality some of the best speed optimized models from Frontier labs. So we'll think about the Haikou models, the Flash models, Mini models from OpenEI. So it's the ADAC quality level. But again, it's about 5 to 10x faster in terms of the time it takes you to get an answer using a diffusion model versus an ultra-aggressive model. Are you able to give us an overview or summary of some of the mathematics that kind of make this work? In a intuitive level, it's somewhat similar in the sense that there is still a neural network that is trained to remove noise. It's just like the noise process is no longer adding small numbers to the pixel intensities. It's more like there is different kinds of noise processes that you can use. One that works pretty well is basically one where you mask out tokens. So you kind of hide them. You take a sentence, and then you remove some of the tokens. You hide them from the neural network. And then you ask the neural network, can you predict what those tokens were? And so it's similar in some sense to next token prediction, except that thing were done out of order. And the network needs to be able to use information from-- you need to use context to the left and to the right and combine it in some interesting ways to figure out how to predict all these missing tokens from the sentence. So in some ways, you're changing the definition of noise to one that makes sense in the context of text. Exactly, exactly. And so actually that kind of training objective is very similar to the birth style models from, again, many years ago. But that was the thing that for a while, was sort of widely used in natural language processing. People were training these neural networks exactly on the same objective. This idea of, oh, let's train the network to predict some of the missing tokens. If it in order to do that, it really needs to understand the meaning of the other tokens. And that's a good way to get through presentations. In that, I see a wallpaper that I mentioned, basically, we show that, well, once you can do that, you can also generate content from scratch. Because essentially, you can start with a sentence where everything is masked. And then you can let the neural network figure out how to fill in pieces. But it does so out of order. So instead of generating left to right, one token at a time, it does it in any order. And crucially, the network can output more than one token at a time. And that's why these models are so much faster. Because in the autoregressive world, if you want to generate 1,000 tokens, you need 1,000 neural network evaluations. In the context of a diffusion language model, the neural network can output many tokens at every step. And so to the extent that you don't need too many steps, 20 denoising steps, then these models can be much, much more efficient. In the image world, I think we're familiar with these progressively enhanced images where you see the image taking shape. Are you able to see the same thing with text? Does text start out horrible and get better over time? Yeah, there is definitely something like that going on. In fact, if you go on our website, you can see some little animations to give you a sense of what's going on under the hood. It's not as interpretable, I would say, as what you see in the image space, where you see really the details emerging as you go through the process. I think text has always been a little bit less interpretable to me. But you can definitely, especially in code, you can see the structure emerging sometimes, at least. You are able to see some interesting patterns in terms of how the model is producing the answer. And for sure, this idea of being able to control the quality of the answer as a function of the number of improvement steps, the number of denoising steps, is actually very exciting because it gives you another direction to do test time scaling, test time inference, use applying compute at the inference time to control the quality of your answers. So if you have an autoregressive model, the only way you can actually control the quality of the answer is by basically producing a longer and longer thinking trace. So that's what these reasoning models are doing. They produce a thinking trace before actually providing the right answer. And the longer you let them think, the better usually the quality of the answer is, and the more expensive it becomes, and the slower, of course, it becomes. A diffusion language model has a different kind of access to do something similar, where you can kind of like control the number of denoising steps, the number of iterations. And the more iterations you do, the higher the quality becomes, but all the edits are essentially happening in place. So you don't necessarily have to make the trace longer. The model is actually able to do other correction. It's able to improve its own answer without having to make it longer and longer, which saves memory and it's significantly more efficient. - When you talk about reasoning models and thinking in the context of diffusion, even beyond this idea that you can change the number of denoising steps, should we be thinking about thinking in the same way with diffusion models as autoregressive models? Like do you still have thinking traces or is all thought, if we're even there yet, is all thought in diffusion models out of band? - Yeah, it's a great question. And in fact, the space of reasoning and diffusion language models, it's pretty new. Mercury is the first Mercury tool, like the model we released this week is the first commercial scale diffusion language model with reasoning capabilities. So it's still this capability, this technology, it's all brand new. In our case, we do still have reasoning traces. They're just produced in a different way and the models have been trained to generate them through denoising through a different training process. But the idea of reasoning trace is still there. And in fact, we're able to provide summaries of the reasoning trace to our users if they want to. So it's actually pretty similar. And in fact, all the API, it's all opening a compatible and you can still use some parameters to decide and to kind of like control how quickly you want your answer, how much you want to trade off compute for quality at inference time. - Another thing that I'm thinking about in comparing these two types of models is that with your traditional LLMs, auto regressive LLMs, to get more tests, you just continue generating tokens because it's auto regressive is next token. For diffusion models, what does that mean? And what are the implications on things like context windows? Like are you doing rolling windows of generation or how does that all translate? - Yeah, that's another kind of capability that it's not, there's many different ways of handling outputs of variable length. We have figured out a way to do it at inception. I'm sure there is also the idea of doing rolling blocks, which also makes sense, has been published in the literature. So there's different ways of handling variable length. It's possible to do it with a diffusion language model. It does not affect sort of like the scaling with respect to context size. That's more affected by the architecture, which is kind of like completely orthogonal to the training objective. So at inception, we're still using transformers as the underlying neural network. And so we're using attention and so we have the same kind of like benefits and downsize our attention in terms of like how it scales with respect to the sequence length. But that's an orthogonal kind of like direction. It's possible to train. And in fact, we have prototypes of diffusion language models where the backbone of the network is not a transformer. It's maybe state space based or mamba based so that you have better scaling with respect to context length, sub-quadratic scaling. But for our main models, we're still transformer based. I guess the question that is coming to mind for me is why now like is this model enabled by particular other things that are happening in the space or is it just the time that it takes or took you to kind of get to this point? Yeah, it's kind of like a combination of both. One is it was just like, it took us a while to figure out how to do it. And it was the right timing to sort of scale things up because finally things were working at least at the sort of like an academic benchmarks, at academic scale. The other one is that people are starting to realize that it's all about the inference scaling, right? So for a while, the main axis that people care about and all the interest was around scaling laws in terms of like, okay, how do things scale at training time, at pre-training time, right? Now everything has shifted to inference time scaling and because of several reasons, one is just like, that's where you're seeing the biggest benefits like RL, post-training, test time inference, but also just like the economics. Like if you need to scale up these models and they are actually getting into production, the price per token or the what needed per token becomes the key metric that you care about. And so what we're seeing with diffusion language models is that they scale better than auto regressive models at inference time. They're cheaper to serve, they're faster, you get more tokens per GPU, which means that the price is actually lower. And so that's why we felt like, yeah, this is the time to do it because if we are just able to match the capabilities in terms of intelligence about the regressive models with a solution that scales better along the axis that actually matters, which is cost and speed, then we would have something that can actually be very, very valuable and that customers would jump to. And in fact, that's what we're seeing. That there is really, really a lot of demand for speed, for cost, and we're seeing also other competitors are sort of like trying to get fast versions of their models, partnering with the, yeah, inference chip companies, a syriprous crocs, and by now, but we're trying to get the fastest models out there, except our solution is software based. So it's much more scalable, we're still running on GPUs, so you can get as much capacity as you can get GPUs for, which is relatively easier compared to specialize their inference chips. - So what degree do all of the techniques that we have learned about and apply regularly now in post training apply also to these diffusion models? - So some do, some, we have to kind of like reinvent from scratch. So if you think about pre-training, mid-training, SFT, a lot of that is actually relatively simple in the sense that you can essentially use the same kind of data sets, rough, the architectures don't have to change too much, and the loss is, you just need to change the loss function essentially, right from next token prediction to denoising. If you think about reinforcement learning, that's where things become more interesting. Because in the context of reinforcement learning, whether you do it from human preferences, or you do it, if you have some kind of like verifiable or non verifiable reward that you need to optimize for the sampling process is quite different, so the way you would propagate that information back into the network is different. And in fact, it's actually beneficial to the diffusion language models, because in the context of RL post training, the real bottleneck is an interest. Like if you're doing RL post training of an LLAM, you're gonna spend most of your time doing rollouts. You get the model to provide a bunch of candidate solutions, and then you score them using your reward function, and then you somehow figure out how to teach the model to do better, to put more probability mass on the rollouts that were good, and avoid rollouts that were not good, as evaluated by the reward function. And because the fusion language models are so much faster at the inference time, then you can kind of really do different things in the RL post training stage, and that's where we're spending a lot of our time right now, which is to figure out what is the right way to do RL post training for the diffusion language model. - In terms of free training, are these models free training from scratch? - Yeah, we're just training our own models. We have our own pipeline. We have not disclosed a lot of detail in terms of how we do it, but we have our own recipe. We have our own stack for training our models. - And are the recipes substantially different from beyond the loss function, or are they like enough is quint? You can kind of see the echoes of the way we train out our aggressive models. - There are some similarities, but I would say, yeah, it's been non-trivial to figure out what is the right way to get these models to work. And yeah, in fact, I mean, there have been attempts over the years to get the diffusion language models to work, including from Google, from other places. And it wasn't, they were not successful for a long time, right? So it is non-trivial to figure out how to train them well, how to get them to scale in the best possible way, to mean the best possible use of the data and the flops that you can have access to. Is it a foregone conclusion that there's no way to, like, does the math, for example, say that there's no way to start from a pre-trained, autoregressive model and somehow, through some magic, transform that into a base that you can then diffusion train. It seems like that would be really interesting giving how much energy and investment has been done. in place to training traditional models. - Yeah, so there's been a number of papers in the academic literature kind of like trying out the various recipes for doing exactly that. And so to some extent embeddings can be still reused and to some extent networks. You know, the real challenges that sort of like the attention mask that you use in a traditional auto regressive model is causal. So the model only knows how to use context to the left as it figures out how to what to do next. And in a diffusion language model, you really wanna be able to relax as to the context to the left and to the right as you decide what to change. It's like one of the key properties that make these models potentially much higher quality than compared to auto regressive models. And so that's like the the challenge a bit and people have explored ways of kind of like annealing the attention mask to make it go from causal to something non-cozal, slowly kind of like making the model drift away from the initial auto regressive thing that are more mathematically sophisticated ways of kind of like converting the likelihood from an auto regressive model to a score function which is what you need in the context of a diffusion model. More broadly, I think one thing that always sort of works is that you can get samples from the auto regressive model that's usually a good way to at the very least generate synthetic data that then you can use to train your diffusion language model. That's almost like a black box that you can always kind of like use to combine and try to get knowledge out of an existing model. And as we know, I mean with the distil gate, I think that's the that has been on the mind of a lot of labs and a lot of researchers. And it seems to be something that is really going on in pretty massive scales in other places, but that's always possible. How does the serving setup change for diffusion models? Yeah, that's a great question. And it's another pretty challenging kind of like aspect. And I think one of the reasons why there are still no other providers that are able to serve diffusion language models in production today, you cannot run a diffusion language model on existing serving engines. So you think about BLLAM, SGLAM, TensorRT, these frameworks that exist and I know I've even opened a source and they are really, really good at serving auto regressive LLAMs very efficiently. So they would handle things like continuous bashing for you. And there is a stream of requests coming in. How do you batch them together to serve them efficiently? And there is all kinds of optimizations that you need to do. Once you have access to multiple GPUs and many requests and there is a lot of existing frameworks and great work that has been done for auto regressive models. The space for diffusion language models is much, much less developed. So we had to build our own serving engine. Over the last maybe month or two, there's been some support for diffusion language models in SGLAMG for the open source models that have been open source diffusion language models that have been developed by the community. So there is starting to be a little bit of ecosystem, a little bit of tooling, a little bit of community support for diffusion language models in the open source community. But it's still not nearly as developed as for auto regressive models. You talked earlier about the ability to change the number of refinement steps and how powerful that is in diffusion zone type of inference time scaling. Is that something that is currently-- you know, I'm thinking on the static to dynamic spectrum. Is it fully dynamic? Is it fully static? Is it per request static? Like, how do you think about that the knobs there? Yeah, so it's a design choice. I think to some extent, it's a choice that as developers, we made to figure out how to expose this kind of functionality to the user. So right now, our Mercury models allow you to select different kinds of efforts, essentially. So we try to basically kept it compatible with the existing auto regressive, open AI kind of like frameworks. So that's very easy for people to essentially plug in our diffusion language models into their existing apps or IDEs and they can just be used. Similarly, you just need to change the API key. Everything works out. So we still basically use the reasoning effort parameters to control how much compute is used under the hood. But potentially, you could think about alternative ways of exposing the knob. It's just like as there is already a very well-developed market right now. And so we've tried to be-- for us, it's very important to be backwards compatible so that our customers can very quickly kind of like switch out whatever they were using before to add diffusion language model. It's very easy for people to try our models and see how fast they are. And so that was kind of like a design choice that we made because it makes it easier for us to go to market with the diffusion language models. Beyond Speed and Costs, which are these metrics that we've talked about, there's also quality. And are you giving the user all three or are they sacrificing? Were they sacrificing? How do you know or how did they know what the sacrifices are? And what's the strongest evidence you have that the speed gains survive under real production load at an acceptable quality? And I think it boils down to-- there are three things that matter when you think about our LAMs, its quality speed and cost. It's always the trade-up between those things. And you can actually plot this where existing LLAMs stand in terms of these three things. And that's what you find if you go on artificial analysis or you look at the providers that are kind of like benchmarking LLAMs in terms of the capabilities that they have in terms of cost, price, cost, speed, and quality. And so we've benchmarked our models using this existing methodology. And so of course, measuring speed is easy. Measuring cost is also easy. Quality is always the hard one. What does it mean that the model is better than another one? It's very tricky to actually measure quality in a good way. But the way it's usually done is there is a number of benchmarks that have been established and that people try to measure things that people care about. Coating ability, a question answering, instruction following, stuff like that tool use. And so what we do is we basically just compare the quality of our models on these existing benchmarks. And we've actually given our models, for example, to artificial analysis, or artificial analysis, did their independent evaluation. They tried the model on a bunch of benchmarks that they use to come up with their own intelligence score, which is exactly like a quality metric. It's basically trying to see how different models compare in terms of their capabilities on these benchmarks, which reflect kind of a real world use cases. And again, the result is that our latest diffusion language model, Mercury 2, is comparable in quality is about the same as the speed optimized models from frontier labs. So high-cose, meany, flash models, but significantly faster. 510x faster, depending on which one you compare against. So the big limitation is that it's not the highest possible quality. So if you have a workload where you want to have the most intelligent model, like the latest opus model, or the latest pro models from Google Gemini Pro, something like that, we are not at that quality level. So we have not yet trained a diffusion language model that matches the quality of the best models from frontier labs. That's kind of like the key limitation at the moment. So we've been able to show that we can shift the parity of frontier of quality versus speed at the level of the speed optimized models from frontier labs. But we need to do more work to basically keep increasing the quality of our models, train bigger diffusion language models, use more data, figure out better training techniques to close a gap. And at that point, we would have something really, really, really valuable. Do you find when you're comparing these models empirically, do you find any qualitative differences between the types of generations that you see? We heard anecdotally from our users and customers that yeah, it does feel different, but it's hard to quantify again. And use the benchmarks as a good way to measure how well these models do. There are some that I think are basically editing like tasks, everything about autocomplete or edit suggestions. That's the kind of task where intuitively, you can kind of see that you really want to be able to use context to the left and to the right. Maybe if you're doing autocomplete in an editor And indeed, we're seeing that the Schusser models do really, really well. well. So, you know, there is this thing called kupai lotarina. It's kind of like the LM arena for code generative models where there is basically an yellow score that they come up with for code generations. So it's literally an IDE and developers get to see auto complete suggestions from two models. They don't know which them what the models are and then they rank them, which one is better. And we are the top of that ranking in terms of like the quality of the completions that you get from a diffusion language model. It's really, really fast. So we're seeing a lot of views right now. Our models are already embedded in a number of IDs. Continue Z a bunch of others, he look code and and and yeah, developers are actually, you know, loving the the experience, the quality of the of the generations that we get and the speed that we can provide. Are there areas where diffusion struggles relative to traditional models and not, you know, granted at a consistent like, you know, tier like if you're comparing the smaller faster models to mercury to, you know, like long arising for here in turn, you know, needle and a big haystack or like, yeah, so the context that we are mercury to model has 128 K context. So that could be, you know, that's, you know, if you have a task where you maybe need more than that, that's again, probably not the best use case. Again, I don't think it's a fundamental limitation of the fusion language models. It's just like we haven't trained models with longer context. We're not multi model yet. So that's another limitation, at least right now. So if you have a task where you need the vision input, so you're thinking about audio or outputting images and video or something multimodal, we, yeah, we do not yet support those kind of functionalities. I mean, there's no technical, you know, fundamental reason we cannot do is just like we didn't have the time to train them multimodal models yet. In terms of getting to a larger scale, like, what does that look like for you? And what are the key, you know, impediments steps that kind of thing? Yeah, it's, you know, it's a process. So like it's a new technology. And so a lot of the time we still have to reinvent new things. And it doesn't make sense to do all the R&D at the largest possible scale. So it's, you know, we can iterate much more quickly. If we try out our ideas, try out our methods at, you know, medium scale, small to medium scale, kind of like models, nice, just because, yeah, iteration is faster. And so there is still a lot of R&D to be done before we kind of just okay, let's just scale up, right? And but fundamentally, yeah, it's a, there's some science questions that still need to be solved. Then there is engineering, of course, every 10x in data parameters comes with a lot of engineering challenges. And it often means that you have to change a lot of the infrastructure because there are a bunch of new problems that didn't show up at that previous scale, then now become, you know, important at the next scale. And so as we go through this process, we're learning a lot about, you know, scaling up to much larger number of GPUs and bigger datasets and kind of like various kinds of engineering problems that any first structure problems that, you know, that they are not, there's a lot of technical risk. He just takes time to figure out how to, you know, get come up with a solution in turn. He talked about some of the open science questions. Yeah, it's still pretty open in terms of like, you know, what is the best way to train one of these models, right? What is the right noise process? That there's many choices there, we have some things that were, but there could be better ones. If you think about the inference, the interesting thing about diffusion language models that training an inference or decouple. So in an auto regressive model, you're trained to predict the next token. And then at inference time, the only thing you can do is to basically reuse exactly the same process over and over. In a diffusion language model, you're essentially solving a differential equation to, to generate samples. And at least for image and video generation, there is a, there is a lot of methods that you can use to accelerate sampling. A lot of techniques from the, a lot of numerical methods techniques like fancy, ODE, ordinary differential equations, solvers, or stochastic differential equations, solvers, a lot of those techniques have been ported over to machine learning and they've led to really, really fast and high quality sampling algorithms for traditional continuous diffusion models. The space of discrete language, diffusion language models, it's, it's, it's still the wild west. Nobody knows what's the best way to do things. Um, architecture wise, I think there is still a lot that can be, that can be changed. If you think about RL, what is the right way to do RL using a diffusion language model, even in the context of just traditional image and video models, there is still a lot of research that's still kind of like wide open, what is the best way to incorporate the information through the diffusion process, what's the most efficient way of doing it? I'm still involved through my lab at Stanford and some research projects there, some collaborations with Nvidia, where we know we're training big video models, Cosmos, we've been working trying to figure out what is the right recipe for RL on this more standard diffusion models that have been around for six or seven years, the space for language, discrete, it's, it's, it's still much less mature. And so there is a lot of research to be done there as well. And presumably, because you're ultimately based on transformer models, all of the limitations of traditionally trained LMS are similar in diffusion models, is that the case? hallucination is one that comes to mind, for example. Yeah, so hallucinations, yes, I think it's not necessarily an issue with the, or is the way I think of it, it's not necessarily, probably the architecture, I think that's just like a fundamental issue, whenever you fit a statistical model, right? You know, there is data you're feeling a statistical model of the result of a gene where you're going to be interpolating and you know, maybe the answers that you get are going to be reliable, and there's always going to be a regime where you're going to be extrapolating and at that point, you know, there's going to be mistakes. And I think that's a strong extent unavoidable, no matter whether it's a diffusion, autoregressive, no matter what is the architecture, we're learning from limited data, we need this model to generalize. And yeah, generalization is very, very, very, you know, nobody really understands it, basically, that's generalization of deep learning work. I mean, even for classification, but there's been, you know, people in the M.L. theory, very, very smart people have spent a lot of time trying to understand how does generalization work in deep learning, the progress has been very, very limited. Like to this day, you cannot, there is no predictive theory that can tell you, will this neural network generalize? It's all very empirical, you have to try and then, and then you know, you know, did it work or not? But there is no theory that is, or that is at any reasonable scale that people would care about, that is predictive and that will tell you how well neural network working practice, even for classification. For generation, generalize models, it's even worse. It's a problem that fundamentally should be impossible to solve, right? There is a, there is curse of the missionality, like we have, there is some pretty good arguments for why what these models are doing should not be possible, yes, they work. So I feel like that it's, that it's a, it's a fundamentally missing there from a, from a scientific point of view in terms of like understanding how these models work, why they work, under what conditions they will work. It's still very, very open. And how about things like explainability or the model's ability to estimate its uncertainty? Are there any, I guess some looking for any fundamental differences, either to the benefit of diffusion models or to transform based models, you know, in these kind of, you know, core dimensions? Yeah, so we've not explored much interpretability. I would not expect particular differences in the sense that it's yet again, one of the spaces where if the moment you start using deep networks, I think I'm personally pretty skeptical about the whole interpretability, spatial direction. And so there's not something that we've invested in at the moment. One interesting direction that I think is actually exciting and it's also practically relevant is controllability. That's a space where people do care about being able to control the outputs of these models. And usually that's known through a prompt, maybe some guard, guard rails at the end, like the reason there's a there's a certain stack and a certain set of things you can and cannot do with another aggressive model. A diffusion model, at least for images, diffusion models are known to be much more suitable for controllable generation. And the reason is that because they're all object, let's say the image that you're generating is sort of like available to the model from the very beginning. It's very easy for the model to check whether or not this object that it's generating is consistent with say some constraints or some kind of some kind of a control signal that you want to use to to make sure that the output is consistent with whatever you want the model to generate. And not only you can check whether it matches your conditions, but you can also steer the generation process in a direction that makes it consistent with these external constraints. And that's only possible because you have the object from the very beginning, the full object, as opposed to generating it token by token, where you can only check whether or not it satisfies the constraint at the end. And so that's why diffusion models have been used a lot as priors for solving inverse problems, medical imaging, like there is a lot of applications where disability of controlling the output through some external signal has been really, really important. So I was on some papers where we were doing medical imaging and the idea is that you know into the CT scan, you're basically taking some projections of your body cross section and then you're trying to reconstruct what your body looks like from some measurements that you get from the machine and the more measurements you get, the higher the quality of the reconstruction, but it also means more radiations for the patient, right? But if you had a good prior model of what the body looks like, which can be given by a diffusion model, then you can kind of force the model to say, okay, produce something that is likely to be the, you know, the to correspond to an actual human body, but it's also consistent with these measurements that I'm getting, you know, for this particular patient. And that can significantly reduce the number of measurements that you need to take for the same quality level, which means less radiations for the patients. And there's a number of problems that kind of like have that flavor where the diffusion models have been really, really good. And so now how to do that for tax, there's some work again there, but that would be pretty exciting. I think people care about being able to, you know, stay on brand or be, of course, safety constraints. Like there is a bunch of settings where you do want to be able to control the output of the model. And so I think that's that's an exciting capability that is pretty unique to the diffusion of language models. Looking forward, what's your kind of mental timeline for maybe I should even ask this, maybe even more in a more open and a way, do you ultimately see diffusion challenging auto regressive models at frontier scale? Yeah, yeah, I think that's our bet. I think there is no reason it shouldn't. I don't know how long it's going to take us to get there. And I think the challenge a bit is that the frontier keeps moving. Like you should tell me, you know, this is the frontier. How long do you need to get there? I think I probably come up with a reasonable estimate for that. The problem is that it keeps shifting. And so the models keep getting better and the speed keeps accelerating. So it's hard to predict how long it's going to take. And again, there is still a lot of R&D, unfortunately, which has a lot of risks, but also a lot of upside. It's entirely possible that we come up with a new algorithm that is way better than what we have. And so that could accelerate progress by a lot, especially because the diffusion language space is very, very unexplored. I think there is still a lot of low hanging fruits, a lot of room for improvement, a lot room for wildly better solutions to what we're currently doing. So it's hard to predict how quickly it's going to take. It's hard to predict how well it's going to work if we were to scale up to those sizes. What's exciting is that it's unlikely that one architecture is going to dominate the other one. So maybe the best case scenario is short. The future models are better. Even if that doesn't happen, there's got to be some use cases, latency, sensitivity, phone device. There's going to be some use cases where an alternative architecture is just going to be better. And it's going to be such a big market that even the worst case scenario is actually pretty good for us. And obviously, there's just so many use cases of these out-elabs that as long as we come in on a reasonable subset of them, that's still going to be extremely valuable. Yeah, when I think about latency sensitive, the things that come to mind most immediately, things like voice interactions. But then I think about all the activity around agents and how they're running a loop. And anytime you have looping, if you can compress the time for one run through that loop, then that compounds. Are you doing a lot or seeing a lot with regards to these diffusion models and agentic applications are they powerful enough to be used in agents now? Absolutely. Yes, yes. So we're already seeing a lot of usage. I mean, you nailed it to main ones that we're seeing. Voice, a lot of voice, customer support, the educational kind of like agents. People love the speed of the diffusion language models. They always had this issue that they would want to be able to use a thinking model, like a reasoning model. But usually the latency is just not enough. And so maybe they use unless they use specialized AI infrastructure, but that's too expensive and it cannot scale to large volumes. So we had a bunch of customers that are building voice agents on top of the fusion language models and agents. That's another one, just general agents. Mercuryity works actually pretty well in open-cloth, for example. So yeah, you can use it if you're in plug it in. It's already, yeah, you can use it also for coding, client, a kilo code. So it's all, you know, it can use tools, it can reason, and it's really quick. So, you know, especially kind of like, it's not the best model. If you're thinking of, okay, I'm going to let it run for 24 hours. I'm going to come back and see where it solves my problem. Then it's probably not a good use case for that. But if you think about fast agentic interactions and loops where you're actually there and you want, you want to be able to get an answer quickly and there is a human in the loop. Then it's a really good model because as you said, it's significantly faster. You can iterate more quickly. And so eventually you can get to the final result in less time, which is the thing that actually matters to developers. I think last year, I think it was in Google I/O, Google announced and kind of previewed, you know, their play in the space. I don't know that I've seen much of it since then. Have you tracked what they've been up to? Can you give us a summary? Yeah, so I mean, I don't have any exciting information in terms of what they're doing. But, as you said, they also announced the diffusion language model, Gemini diffusion. A few months after we announced our first Mercury model. So I'd like to think that maybe that had a little bit of an influence and I hope that we're pushing it to actually show they have something to do. What they published back then, like those numbers, were very comparable to our initial Mercury one model. So I don't know whether it'd been able to improve. What I know is that it's not yet in production. And this is not yet available to customers. So I'm guessing they may be not figuring out or they're still working on figuring out how to survey efficiently and what are the best use cases. My sense is that there is a big switching cost. They're very focused on Gemini and then their main model. That's the issue with these big labs is that they're all in in one direction and then it's hard for them to really focus on an alternative direction. As a startup, we're in a much better position to do that because we are laser focused on one thing and we can really deliver and build everything that's needed to get that technology to succeed. But it's a little bit harder to do these things in this place. I think in a company, big lab, I think they're already kind of like, have a direction set and there's a big opportunity cost if you want to switch. What are the labs or teams, academic or industry do you kind of keep an eye on for doing interesting things with diffusion? Yeah, so there is a lot of good work coming out from China like the Lada models. These are several Chinese universities collaborating with Alibaba. So that's the, I think they get all the computing funding from industry and they're doing good work in terms of thinking about models, architectures, how to train them. There's still a huge gap between these Lada models and what we have internally. But they've been doing good work in terms of doing research and pushing the field forward. Bites dance has a pretty serious effort internally. They've also at least published a few papers with some internally built diffusion language models by Dan Seed, which is kind of like a fundamental research group within Bites dance. A lot of smart people, a lot of good researchers, they've been doing good work in this space too. And then generally, I think the whole academic community, there is a bunch of interesting papers coming out. I was under a rips in December and yeah, it was crazy to see how many papers are there on diffusion language model. And if you were to plot it, you'll see that there's been an explosion since that original paper. from my group in 2004. Now everyone is kind of like looking at this new part of that. And of course, it's exciting because it's a, there is this approach that works really well for image video and music. And this desire approach that works well for text and code and then what's going to be the winning solution is that our way to, you know, this that our way to unify everything and have a single kind of generative model that works well across all modalities. And of course, everyone is excited about other lamps, but it's surprising how similar all the different models from frontier labs are. They're all kind of like clones of each other. There's very, very little differences. And so now there is an alternative approach, an alternative path. So of course, that is generating a lot of excitement in the research community because that's an opportunity to do something new, something at the frontier, something to have really, you know, impact on the conceptual foundations for this approach. Do you see image and text as kind of these two divergent paths or are there techniques, you know, with image being further ahead of the techniques that are created on the image side that you, you know, can pull over or have pulled over to facilitate your work on the text side? Yeah, yeah. Though those are, there is a lot of cross-boilination, I would say. And I myself started out working on images. A lot of the researchers you now team because there was not really a set of researchers that were working on diffusion for language or not that many, a lot of the people on our team actually started out as just, you know, pure old kind of diffusion for images or diffusion for video and researchers. And so a lot of the know how they didn't did transfer reasonably well. And so yeah, we also do pay attention, close attention to what's happening in that community, in terms of like distillation, techniques to accelerate the models. As I mentioned before, inference tricks to make diffusion models go even faster. So all those advances have been pretty exciting. Is there any work happening either, you know, inception or elsewhere that points to an approach to kind of a credible multimodal approach based on diffusion? So yeah, at inception, we've not been prioritizing a multimodal yet, but in the academic community, there's been a number of papers that have come out over the last year or so, including from one of my co-founders, Aditya, who was a former PhD student with his lab. He's done some really, really good work in terms of like showing how to build the diffusion models that are truly multimodal. And so there's been some really, really good results on the in the academic space on getting that, unifying model based on the fusion that can handle the multimodal. Well, Stefano has been great catching up with you and getting a complete download on text diffusion. I feel caught up now. Thanks so much for jumping on and sharing a bit about what you've been working on. Yeah, thank you so much for hosting me. Yeah, it was really fun chat. Same, thank you. (upbeat music)