How Cursor Trained Composer on Fireworks: Distributed Infrastructure for High-Performance RL

You need all the infrastructure to run these environments that have to mimic as closely as possible what a user's computer would look like. And it's very important, it's closely as possible because sometimes the model can actually figure out when it's being run in like a fake environment or a real one and it has like different behaviors during a rail than in production. Are you saying it being conscious so that it's being in a fake environment and it starts behaving differently? Yes, interesting. Like it's like, oh, I'm in a fake environment, I've learned a few tricks to like get the better reward in this environment and let me try them out. Models love to cheat. Our others really good at encouraging cheating. I'm delighted to welcome Federico from Cursor and Dima from Fireworks with Podcasts today. Federico, you are the research lead on Composer 2 at Cursor, Cursor's new agentate coding model. And Dima, you spent how many of the last few months moonlighting at Cursor in order to support a lot of the infrastructure required to make this gargantuan training task happen. And so I'm excited to talk to both of you today about how the training of Composer 2 came together, what hard problems you solved together and what you think it means for the future of a AI and foundation model companies. Exciting. Yeah, excellent. Thank you. Have a nice. Thanks for joining. Okay. Let's dive right in for those who haven't been following us closely. Cursor recently announced Composer 2, which is an agentate coding model meant for long-horizon coding tasks. Federico, up till now, Cursor was mostly enabling other people's coding agents. What was the impetus for Cursor to lean so heavily into Composer 2 and how existential is it for you to become not just an application company but also a foundation model company yourselves? The reason why we started the looking into training our own models is you can sort of think about the model as sort of like a storage drive. It has certain amount of bits that it can store in its weights. And the idea is very simple. You know, like we hear about only one task. We don't even care about coding or programming necessarily. We care about software engineering inside Cursor and inside Cursor only. And so what if we were to allocate all of the bits of information that can be stored inside a model weights to that one particular task? Also as people may have noticed, Composer is order of magnitude the less expensive the no-puss and other like coding models because we can just simply specialize all of the model weights to that particular task. And so we can serve like as a model or something of that sort. So it's about let's make sure every single bit of weight or information we have is dedicated toward the specific problem that we have at hand. Exactly. Got it. That seems like it's an almost generalizable problem. DMI I'm curious your perspective. Do you think that every application company should be looking at Cursor as a harbinger of what's to come? Should they all be looking to do the same thing? Yeah, absolutely. I mean, we actually generally see it as a pattern of evolution of their applications. You may be start prototyping, you might be using kind of of the shelf model to get something running, maybe do some frontage year in, figure out how your harness works. But the most leveraged attribute of your application is actually usage of user data or particular specific aspects of how the application works. Maybe some aspects of your harness, which those do provide, how the application works. Really important bits, which are important for your application and the right way to capture that. You can do a little bit of that through prompting, but really the right way to do this is craft your model to act in your environment. Yeah, absolutely. Like there are certain tools, the agent calls that it's very hard to succinctly describe exactly the behavior of that tool to the model. And you know, we've just like post training, we can bake in the optimal way to use those tools. Like composer, we do serve a prompt to composer, but I think the way we are training it, it would work even without a prompt and it would know what to do just because we are intrinsically pushing the model to the right direction of how it should act throughout our training. Basically there's a per bound of how far you can get this prompt in G-Rin. If you want to craft really great AI products, you have to go through faintly in influence of what we have here. That's one reason. Reason number two is what the recommended is, kind of cost trade-off or XP trade-off. The way we kind of view the fireworks is that when you're trying to do optimization, you have this like three dimensional trade-off between quality, speed and cost. And you can go quite far and we are doing this a lot of cost-vers initially. You can go quite far with just optimizing infrastructure, but when you start getting the model trading, you can really push this trade-off much further and you can get better model at fraction of the cost running much faster. And you know, composer is a great example of. Can I push on this a little bit? I want to ask you if this approach is that they're less in pills. And we were actually all talking about tab nine on the walk-in. I'm remembering before the LLM era, there were these like small specialized coding models. And one of the things that was I think surprising to a lot of people was as you've scaled up, you know, you've scaled up just training on the internet and a lot of bunch of English texts and other languages. Actually, the models themselves got inherently better at coding as well. And so at least the trend line I've seen so far is just like bigger models perform better on everything, including on coding. Is what you guys are saying, does that go against the grain of the better lesson? I think no, but one sort of thing to point out is that the big models trained by the labs train on a lot of code as well. Like code is one of the main tasks the labs are interested in pushing. And so they don't just generalize to it. They're a bit specialized as well. I think for our case, actually, you know, if we believe about the bitter lesson, we are just pushing very hard on the data dimension and we know that the models inherently have finite capacity. And so if we want to saturate all that capacity, we need to scale data. And in order to ingest more data, we need to like free up the weights from distractions the model may have. Okay, got it. Super interesting. Okay, let's dig into the training of Composer 2. You launched a couple weeks ago, immediately grabbed attention. One benchmark number is much lower cost to run in front of. What's the short version of how Composer 2 works and what you guys did to make it so performant? We started from a very strong base, which is Kemi 2.5. It's like a one trillion parameter MOE that's a 32 be active. So very, very sparse, actually. We sort of like looked at the stack and realized there are like two axes. So mainly Composer 1 was just pushing one of these axes, which is reinforcement learning. By Composer 2 pushes in two different axes, one is continual pre-training and the other is reinforcement learning. So the thing that made Composer 2 very good is pushing in both of these directions. So we started off the training run by doing lots of mid training on code tokens, almost sort of pre-training scale, actually. And then coming out of that mid training run, we took the checkpoints and we did the very large scale around on lots of lots of tasks. Okay. And then the premise here would be because cursor sits in the middle of so many interesting coding tokens, you actually pretty uniquely have access to data to be able to train at almost pre-training scale. Yeah. Why not pre-training your own model then? We just think about our approach from top down instead of bottom up. So how do we get a model that's useful to users in the least time possible? If we were to start from the bottom, sort of figure out how we do pre-training and then scale it up to meet the training and then, okay, now we figured out me training. We do reinforcement learning that would take a very long time to get a model out to our users. By doing it the other way around, we were able to give a useful model to our users in very little time. So hopefully, next composer versions are going to be our own model instead of basing it off an open source base. And what is the model roughly learning in the mid-training step? What is the model learning in the post-training step for you? Yeah. So in mid-training, it's just kind of learning about libraries of code and learning about specific code patterns that are very common. Just world knowledge as well. There is web data there as well. And this is sort of just creating a wider distribution that then reinforcement learning can sharpen on. And so during reinforcement learning, the model gets to play directly with the cursor harness. And so it gets to learn about the world the model is going to live in for the rest of its life, right? In some way. And so then during reinforcement learning, that's where it learns how to call tools properly, how to navigate its environment, how to write correct code. Because during mid-training, it learns how to write code. That doesn't necessarily mean it learns how to write correct code. We try to train on code that is largely only correct, but the model doesn't actually know how to differentiate between the two. While in RL, one of the key things that we are doing is we're kind of tuning the feature of the model saying, hey, now you get a right correct code all the time. Exactly. And is the model after mid-training? Is that similar to the model that you guys have on tab auto-complete or is it a different core competency? Yeah, I mean, I think I would put it like that because during mid-training, we are just doing the next token prediction, how well you predict the next token and then the token after that. So why not just post-training your tab auto-complete model then? Why mid-trains the different model? Yeah, I mean tab is a very small model because it's like a super low latency model. We want it to be very fast. So like the core to distinction about. the base models here is that tab is like small and and composer is quite large. I see I see. Okay so it seems like a lot of the focus of what you guys did for composer two was this large scale reinforcement learning run. Can you break that down for us like what goes into that and what are the various hard problems you solve along the way? When you do our relics quite different from from like pre-training of betraying because you know just trying to predict next token you're actually running the entire harness like the entire experiment you're letting the model act in the environment see how how it performs for a given rollout that's the terminology which is called rollout and kind of assign the three word whether it did something correctly or not which might be some L using LM as a judge or maybe something very viable like does the Scott compile or something like this which actually means that compared this regular training you need a bunch of other components like you still need large scale training you still need to orchestrate tens of thousands of GPUs to the forward backward propagation though all the stuff you do in mid-training and pre-training but now you also need to orchestrate a bunch of environments you need to run model inference because when you do this this rollout you effectively running like real cursor session in some sense right so you have you know no rollout is basically you entire like agent session from cursor right so we basically means it might take something like 50 turns a model will take a take you initial prompt then besides call some tools you want to execute those tools then model it generates a bunch of other codes kind of entire session which when you interact with the agent in cursor right you you kind of simulate this entire session as a part of your training run you get to final reward and use the you get use the signal to now go back to trainer and incorporate it in the model weights so you have this kind of very big loop update loop which is very heterogeneous right because you have all this like different components working together and now you're trying to orchestrate all of this to work efficiently it work with high throughput because GPUs are expensive and you want to get your model trained quickly in an economic fashion so that's by itself is like very interesting kind of problem and intersection of algorithms and infrastructure because there are a lot of trade-offs how you can kind of co-optimize and co-design the system one aspect is kind of people call about like this as in-carrel of pipeline or all the idea is basically okay you're trying to update this model in steps right so you have your current model version and you are trying to do a bunch of rollouts visit what does your trainer do while you're doing this rollouts right like native approach would say that okay now I'm gonna stop my trainer I'm gonna do a bunch of sessions and those sessions my run for like five ten minutes or even longer if it's like longer horizon tasks I'm gonna get those outcomes and now I'm gonna pause my inference when I go back to training trying to do updates that's like very theoretically algorithmically robust because you are not precisely simulating everything but it's very system inefficient because half of your capacity is sitting idle as time so you can do all the clever like algorithmic tricks allowing you to yeah you can like kind of pipeline all of this so imagine this as a gigantic like factory right you have this like trainer building and you have a rollouts building they're always churning right there so rollouts always take like latest model version and try to do new sessions and kind of simulate new agent sessions and trainer always takes new outcomes as they come and try to compute updates so everything is moving along all the time the tradeoff is that why I'm saying that algorithmically difference because now by the time you finish some test rollouts in your kind of simulators environment maybe model weights already updated on some other data so you have this kind of staleness like delay between how quickly model can learn updates because by the time you kind of process or some interaction session with a simulators environment your model base changed and that introduces interest in training dynamics and there are clever ways how you can address this but the flip side of that is that you're all uGPUs all your computers kind of law that and chime in all the time which actually you're using more more flops and to your bitter lesson example yeah you you have like higher computer efficiency you can get to a better model in smaller amount of time yeah maybe you're losing a few percent from being asynchronous and not doing like perfect mathematical updates but you weigh compensate for that by effectively not leaving healthier capacity on the table and there are a lot of kind of depths and interest interaction in that part and we're very serious about performance at cursor because unlike the big labs you know we have tons of thousands of GPUs not millions and so yeah we do all sorts of tricks to make get the most out of GPU like we train in production with fp4 even we work with fireworks to like push on inference as well because the thing about a rally infrastructure is just like it's just inherently more complex than pre-training because you need all the pre-training infrastructure that's just like one of the requirements then you need all the infrastructure to run these environments that have to mimic as closely as possible with a user as a computer would look like and it's very important it's closely as possible because sometimes the model can actually figure out when it's being run in like a fake environment and they're a real one and it has like different behaviors during a rail then in production are you saying it being conscious that it's being it's an affect environment is starts being behaving differently. Yes, interesting. Like it's like oh I mean a fake environment I've learned a few tricks to like get the better reward in this environment and let me try them out. A rally is really good at encouraging cheating. Yeah and then we need a really efficient inference so this is really important so there is like actually this kind of myth that during a rail you spend more way more inference flops than training flops this is sort of like just because the open source inference engines are very unoptimized instead of actually being a property of a rail roughly the same ratio is kind of the same interior if you push the GPUs to the maximum you should have one third of your training GPUs are located to inference right because training is effectively three forward passes you have the forward pass you have the data gradient the weight gradient while if you really hit the critical batch size on inference you should only have a single forward pass word of flops. So that's why you guys use fireworks instead of using an open inference engine. Yeah I mean the other alternative is we would build one in house but you know if we have finite engineers like everybody else we would like prefer to have engineers make training more efficient and more precise rather than like spin up like an inference effort yeah. Okay that's super hardcore. What about thinking of mentioning your technical paper paper that you were doing this in a kind of globally distributed way? Why globally distributed and then what makes that hard? Yeah well there are various reasons one you know like this very large continuous clusters are hard to find in the market and so what we can do instead is we have one cluster that's going around all of training you know we can't do global training cluster but then the inference component of reinforcement learning we can globally distribute that across small clusters all over the world. So I think for the composer to run we use the four clusters in total that were all over the world very far away from each other and we even used some of our production traffic when it was least used so like we had the composer 1.5 the previous model served and when it was least used by people we just grabbed some inference GPUs and we put them to speed up training and so we can do this sort of things and sort of easily scale up our training ground without having one large continuous cluster and the thing that enables it maybe the macintok more I mean kind of like the reform late what the radio status basically are all training is like very heterogeneous right and by leveraging heterogeneity how different components like what infrastructures any it's you can actually drive efficiency and you see this pattern kind of across the board everywhere specifically for for training you have all this like high interconnected clusters you need high speed network kind of need to work in lockstep so those clusters are expensive right and actually it's really hard to find big ones right basically a scale with which composer was trained finding like two x larger clusters like significantly harder than finding the current size one and that's why if you can disaggregate this components and put them on different places one you don't need to find such a big cluster too you can actually find like different trade-offs of hardware because for inference you don't need that kind of wide interconnect you can have smaller groups of GPUs interconnected together you can have heterogeneous types of GPUs you can have different generations of GPUs you can kind of play all these gates of a game optimization and finally like inference it's much easier to scale up and down as you go and yes very conventional like when you have off-peak covers you can view all your kind of inference pool as one set of GPUs serving production traffic for real users or serving simulated environments for our own purposes and kind of balance between this of course it's a very interesting systems problem that you can mention like the kimi model is like one one terabyte training step takes somewhere between like five to fifteen minutes so it basically means like every like every five to ten minutes you are producing like one terabyte new snap-throat of weights so the question is like how are you going to ship it to a different cluster on this other side of the world very efficiently right and you want to like do it quickly because remember you don't want to get this staleness to get out of hand so I think that's was probably one yeah the kind of the most fun part which we figured out together is that despite you know full full model being like one terabyte not all the weights change every step right because RL does a lot of very like precise adjustments especially the training going on so actually there are very kind of regular patterns in like which subset of weights gets changed maybe not all of them change every time so if you were to look at like how my model changes within one training step like after ten minutes there is relatively small delta between those you can write write a compression algorithm which basically leverage is a property and now you end up with kind of like database systems problem, which is, okay, I have my delta and I just want to like ship it across across the world. My delta maybe is like 20 times smaller than what ship is full model is and that makes it practical. But of course, now you need to build all this kind of machinery from storage systems of full snapshots and deltas and recovering like reconciliation, etc. We were able to build it kind of in lossless fashion. Basically means that you always end up with bit equivalent models on the other side. So you don't need to worry about any mass aspects of this and you can do it really fast. You can do it under, you know, under a few minutes, even in the worst conditions, usually it's under a minute and most importantly, you like pause only for like maybe 30 seconds to swap the weights in your actual influence. We were like fully like saturated the band, the egress of the cluster while like sharding the upload and the download as well. So you can do all this like system tricks to bring the stand down is it is quite a few complexity, but you can kind of obstruct it out and just make it work great. Like it doesn't interfere with your training algorithm and on a flip side, you have this kind of power to desegregate to leverage other clusters to do that. And that kind of goes against kind of conventional wisdom of how you should do RL infrastructure. Because conventional wisdom is like you, okay, you're gonna have this really huge one cluster connected with our DMA and it's gonna be very expensive and you're gonna probably spend, you know, maybe you're gonna allocate like one sort to training into sorts to inference and sure if you have very expensive network, it's much easier to copy this one terabyte quickly, but now we have like three times larger cluster. Now if your inference engine is more optimized, then maybe you're gonna save one sort of that cluster in terms of GPUs anyway because you're just more efficient and you can take, you know, half of this cluster somewhere else and maybe cheaper hardware in a different region. So your cost comes down quite a bit. I love that you guys are just grinning as you describe this because it's like it's so hard and this is like systems engineers dream, right? And so it's just like a, it's an amazing, amazing system you guys have built. We spend a bunch of nights working on this. Yeah, you'll look like you have so long time to do a lot of time together. What about, I mean, you went to the beginning that Kimi is a very large sparse M.O.E. model. Does that make the RL run tricky in any way? Yeah. How so? Well, when you do inference, you're essentially doing like a forward pass is just kind of like autoregressive. And in this forward pass, it produces like log probabilities of like the tokens it has sampled. When we ship back the like generations of the model to the trainer, we have to rerun that forward pass because as we mentioned, we are doing asynchronous training. So the model that has produced the pass may have been like actually a few steps behind what the trainer is at. And so we have to rerun that forward pass and reproduce log probabilities. Now the problem is in theory, this log probability should be exactly the same if it's the same model version. But even with the same model version, you get slightly or sometimes very different log probability values for the same tokens. So this is often called like a numerical mismatch for inference. You hear this about all the time these days for mixture. Why is that? Why does that happen? I mean, primarily because like fundamentally floating point arithmetic, which is doing this is non deterministic. So if you're a floating point arithmetic is non deterministic. So you know, you learn this call that like if you take A plus B plus C, right? Then like C plus B plus A, it's going to be the same result. If you're doing this with integers, with whole numbers of the computer, that's going to be always true. If you're going to do it with floating point numbers, which are actually like approximate approximation numbers, you have this like many certain exponents etc. A plus B plus C and C plus B plus A is going to give you like different results or even like A plus B and B. So basically like fundamentally it's a accumulation order of like all the operations, which models though is basically like multiplication and additions. And like addition order matters to your final result. It's all like small differences, but they get amplified sort of like exact millions and billions of operations. So when you do inference of models, usually it doesn't matter that much because you portray in your model, you actually pretty robust. If you like flip some bits, it's still going to produce your good results, your benchmarks and then I change. But I really particular because you're using this very, very like weak signal to teach the model. The noise from this numerical differences can make a break your training. And that's like particularly important. And it again, it's an interesting intersection between algorithmic and systems part because you know, you can write a beautiful mess and it just doesn't work in practice. There are ways how you can drive this difference to pretty much zero. There are all this like batch invariant ways. You can be very, very careful and write all your GPU kernels. So they always add numbers in the same order. So you always do like A plus B plus C and not a different order. It's possible, but that always has like trade-offs, right? Basically your system becomes maybe like 2x or 3x slower. Again, it becomes an interesting trade-off. Like okay, what is the 10% of slowdown which we can take or in fact is actually a few percent of slowdown we can take to address 97 of these difference. That's the right trade-off which we find together through iteration. You mentioned that particularly for MOS and Sparkity is hard. The reason for that is that the way MOS work is that you take your activations at every layer and you would run it through gate-in-layer which basically decides, okay, for this token I'm going to run out of 3, 84 experts. I'm going to run this 8, right? So it's going to do like some mess and like top 8 scores. Those 8 experts going to be activated. Other ones will not be activated for this token. This operation amplifies your small numerical differences quite a bit because maybe your hidden states were like difference by like 5th digit after dot. Doesn't really matter. But this difference made it so you picked expert number 7 versus expert number 9 as kind of as a cutoff. And suddenly you went and activated totally different parts of the model and your difference got amplified quite a bit. And my models by definition are more sensitive to this mismatch. Again, when you do inference or when you do regular allowed, it usually doesn't matter in other average out. But now if you're trying to base this model learn, this difference is huge because your inference activated expert number 7. Now in your training you are trying to update expert number 9 which didn't even contribute to that during inference. So were you guys handwriting GPU kernels then to help get around this from? Yes. So you can, again, you can address all of this with GPU kernels and there's always trade-off. Specifically for me, you can do this interesting trick which people call a router replay. But basically you can have your inference just pass extra information to training and say that, hey, I activated expert 7 for this token. This very small piece of information is just one literature saying that like, okay, this is the expert that you activated. So trainer can be aligned with that. And a lot of this numerical alignment is basically doing tricks like that, matching quantization levels, matching kernels etc. to drive the divergence between training inference implementation down. And that makes huge difference in between your run maybe divergent completely or being multiple less compute efficient because you'll need much more data to address to this mismatch. I'd love to maybe chat a little bit more about the RL kind of recipe. Can you say what about the reward signal you're using? Is it like, are you care? Okay, can't say, got it. Top secret stuff. Top secret stuff. Okay, that makes sense. Like it seems like to say, almost like the equivalent of learning in sim. This is simulated rollouts versus like, you have so much actual user data that you could be learning on. Why not just do RL on your actual user data and your actual user harness versus doing this in sim? Yeah, we're also doing that. So that's what we call a real time RL. Okay. And we use the same technology to do like the inference weight sync with like fireworks to do this. We find like user signals where the user was happy or sad about a particular model generation. And we're able to update that model live. And so then ship a new version of the model continuously every few hours. We're working on decreasing that time. Actually at some point we'll have to increase that time because as the horizon of the model gets longer and longer, we'll have to a re-extend that time. So like, I interesting play. Like right now we are trying to decrease the time for stability because we were figuring out the right hyper parameters. And then after we have figured it out, we have to re-extend it again just because we want to lengthen the horizon of these models. Yeah. Do you need to do any of the kind of like pre-training simulated RL? You have so much actual user data. I imagine that's just like much more valuable to train and turn on. Like why not just go straight to the online RL step? Why do you have to do the offline RL? The online RL currently is pretty inefficient. We suffer from this problem that the GPUs are offline for a long time essentially. And beside that there is also like different trade-offs. Yeah, also there was efficiency and user experience. Yeah. If you do simulation, you actually do multiple RLs from the same prompt. Right? You effectively take a task and you ask a model to do 16 tries at a task like 128 tries on task like different RLs from the same prompt. Some of them are going to go well. Some of them are not going to go well. And by doing its multiple RLs in parallel, you are able to get much more precise signal. Maybe like you know, maybe model is very good and it's does it well 90% of the time. Maybe it's not very good. Losses like GRPO, like group policy gradient, like kind of work by doing multiple RLs at the same time. If you're doing online, you have only one RL out having the back. And so trade-offs of how you do it all automatically are different. And most importantly, if a simulated RL out goes from it's not too bad. I mean, you just maybe spend some time with you. If it's an actual user, you have much higher minimum bar on that. Because effectively you're doing A/D test. So if the model produces something weird, that's a bad user experience. Yeah. Okay. So you can go off policy more often when it's not a real user because you can experiment with crazy things and without affecting the user experience. You can do a lot more more relaxed, do GROPO. And then you're going to have a great time. you can basically bootstrap some level of performance that's good enough to even put in front of users. >> Yeah, we teach reasoning through the offline URL, which is actually called online. offline URL is more like DPO kind of technique, the sort of rainforest kind of URL is online. And then there we teach the reasoning to the model, we give it some kind of input of the behavior it should have. We try to give it to new information about the world and we teach it tool calling. And then we put it live to users. Because you could imagine if the model is bad, users don't want to use it, they're not going to give us any feedback, right? So the model has to meet some kind of bar to even be put into online URL. Like we want to be really happy with the model, and this is the model we ship. That's kind of the paradox of online URL or how we like to call it a real time, is that we can't use this to really create the model from scratch because users need to be using the model. And so it has to be good already and we can only make it better. Yeah. It's kind of like cherry on top to really get this super delightful experience for the sessions. Hopefully one day it will be like big, big cherry, you know? Yeah. Yeah, Dan Roberts presented at our conference last year. I think you were there. It's like traditionally it was the big cake and the little cherry-- Yeah, the unicorns cherry, yeah. Little cake, big cherry. I'm curious, the Andre Carpathy line of like right now, RL is still super inefficient. If you do a big, big, long rollout, then you kind of get like, you know, a little bit of information at the end. And it's still like, I think, sloping bits from a straw. What do you think? Have you been able to figure out how to get more bits out of that path? I can't talk about that. OK, OK, got it. Go back on the secret stuff. Good. That's how I'm asking the right questions. You mentioned the rollout to a few minutes at a time. It seems like the whole field is pushing towards making like longer eyes in agents, agents that can work for a long period of time uninterrupted and generally not failing. I love that meter scaling charts. What goes into the RL process to try to get the agents to run for longer? Several things. So one problem about sort of like reinforcement learning is that the longer the trajectory is, the harder it is to do credit assignment. So you can imagine like we are giving thumbs up, thumbs down at the bottom right at the end of its work, and sort of like to simplify the problem is like, the model asks itself, OK, where did I do it right and where did I do wrong? That's basically the problem called credit assignment. It gets harder as this gets longer. So you have to do a bunch of tricks there. The other problem is just like you run out of space, right? Like these models have a finite context window. And at some point they're going to reach that. So actually the way we solve this at Curse or is, we put compaction inside the RL loop. So we call this self-somerization. So during reinforcement learning, the agent actually learns how to continue and go on forever. So in practice, our model is like a 200,000 context window model. But in reality, it can go on for millions of tokens. And just because of this ability that it can summarize its work and then take that summary to restart its context window while still trying to accomplish the task. And through RL, because RL pushes the model to do things correctly towards the goal. At the same time, jointly, we are training the model to produce a good summary. And then we are training the model to listen to that summary very well at the same time. And so this is kind of like a continuation to reason, almost, I feel like. I'm fascinated because usually context management can see the part of the harness, right? In this case, you effectively coped, how part of the harness and model itself work together and throw in all of that into the optimization loop. And we've seen this again and again in AI. The more you throw out computers, the problem is the more you can solve the problem end to end. The magic of computing, guitar lesson, works, and you get much better systems which can work together. Totally. Totally. Do you think every company is going to be rrolling their own harnesses? Do you think that every company has the same shape of problem as cursor? If they are using AI and they're producing lots of tokens and they have a product to optimize against, I think it's the right move and the right direction to train models. Yeah, interesting. Interesting. And so it seems like most of the reinforcement learning I did then was on the harness/tool-use part rather than on the get good at completely next token for code. Is that roughly the pattern that other founders should have in mind when they're trying to think about where to use reinforcement learning. So if you're trying to get an agent to perform tests with tools over long horizon, you need RL. If you're trying to create a model that's good at summarization or next token or whatever, you probably don't need RL. Is that a good framework for when you need RL? I think RL fits everywhere. So even for tab, we use the RL. Personally, this is just my theory and not backed up by anything. When you pre-train a model, the models are just in the ingesting the totality of human knowledge. Let's say you're trying a model for math. The model learns all the math from stuck exchange. The model, when it's presented with a math problem-- and this is a model that hasn't gone through RL-- the model needs to wonder what kind of person it is. Is it the expert or is it the student that's trying to learn? And so one of the things that I think happens during RL is that we are tuning this knob, letting the model know, hey, you are the expert. You need to do things correctly. So that's one thing that happens is we are sharpening this distribution. Sort of like RL has a few phases. So there is the very first phase where the model learns and becomes very good very quickly. And then there is a second phase where it takes a lot of compute to continue to improve the model. And you see the model starts reasoning and have this pattern. So in the very first phase of the curve, I think that's where we're just tuning the knob, telling the model, hey, you should do things correctly here. And so RL in this small compute case is also very useful just to let the model know that it has to do things correctly. That's sort of like my case to this. Yeah. I mean, second data-- I mean, we see this pattern because many of these cases have RL-ventunion generally for many customers. And we see this usually-- you kind of continuous-- let me train regular supervised fine-tunion, is simplifying. You can see this transfer of new knowledge kind of in abstract way. And RL is kind of sharpening the behavior or like particular qualities you would want from the model. And usually you end up needing both. And even to your example of summarization, it's actually like RL may very useful for this, because sometimes it's-- if you want particular style out of summarization, it's really hard to come up with examples of like good and bad summarization, et cetera, like really describing this precisely. But if you use, for example, LM as a judge, you can actually say very precise rubrics. You can kind of prompt you while saying, like, OK, this is a criteria how I am going to evaluate whether summarization good or not. Throw it into RL loop and let the model kind of experiment with different summarization styles figure out what you actually want from it. Well, maybe another LM kind of evaluated whether it's matching particular rubric or not. And that's kind of type of pattern, which you see a lot, not just in coding. I see. OK, I'm going to ask this question to Dima, because Federico is going to plead the fifth. You mentioned LM as judge a couple times. Do you think that ultimately companies will be more successful having experts hand examining RL roll outs and hand coaching the model behavior in some way or do you think LM as judge, other automated rubrics are likely to get us there? You don't really put experts directly in judge in RL roll outs. I mean, that would be some kind of like-- RL time RL, if it's actually users, all like some form of I don't really like RLHF or DPO. I mean, generally, the more verifiable your reward is the better, because it allows you to scale the compute and just get better outcome. In some case, and by verifiable, basically, means that we can automatically produce without the human. Of course, if it's like mess or coding, and you can craft something very deterministic, that's the best. The reason why LM as a judge works is that it's actually kind of a generator discriminator distincter, like it's much easier to judge. I mean, the center of a humans, right? It's easier to judge than to create. Does LM have AC? Yeah. No, no application there. But yeah, it's much easier to judge. And you can craft precisely different criteria so you want to rank some answer. And you see this pattern where you might have very complicated eval from multiple aspects, right? Because if you dump multiple aspects to a single element, you might get confused how to judge. You might break it down, OK? You're going to judge rubric based on style, based on some different aspects, based on the factuality, kind of really craft these rewards. Some of the eval will be the same as some of the BLM based. And that's what guides your model behavior. Then you just turn on more computes and see the graph go up. Do you think that we're going to see RL be more effective in the harder it to verify domains? Like, do you think LM is just sufficient? That's one of the techniques you would start, right? Ideally, you want to figure out what is their actual outcome, what is their actual metric you want to get, right? So kind of trying to approximate this for LM is one way. Trying to get bigger simulated environments is another way. If you can simulate more of your product, if you can simulate more of your environment, usually you have like final metric which you care about. It's just harder to capture. If you can figure out how to capture this, that's great. And to your point about experts, I mean, experts still still need that thread. Because crafting this task can actually encode in the product experience you want. That's what matters, right? We went through software 1.0, 2.0, 3.0. Because we have crafting software directly, we went to crafting. training data right now, you effectively craft in the evaluation rules, but that's still very important. You need to look at examples. You need to look at the data. You need to look at where your product fails and how to nudge the model in the right behavior. I want to ask about RL environments, which is maybe related to what you were talking about. It seems like there's been a huge explosion and just the revenue scale that some of these RL environments companies are reaching. What do they provide that's actually useful? Because I think cursor, for example, you have so much data on how your customers are actually using your environments. What do the RL environment vendors offer you on top of what you already have? Yeah, we don't actually use any of the environment vendors. I think so it's very difficult to construct working environments. It's a valuable product for people that do not have access to these. For coding particularly, there is a very large amount of working coding environments available to everybody. That's GitHub. You can go in and maybe you can have a model just install all of the dependencies for a repository and that's a working environment. I think a lot of the difficulty comes from the infrastructure as well. You can imagine that the work's well for a particular task may need services out. They're making a change that let's say a database migration to test that is actually working. You need the database up. Those kinds of things are very tricky. I think these environment companies are quite helpful for that kind of stuff. There are two aspects to this. First, if you look at Frontier Labs, they're trying to build generic model which is good at everything. They need to cover all these different tasks underneath, package up in one model and encourage them to generalize. That's one part and that's very helpful. In cases like Composer, you have your actual product. Anything that's also video fireworks. If you have your actual product, you should do it all against it. That's the most powerful environment as your own product. Exactly because that's where your model will be used. Of course, if you have Frontier Labs, you're not going to do it across all the products. If you're trying to build the best model for your product, specialize in Taylorit, you should just use your production environment. Of course, you want to isolate it properly. You don't want to have a production database. You want to clone it, etc. There are some tools from the environment companies, such as from General Infrastructure, which makes it easier. Generally, you want your rural environment to be as close to real production as possible. That's what, as an example, we see it is if you look at Toy RL examples, the RL from Rooks, they always start like, "Oh, there's this toy environment. I'm going to spin up a Docker container and run everything in it." Which is great for Toy Examples, if you're trying to teach model how to play Atari or whatever. But if you're actually transitioning to production cases, you can't just put your real production application in a Docker container. We found it pretty early, as well, like working with many folks. In case, of course, our trainer on their side, some other customers, we run trainer on our training platform. But for environments, we actually default to running them on the customer side because that's where the actual implementation is. Even if it's part of our work platform or on the customer side, calling the actual production environment, not trying to kind of wrap it and componentize it on the host platform. Because that's really hard and that introduces differences. Yeah, like, I mean, what we call a rural environment is really three components. One is the harness. So the harness is like where the model can submit tools and its tools get executed. And the second thing is, let's call it the kind of operating system, right? So what is the actual world in state where the model is interacting with? And then there is the reward component. Which needs to check at the end that the work is done correctly. And generally, the harness is pretty portable. You can take the harness and put it in many different environments. The thing that's key is the operating system. And to replicate this, just normal containers don't really work very well. So at Curcer, we actually built like a whole virtual machine stack. And so we can spin up like virtual machines really quickly. And it has to be super bursty because you can imagine like, we are asking this system, please give me 100,000 virtual machines now. And it has to come all come up. And yeah. Awesome. I really enjoyed this conversation today. I think Curcer is such an inspiration in what you all are doing as a company. It's going from application company to really a frontier model lab. And I think the work you do with Composer 2 really leads that charge. So really special to hear about it. And then Dima, really cool to hear about the hardcore infrastructure problems. Actually, that the two of you solved together in the trenches over many, many late nights to make it all possible. So thank you. Thank you guys for joining today. Thank you so much for having us. Thank you. [MUSIC PLAYING]