WEKA's Val Bercovici: KV Cache, DeepSeek V4, HBF, SLC vs QLC NAND, CXL, NVLink, Tokenomics

[MUSIC PLAYING] Welcome to another semi-dope podcast. I'm Vik from Vik's newsletter. And with me is Val Burkowici, Chief AI Officer from Weka. Weka is an AI data and memory infrastructure company. Val has been on semi-dope before. And a few months ago, we spoke about context memory storage right after NVIDIA announced their platform. So if you want to check out that whole conversation, it's right in the back catalog. And we'll try to link it up here as well. But yeah, today's conversation is more about more recent developments in the world of memory and storage and where things are going in AI inference. Val, thanks for being on the podcast. So much for us to be back, Vik. And as we predicted, the only constant is change. So lots of updates to talk about since I was last on. Awesome. Isn't that so exciting? We have things that are constantly changing. And since we spoke, what was it like a few months ago? Seems like everything has changed. And you were mentioning before the show, seemed like 10 years have gone by in AI land. Exactly. In fact, I actually-- I don't know if I made this prediction last time. I expected a mythos-fable moment at the end of the year. And it happened in what May or April, I forget exactly when. So it's definitely happening faster than we saw. That's amazing. Yeah, yeah. It's really picking up now. And memory has reached an all-time high. It's become extremely expensive. I think a lot of people who are deploying AI hardware are finding ways now to make the best of what they've got. Increase utilization rates, pool stuff, offload to different tiers, whether it's other kinds of RAM, not just HBM, or go to NAND Flash. So there are so many techniques that have sprouted up in a matter of mere months because we have to do something about the memory situation. Exactly. I would say even-- one of-- I wouldn't say it's the simplest, but one of the most obvious and coarse-grained techniques in my mind has been model routing. If you're not always using the most memory hungry model for every prompt, especially in an agent's form, with hundreds or thousands of turns, and you're intelligently routing requests to medium and small models, implicitly there, you've got lower memory requirements. So that's kind of an easy button, if you will, for reducing memory and aggregate. But then we get obviously much more specific here with regards to each model's memory requirements, working memory, KVCache, and so forth. Yeah, exactly. So that's what we were talking about in some sense last time because the KVCache started getting so big when the agents arrived on the scene that you couldn't keep it on expense of HBM because it's a very limited resource and it has to be preserved and used judiciously. So there was like a lot of talk about offloading to DRAM. And then as agents started working more and more and having greater context length, they started talking about offloading to context memory storage, which is what was this whole rack, which Nvidia calls what was that STX, right? Yeah, the new storage platform is STX. The biggest use case of STX is going to be context memory storage, CMX, context memory extensions, and Nvidia terminology. And that is within the STX hardware framework, hardware and software framework. Awesome. So yeah, I know Weka has hardware. We spoke about this last time too, where you have got Weka's AMG, which provides really a lot of storage directly to the GPU. We are a very fast network connection. And it occupies the speed that you get out of it is like really good, comparable to something like faster than NAND, but slower than DRAM, right? So that's where we were. Let's double click on that because it's actually different than that even. So the standard positioning is you can be faster than NAND with higher bandwidth connections, you know, faster than local storage effectively. But in a standard hierarchy, you could slot in a slower than DRAM. However, depending on your architecture, and whether you can take advantage of the full line rate network performance or something like NVLink, the way Weka does, you can be faster than DRAM. So if you actually take a look at the number of PCI lanes that we have in these wonderful scale up domains with NVLink on and video servers in particular, there are far more PCI lanes, like 128, I think, in NVLink than there are PCI lanes from the CPU to DRAM, which is only 32 on the motherboard. So there's more-- this is one of the cool architectural, I would guess, leverage points of Weka, is we recognize at company founding inception about 13, 14 years ago for high performance computing, which is what AI factories are, there is more bandwidth on the network than the motherboard. And PCI is a bottleneck. It's not an accelerant. And this is still something that I find a lot of systems architects and developers struggle with. They assume nothing is faster than what I can access on the motherboard. And they design entire database hierarchies, entire application, latency profiles, based on that invalid assumption today, because the network is faster than the motherboard in a GPU network and a high performance network. And so yes, I think the memory tiering can have whatever you connect on the other side of that high bandwidth network, the NNV link style network, East, West, Compute, Memory Network, Nickel Network. It goes by many names. Whatever you connect on the other side of that can be faster than DRAM. And the way Weka connects optionally, we don't require that network, but we certainly leverage it maximally. But the way Weka connects, we're faster than DRAM on a raw bandwidth basis. Another confusing part is the way CPU and GPU memory transfers-- KV transfers, for example, and others-- operate on most of these servers, GPU servers. Depending on a generation, hopper has something called a bounce buffer between the GPU and the CPU. Blackwell introduced, especially with Grace Blackwell, more sophisticated chip to chip interconnects that are faster between GPU and CPU transfers. Very rubin' yet again, continue to advance that with higher bandwidth, far more scalable memory transfers between GPUs and CPUs. But you can still be faster than DRAM on these servers because of NV link. There's a ratio in terms of NV link bandwidth relative to memory bandwidth relative to front-end network bandwidth. And as long as it can be line rate on NV link, you can be faster than DRAM, which is counterintuitive. But the math is math. It proves out. That's awesome. OK, that's great. Because just to recap, what we are doing is that the networking on modern hardware is so fast that when you hook up even storage using that, you can get bandwidths that are faster than DRAM because motherboards aren't the fastest. And that is an assumption that a lot of people are making right now. But instead, if you view the NV link as a very high bandwidth interconnect, you can actually bring storage to operate with much higher capacity, but at speeds that are better than DRAM speeds. Yeah. And these networks are really critical right now. We've seen a rise in interest in networks, not just with co-pagogoptics, CPO, but what Google does with their Taurus network, AMD obviously with Pensando and that line of networking that they're going to continue to improve with Helios. But just what Melanox has done within VLink has been wonderful. And it's not just in VLink, right? Melanox offers CX7, CX8, CX9 network adapters, Bluefield's three's in four's. There's multiple varieties of Bluefield four's. And so we're not picky at least at WECA with regards to where the bandwidth comes from. As long as systems architects configure the bandwidth they want for the memory tearing they want, WECA can support it and continue to deliver these radical economic benefits in terms of more tokens per second, more concurrent users per second, which is really hitting the market sweet spot right now of agent swarms. Awesome. So since we last spoke like WECA, right now, it shouldn't be viewed so much as like a storage company anymore. Because in the AI era, because of what solutions WECA provides, I think it's far more than that. We were speaking a little bit about it before the show. Do you want to double click a little bit more into why WECA should be seen more as a memory company and a company that does AI data infrastructure instead? Yeah, very much. Two of our industry partners, if you will, one is GROC with a queue that Nvidia Acquie hired last December for $20 billion in a non-exclusive license to their technology and most of their founders and engineers. And then, of course, a very popular and successful cerebra site, P.O. were clear indicators that the inference market, the inference infrastructure market, is very distinct, very different than the training market that Nvidia dominates with GPUs. And even Google themselves, for me, for some reason, this is like the clearest example. Google has had seven generations of TPUs. that were applied both the training and inference, right? And radically different markets back then when you know, TPU123, et cetera, through seven was released. With TPU8, which they announced just a few months ago, there were very explicit as saying there is now a TPU8 for training, but there's a completely different TPU8i for inference. So that's just another example that, you know, inference is different. Infra-structures different. It's great when you can leverage, you know, the same infrastructure for training and inference. There's nothing wrong with that, but understand that, you know, whether it's really cool companies coming back out of stealth like etched, you know, or cerebrus, or all sorts of other LPU-based, you know, ASIC-based, SRAM-based companies, inference will continue to diversify in terms of infrastructure and diversify away from training. So, WECA is very much following in that trend. WECA has that great legacy of high performance computing HBC storage, but the new tagline for the company data and memory infrastructure reflects the fact that at its first principles raw basis, inference is not storage-centric. Infra-ence is a little bit compute-centric, as we know, for pre-fill and extremely memory-bound for decode and feed-fold and so forth. And that's really where the WECA augmented memory grid product line fits in is it's very much you can use it as an inference-only product without any WECA storage whatsoever. It just happens to be backed by NAND at the bandwidth of memory, and as we just discussed, when that particular context memory network happens to be large, then it's actually more bandwidth than DRAM. - Yeah, that's awesome. So, depending on the kind of hardware that is running, now you mentioned that cerebrus is one, and the whole inference landscape is really, seems like there's no one right way to do inference. You could run it all on SRAM, you could run it with LPUs, which is also SRAM, then you could run it with a combination of HBM and SRAM, or if you look at San Bernova, they use all three. So it's like breaking up all over the place. And so how do all these compare, like in terms of performance, or where does storage play a role into this whole thing? - So there's a memory hierarchy, and it's interesting now, because when we last talked about this memory hierarchy, it's funny. We talk about it as if it's been run forever, but it's really only about a year old. But if you're already well established for tier hierarchy, that's gonna have to change now. And Nvidia's Dynamo team is done a really good job of documenting this. They even label it G1, the top tier in the memory hierarchy for high bandwidth memory. G2 is for CPU DRAM, but it also goes by LPDDR and SOCAM, but it's for the uninitiated, it's all the same thing. And then, so that's G2, G3 is what's called local storage, and sometimes it's called like the RAC SSD that's in the servers. And then the G4 tier is remote storage. Often it's NFS or S3 type storage layers. And the one thing to really not maybe confuse about this memory hierarchy is there's no smooth graduation from HBM to DRAM to storage, local or remote storage. These are very, very rough sort of jagged transitions, almost a grand canyon sometimes between them. Because we're talking about orders of magnitude higher latency between those memory tiers, orders of magnitude different bandwidth between those memory tiers, and you very quickly reach these cliffs, where if you're okay providing one token or 10 tokens per second to a user, and you're okay with latencies, time to first token of like 10 seconds and 10 latency of hours where they should be minutes, then you don't have a problem. But in the real world, no one tolerates five or 10 tokens per second output. Kind of the human eye demands human attention span, demands about 35 to 50 tokens per second about put minimum and agents of course and machine speed will take thousands of tokens per second about put in a multi turn agent swarm. So SLO service level objectives matter. And that's where we quickly find right now that even though there's a lot of vendors talking about memory tiering, including NAND and storage in the KVKash hierarchy, it's kind of irrelevant. Most of the benchmarks we see out there today that includes storage in the KVKash tiering hierarchy, are workloads that you could just run on a DGX Spark or a Mac Studio today. You don't even need a big server to run some of these smaller models with small context windows and single turn chat sessions or two to three turn agent sessions that are not representative of reality. When you're running real world workloads today and semi analysis is actually working on updates to inference X that they've nicknamed agent X after my suggestion actually. So agent X when it comes out soon, I'm not gonna steal their thunder, but when that comes out soon, that's gonna reflect something called an input sequence length and ISL, which is the context window of much larger than 8K, which has been the limit in the past. Hundreds of K, they themselves have published analysis of the fact that I think the median traces for coding agents now are two, 300 K tokens. Because of course we're in the era since we last spoke of million token context windows instead of 100 K or 200 K token context window limits. So when you reflect the reality of large models, trillion parameter models or models in that class, many Macs is in that class even though it's not quite a trillion parameters, but Kimi is in GLM 5.2, then you hotness is and so forth. When you reflect large models with large multi turn sessions with high concurrency, the way, you know, cloud code, open code, codex, actual pi, you know, Hermes, open clock, actual agents work, then you end up with a radically different workload profile. And on the one hand, the models have gotten so much better in terms of KV cache consumption individually, right? I use a number I think I remember of for every 100 K tokens without optimized compression like turboquant, without optimized compression like sliding window attention and others from deep seek before, you are up to 50 gigabytes of KV cache usage for that 100 K tokens. That's been reduced on a unit basis right now with turboquant, with optimizations from deep seek and others to about five, five gig of instead of 50 gig. So that 90% reduction is great. But what's happened on the other side? Context windows 10x. And then agents want 10x, you know, 100x the number of concurrent sessions. And on and on and on, we've got a multi-modal. And so the net effect we always seem to see is that for pretty much every 100x reduction in unit KV cache size, there's a 10,000 x increase in consumption of KV cache. Yeah. There were always looking about that 100x more token volume. And it's reflected in the pricing pushback we're seeing by enterprises today. Maybe they're not paying 100x times more than expected, but they're paying a lot more than expected because of that token volume. Yeah, yeah. So this is great, right? Because clearly KV cache optimizations are on the way. Like turboquant was one which scared the daylights out of the market, everybody thought, like memory is dead, HBM is dead, everything is dead. But it turns out that the quantization is a good thing because we only need like you said, one tenth, or 90% lower KV cache storage for 100k tokens. So 100k tokens was taking 50 gigabytes. Now it takes five gigabytes. But this is great because which means with our existing models, we can run far more agents and do a lot more work. And so the demand really never went away. In fact, it got more, right? It increased. Yeah, in fact, that you can almost make a meme out of this. That 50 gig that went down to five gig went right back up to 50 gig because the context when their size went from 100k to a million. So we're going to give it a take here. And then that result is just always more memory demand. So just because KV cache is being compressed like this, do you see that it's going to continue to be so? Do you think that we'll go to like maybe one gig for 100k tokens in the future? And what does that mean for KV cache? Because if you say like we dropped a usage by quantization techniques, and then we brought it back up because of like agents, I guess we don't have a net increase. Is that something that the memory and the storage industry should worry about? Because I don't know if the increase in the usage is exceeding the benefit coming in from the quantization. That cliche this time, it's different applies here too, right? Every time there's this massive new innovation in KV cache compression, it's not different. It just Jevon's paradox keeps kicking in over and over again. And we're seeing that that higher increase in demand, every time there's a reduction in unit consumption. And so for me, let's look at context windows. We're not stopping at 1 million. It won't be long before we're two and five and 10 million maybe by the end of this year. Everybody wants more context. And we're going again longer horizon and our agents. We're giving agents far more ambitious goals being encouraged to buy the agent and model providers. The goals now span hours and days. Soon, I think some agents will run weeks quite regularly. In fact, the cyber security, which we can get into, they run forever. You've got to run the security operations center persistent Louis agents, once forever now. So we're seeing longer multi-turn horizons. Of course, we're seeing more parallelism, more concurrent sub-tasks, plot workflows was one sort of feature introduction that became very popular to just solve a problem in parallel 10 times faster. And then again, we're going multi-modal. So it's not just language, but we're inserting more video and audio frames and image frames into these agents. And it all just results in a continuous explosion of overall token demand and overall KV Cash consumption. Yeah, that's awesome. Okay, so now that we have established that this is only going to get more from here. So NAND and DRAM or whatever, like, form you would want to look at it as pool, not pooled, we're going to need more of this because there is, we're only going to run more of these things, so we're going to need more. So that doesn't spell any decline or leveling off in the near term, because context windows are going to get longer agents are going to run longer. And more agents are going to run per task because you can like have parallel ones run. But I think one interesting thing that you mentioned earlier was like deep seeks optimizations because when deep seek V4 came out, there was a lot of emphasis on SSD first approach to inferencing, which I think really helped in terms of their token pricing, especially when you end up hitting cash. And they showed that for some workloads, you could hit like 95% cash hits. And what happens then is that as long as you keep hitting KV Cash, you already have the tokens stored in a high bandwidth, the network connection to an SSD, which means now you can basically get like free, it's free now, because their token pricing is so low for cash hits, it's insane. Like, how is this playing out into like other models are other models also hitting cash hits at this level or what do you think of that? Yeah, so on a one hand, I'm glad you noticed that because we really want to emphasize this key point now. The majority of token volume is agents forms by far 80, 90% to some longer chat sessions. So chat sessions are almost irrelevant to the conversation from an infrastructure perspective. And agents, it's interesting, even the Sonnet 5 announcement just the other day, right? Everyone looked even anthropic. Interestingly enough said, here's the new pricing for input tokens, and here's a new pricing for output tokens, and it's a promotional price. And the reason why I don't even quote the numbers, it's really irrelevant because of all the tokens used by agents, most of them are cash read tokens. And some of them with anthropics, particular, you know, unique pricing models, you have to pay for cash rights as well to benefit from cash reads. But we should only be talking about cash read pricing. The other pricing is a rounding error, and deep seeks pricing reflects that, right? They're dramatically new, you know, ex inserting two zeros, right? Basically, after the decimal point, or price per million cash read tokens, really is a shock to the industry. Not so much the memory industry. It's a shock to the other models and the other inference providers. But the very important nastrix there is that pricing is only available out of China, right? You have to use deep seek hosted, you know, the rumor is I think among Goli and data center with very, very low energy costs, but also some secret sauce that deep seek has with regards to how they've integrated version four flash and pro with the, you know, fat high flyer file system, a three FS file system, parts of which is open source. They famously open source that last year and wrote some wonderful papers and blogs about it. Curiously, unless I've missed something, they haven't updated those blogs with what what definitely are some new innovations and improvements that factor into that that dramatically low pricing. The net of it though, I think venture B did some math, which I was quoted on earlier on is it's about 87 times lower cash read pricing from China than the same model VIV4 Pro or V4 Flash hosted in Singapore or hosted in the US or Europe. And so the real conversation is yes, deep seek has completely set the bar for global pricing for people that are able to inference out of China, but for people that can't inference out of China, it's still by far the best pricing, but there's a massive margin opportunity for open weights inference providers to differentiate on cash read pricing because you you may not be able to match deep seeks subsidized pricing, but you can still leverage their innovations, their sliding window attention, hybrid compress attention, HCA and so forth. You can still leverage those innovations and offer very important reductions in token cash read pricing and really drive effective aggregate agent pricing down because that is the one pricing metric that dominates the cost of agents. Yeah, so I have a couple of questions on this one. The first one is why is it's news to me that so that pricing comes only out of China. So if I run a deep seek model out of I don't know, open router, I don't have that pricing. Open router is wonderful because it shows you the difference, right? In fact, for some reason, I don't even know exactly what this signifies, but there's a little a little slider button you have to slide to say show ignored. And I'm not exactly sure why they label the button that the button that way, but when you click on that, it does expose the deep seek pricing out of China and then you can compare. In fact, your agent, of course, your Hermes, your open client can compare side by side in real time the pricing for the exact same model, the exact same input pricing, the exact same output pricing, the exact same cash read pricing. And I should double check because I tend to to stand that open router side quite a bit daily. I should check whether there's more and more cash right pricing beginning to appear because I know for some models it has and it escapes me whether it's appeared for deep seek providers as well. But you're going to see some providers now differentiate not just on cash read pricing, which is a major point of differentiation, but also on cash right pricing. Okay. So in terms of cash read, the real workloads that we're running today, it could be agent workloads, because like you were saying, any other kind of workloads like chatting and typing in on the chat window is an negligible portion of the inference market today. So let's just talk agent workloads. How does somebody make sure that they have a cash iterate of I don't know 95% because that will drive the pricing down enormously and create meaningful differentiation like you were saying. And where is the industry right now in terms of cash iterates? Yeah, you know what we could spend a whole part just on this question because it's it's very opaque and a little bit confusing. So let me let me go through this because this comes up a lot in other pods as well. When you look at your agent dashboard, your cloud code or just your cloud dashboard. If you have both code and cowork and another cloud products or open claw pick your dashboard right. It'll have a cash rate that is a logical cash rate that that reflects the cash ability of the tokens and your agent's warms. And that is often very, very high like every agent pretty much has a cash ability of 95% right because you are reusing a lot of context for agents, especially agents warms and so forth. However, the providers, you know, the actual cash rate from the provider is not one to one the cash ability of your tokens right. The effective cash rate is very much a function of the memory tiers you have and everyone are buying large, you know, most people in the first quarter of this year and some in the second quarter have had very finite fixed memory tiers. There's only so much hbm that comes packaged on your GPUs. There's only so much of the RAM that's on the GPU servers. And as I mentioned before, you can try and add storage tiers and cave the offloading to storage, but they ruin your SLOs. So they're it's really not that common in production for the popular models and popular token consumption. And so the way to really understand where people actually have true effective cash rate that are as high, maybe not as high as the logical cash rate, but close is in the pricing. So open router again is a great proxy for that because it's the only way to introduce transparency to the real world cash rate and even open router themselves. They publish actual buy provider by model by provider. They publish some cash rate rates. And I like them because if you look every few minutes, they do change, right? Based on the actual token traffic of the moment. But even with some of the numbers I see there, I don't think there's a real effective infrastructure. hardware memory, cashier rates, there's some blend, but nevertheless they're much closer than what your own, your own agent dashboard reflects. And yes, for me, I like to joke now. We're seeing a lot of, you know, bench maxing and a lot of the models right now, you know, for example, like sweet bench, everyone sort of trains for. So it's not long to that relevant to benchmark, but deep sweet is still a good benchmark. We see the same thing obviously when when when vendors sort of, you know, benchmark the KV offloading solutions, there's not a lot of truth left in benchmarking. So my personal belief is that profit and loss, right, pricing, transparent pricing is the ultimate benchmark. And then you're going to see what could be more explicit in that space to do that, right? We want to continue to prove out our own advantages. And we think that, you know, real world metrics reflected by pricing is probably the best way to actually benchmark a solution now as opposed to something in a lab. You also mentioned that deep seeks sliding window attention is something really to look into. What's unique about that sliding window attention in deep seek? It's actually if you ever paid attention to, you know, where compression of files in general has evolved from simple like PK zip compression or simple block based deduplication towards the similarity hashing algorithms and a combination of, you know, sort of global similarity and global hashing and local hashing. The same thing is happening now with KV cash. The same concepts are being introduced at the token at the attention level. And sliding windows just that it's a way to take a look at just, you know, the more recent tokens and compress them as much as possible and just assume the tokens that your particular attention head hasn't paid much attention to recently aren't as relevant aren't as compressible. And this it's just cumulative right. I think if you count it, there's a formally about five different attention mechanisms in deep seek V4 now all simultaneously applied in each one specializes in, you know, short term compression, long term compression, you know, context relevant compression and so forth. And then that result is that impressive 90% real world reduction in KV cash usage per request. But again, the requests now are just coming faster and bigger. So it's kind of necessary to support. In fact, I think the reason why they continue to innovate so aggressively on KV cash consumption as they want to support million token context windows in soon to and five and 10 and the only way to do that effectively is to keep being more and more efficient intelligent current, if you will, on how you consume KV cash. Okay. So deep seek what it does is sliding window attention idea is that you just keep the window of attention to what is most relevant right now and keep discarding what stuff is is is relevant anymore. And that saves you KV cash almost assume what's what's been attended to before that's not being attended to now has already been compressed to some extent. So now let's really be aggressive on what we're tending to right now and see where the opportunities to compress token memory, you know, is a token attention in KV cash is. Okay. In terms of just like NAND flash storage even I wrote an article recently about what it really takes for an SSD to be like AI ready like SSDs in the past were meant for a different use case really than what they're being used for now. And one theory I have which I want to run by you is that NAND itself is now breaking up into several tiers. Like we always in the past wanted to get more capacity out of NAND. So the industry was like trying to go from single level cells which has lower capacity but you know more endurance and faster right speeds are to going to QLC or you know quad level cells on the other hand where you can save you know like basically store four bits or 16 different states in just like one cell. It really gives you a lot more capacity but endurance is a concern and I'd say it's harder to write to because you have to make sure you can differentiate between 16 states. How do you see these NAND tiers working out our single level cells making a comeback. How is that working? Yeah it is making a comeback and so just like we talked about earlier you know training infrastructure used capacity NAND because you wrote a lot of training data infrequently and you read it very frequently so that was ideal for QLC type of roaches. Whereas inference is just fundamentally different right it's really fundamentally memory centric. So you're really trying to make NAND appear more like memory and a lot less like storage and memory doesn't care whether you write or read at the same rate because the assumption is there's no endurance issue right there's a power issue but there's no endurance issue a persistence issue with no endurance issue with memory with with memory with DRAM in particular or HBM and that's not true. So this is where NAND flash struggles to act more like true memory from a performance and a consumption perspective as opposed to just a capacity perspective. Again something this is something that WCA anticipated a long time ago and so you're seeing that largely because of inference alone Intel had this really interesting technology with micron called Optane or 3d cross point. It's a real game. It's a real shame. Oh my god. Yeah. You know today inference is the killer app for Optane. But it's gone now right based on different materials phase change memory and so forth. And what is tried to replace it as SLC flash right. Okay. If you have to if you can't optimize your rights your KV cache rights well enough then you have to buffer that endurance issue that drive wear issue with a more endurance form of NAND flash which is a single layer cell SLC tier. And sometimes I can be complemented by QLC so you buffer a lot of rights and SLC and then you destage them down to QLC later. Again that adds complexity that adds latency that's no free lunch there. Another thing you can do and this is something that WCA does is kind of anticipate that you're going to write to NAND flash a lot you're going to read to it a lot basically use NAND flash as opposed to just keep it for capacity. Yeah. And in that case yes. Well WCA is done as we've always amortized rights across a whole fabric of NVME drives. My joke is there's no S there's no storage in NVME. It's not about tell memory express or memory extensions. And when you treat it like a true memory protocol and you amortize the rights across a whole fabric of NVME devices you don't need SLC. We don't have SLC for example never have in WCA. We literally can use TLC for a very high right endurance workload like KV cache. And of course in the future as we're able to influence the rest of the inference stack so the scheduling and the routing not just the KV transfer storage layer we will be adding support for QLC drives as well. But again we got to be careful about that because you don't want to break those drives by doing unintelligent things on scheduling rights or unintelligent things on just routing tokens to somewhere that needs a quick right but is really kind of not worth it. So you can actually optimize the controller layer and make sure that you don't clobber the drive whether it's a TLC or QLC drive and intelligently right to the whole TLC/QLC array and still keep endurance. Exactly. This is something that's a category of storage technology called shared everything as opposed to the Hadoop style shared nothing which you still see in parallel files as it was like GPS or Luster etc. And with a shared everything approach the client if you will has global awareness using a raft protocol of the Q status the work Q status there's multiple Qs in every NVMe device every SSD controller basically. So when every client has global visibility into the Q depth of every NVMe device in a fabric tens of thousands hundreds of hundreds of thousands of Qs on tens of thousands of drives in a fabric then you can treat NVMe like a cache line for DRAM and you can be very intelligent about where you you you you you shard the rights right where you load balance the rights very very intelligently. And at that point you don't need expensive SLC tiers to buffer rights you're not being you know suboptimal you're not being unintelligent or treating devices as blunt storage devices you're really you're digging into the devices and you're making sure that you're you're you're optimizing the actual underlying that flash well and cooperating with the controllers versus offloading work to the controllers. So you don't really see SLC as coming in on its own tier maybe it's just going to be used as a as a buffer. Well let me be you know let me disclose here that I'm talking about how WECA has decided to optimize flash tiers. I think the industry needs SLC right. The industry doesn't have our patents doesn't have you know our implementation by and large except for you know the server partners that we partner with but generally that that technology is not available outside of WECA so you do need SLC there will be a rise you know Nvidia is forecasting this themselves even for CMX there will be a rise in the need for SLC to buffer the rights when you can't amortize them over TLC. But the upside of that is that if you buffer correctly the SLC, you can still use QLC at WCAW or skeptical, right? Because again, these are not capacity workloads where you have the luxury of time to D-stage SLC, the QLC. These are very bursty, very intensive workloads that are attempting to emulate memory. We think that ultimately one tier, whether it's SLC or TLC, is the only safe and fast way to offload KV offload the storage. But the industry, there's a lot of smart engineers across the industry and many vendors. The market will prove what works over time. We just know what works for us right now. In a NAND storage rack, you could amortize rights over many drives. But you also over-provision, and to what ratio? If you have 100 drives in a rack, and to make sure that you have the right endurance, even after amortizing over TLC drives, do you actually have 20% more? These are tricks we can always play right. So if we don't have SLC, and economically, or just supply chain issues, we need to use TLC or God forbid to take QLC only, then yes, you would really have to over-provision. And the net effect is you would be buying, let's say, a petabyte of storage, but only C300 terabytes. It's like a fact there were 500 terabytes of effective capacity from that petabyte that you purchased and are powering because the over-provisioning is needed not to break those drives. Yeah, so it could be even a two or three to one over-provisioning. Yeah, it's tough, because these are extreme workloads. There's nothing gentle about emulating memory within VME with NAND Flash, right? It's a very intense workload. Yeah, yeah. Well, that's great insight. That's a really great insight for me on how SLC and TLC tears work and what the trade-offs are. So I definitely has added to my body of knowledge today. It's nuanced. That's why I said, we could spend a lot of time on this alone. Yeah, yeah, yeah. We should move on. I think a lot of interest in the market right now is for high bandwidth Flash, because now if people can put in Flash right next to a GPU and have it store some stuff, it would be a useful thing, I would imagine. So my two questions I think around that are, does HBF use SLC, like single layer SLC? That's one thing. And secondly, what are the really the use cases for this thing? Yeah, great question. I think HBF will probably have to use SLC in the most common configuration for the reasons we just discussed. It really is trying to be a great offload tier. It's trying, in fact, in many cases to replace the RAM, right? Right. And complement high bandwidth memory directly. For me, it's always come down to a packaging decision. I think SK Heinex has been public about the fact that they want to package it, directly on the GPU package itself. They don't want to go through buses. They don't want to go through networks. They want it to be really tightly bound to GPUs. I think it's a great vision. I haven't seen any GPU vendor directly committed that yet or adopt that yet, but I think in the future, maybe 2028, I think we'll see that. But there's other vendors that I can't discuss and reundit or packaging it differently on a PCI card, for example, and not even packaging it with GPUs, but with ASICs, non-GPU accelerators. There's going to be different packaging form factors, but ultimately, I think it's a necessary thing. Because the opportunity to get this right, the opportunity to optimize the KV transfer layer, the opportunity to schedule correctly, it was really acquisition, I think, a Qualcomm of modular, right? And be able to get that compiler expertise, what Nvidia acquired with GROC was a lot of compiler expertise from the TPU team over at Google. That is going to be more important over time to making HBF really usable because as we recompile models to understand HBM and HBF tiers and maybe bypass, this is a prediction we have inside WCA engineering, bypass DRAM tiers altogether because you can get a lot of bandwidth at an NVMe, sorry, out of NAND flash devices over NVMe or other protocols, or without NVMe, but just NAND flash. There are optimizations possible now, and there's definitely a trillion dollars of opportunity that encourages those optimizations between HBM and HBF with raw NAND flash and/or NVMe. You can always replace drives if you can run that bandwidth over a high speed network and NVMe drives. And like we spoke about over provisioning and something dies, you change it out or whatever. HBF, I don't know, the one thing that always is on my mind is, it is NAND flash after all, and it will, even if it's SLC, it ultimately has an endurance to it. You can't change it out if it's packaged next to the ASIC or GPU. So is that like something that's going to play out or the lifetime of this thing is, I don't know, 50 years doesn't matter, we probably throw the chip away by then anyway. You've already done a good job in covering this. There's a bag of tricks engineers are throwing to mitigate this problem. It's not one solution or one work around. There's over provisioning is one of the tricks for sure. Then again, there's proper amortization just at the storage at the NVMe layer and NVMe fabric layer. Then there's very explicit scheduling at the inference time, but also very explicit scheduling at a model, a compilation time, and very intelligent token routing inside the MOE level and even just at the gross model level as well. So you could engineer the whole thing to a level that this is not really a problem and you can still use it. Not only can you do that, I've already seen anecdotes of people applying fable to very complex engineering problems and seeing months worth of sophisticated systems engineering completed in four hours. So one optimistic scenario is that these agent harnesses and the Eval loops, the quality Eval loops and so forth, the judgment models. And of course the guardrail loops and the guardrail tokens. When you package that all together, some of these really deep tech engineering advances can happen much faster. But ironically, we need more engineering and the harness to make sure they happen reliably and safely. Okay, that's amazing. So much stuff is happening that like we really interesting see how all this plays out. So that's the whole fun of this thing. And the one other thing that has recently cropped up is, and I know like we spoke about this last time and you had ideas around this that I want to see if it has evolved or changed ever since is the use of like CXL. There's a lot of talk about using CXL now because some of the RAM DRAM isn't really being used on every server. I think maybe like 50% is being used. So because memory so expensive, even Google has had a change of heart. It looks like to actually start using CXL and reclaim some of the unused DRAM. What do you think like is CXL a thing now? So CXL has a lot of fans. I was one of them 10 years ago. I'm not a fan anymore, right? And it's not because I don't like the technology. It's because in the real world there's alternatives. So personally I think Melanox and of course Nvidia's acquisition and really great execution of scale up domains kind of kills CXL in one sense. The ability to have this great NV link style network and have it used for memory, have it used for high bandwidth as well as regular DRAM memory. Has been a real glass ceiling or concrete ceiling really for CXL growth, market growth. There's been less and less need. Then you've got Rocky right, RDA may over Ethernet and you never bet against Ethernet in this industry, right? So the challenge CXL has is another bus. It's another bus you have to engineer. It's another bus you have to debug. It's another bus you have to maintain and power. And it's not the in isolation, it's bad. And again, it's got a great killer app today of utilizing this really precious underutilized resource in some cases of DRAM. But in the context of real world alternatives, I'm just pessimistic about the future of CXL. Largely because again, I'm biased. I'm able to leverage things like Rocky or NV link over in Finabend and deliver better than CXL performance with NAND Flash economics cost of goods and capacities. So it's one thing to pool underutilized terabytes of DRAM. It's another thing to pool petabytes and exabytes of NAND Flash at the same or better performance. Yeah, want to fast the network bandwidth also benefits CXL like it does. The first principles are simple until you have to engineer the real world issues into it. Yeah, the first principles are more bandwidth is good. And so if you can have faster than PCI bandwidth. to a pool of DRAM as Google discovered, there's benefits to that. It's just that if they're really where benefits to that, I think you'd have seen it in Blackwell, you'd have seen it in Vera Rubin from Invidia or in Helios from AMD or even in Feynman, the next generation from Invidia that's already pre-announced and you haven't seen it, right? And then you haven't seen it as a standard supermicrofering and you haven't seen it as a standard Dell or HPE or Lenovo offering. And so I think it's that absence that speaks volumes, right? There definitely are use cases for it, but for some reason it hasn't broken out into mainstream. - Yeah, yeah, for yet probably. There seems like, yeah, it seems like the shortage of HBM is now causing some players like Google to actually consider it now because it's just like, capacity issues are pushing hardware makers towards solutions they probably didn't consider before. So yeah, that's another interesting thing to see how it'll play out 'cause you're right. So all this time it hasn't been in there, there's a reason for that, right? - There's a reason for that. And I think there's one important hint, right? So it was reported, I think by semi-analysis about two or three weeks ago that Invidia changed the bomb on Vera Rubin and they cut the actual amount of DRAM in half, either from two, four to two or three to one and a half depending on the model. But that's a clear indicator that A, DRAM has gotten too expensive and B, Invidia's projecting with CMX solutions in the marketplace are a number of vendors including Weka that there will be less need for DRAM for inference. And so you're seeing that people are applying solutions to this problem and it's not always just pooling it better, it's just reducing it overall. - Yeah, yeah, yeah. That's a part of what we spoke about in the beginning, right? Like you could make optimizations to the algorithm, you could use sliding window, you could do whole lot of different things that maybe uses less memory overall going forward rather than just use existing architectures but start pooling stuff together. So yeah, you know, it could go either way. If we find better algorithms then we probably don't need to pool it, could be. - Yeah, yeah, that's I think the prediction we're making. - Speaking of better algorithms, what do you, what's your take on AMD's mext acquisition and for people who may not have heard of this? Mext is essentially a software company that found a way to optimize the use of DRAM by dynamically offloading all the unused parts of DRAM to NAND Flash. And then using AI to predict when that same information is going to be needed back in the DRAM and preemptively moving it back before the GPU even notices it's gone, right? So it looks like it's a cool way to use NAND but what's your engineering interpretation of what's going on? - That was a fun one to review because I wasn't familiar with Mext beforehand but it was clear in reviewing their use cases pre acquisition and the initial positioning, post acquisition is, it's AI technology, it's machine learning specifically small language models and small neural networks, again optimizing caching algorithms and being more semantically aware than just recently used or so forth simplistic heuristics. So it's the application of machine learning and deep learning into caching algorithms but the actual use case is ironically enough not yet for KVKash offloading, it has potential to be very good there. We've seen even things like popular in memory databases, Redis and so forth implement algorithms that aggressively destage the RAMT and NAND Flash and so forth and retrieve it back. And Redis does position that for KVKash offloading as well. So I think it's an active space but right now I think the AMD initially is targeting scientific computing so whether it's life sciences, whether it's Monte Carlo simulations and other kinds of seismic analysis, whether analysis, those are the kinds of applications that technology has proven itself in and we may see it happen, we may see it appear in KVKash offloading as well. - Nice, yeah, I thought it was an interesting use of the predictive nature of an LLM because if you can predict what the next word is, why not use it to predict what the next page of memory is required and quickly pull it from Flash to DRAM? - I don't know how practical or useful it is but I found the idea was interesting. - I think your instinct is right, why is cursor customized KVK 2.5? It's a composer, as many companies are realizing now that the bar towards being able to train your own model has come down, it's a much more accessible thing from any companies right now. You don't need a million dollar ML researchers to train your own models anymore and when you can customize a model for your domain, it actually doesn't have to be an LLM at all. It can be an ML, it can just be a very tight neural network, it can be influenced on a CPU, it can be influenced on a small low power CPU if the model is really domain specific and it's just a neural network at that point. It's not a large language model and there's I think gonna be again, another Cambrian explosion of use cases and applications for clever small models that do things at heuristics, you know, peak that and can no longer optimize or improve. - Yeah, do you think that this use case is basically physically in robotics where you could have those sensors at the edge like process, very specific amounts of information, it's only one kind of information from a sensor, right? So it's not like a large model you need. So is that a useful use case you think going forward? - 100%. I think it's probably gonna be the reference architecture for robotics and edge inference. We don't need large language models. There will be some aggressive cloud connection whether it's through Starlink and remote locations or just broadband. If you really have to burst to some kind of complex decision that a large language model has to make, but 90, 95, maybe 99% of inference for robotics will be local and disconnected air gaps, so to speak. - Yeah, that's the fact that like you could make inference decisions at the point of sensing or just like put in intelligence anywhere actually is a very useful edge use case. How useful or how good that intelligence is, I think yet to be seen, but in principle, you could deploy these little models like everywhere. - Yeah, the cost of these Raspberry Pi style system on a chip motherboard are really plunging. And fortunately again, whether it's KB cache optimizations or just small model quantization and just custom neural network training that doesn't have to be a large language model at all, is intersecting really well with really affordable system on chips, SOCs. And yes, that results in some really interesting robotics and drone use cases, yeah. - Yeah, Jensen mentioned something about the AI flywheel in this context. - Exactly, so this is a general concept of, it's all about being able to capture data, domain-specific data, train models, and then customize that with more either real world domain-specific data or synthetic data now that you have enough real data to create useful synthetic data and just keep iterating on that loop of you're pushing the frontier with big models. You're customizing either through fine tuning, distilling, you know, quantizing, low-rank adapting, et cetera, all sorts of customizations. You're customizing smaller, smaller versions of those models. You're able to maybe retrain entire small neural networks that are very domain-specific of those models and just get more and more efficient at processing inference, retaining some of the new fresh data and keeping the flywheel going. So it's a mix. It's definitely a whole ecosystem. It's a thriving ecosystem of different model types, different phases of data, different types of data. But if you keep the flywheel going, it stays relevant with the nature's natural entropy. So yes. - Yeah, it's a fascinating idea. Since we're coming up on time, I want to pick your brain for like one prediction. What do you think is going to happen in the next 12 months? What are you most excited about? - So zooming out a bit, again, this is something that Jensen reference very, very often is that software is fundamentally changing, you know. A year ago, we wouldn't have predicted that all of our engineers really would be using AI for most of their daily work right now was it was heresy even a year ago. And so the not only the rate of change that's happening right now, but the fundamental change in software is that more and more software now is not compiled and run. It's basically compiled run and inference, right? It's really agents now. As we said, being much more intelligent in their token consumption, we're not using opus for everything. We're not using GPT-555 for everything. We're definitely using a now model routing in a mixture of models and a mixture of experts within models right now to be very token efficient. And what that means is now the cost of running a software business is radically different than before. It is a high marginal cost. You can't just leverage, even with, you know, maybe KVCache is the way to leverage, but you can't just leverage the cost of tokens across users the way you could leverage, you know, cloud instances and databases and VMs and micro VMs and containers across users. And SaaS companies, the reason I believe the SaaS PoccoLips is real and is a problem is that no matter how much SaaS companies figure out, you know, new pricing models and new values and you know, value-based pricing and so forth, their op-ex is going through the roof. Their op-ex now is token op-ex. It's tokenomics, it's token consumption. And yes, all of these engineering solutions we just discussed are ways to manage those. But if you just take a look at token volumes on OpenRider a week after week after week, it keeps rising and rising. And we've really just barely begun mainstream token consumption and persistent agent swarms. The only way to run a profitable gross margin business in software will be to own more of the token stack and you can continue to outsource that to an inference provider to a model provider or you can acquire it. You can merge within the cloud, you can merge with the token factory and you can essentially can you know, vertically integrate more of that very expensive token generation stack and continue to run the high gross margin software business. The real question is, with cash loads what they are, will SaaS giants acquire Neo Clouds before Neo Clouds are able to acquire the SaaS giants? That's fascinating. You know, I always spoke about like the best way for large companies to save on token cost is like you bring inference on premises, right? Exactly. So you could run that at the edge. What you're suggesting is like one level, that concept on steroids. A big enough software company can go acquire a Neo Cloud and say that this is my token factory and now I can like, there's still a cost to run the token factory, but the tokens are yours to use. It's entirely yours. Now if every software, big software company starts acquiring Neo Clouds of some size, I mean, they don't have to be like multi gigawatt data. That's amazing. I was predicting, you know, I made this prediction before X acquired cursor or SpaceX AI acquired cursor. I predicted, you know, like workday or Monday dot com or something like that would merge with like a mid to your Neo Cloud. But now of course, first dominoes fallen with the, with X-Face X AI and cursor and older established as companies are going to have to react as well. So yes, I think whether the dominoes start falling in the middle or one side or another, it's kind of inevitable now that most SaaS companies and most Neo Clouds will have to merge. That's a fascinating prediction. I would love to see how that works out. Yeah. Thanks so much, Val. Like it's always a pleasure chatting with you. You're like a fire hose of information that I know I'm going to listen to this podcast later myself as I'm reviewing the edits and stuff and be like, oh my god, I missed that when I spoke to Val. But you know, I hope this helps all of you as well. It's really a pleasure. Always a pleasure. We said we'd enjoy it the next time, last time we did it and I'm definitely looking forward a few months from now from coming back. If you come back well before the end of the year because by the end of the year again, we're going to be very surprised by what happens. It's an eternity. Every three months is an eternity, AI time. So yeah, we should do this more often. All right, guys. That's it for today. Thanks for listening. If you're enjoying semi-doped, please share it with your friends. And we also have a daily newsletter on semi-doped.com where we put our daily takes on the news. It helps us keep a brist of what is happening in this fast, paste AI landscape we are in. And it's entirely free, so make sure to check it out. And thanks for everyone who puts comments on YouTube. We do read all of them. Some of them are like really amazing. Some of them are really funny. We have a good laugh. But we read all the comments even if we don't respond. We promise. And it helps us plan all the future episodes, so definitely keep them coming. And if you can, leave us a five-star review on the Apple Podcast that really helps us out. and catch you on the next one.