What is vLLM? | Agentic AI Podcast by lowtouch.ai

Welcome back to the Egenic AI podcast. Today we are going to talk about the hangover. The hangover. Yeah, the hangover. You know, the tech world has been at this massive party for the last two years. Everyone's celebrating the training runs. We hear about $100 million clusters, trillion parameter models, the sheer scale of just building these brains. It's exciting. It's flashy. It is. It's sort of the space race phase of AI. Exactly. The party is winding down, the bill is arriving, and we have to actually run these things. And that's where the hangover sets in. Because it turns out, serving these models, getting them to do work for enterprises at scale is wildly expensive and frankly inefficient. It's the classic iceberg problem. You know, you see the training costs above the water, but the operational costs, the inference, that's the massive chunk of ice lurking underneath it sinks the ship. And if you're building agents, which is what we care about here, this is a crisis. Because agents aren't just one in done chatbots. They loop. They think. They plan. It's been inferentially crazy. So today, we're doing a deep dive into the infrastructure that fixes this. We're talking about VLLM. VLLM. It sounds like just another library and the Python stack, doesn't it? It does. But based on the research we've been looking at, specifically from UC Berkeley Sky Computing Lab and some great insights from Pradeep Chandran at lowtouch.ai. This isn't just a library. This is the engine that might actually make a jetic AI profitable. We're talking of some pretty wild numbers, like potentially 24x higher throughput and cutting memory waste from something like 60% down to four, which is it's astronomical. I mean, if you told a standard cloud architect, they could reclaim 50% of their hardware efficiency just by swapping an engine, they wouldn't believe you. So let's unpack this. We need to get a little technical today because if you don't understand why LLMs are slow, you can't understand how VLLM fixes it. We're going to talk about page detention, continuous batching, and what this all means for the private AI stack. All right. Let's get into it. Okay. So start with the pain. Why is serving an LLM so much harder than say serving a website or a standard database query? I mean, it's just text in text out. Right. We should wear that simple. The fundamental difference really lies in state. Most web traffic is stateless. You ask for a page, the server sends it, and it completely forgets you existed. But an LLM is autoregressive. Autoregressive, meaning it generates one token at a time based on everything they came before it. Right. And to do that efficiently, it can't reread the entire conversation history from scratch for every single new word it generates that would be, well, insanely slow. So it uses a cache, the KV cache. KV cache, key value cache. I see this term everywhere in the papers. What is physically happening in the GPU memory when we say KV cache? So for every single token, the model processes, whether it's in your prompt or part of the answer, the model calculates a set of, well, they're mathematical vectors, a key vector and a value vector. Okay. These represent the meaning and context of that token for the attention mechanism. So it's not just storing the word Apple. It's storing a really heavy mathematical representation of Apple. Exactly. And these tensors are large. They take up a huge amount of space in the GPU's high bandwidth memory, or HBM. And HBM is the most expensive real estate in your data center. It's incredibly scarce. So the GPU has to hold all these heavy tensors in memory for the whole conversation. Yes. And here is where it's really messy. This is the fragmentation trap. When a request comes in, the serving engine, let's do a traditional one like the default hugging phase setup, it has to allocate memory for this KV cache. But it has no idea how long the conversation is going to be. Right. So the question to get a one word answer, I might get a five paragraph essay. The system can't predict the future precisely. So standard frameworks, they operate on paranoia. They say, well, the model allows for a context window of say 2048 tokens. So to be safe, I am going to reserve a contiguous block of memory big enough for all 2048 tokens right now, just in case. Wait, contiguous is the key word there, right? It is the fatal flaw. It demands a single, unbreakened block of physical memory. Okay. Yes. The sources use a hotel floor analogy. What me through that? Okay. So imagine you run a hotel, a guest checks in. Let's call him request a, he's just one guy. But he tells you, hey, I might invite some friends over later. Maybe one friend, maybe a hundred. In the traditional serving model, because you need to keep his party contiguous, you basically have to evacuate an entire floor of the hotel and reserve it just for him. You lock the doors to 50 empty rooms, just in case his friend show up. It sounds like a terrible way to run a business. It's catastrophic. I mean, even if he never invites anyone, those rooms sit empty, you can't put other guests in them because the space is reserved. In technical terms, this is called fragmentation. You have internal fragmentation, which is the empty space inside his block, an external fragmentation, which is all the little unusable gaps left between the blocks. And what is the data say? Like, how much space are we actually wasting with this paranoia? It's shocking, really. The analysis from the Berkeley team showed that in existing systems, anywhere from 60 to 80% of the KVCache memory is just wasted. 60 to 80%. Yeah, it's just empty slots reserved for tokens that never get generated. So if I go out and buy a massive NVIDIA H100, more than half of its memory is effectively doing nothing. Basically, yes, you are lighting money on fire. And because your memory is full of these reservations, you can't accept new requests. Your throughput hits a ceiling, not because you're out of compute power, but because you literally ran out of memory slots. This brings us to the hero of the story, VLLM. It comes out of Berkeley in 2023, and they introduced page attention. Now the name sounds familiar. It should, if you've ever taken an operating systems class, or even if you've just used a computer in the last 30 years. It's virtual memory, right? Exactly. The researchers looked at this problem and said, wait a minute, operating systems solved this decades ago. Your laptop doesn't reserve a huge, contiguous block of physical RAM for Chrome, just because you might open 50 tabs. It breaks the data up into pages. So how does page attention apply that to the GPU? Instead of demanding that giant contiguous hotel floor, VLLM breaks the KV cache into fixed size blocks. Just small chunks. Say, 16 tokens or 32 tokens worth of data per block. Okay, so we're smashing the long strip of memory into little bricks. Yes. The magic is that these bricks, these blocks, can be stored anywhere in the physical GPU memory. They don't need to be next to each other at all. Block one can be at a dress 100, block two can be at address 5000. It doesn't matter. But hold on, the attention mechanism in the model, the math part. Yes. It needs to see the whole sequence to make sense of the context. If the data is scattered all over the place, doesn't the model get confused? How does it know that block two follows block one? That's the virtual in VLLM. They use a block table. It works exactly like a page table in a OS. It's a look up map. When the model tries to look back at the history, the block table just intercepts the request and says, oh, you're looking for the second part of the sentence. That's over here in physical block number 5000. So it just ditches it together virtually. Right. It creates a continuous view for the model while the physical reality is actually scattered. So going back to our analogy, we're moving from the hotel floor model to what, an apartment complex. It's just a really efficient room assignment system. Now, when a guest checks in, you give them one room. If they invite a friend, you give them another room. Maybe it's down the hall. Maybe it's on a different floor. It doesn't matter. You just write down the room number in your ledger. You fill all the cracks. Exactly. You utilize every single empty room in the building before you turn away a new guest. And the waste reduction numbers are the headline here, right? We go from wasting 60 to 80% of memory to wasting less than 4%. Less than 4. That is optimization at its finest. And that freed up memory isn't just for show. In the world of serving, memory equals capacity. Since you aren't reserving all that empty space, you can jam way more concurrent users onto the same GPU. That's where that 24x throughput claim comes from. Okay. So page attention solves the memory bottleneck. But there's another layer to this, the source is highlight continuous batching. Yes. Pageing is one of those words that makes people's eyes glaze over. But for agent AI, this is critical. Explain the old way first. Why was batching such a problem? So traditionally we used what's called static batching. Think of it like a bus. The bus driver waits at the station until say eight people get on. Then he shuts the doors and drives the route. Okay. Makes sense. But the bus can't stop until everyone is finished with their ride. In AI terms, a ride is generating a response. Now imagine one person on that bus just wants a quick yes-no answer. But another person is asking the AI to ride a 50 page screenplay. Uh oh. I see the problem. The yes-no guy is stuck on the bus. Exactly. He's held hostage. This is called head of line blocking. The entire batch, the whole bus, is locked up until the longest request finishes. Which just kills latency. It destroys it. If you're waiting for a simple response, you might wait seconds just because someone else in your batch is doing some heavy lifting. And VLLM changes this to what a conveyor belt. And you is batching or dynamic batching? And yeah, a conveyor belt is a great way to put it. It operates at the token level. As soon as that yes-no request is finished, it hops off the belt immediately. And this is the key. A new request hops on to take its slot right away. So the GPU is never waiting. Never. It's constantly saturated. But to really appreciate why this matters for agents, we have to distinguish between two phases of generation, pre-fill and decode. Okay. Let's get into the weeds. What's the difference? Pre-fill is when the model first reads your prompt. takes all the text you sent, processes it and parallel and builds that initial KV cache. This is compute bound. It uses the heavy math cores of the GPU. Got it, in decode. Decode is the generation part, producing one token at a time. This is memory bound. It's spending most of its time just moving data back and forth from memory. So they require different things from the hardware? Exactly. And in the old way, you were often stuck doing just one or the other. But with VLLM scheduler, it can weave them together. While one request is stuck waiting on memory to generate a token, that's the decode phase. The GPU can use its compute cores to start processing the prompt for a brand new request. That's the pre-fill. It's multi-tasking. It's pipelining. It fills the compute gaps of the decode phase with the pre-fill work of new requests. I want to pause here and connect this directly to our audience at people building agents. Because we talk a lot about latency as just a number. But for an agent, latency is behavior. That's a really great distinction. An agent isn't just a chatbot. It's a loop. It reasons, it acts, it observes, it reasons again. If I have a chain of thought agent that needs to take, I don't know, 10 steps to solve a problem. And each step has a latency penalty. Right. If you're serving infrastructure is sluggish, if you're stuck on that bus, we just talked about that 500 millisecond delay doesn't just happen once. It happens 10 times. And suddenly your real time agent takes 30 seconds to do a basic task. The user experience just completely collapses. Exactly. It feels broken. This is why VLLM is so critical for the agentic future. It keeps that loop tight. By using continuous batching, the time per token stays consistent even under a really heavy load. So the agent feels responsive. It feels intelligent. And we're seeing this adoption in the wild. This isn't theoretical. Chatbot Arena, the site where everyone goes to test models, they switch to VLLM. And cut their GPU usage by 50%. Roblox is using it for content moderation. Amazon is using it for roofless. They're shopping assistant. These are high volume real time environments. It's becoming the standard. I mean, if you're building a production inference stack today and you aren't looking at VLLM or something derived from it, you're probably doing it wrong. I want to pivot now to the broader ecosystem. We mentioned Pradeep Chandren from lowtush.ai earlier. He frames his efficiency in a really interesting way. He's not just talking about saving money. He's talking about enabling private AI. This is the strategic angle. For a long time, enterprises were kind of stuck. They wanted AI, but they couldn't send their sensitive data, medical records, financial data to a public API like open AI or inthropic. By compliance, we'd shut that down and meet it. So the only alternative was host it yourself. But historically hosting a massive model on premise was nightmare. It was slow. It was expensive. And you needed a team of PhDs just to manage the cluster. They're eye touch. Exactly. But VLLM changes the economics because it makes serving so much more efficient because you can run more throughput on less hardware. It suddenly becomes viable to deploy what Pradeep calls a private AI appliance. Which essentially means running a powerful open source model like Lama 3 or MixedTroll inside your own virtual private cloud, your VPC. Right. You keep all the data within your firewall. But because of VLLM, you don't need a massive sprawling cluster to do it. You can achieve low touch automation. You can have a high performance agent running on a much more modest hardware footprint. So efficiency leads to privacy efficiency enabled privacy. If self-hosting costs, you know, $100,000 a month, most companies won't do it. If VLLM brings that down to 10,000 a month, suddenly every bank and hospital can afford their own private brain. That's a massive shift. It moves us away from the one giant model to rule them all centralized approach toward a world of millions of specialized private agents running everywhere. And Adalines perfectly with where the hardware is going too. I mean, VLLM isn't just for Nvidia anymore. It supports AMD. It supports Intel Gaudi. It's commoditizing the inference layer. Speaking of where things are going, we have to mention VLLM Omni. We've been talking about text, but the world is going multimodal. It is. The future isn't just text in, text out. It's video, audio, images. And VLLM is expanding to handle this. Yes. VLLM Omni is designed to handle multimodal streams simultaneously. Simultaneous. You mean like watching a video and listening to audio at the same time? Exactly. Imagine an agent that's watching a security camera feed, which is video listening to a radio channel. That's audio and reading a log file, which is text all in real time and synthesizing that into a single decision that requires insane throughput. It requires massive memory bandwidth. And remember, page attention. The blocks. That memory management becomes even more critical when you are dealing with huge video frames and audio tensors. I mean, if you think text-waste memory, video is a memory hog. The efficiency gains from VLLM are pretty much the only thing that makes real time multimodal agents economically feasible. It's wild to think that this memory management technique, literally just how we chop up data blocks, is the key that unlocks that kind of sci-fi future. It's always the infrastructure isn't it? We love to talk about the intelligence of the model, but it's the plumbing, the memory paging, the batching schedules that actually lets the intelligence get to work. So let's wrap this up. We've covered a lot of ground today. If we had to distill the VLLM revolution down to the essentials for our listeners, what are the takeaways? Okay. Pillar one is page attention. That's the memory fix. It stops the fragmentation bleeding. We stop treating memory like a hotel floor and start treating it like a flexible filing system. We go from 60% waste to under 4%. Pillar two is continuous batching. That's the speed fix. We stop the bus and we start the conveyor belt. We mix the pre-fill and decode phases so the GPU never has a spare moment to be idle. And pillar three. Democritization through efficiency. By cutting the hardware attacks, we make it possible to run private AI. We enable enterprises to build agents that are secure, fast, and actually affordable. It really is the silent engine behind the eugenic revolution. Without it, agents are just expensive toys. With it, they become viable employees. So here's the question I want to leave you with today. We talk about the hidden tax of AI serving. VLLM effectively gives you a tax rebate. It hands you back 50% of your compute capacity. So if your infrastructure capacity doubled overnight, what would you build? Would you run a smarter model? Would you spawn 10 times as many agents? Or would you finally bring that sensitive data in-house? That's the so what? The ceiling has been lifted. Now it's up to you to fill the space. Huge thanks to the open source community, the team at Berkeley, and the insights from lowtouch.ai for fueling this discussion. It's an exciting time to be building. We'll catch you on the next episode of the Agentec AI Podcast. Keep building.