How to design a highly scalable Distributed cache

Unveiling the Engine: Why Distributed Caching Powers Modern Internet Welcome back to the Deep dive. Today we are peeling back the layers. We're going to set aside the application code, the user interfaces, all that front end gloss. We are plunging right into the invisible infrastructure, the stuff that really defines the modern Internet experience, the engine, the engine that makes these, you know, multi billion dollar services feel absolutely instantaneous. We're talking about speed, latency, and just massive scale. Speaker 2 That's exactly it. And when you interact with any application that's handling I mean monumental traffic think about the busiest e-commerce sites, the big social networks, you run into this kind of engineering paradox, right? How do these systems dealing with billions of requests every single day get you the data you need faster than their primary database can possibly respond? Speaker 1 Because the database is built for safety, not necessarily for pure speed. Speaker 2 Exactly. Its job is safety persistence, and that inherently adds latency it has to. Speaker 1 And the answer to that paradox is this highly specialized lightning fast layer that's built entirely for reuse for rapid retrieval. Speaker 2 The cat, but not just any cache. This is not, you know, a simple file on a server disk somewhere. Speaker 1 No, today our whole focus is on designing the architectural backbone of these high speed services. We're talking about the distributed cache, and our mission for you listening is to really understand the complex engineering decisions you have to make to manage truly massive scale, specifically a peak traffic load that demands 1,000,000 operations per second. A. Speaker 2 Million queries per second QPS. It's an enormous mandate A. Speaker 1 Sustained million QPS. It's mind boggling. Speaker 2 It is, but it's the kind of engineering that underpins almost every major service you use, you know, every single day. So to tackle it, we need to be really clear. Caching. At its heart, it's just a high speed storage layer. Speaker 1 Usually in RAM, right? Speaker 2 Usually in volatile memory like RAM. Yeah, and it's dedicated to storing transient data. It's only purpose is to reuse data that's already been fetched or computed so you can cut out that slow, expensive trip back to the main database. Speaker 1 And when we put the word distributed in front of that, what does that change? Speaker 2 Well, distributed caching is the mandatory next step. It's about spreading that high speed transient data across a huge network of dedicated servers. Speaker 1 It's the only way. Speaker 2 It's the only path to achieving what we call horizontal scalability. It lets us grow our capacity just by adding more machines. By decoupling the cache onto its own cluster, we relieve that crippling load that would otherwise just flatten the primary database. Speaker 1 So our objective is clear. We're designing a highly available ultra low latency system. It has to deliver sub millisecond read and write latency, handle these enormous loads and, you know, stay resilient when things inevitably fail. Speaker 2 Right, let's get into it so. Beyond Limits: Why Single Servers Fail at Massive Scale Let's start with the first hurdle. Why do we have to go distributed? Why can't we just, you know, buy a single monster server loaded up with a colossal amount of RAM and just call that our cache? Speaker 2 That's the perfect first question, because it gets us right into the reality of enterprise scale, both in terms of memory and traffic. First, let's just talk physics. Modern servers are powerful, but the amount of data you need to cache often goes way beyond what's practical or even economically sensible for a single machine. Speaker 1 You just hit a wall. Speaker 2 You hit a hard ceiling. A high end server might top out at what, 128 gigs? Maybe 200 and 56512 gigs of RAM if you're lucky. If a major platform needs to cache terabytes of user profiles or product catalogs or session tokens, a single machine just isn't an option. Speaker 1 So the physical hardware limitations force your hand. You have to scale horizontally. Speaker 2 Precisely. And that leads you to the second reason, which is architectural decoupling. What do you? Speaker 1 Mean by? Speaker 2 That I mean you need the ability to scale your memory capacity separately from your applications processing power. If your application needs more CPU cores to handle business logic, fine, you scale up the application tier. But if you see a surge in the volume of cache data, you should be able to just scale the cache cluster by adding more nodes without ever touching the app server. Speaker 1 That makes so much sense. You don't want those two things coupled together. Speaker 2 It's vital for operational efficiency. It prevents all sorts of system wide problems. Speaker 1 And then there's the traffic. Those numbers we mentioned, 100 billion operations daily, that 1,000,000 QPS peak, that is just a constant enormous barrage of requests that no single network card or CPU could ever hope to handle. Speaker 2 Yeah, let's ground this in some real numbers for you, just to see how quickly this explodes. Let's say our goal is to store 100 terabytes of data in memory. That's just the working set, right? Speaker 1 Just the data itself, and then you have to account for resilience. Things fail. Speaker 2 Exactly. For high availability you need replication. Let's say we use a standard 3X replication factor. Speaker 1 So every piece of data exists in three places. Speaker 2 Right. Which means means you actually need 300 terabytes of physical memory across the whole cluster. And you can't run your nodes at 100% capacity, you need some headroom. So let's target say 80% utilization. And if we choose pretty standard cache nodes, maybe servers with 64 gigs of RAM each, well, the number of nodes you need just explodes. Speaker 1 OK, walk us through that math. Speaker 2 You take the total required memory, which is 300 terabytes or 300,000 gigabytes. Then you divide that by the effective capacity of each node. Speaker 1 Which is 64 gigs times 0.8, so 51.2 gigabytes. Speaker 2 Right, so 300,000 / 51.2 that comes out to about 5859 active nodes. Speaker 1 Almost 6000 servers. Speaker 2 And that's just to store the data and handle redundancy. You factor in a buffer for failures and you land somewhere near that 6250 total node requirement. Speaker 1 That just makes it so clear a single machine is an absolute non starter. Speaker 2 It's not even in the conversation. Speaker 1 So now that we know why it must be distributed, let's lay out the ground rules. Understanding Cache Needs and When to Strategically Bypass It What must this maths of thousands of node system actually be engineered to do? Speaker 2 We can break these down into functional and non functional requirements. Functionally it's pretty straightforward. It has to 1 read and write data quickly with key value pairs 2. It needs a smart eviction policy like LRU to manage its finite memory. Speaker 1 We'll get into that. Speaker 2 Three, it has to support replication for fault tolerance, 4 maintain consistency across all the nodes, and five, allow for dynamic node management so you can add or remove servers without everything crashing and. Speaker 1 The non functional requirements, OK, this is the quality metrics right? The how well it has to do its job. Speaker 2 These are the really challenging ones. First performance, non negotiable, sub millisecond read, write latency. But here's the nuance. We don't just care about the average. Speaker 1 This is so important. Speaker 2 We have to focus on tail latency, specifically the P-95 and P99 response times. Speaker 1 So for anyone listening, that's the experience of your 95th and 99th percentile of users, the worst experiences. Speaker 2 Exactly. Then scalability. It has to be truly horizontal. Add end nodes. You should get roughly end times, the capacity, reliability of course, 99.99% uptime or better, 4 nines, 4 nines and finally elasticity, the ability to scale up and down with demand with minimal disruption. Speaker 1 You mentioned tail latency and I want to double click on that. Why is optimizing the P99 that worst 1% of requests, so much more critical than just looking at the average? Speaker 2 Because the average is so deceptive. I mean, if 90% of your requests take 5 milliseconds, but 10% take 500 milliseconds, your average might still look pretty good on a dashboard, right? But that tail, those 500 millisecond responses, that's a huge number of your users having a painful, slow experience. At a million QPS, the P99 is the most critical indicator of your users worst case experience. Speaker 1 So optimizing for it means you're actively hunting down those internal problems, garbage collection, network hiccups. Speaker 2 Slot shards, all of it. A successful cache design is 1 where the P99 latency is almost as low as the average. Speaker 1 That sets the bar incredibly high. OK, before we move on, let's address the flip side. Caching adds complexity. When should an engineer look at a problem and consciously decide not to use a distributed cache? Speaker 2 This is a real test of architectural contextual maturity. You avoid it when the complexity and cost outweigh the benefit. Speaker 1 Give us some scenarios. Speaker 2 1st, when stale data is completely unacceptable. If your system demands absolute real time consistency, think of a Stock Exchange. Financial ledgers relying on a cash which inherently introduces some risk of staleness is just a bad idea. The performance gain doesn't justify the data integrity risk. Speaker 1 OK, what else? Speaker 2 2nd, if your traffic patterns are non repetitive, caching is all about reuse. If every request reads a unique massive object, or if you're just processing log files line by line, your cache hit rate will be near 0. Speaker 1 So you've added this whole complex layer for, no? Speaker 2 Reason you've actually just added latency and cost. And 3rd, if your system is super write heavy, say 90% writes, 10% reads, the overhead of keeping the cache consistent might actually slow down the whole system. Speaker 1 So it's not a silver bullet, it's a strategic tool for read heavy, latency sensitive workloads that can handle a little bit of staleness. Speaker 2 Exactly right. It's a calculated engineering decision, not a default choice you just reach for. Choosing Your Strategy: Cache Aside, Read Through, Write Back OK, this is where it gets really interesting, because once you decide to use a cache, the way the application, the cache and the database talk to each other, Yeah, that relationship defines everything. Speaker 2 It's the central negotiation of the entire system design and there are really four primary patterns we need to look at and each one comes with a different set of guarantees and frankly different failure modes. Speaker 1 Let's start with the one that puts the most work on the application itself, cash aside. Speaker 2 Yeah, so with cash aside, the cash literally sits aside from the main data flow. The application code is the puppet master. When it needs data, the application first sends a GET request to the cash. If it gets a hit, great. Data is served. If it's a miss. Speaker 1 Application has to go to the database. Speaker 2 It goes to the database, gets the data, and then this is the critical part. It has to do a second PUT operation to write that data back into the cache before it can finally return it to the user. Speaker 1 So the application is the manager. What's the benefit of the app doing all that heavy lifting? Speaker 2 Well, the main benefits are simplicity and resilience. Since the application knows about the database, if the entire cache cluster just disappears. Speaker 1 A network partition or something? Speaker 2 Right, the application can just seamlessly bypass the cache and keep working by hitting the database directly. That's a huge win for fault tolerance. Speaker 1 But the downside has to be managing all that consistency logic in the application code. Speaker 2 It's a massive headache. The developer is now responsible for managing time to live or TTL to expire stale data. When the application writes new data to the database, it has to remember to also invalidate or update the cache. Speaker 1 And if a developer forgets? Speaker 2 You get what's called the dual write problem, and now your cache is permanently out of sync until the TTL happens to expire. Speaker 1 OK, so if cache aside puts the burden on the application, how do we shift that responsibility onto the cache itself? Speaker 2 That brings us to our second pattern. Read through. OK. With read through, the application only ever talks to the cache. The cache becomes the single source of truth for reads. Speaker 1 So the app is totally blind to the database. Speaker 2 Completely, when the application issues a GT, if there's a cache miss, the cache component itself contains the logic to go fetch the data from the database, update its own storage, and then return the result. Speaker 1 That sounds like it simplifies the application code immensely. Speaker 2 It does a ton. The application basically treats the cash as if it were the database. But you know that complexity doesn't just vanish, it moves into the cash component. Speaker 1 So now the cash has to know how to talk to the database. Speaker 2 It needs connection details, credentials, it has to understand the data model and more importantly, you lose some resilience. If the cash layer goes down, the application has no fall back. It can't bypass it. So your downtime risk goes up, right? Speaker 1 Let's talk about rights now. The first one is right through. Speaker 2 Right through is all all about strong consistency. The application writes to the cache. The cache then synchronously writes that data to the database and updates its own local copy. Speaker 1 Synchronously is the keyword there. Speaker 2 It is. The cache only sends an OK back to the application once both the cache and the database have confirmed the write was successful. Speaker 1 Which guarantees that the data is always fresh. Speaker 2 Precisely. You get strong consistency between the two layers. This is great for things that are likely to be read immediately after being written, like a user profile update. Speaker 1 The trade off is latency. Speaker 2 A big one. You're right. Latency is much higher because the application is waiting for two network round trips, one to the cache and then another from the cache to the database. It's substantially slower. Speaker 1 And then we get to the polar opposite, the one that's optimized purely for speed, right back, also called right behind. Speaker 2 Right right back basically throws synchronous consistency out the window in the name of speed. The application writes to the cash and the cash immediately says Yep, got it, success. Speaker 1 It's lying to the application. Speaker 2 It's totally lying. The actual right to the database is deferred. It happens asynchronously, maybe in a batch every few seconds or even minutes. Speaker 1 So the user gets this lightning fast confirmation, but the data isn't actually safe in the durable store yet. Speaker 2 Yes, and that's super efficient for certain things. It's great for extreme right heavy workloads, sensor data, location updates, logging where you're generating a huge volume of data and you don't want to bombard the database. Speaker 1 But the immediate consequence of that speed is the frankly terrifying risk of data loss. Speaker 2 That is the immense trade off. If that cash note crashes before it performs the asynchronous write to the database, that data is gone forever. Speaker 1 So how do you even use this pattern safely? Speaker 2 You have to implement some very sophisticated safeguards. Usually you'd use something like a write ahead log or a wall on the CAF server's local disk. That log records the intended write, so if the process crashes, it can replay the log on restart and complete the write to the database. Without that, write back is just too dangerous for most use cases. Speaker 1 And there's one more, a simpler one. Write around. Speaker 2 Yeah, write around is simple. The application writes directly to the database, completely bypassing the cache. The data only gets into the cache later when a read request misses the cache and forces a load from the database. Speaker 1 Why would you do that? Speaker 2 It's great for preventing what we call cache pollution. You avoid loading data into your expensive cache that gets written once and then is never or very rarely read again. It keeps your limited cache space reserved for the truly hot, frequently accessed keys. Speaker 1 So the choice of strategy is, it's not arbitrary, it's a core decision that defines your system's performance and its data safety guarantees. Speaker 2 It is the exact moment where the rubber meets the road on the CAP theorem trade-offs for your specific application. Scaling Without Chaos: The Power of Consistent Hashing OK, so we've established our system needs thousands of nodes. We mentioned 6250. Now we need a mechanism to know for any given key, exactly which of those thousands of servers holds the data, right? And if we tried the naive approach, something like hash key MODN where N is the number of servers, yeah, that just fails instantly if N changes. Speaker 2 The failure of that modulo hashing is the whole reason distributed systems design had to evolve. If you lose just one note, so N drops from 6250 to 6249, the result of the modulo changes for almost every single key in the system. Speaker 1 And the consequence is a. Speaker 2 Global catastrophic cache invalidation. All 1,000,000 of your quick queries per second are suddenly routed directly to the back end database and your entire system crashes. It's a guaranteed outage. Speaker 1 We need a system that minimizes that disruption, and that system is consistent hashing. Speaker 2 Consistent hashing is the non negotiable architectural backbone for scaling any modern distributed key value store. It's a really elegant solution built on a logical ring structure. OK, a wing. Imagine a continuous space, say from zero to two to the 64 mapped onto a circle. We use the same hash function to map both our cache servers and our data keys onto points along this ring. Speaker 1 So everything, servers, keys, they all get an address on this circle. How does the assignment work then? Speaker 2 A key is assigned to the very first cache noted encounters as it moves clockwise around the ring, and the crucial advantage of this geometry is how little disruption there is when things change. If a server fails and is removed from the ring, only the keys that were previously assigned to that server need to be remapped. Speaker 1 And where do they go? Speaker 2 They just get reassigned to the next server clockwise. So mathematically, the data migration is localized to roughly one over north of the keys, where N is the number of servers. Speaker 1 That reduction is incredible. You go from basically 100% of keys being remapped to less than .01%. That allows for true elasticity, but I've heard even this can have problems like if servers cluster together on the ring. Speaker 2 That's the classic problem of non uniform distribution. You get hotspots where one or two nodes end up with way more data or traffic than their neighbors. Speaker 1 How do you solve that? Speaker 2 The solution is to introduce virtual nodes or V nodes. Instead of mapping a physical server to just one point on the ring, we map each physical server to hundreds of virtual positions, maybe 150 to 200 V nodes per physical machine. Speaker 1 Why do multiple virtual nodes fix the distribution? Speaker 2 It's just a statistical averaging trick. By spreading a server's identity across all these different points on the ring, we dramatically increase the probability that the data distribution across the entire cluster will be smooth and even. Speaker 1 It just averages out the randomness. Speaker 2 Exactly, the trade off is a bit more memory overhead and look up complexity, but the gain in balance load management is absolutely essential for a system doing a million QPS. Speaker 1 OK, so we have keys and nodes on the ring. The application still needs to know which of the thousands of nodes to connect to for a specific key. What component handles that? That is the job. Speaker 2 Of the Cache client library. This is a lightweight piece of code that you integrate directly into your application. It runs on the client side and it's not just a. Speaker 1 Dumb connector? Not at all. It's an. Speaker 2 Intelligence Hub. Its job is to maintain an accurate sorted list of all the known healthy cache servers in the cluster. When the application requests a key, the client library runs the consistent hashing algorithm on the key, uses a super fast method like binary search on its sorted list of hosts, and then routes the request directly to the correct physical node so the client is. Speaker 1 Doing all the heavy lifting which avoids a central routing bottleneck. But that means every single application server's client library needs to have a perfect synchronized view of the cluster. How does that happen? This is where dedicated. Speaker 2 Cluster management and health services become absolutely critical. The state of the cluster, which nodes are alive, where they are on the ring? That has to be stored consistently somewhere, and you have a few. Speaker 1 Ways of doing that right, we rely. Speaker 2 On three key mechanisms. First, heartbeats. Just simple low latency ping pong checks between a manager and the nodes to make sure they're alive OK. Second, the gossip rotocol. This is really cool. Nodes constantly and randomly exchange membership info with their neighbors. It helps the whole cluster build a decentralized consensus view of itself really quickly. And the third one is. Speaker 1 For the authoritative state, exactly. Speaker 2 You use a configuration service like Zookeeper or etcetera. These systems are the source of truth, and they use complex, robust consensus protocols like Raft or Paxos to make sure that even if the network gets weird, every node agrees on the exact current state of the cluster that consensus. Speaker 1 Service is the safety net that prevents total chaos. It is and. Speaker 2 Once you have that monitoring, it lets you handle change gracefully. When a new node is added, the configuration service updates, the client libraries are notified and you trigger an automatic rebalancing. And because of consistent. Speaker 1 Hashing that rebalancing isn't a disaster, it's smooth. Speaker 2 The system calculates the minimal set of keys that need to move to the new node, and it does it gradually in the background with minimal impact. That is the definition of elasticity at this scale. Protecting Data: Replication, Eviction, and Cache Persistence We've built the scaling. Speaker 1 Engine, but it's all running on volatile RAM and distributed systems are, you know, inherently prone to failure. We need strategies for fault tolerance and for managing that finite memory. Let's start with that 99.99% availability requirement. High availability. Speaker 2 Absolutely requires redundancy, which we get through replication. The most common strategy here is leader follower replications, sometimes called master. Speaker 1 Slave right the. Speaker 2 Leader or master is the source of truth. It handles all the right operations. The followers or replicas just mirror the data and critically they help share the read load, so the replication. Speaker 1 Ensures the data survives if a master fails. But what's the consistency model here for a cache? Do we sacrifice speed for total safety? For caching? Speaker 2 The priority is almost always speed, so asynchronous replication is the default. What does that mean in? Speaker 1 Practice. It means the leader. Speaker 2 Acknowledges the right to the client immediately, without waiting for the followers to confirm they've received the data, which makes right. Speaker 1 Super fast. Very fast. Speaker 2 But it introduces the possibility of eventual consistency. A follower might, for a very brief period, serve stale data until the right propagates to it. And for a cache, that small window of staleness is usually an acceptable trade off for the massive performance boost. And if you needed. Speaker 1 Absolute consistency. You'd use synchronous replication, but that would slow everything down. It would make rights. Speaker 2 Much, much slower. The master would have to wait for confirmation from a quorum of replicas across the network before it could acknowledge the client. It's just too slow for our submillisecond goal. OK, so a leader. Speaker 1 Fails. How does the system recover automatically? This is where. Speaker 2 That automated failure handling kicks in. The configuration service like Zookeeper is constantly monitoring the leader with heartbeats. If it detects a failure, maybe 3 missed heartbeats in a row, it initiates an automatic failover. It elects a new. Speaker 1 Leader. It triggers an. Speaker 2 Election protocol promotes the most up to date follower to become the new leader and updates the cluster configuration notifying all the clients of the change. And this whole process has to happen in milliseconds to maintain those high availability targets. Now let's talk about. Speaker 1 Managing that finite, expensive memory. We can't store everything forever. We need a way to intelligently evict data, right? We need. Speaker 2 Robust eviction policies. The first one is simple time to live or TTL. It's essential for data that is naturally time bound, like a user session. It just automatically expires after a fixed duration. But what happens when? Speaker 1 The cache just fills up before things expire. Then we turn to. Speaker 2 Algorithms and the industry standard is least recently used or LRU, so tell us. Speaker 1 Not just what it is, but how it's implemented under the hood to get that O of 1 performance that you need for millions of operations a second. OK, so. Speaker 2 LRU evicts the item that was accessed furthest in the past. It seems simple, but making it performant is a really elegant data structure puzzle. You can't just search a. Speaker 1 Huge list? No, that would. Speaker 2 Be O of N Way too slow. The solution is to combine two data structures, a hash map and a doubly linked list. OK, how does that? Speaker 1 Combination work. The hash map gives. Speaker 2 You O of one constant time look up for any key. The value stored in the hash map isn't the data itself, it's a pointer to a node within the doubly linked list and the list. Speaker 1 Maintains the order exactly. Speaker 2 The list maintains the order of recency. The head of the list is the most recently used item, and the tail is the least recently used. Got it? So when an item is access to cash hit, you use the hashmap to find its node in the list in O of one time. Then you just manipulate a few pointers to move that node to the head of the list, which is also O of one. When you need to evict something, you just remove the node at the tail. That's brilliant. Speaker 1 That combination gives you O of one for all the critical operations. What about the alternative? Least frequently used? Yeah, least. Speaker 2 Frequently used or LFU VIX the item that's been accessed the fewest number of times. It's theoretically better for applications with really stable access patterns because it protects popular items that maybe haven't been touched in a minute, but it's harder. Speaker 1 To implement right it's way. Speaker 2 More complex to implement in a performant distributed way. I mean, think about trying to track the true frequency count for billions of keys across thousands of nodes. It's very resource intensive. O how do modern? Speaker 1 Systems do it without, you know, burning all their CPU. They almost. Speaker 2 Never calculate the true LFU count perfectly. They use approximations. A common technique is something called a count min sketch. What's that? It's a probabilistic data structure. It uses hashing in a few arrays to estimate frequency counts with very high accuracy, but it uses vastly less memory than storing A precise counter for every single key. So for Massive. Speaker 1 Scale. You trade mathematical certainty for high probability estimates to maintain speed. That's a huge. Speaker 2 Theme in distributed systems? Exactly. Let's talk. Speaker 1 Durability. If the whole cluster restarts at once, a cold start, we lose everything in memory and we get a massive Stampede to the database. How do you prevent that? You have to rely. Speaker 2 On disk persistence, even though the cache is supposed to be transient, the two main strategies are snapshots and logging. First is RDB or snapshots. This takes a periodic point in time snapshot of the whole data set and writes it to disk. Recovery is fast, you just load one big file, but you can lose any data written since the last snapshot and the other is logging. Speaker 1 Right AOF. Speaker 2 Or a pend only file. This logs every single write command the cache receives. It offers much better durability because every transaction is recorded, but recovery is way slower because you have to replay the entire log file to rebuild the state. So the hybrid? Speaker 1 Approach is probably best, yes. Speaker 2 A resilient system will use periodic RDB snapshots for fast initial recovery, and then supplement that with continuous AOF logging for the most recent data. That protects the primary database from getting hammered after a failure. Resilience at Scale: Hotkeys, Stampedes, and P99 Latency OK, finally. Speaker 1 Let's address the failures that are caused by popularity and concurrency. Hotkeys and cash Tampedes. These are huge problems. At a million QPS, a hotkey is. Speaker 2 Just a consequence of unbalanced traffic. One single key, a viral news story. A celebrity's profile gets a totally disproportionate number of requests. And if that key lives on a single node, that node just gets overwhelmed. So how do you? Speaker 1 Stop one key from syncing an entire server. You have to distribute. Speaker 2 The key itself. Since you can't stop the traffic, you spread the load. You use key splitting. OK instead of one key like viral article .123, you generate multiple sub keys like viral article dot 123.0.1.2 and so on. You distribute those sub keys across different physical nodes. The application client then randomly queries one of them. You're effectively load. Speaker 1 Balancing the request for a single item across the cluster exactly you turn. Speaker 2 One key into many traffic lanes and the cache DMP. Speaker 1 That's a concurrency problem when a key expires. It's a classic. Speaker 2 Race condition failure. A super popular key expires. Thousands of clients all try to read it at the same time. They all get a cache miss and they all rush to the back end database at the exact same moment to get the fresh data and the database. Speaker 1 Just collapses. It collapses. Speaker 2 From the overload. So you need to ensure that only one client is responsible for regenerating that data while everyone else waits patiently. How do you do that? Speaker 1 The most common? Speaker 2 Way is with a distributed lock. The first client to see the miss acquires a lock for that key. All the other clients who come along see the lock is taken and they wait. The client with the lock goes to the back end, updates the cache and then releases the lock. And there are other ways. Speaker 1 Too right? Yeah, you can do. Speaker 2 Request coalescing which is similar or a strategy I really like called stale while revalidate. How does that work? Speaker 1 When a key expires. Speaker 2 The system serves the old stale data back to the user immediately, so the user sees no latency, but at the same time it triggers a single asynchronous background process to go fetch the fresh data. It completely protects the database these. Speaker 1 Defenses really show that designing a high speed system is all about anticipating how massive scale will cause failures and then engineering specific defenses for those scenarios. Absolutely. Speaker 2 Resilience at a million QPS means you have to assume every protective layer will fail at some point, and then you build redundancy into the failure handling itself. This has been an. Speaker 1 Incredibly deep and very technical dive into the architecture running right beneath our favorite apps. We've gone way beyond a simple hash table to build a globally distributed fault tolerant system. And the key. Speaker 2 Takeaways are really foundational for anyone serious about this stuff. Consistent hashing is not optional. It's the mathematical guarantee that lets you scale without catastrophic failure. And that choice of cash access pattern, especially right through versus right back, that's the line you draw in the sand that defines your system's latency versus it's data consistency guarantees. And as we wrap up. Speaker 1 Let's just tie this back to the philosophical foundation of all distributed systems design. The CIP theorem indeed. Speaker 2 The CIP theorem forces you to make an inescapable choice, and when you're designing caching systems for speed and scale, the standard architectural decision is to intentionally sacrifice strict consistency, so we accept that. Speaker 1 Data might be a little stale sometimes. We accept eventual. Speaker 2 Consistency. We accept temporary staleness via TTLS and async replication all in favor of maximizing availability and partition tolerance. That trade off is what unlocks the sub millisecond performance. You need that ability to. Speaker 1 Tolerate imperfection is really the secret to modern high speed design. It is. Speaker 2 And a final provocative thought for you to take away. While we focused a lot on the mechanics of speed and capacity, remember that these systems are living, breathing things. The real measure of success isn't just the average speed when things are going well, it's how it behaves. Speaker 1 When things go wrong, it's the deep. Speaker 2 Continuous observability of its failures. Managing a system at the scale means you have to track things like replication lag and error rates, but most importantly, you have to be obsessed with your P99 latency. Your worst case. Speaker 1 User experience optimizing. Speaker 2 That 99th percentile response time is the active pursuit of fixing that experience, and that commitment is the real difference between a system that just works and a system that is truly resilient, that focus. Speaker 1 On the P99 is where engineering elegance really meets the user experience. Thank you so much for joining us for this deep dive into designing the backbone of high speed systems.