Pillars of System Performance: Throughput, and Latency

Introduction Welcome to the deep dive. Today we're jumping into two really fundamental concepts for system performance, latency and throughput. Speaker 2 Yeah, terms you hear all the time. Speaker 1 Exactly. But you know how they actually interplay the trade-offs you have to make when designing systems. That's what we want to unpack today. Speaker 2 Because it's not always straightforward. Speaker 1 Right, we'll look at some real world stuff and try to give you a clear handle on it without, you know, getting totally lost in the jargon. Speaker 2 Is good because latency, essentially the delay and throughput, how much stuff gets done. They seem simple, but they're relationship that defines so much about how a system feels and what it can handle. Speaker 1 OK, so let's start with latency, then simple terms. Network Latency First, it's just the time it takes for something to happen, right? Like you click a link and the time until the page starts loading that delay. Speaker 2 Exactly that is that lag, that waiting time. But even that simple idea breaks down a bit. OK, well, you've got network latency. That's purely the time it takes for data to physically travel across the network. Speaker 1 Like sending a signal down a wire. Speaker 2 Pretty much it includes the propagation delay, just the travel time plus transmission delay, getting the data onto the wire, and any queuing delay like waiting it's turn out a router, right? And just for perspective, sending a small packet from say California to the Netherlands and back, you're easily looking at around 150,000 microseconds just for that round trip over the network. Speaker 1 Wow, OK, that's actually noticeable. Speaker 2 It can be, yeah. And that's before you add in the other part, system latency. Speaker 1 Which is the thinking time. Speaker 2 Kind of. It's the processing time on both ends, The client machine getting the request ready, the server running its code talking to the database figuring out the response, and the client rendering it. All that computation. Speaker 1 Gotcha. So it's not just the travel time, it's the processing of both ends too. Speaker 2 Precisely, and loads of things can affect both. For the network side, congestion is a big one. Too much traffic jamming things up. Speaker 1 Like a traffic jam on the highway. Speaker 2 Exactly. Then there's just the raw bandwidth limits, the physical distance the data has to cover, the overhead of the network protocols themselves like TCP needing acknowledgments, and even things like packets getting lost and needing to be resent. Speaker 1 That all adds delay. Speaker 2 Definitely the number of hops, the type of connection, fiber versus copper, it all plays a part. Speaker 1 And on the system side, what slows things down there? Speaker 2 Oh, lots of things. Server load is a major one. If the server's already busy, it'll take longer. Slow database queries maybe inefficient application code fighting over resources like CPU or memory, Even whether the needed data isn't a fast cache or needs fetching from a slow disk. Cache misses can really hurt latency. Speaker 1 Right. OK, so if we want to actually put a number on this latency, how do we measure it? How to measure latency What tools do people use? Speaker 2 Well, for basic network checks ping is the classic tool. It gives you the round trip time. The RTT then trace route shows you the path the data takes hop by hop and the delay at each point. MTR is kind of neat. It combines ping and trace route for ongoing monitoring more. Speaker 1 Advanced stuff. Speaker 2 Yeah. And then for digging into the application itself, APM Tools, Application Performance Monitoring are invaluable. They can show you where time is being spent inside your code or in database calls, right? Or you know, you can always add your own timing code custom instrumentation if you need really specific measurements. Speaker 1 And when we get these measurements, what are the key numbers, the metrics we should look at? Speaker 2 RTT is fundamental. As we said time to 1st byte or TTFB is useful. How long till the first bit of the response arrives? And often we look at the total end to end latency for a full request response cycle. But really importantly, especially for user experience, you need to look get percentiles. Speaker 1 Percentiles like P90, P 99. Speaker 2 Exactly. P50 is the median. Half your requests are faster, half are slower. But P90 tells you the latency experienced by 90% of requests. That means 10% are slower than that number. Speaker 1 So it shows the worst end of the experience. Speaker 2 Right, and P99 shows the latency for the slowest. 1% averages can hide really bad outlier experiences. If your average latency looks OK, but your P99 is terrible, that means 1% of your users are having a really bad time. Speaker 1 Which could be thousands of users for a popular service. Speaker 2 Precisely, And that P 99 might be the person trying to complete a purchase who gets frustrated and leaves. So understanding those tail latencies is crucial for SLA's for alerting a spike in P 99 is often your first sign of trouble. Speaker 1 That makes perfect sense. You mentioned RTT and gave that California, Netherlands example. How to estimate latency between different cities Could we quickly walk through how you'd estimate that, say, between New York and London? Speaker 2 Sure. Let's say the distance is roughly 5570 kilometers. Data in fiber optic cables travels at let's say around 200,000 kilometers per second slower than light in a vacuum because of the glass. So for one way you divide the distance by the speed 55170km / 200,000 kilometers, that comes out to about .02785 seconds or 27.85 milliseconds. Speaker 1 Just for the signal to get there. Speaker 2 Yeah, that's the propagation delay. The round trip time RTT is roughly double that, so around 55.7 milliseconds. Speaker 1 OK, so nearly 56 milliseconds. Speaker 2 Oh, remember, that's the absolute minimum. That's just the speed of light limit in fiber. It doesn't include any processing time, router delays, congestion, the the actual latency will always be higher. Got it. Speaker 1 That's a good baseline to understand, though. All right, let's switch gears then. What is throughput? What about throughput? What's the essence of that? Speaker 2 Throughput is all about volume or read. It's how much work your system can actually get done in a given period. Speaker 1 So not how fast one thing happens, but how many things happen. Speaker 2 Exactly. Think of it as the capacity. How many requests can the server handle per second? How much data can you transfer per second? Speaker 1 And the units reflect that right? Bits per second, requests per second. Speaker 2 Yeah, for networks, you see bits per second, BPS, megabits, Mbps, gigabits, GBPS, like your Internet connection speed. For systems, it's often requests per second RPS or transactions per second TPS, or maybe operations per second, depending on what the system does. It's quantifying the rate of completion. Speaker 1 Does throughput also breakdown into network and system components? Speaker 2 It does, yeah. Network throughput is the actual data rate you achieve over the network. It's often less than the theoretical maximum bandwidth because of, you know, network conditions and protocol overhead. Speaker 1 Right, reality versus the label on the box. Speaker 2 Kind of, yeah. And then system throughput is the rate your back end servers, databases, applications can process the work. That depends heavily on your resources and architecture. Speaker 1 OK. How to increase system throughput So what limits in its throughput? What are the bottlenecks usually? Speaker 2 On the system side, resources are key CPU power, available memory, how fast your disks can read and write data, that IO performance, and of course the network bandwidth to the server. Speaker 1 The fundamentals. Speaker 2 Right. But also the application design is huge. How does it handle concurrency? Is it using threads efficiently? Is it doing asynchronous operations? How efficient are the algorithms or the database queries optimized? All that matters. Speaker 1 So if throughput is too low, what can we do? How do we increase it? Speaker 2 Well, there are a few main strategies. 1 is vertical scaling just making your existing servers beefier? More CPU, more RAM. Speaker 1 Throw more hardware at it. Speaker 2 Basically, but that only goes so far. Often horizontal scaling is more effective long term. That means adding more servers, not just bigger ones. And if you do that, you need load balancing to spread the traffic evenly across those servers. Go cancel other things. Using asynchronous processing helps a lot so the system isn't just waiting around for slow tasks. Batch processing can be good for certain workloads. Grouping similar things together for efficiency and optimizing resource use is always important. Connection pooling to reusing connections instead of creating new ones constantly. That reduces overhead and helps throughput. Speaker 1 OK. That's a good overview. So we have latency for speed, throughput for volume. How latency for speed and throughput for volume relate How do they relate? Where do the trade-offs usually happen? Speaker 2 Yeah, the trade-offs. This is where system design gets interesting, because optimizing for one often negatively impacts the other. You usually can't Max out both simultaneously. Speaker 1 It's a balancing act. Speaker 2 It really is. It depends entirely on what the application needs most. Speaker 1 Can you give some examples of that tension where you prioritize one over the other? Speaker 2 Sure. Think about large scale batch processing, like generating payroll or crunching huge data sets overnight. These systems are built for massive throughput. They process a ton of data. But the latency for any single piece of data might be quite high. It might sit in a queue for hours before it even gets processed. My throughput? High latency OK. Speaker 1 Opposite example. Speaker 2 Financial trading systems High frequency trading. Here latency is everything. A millisecond delay can cost millions, so they're incredibly optimized for speed for low latency, but they might not handle the same sheer volume of data per second as a big batch system. Their throughput might be lower, relatively speaking, because the focus is absolute speed for each transaction. Low latency, possibly lower throughput. Speaker 1 And what about something in the middle, like a website? Speaker 2 Yeah, web applications often need a balance. You need decent responsiveness, low enough latency so pages load quickly and feel snappy. But you also need enough throughput to handle potentially thousands or millions of users browsing, searching, buying things all at once. So you're trying to find a sweet spot. Speaker 1 OK. So when we're designing and we know we need to maybe lean one way or the other, what strategies help reduce latency specifically? Strategies to reduce latency Caching is probably the number one latency killer. Storing data closer, avoiding slow lookups makes sense. CDNS content delivery networks do this for static files. Geographically optimizing database queries is huge. Minimizing network hops maybe designing your network path better using efficient algorithms in your code. Connection pooling helps latency too by avoiding connection setup time. Speaker 1 And if the priority is pushing more volume, boosting throughput? Speaker 2 Then horizontal scaling with load balancing is often the go to. As we mentioned, spread the workout. Speaker 1 More lanes on the highway. Speaker 2 Good analogy. Asynchronous processing is key for throughput, letting the system work on many things at once without blocking batching operations where it makes sense, really squeezing efficiency out of your resources. CPU, memory, network connection pooling helps here too. Less overhead per request. Speaker 1 These are great practical tips. Could you maybe ground this further with specific application types and their typical priorities? Speaker 2 Absolutely low latency champs, high frequency trading. We mentioned microseconds matter. Real time multiplayer games. You need latency under say 50 milliseconds or it feels laggy and unplayable. Speaker 1 Yeah, definitely felt that. Speaker 2 Video conferencing like Zoom or Meet. For a natural conversation you need latency below maybe 100 milliseconds. Any higher and it gets awkward talking over each other. OK. Speaker 1 Those are all about speed. What about high throughput examples? Speaker 2 Think content streaming, Netflix, YouTube. They need to pump out enormous amounts of data gigabits per second to millions of users simultaneously. The initial buffer might have some latency, but the sustained throughput has to be massive. Speaker 1 Right, smooth playback. Speaker 2 Social media feeds, Twitter, Facebook. Handling millions of requests per second for scrolling, posting, liking. Huge throughput needed there. And ecommerce platforms like Amazon, especially during sales, they need to handle thousands of transactions per second reliably. Speaker 1 OK, that paints a clear picture. So pulling this all together, what are the essential best practices when designing systems keeping both latency and throughput in mind? Best practices for designing systems with both latency and throughput in mind First, define your goals. What latency is acceptable? What throughput do you need? You can't optimize that target. Speaker 1 You know what you're aiming for. Speaker 2 Exactly. Then a monitor everything. Track latency and throughput. Critically monitor those percentiles P90P99, set up alerts based on them, use APM tools. Speaker 1 Measure, measure, measure. Speaker 2 You got it design wise. Choose the right architecture for your goals. Use caching smartly. Plan for load balancing. If you expect scale, optimize your database access patterns and test performance. Testing loads stress endurance tests is non negotiable. Find the bottlenecks before your users do. Speaker 1 And keep optimizing. Speaker 2 Yeah, it's an ongoing process. Review code, queries, configurations, performance isn't a one time fix. Speaker 1 Are there common mistakes people make here? Common mistakes Things to definitely avoid. Speaker 2 Oh for sure a big one is ignoring the percentiles. Just looking at averages, that hides real pain. Speaker 1 Right, the P99 problem. Speaker 2 Exactly. Another is premature optimization, trying to make things fast before you even know where the slow parts are. That just adds complexity for no reason. Speaker 1 Wasted effort. Speaker 2 Yep, and finally, poor monitoring, flying blind, not having the right metrics or alerts means you won't know you have a problem until it's a catastrophe. Speaker 1 OK, this has been super helpful. So just to wrap up, what's the key take away? Speaker 2 I think it boils down to this. Latency is about time, throughput is about amount. They're both critical for performance, but they often compete. Understanding that trade off, knowing how to measure both, and making conscious design choices based on your specific applications needs. That's essential for building good systems. There's no single right answer, It always depends on the requirements. Speaker 1 That's a great summary. And maybe one final thought for everyone listening. As we see more and more real time applications, more massive data streams, how might these traditional trade-offs between latency and throughput start to change? Will we need new architectural patterns to cope? Definitely something to ponder. Thanks for joining us for this deep dive.