How to Design a Scalable Metrics, Logging and Alerting Platform?

Defining Functional and Non-Functional Observability System Requirements Welcome to the deep dive. Today we're really getting into the weeds on a huge system design challenge. It's something that's absolutely core to pretty much any big application nowadays. We're talking about designing A massively scalable observability platform. Speaker 2 Exactly. And this isn't just, you know, tailing a few log files on a server. Speaker 1 No, definitely not. Speaker 2 We're architecting the whole system for metrics and logging. Think thousands of micro services all pumping out data. Speaker 1 It's like building the nervous system for a giant tech company, isn't it? It needs to handle this incredible fire hose of data. Speaker 2 That's a great analogy, and you, our listener, need to understand the critical choices here, the trade-offs that mean the difference between a system that works under pressure and one that just falls over. Speaker 1 Right. So our mission today is to break this down piece by piece. We'll look at the requirements first, then the ingestion pipeline, how you actually store this stuff. Speaker 2 Which is trickier than it sounds. Speaker 1 Oh, I bet. And finally, the alerting system. We want to explain the why behind each decision. How do you reliably handle millions of events per second without, you know, creating a monster of technical debt? Hashtag, hashtag I defining the system requirements and scale. OK, let's start unpacking this beast before we even think about specific tech. Kafka this elastic search that we need to ground ourselves. What's the actual scale we're designing for and what does the system need to do functionally and non functionally? Speaker 2 The scale, yeah, that's, that's the absolute dictator here. It drives every single decision. We have to assume realistically a peak ingestion rate of something like maybe 1,000,000 events per second. Speaker 1 1,000,000 per second. Wow. Speaker 2 And just to put that into perspective, if you sustain that, you're looking at potentially over 10 billion log lines or metric data points every single day. Speaker 1 10 billion, OK. Speaker 2 So if your architecture can't swallow that volume reliably, consistently, it's just not fit for purpose. It fails right there. Speaker 1 That kind of volume forces some really tough choices, doesn't it, Around efficiency, compression, everything. Let's start with the functional side. What do the users, typically engineers or maybe product managers, actually need this system to do? Speaker 2 Functionally, I think it boils down to three main pillars. First is collection. The system has to be able to gather logs and metrics from everywhere. Speaker 1 Everywhere meeting. Speaker 2 Meaning Kubernetes pods, old school VMS, serverless functions, databases, network gear, you name it, and they're all potentially spitting out data in different formats and needs to handle that diversity. Speaker 1 OK, so collection is pillar one if the inputs are that varied, you know structured Jason here, plaintext server logs there, maybe some metrics format. Doesn't that make the second pillar, which I'm guessing is accessing the data, really complicated? Speaker 2 Exactly. Pillar 2 is Access. Users need fast, powerful ways to query this mountain of data. For logs, that means searching effectively, maybe by service name or a trace ID that links requests across services or specific error codes. Speaker 1 Stuff that helps you debug quickly. Speaker 2 Precisely. And for metrics, it's about visualization, generating time series graphs, dashboards showing key performance indicators, calculating things like 99th percentile latency for a specific region over the last hour. And it needs to be fast. Speaker 1 Right, nobody wants to wait 5 minutes for a dashboard to load when the site's on fire. Speaker 2 Nobody. Which brings us to the third pillar, Action. This system isn't just a historical archive, it has to drive action. Real time alerting is crucial. Speaker 1 So it's setting up rules. Speaker 2 Setting up rules. Yeah, things like if the rate of 500 errors from the checkout service exceeds X for Y minutes, page the on call engineer or Alert me if disk space on the database cluster drops below 10%. It needs to trigger notifications immediately when thresholds are breached or anomalies are detected. Speaker 1 Collection access action. OK, that makes sense functionally. But hitting those functions at the scale of what was it, 10 billion events a day, that slams us right into the non functional requirements, the NFRS, yeah, what are the absolute must haves in terms of performance and reliability? Speaker 2 OK. NFRS #1 And arguably the most critical for actual usability during an incident is low ingestion latency. Speaker 1 How low are we talking? Speaker 2 The target is usually pretty aggressive. We're aiming for under 100 milliseconds. That's from the moment the event happens on the source server to the moment it's actually queriable in the storage layer. Speaker 1 Under 100 milliseconds. Why is that specific number so important? I mean if a log shows U in say 3 seconds, isn't that still pretty fast? Speaker 2 It sounds fast, but in a real debugging scenario, especially with distributed systems, those few seconds can be an eternity. You might be trying to catch a really transient error, something that flickers and disappears. If you're monitoring data is lagging by seconds, or worse, minutes, you might miss the crucial context. Also, there's a trust factor. If engineers know the monitoring system is slow, they lose confidence. They might start reacting to problems that have already resolved themselves, which wastes time and effort. So sub 100 meters, a goal is really about operational confidence. Got it. Speaker 1 Near real time feedback is key for trust and effectiveness. What about resilience? The system can't go down right? Speaker 2 Absolutely not. So scalability and availability are paramount. Must be horizontally scalable. If traffic double s, you should just be able to add more machines, more nodes, and the system handles it. No vertical scaling bottlenecks. Speaker 1 Add more commodity hardware. Speaker 2 Exactly. And availability needs to be extremely high. This platform is often the source of truth for the health of everything else. So we're talking 99.99%, maybe even higher uptime targets. And link to that is durability. This is non negotiable. You cannot lose data. Speaker 1 Especially not critical error logs. Speaker 2 Precisely. If you lose the log that explains why your entire checkout flow failed, you might never figure it out. So every message, every metric needs to be persistently stored, replicated and recoverable even if hardware fails. Building a Scalable Data Ingestion Pipeline with Kafka OK, let's move it to the architecture. Now we've got this requirement in just a million events per second. Just slamming that fire hose directly into a database, that feels like a terrible idea. It's going to get overwhelmed instantly, especially during spikes. How do we build a reliable front door? A buffer. Speaker 2 Yeah, direct connection is a recipe for disaster. The key architectural pattern here is decoupling. We absolutely need an intermediary layer, a durable log based message broker. Speaker 1 Like Kafka. Speaker 2 Like Apache Kafka or maybe Apache Pulsar, these systems are designed for exactly this. They act as a massive, highly available high throughput buffer. They absorb the spiky, unpredictable ingress traffic and provide a much more orderly, manageable flow of data to the downstream processing and storage layers. Speaker 1 You specifically said log based message broker. Many people might know simpler message queues, maybe Rabbit MQ or something similar. Why the emphasis on a distributed log like Kafka here rather than maybe an in memory queue that could potentially be faster? What's the deciding factor? Speaker 2 It really comes down to durability and resilience, especially at the scale in memory. Queues can be very fast, sure, but they often hold messages well in memory. If that broker crashes, you can lose data unless you've configured persistence very carefully, which adds complexity. Kafka and systems like it are designed from the ground U around a persistent replicated log on disk. Messages are written committed to disk an replicated across multiple brokers before they're acknowledged. This gives you incredibly strong guarantees against data loss even if nodes fail. Speaker 1 And that persistence, that log structure, enables something called replayability, right? Which sounds important for later processing. Speaker 2 It's huge. Replayability is 1 of Kafka's superpowers in this context. Let's say you have a downstream application, a stream processor that's calculating real time metrics. If that processor crashes, which happens, which definitely happens, you can simply restart, connect back to Kafka and say OK, the last message I successfully processed had offset #1,000,200 34,567. Send me everything. After that it picks up right where it left off from the durable log. No data lost. Speaker 1 And you could add new consumers later too. Speaker 2 Exactly maybe six months down the line you realize you need a completely new type of analysis on your historical logs. You can spin up a brand new consumer application, tell it to start reading from the beginning of the relevant Kafka topic or from a specific point in time, and process all that historical data without disrupting any of the live real time systems. That flexibility ability is incredibly valuable for evolving the platform. Speaker 1 OK, so Kafka acts as the durable buffer. Let's trace the data back even further. How does it actually get from the application server the source of the logger metric to Kafka without slowing down the main application itself? That seems critical. Speaker 2 Yeah, you absolutely cannot impact the performance of the core business logic. The standard approach is to use lightweight data collection agents. These run locally on each server or container that's generating data. Speaker 1 Like Fluent or Telegraph? Speaker 2 Exactly. Tools like Fluent, fluent bit, Telegraph, maybe Vector, or even the older Jersey Slog for system logs. These agents are designed to be efficient. Often the application itself writes logs or metrics using a non blocking protocol like UDP, just sending the data to local hosts. Speaker 1 So the app just fires and forgets. Speaker 2 Very quickly. Pretty much. It dumps the data locally almost instantaneously. The agent running on that machine picks it up. The application code doesn't have to wait for network round trips or broker acknowledgments. It's the agent's job to handle the reliable delivery to the central Kafka cluster. Speaker 1 That makes a lot of sense, decoupling even at the source. What does the agent do before it actually sends the data over the network to Kafka to just forward messages 1 by 1? No. Speaker 2 That would be incredibly inefficient. Sending millions of tiny messages individually creates huge network overhead, TCP handshakes, packet headers, all that stuff. So the agent performs 2 crucial optimizations, batching and compression. It buffers messages locally for a very short period, maybe a few 100 milliseconds, or until a certain size is reached. Then it bundles maybe hundreds or thousands of messages into a single larger payload. And then it applies compression G, zip, snappy, LZ 4, something like that to that batch before sending it over the network to the Kafka broker. This dramatically reduces the number of network connections, the bandwidth consumed, and ultimately the cost. It's a critical optimization for throughput and operational expense. Speaker 1 So the compressed batches arrive at the Kafka broker. Now we have to deal with the data format inside those batches. You mentioned earlier we might have structured data like metrics alongside unstructured text logs. Our sources strongly push for using efficient encoding for the structured stuff. Can you talk about things like Protocol Buffers or Avro versus just sending Jason? Speaker 2 Yeah, this is another layer of optimization focused on payload size. When you have structured data and most metrics are highly structured, using formats like Jason is actually quite wasteful. Speaker 1 Wasteful how? It seems easy. Speaker 2 To use. It's easy for humans to read, yes. But think about a typical Jason metric object. You might have a field name like Serverlets and see in milliseconds, that's 30 characters, right? And you send that same string in every single message for that metric, millions, billions of times a day. It's incredibly redundant. Speaker 1 I see so protobuf or avro get around. Speaker 2 That they do. They rely on a predefined schema. You define your message structure once. Say field number one is timestamp, field #2 is server at, field #3 is latent Sims. Then in the actual message you send over the wire, you don't transmit the long string names, you just send the field numbers 123 and their corresponding values. Speaker 1 So it's just numbers and values much smaller. Speaker 2 Way smaller. We're often talking reductions of 50% or more in message size compared to Jason. That translates directly into savings on network bandwidth, Kafka disk usage, and long term storage costs. It's a massive efficiency gain at scale. Speaker 1 But wait, if the message just contains field number 3, value 150, how does the system that reads this data from Kafka know that three actually means latent Sims? Doesn't that create a dependency? Jason seems simpler there. Speaker 2 You've hit on the necessary trade off. Yes, it introduces a dependency on the schema. To manage this, you absolutely need a schema registry. Speaker 1 OK, what's that? Speaker 2 It's a centralized service like Confluence Schema Registry or an open source alternative. It acts as the authoritative source for all your schemas and their versions. When a producer application wants to send protobuff data, it registers the schema first. When a consumer application reads that compact binary data from Kafka, it looks at a schema ID embedded in the message. It then queries the schema registry using that ID to fetch the correct SEMA definition. Then it can correctly deserialize the binary data back into meaningful fields like latent sums. Speaker 1 So we're treating the human readability and self descriptiveness of Jason for significant performance games, but it comes with the operational overhead of managing the central schema registry and handling schema evolution carefully. Speaker 2 That's the scale tradeoff exactly. At a million events per second, the cumulative savings from compact binary encoding massively outweigh the operational cost of the registry. But you're right, for data where the structure is unknown or maybe coming from third party AP is where you don't control the schema. You might still have to use Jason or even XML and just accept the higher overhead and the need for more parsing later. Real-time Data Enrichment and Aggregation with Stream Processors Hashtag tag three real time stream processing and aggregation. Speaker 1 OK, so we have this efficient, reliable firehose of potentially binary encoded data flowing into Kafka. Now what? We can't just dump it straight into storage necessarily. We often need to process it in real time. This is where strain processors come in, right? Like a patchy flank or Spark Streaming. Speaker 2 Precisely this layer is often the brains of the real time part of the operation. These aren't just passive consumers. They are stateful applications that read from Kafka, perform transformations, enrich the data, and compute aggregations on the fly before the data even hits its final storage destination. Speaker 1 What are the main jobs this stream processing layer handles? Speaker 2 I'd say there are three core functions here. First, data enrichment, Second, basic transformations like parsing unstructured logs or filtering out noise. And 3rd crucially for metrics, time based aggregation using windowing. Speaker 1 Let's start with enrichment. We talked about agents sending minimal data to be efficient, maybe just a user RID and an action. How does the stream processor add more context like the user subscription tier or geographic region in real time? Speaker 2 This is a really powerful capability. The stream rocessor, let's use Flink as an example, can maintain its own internal fast access state. This state often holds look U tables or cache data relevant to the incoming stream. Speaker 1 Where did that lookup data come from? It can't query the main production database for every single event, can it? Speaker 2 No, that would kill the database. Instead, we often use Change Data Capture CDC, a separate process, tails the transaction log of the main operational database, like the user database, and streams any changes, new users, updated subscription peers into a dedicated Kafka topic. Speaker 1 Ah, so Flank listens to both the main event stream and this user update stream. Speaker 2 Exactly. Flank consumes that CDC stream and keeps its local state cache like a map of user into user detail, constantly up to date. So when a log event comes in from the mainstream with just user as one ton 3, Flint can do an extremely fast look up against its local state, find the user's region in tier, and add those fields to the event before passing it downstream. Speaker 1 So you're effectively doing the join enrichment at ingestion time in the stream, rather than forcing expensive joins at query time when someone's loading a dashboard. Speaker 2 That's nicely it makes the data richer and more immediately useful in the final storage system, and dramatically improves query performance later on. Speaker 1 Okay, enrichment makes sense now. You mentioned flank and spark streaming. What's the key difference in how they process data? Flank is often called true real time while Spark uses mini batches. What does that mean practically? Speaker 2 It's about the processing model and its latency implications. Flank aims for true event at a time processing. As soon as an event arrives from Kafka, Flink tries to process it immediately, aiming for very low sub second latencies. Spark Streaming operates on a micro batch or mini batch model. It collects events from Kafka for a very short configured interval, maybe one second, maybe 500 milliseconds. Then it processes all the events collected in that tiny batch together. Speaker 1 So there's an inherent delay while the batch fills up. Speaker 2 Exactly. Spark streaming will always have slightly higher end to end latency than flank because of that microdatching interval. However, processing things in small batches can sometimes be more efficient in terms of throughput, especially when writing results out to external systems, as you can batch those rights. Speaker 1 So if our NFR for ingestion latency is super strict like that 100 meters target, Flink is probably the better choice. Speaker 2 Generally, yes. Flink's event at a time model is better suited for ultra low latency requirements. If you can tolerate maybe half a second or a second of latency, Spark Streaming's micro batching approach might offer simpler operational characteristics, better write efficiency to the database. Both are viable. It depends on the specific NFRS and trade-offs. Speaker 1 Makes sense. The most common task for metrics though, is aggregating over time. You know requests per minute, average latency over the last hour. This involves windowing. Can you break down the common types of time windows used in stream processing? This trips people up. Speaker 2 Absolutely. Windowing is fundamental for turning raw event streams into meaningful aggregated metrics. There are three main types you'll encounter, tumbling, hopping and sliding windows. Speaker 1 OK, let's start with tumbling. That sounds simplest. Speaker 2 It is a tumbling window. Has a fixed size and the windows are distinct, they don't overlap. Imagine you define a 5 minute tumbling window. The first window covers events from 12.00 00 to 12.04 point 59. It calculates its result, EG average latency at 12.05. The next window starts completely fresh at 12.05 band 00 and covers events until 12.09 point 59. Speaker 1 So each event belongs to exactly 1 window. Speaker 2 Exactly. It's simple, computationally easy, good for basic periodic reporting, like calculating the total errors every minute. Speaker 1 OK, what about hopping windows? How do they differ? Speaker 2 Hopping windows introduce overlap. They have a fixed size like tumbling windows, but they also also have a hop or slide interval which is smaller than the window size. Let's say you have a 10 minute window that hops every 5 minutes. OK. The first window covers 12.00 to 12.10. The second window starts at 12.05 and covers 12.05 to 12.15. The third starts at 12.9 and covers 12.10 to 12.2. See the overlap? Speaker 1 Yeah, so an event that happens at say, 12.08 would be included in both the 1st and the second Windows calculation. Speaker 2 Precisely this overlap stepping nature is useful for smoothing out metrics and seeing trends more clearly as you're recalculating more frequently over a rolling period, but it means the stream processor has to manage more state because events contribute to multiple windows. Speaker 1 Right, more complex state management and the third type, sliding windows. Speaker 2 Sliding windows are conceptually different. They aren't defined by fixed start times, but rather relative to the current time or event. A common example is calculate the average latency over the last five minutes of activity. Speaker 1 So the window slides continuously. Speaker 2 Exactly as each new event arrives, the window slides forward. If the window duration is 5 minutes, when an event arrives at 12.23 point 01, any event in the window with a timestamp before 12 point 88.01 is immediately evicted. The stream processor effectively maintains a rolling buffer of the last N minutes of data. Speaker 1 That sounds like the most real time view, but also the hardest to manage statewise. Speaker 2 It is. It gives the most up to the moment aggregation, but requires constant state updates, adding new events, removing old ones. It's often implemented using efficient data structures like link lists or time ordered queues within the stream processor state. Speaker 1 OK. Choosing Specialized Databases for Logs and Time Series Metrics We've ingested efficiently. We've enriched and aggregated in real time now. The data needs place to live and we established early on. Logs and metrics are different beasts. Tech logs need searching, metrics need fast time based querying and high rate rates. We can't just throw it all into a standard SQL database, right? Why do traditional databases struggle here, especially with the metrics? Speaker 2 Yeah, a standard relational database, especially 1 using traditional B plus tree indexes, really falls down hard with high volume time series data like metrics. The core problem is the right pattern. How so? Time series data is almost always inserted in time stamp order. You're constantly appending new data points at the end of the time range in AB plus tree. Inserting sequential keys forces the tree to constantly rebalance nodes, split pages. It's just a huge amount of overhead and contention, especially on the right most edge of the index. It simply doesn't scale well for the relentless right load of millions of metrics per second. Speaker 1 OK, so standard B plus trees are out for high volume rights. We need specialized stores. Let's tackle text logs first. Billions of lines, potentially terabytes or petabytes. How do we store them so we can actually search them quickly for say an error message or a specific transaction ID? Speaker 2 For the log search use case, the go to solution is a distributed search index. The most common examples are Elasticsearch or it's open source fork open search. Sometimes people use solar too. Speaker 1 And what's the underlying technology that makes them so fast for searching text? Speaker 2 The magic is the inverted index. Instead of storing data row by row like a database, a search index fundamentally works like the index at the back of a book. It scans all the log messages, breaks them down into individual words or tokens. Speaker 1 Like error database connection failed. Speaker 2 Exactly. And for each unique token it builds a list of all the document IDs, the specific log lines that contain that token. So when you search for database connection failed. Speaker 1 It doesn't scan every log line. Speaker 2 No. It goes to its index, finds the lists of documents for database connection and failed, and then quickly computes the intersection of those lists to find the log lines containing all those terms. It allows you to pinpoint relevant logs in milliseconds, even across massive data sets, because you're searching the index, not the raw data. Speaker 1 Clever. OK, that handles logs. Now for the metrics, CPU usage, request counts, latencies. We need something optimized for those high rate time stamp rights and also fast range queries like show me CPU usage between 2:00 PM and 3:00 PM. What's the specialized solution here? Speaker 2 For metrics we need a time series database or TSDB. There are quite a few options out there in Flux DB, time scale DB which builds on PostgreSQL, Open, TSDB. Prometheus has its own built in TSDB. Speaker 1 And what's special about their architecture? How do they solve that B plus tree right problem we talked about? Speaker 2 They use different internal structures, but a common architectural pattern involves partitioning data much more aggressively, especially by time and by source or tag set in flux DB and time scale DB use concepts often referred to as hypertables and chunk tables. Speaker 1 Hypertables and chunk tables, What does that mean? Speaker 2 Think of the Hypertable as the logical table you interact with like CQ metrics, but behind the scenes the TSDB automatically partitions this logical table into many smaller physical tables called chunks. Each chunk typically holds data for a specific time range, for example one day or one week, and potentially for a specific source, EG hosts server 123. Speaker 1 So when a new metric comes in for server 123 today. Speaker 2 It only needs to write to the one specific chunk that covers server 123 and today's date. The index for that single chunk is much, much smaller than a global index for all metrics ever. It's small enough to be highly optimized, probably fit entirely in memory cache. This makes writes incredibly fast because you're always writing to a small localized hot index. Speaker 1 That makes sense for write performance. Does this chunking approach offer other benefits, maybe for managing old data? Speaker 2 Oh absolutely, it's a huge win for data retention and deletion. Think about trying to delete data older than 30 days from a traditional database or even some no SQL stores using log structured merge trees, LSM trees. You often have to mark rows for deletion tombstones and then run expensive background compaction processes to actually reclaim the disk space. Speaker 1 Right compaction can be resource intensive very. Speaker 2 But with the chunk table architecture, if your retention policy says drop data older than 30 days, the TSTB can simply identify the entire chunk tables that can only data older than 30 days, like the chunk for server 12331 days ago, and issue a simple instantaneous drop table command for those chunks. Speaker 1 Wow, so deleting terabytes of old data becomes almost free operationally. Speaker 2 Exactly. No expensive seeking, no tombstones, no compaction for deletion. It's incredibly efficient for managing data life cycle and controlling storage costs. Speaker 1 That's a really key architectural insight. OK, before we leave storage, there's one critical trap with TSD BS we need to discuss. Cardinality metrics usually have tags or labels like region, Easiest one Service, Appia, Gateway, Host, Ashway Z. Why is it absolutely vital to control the number of unique values for these tags? What is high cardinality and why is it bad? Speaker 2 This is probably the number one way people accidentally cripple their TSDB performance. Cardinality refers to the number of unique combinations of tag values that exist in your data set. So low cardinality tags might be environment, prod, staging dev or region, UC, Swan, US1. There are only few unique values. OK, high cardinality happens when you use tags that have a huge, potentially unbounded number of unique values. The classic example is using something like userid, sessionid or requested as a metric tag. Speaker 1 Why is that so bad? Speaker 2 Because the TSDB typically creates an index entry or some internal tracking structure for every unique combination of tags it encounters. If you tag your metrics with user ID and you have millions of users, you suddenly create millions or even billions of unique time series the database has to index and manage. Speaker 1 What's the impact of creating millions of series? Speaker 2 It causes an explosion in memory usage for the indexes. The TSDB needs to keep track of all these unique series, often in memory for fast look UPS. High carnality leads directly to memory exhaustion on your TSDB nodes. It also slows down all queries and writes because the database has to sift through this gigantic index space for every operation. Speaker 1 So the rule is absolutely do not use high cardinality identifiers as metric tags. Stick to things that group data into manageable buckets. Speaker 2 Precisely. Use tags for dimensions that have a relatively small bounded number of unique values. Region, availability, zone, service name, environment, maybe instance type. Be ruthless about preventing things like user ID's or request ID's from becoming metric tags. Managing Historical Data with Columnar Storage and Tiering This requires policy, education and sometimes technical controls, hashtag tag V analytical storage and data life cycle management. Speaker 1 Right. So we've got fast real time storage for logs, search index and metrics TSTB. What about the massive amount of historical data, data we need for say business intelligence reporting, long term trend analysis, capacity planning, maybe machine learning model training? This data doesn't need sub second access, but we need to store potentially petabytes cost effectively and query it reasonably well. Sounds like OL app Online analytical processing. Speaker 2 Exactly. We're moving from the real time operational plane to the batch analytical plane. The access patterns are different here. Queries often scan large amounts of data but only select a few columns. For example, calculate the average session duration for all users in Europe over the last year. You're reading the session, Duration, and Region columns for potentially billions of rows. Speaker 1 So row oriented storage like in traditional databases isn't ideal for. Speaker 2 That no, it's inefficient. You'd have to read the entire row, including columns. You don't need just to get the session duration and region. For analytical queries, column oriented storage formats are far superior. Speaker 1 Like parquet? Speaker 2 Apache Parquet is the de facto standard here. ORC is another popular one. These formats physically store all the values for a single column together on disk. So all the session duration values are contiguous, all the region values are contiguous, and so on. Speaker 1 How does that help analytical queries? Speaker 2 2 main ways. First, the query engine only needs to read the data blocks for the columns actually mentioned in the query, session, duration and region. In our example, it can completely ignore the blocks for all other columns, drastically reducing the amount of IO required. Second, data within a single column is often highly similar, which means it compresses extremely well, further reducing storage size and IO. Speaker 1 OK, better compression and only reading needed columns. Our sources also mentioned something called predicate pushdown with parquet. What's that? Sounds like another optimization. Speaker 2 It's a huge optimization, yeah. Parquet files don't just store the column data. They also store metadata within the file itself, often at the level of blocks or row groups. This metadata includes things like the minimum and maximum value for each column within that block. Speaker 1 How does the query engine use that? Speaker 2 Let's say your query is find sessions where region equals Europe. The query engine like Spark, Presto, Trino or Snowflake reads the parquet file metadata first if it sees a particular data block where the metadata says the min region value is Asia and the Max region value is Asia. Speaker 1 Knows Europe can't possibly be in that block. Speaker 2 Exactly. It can completely skip reading that entire block of data from disk or from S3. It pushes down the filter predicate Europe into the storage layer, allowing it to avoid reading massive amounts of irrelevant data. This dramatically speeds up analytical queries. Speaker 1 Very cool. So we process our older data, maybe from Kafka or even the TSDB, transform it into parquet files. Where do we actually store these potentially massive parquet files cost effectively? Speaker 2 This is the classic data lake storage question. The two main contenders are Cloud Object storage like Amazon S3, Google Cloud Storage, Azure BLOB Storage or a distributed file system like HDFS Hadoop Distributed File system. Speaker 1 How are the pros and cons? Why is S3 often preferred now? Speaker 2 S3 and its equivalence has become hugely popular, primarily because of cost and decoupling. Object storage is incredibly cheap per GB, and it lets you scale your storage capacity completely independently from your compute resources. You can store petabytes cheaply in S3 and then spin up a compute cluster like Spark on EMR or data bricks only when you need to run queries, paying for compute only when you use it. Speaker 1 What's the downside of S 3 then? Speaker 2 The main drawback is data locality. The compute cluster has to read the parquet files over the network from S3. While networks are fast, this network transfer always introduces some latency compared to reading from local. Speaker 1 Disk and HDFS, What's the argument there? Speaker 2 HDFS traditionally offers better data locality. In a typical Hadoop or Spark on YARN cluster running HDFS, the compute nodes are the storage nodes. So when a task needs to process a block of data, there's a high chance that data is already on the local disk of the machine running the task. This can significantly speed up job execution times by avoiding network transfer. Speaker 1 The HDFS is generally more expensive, right? Yes. Speaker 2 With HDFS, compute and storage are tightly coupled. You have to provision and pay for a cluster of machines that provide both, and those machines are running and costing money 247 whether you're actively querying or not. It's generally less elastic and more operationally complex to manage than relying on S3, so most modern data lakes lean towards S3 for cost and flexibility, accepting the network latency trade off. Speaker 1 OK, S3 usually is now. Even with cheap storage, keeping raw high granularity data forever gets expensive and might not even be useful. How do we manage the life cycle? You mentioned downsampling earlier. Speaker 2 Yeah, data retention and downsampling are critical for cost management. You define tiers, for example tier one keep raw maybe per second or per minute metrics and full logs in the hot stores, TSDB, Elasticsearch for say 7 days or maybe 14 or 30 depending on operational needs. OK. Speaker 1 The recent high fidelity stuff. Speaker 2 Tier 2. After seven days, aggregate the metrics. Roll up the permanent data into maybe 5 minute or one hour. Averages, percentile sums, counts. Store this downsample data in the analytical store, for example Parquaeon S3 for maybe 90 days or six months. You might do something similar for logs, perhaps extracting key fields and discarding the raw messages. Speaker 1 Lower granularity, longer retention, cheaper storage. Speaker 2 Exactly tier 3. After that period, maybe you aggregate again into daily summaries. Store this very coarse grain data for a year or several years for long term trending Tier 4. Finally, the oldest, least frequently accessed data may be the raw logs older than 30 days or the hourly aggregates older than a year. It gets moved to the cheapest possible archival storage like AWS Glacier Deep Archive. This is purely for compliance or occasional forensic analysis, accepting that retrieval will be slow and might cost extra. Speaker 1 So it's a continuous process of reducing granularity and moving data to colder, cheaper storage tiers over. Speaker 2 Time precisely, it balances accessibility, cost, and compliance requirements. Implementing Intelligent Monitoring and Alerting at Scale Hashtag tech #zix. Monitoring, alerting and critical path. Speaker 1 All right. We've covered the whole data journey, ingestion, processing, real time storage, analytical storage, life cycle management, but the whole point of this observability system is to tell us when things break. So the final piece, monitoring the data and triggering alerts, how does that work at scale? Speaker 2 This is closing the loop, turning data into actionable signals. It starts with defining the alerting rules. These rules are essentially metadata. If this condition is true for this long, do that. Speaker 1 Where do these rules live? Speaker 2 They're typically stored in a reliable database, often a standard relational database like PostgreSQL, because you need consistency and the ability to manage potentially thousands of rules. The rule definition would include things like the metric or log query to run the condition, example lane CP 99500 meters, the time window the condition must persist for for example for 5 minutes, the severity, and crucially where to send the notification, example pager, duty service, Slack channel. Speaker 1 OK, so we have a database full of rules. What actually executes these rules against the metric or log data? Speaker 2 That's the job of the alerting engine. This could be a dedicated component like Prometheus alert manager, Grafana alerting a last alert for Elasticsearch, or a custom built application. It's job is to periodically query the relevant data, store the TSDB for metrics, elastic search for logs based on the definitions in the rules database. Speaker 1 And it checks if the conditions are met. Speaker 2 Exactly. It runs the query, evaluates the result against the threshold and duration, and if a rules condition fires, it proceeds to the notification stage. Speaker 1 Are these rules usually just simple static thresholds like CPU 90% or do we need more sophisticated detection? Speaker 2 Simple static thresholds are the foundation, but they're often not enough, especially in complex systems with natural fluctuations. A sudden spike in traffic might push latency over a static threshold, but it might be perfectly normal behavior for that time of day. Speaker 1 So you get alert fatigue from false positives. Speaker 2 Tons of it. That's why more sophisticated systems incorporate anomaly detection. Instead of just checking against a fixed number, these systems use statistical models or even basic machine learning to establish a baseline of what normal behavior looks like for a given metric at a given time. Speaker 1 Like understanding seasonal patterns. Speaker 2 Exactly. It learns the typical daily or weekly patterns, then it alerts when the metric deviates significantly from that learned baseline, even if it hasn't crossed a hard static threshold. Algorithms like Holt Winters or even simpler ones looking for standard deviation changes can be used. This helps catch subtle problems and reduces noise from predictable variations. Some tools might even use more complex algorithms, maybe referencing things like a star in the context of finding deviations from expected paths or performance, though that specific algorithm is more pathfinding. The principle is detecting unexpected change. Speaker 1 OK, anomaly detection helps. Now, if we have potentially 10s of thousands of rules that need to be evaluated maybe every minute against terabytes of data, how do we make sure the alerting engine itself doesn't become a bottleneck or a single point of failure? How do we scale the rule execution? Speaker 2 That's a critical point. You can't have one massive process trying to run all the rules. Modern approaches often leverage serverless functions or distributed job scheduling. Speaker 1 How does serverless help here? Speaker 2 Rule evaluation is often a very parallelizable task. Each rule check is largely independent, so you can have a scheduler, maybe a simple con job triggering A orchestrator or something like AWS Step Functions or Argo workflows that fans out the execution of individual rules, or small groups of rules to hundreds or thousands of concurrent serverless functions like AWS Lambda or Google Cloud Functions. Speaker 1 So each function runs one real check, queries the data and decides whether to fire. Speaker 2 Exactly. It provides massive parallelism and elastic scaling. You only pay for the compute time when the rules are actually running and you don't have to manage a fleet of dedicated alerting servers. The cloud platform handles the scaling automatically. Any state needed like has this alert already fired recently for alert silencing or grouping can be managed in a fast external cache or database like Redis or Dynamo DB. Speaker 1 OK, distributed scalable rule execution. Finally, an alert condition is met. The system needs to send a notification. How do we make sure that critical notification actually gets delivered reliably? Speaker 2 The notification step needs its own layer of reliability. The alerting engine doesn't usually send emails or Slack messages directly. Instead, it typically sends the alert event with all its context, severity, service name, relevant links, maybe a snapshot graph to a dedicated notification router or dispatcher. Often this is part of the alerting engine itself, like Alert manager and that. Speaker 1 Dispatcher handles the actual delivery. Speaker 2 Yes, it handles things like grouping related alerts together so you don't get 100 identical alerts, silencing alerts that are known issues or in a maintenance window, and routing the alert to the correct destination based on the rules. Maybe. High severity goes to Pager duty in e-mail, Medium goes to Slack, low goes to a ticket system. It also handles retries if a destination like Slack is temporarily unavailable. Using established tools like Pager Duty or OPS Genie for the critical paging part is essential as they are built for reliable escalated delivery. Hashtag tag outtrick. Wow. Speaker 1 OK, we started with this huge challenge, building the observability nervous system for a massive application, and we've walked through the entire architecture layer by layer. It really highlights how you absolutely need specialized components at each stage, doesn't? Speaker 2 It it really does. Speaker 1 You need that durable message broker like Kafka for decoupling and buffering the insane ingestion rate. You need separate specialized storage. A search index like elastic search for logs, A TSDB like influx DB or time scale for metrics because their access patterns are fundamentally different. You need stream processing for real time enrichment and aggregation. You need columnar storage like parquet for cost effective analytics. Speaker 2 The whole design philosophy hinges on decoupling and choosing the right tool for the specific job. You wouldn't use a relational database for high volume time series rights, just like you wouldn't try to do full text search efficiently in a basic key value store. Speaker 1 It's like that saying when all you have is a hammer, everything looks like a nail. You can't build this system with just one type of database. Speaker 2 Exactly. Understanding that these specialized components exist and more importantly why they're specific architectures that inverted index the chunk tables, predicate push down, binary encoding are necessary to achieve scalability. That's the key take away for mastering this kind of system design. Speaker 1 Right. Let's leave our listeners with a final thought to chew on, something to take away and maybe debate. We spent a lot of time today focused on a push model, agents on servers, pushing logs and metrics towards a central broker. But there's another popular model particularly championed by Prometheus for metrics, which is the pull model. The central monitoring system actively scrapes endpoints on the monitored services periodically. So the provocative question for you is, if you were designing the system from scratch for a truly massive complex organization with thousands of micro services, maybe like an eBay or an Amazon, which model do you think ultimately leads to less operational complexity and greater resilience at extreme scale? Is it the push model where every single service or agent needs potentially complex logic for buffering, retries and back pressure if the central district is slow? Or is it the pull model, which might seem simpler operationally because the logic is centralized in the scraper, but also concentrates the risk if the scraper fails or gets overloaded, you lose visibility across many services at once. Which approach scales more gracefully, not just technically, but operationally? Speaker 2 That's a great question. There are strong arguments both ways involving service discovery, network policies, failure modes. Definitely something worth thinking deeply about.