Episode 20: DynamoDB Deep Dive - Keys, Capacity, DAX, Streams & Global Tables | SAA-C03

Hey everyone, Balu here. Welcome back to Tech Talk with Balu, your complete guide to async the AWS Solutions Architect Associate Exam. Today, we are tackling episode 20 and we are doing a deep dive on DynamoDB. That's AWS's flagship NoSQL database. Back in episode 8, we covered the relational databases, Adius, Aurora, and Elasticash. But DynamoDB only got a brief mention and honestly, it deserves its own episode. DynamoDB questions show up virtually on every exam and it powers a huge portion of modern serverless architectures. Now here's why DynamoDB matters. Remember our serverless episode, Lambda and API Gateway? DynamoDB is the database of choice for those architectures. It's fully managed, scales to millions of requests per second, deliver single digit millisecond performance, and you never patch a server. But it works completely differently from relational database and the exam tests those differences hard. Today, we'll cover DynamoDB fundamentals, partition keys and sort keys, the two capacity modes, DAX for caching, DynamoDB streams, global tables for multi-region, TTL, backups, and the decision framework for when you choose DynamoDB over Adius. We follow the same intractive format as always, pulse checks, crops, spotlights, and memory hooks. And we are keeping the puncture shorter format from the last episode. So let's get started. Onto section one, DynamoDB fundamentals. Let's start with what makes DynamoDB special. DynamoDB is a fully managed nose equal database. Fully managed means AWS handles everything. There is no servers to provision, no operating systems to patch, there is no maintenance windows and so forth. It's highly available by default, with data automatically replicated across multiple availability zones. Notice that you don't configure multi-easy like you do with Adius, it's just all built in. The scale here is staggering. DynamoDB handles millions of requests per second, trillions of rows, hundreds of terabytes of storage. The performance stays fast and consistent, single digit millisecond response times, regardless of the scale. That consistency at scale is DynamoDB's superpower. Now, nose equal means it's not a relational database. There are no joins, no complex SQL queries that you need to do across the tables, no foreign keys to manage and so forth. Instead, you get a flexible schema. Let me explain the structure. DynamoDB is made of tables, and each table has items, think of them as rows. Each item has attributes, think of them as columns, except here's a twist. Attributes can be added over time and they can be null. Different items the same table can have completely different attributes. You don't declare a schema upfront like a relational database. This is called schema flexibility and it lets you evolve your data model rapidly. One hard limit though to memorize is that the maximum size of a single item is 400 kilobytes. That's kilobytes, not megabytes. If you need to store larger data, you store it in S3 and keep a reference in DynamoDB, the same claim check pattern we discussed with SQS. Data types include scalars, string numbers, binary, boolean, null, document types include lists and maps for nested data, and set types include string sets, number sets, and binary sets. DynamoDB also supports transactions which surprised a lot of people when it launched that you can have acid transactions in DynamoDB across multiple items and tables. And there are two table classes. There is the standard type and an infrequent access for tables where storage cost matters more than the access cost. On for memory hook here, you want to think of DynamoDB as a massive filing warehouse with infinite drawers. Each drawer opens in milliseconds no matter how many drawers exist. But unlike a relational database, library where everything is cross referenced, each drawer is independent. You get speed and scale by giving up joins. On to section two, primary keys, partition and slot. Now the most important concept in DynamoDB is the primary key. This is heavily tested so let's nail it all down. Every DynamoDB table has a primary key and you must decide it at table creation time. You can't change it later. The primary key uniquely identifies each item and determines how data is distributed and accessed. There are two types of primary keys. Option one, a simple primary key. This is just a partition key also called a hash key. The partition key determines which physical partition your data lives on. DynamoDB hashes the partition key value and uses that to distribute data across its internal storage nodes. For example, a user's table with user_id as the partition key. Each user id hashes to a location and looks up by user id are lightning fast. Option two, we have a composite primary key. This is a partition key plus a sort key also called a range key. The partition key still determines whether data lives but now multiple items can share the same partition key and the sort key orders them within that partition. Let me give you the classic example. A gaming table where the partition key is user_id and the sort key is game_id. One user can have many game records. All games for a user live together in the same partition that sorted by game_id. You can efficiently query give me all games for user x or give me games for user x where game_id is greater than say y. Here's where the partition key choice matters so much. DynamoDB distributes your data and traffic based on the partition key. If you choose a partition key with few distinct values with only active and inactive, all your data crams into two partitions and you get what's called a hot partition. Performance suffers here. You want partition keys with high cardinality, many distinct values like user_id, order_id, device_id and so forth. This spreads data and traffic evenly. On for a quick pulse check here, there's a quick scenario. You're designing a DynamoDB table for an IoT application. 10,000 sensors each send a reading every minute. You need to query readings by sensor and time range. What's your primary key design? Let's pause and think. The answer is to use a composite key. Partition key is sensor_id, sort key is timestamp. Sensor_id has high cardinality. That's 10,000 distinct values, so data spreads evenly. The timestamp sort keys let you query time ranges within each sensor. So you can say give me readings for sensor 42 between let's say two PM and three PM. This is the classic IoT pattern and appears on the exam regularly. On for a cloud spotlight, watch for questions about poor partition key choices. If the question describes throttling or hot partitions despite provisioned capacity being adequate, the problem is usually a low cardinality partition key concentrating traffic. The fix is to redesign the keys for better distribution, not just adding more capacity. For a memory hook here, the partition key is the file cabinet drawer. It determines where data lives. The sort key is the ordering of files within the drawer. Choose a drawer system where files spread across many drawers, not just one overstuff drawer. On to section 3 now, capacity modes. Next up, how you pay for dynamic DB throughput. There are two capacity modes and choosing between them is a classic exam scenario. Provision mode is the default. You specify the number of reads and writes per second your table needs. You plan capacity beforehand. You pay for what you provision, whether you use it or not. Capacity is measured in what we call RCUs, read capacity units and WCUs that's write capacity units. The good news is that you can add auto scaling to provision mode. You set a target utilization and dynamic DB adjust your provisioned capacity up and down within limits you define. This handles gradual traffic changes nicely. On demand mode is the alternative here, that reads and writes automatically scale with your workload. Zero capacity planning, you pay per request, there's no wasted provisioned capacity. What's a catcher? It's more expensive per request, roughly 2.5 times the cost of fully utilized provisioned capacity. So when do you use which? Use provisioned mode when your traffic is predictable and steady. You can forecast capacity, cost optimization matters and you will actually use what you provision. This is the most production workloads with known pattern. You want to use on demand mode on the other hand when your workload is unpredictable, that you have steep or sudden spikes that auto scaling can't react to fast enough. You have brand new application let's say with unknown traffic, all your traffic is spikey and infrequent like a report that runs once per day. Paying per request beats paying for idle provisioned capacity. So the exam phrase to watch for is unpredictable workload or sudden spikes, that's on demand. Predictible or steady or cost-optimized think provisioned. One more concept I want to introduce is read consistency. Dynamic DB offers two read types. Eventually consistent reads are the default, they are cheaper and faster but might return slightly stale data if a write just happened. Strongly consistent reads guarantee you you get the latest data but it costs twice as much in RCUs and have slightly higher latency. Most applications are fine with eventual consistency. Banking style scenarios that must read the latest write need strong consistency. On for a pulse check a scenario. A startup is launching a new mobile app next week. They have no idea what traffic will look like, and it could be hundreds of users could be going viral with millions, which capacity mode should they use. The answer here is on demand mode, because the traffic is unknown. There is potential for certain spikes, new application, all the on demand signals. And once the app measures and traffic patterns become predictable, they can switch to provision mode with autoscaling to optimize costs. And yes, you can switch between the two modes once per 24 hours. On for a memory hook, think of provision mode as a monthly gym membership. It's cheaper if you go regularly, but wasted if you don't. On demand, on the other hand, is paper visit? There's more per session, but it's perfect when you don't know your schedule. Right, on to section 4, DAX DynamoDB Accelerator. DynamoDB is already fast. It provides single digit milliseconds latency. But what if that is not fast enough? Enter DAX DynamoDB Accelerator. DAX is a fully managed, highly available in-memory cache builds specifically for DynamoDB. With DAX, cache reads drop from single digit milliseconds to microseconds. That's a thousand times faster. And here's the beautiful part. DAX requires no application logic changes. It's APA compatible with DynamoDB. You point your application at the DAX cluster instead of DynamoDB directly, and caching just happens. Cache hits return in microseconds. Cache misses pass through to DynamoDB and get cached for the next time. The default cache detail is about 5 minutes. DAX runs as a cluster of nodes for high availability. The primary use case is solving read congestion. If you have a hot item being read thousands of times per second, example think of a vital product page or a popular leaderboard. DAX absorbs those reads from cache protecting your table and saving RCU costs. Now the exam loves comparing DAX with elastic cache, so let's be clear about the differences. DAX caches individual DynamoDB objects and query results. It sits transparently in front of DynamoDB. You use it when your caching need is specifically DynamoDB reads with no code changes. Elastic cache in the other hand, redis or memcache is a general purpose cache. You use it when you need to store aggregation results, computed values, session data, or cache data from multiple sources. Elastic cache requires application logic, so your code decides what to cache and when. They can, by the way, work together, DAX for individual object caching and elastic cache for storing aggregated computational results. So for a trap spotlight here, the keyword distinction. If you hear a cache DynamoDB reads without changing application code, then that's DAX. If its store aggregation results or cache computed values or session storage, then that relates to elastic cache. If the question is purely about accelerating DynamoDB reads, DAX wins every time. For a memory hook here, think of DAX as a turbocharger bolted onto DynamoDB. So it's like the same car, same controls, but it's just faster. Elastic cache on the other hand is a separate storage unit. You have to manage all by yourself, but it can hold anything. Now we move on to section 5, DynamoDB streams. Now let's talk about DynamoDB streams, one of the most powerful features for event driven architectures. DynamoDB streams is an ordered stream of item level modifications in your table. Every create, update and delete gets captured in the stream in order. Think of it as a change log for your table. Now what can you do with it? You can react to changes in real time. The classic example of a new user signing up and the item is created in your user's table. The stream captures it, a lambda function triggers, and then sends him a welcome email. There is no polling, there is no batch jobs here, it's instant reaction to data changes. Other use cases include real time analytics on usage buttons, like inserting into a derivative tables, feeding open search for text search on your DynamoDB data, and implementing cross region replication. So the specs to remember here is DynamoDB streams have 24 hour retention. Records are available for one day, then it's gone. There's a limited number of consumers. You process the streams using lambda triggers or the DynamoDB Kindness's client library adapter. If you need more, you can send changes to Kindness's data streams instead. That's that you get one year retention, a higher number of consumers, and processing with the full Kindness's ecosystem. Kindness's data analytics, data firewalls, glue streaming, ETL, so and so forth. How are the most common pattern by far is DynamoDB streams plus lambda. The stream captures changes and lambda processes them. This combination appears constantly in serverless architectures and exam questions. For a pulse check here, a quick one. If your application needs to send a notification every time in order status changes in your DynamoDB order's table, and the solution must be real time and serverless, what's the architecture you want to use? The answer is to enable DynamoDB streams on the order's table, then configure a lambda function triggered by the stream and have lambda send the notification via SNS. Stream captures the changes, lambda reacts and SNS notifies. It's fully serverless real time and there is no polling involved. For a memory hook here, think of DynamoDB streams as a security camera pointed at your table. Every change gets recorded in order and lambda is the guard watching the feed, ready to act on anything it sees. But the tape only keeps 24 hours and you want to set up archiving to kinases if you need more. Now let's move on to section 6, global tables, TTL and backups. Let's cover three more features, rapid fire style, global tables, TTL and backups. First, global tables. This makes a DynamoDB table accessible with low latency in multiple regions, and here's a key phrase, it's active active replications. That's application can read and write to the table in any region and changes replicate automatically both ways. Remember from our DR episode, active active is a gold standard. That's users in Sydney, write to the Sydney replica, users in Virginia, write to the Virginia replica, both stay in sync with typically sub-second replication. So for a critical exam fact, global tables require DynamoDB streams to be enabled. Streams is the mechanism that captures changes for replication. So if a question asks about prerequisites for global tables, the answer is streams. Use global tables for globally distributed applications, low latency access worldwide and multi-region disaster recovery. Number two, we have TTL, time to live. TTL automatically deletes item after an expiry timestamp. You add an attribute containing an epoch timestamp, tell DynamoDB that's your TTL attribute, and the item gets deleted automatically after that time passes. The use cases are web session handling, where sessions expire after inactivity, keeping only current data to reduce storage cost, and regulatory compliance where data must be deleted after a retention period. The deletion is free, it doesn't consume right capacity. Number three, backups. DynamoDB has two backup options, there's point in time recovery or called PITR, that's continuous backups for the last 35 days. You can restore to any second within that window, it's optionally enabled, and the recovery process creates a new table, it doesn't override the existing one. Then there is on demand backups, that's full backups for the long retention time period, it is kept until you explicitly delete them. There is no performance impact, this can be managed through AWS backup, which enables cross-region copies for DR. There's also the S3 integration worth noting, you can export a table to S3, it requires PITR enabled, and analyze it with Athena without touching your table's read capacity. And then you can import from S3, that CSV or JSON, which creates a new table without consuming right capacity. For a trap spotlight here, two facts get tested here, number one, global tables require streams enabled, number two, PITR covers a maximum of 35 days, and restores always create a new table. If a question mentions restoring to an existing table in place, that is wrong, DynamoDB restores always create new tables. From a memory hook, think of global tables as identical twins living in different countries, instantly sharing everything both ways, think of DTL as a self-destruct timer on each item, and think of PITR as a 35-day rewind button that always plays back onto a fresh table. Now let's move on to section number seven, DynamoDB versus the RDS decision. Now the big architectural decision, when do you choose DynamoDB versus RDS, this framing appears all over the exam. So you want to choose DynamoDB when you need massive scale with consistent performance, that's millions of requests per second. Your access patterns are simple and known upfront, you get item by key query by partition, your schema is flexible or is evolving, you're building serverless architectures with Lambda, you need single digit millisecond latency at any scale, and you want zero database administration. You choose adias or Aurora when you need complex queries, joins across tables or ad hoc analytics. Your data is highly relational with foreign keys and constraints, you have existing SQL based application, or when your team thinks always in SQL, or you need complex transactions across many entities. Here's the mental model, DynamoDB requires you to know your access patterns before designing the table. You design the table around the queries, relational database on the other hand lets you design around the data and figure out queries later. That's the fundamental trade-off. DynamoDB gives up query flexibility for unlimited scale. Also I want to mention the classic serverless stack that appears constantly on the exam. API Gateway, Lambda and DynamoDB. This client sends rest requests to API Gateway, Lambda handles the logic, DynamoDB finally stores the data. There's no servers anywhere here. This trio is the backbone of serverless on AWS. On for a pulse check, a scenario, an e-commerce company needs a database for their product catalog. The requirements are, it should provide millions of reads per second during sales events, single lookups by product ID, single digit millisecond latency, and no database administration. But they want to use DynamoDB or adios, let's pause and think. The answer is of course, DynamoDB, every signal points there, massive read scale, simple key-based access pattern, latency requirement and the zero administration requirements. If the requirements had mentioned complex reporting queries or joins across inventory, orders or supplies, adios would enter the conversation. Simple access at scales equals to DynamoDB. For a memory hook, think of DynamoDB as a "wending" mission. You insert the key, get item, instant works for millions of people but only does exactly what it's built for. Think of adios as a restaurant kitchen. It's flexible can make anything but it's slower and needs staff. Onto section number eight, exam traps and gotchas. Let's now run through common DynamoDB traps on the exam. Trap number one, partition key design. Trap partitions come from low cardinalty partition keys. The fix is high cardinalty keys, not more capacity. Watch for throttling questions where capacity seems adequate. Trap number two, the 400kb item limit. Items max out at 400kb, for larger objects store industry with the reference in DynamoDB. Trap number three, provisioned versus on demand. Unpredictable or sudden spikes mean on demand, predictable and cost-conscious means provisioned with auto scaling. Trap number four, DAX versus Elasticash. Accelerating DynamoDB reads with zero code changes? Think DAX, storing aggregations or general caching, think Elasticash. Trap number five, global tables prerequisites. Global tables require DynamoDB streams enabled. This exact fact gets tested. Trap number six, streams retention. DynamoDB streams keep records for 24 hours only. If you need it longer, you want to use kindness's data stream for at least an yield retention. Trap number seven, PITR window. Point in time recovery covers a maximum of 35 days and every restore creates a new table. Trap number eight, strongly consistent reads cost double. They consume twice the RCUs of eventually consistent reads. Default is eventual consistency. Trap number nine, TTL deletions are free. They don't consume right capacities great for session data and compliance driven cleanup. And finally Trap number 10, the serverless trio. API gateway plus lambda plus DynamoDB is the serverless pattern. If the question describes a fully serverless rest API with the database, then this is the answer. Onto our final section, number nine, exam quick tips. This is the rapid fire summary time. Let's consolidate what we have covered so far in this podcast. DynamoDB is fully managed, no sequel. It's multi-easy by default. It can handle millions of requests per second. It provides single digit millisecond latency, but it can handle 400kb max item size. Primary keys, they are simple, they are partition key only or composite, which is partition plus sort key. Remember that high cardinality partition keys prevent hot partition. You want to design tables around access patterns. In capacity modes, there are provision that's plan ahead, it's cheaper, auto scaling is available, versus on demand, there is no planning, it's paper request, and it's for unpredictable workloads. DAX is an in-memory cache, provides microsecond reads, there is zero core changes. Elastic cache is used for aggregations and general caching on the other hand. Streams are used for order change log, provides 24 hour retention, it triggers lambda, use kinases data streams for one year retention. Total tables are active active multi-region and require streams enabled. TTL, auto deletes expired items, it's free and it's great for session management. Backups, use PITR for 35 days continuous and on demand for long term. Restore creates new tables, you export to S3 for Athena analysis. So finally the decision framework I want you to take away, if the scenario refers to simple access patterns at massive scale, then think DynamoDB. If it talks about complex queries and joins, then it's obvious. If it's related to serverless architecture, then think DynamoDB with lambda. And if the requirement is global, low latency, think global tables, faster reads related DAX, react to changes, relates to streams plus lambda. Alright, so that is episode 20 on DynamoDB. We started with covering the fundamentals that's fully managed, no sequel, with built-in multi-az, primary key design with partition and sort keys, the single most important DynamoDB skill. Then we went onto two capacity modes and when each wins, DAX for microsecond caching, streams for reacting to changes, global tables for active active multi-region, TTL, backups and S3 integration and the DynamoDB versus RDS decision framework. So the big takeaway is that DynamoDB trades query flexibility for unlimited scale. To design your table around your access patterns, choose high cardinality partition keys and DynamoDB will deliver consistent single digit millisecond performance, no matter how big you grow. It's the database backbone of serverless AWS. We have now completed 20 main episodes, combined with the knowledge of RDS and Aurora from episode 8, you now have the complete AWS database story, relational and no sequel. If this episode helped DynamoDB click for you, please consider to leave a five star rating on my podcast and share it with anyone studying for the AWS exam. It generally helps the channel grow and keep the feedbacks coming on the shorter format. It seems to be working well and your input shape what comes next. So until next time, keep building, keep scaling and I will see you in the next episode. This is Baloo signing off. Bye.