Go back

Episode 13: Messaging & Event Architecture - SQS, SNS & EventBridge Explained | SAA-C03

40m

Episode 13: Messaging & Event Architecture - SQS, SNS & EventBridge Explained | SAA-C03

This episode explores core AWS messaging and event-driven architectures essential for the AWS Solutions Architect Associate Exam. It begins by explaining the need for decoupling to handle traffic spikes and enable independent scaling. Amazon SQS is introduced as a queue service that buffers messages, with standard queues offering high throughput but risking duplicates and out-of-order delivery, while FIFO queues guarantee ordering and exactly once processing at reduced throughput. Amazon SNS enables one-to-many message broadcasting, ideal for notifications and alerting, with support for message filtering to route messages to specific subscribers. A key pattern, the fan-out, uses SNS to deliver a single event to multiple SQS queues, ensuring reliability and scalability, but requires proper access policies. Amazon EventBridge, formerly CloudWatch Events, acts as a central event bus that reacts to AWS service events, schedules actions, and integrates with third-party SaaS applications using event patterns and rules. It supports event archiving, replay, and schema registry for structured, versioned event data. The episode concludes with a clear decision framework: use SQS for queuing and buffering, SNS for notifications and fan-out, and EventBridge for reacting to AWS events or scheduling. Critical exam traps—such as visibility timeout misconfigurations, missing access policies, S3 event limitations, and incorrect throughput assumptions—are highlighted to ensure deep understanding and accurate application of these services.

Transcription

6379 Words, 37314 Characters

English
Hey everyone, Baloo here, welcome back to Tech Talk With Baloo, your complete guide to async the AWS Solutions Architect Associate Exam. We are on episode 13 and today we are tackling something that sits at the heart of modern cloud architecture, its messaging and event driven systems. Specifically we are covering Amazon SQS, Amazon SNS and Amazon Event Bridge. Now why does this matter so much? Let me paint you a picture. Imagine you have built an application and its just one big monolith, everything talks directly to everything else. The friend end calls the backend directly, the backend writes to the database directly. This works fine when traffic is steady. But what happens when you suddenly get a massive spike? Say you normally process 10 video uploads but suddenly a thousand come in at once. Your system gets overwhelmed, requests start failing and transactions get lost. The solution to this problem is what we call decoupling, instead of having your applications stock directly to each other you put a messaging layer in between. This way each part of your system can scale independently and if one part slows down the others don't crash, they just wait patiently. Today we are going to explore the three main services AWS gives us for this SQS which is a Q model where messages wait to be processed, SNS which is a published subscribe model where one message goes to many receivers and finally the event bridge which is an event bus that reacts to things happening across your entire AWS environment. By the end of this episode you will understand not just what these services do but when to use each one of them and how they work together and all the architecture patterns that show up again and again on the exam. So let's dive straight in. Now before we get into specific services let's really understand the problem we are solving here because this is the foundation of everything else that we build on. There are two fundamental patterns for how applications communicate with each other. The first is synchronous communication. This is when one application talks directly to another and waits for a response we covered this earlier. The buying service calls the shipping service directly and it waits until shipping response before it moves on. The second pattern is asynchronous or event based communication. This is when an application sends a message to a Q or a topic and then goes about its business. Another application picks up that message later and processes it. Here's why synchronous communication can be problematic. When everything is directly connected and you get a sudden spike in traffic problems cascade through your entire system. Let me give you that video encoding example again. Normally your application encodes 10 videos at a time. There's no problem but one day a thousand encoding requests arrive simultaneously. If your encoding service is directly connected to whatever is sending those requests it gets absolutely flooded. It can't keep up. It starts failing and worse the failure ripples back to the service that was calling. Now here's the beautiful thing of decoupling. If you put a Q in between those thousand requests then those requests just sit in the Q waiting. Your encoding service pulls them off one at a time or a few at a time processing them at whatever rate it can handle. Nothing crashes here, nothing gets lost. The Q here acts as a buffer absorbing the spike and here's the key inside for the exam. These messaging services can scale independently from your application. The Q can hold a virtually unlimited number of messages while your processing application scales up at its own pace. This is the entire philosophy behind these services. SQS gives you the Q model, SNS gives you the published subscribe model and kinases which we will touch on later gives you the real-time streaming model. Each one decouples your application in a different way and choosing the right one is what the exam tests you on. So let's start with Amazon SQS which stands for symbol Q service. This is the older service in all of AWS and it's been around for over a decade and it's the foundation of Q based coupling. So what exactly is a Q? Think of it like a line at a coffee shop, people join at the back of the line and the barista serves them from the front one at a time. In SQS terms, we have producers who send messages into the Q and we have consumers who pull the Q to receive and process those messages. The interesting thing is the producers and the consumers don't know about each other at all. They only know about the Q. That right there is decoupling. Let me walk you through how messages flow through SQS. A producer creates a message and sends it to the Q using an API called send message. That message then sits in the Q, persisted safely until a consumer comes along to process it. The consumer pulls the Q, receives the message, does what to a work that needs to be done. Maybe it inserts the data into a database and then critically it deletes the message from the Q using the delete message API. That delusion is important. The message stays in the Q until a consumer explicitly says, "I have processed this, you can remove it now." Now let's talk about the attributes of a standard SQSQ because these numbers come up on the exam. Standard Qs offer unlimited throughput, meaning you can send as many messages per second you want. They can hold an unlimited number of messages waiting in the Q. The default message retention is 4 days, but you can configure that anywhere up to a maximum of 14 days. After that unprocessed messages get deleted automatically. Latency is very low, under 10 milliseconds on both publishing and receiving. And each individual message can be up to 256 kilobytes in size. But here's where you need to pay attention because standard Qs have two characteristics that trip people up. First, standard Qs provide at least one delivery, which means you can occasionally get duplicate messages. The same message might be delivered twice. Second, standard Qs only make a best effort to keep messages in order, which means messages can sometimes arrive out of order. So if your application absolutely needs messages processed exactly once and in perfect order, a standard Q isn't going to cut it, and we'll talk about the solution to that shortly. When it comes to consuming messages, your consumers can run anywhere on easy to instances on your own servers or as lambda functions. They pull sqs and can receive up to 10 messages at a time. They process those messages and then they delete them. And a powerful feature here is that you can have multiple consumers processing messages in parallel. If you've got a lot of messages piling up, you just add more consumers and they all pull from the same Q simultaneously. This is how you scale your processing horizontally. Now for the exam, remember these key sqs facts. It's got unlimited throughput and unlimited messages. The default retention is four days. It's a maximum of 14 days that you can configure. Mesh size can be supported up to 256 kilobytes and it supports at least one delivery, so there are possible duplicates. It does best effort in ordering, so possibly you might receive out of order messages. Now that you understand the basics, let's dig into the sqs features that the exam loves to test. These are the details that separate someone who knows sqs exists from someone who really understands it. Let's start with the most important one, the visibility timeout. This is a concept you absolutely must understand. Here's the scenario. A consumer pulls the Q and receives a message. The moment that happens, sqs makes that message invisible to all the other consumers. Why? Because we don't want two different consumers grabbing the same message and processing it twice. So sqs hides it temporarily. By default, this visibility timeout is about 30 seconds. That means the consumer has 30 seconds to process the message and delete it. Now here's what gets interesting. If the consumer processes the message and deletes it within those 30 seconds, that's great. Everything works perfectly. But what if the consumer is too slow or it crashes? When the visibility timeout expires and the message hasn't been deleted, sqs makes it visible again and another consumer can pick it up. This is good for reliability. It means a crashed consumer won't cause a message to be lost. But it creates a trade off you need to understand. If your visibility timeout is set too low, say just a few seconds and your processing actually takes longer than that, then the message becomes visible again. And while you're still working on it, a second consumer can grab it. Now you're processing the same message twice. On the other hand, if you do the opposite, if you set the visibility timeout very high like several hours and the consumer crashes right after receiving a message, that message stays invisible for hours before anyone can retry it. So you need to tune this carefully. And there's an API called Change Message Visibility that lets a consumer request more time if it needs it. Next, let's talk about long polling versus short polling. When a consumer asks the queue for messages but the queue is empty, it has to option. With short polling, it gets an immediate response saying nothing here and it has to keep asking over and over again. Some sort of polling mechanism. This generates a lot of unnecessary apicals as you can imagine. With long polling, the consumer says I'll wait up to 20 seconds, let's say for a message to arrive. If a message shows up during that wait, the consumer gets it immediately. If not, it returns empty after the wait. Long polling is almost always preferable because it dramatically reduces the number of apicals, which reduces cost thereby and it reduces latency because messages get picked up the instant they arrive. You can set the wait time anywhere for 1 to 20 seconds and 20 seconds is generally the recommended setting here. Now, let's address that ordering and duplication problem I mentioned earlier. The solution is the FIFO Q, FIFO stands for first in first out and it guarantees that messages are processed in the exact order they were sent. FIFO Qs also provide exactly once processing by removing duplicates using something called a D duplication ID. So if your application sends the same message twice by accident, the FIFO Q recognizes the duplicate and only keeps one copy. The trade-off is that, well, this is the exam point, is the throughput. FIFO Qs are limited to 300 messages per second without batching or 3000 messages per second if you batch them. So you sacrifice some throughput in exchange for guaranteed ordering and exactly once processing. There's also a mandatory parameter here called the message group ID which controls how messages are grouped for ordering. Let me give you one more powerful pattern, FIFO Qs as a buffer for database rights. Imagine your application receives a flood of transactions that all need to be returned to a database. If you write them directly and the load is too high, some transactions get lost because the database can't keep up. The solution is to put an FIFO Q in front of the database. The front of the application sees all the transactions into the queue and then a separate processing tier pulls them out at a control rate and inserts them into the database. The queue absorbs the spike and nothing gets lost here. This is called using SQS as a buffer and it's a classic exam scenario. And finally one important pattern I wanted to remember combining SQS with auto-scaling groups. SQS publishes a cloud watch metric called approximate number of messages which tells you how many messages are sitting in the queue at that point in time. You can create a cloud watch alarm on this metric and when the queue length breaches a threshold, the alarm triggers your auto-scaling group to add more easy-to-instances. More instances mean more consumers which means the queue gets trained faster. When the queue empties out, you scale back down. So this is how you build a system that automatically scales its processing power based on how much work is waiting. Now let's quickly cover security for SQS because it follows a pattern you will see across all messaging services. For encryption, SQS gives you three options. In-flight encryption happens automatically using HTTPS API so your messages are encrypted as they travel over the network. Addressed encryption on the other hand uses KMS keys to encrypt the messages while they are sitting in the queue. And if you want full control you can do client-side encryption where your application encrypts the message before sending it and decrypts it after receiving it. In that case, AWS never sees the unencrypted data. For access control, there are two layers. First, IAM policies regulate who can access the SQS API, this controls which users and roles in your account can send or receive messages. Second, and this is the interesting one, SQS has its own access policies that work very much like S3 bucket policies. These SQS access policies are incredibly useful for two scenarios. One is cross-account access where you want to let another AWS account read from or write to your queue. The other, and this is the one that matters most of the patterns we're about to discuss, is to allow other AWS services to write directly into your queue. For example, if you want SNS or S3 to send messages directly into your SQS queue, you need an access policy that grants them permission. Keep this in mind because it becomes essential for the fan-out pattern we are going to cover shortly. Now let's shift to a completely different model with Amazon SNS which stands for Simple Notification Service. Here's the question SNS answers. With SQS, one message goes to one consumer, who pulls it first, gets it. But what if you want to send one message to many receivers at the same time? Say a customer places an order, and you need to notify the shipping service, the fraud detection service, and then send an email, confirmation, all from that one single event. You don't want to send three separate messages here. You want to send one message, and have it to fan out to everyone who cares about it. Now that's exactly what SNS does. SNS uses what's called a publish/subscribe model, often shortened to Pub/Sub. Now here's how it works. You have an event producer that sends a message to just one place, which is called an SNS topic. Think of a topic as a communication channel. Then you have any number of subscribers who have signed up to it to listen to that topic. When a message gets published to that topic, every single subscriber receives a copy of that message. The producer doesn't need to know who the subscribers are or how many there are. It just publishes ones and SNS handles delivering to everyone that's interested. Let me give you the scale numbers because they show up on the exam. A single SNS topic can have up to 12 and a half million subscriptions, and your account can have up to 100,000 topics. So this scales enormously. What can subscribe to an SNS topic? This is important. Subscribers can be sqs, lambda functions, kinases data, firehose, HTTP, HTTPS endpoints, email addresses, and SMS or mobile push notifications for that matter. So a single publish message could simultaneously trigger a lambda function, drop a message into an sqs queue, send an email, and fire off a text message. That's the power of pubs of model right there. SMS also integrates directly with a huge number of AWS services that can send data into it. CloudWatch alarms publish to SNS when they trigger, S3 buckets can send even notifications to SNS, autoscaling groups, send notifications, cloud formations, and state change notifications. AWS budgets are these events, DynamoDB, all of these can publish to SNS. So SNS often acts here as a central notification hub for your entire AWS environment. In terms of how you publish to SNS, there are two main approaches. The standard approach is topic publish where you create a topic, create your subscriptions. The other approach is direct publish, which is specifically used for mobile apps. With direct publish, you create a platform application and platform endpoints, and you publish directly to those endpoints. This works with Google's messaging services, Apple's push notification services, and Amazon's device messaging. Security for SMS works exactly like sqs, by the way, which makes it easy to remember. You get in-flight encryption with HTTPS, address encryption with KMS keys, and optional client-side encryption. For access control, you have IAM policies regulating API access and SNS policies that work like S3 bucket policies, useful for cross-account access and for letting other services like S3 write into your topics. SNS also has a fee for worship, just like sqs, SNS fee for topics provide ordering by message group ID, and deduplication using a deduplication ID. They can have both sqs standard and sqs fee for cues as subscribers. And just like sqs fee for, they have limited throughput compared to standard topics. So if you need ordering and deduplication in your pop subsystem, SNS fee for is the answer here. For the exam, remember the core SNS concept. One producer publishes to one topic, many subscribers each receive every message. There is up to 12.5 million subscriptions possible per topic, subscribers include sqs, lambda, email, sms, http, endpoints, and kinesis data fire hose. Now we get on to one of the most important architecture patterns on the entire exam, the fan out pattern. This is where SNS and sqs come together, and you will almost certainly see a question about this in the exam. Let me set up the problem first. Suppose you want to send one message and have it processed by multiple sqs cues. Your first instinct might be that you have your applications send the message to each queue individually. Send to queue one, then send to queue two, then send to queue three, and so on and so forth. But this is fragile. What if send to queue two fails after queue one succeeds? Now you have an inconsistent state right there. What if you want to add a fourth queue later, you have to change your application code, and this approach doesn't scale well and it's error prone. The fan out pattern solves this very elegantly. Instead of your application sending to each queue, you push the message just once to an SNS topic. Then you subscribe multiple sqs cues to that topic. When the message arrives at the topic, SNS automatically delivers a copy to every subscribed queue. Your application only does one thing. publishing ones to SNS. SNS handles the fan out to all the queues. This pattern has some really nice properties. It's fully decoupled, and there is no data loss. Further, each SQS queue gives you all the benefits of a queue that's data persistency, the ability to do delayed processing, the automatic retries of fail work, and so and so forth. You can add more SQS subscribers over time without ever touching your producer application. Just subscribe a new queue to the topic, and there you are. And here's the detail that's easy to forget, but shows up on the exam all the time. You must make sure your SQS queue access policy allows SNS to write to it. Remember those access policies I mentioned earlier? This is where they become essential. SNS needs explicit permission to deliver messages into each queue. Now one more powerful feature. The fan out pattern even supports cross region delivery. SNS can deliver two SQS queues that live in completely different AWS regions. So you could publish a message in one region and have it processed by queues in multiple regions around the world. Let me give you a concrete real world example of fan out that the exam loves. S3 events going to multiple queues. Here's a constraint. For a given combination of an event type, like object created and a prefix, like the image folder, S3 allows you to define one even notification rule. So you can't natively send the same S3 event to three different SQS queues. But with the fan out, you solve this very elegantly and beautifully. You first configure S3 to send its event to the SNS topic, then you subscribe as many SQS queues as you want to that topic. Now a single S3 upload event fans out to all your queues and each one can process it independently. One queue might trigger image resizing, another might update a database and another might send a notification, all of that from just that one S3 event. And if you need that fan out combined with ordering and deduplication, you use SNS fee for topics with SQS fee for queues as subscribers. This gives you the fan out distribution plus the guaranteed ordering and exactly once processing logic. That is the premium version of the pattern. There's also a neat variation where SNS sends data to Amazon S3 through kinases data firehose. Since SNS can publish to kinases data firehose and firehose can write to S3 and other destinations, you can build a pipeline where messages publish to SNS end up archived in S3 automatically. Okay, now before we move to the even bridge topic, let me cover one more SNS feature that's both powerful and commonly tested in the exam. It's called message filtering. By default, every subscriber to an SNS topic receives every single message published that topic. But sometimes that's not what you want. Sometimes different subscribers only care about certain types of messages, and this is where message filtering comes in. Message filtering uses a JSON policy that you attach to a subscription. This filter policy determines which messages that particular subscription actually receives. Here's a great example. Imagine you have an SNS topic that receives messages about orders and each message has a state which could be placed, cancelled or declined. You have several SQS queues subscribed to this topic. Now using filtered policies, you can set things up so that one queue only receives messages, where the state is placed, another queue only receives messages, where the state is cancelled, and the third queue only receives declined orders. Meanwhile, you could have one more queue with no filtering policy at all, and that queue can receive everything now. The key rule to remember is that if a subscription doesn't have a filter policy, it receives every message, but if it does have a filter policy, it only receives the message that matches to that policy. This lets you route different messages to different places, all from a single topic without writing any routing logic in your application. The filtering happens inside SNS itself. This is a really elegant way to build sophisticated routing, instead of one application examining every message, and deciding where it should go, you let SNS do the filtering based on the message attributes. For the exam, just remember that SNS message filtering uses JSON filter policies on subscriptions, and that a subscription with no filters gets everything. Now let's move to our third major service, it's called the Amazon Event Bridge, and here's a little piece of history that helps you on the exam. Event Bridge was formerly called as CloudWatchEvents, so if you see CloudWatchEvents mentioned anywhere, just know that's the old name for Event Bridge, they're the same thing, and Event Bridge is the newer, more powerful version. So what does Event Bridge do? At its core, Event Bridge serves two main purposes, the first is scheduling. You can use Event Bridge to run things on a schedule, like a cron job in the cloud. You set up a rule that says every hour or every day at 2am, and the Event Bridge triggers whatever action you have configured. This is exactly the serverless cron job pattern we discussed in the Lambda episode, even Bridge on a schedule triggering a Lambda function. The second purpose, and this is the really powerful one, is reacting to those events. Event Bridge can watch for things that happens across your entire AWS environment, and react to them automatically. You define an event pattern, a rule that says, "When this specific thing happens, do that." For example, when someone signs in as a root user of your account, Event Bridge can detect that event and immediately send an SNS notification to alert you. That's a security use case. Or when an EC2 instance changes state, that is when a code build fails, when an object is uploaded to S3, when any API call is made that's recorded by CloudTrail, even Bridge can react to all of these kind of scenarios. Let me explain how Event Bridge rules work. You have a source, something that generates events. That source could be an EC2 instance changing state, a code build that's failing, an S3 upload, a trusted advisor finding, any API call captured by CloudTrail or a schedule. Event Bridge takes the event from that source, optionally filters it based on the event pattern you defined, and then routes it to a destination. And the list of possible destinations is enormous. You can trigger Lambda functions, AWS batch jobs, ECS tasks and so forth. You can send to SQS, SNS or Kindness's data streams. You can kick off step functions, code pipeline or code build. You can run system manager actions or EC2 actions. Event Bridge is essentially the central nervous system that connects events to actions across all of AWS. The events themselves are JSON objects, when an EC2 instance changes state, for instance, Event Bridge generates a JSON document describing exactly what happened, the event type, the instance details, the timestamp and so on, and your rules examine this JSON and decide what to do. Now let's talk about a concept that's central to Event Bridge. It's called the event bus. An event bus is the pipeline that receives events. There are three types you need to know about. The default event bus is where all standard AWS service events flow. This exists automatically in your account. Then there are partner event buses which receive events from third party SAS providers like ZenDesk or DataDog. So when something happens in one of those external services, it can flow into your AWS environment through a partner event bus. And finally, there are custom event buses which you create for your own applications to send their own custom events. Here's a powerful capability for the exam. Event buses can be accessed by other AWS accounts using resource-based policies. This means you can aggregate events from across your entire organization. Imagine you have dozens of AWS accounts and you want all their important events to flow into one central account for monitoring. You set up resource-based policies on a central event bus and all those accounts send their events to it. That's a common enterprise pattern. Event bridge has two more features worth knowing. First, you can archive events that are sent to an event bus, either all of them or a filtered subset. And you can keep that archive indefinitely or for a set period. And then you can replay those archive events later. Now, why is this useful? Maybe you deployed a bug in your event processing. You fixed it and now you want to replay all those events from the last few hours to reprocess them correctly. Replay makes that possible. The second feature is called the schema registry. Event bridge can analyze the events flowing through your bus and automatically infer their structure, their schema basically. The schema registry stores these schemas and can even generate code for your application. So it knows in advance exactly how the event data is structured. And these schemas can be versioned. So as your events evolve over time, you can track those changes. For the exam, here's how to keep event bridge straight. It does scheduling like cron jobs and it reacts to events using event patterns. It routes events from many sources to many destinations. It uses event buses, default for AWS events, partner for SaaS and custom for your apps. And it offers archive, replay and schema registry capabilities. All right, now that we have covered all the three services, let's tackle the question that really matters for the exam. How do you choose between SQS, SNS and event bridge? bridge. The exam will give you a scenario and expect to pick the right one, so let's build a clear mental model. You want to use SQS when you need a queue where messages wait to be processed by consumers and where typically one consumer processes each message. The key signals in an exam question are decoupling two components, buffering requests to handle spikes and processing work asynchronously where each task is handled once. If you see phrases like process orders one at a time, buffer database writes, decoupling the application tiers or ensure no messages are lost during a traffic spike, you are looking at SQS. And if the question adds must be processed in the exact order or exactly once with no duplicates, then it's specifically referring to SQS FIFA. You want to use SNS when you need to send one message to multiple receivers simultaneously, the published subscribe model. The signals here are notifications finding out to multiple systems or use for alerting. If you see notified multiple services, send the same message to several systems, send email and SMS notifications or alert administrators that points to SNS. And when the question describes sending one event to multiple SQS queues, that's the classic fan out pattern, SNS publishing to multiple SQS subscribers. Use even bridge when you need to react to events happening across AWS services or schedule actions or connect to third party SaaS applications. The signals are reacting to a state change in an AWS service, running something on a schedule or filtering events with complex rules and routing them to many possible targets. If you see trigger an action when an EC2 instance changes state, run a lambda function every hour, react to an API call recorded in cloud trail, integrate events from a SaaS partner or aggregate events across multiple accounts, that's even bridge. Let me now highlight the subtle distinctions that the exam exploits. SQS and SNS are both about your application messages flowing between your components. Even bridge is more about reacting to events from AWS services themselves and from SaaS partners with the rich filtering and a huge range of destinations. SNS and even bridge can look similar because both can send to multiple targets, but SNS is a simpler Pub/Sub broadcast to subscribers, whereas even bridge has sophisticated event pattern matching, the schedule capacity, partner integrations and the archive and replay features. So when a question emphasizes complex even filtering scheduling or reacting to AWS service events lean towards even bridge. When it's a straightforward broadcast of a message to multiple subscribers lean towards SNS. Right, now let's go through the common traps that catch people on these messaging topics. Trapp number one, SQS standard cues can have duplicates and out-of-order messages. If a question requires exactly once processing or strict ordering, the answer is SQS feeful, not standard. Don't get lured into picking standard just because it has hired through it. Trapp number two, the visibility timeout. If a question describes the same message being processed twice, the cost is almost always that the visibility timeout is too short. The message became visible again before processing has finished. The fix is to increase the visibility timeout or call change message visibility to extend it. Trapp number three, SQS message retention. Messages are kept for a default of 4 days and a maximum of 14 days. If a question asks about keeping messages longer than 14 days, SQS alone can't do it. You at need to move them somewhere persistent like S3. Trapp number four, the fan out pattern requires SQS access policies. If you're sending SNS messages to SQS cues and the messages aren't arriving, the trap is that the SQS cues access policy doesn't allow SNS to write to it. The fix is always updating that access policy. Trapp number five, S3 can only have one even notification per event type and prefix combination. If a question wants the same S3 event sent to multiple destinations, you cannot do it with native S3 notifications alone. The answer is fan out, S3 to SNS, then SNS to multiple SQS cues. Trapp number six, SNS by itself doesn't persist messages for later. If a subscriber is offline when a message is published, a plain SNS to endpoint subscription can lose that message. If you need persistence and retries, you subscribe an SQS cue to the SNS topic. The cue holds the message until it's processed. This is another reason the SNS plus SQS fan out pattern is so popular. Trapp number seven, event bridge versus cloud watch events. Remember, they are the same service. If a question uses the old name, cloud watch events, don't get confused. It's simply event bridge. Trapp number eight, choosing SNS when you really need event bridge. If the scenario involves reacting to an AWS service event, like an easy to stay change or a cloud trail API call or involves scheduling, the answer is event bridge. Even though SNS can send notifications too, even bridge is one that natively understands AWS service events and event patterns. Trapp number nine, SQS fee for throughput limits. fee for cues are kept at 300 messages per second without batching or 3000 with batching. Remember that. If a question needs extremely high throughput and mentions millions of messages and ordering isn't strictly required, standard cues with their unlimited throughput are the answer. Trapp number 10, the SQS plus auto scaling pattern. When a question asks how to automatically scale your processing based on workload, the answer involves the approximate number of messages, cloud watch metric, driving a cloud watch alarm that scales an auto scaling group, watch for this exact pattern. Trapp number 11, message size limits. SQS messages max out are 256 kilobytes. If you need to send something larger like a big file, the pattern is then to store the large payload in S3 and send only a reference to it through SQS. This is sometimes called the claim check pattern. And finally for Trapp number 12, even bridge cross account aggregation. If the question asks how to collect events from many AWS accounts into one center place, the answer is even bridge event buses with resource based policies, not building some custom solution. Right, now let me give you the rapid fire summary to lock all of this in. SQS is a cue, one message typically one consumer. Decables and buffers, standard means unlimited throughput but possible duplicates and out of order. FIFA means ordered and exactly once but limited throughput. SNS is published, subscribe, where you can send one message to many subscribers, it's used for notification and fan out. Subscribers include SQS, Lambda, email, SMS and HTTP endpoints. Event bridge is the event bus. It schedules ground jobs and reacts to AWS events with rich filtering. It can connect to SaaS partners and offers archive and reply. Fan out equals to SNS publishing to multiple SQS cues. Remember that SQS access policy must allow SNS. SQS buffer in front of a database protects against losing transactions during spikes. SQS plus auto scaling uses the cue length metric to scale your consumers. You need ordering and exactly once behavior, then use FIFA. You need to broadcast to many, then use SNS. Do you need to react to AWS events or schedule, then go for event bridge. And that's it for episode 13 on messaging and event architecture. Let's recap now what we've covered. We started with why decoupling matters that putting a messaging layer between your applications lets them scale independently and protects you from traffic spikes. Then we dove into SQS, the cue service covering everything from visibility timeouts and long polling to FIFA cues and the buffer pattern. We explored SNS, the publish subscribe service and how one message fans out to many subscribers. We then brought SQS and SNS together in the fan out pattern, which is one of the most tested architectures on the exam. We went on to cover SNS message filtering for routing messages intelligently and we finished with event bridge, the event bus that schedules jobs and reacts to events across your entire AWS environment. The big takeaway is this. These three services solve different problems. SQS is for queuing and buffering work. SNS is for broadcasting one message to many, even bridge is for reacting to events and scheduling. Master went to use each one, understand the fan out pattern deeply and you will handle the messaging questions on the exam with confidence. We have now completed 13 main episodes covering the core of the exam that compute storage, databases, networking, serverless and now messaging. You are building a really comprehensive foundation. If this episode helped you learning messaging and event architecture, then please leave a five star rating on Spotify and share it with anyone studying for the AWS exam. It genuinely helps this channel grow. Until next time, keep building, keep decoupling and I will see you in the next episode, this is Baloo signing off, bye.

Podcast Summary

Key Points:

  1. Decoupling applications using messaging services allows independent scaling and prevents system failures during traffic spikes.
  2. Amazon SQS provides a queue-based model where messages wait to be processed, supporting unlimited throughput and message retention up to 14 days.
  3. SQS standard queues offer at least one delivery and best-effort ordering, leading to possible duplicates and out-of-order messages; FIFO queues provide exactly once processing and strict ordering at the cost of limited throughput.
  4. Amazon SNS enables one-to-many message broadcasting via a publish/subscribe model, allowing notifications to be sent to multiple endpoints like SQS, Lambda, email, or SMS.
  5. The fan-out pattern uses SNS to broadcast a single event to multiple SQS queues, enabling scalable, fault-tolerant, and decoupled processing without modifying application code.
  6. Amazon EventBridge reacts to AWS service events, schedules actions, and integrates with SaaS partners, using event patterns and destinations to automate workflows.
  7. Key exam distinctions include
  8. Common exam traps include ignoring visibility timeouts, missing SQS access policies, assuming S3 can deliver to multiple queues, or overlooking the need for persistence in SNS subscriptions.

Summary:

This episode explores core AWS messaging and event-driven architectures essential for the AWS Solutions Architect Associate Exam. It begins by explaining the need for decoupling to handle traffic spikes and enable independent scaling. Amazon SQS is introduced as a queue service that buffers messages, with standard queues offering high throughput but risking duplicates and out-of-order delivery, while FIFO queues guarantee ordering and exactly once processing at reduced throughput.

Amazon SNS enables one-to-many message broadcasting, ideal for notifications and alerting, with support for message filtering to route messages to specific subscribers. A key pattern, the fan-out, uses SNS to deliver a single event to multiple SQS queues, ensuring reliability and scalability, but requires proper access policies. Amazon EventBridge, formerly CloudWatch Events, acts as a central event bus that reacts to AWS service events, schedules actions, and integrates with third-party SaaS applications using event patterns and rules.

It supports event archiving, replay, and schema registry for structured, versioned event data. The episode concludes with a clear decision framework: use SQS for queuing and buffering, SNS for notifications and fan-out, and EventBridge for reacting to AWS events or scheduling. Critical exam traps—such as visibility timeout misconfigurations, missing access policies, S3 event limitations, and incorrect throughput assumptions—are highlighted to ensure deep understanding and accurate application of these services.

FAQs

Decoupling allows applications to scale independently and prevents system failures during traffic spikes by introducing a messaging layer between components.

Use SQS when you need to buffer requests, process messages one at a time, or decouple application components to handle traffic spikes and ensure message persistence.

Standard SQS offers unlimited throughput but can deliver duplicates and messages out of order. FIFO queues guarantee exactly once processing and message ordering, but have limited throughput (300 or 3000 messages per second).

The fan-out pattern uses SNS to publish a message to multiple SQS queues, enabling one event to trigger processing in several places. It's used when a single event needs to be processed by multiple systems without code changes.

SNS sends one message to all subscribers of a topic, allowing one publish to trigger actions in multiple services like SQS, Lambda, email, or SMS simultaneously.

Event Bridge reacts to AWS service events, schedules actions, and integrates with third-party SaaS tools, acting as a central event bus that routes events to destinations like Lambda or SQS.

Chat with AI

Loading...

Pro features

Go deeper with this episode

Unlock creator-grade tools that turn any transcript into show notes and subtitle files.