Episode 21: Hybrid Cloud & Migration: Direct Connect, VPN, Snow Family & More | SAA-C03

Hey everyone, Balu here. Welcome back to TechTalk with Balu, your complete guide to asing the AWS solutions architect associate exam. Today we're tackling episode 21, hybrid, cloud and migration. This is where AWS meets the real world of enterprises. Very few companies flip a switch and move everything to cloud overnight. Instead, they run part of their infrastructure on premise and part in AWS. They connect the two together and they migrate gradually over time. The exam tests these hybrid scenarios heavily. So this episode really matters. Let me pay in the landscape for you first. A company has a data center full of server, storage and databases. They want to start using AWS, but they can't abandon their existing infrastructure immediately. Maybe they have compliance requirements, maybe they have long migration timelines, or maybe their IT strategy keeps some workloads on premise permanently. Whatever the reason, they need two things. First, they need connectivity, which is reliable way to link their data center to AWS. Second, they need migration tools, which are ways to move data and workloads into the cloud. Sometimes terabytes or even petabytes at a time. Today we'll cover the connectivity options, which are site to site VPN and direct connect along with transit gateway for connection at scale. Then we'll move on to the data transfer and storage services, which include storage gateway, data sync and snow family for physically shipping data. And finally, we'll cover the migration services, which are DMS for databases and outposts for running AWS hardware inside your own data center. We are keeping the same interactive format with pulse checks, traps, portlights and memory hooks. And we are staying with the shorter puncture length. So let's get started. Let's start with the most common way to connect the data center to AWS, which is called site to site VPN. A site to site VPN creates an encrypted tunnel between your own premise network and your AWS VPC and that tunnel runs over the public internet. Your traffic is encrypted the whole way here. So even though it travels across the public internet, it stays secure. There are two components you need to know for the exam. On the AWS site, you have the virtual private gateway, which is often abbreviated as VGW. This is the VPN concentrated that lives on the AWS site of the connection. You create it and then attach it to the VPC you want to connect it to. You can also customize it's autonomous system number or ASN for routing purposes. On your site, you have the customer gateway, abbreviated as CGW. This is either a software application or a physical device in your data center that represents your end of the VPN connection. You configure it with the details of your AWS VPN. And once both gateways are set up and the site to site VPN connection links them together, your own premise servers can talk to your AWS resources privately and securely. The big advantage of a VPN are that it's quick to set up and relatively cheap. You can have a VPN running in a matter of minutes because it uses your existing internet connection and there is no dedicated hardware or special circuits to provision here. The downsides come from the fact that it travels over the public internet. Because of that, you're subject to internet conditions. Your bandwidth varies, your latency is in guaranteed and your performance can fluctuate depending on what's happening on the internet at any given moment. For many workloads though, that's perfectly fine. But for consistent high throughput low latency needs, it becomes a real limitation. One more detail worth remembering is that VPN connection actually consists of two tunnels for redundancy and a single VPN connection to virtual private gateway gives you about one and a quarter gigabits per second of bandwidth. And here's a memory hook to lock all of it in. Think of a site to site VPN as an encrypted tunnel running through public roads. You're sharing the highway with everyone else's traffic, but your cargo is locked inside an armored tunnel. It's quick to build and it's secure, but it's still subject to traffic conditions. Now, let's talk about direct connect. Sometimes just called DX, which is the premium connectivity option. Direct connect is a dedicated private physical connection between your data center and AWS. Those three words really matter. So, let me say them again. It's dedicated, it's private and it's physical. Unlike a VPN which travels over the public internet, direct connect gives you your own private line running straight into AWS and your traffic never touches the public internet at all. Now, why does this all matter? There are three big reasons. The first is consistent performance. Because it's a dedicated connection, you get both predictable bandwidth and predictable latency with no internet congestion getting in the way of your traffic. The second is higher bandwidth. Direct connect supports connections ranging from one gigabit all the way up to 100 gigabits per second, which is far more than a single VPN can offer. The third is reduced data transfer costs. When you're moving data at high volumes, transferring it over direct connection is often cheaper than transferring it over the internet. Of course, there are trade-offs here. Direct connect takes time to provision, typically anywhere from few weeks to a month. Because a physical connection has to be established at an AWS direct connect location, it's also more expensive with both setup costs and ongoing port hour charges. And here's something a lot of people miss. Direct connect is not encrypted by default. It's private, but private is not the same as encrypted. So if you're carrying out sensitive data, you can run a VPN over your direct connection to get both privacy and encryption at the same time. There is also a critical exam pattern around resilience. A single direct connect connection is not highly available on its own. If it fails, you lose your connectivity entirely. There are two common ways to design around this. The first is to use two direct connect connections for full redundancy. The second is to use a direct connect connection as your primary link and a site to site VPN as your backup. The VPN is cheaper and simply kicks in if the direct connect connection fails. The VPN as backup pattern is very cost effective and shows up on the exam regularly. Let's do a pulse check here. Here's a scenario. A financial services company needs to connect their data center to AWS. They require a consistent high throughput performance with predictable low latency for a data intensive application and internet based connectivity has been unreliable for them. What should they use? Take a moment and think about it. The answer is AWS direct connect of course. The keywords all point in that direction. They mentioned consistent performance, predictable low latency and high throughput and they told you that internet based connectivity has been unreliable. All that points to direct connect dedicated private connection. A VPN travels over the internet so it wouldn't solve their consistency problem here. Here's a trap to watch out for. Pay close attention to the keyword distinctions. When you see words like encrypted, quick to setup, cost effective or temporary, the question is leading towards VPN. When you see words like dedicated, private, consistent performance, predictable latency or high bandwidth, it's generally leaning towards direct connect. If a company needs connectivity quickly but ultimately wants direct connect, the answer is often to use a VPN right now as an interim solution and move to direct connect once it's been provisioned. For your memory hook, think of direct connect as your own private highway built straight to AWS. There is no traffic, no congestion and your speed is guaranteed. But it takes weeks to build that road and it costs more to use. A VPN by contrast is the public road you can start driving on today. As companies grow, they end up with many VPCs and multiple on-premise connections. Trying to connect them all individually becomes a nightmare very quickly and that's exactly the problem transit gateway solves. Transit gateway is a network hub that connects thousands of VPCs and on-premise networks through a single gateway. It uses a hub and spoke model which you will also hear called a star topology. Instead of connecting every VPC to every other VPC individually which explodes in complexity as you add more, everything connects to the transit gateway and the transit gateway handles the routing between them. Let me highlight the key features. First, it enables transitive peering. Normally VPC peering is not transitive which means that if VPCA appears with VPCB and VPCB appears with VPCC, then A still can't talk to C. The transit gateway solves this problem because everything connected to it can route to everything else and you control that routing with route tables. Second, it's a regional resource but it can work across regions since you can peer transit gateways in different regions together. Third, you can share it across accounts using resource access manager or call RAM which makes it great for large organizations that have many AWS accounts. Fourth, route tables let you control which VPCs are allowed to talk to which other VPCs, so you get segmentation and security built in. Fifth, it works with both direct connect gateway and VPN connections which means it unifies your cloud connectivity and your hybrid connectivity in all one place. And finally, it has one unique feature worth remembering which is that it supports IP multi-cast and no other AWS service desktop. There's also a neat bandwidth feature. When you connect a site to site VPN to a transit gateway, you can use equal cost multi-path route. routing, abbreviated as ECMP, to combine multiple VPN connections for more total bandwidth. Each VPN connection adds about two and a half gigabits per second with ECMP, and you can scale up your VPN throughput simply by adding more connections. For your memory hook, picture transit gateway asks the central train station. Instead of building separate tracks between every pair of cities, every line runs into one big hop, and that from that hop, you can get routed anywhere you need to go. You have one connection point, and everything becomes reachable. Now let's shift over to storage and data, starting with AWS storage gateway. Here's the problem that storage gateway solves. S3 is proprietary AWS object storage, which means your on-premise applications can't natively mount it in the way they mount a regular file system for a local disk. So the question becomes, how do you give on-premise systems access to cloud storage? Storage gateway is the bridge that makes that possible. Storage gateway is a hybrid storage service that connects your on-premise environment to AWS cloud storage. Common use cases include disaster recovery, backup and restore, tiered storage, and giving on-premise applications low latency cashed access to cloud data. There are three types of storage gateway, and the exam expects you to know the differences between all three of them. The first is called S3 file gateway. This presents cloud storage as a file system that your on-premise applications can mount using either NFS or SMB protocols. Behind the scenes, those files are stored as objects in S3. You also get a local cash for low latency access to recently used files. It's great for extending file storage to the cloud and for handling backups, and it can also work with FSx file gateway when you need Windows file shares. The second is volume gateway. This presents cloud backed storage as block storage volumes using IECSI protocol, so it looks and behaves like a local disk. There are two modes here. Cached volumes keeps your primary data in S3 while caching your frequently accessed data locally. Stored volumes do the opposite, keeping your primary data on-premise and asynchronously backing up to S3 as snapshots. So you use cached mode when you want your primary data in the cloud and stored mode when you want your primary data on-premise with the cloud as your backup. The third is called a tape gateway. This one replaces physical tape backup infrastructure. If your company uses tape-based backup software, tape gateway presents a virtual tape library over IECSI. Your backup software thinks it's writing to physical tapes, but the data actually goes to S3 and can archive to glacier. This lets companies keep their existing tape-based backup workflows completely unchanged while moving to actual storage to the cloud. Here's a scenario. A company uses tape-based backup software in their data center and they want to eliminate their physical tape infrastructure while keeping their existing backup workflows unchanged. Which storage gateway type should they use? Let's take a moment and think about it. The answer here is tape gateway. The signals in the question are that they use tape-based backup software. They want to eliminate physical tapes and they want to keep their existing workflows. Tape gateway presents a virtual tape library so the backup software keeps working exactly as before, but the data lands in S3 and glacier instant on physical tapes. So here's the trap to watch out for. Mash the storage gateway type to the protocol and the use case described in the question. If you see file system access using NFS or SMB, that's file gateway. If you see block storage or an ISCSI disk volume, that's definitely volume gateway. If you see anything about replacing physical tape backups, that's tape gateway. The protocol mentioned in the question is usually your biggest clue. For your memory hook, think of storage gateway as a bridge between your data center and AWS storage. While gateway carries files across that bridge, volume gateway carries disk blocks and tape gateway carries virtual tapes. It's three lanes on the same bridge, each one designed for a different kind of cargo. Now let's cover moving large amounts of data and there are two very different approaches to this. DataSync handles online transfer over the network, while the snow family handles offline transfer by physically shipping devices. Let's start with AWS data sync. Then moves large amounts of data to and from AWS over the network. When you're moving data from on premise to AWS, it works with NFS, SMB, HDFS, and the S3 API, and it requires an agent installed on your own premise site. For AWS to AWS transfers between different storage services, you don't need an agent at all. DataSync can sync your data to Amazon S3 in any storage class, including glacier, to Amazon EFS, and to Amazon FSx in all of its variants. You can schedule your replication task to run hourly, daily, or weekly. One really important detail is that DataSync preserves file permissions and metadata, including the POSX permissions and the SMB attributes. A simple agent can push up to 10 gigabits per second and you can set a bandwidth limit so that it doesn't saturate your connection and disrupt the rest of your business. The key thing to remember is that DataSync is for online, scheduled, recurring data transfer over the network, in situations where you want your metadata preserved. If you're migrating a file server to the cloud or you need ongoing replication between storage services, then DataSync is your answer. But what happens when you have so much of data that's transferring it on the network that would take forever? That's where SnowFamily comes in, offering offline, physical data transfer. AWS Snowball and specifically Snowball Edge is a highly secure portable device that AWS ships directly to you. You load your data onto it, ship it back to AWS and AWS loads that data into S3 for you. It can handle up to petabytes of data. Here's a rule of thumb that the exam absolutely loves. If transferring your data over the network would take more than a week, you should use Snowball instead. Let me give you a sense of the numbers. Having 100 terabytes over 100 megabit connection takes around 124 days. With Snowball, it's just the time it takes to copy the data locally plus the shipping time, which comes out to a matter of days. So when the internet is simply too slow, you ship a physical device instead. Snowball Edge comes in two flavors. The storage optimized version gives you maximum storage for data migration. The compute optimized version gives you more compute power for edge computing. That brings up the idea of edge computing. Snowball Edge devices can actually run easy to instances and Lambda functions right there on the device. This is designed for locations that have limited internet or no internet at all, like a ship out at sea, a truck on the road, or a mining station deep underground. You process the data right where it's being created, doing things like pre-processing machine learning or media transcoding, and then you ship the device back or send the results where never connectivity becomes available. For truly massive migrations, there is Snow Mobile, which is literally a shipping container on a truck that can hold up to 100 petabytes. That one is for exabyte scale migrations. And here's one exam gotcha to tuck away. Snowball cannot import directly into glacier. You have to import your data to S3 first and then you use an S3 life cycle policy to move that data over to glacier. Here's the scenario. A company needs to migrate 200 terabytes of data to AWS. Their internet connection is 100 megabits per second, and it's shared with the rest of the business. They need the migration done quickly, should they use network transfer or Snowball? The answer of course is Snowball, at 100 megabits per second, 200 terabytes would take months to transfer. On top of that, it's a shared connection so the transfer would cripple the bandwidth that the rest of the business depends on. Since network transfer would take far more than a week, Snowball here is clearly the right answer. You ship the data physically. So for a memory hook, think of data sync as sending your data by high speed internet career, it's fast, it's scheduled and it keeps all the labels intact. Snowball on the other hand is like shipping a truck full of hard drives. When your data is simply too big for the wires, you put it on the wheels instead. Let's cover two more migration focused services. DMS handles databases and outposts let you run AWS inside your own data center. First up is AWS data migration service or DMS for short. DMS migrates databases to AWS while the source database stays fully operational throughout the migration. That's the key benefit because it means minimal downtime. Your production database keeps serving traffic while DMS copies everything over. DMS runs on a replication instance inside your VPC and that instance connects to both the source and the target databases. It performs a full load of your existing data and then it can optionally use change data capture or CDC to keep replicating ongoing changes. That continuous replication means your target database stays in sync with the source right until you're ready to cut over. DMS supports two kind of migration. A homogeneous migration is when the source and target use the same database engine like going from Oracle to Oracle or from MySQL to MySQL. A heterogeneous migration on the other hand is where the engines differ. We first use the AW schema conversion tool, abbreviated as "A". a CT to convert the schema and the core from the source engine over to the target engine. For resilience, DMS supports multi-AZ deployment. When you enable it, DMS maintains a synchronous standby replica in another availability zone, which gives you data redundancy and eliminates the IO freezes that could otherwise happen during migration. Now let's talk about AWS Outposts. This one is for companies that want to run actual AWS infrastructure inside their own data center. Outposts are physical server acts that AWS delivers, installs and manages right there in your facility. They run the same AWS services, the same APIs and the same tools you already used in the cloud, except that the hardware physically sits on premise. Why would you want this? There are a few reasons. You get low latency access to on-premise systems because your workloads run right next to your local systems. You get local data processing, you get data residency, which means keeping your data physically in your own building to satisfy compliance requirements, and it makes migration to the cloud easier over time. It's a fully managed service, although you're responsible for physical security of the racks themselves. The service that run on Outposts include EC2, EBS, S3, EKS, ECS, RDS and EMR. So you get the AWS experience just running on hardware that lives in your own building. Here's a trap to watch out for, though, covering two distinctions. First, remember that DMS keeps the source database running during the migration and that heterogeneous migrations need the schema conversion tool. Second, remember that Outposts is specifically for running AWS services on premise using AWS managed hardware. If a question mentions needing AWS services physically located in your own data center for latency or other data residency reasons, that's pointing you towards Outposts. Before your memory hook, think of DMS as a moving company that keeps your business open while they relocate you. So your customers never even notice that the move is happening. And think of Outposts as AWS building a miniature cloud branch office right inside your own building. Okay, let's pull everything together with the decision framework and the common exam traps now. Let's start with connectivity. If you need quick encrypted cost-effective connections over the internet, that site to site VPN. If you need a dedicated private connection with consistent high bandwidth performance, that's direct connection. If you need both privacy and encryption together, that's a VPN running over direct connect. If you need a backup for your direct connect connection, that's a site to site VPN acting as failure. And if you're connecting many VPCs and on premise networks together, that's transit gateway. Now think about data transfer. If you need ongoing scheduled online transfer with your metadata preserved, that's data sync. If you need a one time huge transfer, that would take more than a week over the network, that's noble. If you're operating at xabyte scale, that's no mobile. And if you need edge computing in a disconnected location, that's no ball edge compute optimized. For storage bridging, when you need to expose cloud storage to your own premise systems, that's storage gateway. File access over NFS or SMB is file gateway. Block storage over ISCSI is volume gateway. Virtual tapes is tape gateway. For databases, when you need to migrate with minimal downtime, that's DMS. And when your source and target engines are different, that is DMS combined with schema conversion tool. And for running AWS on premise, that's our posts. Okay, now let's run through the exam traps. The first trap is VPN versus direct connect. Remember that encrypted quick and cheap means VPN while dedicated, private, consistent, and high bandwidth means direct connect. And don't forget that direct connect is not encrypted by default. The second exam trap is the pattern where a company needs connectivity fast, but wants direct connect. The answer there is to use a VPN now as an interim solution and move to direct connect once it's provisioned because direct connect takes weeks to set up. The third trap is direct connect resilience. A single connection isn't highly available, so you use either dual connections or a direct connect connection paired with a VPN backup. The fourth trap is the one week rule. If network transfer would take more than a week, you use snowball. Do the mental math on bandwidth versus data size to figure this out. The fifth trap is that snowball can't import directly to glacier. You import to S3 first and then use a lifecycle policy to move the data to glacier. The sixth trap is identifying storage gateway types with protocol. NFS or SMB means file gateway. ISCSI volumes mean volume gateway, virtual tapes mean tape gateway. The seventh trap is remembering that data sync preserves metadata and permission and is meant for scheduled online transfer. Don't confuse it with snowballs offline physical transfer. Data trap is the heterogeneous data migration requires the schema conversion tool, while homogenous migration does not. The ninth trap is that transit gateway enables transitive routing and supports IP multicast, and it's the only AWS service that supports multicast. It's your hub for connecting many VPCs together. And finally for the tenth trap is that outputs mean AWS managed hardware physically located in your data center, for when you have latency or data residency needs. Alright, now let's do a rapid fire summary to lock everything in. Site to site VPN is an encrypted tunnel over the internet. It's quick and cheap to set up. The virtual private gateway sits on the AWS site, the customer gateway sits on your site, and you get about one and a quarter gigabits per connection. Direct Connect is a dedicated private physical connection with consistent performance. It ranges from one to 100 gigabits, it takes weeks to provision, and it's not encrypted by default. For resilience, you use either a dual connections or a VPN backup. Transit gateway is your hub and spoke solution for connecting thousands of VPCs and on-premise networks. It provides transitive routing, it can be shared across accounts using RAM, and it uniquely supports IP multicast. Storage gateway is the bridge to cloud storage. File gateway handles NFS and SMB. Volume gateway handles ISCSI block storage in either cached or stored mode, and tape gateway replaces your physical tapes. Data sync handles online schedule data transfer. It preserves your permissions and metadata, and it works both from on-premise to AWS and from AWS to AWS. The snow family handles offline physical transfer. Snowball edge handles up to petabytes and can do edge computing while snow mobile handles exabyte scales. You use them when network transfer would exceed a week, and you always import 2S3 first before moving anything to glacier. CMS handles database migration and minimal downtime, keeping your source operational the whole time using a full load plus CDC. You bring in the schema conversion tool for heterogeneous migrations, and you can enable multi-AZ for redundancy. An outpost finally gives you an AWS managed rocks inside your own data center, running the same AWS services on-premise for low latency and data residency. All righty, and that's episode 21 on hybrid cloud and migration. We have covered the connectivity options, which were site to site VPN for quick encrypted links, direct connect for dedicated private performance and transit gateway for connecting at scale. We covered the storage and data services then, which were storage gateway as the bridge to cloud storage, data sync for online transfers, and snow family for shipping data physically. Then we covered the migration services, which were DMS for databases with minimal downtime, and outposts for running AWS in your own data center. So here's the big takeaway, hybrid architectures are the reality for most enterprises, so your job is to mash the tool to the requirement. Consistent performance means direct connect, quick and encrypted means VPN, too much data for network means snowball, and minimal downtime database migration means DMS. Read the keywords and you will get the answer. We have now completed 21 main episodes, and you've called virtually the entire breadth of the solutions architect exam, including compute, storage, databases, networking, serverless, security, and now hybrid and migration. If this episode helped hybrid concepts click for you, please consider to leave a five-star rating on my podcast, and please do share it with anyone who is studying for the AWS exam. It genuinely helps the channel grow. And thank you for the continued feedback on the shorter format. It's clearly working well, so we'll keep it up. So until next time, keep building, keep connecting, and I will see you in the next episode. This is Baloo signing off, bye.