#14 AWS IAM Series: Part 2 - AWS Storage Services

Hello and welcome to another episode of the Identity Navigator. My name is Rohit. This is the second part of our AWS IM series in which we are deep diving into cloud computing, making AWS, I make an example, AWS IM, and compare and contrast the cloud concepts with the legacy concepts, giving the listeners a foundation knowledge of all things cloud IM. So first of all, it has been a while since I published my last episode. So I would like to apologize to you all. Life just got in the way and that made me realize that I need to have a backlog of episodes ready to go. Also I received some of your feedback. Thank you so much for your LinkedIn messages and your emails. A couple of them at least stood out. One was a listener said that they actually enjoyed the background music. So now I'm little conflicted and another one said that maybe 35, 45, 35 to 40 minutes of episodes are a little too long and they would like me to cut down to about 20 minutes. So I will try to look into it, see what all of you have to say if you have any preferences, please do let me know and we can absolutely adjust. But back to AWS IM. So this is a multipart series because we did not really just wanted to deep dive into AWS IM because as we spoke about for a very, very experienced IM professional, it's easy to get into the cloud and implement those core concepts within the cloud. For somebody starting new who has always seen cloud, it is easier for them to adapt. I'm talking about Gen Z basically, but for mid level, I am practitioner jumping to cloud I am could be overwhelming right. So in this multipart series, we are deep diving into cloud computing, AWS cloud, AWS I am and comparing and contrasting the cloud concepts with the legacy concepts. So in the last episode, we spoke about cloud computing, major cloud providers and foundational AWS services introduction. So what were the foundational AWS services? They were compute storage, database and network. Now we deep dive into compute and AWS global infrastructure. So we spoke about when we when you think compute, think processing powers in traditional terms, think CPU and RAM and we then deep dive into a few compute services. We also made a point of you should be aware of the concepts, but you don't need to memorize the cloud concepts. We will use the same pattern today as we deep dive into storage and other foundational services. And you will know what I mean in just a second. In the last episode, we also talked about AWS global infrastructure, the ability zones, the regions, the edge locations and regional edge caches. So if you have not listened to that episode, I would highly recommend going and hearing this, but in case you are just interested about storage and databases and networks, that is not a prerequisite. Absolutely. So today, let's deep dive into other fundamental concepts and see how far do we go? Because now at the back of my mind, I have this thing, what is the right length for an episode? So thank you for giving me that doubt. Also funny thing happened with me a few weeks back, you know, for for the listeners who have been with us through this journey, they know about the funny side about me talking about sponsorship opportunities and somebody who was in an official call with me actually mentioned it. And it was embarrassing for me to say the least and it made me aware that not everybody might be getting the jokes. So it is just a joke. I am in not looking for sponsorship. It is just a hobby podcast so that we can all connect have a way to talk about identity and access management. And you know, we'll figure out where it takes us as you all know, my goal is to get to 100 episodes and I am not looking at the list size, how many times the download happens, all those things will take care of itself. That is not something that I would control or I could control and not something that I need to worry about. However, what I can control is publishing 99 episodes and that is what I would do. I would not lie though, getting some of your feedback and your subscriptions really helps me pick up my spirit and makes this effort worthwhile. So thank you so much. All right, let's talk about things that I know about identity and access management. So storage. Imagine AWS storage as a virtual warehouse where you can store all kinds of digital stuff. Just like having a giant locker in the cloud where you can keep your files, your photos, videos and more instead of storing them on your computer or phone. Remember, I do have an open question to you all. When we spoke about foundation, I AWS services, we spoke about both storage and databases. And my question to you all is, how is storage different from databases? You possibly get to it in the next episode. But in case you want to let me know, please feel free to reach out. So AWS storage obviously provides a range of options to safely store and manage your digital assets. Offering flexibility, scalability and reliability. So all those things are those good things that comes with cloud. So you don't have to worry about physical storage space or hardware maintenance. So there would be multiple AWS storage services that you would come across in your day to day work. Your Amazon S3s, your Glacier Cloud Front, Storage Gateway, EFS, EBS, Snowball, Amazon EC2 instance store. So the trick is to have a vague idea of or a good idea of what Amazon storage is. Have a general idea of what are the type of services that exist. Now believe me, you will never be able to keep up to all the new services that I am releases on a very frequent basis. And that is right, it's good to have a vague idea that care could be something out there. And you just need to have those basics of what is storage, what are the primary services that it provides. And then you can obviously look at Amazon docs or Google your way through identifying, which is the best service that satisfies your use case. So do not try to learn these facts. Just try to understand that there is something that is out there that might be suitable for your use case and you would you would cross that bridge once you get to it. So let's deep dive into some of these individual storage services. And that will give you a pattern of how do you identify one that best suits your use case. So let's start with the one that is the most famous one. Amazon simple storage services are affectionately and professionally called as S3. So Amazon S3 is a fully managed object based storage that is highly available, highly durable, very cost effective and widely accessible. So think of Amazon S3 like a giant secure online storage space where you can keep all sort of things. So in the real world, think of it like a storage locker. So imagine you have a huge closet or a storage locker. You can put anything that you want in there. Your photos, your videos, your documents, your backups, your furniture. So this closet is super organized and can grow as big as you need it to be. This is accessible from anywhere. This storage locker isn't in your home. It is in the cloud which means you can access it from anywhere anytime as long as you have an internet connection. Whether you are at home, at work, on vacation, you can get to your stuff. It is obviously safe and secure. It makes sure that your data is protected and backed up. So even if something goes wrong, your information is still secure. The best part is that you only pay for what you use. So you only pay for the space you use in this locker. If you store a little, you pay a little. If you store a lot, you pay more. So it's flexible and cost effective. And you can organize it with label. You can organize your items using labels or tags. These help you find and manage your stuff easily. For example, you might have a section for family photos, another for work documents and another for videos. So what are the key features of Amazon S3? Scalable, durable, accessible, flexible and cost effective. Everything that you associate with a cloud computing or benefits of cloud computing. Now why use Amazon S3? It would be used for backup and recovery. So keep copies of your important data safe. You might have an on-premise data storage. You want to back it up for recovery purposes, ransomware protection. It could be used for that. Content storage, obviously, you can store and retrieve media files, documents, and other content. And data are caving. So archiving old data that you don't need to access frequently, but still want to keep. Think about audit requirements. So key facts. Getting into a little bit of technical space here. So the smallest file size that it supports is zero bytes. And the largest file size is five terabytes. The last time I checked. You can specify the region. The data is then duplicated over multiple availability zones in that region. I hope you know, data center region, availability zones. And to store object in S3, we have to define and create a bucket. And this bucket needs to have a globally unique name. Right. There are multiple storage classes, like standards, standard infrequent access, intelligent tiering, one's own infrequent access, reduced redundancy. So basically, Amazon is doing a great job. It's not overselling you and really is helping you out in terms of S3 of you can identify what you need. And I believe the payment is very, very fair. Now, if you enable worsening, you cannot disable it. So make sure that you identify whether you need worsening or not. And then life cycle rules in AWS provides an automatic method of managing the life of your data while it is being stored on an Amazon S3. So you can define those rules. As we spoke about, when you think of S3, use or think in terms of data backup, static content for websites and large data sets. Amazon S3 is like having a massive secure and accessible online storage locker where you can store any type of data, access it from anywhere and pay only for what you use. Now you might have caught on that I am using storage locker a lot. It is because I want you to create this connection. Even if you forget everything about S3, if you just remember hey, S3 is like an online storage locker, you would be able to identify at the highest level whether this could satisfy your use case or not or whether it is worth your time to deep diving into some of the other features of S3. And let's keep doing it for all the other storage services then. Let's see if we can associate them with a word that would remind you of what it does. So you don't have to remember or learn or memorize actual details of all these storage services because once you work on them, then it would be like a second nature to you. But until then let's do this word association. Now Amazon Glacier think of it like a deep secure storage basement where you keep things you don't need to access frequently but want to keep safe for the long term. So what's the keyword association here? Amazon Glacier imagine storage basement. So imagine you have a basement where you store items you don't need to access often old photo albums, important documents, seasonal decorations or historical records. The basement is defined to keep things safe for a long time. Unlike your main storage closet, Amazon S3 where you can quickly grab things as you use often, the basement is for items you rarely need. When you do need something from the basement, it takes a bit longer to get to it. So you know where we are going at, hey, it takes longer to retrieve but it cheaper to store. So if you can give me some time for me to retrieve data, I can give you a cheaper way to store your data and content. The basement is extremely secure obviously and built to protect your items from damage, loss or theft. Amazon Glacier actually can ensure that your data is stored safely and can last for decades. It is very cost effective because you don't access the storage frequently. It's cost much less to store things in the basement. It offers a very low cost solution for storing large amount of data that you don't need to access regularly. Now obviously because it is cheap, you can organize your basement storage with labels and sections but getting something out takes more time compared to your main closet. This store, this slower retrieval time is fine because you don't need to access these items urgently. That is the use case here. So in terms of key features, we discussed long-term storage, cost effective, highly, highly durable and retrieval time is slower. So why use Amazon Glacier? Perfect for storing historical records, compliance data and backups that don't need to be accessed regularly. It is a cost saving option and then data durability. Let's look in some of the key facts. Obviously cheaper than S3 used to store large data sets which is used, accessed and updated infrequently. The data structure within Glacier is centered around walls and archives. So this is just a way of how it stores data and these walls can be created from Glacier dashboard but data operations are our API and CLI based. You would like to know this that data is encrypted by default. If you have a vault, you would also have something like a vault access policy which governs access control to the vault and each vault can have only a single associated vault access policy. Vault lock policies are similar to vault access policies. However, once they are set, they cannot be changed. So we spoke about versioning in S3. Once that is enabled, it cannot be disabled. Similarly, vault lock policies. Just similar to access policies, not entirely but once they are set, they cannot be changed. So word association, Amazon Glacier is a deep secure basement storage. For items, you want to keep safe for the long term but don't need to access frequently. It's very cost-effective more than S3 and provide excellent protection for your data, making it ideal for archival and long term storage needs. Let's talk about another AWS service, EC2 instance storage. Think of EC2 instance storage like the hard drive. Word association EC2 instance storage, hard drive. It is where your computer stores data. It needs to access quickly while it is running. EC2 instance storage, imagine your computer's hard drive. Imagine the hard drive inside your computer. This is where you store your operating system, your software applications and any files that you are currently working on. It is fast and right there for quick access. However, just like the files and data on your computer's hard drive, EC2 instance storage is great for temporary work. When you are done, you can save what's important elsewhere and clear the rest. This hard drive is physically attached to your computer. So it is always there when your computer is on. Similarly, EC2 instance storage is attached to the virtual machine. The EC2 instance you are using in the cloud. It is physically attached to that EC2 instance. Now, if you turn off or restart your computer, any unsaved data can be lost. EC2 instance storage behaves similarly. It is temporary. When you stop or terminate the EC2 instance, the data on the storage can be lost unless you save it somewhere else. What could be that somewhere else? We will get to it. But because it is directly attached, the storage is very fast. It's perfect for applications and tasks that need quick access to data like running databases, processing data or temporary storage during computations. So what are the key features of EC2 instance storage? Temporary storage because it is directly attached to your EC2 instance. High performance again because it is directly attached to your EC2 instance. And aphemeral because data can be lost if the instance is stopped or terminated. So it is not for long term storage. So why would you use it? For temporary data processing, ideal for data that is needed during the runtime of your applications or processing tasks. High speed access. Perfect for applications that require very fast read-write speeds like databases or big data processing. And cost effective for temporary tasks. Use it for tasks where you don't need to keep the data permanently helping to save costs. So what are some of the other key facts? The volume physically resides on the same host that provides your EC2 instance itself, acting as a local disk drive, allowing you to store data locally to that instance. Any data stored on these values is considered temporary. If your instance is either stopped or terminated, then any data that you have stored on that instance, store volume associated with that instance will be deleted without any means of data recovery. However, if your instance was simply rebooted, your data would remain intact. Right? From a cost perspective, the storage used is included in the price of EC2 instance itself. So it's free. And not all instance types support instant store volumes. So these are all the things that is great to know. When you are working on it, you will come across it. What we are trying to create here is identify the patterns. So EC2 instance storage, think of like the hard drive inside your computer. It's fast, temporary storage directly attached to your virtual machine in the cloud. Great for temporary tasks and high speed data processing, but not meant for long term data storage. Let's talk about EBS or elastic block store. So if EC2 instance storage is temporary, what do I want to do if I need something permanent? And this is where EBS or elastic block store comes into picture. So think of this like an external hard drive that you can attach to your computer. It provides storage that you can use to save data and it remains available even if you turn off your computer. So think of it as an external hard drive that you can attach to your computer. It provides extra storage space for your needs like for your storing files, applications and operating systems. It is a persistent storage. So unlike your temporary storage inside your system, an external hard drive keeps your data safe even if you turn off or restart your system. Similarly, EBS keeps your data safe even after you stop or restart your EC2 instances. It is flexible and removable. So just like you can plug an external hard drive from your computer, you can attach and detach EBS volumes from your EC2 instance. It is obviously scalable. If you need more storage, you can easily get a bigger external hard drive or add more drives. And backup a snapshot. So you can take a snapshot of your external hard drive to backup your data. So EBS allows you to take snapshots which are like photos of your data at a certain point in time. And these snapshots could be used to restore your data if needed. So what are the key features that we discussed? Persistent storage, high performance, flexible and scalable and backup and recovery. So why use Amazon EBS reliable storage for applications? Use it to store data for applications that need to keep data data even when they are not running, such as databases and enterprise applications. Flexible and scalable storage adjusts storage size and performance as your needs changes without needing to replace the storage device. And then obviously data backup and recovery. But if the runtime data needs for an application are pretty high, you have typically seen Amazon EBS being used for databases. I would suggest not using this as a storage service. We have other better ways in AWS that could be used to leverage those functionalities. So what are the other key facts? EBS volumes can be attached to your EC2 instances, but they are independent of the EC2 instances, meaning they exist as two different resources. And EC2 instance is a different resource than an EBS volume. But unlike instance store volumes, EBS does offer persistent and durable data storage. It also offers the ability, ability to provide point in time backup of the entire volume known as snapshots. Now this is really interesting. The snapshots themselves are then stored on Amazon S3. So you know they bring along the durability, reliability and incremental snapshots or features that could be associated with Amazon S3. And there are two types of EBS volumes available. These being SSD, back storage, solid state device and HDD back storage, hard disk drive. Very similar to how you have it for your computer. What great feature of EBS is its ability to enhance the security of your data, both at rest and when it runs it through data encryption. One thing to note here, especially for global companies, is EBS volumes can only be attached to EC2 instances that exist within the same availability zone. So it could only be in the same AZs. So word association, you think Amazon EBS think of an external hard drive for your virtual machines in the cloud. It provides persistent high performance storage that remains available even when you stop or restart your instances. It is flexible, scalable and support data backup and recovery, making it ideal for applications that needs reliable and durable storage. Let's talk about the next storage services, EFS or elastic file system. So you would see a pattern here, right? We started with EC2 instance storage, then we went on to an external hard drive, elastic block store and now we come to elastic file store. So think of Amazon EFS or elastic file store like a shared drive, then you can access from multiple computers, allowing everyone to work on and share file together as easily. So word association, elastic file store, think of shared office drive. So shared drive in an office where all employees can access and store files. It is a common space where employees can work on and share documents, spread sheets and other files, one drive. It is accessible from anywhere. The shared drive is not just limited to your office. So you can collaborate and access files from different places. It obviously automatically scales. So you don't ever have to worry about running out of space or manually increasing storage capacity. High availability and durability is a given and supporting multiple users is inbuilt. This is what it is for. It is perfect for collaborative projects where team members needs to share and updates file frequently. So what are the key features that we spoke about? Scalable storage, high availability, multi access and simple management. So it is easy to set up and use with minimal management required. Why use it? For all the things that we discussed, collaboration, if when you need to share and work on files together, scalability, storage needs are growing, you don't need manual interventions. reliability and then obviously ease of use. It's very simple to use. If you haven't tried it out, it could be set up in a matter of minutes. What are some of the other key facts for EFS? It is considered file level storage and is also optimized for low latency access. But unlike EBS, it supports access by multiple EC2 instances at once and this is the major difference. So EC2 instances can be configured to access EFS instances using configured mount points. Now, mount points can be created in multiple AGEs that attach to multiple EC2 instances. So once you are actually configuring them to be used with your EC2 instances, you would have multiple options available to you, but you would actually associate them using or configuring the mount points. Now, EFS is fully managed. This is highly available and durable services. So good. We don't have to do anything. And as the file system can be accessed by multiple instances, it makes it a very good storage option for applications that scale across multiple instances, allowing for parallel access of data. Unfortunately, EFS does not currently support the Windows operating system or the last I checked. If it has changed, please let me know in the comments or in the emails. But last I checked, it does not currently support the Windows operating system. So how do you remember this? Think of Amazon EFS like a shared office drive that multiple people can access from anywhere. It automatically scales to meet your storage needs, ensure your files are always available and protected and support collaboration among the P members. So we went through some of the storage services. We deep dive into what is an Amazon storage. Then we looked into some of the services like or we spoke about services like S3 or Glacier and CloudFran storage gateway. So I think we are at around 33-34 minutes. Maybe a good time to start. I still have more Amazon storage services to discuss with you. But we can take it up in the next episode. For now, what are the key takeaways from this episode? Amazon storage is like a virtual warehouse where you can store all kinds of digital stuff. Some of its key services are S3, Glacier, EBS, Snowball, Instant Store, EFS, what not. Then we deep dive into Amazon S3 which was a what? A storage locker. Then we looked into Glacier which was a storage basement. Then we looked into EC2 instance storage which was a computer's hard drive. We moved on to elastic block store an external hard drive. We then moved on to elastic file system, a shared office drive. In the next episode, some of the other key services that we would touch upon are CloudFront. We will talk about AWS storage gateway, Snowball and potentially some more. Let me give it some thought. We will also find an answer to what the difference between AWS database, foundational services and AWS storage. So get me your answers by then. I am also speaking at European Identity and Cloud Conference in the first week of June. So I would be virtual but if you are there, let me know. I would love to meet up for a coffee. You can always reach out to me via LinkedIn or you can always email me at the identity navigator at gmail.com. Thank you for listening. This is Rohit, your identity navigator.