#166 Optimizing Cloud Data Warehouses with Salim Syed, VP, Head of Engineering at Capital One Software

Hi, I'm Maggie, VP of Curriculum at Data Camp, and I know firsthand how overwhelming it can be to keep up with the rapid changes in data and AI. But staying ahead doesn't have to be hard. That's why our Curriculum team has curated the best learning content to help you and your teams stay ahead, covering everything from data and AI literacy for busy executives to cutting edge generative AI for developers and everything in between. Whether you're upskilling yourself or your team, we make learning practical, engaging, and flexible. It just works. Power your data and AI transformation journey on DataCamp.com. When it comes to cost optimization, it's never once and done deal. You have to be vigilant. It's a constant battle. So you need to be vigilant. You need to be watching for new users, new usage, new usage pattern. Work loads are always introduced in the system. So don't focus on reducing cost. Hello everyone. Welcome to DataFrame. I'm Adele. Data Evangelist and educator at DataCamp. If you're new here, DataFrame is a weekly podcast in which we explore how individuals and organizations can succeed with data and AI. I think it's safe to say that effective data management is essential for any organization looking to succeed with data. In most of the time, organizations look to storing their data in the cloud to begin that journey of data management. However, things can get easily out of control as you're transitioning to the cloud. Costs can be loaned as you set up your infrastructure, data governance becomes more and more challenging and a lot more. So how do you make sure you navigate that cloud journey effectively? Enter Selim Sayed. Selim is the VP and head of engineering for Capital One Sling Shop, which is a tool Capital One software developed to enable organizations to optimize their data and the cloud. He led Capital One's Data Warehouse migration to AWS and is a specialist in deploying snowflake to a large enterprise. Selim's expertise lies in developing big data lakes and data warehouse strategies on the public cloud. He leads an organization of more than 100 data engineers, support engineers, database administrators and full stack developers in driving enterprise data lake, data warehouse, data management and visualization platform services. Throughout the episode, we explore cloud data management and the evolution of Capital One Sling Shop into a major multi-tenant SaaS platform. The shift from on premise to cloud-based data governance, the role of centralized tooling as you transition to the cloud, strategies for effective cloud data management, including data governance, cost optimization and waste reduction, as well as insights into navigating the complexities of data infrastructure security and scalability in this data driven age. Enjoy this episode, make sure to subscribe to the show, give it a rating, share it on social, we'd love to hear your feedback. And now on to today's episode. Selim Sayed, it's great to have you on the show. Yeah, I'm happy to be here. So you are the VP and head of Sling Shop Engineering Capital One software. So maybe first of said the stage, walk us through Capital One software and Sling Shop and your role as its head of engineering. So Capital One software is an enterprise B2B software business of Capital One. It's dedicated to providing data management solutions to companies that are operating in the cloud. And our foray into software business, it started a long time ago with building so many different software in-house software. At the time we were in the cloud, we just didn't have those abilities. How do you manage cost? How do you manage efficiency? How do you do data governance in the cloud? All these we had to build ourselves. And then we also realized that we're not unique to this challenge. There are other companies that are facing similar challenges while they're going to the cloud or already in the cloud and now experiencing these challenges. So our offering is to help other companies be more efficient and in the cloud. That's great. And maybe a bit expand a bit on Sling Shop as well and what Sling Shop is? Sling Shop is our first B2B software and it's designed to help. It's a data management solution that helps companies optimize cost and accelerate their journey into snowflake as well as it really allows you to remove your wastage and inefficiencies from the way you're using your data platform. And I want to center today's discussion on really how organizations can maximize the value of managing their data in the cloud. In many ways, centralizing your data in a cloud data warehouse and building a modern data platform is really table stakes for any organization today trying to be data driven. A lot of data leaders have this all of mine. So maybe kind of getting a bit of background as well walks through why that is and why nailing data management and data infrastructure should be top of mind for every data leader today. The way you get value in a business, it really starts from data, right? It's understanding data that insights the insights that are coming out of data is going to drive your business. It's going to help you with your customers. It's going to help make their lives easier. The data is very crucial. And what we're seeing is that when you move to the cloud, the amount of data, the type of data, all this increases exponentially. If you have all that data coming in and it's not well managed and well governed, you don't know where the data is and the right access is around it, right security. All that will make it more challenging for you. Either you will not have business cannot get insights in a timely manner or you will create silos that will prevent you from accessing each other's data and seeing the value that comes with sharing of data across your organization. So to have a good data management is really crucial from the start. I couldn't agree more because ultimately you're creating that pipeline right from collection to enable people to draw insights and that pipeline needs to be resilient needs to be highly efficient, well managed pipeline. So maybe zooming out on the broader context of the industry, you work with a lot of industry partners and worked on data management extensively at capital one internally. Where do you think we are today when it comes to organizations effectively managing their data, where are we on this journey as organizations mature their data management. This is a very good question. Different companies are different places in their maturity of data governance and adoption of cloud. But what I'm seeing is that one of the places where I see some some friction is that a lot of companies are moving to the cloud, but they still have the mindset of on premise. And let me explain what that means. That means that in the on premise world, you had a centralized team managing all of your data, right. They're the ones who are saying, this is the data you can have. This is they're doing the data governance data, data publishing, all of that. And that really is hard to maintain in the cloud, especially because now you're going to have an explosion of data as I explained. And this is all the businesses really wants to go at the speed of their demand and they cannot be dependent on a central team and the central team doesn't have the domain expertise across the lines of business. So they become a bottleneck in the cloud. So that's where we see a lot of friction around adopting this new mindset where you want to allow the businesses to move at the speed that they need. But also there's two things you have to really be careful of. One is creating silos. And the second is how do you enforce enterprise centralized policies governance policy across uniformly across your organization. And that to find that balance capital one has built the way we look at things are, you know, you have central policy and central tooling, but then federated ownership. What that allows you to do centralized team to do is create the policies, make sure the tool enforces them or puts the right guidelines or a guardrails around how you publish data, how you access data, you govern data. But then gives the ownership for the line of business so they can move they can publish on their own they can find data on their own without having a bottleneck on the central team. And I see that as the as the biggest challenge for companies to really truly adopt the cloud mindset. Okay, that's really great and maybe kind of expanding on that cloud mindset. What have you seen our ways to deal with this particular challenge that you mentioned here on having this federated approach while still centralizing data governance and data controls maybe walks through this kind of solution solution landscape in a bit more detail on how data should be approaching this aspect or this challenge of data management. It starts with what capital one believes in that how to solve this problem, which is centralized tooling and centralized policy. What we've built is we've been centralized tooling for our imagine our data publishing. I'll give you two examples of publishing to your data environment and then what consumption needs are first let's talk about publishing. So one way to publishing is you create centralized policies you give it to the line of businesses they publish data into their data environment. It's very becomes from a central team it becomes very difficult to enforce all those to guarantee that everyone's following the practices. So what we've done is we've built central tooling that allows you to publish. your data and making sure that the publishing, all the governance need for publishing data, for example, registering your data. Are you checking for data quality? Are you identifying what is sensitive, what is not? Are you making sure that the sensitive columns like credit card or personal information are encrypted or tokenized? All that is governance need for publishing a pipeline. And what we've done is we made that transparent to the business so that if they're using our central tooling, all that comes free. So not only is it going to help you be more compliant, but also by making sure that everything is registered, the data quality is collected, it makes it on the consumption side for our users and employees when they're accessing data, they have, first of all, they can find data very quickly, find relevant data very quickly because now we have all the data registered in the right places, we're collecting the right metadata. And also we provide the consumers of the data enough information about the data that they can trust it. So think about what is the result of your data quality, what is the sample data, sample data, min, max, data profile, all that is given to you so that you can trust it. And then in the same experience, we allow you to access the data, request access to data. So that it goes to the right owner of the data and they can improve and you provide your business justification. The point I'm trying to make is you want to make sure that all the data governance needs are as easy as possible as transparent as possible. So they don't become hindrance to the developers or the data producers or the consumers to getting their job done, right? And you get both benefits, you get the benefit of compliance and meeting all the regulatory requirements as well as it's easy to find the data, it's easy to get to the data quickly and get the value out of the data. That's really great. You mentioned a lot here, the data governance aspect to it, but one thing you mentioned earlier was the need to develop this federated mindset and to let business teams also lead with their data while having kind of centralized policies. Maybe what have you seen are effective ways teams can organize themselves around this model as data management as well evolves. Maybe walk us through the different ways. Successful data teams have been able to organize themselves around and successful data producers and consumers have been organized as a consequence of this shift in mindset. The best people who know the data are the businesses. They have all the domain expertise. They understand what the data means, how to use it and you want them to own it, right? And that's where the mindset comes at. They are the ones who know it, give them the tooling that allows them to publish data quickly, allows them to ingest the metadata about the data so that everyone can get benefit. And the other thing you need to make sure is that data has to be with proper access control. It needs to be, you don't want the architecture to limit data sharing between organizations. If architecture allows that, then you can build access controls on top of it. You can build that. But as soon as you create an architecture where you have to copy data out of one place into another and go through multiple hoops, it becomes very hard and it slows down the data sharing. So data sharing is going to be a very key component of any organization. And you want to make that easy, but with the right control in place. And we're talking about controls. We're talking about data governance, right? This often makes me angry. As you mentioned, between having centralized data governance policies, but also a federated model, maybe who should be the stakeholders involved in setting up a data governance strategy? Maybe walk us through in a bit more detail what a successful data governance journey looks like here for organizations as they set that up. You have to have buy-in from your leadership. It requires buying from the businesses. Their own governance, own risk officers that allow them to manage their own risk. Because every line of business has different risk around it, right? You want to make sure that they have the right ownership. They have the right risk reviewers within their line of businesses as well as essential policy. And the way it really works is if you have a leadership that says, this is the way we're going to do it, but this is the benefit all organizations going to get it. Then you have a lot of buying. But without the buying from leadership, it becomes very hard to start this from the grounds up. So that I think is a requirement for success. I couldn't agree more here, especially on the importance of buying for leadership. Now speaking of buying, I think in a lot of ways, the other side of cloud transformation and effectively managing data in the cloud is actually ensuring that people have the tooling, scales, the ability to adopt data, right? And which is this is much more of a people in tooling problem than it is only a data management problem. So maybe walk us through good examples of data adoption that you've encountered at Capital One and Capital One software customers. It comes really down to, again, I said the leadership, but it's a data mentality, right? You want to have everyone be educated on making data driven decisions. And it needs to show from up down everywhere, even during development processes, you're making decisions based on data. So that's the culture you want to create first. But on top of that, you want to make sure that it's very easy to find access and trust data. So the democratization of data is very important in any organization to allow just data engineers to get access to data, but make it so easy that anyone with the right access and need can get value out of the data. So that and then third piece would be just education, train the trainers and make every data, and during experience where this inefficient season, the way you're accessing data, make it a teachable. So teachable ones create more, more, it's got a trickle down effect across the organization. So that's how I see it. Okay, that's great. And maybe expanding here on the aspect of providing access to data, you mentioned here in effective ways of surfacing data. I think one common anti-pattern we see in organizations today is that there's really limited context on how this data is useful and could be used. And a lot of organizations are trying to solve this with a metadata platform, right? Or a data catalog. I would love if you can comment here on the importance of surfacing metadata and providing that data catalog for the wider organization. And what are effective ways that you've seen as well to provide that context for the wider work? Yeah, metadata is everything. And a lot of catalog is the key on that metadata allows you to know about your data. It allows you to know not just the business context or the technical context of the data, but it's actually becoming bigger than how metadata was used in the past. It was about catalog due to you kept the business and technical metadata. But now there is a concept called passive metadata, which is all the cost associated, security, all the resiliency associated with the data set. How do you track that along with the metadata along with the data? So think about a table you have in your environment. How often is it used? Is anyone using it? How often is it updated? How much does it cost to maintain that table? How much does it cost to access that table? Those are all passive metadata. And the bigger view you have across all the metadata gives a company a lot of edge around where not only about providing information or what data you have in your environment, but also what's valuable, what is used more, what two data sets are relatable. How often people join the two data together to create the insights. All that becomes very important. So you always want to start with the catalog, but it needs to expand on just static metadata. It needs to be alive. Everything that's happening to that data needs to be collected at the same time to give you even more insight into your operational excellence. Right. Yeah, this often makes me think about really the importance of providing as well the business context on how data is being used, like effectively which teams are using it, which queries, which tables is this data used. Yeah, I couldn't agree more here on the importance of seeing how that data is being transformed. Now as we're discussing about a lot of the challenges organizations face when it comes to data management, I'd love to also learn more about capital one slingshot. Right. So maybe walk us in a bit more detail what capital one slingshot aims to solve. In a lot of ways of this challenges that we're discussing here today were challenges faced by capital one. So I'd love to learn how capital one slingshot aims to solve a lot of these challenges. Before I get into capital one, the slingshot, I want to explain the concept of optimization. One of the things that happens in the cloud is that when you are on premise, your data platform cost are pretty much static, right? You buy a certain number of servers and then you have some constraint on how much you can use, but the cost doesn't fluctuate until you buy a big urban or expand. It took four to five months to do that. But when you move to the cloud for the first time, cost is now dependent on how you use it and how much you use. So the more compute you use, the more queries you run, the more your cost is going to be. And what that means is there's a room for a lot of inefficiencies that you probably had in the oil environment, but it didn't affect your cost, it affected your performance, but it didn't affect your cost. But now it does affect your cost. So what Slingshot does is it tries to help you optimize your cost. But the way it does is in a few steps, right? And first is it provides you the visibility into your cost. Where are the costs spiking? Where where are the cost? High, low, all that. Second, it gives you near real time alerts and insights into the cost drivers. The last thing you want to do is wait until a month and then build comes and realize that you had one query running for 30 days, I run away query. So you want to know about that as quickly as possible. Then the last one is not everyone is going to be an expert in Snowflake to know how to optimize their queries or optimize their server configuration. So what we try to do is we try to recommend you, when we give you a, we find the inefficiencies we try to give you a recommendation on this is why we're seeing the inefficiency and this is exactly what you can do or give you a couple of options. And then in the tool allow you to make the changes to your settings, right? And all three together allows you to not only save cost, remove inefficiencies, but also get everything that happens in the Snowflake environment is basically all managed and well governed because we also have a way for you to provision resources, change resources, all through approval process and the proper tagging. So it's well managed, well governed as well. It's in the nutshell what Slingshot does. >> That's great. And when you mention your on-console optimization, maybe we should have covered that a bit more depth. Maybe walk us through the drivers behind ballooning costs when managing data and the cloud. >> Yeah, there's a way I see it as there are four different areas where cost and we've so variable that I haven't explosion if you don't manage it well. First one is the compute cost, which is in the modern data warehouses as data platforms, you always have a compute aspect which is separate from your storage. And the more queries you run, the more you spend. The second one is just the query itself, the query optimization. Third one is your data set optimization, which is how much data do you store, how is it being modeled correctly. And the last one is environment optimization, which is around there's gonna be a lower environment, dev, QA and production. And in these, there's a lot of inefficiencies that can happen in the lower environments as well. But let's go back to the first one, the compute. This is where what we understood is that there's no one size that fits for all times of the day. What we see is that workload usually goes up and down based on the time of the day, day of the week. And you want or snowflake resources to size up and down accordingly. And today, snowflake doesn't let you do that. It gives you one size and you have two. So it's very important to know the ups and downs and making sure your compute goes with it that way you are spending the most effectively. Also making sure that your servers are shut down and not in use and you have some timeouts in your queries so that it will run away query keeping a server up. So watching for those is very important. The second one is query optimization. Like I said, queries the longer the query runs, the more it costs. So if you have a badly written query, that does, for example, a Cartesian data join, that can forever. All that will cost you a lot of money and you won't even know about it unless somebody's watching that. So it's very important to not only build in alerts or run away queries or but also provide a way to give you advice on how do we write the query and get the best performance and lower the cost. And our sling shot tool does that also. And the third one is data set optimization. I call it data set optimization because it's got a few aspect to it. One is storage costs. Even though you think storage costs is much cheaper in the cloud compared to on premise. But when you're dealing with multi-petabyte of data, then which cart will sneak up on you. So it's very important to have a retention strategy on your data. That means know how long you want to keep it. It could be based on business requirement, regulator requirement. Figure out a way that your storage will not keep going up and up. Otherwise, what happens is even if you start with a 5% storage cost in year three, it'll be 40% storage cost because you're not purging anything old or you're not archiving anything old. So it's very important to keep a retention strategy. Second is a lot of data that we're loading today in our data warehouses, you have to understand the consumption patterns as well. So there's no loading something in real time when you're consuming it once a month. Because there is a cost to loading data as well. So these are the insights you want to draw from a tool that allows you to know, OK? What is the best way to load the data that helps with the usage? And the last point is on the environment cost. One of the things you notice is when you're building data pipelines in lower environments, there's a lot of room for efficiencies there. For example, one of the things we did was we made a rule that says you cannot get a dev environment, you can't have a server larger greater than small, for example, size. And just by enforcing that rule, we saw significant savings. And if there is a need for something larger than that, you go through a certain approval process. But to enforce rules like that, to enforce the jobs that are when you're testing something, when you're developing something that are not just running when you're not working, right? And those things can happen. And so it's very important that you pay attention to lower environment, even though that's a much smaller percentage of total cost. But that can also balloon out of control if you're not putting in the same inefficiency checks that you're going to put in production. So what I'm trying to make is don't forget about the lower environment cost. That can sneak up as well. Yeah, I love this overview. And I love how you thought about all of these edge cases and all of these drivers of bullying cost when designing, capped one's link shop. But maybe you're switching gears slightly and thinking about how leaders and organizations should think about their data management journey and to avoid ballooning costs. A lot of organizations are so early in their cloud journeys. A lot of organizations are still setting up their cloud infrastructure. And maybe what advice would you give them to make sure that they're driving as much as ROI as possible when it comes to data management? How do they avoid being in a mode where they're increasing investment and ballooning and costs are ballooning without necessarily driving a lot of ROI from data activities? There are times that the companies that are moving to the cloud don't see what's right around the corner is they go. And then they face all these different challenges around governance or on cost or on accessing data. So my recommendation and advice has always been think about data governance, data management when you're starting off the journey to the cloud. Sometimes it's much more harder once you open the Pandora as fast to put everything back in because once it's much harder to do that if you don't have your data registered across your organization now you're going to have to go through and collect all that. But if you think about a central tool, central policy, then from any new data that's created, automatically will be well managed while governed. So that's the first piece that you should think about is invest in a good tooling that allows you to do the centralized policies, enforce the centralized policies, but give ownership to the lines of businesses. As well as on the cost side, I would wreck the same thing. It's very important to from the beginning put in the right policies on how you're going to do charge back to your line of businesses. Are you going to have a budget or not? Are you going to have a way to request more funding? All that needs to be part of the data platform in this modern world. You can't just have-- since it's a variable cost, you need to have a proper funding, proper budgeting, and proper way to asking for more resources. The right approval workflows should be built in. And then visibility is very important on the way you're spending your money. Yeah, that's correct, Greymor. And what's interesting hearing about your journey now leading capital one's link show, and as we discussed a lot of ways and a lot of ways link show was built upon, capital one solving these problems that we're discussing right now. Maybe walk us through briefly the journey of productizing a tool built internally and productizing it to the wider market. What changed if any in your approach as you were productizing slingshot? The first thing what we had to do was we had to build-- the tool we built within capital one was very specific to capital one. So it was an integration with capital one infrastructure already. So we had to build a SAS. platform that was supposed to be multi-tenant, that is supposed to be a multi-tenancy. It has to protect the customer's data. It has to make sure that the data is done overlap each other. So we had to create a whole new SaaS framework. We had to create a different software development practice to help with this journey. And one of the things that you're going to see from CapurOne build software is that it's security, moments, scale, resiliency. These things are just built into our DNA. So everything we build is built for a hardened platform. And that's how we've always started, even though other startups will start with just providing you the features, then figure out how to harden it later. We've always made sure that the product you see is going to be a hardened enterprise-grade software. And as we are closing our episodes, today's Sunday, when looking ahead, maybe what are friends that you're excited about when it comes to organizations managing their data effectively? How do you think the landscape will evolve over the foreseeable future? No, I think it's already evolving into that federated mindset. The more companies I say, they realize that it's going to be very hard to manage it from a centralized point of view. So that's the shift that's happening, but it's not happening as fast as I'd like to see. But it's clearly evident from the strategies of different companies. The way they execute it is a little different. They may keep data across multiple platforms. But what's interesting is you have a central catalogue across all of central access policy across all your data. And those are the key part. That way you break the silos and you have a single way to get data even though it's owned by different line of businesses. Yeah, there's certainly an exciting development. As we see that federated mindset, evolving and empowering organizations and different lines of businesses to use their data effectively. Finally, Selenium is great having you on the show. Before we wrap up, you have any final call to action or notes to share with listeners before we end today's episode. Yeah, absolutely. This is what I say quite a lot is when it comes to cost optimization, it's never once and done deal. You need to be watching for new users, new usage, new usage pattern workloads are always introduced in the system. I would say focus on reducing waste and reducing inefficiencies in the system. And then you will be able to scale with with peace of mind knowing that the money you're spending is going to the value generating for the business. That's really great. Selenium, it was great having you on data. Thank you so much. It's been my pleasure.