Go back

Etsy's Big Data Cloud Migration

from The Enterprise AI Show

32m 27s

Etsy's Big Data Cloud Migration

Etsy’s cloud migration journey highlights a strategic shift from on-premises Hadoop to Google Cloud, driven by the need for scalability, flexibility, and team autonomy. The process was carefully phased, using isolated sandboxes to ensure data integrity and prevent disruptions during the transition. Despite initial cost overruns, the move unlocked significant operational efficiencies, enabling teams to build complex data and machine learning systems more rapidly. Key benefits include auto-scaling with Kubernetes, streamlined access to BigQuery and Dataflow, and reduced reliance on specialized infrastructure teams. While Google Cloud’s AI tools are impressive, Etsy’s unique marketplace data demands custom solutions. The migration also reinforced that cloud transitions are inherently complex and unpredictable, requiring patience and adaptive planning. Ultimately, Etsy now enjoys faster experiment velocity, improved system resilience, and a more agile data platform—proving that thoughtful, incremental cloud adoption delivers long-term value. The episode underscores broader industry trends: cloud platforms are evolving beyond just infrastructure to become developer and machine learning ecosystems, with companies increasingly prioritizing platform evolution over simple migration.

Transcription

6608 Words, 36253 Characters

English
Today's show is sponsored by DigitalOcean. DigitalOcean offers the simplest, most developer-friendly cloud platform. It's optimized to make managing and scaling apps easy within two out of API, multiple storage options, integrated firewalls and load balancers, a new managed Kubernetes service, and much more. From predictable pricing, to flexible configurations, to world-class customer support, you'll get access to all the infrastructure services you need to grow your business. Plus, DigitalOcean's community provides over 2,000 tutorials to help you stay up to date on the latest open source software, languages, and frameworks. So to get started on DigitalOcean for free, with a free $50 credit, go to dio.co/cloudcast. That's dio.co/cloudcast. Cloudcast Media presents from the massive studios in Raleigh, North Carolina. This is The Cloudcast with Aaron Delb and Brian Graceley, bringing you the best of cloud computing from around the world. Good morning, good evening wherever you are. Welcome back to The Cloudcast. We are coming to you live from the massive cloudcast studios here in Raleigh, North Carolina. Kind of a slow week in terms of cloud news of the week. I suspect that is because we are, in August, it is summer vacation for a lot of folks, and probably particularly the venture capitalist and the M&A folks that are probably summering in Nantucket or Martha's Vineyard or somewhere in the Caribbean or somewhere in the Mediterranean or something, but we'll give those folks a few weeks off, a little bit slow news week. There was one interesting story that I noticed this week. Not a lot of acquisitions, not a lot of big announcements, but kind of a smaller announcement that I think will probably end up becoming a little bit bigger over time. GitHub announced that they are now offering CICD as a capability or a feature for GitHub actions. And this is sort of important because I think it's one of those things where GitHub is still trying to remain as neutral as possible. They are beginning to slowly do more integration with Azure. And I think as we see people beginning to make some decisions about our things like serverless or functions or just code-based, low-code, no-code types of things. Becoming bigger, we are seeing slowly GitHub is becoming not only just a place to store code, but a place where actually actionable things happen around that code, and not just collaboration, but things like actions, which are sort of a serverless type of function. We're now seeing CI kind of slipping in there. And I think we'll begin to see over time that GitHub becomes a more important sort of cloud within a cloud, right? So we will often think about Google Azure and AWS as the three major clouds. I think we're going to begin to see GitHub really begin to emerge as sort of more of a developer cloud. So that'll be an interesting thing to watch over time. Definitely, they sort of, I don't want to say they slow-rolled it. It wasn't a massive, massive announcement, but for those kind of in the know, it kind of went out there. And I think it's something for us to keep an eye on. We'll do something we'll dig into in a later show. One other thing I thought was interesting and I'll put it in the show notes. Gartner released their low-code, no-code magic quadrant. And this is an area that's always been sort of interesting to me. I think, as we talk about, software eating the world, developers being king makers. At some point, the world can't just operate on the developers, the highest end of developers, the ones that make hundreds of thousands of dollars, millions of dollars, it's got to be sort of democratized a little bit more. You know, some of the low-code, no-code platforms don't necessarily get as much hype. But I think it's an area that we will see over time become more and more popular as data becomes more readily available. Development skills are sort of taught in colleges and elementary schools and high schools and so forth. And some of these low-code, no-code tools become more prevalent. You know, we talked at one point about AWS planning to roll out a low-code, no-code option at some point. Maybe they'll come out at re-invent. But I'll put that in the show notes. We're going to link to the Gartner thing. I know folks had different opinions on Gartner, but it is always kind of interesting to kind of get some perspective on, you know, who are the companies that are in general leading and also some of the up and comers in that space? So that's it. Gonna wrap it up for cloud news the week. Again, kind of mentioned as a slow news week. We do have a very interesting show for you today. We're gonna talk with the folks from Etsy about their cloud migration from a lot of their data science, centric services from being on-prem into the cloud and how they went about doing that with Google Cloud. So look forward to that interview and we will get to that next. Today's episode is sponsored by DataDog, the cloud scale monitoring service that provides comprehensive visibility into public cloud, hybrid cloud, multi-cloud environments with over 250 integrations. DataDog unifies your metrics, logs, and distributed request traces into one platform. So you can investigate and troubleshoot issues across every layer of your stack. Use DataDog's rich, customizable dashboards, and algorithmic alerts to monitor cloud migrations in real time. To start a free trial today and DataDog sending you a free t-shirt, visit dataDogHQ.com/cloudcast to get started. That's dataDogHQ.com/cloudcast. And we're back. Aaron, it is good to have you back. How you been, man? Been good, been good. How about yourself? Doing well, you know? Just got back from some of the summer break stuff. So summer is in full effect and the heat is overwhelming at this point. Yeah, no kidding, absolutely. So, you know, we, I think we're always very lucky to get a lot of the guests that we have on our show. And a lot of times we get, you know, technology vendors and they're talking about new technology. And not enough time do we get with some folks that are not only kind of on the building side of things. They're living and breathing the technology. But, you know, we get a chance to talk to some of the companies that just really shape the internet. We're incredibly lucky. So, you know, today we get a chance to do one of those things. So very excited to have Kevin McCale, who is one of the senior staff engineers at Etsy, especially on the data team, Kevin, welcome to the show. Hi, yeah, thanks a lot. Thanks for having me. It's a pretty intimidating introduction. I hope I can live up. No, we've got it. Oh, it's, it'll be great. We're looking forward to it. You know, so before we dive into some of the work that you're doing at Etsy, you know, we look at your background. You've done a lot of things that have kind of nothing to do with online retail. And you seem to be kind of incredibly good at math. So give, give folks a little bit of your background. Just some of the things that you've worked on, kind of some of the scale of things that you worked on even before coming to Etsy. Yeah, sure. So, you know, I don't know if I'd say I'm good at math, but I certainly studied it long enough to know how much I don't know. You know, my undergrad training was basically a math. I did a PhD in experimental physics. All of which taught me, you know, a lot about things that might not necessarily seem relevant every day to e-commerce, but that I almost feel like I do look back on a lot every day. I mean, just a lot of basic ways of thinking about things and solving problems and project management and stuff like that have all been really important to me. But, you know, obviously, really useful and things that I picked up during that period of time. But since then, I've kind of moved into the tech industry kind of slowly starting as an applied mathematician working for the US government and then kind of gradually getting drawn into data science and while doing that, learning a lot more about software engineering and ultimately kind of culminating in my current position, which was the first time I've actually been formerly a software engineer and not a data scientist or whatever. - Yeah, so it makes sense 'cause it's so fascinating to hear, we're hearing more and more about backgrounds like that or backgrounds that maybe are less traditional making into these roles. And then you get to be involved in some really, really cool in interesting projects, right? So there's two very interesting projects at Etsy, both kind of building evolving the data platform, if you will. And then helping to manage this migration to Google Cloud platform that we're going to talk about a good bit probably. So let's start by talking about the data platform, though. First, tell everyone a little bit about the evolution behind it and then the process and what happened during all of that. - Sure, yeah, so, you know, our data platform, well, so, you know, so Etsy, first of all, is a two-sided marketplace for vintage and handmade goods. We've got, yeah, we are kind of making an assumption there aren't we? (laughing) - You know, it's good to just sort of set the tone a little bit. You know, so we have something like 50 or 60 million items on our site, all of which are, or most of which are unique. Like these aren't things out of a catalog, like an iPhone or something. These are all handmade items or vintage items. And we've got, you know, millions of sellers, millions of buyers who are shopping on the site and trying to sift through this giant mass of stuff to find what it is that they're looking for for, you know, for whatever the special, you know, sort of gift or whatever it is that they're trying to find that they will enjoy themselves or give to people or whatever. So you can imagine like connecting people with these items, helping them find what they're looking for is a big source of machine learning problems for us. So we've basically over the past couple of years at the identified machine learning as one of the core kind of company wide initiatives to help try to improve that search and discovery process. And so in order to do this, we need to collect a whole lot of data. So basically every action that a buyer takes on our site will generate information that we collect on the backend. So, you know, things like clicking things, viewing things, favoriting things, searching for things. All of that is collected. And we generate over a terabyte of data a day. And we need to process that in order to, you know, produce better machine learning models or better recommendations and add placement. We have all sorts of in-house machine learning products now that are on the site and that are powered by this platform. And so now getting, you know, some more specifically to the platform itself, we're pretty reliant on Hadoop right now. And specifically we use Scolding, which is a scholar wrapper on top of Hadoop. And we have something like, I don't know, hundreds of workflows, thousands of jobs that are running daily and processing through all of this stuff. And you know, going back a few years, we were doing all of this on-prem in a data center that Etsy was paying for and we had hardware that we were sort of internally managing. And as we identified this machine learning priority and the volumes of data that we were collecting started going up and up and up and our in-house resources and in-house people just were not enough to keep up with this growing demand. And so that kind of inspired us to start looking at the cloud as one, you know, Etsy generally was looking at a cloud migration for other reasons too, but for the data processing side of things, you know, this was a big motivator that we basically were just kind of at capacity or even above capacity and we didn't have enough people or resources to catch up with where we already were. So now, you know, we basically over the past, say, year and a half have spent a lot of time migrating all of this stuff that we had, all of these Hadoop jobs and all sorts of others kind of scheduled operations and databases and stuff on the data platform into the cloud. And in order to do this effectively, we had to kind of rethink the way things were constructed because if we had just kind of lifted and shifted a big Hadoop cluster and turned it into a bigger Hadoop cluster in the cloud, we really wouldn't be, you know, we really wouldn't be gaining a lot of the benefits that are available to us by being in the cloud such as, you know, elastic clusters that spin up and spin down as needed, things like, you know, basically unbounded storage available through, you know, these big object stores that are available in the cloud. And another thing that was also a big motivator was with our big sort of single in-house cluster, everybody who was doing data science, which at this point, I checked yesterday, over a quarter of our engineers have contributed something to our data platform in the past year. So basically, everybody at the company or, you know, every team at the company is affected by this platform and having everybody on a single cluster with a single scheduler just created really tight coupling between teams and, you know, consequently there were lots of problems when, you know, failures cascaded all throughout the system or, you know, well, you can imagine how things can go wrong. And so by moving into the cloud, we've also kind of focused on decoupling a lot of our systems so that, you know, individual teams, like data science teams and the, you know, recommendation teams can all have their own resources that operate independently of each other and, you know, and give them the flexibility to do what they need to do while not imposing constraints on anybody else. We've had a number of people on the show, you know, guests in the past who have talked about, like, application migration. So it was, hey, you know, we had, we had front end or back end, we migrated that into the cloud. And we haven't necessarily had anybody with really deep expertise in sort of moving the data side of that into the cloud. Can you give some insight? I mean, you're still probably in the process of doing that, but when you're talking about that much information, moving it is not a trivial task. Can you talk, kind of, talk about how you get it there, how you sort of do it in stages so you can keep using it in day-to-day operations? Kind of what goes on behind the scenes there? Sure. Yeah. That was an interesting balancing act that took about a year and a half. Like, basically, yeah, basically, we knew for a long time that we were going to have to develop this new platform in the cloud. And that was going to have to coexist with stuff that was on-premise for, however long it took to move everything. And with a lot of these data processing things, there's, especially us as a platform team, we don't necessarily know what's going on inside of all of these jobs. We don't necessarily know. We don't have assurances that we can run them in some new environment without clobbering data in some old environment or making API calls to things that we shouldn't be making API calls to, et cetera. So we really spent a lot of time, and I think actually this goes back, you know, I mentioned earlier that my time as a scientist taught me a lot of things that are useful today. It certainly taught me like paranoia about data quality and data integrity and stuff. Paranoids may be a strong word, but I think that, you know, we treated this migration process as one in which basically none of these jobs could be trusted. And we needed the ability to run any of them, to test any of them in the cloud, but none of them could be trusted to not break anything on-prem. So we basically created our cloud environment with really, really tight boundaries everywhere. And so, you know, we can run individual jobs in really tightly sandboxed environments and gate whatever comes in and out of those environments strictly through things like network bounds or firewalls or whatever. And kind of gradually over time, we built up this system wherein, you know, we had processing happening on-premise. And generally, the on-premise stuff was, you know, always the highest level pieces of the pipeline. So like big data ingest jobs and big common share jobs that are used by everybody all across the platform. Those things remained on-premise for a really long time. And we would copy data up from on-premise into, you know, we're using Google Cloud. So into their cloud storage platform, you know, bit by bit. And then these sort of, these little sandboxes in the cloud would kind of run an isolation process that data right there output out. You know, we'd look for errors, see what they're trying to do and figure out what sorts of patterns we need to sort of, like, re-enable in these cloud environments. Kind of gradually over time by just taking this really, really, strictly, you know, bounded approach, we kind of just chipped away at the stuff that was kind of at the lowest levels of our processing pipelines and kind of moved stuff upward. And, you know, as of, I think two or three months ago, our data platform team finally finished moving the last jobs over. We, you know, turned off our on-prem clusters and we're fully in the cloud right now. Interesting. Very cool. Now, let me ask you this kind of a follow-on to that because there was certainly an analysis, if you will, of, okay, like you mentioned data integrity. I actually like that idea of paranoia. But the, the, that is certainly a big lesson learned and something you kept in mind very thoughtfully throughout. What are some of the other kind of lesson learned that maybe you didn't anticipate or didn't expect that maybe, you know, because you always hit these bumps, right? In some, what are some of the advice of some things to look out for for others that are considering migration such as this? Sure. Yeah. I mean, I think, I think probably the most obvious one that I think everyone in the company now is aware of is that everything cost more than we thought it would. In the, you know, in the public cloud? In the public cloud. Yeah. We, you know, obviously, you know, it's difficult to sort of anticipate or sort of translate what our jobs were doing on-prem with, with what that will mean for, for our cloud costs in the future. And so, you know, we performed all this migration primarily focused on just getting things into the cloud and, you know, sort of figuring that once the jobs themselves were running in the cloud, we can do whatever optimizations were necessary. But, but it definitely turned out to be true that, you know, that there was a little bit of sticker shock. Just primarily stemming from the fact that I think, you know, when we were in an on-prem environment, there was never really a forcing function to to encourage us to try to optimize these jobs and to make them more efficient. You know, it was just, you know, the forcing function would only be you get more failure emails or whatever if you're running out of resources. Whereas now the cloud will happily run whatever it is that we tell it to do. And, you know, that's great in some ways, but then in other ways, you know, sometimes you tell it to do something that maybe is a little bit too pricey. So I think that, that was probably the biggest maybe lesson learned that the things are going to be expensive. But I also view it as an opportunity in that, you know, it now does give us a real business case for spending time to optimize these things and make them more efficient because we can, you know, directly impact Etsy's overall financial numbers if we can, you know, take a big chunk out of our costs by building more efficiency. Oh, that makes perfect sense, yeah. Now, kind of a follow-on to that with it, you know, you mentioned you didn't want to do kind of a lift and shift and all of this. And so in going to Google Cloud or just really honestly, public cloud in general, right, what are some of the advantages like it is, it is funny, you mentioned that that cost was a bit of a surprise because in the early days of cloud, everyone was kind of like, oh, you know, move it there because it'll be cheaper and then kind of everyone kind of learned that lesson over time, right? So, so what is some of the advantages you've seen? It's, you know, there's obviously like kind of the obvious one is the acceleration of some of those work in greater velocity and flexibility, but, but tell us a little bit about, you know, some of the big advantages that that happened there from both like business and technical, as this, you know, the data platform has evolved. Yeah, that's a, that's a great question. I think that, you know, I would have to say that, I mean, you know, maybe this is obvious, you guys are the cloudcast, you guys probably, this is obvious to everybody, but you know, for us, moving to manage services has been a huge win. You know, we, I mentioned earlier, you know, we have these big hudu clusters on premise that were, you know, we're reaching a scale that we just didn't have the expertise to handle, I think. We did have to hire contractors occasionally to help us out with things. And now basically by being in the cloud, you know, Google is our, our ops team, like Google are the experts on how to configure these things. And so we don't need to have, you know, as specialized, as like, you know, single function specialized teams anymore, we can have teams that are, you know, sort of broader in scope and, you know, take on a bigger mandate in terms of what it is that they're doing for the product without having to like, you know, hike or specialize in, you know, hudu hardware configurations or whatever. So, so that I think, you know, I've given the hudu example, but that has been a sort of a broad win all across the Thank you. entire Etsy platform migration. Our search teams are using Kubernetes in the cloud. And this gives them all sorts of advantages, like self-healing and things like that. And it's auto-scaling that they didn't really have access to on-prem. And it's also, aside from broadening the mandate by using these things, let's sort of punch above our weight class or whatever the analogy is. I have a team that I'm working on right now. So I'm working on the machine learning team specifically on a project that's building out a sort of streaming Lambda architecture. I was kind of going through, while thinking about things I would say today, going through all of the services that we're using in Google Cloud right now and thinking through what it would take for us to manage these things where their open source counterparts ourselves, and it would just be impossible. So our team is four people. And we're using big table at a massive scale, which is a giant nose-equal database. We're using all sorts of streaming processing engines, all sorts of other analytics databases. And these are all things that we just create by writing a few lines of infrastructure code instead of having to research it and figure out what it takes to configure it and make them all work together. So I think that now that our migration is complete, and we're really able to start focusing on building new applications and building new logic on top of this layer, I really feel like the dividends are finally starting to become just really, really evidence in that we can use all these great technologies without having to have all the specialization needed to run them. Yeah, it's an interesting point. If I kind of, you know, correlate a little bit of what you just said, you said you said the beginning somewhere in the neighborhood of 50 million items on there and then a staff of four. So that's a hell of a summary, if you will. I'll ask you a quick follow-up on that. So it's not often I get to talk to somebody who's both familiar with Google Cloud and has a data science background. And you know, one of the big things we've talked about on this show a number of times over the years is Google Cloud and kind of each cloud has their kind of specialty in their services. And so, you know, can you give us some kind of thoughts on like the data science, AI, ML aspects of Google Cloud that tends to be one of the things they hang their hat on, at least, you know, publicly at like the Google next shows and some of these other things. You know, do you have any kind of opinion? We hadn't really covered up until now. So it's kind of off the cuff question, but I just wondering on that. Sure, yeah, so I do have opinions. Actually, it's interesting, you know, so I'm sort of familiar with some of the products that they offer like they have their Vision API, which is a really awesome idea, you know, you just send it an image and it will send you back tags that it has applied to that image and you don't have to train any models or anything. And it's a great way like off the shelf to just kind of try something out and see if it's going to work. And it's, you know, that and other things like it are really, really cool. And you know, I don't know of any other company that makes quite as many things just so easily available like that. But what's funny is actually in Etsy's case, you know, we've tried using some of these things already. And what ends up happening is that we're always sort of bitten by the uniqueness of our items that, you know, we're not sending in, you know, simple photos and asking, like, is there a car in this image or not or whatever? You know, we're looking for really like nuanced analyses of, you know, what our images might contain or whatever. And so we haven't yet found ways to use those APIs effectively. I think in large heart, just because of the kind of specialization that's needed to sort of, you know, to work with Etsy's data generally. But then also to, you know, especially make improvements over what we're already doing. But so aside from that AI stuff that they offer, though, they do have some really great products, like BigQuery and Google Cloud Dataflow and Pub/Sub, which are all, you know, in the big data realm and they're all things that we're using pretty heavily now. And these things, I think, are really compelling, you know, systems in that, you know, they're both really technically powerful, like they all, you know, they all are, I think, based on Google's in-house tools, which are really impressive and which work really well. You know, and again, they get back to that sort of, like, punching above your weight class kind of analogy. Like we, you know, I don't know if there are any open source or other cloud provider tools that can rival these things, you know, and even if there were, certainly if there were an open source, we would have to figure out how to run them ourselves and we'd require more specialists to have access to them. We're here, we just kind of turn them on, it's great. >> Give us a sense. So you've talked about your background. So lots of background in, you know, big computational environment, lots of math. But, you know, you also mentioned, hey, hadn't necessarily worked for, you know, big sort of software environment before. You've got a smaller team. You now have access to these really powerful tools and services. Like, what's your interaction with Google Cloud in terms of going like, hey, how can we best use your tools? Like, what's the thing that's gonna make you the most successful? How do they, how do you communicate with them? How do they kind of give you guidance? Or like, how does that relationship work? >> Sure, yeah, that's a great question. I mean, we have a pretty high level, high tier of support with them. So we actually have some, some Google, you know, I don't know what their titles are, but we have some Google people who are like, you know, some technical people, some not as technical, but who are kind of embedded in our company at this point. So we, I can send Slack messages to people and ask them for things. And so that's kind of like the steady state that mode of operation right now. And they are, I think, their fewer, you know, day-to-day interactions like the ones you described, but earlier on when we were kind of planning out the migration and thinking through what we were doing, we had a lot of meetings with a lot of Google's, you know, applications engineers or, again, I don't know what their titles were, but, you know, people who were able to sort of look at our architecture designs, give us feedback on, you know, all sorts of aspects of them. I mean, personally, I've learned some, you know, just general architectural principles from some of these people in addition to, you know, best practices on Google. And, you know, and we did take quite a number of their suggestions from, you know, in order to make sure that what we were doing would have the highest probability of success while we were doing it. And it's mostly worked out. So yeah, we, now that things are pretty stable, we don't interact so much generally, but we can if we want to, they're right there on Slack. - Yeah, and, you know, I don't know if this is necessarily something that your team does. Obviously, you know, you've been part of the, kind of, re-platforming of the data platform. You've been part of the migration. You know, at some point, the folks, you know, sort of the executive staff, the business side of Etsy goes, you know, we're making a big investment in this. We're taking on the risk of going through these changes. How do they, how do you communicate back to them, you know, progress, success, that things are getting better? Like, what's the, I don't know, what's the metrics you use? What's the language that you use to kind of communicate how this data platform's evolving? - Yeah, that's a good question. So, you know, I think during the migration, I mean, this is like a lesson learned for me. I like, so during the migration, I think that there was a lot of, there was a lot of, there were always a lot of unknown unknowns, you know? You know, I mean, despite the fact that, you know, the cloud was great, Google support was great. We met with their data proc people all the time who were the people behind managed to do service. You know, still, we would kind of continually run into, you know, use cases that weren't covered or whatever. And so I, we always found it kind of hard to project outward while working on the migration very far. So, you know, we were able to give them, say, a roughly six month window during which we thought things would be able to be moved. And, you know, that roughly ended up working out, but it certainly wasn't the degree of precision that I think they were hoping for. But I think now that we, now that things are more stable, there are a lot fewer unknown unknowns. You know, I think the types of things we can start to really quantify are things like our experiment velocity. You know, with this, with this big corporate focus on improving the marketplace and getting, you know, making it easier for people for buyers to find what they're looking for, you know, the only way we can really do that is by running lots of experiments on the site, be those, you know, things as simple as UI changes and to things as complicated as, you know, fundamentally redesigning machine learning models. And I think, you know, it's a good question how would we communicate the benefit that the cloud has given us in a way? Because there are certain things that we just couldn't even do on premise that we're now able to do really easily in the cloud, which contribute to an enhanced experiment velocity. And so, you know, I don't know whether the business site was tracking sort of like experiment release rates or they were certainly tracking like GMS growth rates and things like that. And I imagine over time, you know, hopefully all of these rates, if tracked, will, you know, will show an upward trend by running in the cloud. But yeah, that's a really good question. And it's not something that like I necessarily concretely interact with so much on a day-to-day basis. - Well, and I think you, I think you really, you kind of hit the nail on the head, it's, it's, you know, migrations always, they're hard to scope. You know, people always think they're going to be easier than they think they are. And I mean, I think in general people are still trying to figure out, you know, what's the, what's the, you know, whether you're, you call what you're doing, a transformation, a digital transformation of whatever it is. Like, trying to, trying to communicate to people like, I need you to be somewhat patient 'cause we're learning, we're making changes, you know, like there isn't sort of like one language yet. You know, a lot of that I think is kind of, it depends on the situation. So I was curious if you would figure it out. I think what you went through is exactly what we've seen from a lot of people, which is, you know, we did the best, we could, we didn't necessarily always know what the right thing was, but, you know, we, we learned it as we went along. - Yeah, part of the experience makes me wonder whether, you know, another lesson learned might be that, you know, like often it seems like people choose either lift and shift, or they, I mean, I don't know how often people choose to do what we did, which was basically remake a new platform that, you know, can support everything the old platform did and move things on to it. But in thinking it through, I wonder if we would have a little bit more by looking at something sort of in the middle and asking, "What kind improvements can we make before this migration just sort of set us up for you know quicker success during it but no I don't know it's not something that you do every day so yeah hindsight's always 2020 yeah air before air before we wrap it up Kevin let me ask you one last thing before we go and we'll kind of wrap it up do you have a do you have a favorite item or favorite seller on Etsy I mean it's you know obviously you can't be too biased but do you have something that you're just like man we saw this thing pop up and it's just really cool so in terms of a single thing I'm not sure but I'm kind of a sucker I've become a sucker for mechanical watches you know I just I think the part of me that's like fully embedded in tech every day like all this virtual stuff kind of has a really I'm drawn to these like super super complicated things that were made you know a hundred years ago that are still ticking accurately today and we have a ton of great sellers you know a number of whom you know I've got favorite data and I see their updates and stuff who are just always finding cool items that you know somebody thought was a funky idea in the 70s or whatever and turned into a really cool mechanical watch so that that's what that's my personal Etsy go to but yeah we have tons of cool stuff I recommend everybody checking it out that's awesome very cool yeah so Kevin where can everyone find out more about you you know of course I think at this point everyone's gonna know the website and we're gonna have links but tell everyone a little bit in closing how it how they can find out more about you about me personally yeah sure so I mean you know I have my LinkedIn I don't know how many Kevin MacKale's around LinkedIn is a pretty popular name but I've also got my my GitHub MKLK and go few maybe a few somewhat interesting things on there I don't know we've got an open source project that we release as part of this whole effort so people can maybe check that out and we have we'll have a link to the engineering blog as well it's got it is called code as craft and you had mentioned it earlier but it is actually the name of the blog as well so fantastic well Kevin thank you very much for your time it was it was great to get your insight this week and on behalf of Brian and myself thank you again and thanks to everyone for listening and we'll talk to everyone next week thank you for listening to the cloudcast please visit the cloudcast.net to find more shows show notes videos and everything social media

Podcast Summary

Key Points:

  1. GitHub is expanding its capabilities by introducing CI/CD features in GitHub Actions, signaling a shift toward becoming a developer-centric "cloud within a cloud."
  2. Etsy migrated its data platform from on-premises Hadoop clusters to Google Cloud over a year and a half, prioritizing decoupling, scalability, and data integrity over a simple lift-and-shift approach.
  3. The migration was executed in stages, with tightly sandboxed environments ensuring data safety and minimizing risks during the transition.
  4. A major lesson learned was that cloud costs exceeded initial expectations, creating a strong incentive to optimize jobs for efficiency and reduce expenses.
  5. Google Cloud’s services like BigQuery, Dataflow, Pub/Sub, and Kubernetes provided significant technical and business advantages, enabling faster experimentation and broader team autonomy.
  6. Etsy’s data science and ML teams now leverage cloud tools to build complex, scalable systems—like a streaming Lambda architecture—without needing deep infrastructure expertise.
  7. While Google Cloud’s AI APIs (like Vision) are powerful, Etsy's unique item data requires deep customization, limiting their immediate effectiveness.
  8. Close collaboration with Google Cloud engineers through meetings and Slack support helped shape the migration strategy and improve long-term platform stability.

Summary:

Etsy’s cloud migration journey highlights a strategic shift from on-premises Hadoop to Google Cloud, driven by the need for scalability, flexibility, and team autonomy. The process was carefully phased, using isolated sandboxes to ensure data integrity and prevent disruptions during the transition. Despite initial cost overruns, the move unlocked significant operational efficiencies, enabling teams to build complex data and machine learning systems more rapidly.

Key benefits include auto-scaling with Kubernetes, streamlined access to BigQuery and Dataflow, and reduced reliance on specialized infrastructure teams. While Google Cloud’s AI tools are impressive, Etsy’s unique marketplace data demands custom solutions. The migration also reinforced that cloud transitions are inherently complex and unpredictable, requiring patience and adaptive planning.

Ultimately, Etsy now enjoys faster experiment velocity, improved system resilience, and a more agile data platform—proving that thoughtful, incremental cloud adoption delivers long-term value. The episode underscores broader industry trends: cloud platforms are evolving beyond just infrastructure to become developer and machine learning ecosystems, with companies increasingly prioritizing platform evolution over simple migration.

FAQs

Moving to the cloud enables elastic scaling, unbounded storage, improved fault tolerance, and decoupling of teams to avoid system-wide failures. It also allows for greater flexibility and faster innovation in data processing and machine learning workflows.

Etsy used a phased, sandboxed approach where jobs were tested in isolated, secure environments. This ensured data integrity and prevented disruptions to on-premises systems, allowing gradual migration of processing pipelines without risk.

Cloud costs initially exceeded expectations due to the lack of resource constraints on-premises. This led to a stronger focus on job optimization and efficiency, turning cloud costs into a direct business driver for performance improvements.

Etsy uses BigQuery, Cloud Dataflow, and Pub/Sub for data processing and analytics. These services offer technical power, ease of use, and integration, enabling teams to build complex data pipelines without deep infrastructure expertise.

The cloud enables faster experiment velocity, auto-scaling, and access to advanced data processing tools. This allows Etsy to test and deploy machine learning models more frequently, improving search relevance and discovery for users.

Success is measured through increased experiment velocity, faster model deployment, and improved user engagement metrics like search conversion and marketplace growth rates, which are tracked over time.

Chat with AI

Loading...

Pro features

Go deeper with this episode

Unlock creator-grade tools that turn any transcript into show notes and subtitle files.