Etsy’s cloud migration journey highlights a strategic shift from on-premises Hadoop to Google Cloud, driven by the need for scalability, flexibility, and team autonomy. The process was carefully phased, using isolated sandboxes to ensure data integrity and prevent disruptions during the transition. Despite initial cost overruns, the move unlocked significant operational efficiencies, enabling teams to build complex data and machine learning systems more rapidly. Key benefits include auto-scaling with Kubernetes, streamlined access to BigQuery and Dataflow, and reduced reliance on specialized infrastructure teams. While Google Cloud’s AI tools are impressive, Etsy’s unique marketplace data demands custom solutions. The migration also reinforced that cloud transitions are inherently complex and unpredictable, requiring patience and adaptive planning. Ultimately, Etsy now enjoys faster experiment velocity, improved system resilience, and a more agile data platform—proving that thoughtful, incremental cloud adoption delivers long-term value. The episode underscores broader industry trends: cloud platforms are evolving beyond just infrastructure to become developer and machine learning ecosystems, with companies increasingly prioritizing platform evolution over simple migration.
Today's show is sponsored by DigitalOcean.
DigitalOcean offers the simplest, most
developer-friendly cloud platform.
It's optimized to make managing and scaling apps easy
within two out of API, multiple storage options,
integrated firewalls and load balancers,
a new managed Kubernetes service, and much more.
From predictable pricing, to flexible configurations,
to world-class customer support,
you'll get access to all the infrastructure services
you need to grow your business.
Plus, DigitalOcean's community provides over 2,000 tutorials
to help you stay up to date on the latest open source software,
languages, and frameworks.
So to get started on DigitalOcean for free,
with a free $50 credit, go to dio.co/cloudcast.
That's dio.co/cloudcast.
Cloudcast Media presents from the massive studios
in Raleigh, North Carolina.
This is The Cloudcast with Aaron Delb and Brian Graceley,
bringing you the best of cloud computing
from around the world.
Good morning, good evening wherever you are.
Welcome back to The Cloudcast.
We are coming to you live from the massive cloudcast
studios here in Raleigh, North Carolina.
Kind of a slow week in terms of cloud news of the week.
I suspect that is because we are, in August,
it is summer vacation for a lot of folks,
and probably particularly the venture capitalist
and the M&A folks that are probably
summering in Nantucket or Martha's Vineyard
or somewhere in the Caribbean or somewhere in the Mediterranean
or something, but we'll give those folks
a few weeks off, a little bit slow news week.
There was one interesting story that I noticed this week.
Not a lot of acquisitions, not a lot of big announcements,
but kind of a smaller announcement
that I think will probably end up
becoming a little bit bigger over time.
GitHub announced that they are now offering CICD
as a capability or a feature for GitHub actions.
And this is sort of important because I think
it's one of those things where GitHub is still trying
to remain as neutral as possible.
They are beginning to slowly do more integration
with Azure.
And I think as we see people beginning to make some decisions
about our things like serverless or functions
or just code-based, low-code, no-code types of things.
Becoming bigger, we are seeing slowly GitHub
is becoming not only just a place to store code,
but a place where actually actionable things happen
around that code, and not just collaboration,
but things like actions, which are sort of
a serverless type of function.
We're now seeing CI kind of slipping in there.
And I think we'll begin to see over time
that GitHub becomes a more important sort of cloud
within a cloud, right?
So we will often think about Google Azure and AWS
as the three major clouds.
I think we're going to begin to see GitHub really begin
to emerge as sort of more of a developer cloud.
So that'll be an interesting thing to watch over time.
Definitely, they sort of, I don't want to say they slow-rolled it.
It wasn't a massive, massive announcement,
but for those kind of in the know, it kind of went out there.
And I think it's something for us to keep an eye on.
We'll do something we'll dig into in a later show.
One other thing I thought was interesting
and I'll put it in the show notes.
Gartner released their low-code, no-code magic quadrant.
And this is an area that's always been sort of interesting to me.
I think, as we talk about,
software eating the world, developers being king makers.
At some point, the world can't just operate
on the developers, the highest end of developers,
the ones that make hundreds of thousands of dollars,
millions of dollars, it's got to be sort of democratized
a little bit more.
You know, some of the low-code, no-code platforms
don't necessarily get as much hype.
But I think it's an area that we will see
over time become more and more popular
as data becomes more readily available.
Development skills are sort of taught in colleges
and elementary schools and high schools and so forth.
And some of these low-code, no-code tools become more prevalent.
You know, we talked at one point about AWS planning
to roll out a low-code, no-code option at some point.
Maybe they'll come out at re-invent.
But I'll put that in the show notes.
We're going to link to the Gartner thing.
I know folks had different opinions on Gartner,
but it is always kind of interesting
to kind of get some perspective on,
you know, who are the companies that are in general leading
and also some of the up and comers in that space?
So that's it.
Gonna wrap it up for cloud news the week.
Again, kind of mentioned as a slow news week.
We do have a very interesting show for you today.
We're gonna talk with the folks from Etsy
about their cloud migration
from a lot of their data science,
centric services from being on-prem into the cloud
and how they went about doing that with Google Cloud.
So look forward to that interview and we will get to that next.
Today's episode is sponsored by DataDog,
the cloud scale monitoring service
that provides comprehensive visibility
into public cloud, hybrid cloud, multi-cloud environments
with over 250 integrations.
DataDog unifies your metrics, logs,
and distributed request traces into one platform.
So you can investigate and troubleshoot issues
across every layer of your stack.
Use DataDog's rich, customizable dashboards,
and algorithmic alerts to monitor cloud migrations
in real time.
To start a free trial today and DataDog sending you
a free t-shirt, visit dataDogHQ.com/cloudcast
to get started.
That's dataDogHQ.com/cloudcast.
And we're back.
Aaron, it is good to have you back.
How you been, man?
Been good, been good.
How about yourself?
Doing well, you know?
Just got back from some of the summer break stuff.
So summer is in full effect
and the heat is overwhelming at this point.
Yeah, no kidding, absolutely.
So, you know, we, I think we're always very lucky
to get a lot of the guests that we have on our show.
And a lot of times we get, you know, technology vendors
and they're talking about new technology.
And not enough time do we get with some folks
that are not only kind of on the building side of things.
They're living and breathing the technology.
But, you know, we get a chance to talk to some of the companies
that just really shape the internet.
We're incredibly lucky.
So, you know, today we get a chance to do one of those things.
So very excited to have Kevin McCale,
who is one of the senior staff engineers at Etsy,
especially on the data team, Kevin, welcome to the show.
Hi, yeah, thanks a lot.
Thanks for having me.
It's a pretty intimidating introduction.
I hope I can live up.
No, we've got it.
Oh, it's, it'll be great.
We're looking forward to it.
You know, so before we dive into some of the work
that you're doing at Etsy, you know,
we look at your background.
You've done a lot of things that have kind of nothing to do
with online retail.
And you seem to be kind of incredibly good at math.
So give, give folks a little bit of your background.
Just some of the things that you've worked on,
kind of some of the scale of things
that you worked on even before coming to Etsy.
Yeah, sure.
So, you know, I don't know if I'd say I'm good at math,
but I certainly studied it long enough to know
how much I don't know.
You know, my undergrad training was basically a math.
I did a PhD in experimental physics.
All of which taught me, you know, a lot about things
that might not necessarily seem relevant every day
to e-commerce, but that I almost feel like I do look back
on a lot every day.
I mean, just a lot of basic ways of thinking
about things and solving problems and project management
and stuff like that have all been really important to me.
But, you know, obviously, really useful
and things that I picked up during that period of time.
But since then, I've kind of moved into the tech industry
kind of slowly starting as an applied mathematician
working for the US government
and then kind of gradually getting drawn into data science
and while doing that,
learning a lot more about software engineering
and ultimately kind of culminating in my current position,
which was the first time I've actually been formerly a software
engineer and not a data scientist or whatever.
- Yeah, so it makes sense 'cause it's so fascinating
to hear, we're hearing more and more about backgrounds
like that or backgrounds that maybe are less traditional
making into these roles.
And then you get to be involved in some really, really cool
in interesting projects, right?
So there's two very interesting projects at Etsy,
both kind of building evolving the data platform, if you will.
And then helping to manage this migration
to Google Cloud platform that we're going to talk about
a good bit probably.
So let's start by talking about the data platform, though.
First, tell everyone a little bit about the evolution
behind it and then the process
and what happened during all of that.
- Sure, yeah, so, you know, our data platform,
well, so, you know, so Etsy, first of all,
is a two-sided marketplace for vintage and handmade goods.
We've got, yeah, we are kind of making an assumption
there aren't we?
(laughing)
- You know, it's good to just sort of set the tone
a little bit.
You know, so we have something like 50 or 60 million items
on our site, all of which are, or most of which are unique.
Like these aren't things out of a catalog,
like an iPhone or something.
These are all handmade items or vintage items.
And we've got, you know, millions of sellers,
millions of buyers who are shopping on the site
and trying to sift through this giant mass of stuff
to find what it is that they're looking for for, you know,
for whatever the special, you know, sort of gift
or whatever it is that they're trying to find
that they will enjoy themselves or give to people or whatever.
So you can imagine like connecting people with these items,
helping them find what they're looking for
is a big source of machine learning problems for us.
So we've basically over the past couple of years
at the identified machine learning as one of the core
kind of company wide initiatives
to help try to improve that search and discovery process.
And so in order to do this,
we need to collect a whole lot of data.
So basically every action that a buyer takes on our site
will generate information that we collect on the backend.
So, you know, things like clicking things, viewing things,
favoriting things, searching for things.
All of that is collected.
And we generate over a terabyte of data a day.
And we need to process that in order to, you know,
produce better machine learning models
or better recommendations and add placement.
We have all sorts of in-house machine learning products now
that are on the site and that are powered by this platform.
And so now getting, you know,
some more specifically to the platform itself,
we're pretty reliant on Hadoop right now.
And specifically we use Scolding,
which is a scholar wrapper on top of Hadoop.
And we have something like, I don't know, hundreds of workflows, thousands of jobs that
are running daily and processing through all of this stuff.
And you know, going back a few years, we were doing all of this on-prem in a data center
that Etsy was paying for and we had hardware that we were sort of internally managing.
And as we identified this machine learning priority and the volumes of data that we were
collecting started going up and up and up and our in-house resources and in-house people
just were not enough to keep up with this growing demand.
And so that kind of inspired us to start looking at the cloud as one, you know, Etsy generally
was looking at a cloud migration for other reasons too, but for the data processing side
of things, you know, this was a big motivator that we basically were just kind of at capacity
or even above capacity and we didn't have enough people or resources to catch up with
where we already were.
So now, you know, we basically over the past, say, year and a half have spent a lot of
time migrating all of this stuff that we had, all of these Hadoop jobs and all sorts of
others kind of scheduled operations and databases and stuff on the data platform into the cloud.
And in order to do this effectively, we had to kind of rethink the way things were constructed
because if we had just kind of lifted and shifted a big Hadoop cluster and turned it into
a bigger Hadoop cluster in the cloud, we really wouldn't be, you know, we really wouldn't
be gaining a lot of the benefits that are available to us by being in the cloud such as,
you know, elastic clusters that spin up and spin down as needed, things like, you know,
basically unbounded storage available through, you know, these big object stores that are
available in the cloud. And another thing that was also a big motivator was with our big
sort of single in-house cluster, everybody who was doing data science, which at this point,
I checked yesterday, over a quarter of our engineers have contributed something to our
data platform in the past year. So basically, everybody at the company or, you know, every
team at the company is affected by this platform and having everybody on a single cluster with
a single scheduler just created really tight coupling between teams and, you know, consequently
there were lots of problems when, you know, failures cascaded all throughout the system
or, you know, well, you can imagine how things can go wrong. And so by moving into the cloud,
we've also kind of focused on decoupling a lot of our systems so that, you know, individual
teams, like data science teams and the, you know, recommendation teams can all have their
own resources that operate independently of each other and, you know, and give them the
flexibility to do what they need to do while not imposing constraints on anybody else.
We've had a number of people on the show, you know, guests in the past who have talked
about, like, application migration. So it was, hey, you know, we had, we had front end
or back end, we migrated that into the cloud. And we haven't necessarily had anybody with
really deep expertise in sort of moving the data side of that into the cloud. Can you
give some insight? I mean, you're still probably in the process of doing that, but when you're
talking about that much information, moving it is not a trivial task. Can you talk, kind
of, talk about how you get it there, how you sort of do it in stages so you can keep
using it in day-to-day operations? Kind of what goes on behind the scenes there?
Sure. Yeah. That was an interesting balancing act that took about a year and a half. Like,
basically, yeah, basically, we knew for a long time that we were going to have to develop
this new platform in the cloud. And that was going to have to coexist with stuff that
was on-premise for, however long it took to move everything. And with a lot of these
data processing things, there's, especially us as a platform team, we don't necessarily
know what's going on inside of all of these jobs. We don't necessarily know. We don't
have assurances that we can run them in some new environment without clobbering data
in some old environment or making API calls to things that we shouldn't be making API calls
to, et cetera. So we really spent a lot of time, and I think actually this goes back,
you know, I mentioned earlier that my time as a scientist taught me a lot of things that
are useful today. It certainly taught me like paranoia about data quality and data integrity
and stuff. Paranoids may be a strong word, but I think that, you know, we treated this
migration process as one in which basically none of these jobs could be trusted. And we
needed the ability to run any of them, to test any of them in the cloud, but none of them
could be trusted to not break anything on-prem. So we basically created our cloud environment
with really, really tight boundaries everywhere. And so, you know, we can run individual jobs
in really tightly sandboxed environments and gate whatever comes in and out of those environments
strictly through things like network bounds or firewalls or whatever. And kind of gradually
over time, we built up this system wherein, you know, we had processing happening on-premise.
And generally, the on-premise stuff was, you know, always the highest level pieces of
the pipeline. So like big data ingest jobs and big common share jobs that are used by
everybody all across the platform. Those things remained on-premise for a really long time.
And we would copy data up from on-premise into, you know, we're using Google Cloud. So
into their cloud storage platform, you know, bit by bit. And then these sort of, these
little sandboxes in the cloud would kind of run an isolation process that data right there
output out. You know, we'd look for errors, see what they're trying to do and figure out
what sorts of patterns we need to sort of, like, re-enable in these cloud environments.
Kind of gradually over time by just taking this really, really, strictly, you know, bounded
approach, we kind of just chipped away at the stuff that was kind of at the lowest levels
of our processing pipelines and kind of moved stuff upward. And, you know, as of, I think
two or three months ago, our data platform team finally finished moving the last jobs over.
We, you know, turned off our on-prem clusters and we're fully in the cloud right now.
Interesting. Very cool. Now, let me ask you this kind of a follow-on to that because there
was certainly an analysis, if you will, of, okay, like you mentioned data integrity. I actually
like that idea of paranoia. But the, the, that is certainly a big lesson learned and something
you kept in mind very thoughtfully throughout. What are some of the other kind of lesson learned
that maybe you didn't anticipate or didn't expect that maybe, you know, because you always hit
these bumps, right? In some, what are some of the advice of some things to look out for for others
that are considering migration such as this? Sure. Yeah. I mean, I think, I think probably the most
obvious one that I think everyone in the company now is aware of is that everything cost more than we
thought it would. In the, you know, in the public cloud? In the public cloud. Yeah. We, you know,
obviously, you know, it's difficult to sort of anticipate or sort of translate what our jobs
were doing on-prem with, with what that will mean for, for our cloud costs in the future. And so,
you know, we performed all this migration primarily focused on just getting things into the cloud
and, you know, sort of figuring that once the jobs themselves were running in the cloud, we can do
whatever optimizations were necessary. But, but it definitely turned out to be true that, you know,
that there was a little bit of sticker shock. Just primarily stemming from the fact that I think,
you know, when we were in an on-prem environment, there was never really a forcing function to
to encourage us to try to optimize these jobs and to make them more efficient. You know,
it was just, you know, the forcing function would only be you get more failure emails or whatever
if you're running out of resources. Whereas now the cloud will happily run whatever it is that we
tell it to do. And, you know, that's great in some ways, but then in other ways, you know, sometimes
you tell it to do something that maybe is a little bit too pricey. So I think that, that was probably
the biggest maybe lesson learned that the things are going to be expensive. But I also view it as
an opportunity in that, you know, it now does give us a real business case for spending time to
optimize these things and make them more efficient because we can, you know, directly impact Etsy's
overall financial numbers if we can, you know, take a big chunk out of our costs by building more
efficiency. Oh, that makes perfect sense, yeah. Now, kind of a follow-on to that with it, you know,
you mentioned you didn't want to do kind of a lift and shift and all of this. And so in going to
Google Cloud or just really honestly, public cloud in general, right, what are some of the advantages
like it is, it is funny, you mentioned that that cost was a bit of a surprise because in the early
days of cloud, everyone was kind of like, oh, you know, move it there because it'll be cheaper and then
kind of everyone kind of learned that lesson over time, right? So, so what is some of the
advantages you've seen? It's, you know, there's obviously like kind of the obvious one is the
acceleration of some of those work in greater velocity and flexibility, but, but tell us a little
bit about, you know, some of the big advantages that that happened there from both like business
and technical, as this, you know, the data platform has evolved. Yeah, that's a, that's a great
question. I think that, you know, I would have to say that, I mean, you know, maybe this is obvious,
you guys are the cloudcast, you guys probably, this is obvious to everybody, but you know, for us,
moving to manage services has been a huge win. You know, we, I mentioned earlier, you know,
we have these big hudu clusters on premise that were, you know, we're reaching a scale that we
just didn't have the expertise to handle, I think. We did have to hire contractors occasionally
to help us out with things. And now basically by being in the cloud, you know, Google is our,
our ops team, like Google are the experts on how to configure these things. And so we don't need to
have, you know, as specialized, as like, you know, single function specialized teams anymore,
we can have teams that are, you know, sort of broader in scope and, you know, take on a bigger
mandate in terms of what it is that they're doing for the product without having to like, you know,
hike or specialize in, you know, hudu hardware configurations or whatever. So, so that I think,
you know, I've given the hudu example, but that has been a sort of a broad win all across the
Thank you.
entire Etsy platform migration.
Our search teams are using Kubernetes in the cloud.
And this gives them all sorts of advantages,
like self-healing and things like that.
And it's auto-scaling that they didn't really
have access to on-prem.
And it's also, aside from broadening the mandate
by using these things, let's sort of punch above our weight
class or whatever the analogy is.
I have a team that I'm working on right now.
So I'm working on the machine learning team specifically
on a project that's building out a sort of streaming
Lambda architecture.
I was kind of going through, while thinking about things
I would say today, going through all of the services
that we're using in Google Cloud right now
and thinking through what it would take for us
to manage these things where their open source counterparts
ourselves, and it would just be impossible.
So our team is four people.
And we're using big table at a massive scale,
which is a giant nose-equal database.
We're using all sorts of streaming processing engines, all sorts
of other analytics databases.
And these are all things that we just create
by writing a few lines of infrastructure code instead
of having to research it and figure out
what it takes to configure it and make them all work together.
So I think that now that our migration is complete,
and we're really able to start focusing on building
new applications and building new logic
on top of this layer, I really feel like the dividends
are finally starting to become just really, really evidence
in that we can use all these great technologies without having
to have all the specialization needed to run them.
Yeah, it's an interesting point.
If I kind of, you know, correlate a little bit of what
you just said, you said you said the beginning
somewhere in the neighborhood of 50 million items on there
and then a staff of four.
So that's a hell of a summary, if you will.
I'll ask you a quick follow-up on that.
So it's not often I get to talk to somebody
who's both familiar with Google Cloud
and has a data science background.
And you know, one of the big things we've talked about
on this show a number of times over the years
is Google Cloud and kind of each cloud
has their kind of specialty in their services.
And so, you know, can you give us some kind of thoughts
on like the data science, AI, ML aspects of Google Cloud
that tends to be one of the things they hang their hat on,
at least, you know, publicly at like the Google next shows
and some of these other things.
You know, do you have any kind of opinion?
We hadn't really covered up until now.
So it's kind of off the cuff question,
but I just wondering on that.
Sure, yeah, so I do have opinions.
Actually, it's interesting, you know,
so I'm sort of familiar with some of the products
that they offer like they have their Vision API,
which is a really awesome idea, you know,
you just send it an image and it will send you back tags
that it has applied to that image
and you don't have to train any models or anything.
And it's a great way like off the shelf
to just kind of try something out
and see if it's going to work.
And it's, you know, that and other things like it
are really, really cool.
And you know, I don't know of any other company
that makes quite as many things
just so easily available like that.
But what's funny is actually in Etsy's case,
you know, we've tried using some of these things already.
And what ends up happening is that we're always sort of bitten
by the uniqueness of our items
that, you know, we're not sending in, you know,
simple photos and asking, like,
is there a car in this image or not or whatever?
You know, we're looking for really like nuanced analyses
of, you know, what our images might contain or whatever.
And so we haven't yet found ways to use those APIs effectively.
I think in large heart, just because of the kind of specialization
that's needed to sort of, you know,
to work with Etsy's data generally.
But then also to, you know,
especially make improvements over what we're already doing.
But so aside from that AI stuff that they offer, though,
they do have some really great products,
like BigQuery and Google Cloud Dataflow and Pub/Sub,
which are all, you know, in the big data realm
and they're all things that we're using pretty heavily now.
And these things, I think, are really compelling, you know,
systems in that, you know, they're both really technically
powerful, like they all, you know,
they all are, I think, based on Google's in-house tools,
which are really impressive and which work really well.
You know, and again, they get back to that sort of, like,
punching above your weight class kind of analogy.
Like we, you know, I don't know if there
are any open source or other cloud provider tools
that can rival these things, you know,
and even if there were, certainly if there were an open source,
we would have to figure out how to run them ourselves
and we'd require more specialists to have access to them.
We're here, we just kind of turn them on, it's great.
>> Give us a sense.
So you've talked about your background.
So lots of background in, you know,
big computational environment, lots of math.
But, you know, you also mentioned,
hey, hadn't necessarily worked for, you know,
big sort of software environment before.
You've got a smaller team.
You now have access to these really powerful tools and services.
Like, what's your interaction with Google Cloud
in terms of going like, hey, how can we best use your tools?
Like, what's the thing that's gonna make you the most successful?
How do they, how do you communicate with them?
How do they kind of give you guidance?
Or like, how does that relationship work?
>> Sure, yeah, that's a great question.
I mean, we have a pretty high level,
high tier of support with them.
So we actually have some, some Google, you know,
I don't know what their titles are,
but we have some Google people who are like, you know,
some technical people, some not as technical,
but who are kind of embedded in our company at this point.
So we, I can send Slack messages to people
and ask them for things.
And so that's kind of like the steady state
that mode of operation right now.
And they are, I think, their fewer, you know,
day-to-day interactions like the ones you described,
but earlier on when we were kind of
planning out the migration and thinking through
what we were doing, we had a lot of meetings with a lot of
Google's, you know, applications engineers or, again,
I don't know what their titles were, but, you know,
people who were able to sort of look at our architecture designs,
give us feedback on, you know, all sorts of aspects of them.
I mean, personally, I've learned some, you know,
just general architectural principles
from some of these people in addition to, you know,
best practices on Google.
And, you know, and we did take quite a number of their
suggestions from, you know, in order to make sure that
what we were doing would have the highest probability
of success while we were doing it.
And it's mostly worked out.
So yeah, we, now that things are pretty stable,
we don't interact so much generally,
but we can if we want to, they're right there on Slack.
- Yeah, and, you know, I don't know if this is necessarily
something that your team does.
Obviously, you know, you've been part of the,
kind of, re-platforming of the data platform.
You've been part of the migration.
You know, at some point, the folks, you know,
sort of the executive staff, the business side of Etsy
goes, you know, we're making a big investment in this.
We're taking on the risk of going through these changes.
How do they, how do you communicate back to them,
you know, progress, success, that things are getting better?
Like, what's the, I don't know, what's the metrics you use?
What's the language that you use to kind of communicate
how this data platform's evolving?
- Yeah, that's a good question.
So, you know, I think during the migration,
I mean, this is like a lesson learned for me.
I like, so during the migration,
I think that there was a lot of, there was a lot of,
there were always a lot of unknown unknowns, you know?
You know, I mean, despite the fact that, you know,
the cloud was great, Google support was great.
We met with their data proc people all the time
who were the people behind managed to do service.
You know, still, we would kind of continually run into,
you know, use cases that weren't covered or whatever.
And so I, we always found it kind of hard
to project outward while working on the migration very far.
So, you know, we were able to give them, say,
a roughly six month window during which we thought things
would be able to be moved.
And, you know, that roughly ended up working out,
but it certainly wasn't the degree of precision
that I think they were hoping for.
But I think now that we, now that things are more stable,
there are a lot fewer unknown unknowns.
You know, I think the types of things we can start
to really quantify are things like our experiment velocity.
You know, with this, with this big corporate focus
on improving the marketplace and getting, you know,
making it easier for people for buyers to find what they're
looking for, you know, the only way we can really do that
is by running lots of experiments on the site,
be those, you know, things as simple as UI changes
and to things as complicated as, you know, fundamentally
redesigning machine learning models.
And I think, you know, it's a good question
how would we communicate the benefit
that the cloud has given us in a way?
Because there are certain things
that we just couldn't even do on premise
that we're now able to do really easily in the cloud,
which contribute to an enhanced experiment velocity.
And so, you know, I don't know whether the business
site was tracking sort of like experiment release rates
or they were certainly tracking like GMS growth rates
and things like that.
And I imagine over time, you know, hopefully all
of these rates, if tracked, will, you know,
will show an upward trend by running in the cloud.
But yeah, that's a really good question.
And it's not something that like I necessarily
concretely interact with so much on a day-to-day basis.
- Well, and I think you, I think you really,
you kind of hit the nail on the head, it's, it's, you know,
migrations always, they're hard to scope.
You know, people always think they're going to be easier
than they think they are.
And I mean, I think in general people are still trying
to figure out, you know, what's the, what's the,
you know, whether you're, you call what you're doing,
a transformation, a digital transformation of whatever it is.
Like, trying to, trying to communicate to people like,
I need you to be somewhat patient
'cause we're learning, we're making changes, you know,
like there isn't sort of like one language yet.
You know, a lot of that I think is kind of,
it depends on the situation.
So I was curious if you would figure it out.
I think what you went through is exactly what we've seen
from a lot of people, which is, you know, we did the best,
we could, we didn't necessarily always know
what the right thing was, but, you know, we,
we learned it as we went along.
- Yeah, part of the experience makes me wonder
whether, you know, another lesson learned might be that,
you know, like often it seems like people choose either
lift and shift, or they, I mean, I don't know how often
people choose to do what we did, which was basically
remake a new platform that, you know,
can support everything the old platform did
and move things on to it.
But in thinking it through, I wonder if we would have
a little bit more by looking at something sort of in the middle and asking, "What kind
improvements can we make before this migration just sort of set us up for you
know quicker success during it but no I don't know it's not something that you do
every day so yeah hindsight's always 2020 yeah air before air before we
wrap it up Kevin let me ask you one last thing before we go and we'll kind of
wrap it up do you have a do you have a favorite item or favorite seller on Etsy
I mean it's you know obviously you can't be too biased but do you have something
that you're just like man we saw this thing pop up and it's just really cool
so in terms of a single thing I'm not sure but I'm kind of a sucker I've
become a sucker for mechanical watches you know I just I think the part of
me that's like fully embedded in tech every day like all this virtual stuff
kind of has a really I'm drawn to these like super super complicated things
that were made you know a hundred years ago that are still ticking
accurately today and we have a ton of great sellers you know a number of
whom you know I've got favorite data and I see their updates and stuff who are
just always finding cool items that you know somebody thought was a funky idea
in the 70s or whatever and turned into a really cool mechanical watch so
that that's what that's my personal Etsy go to but yeah we have tons of cool
stuff I recommend everybody checking it out that's awesome very cool yeah so
Kevin where can everyone find out more about you you know of course I think at
this point everyone's gonna know the website and we're gonna have links but
tell everyone a little bit in closing how it how they can find out more about you
about me personally yeah sure so I mean you know I have my LinkedIn I don't know
how many Kevin MacKale's around LinkedIn is a pretty popular name but I've
also got my my GitHub MKLK and go few maybe a few somewhat interesting things
on there I don't know we've got an open source project that we release as
part of this whole effort so people can maybe check that out and we have we'll
have a link to the engineering blog as well it's got it is called code as
craft and you had mentioned it earlier but it is actually the name of the blog as
well so fantastic well Kevin thank you very much for your time it was it was
great to get your insight this week and on behalf of Brian and myself thank
you again and thanks to everyone for listening and we'll talk to everyone next
week thank you for listening to the cloudcast please visit the cloudcast.net to
find more shows show notes videos and everything social media
Podcast Summary
Key Points:
GitHub is expanding its capabilities by introducing CI/CD features in GitHub Actions, signaling a shift toward becoming a developer-centric "cloud within a cloud."
Etsy migrated its data platform from on-premises Hadoop clusters to Google Cloud over a year and a half, prioritizing decoupling, scalability, and data integrity over a simple lift-and-shift approach.
The migration was executed in stages, with tightly sandboxed environments ensuring data safety and minimizing risks during the transition.
A major lesson learned was that cloud costs exceeded initial expectations, creating a strong incentive to optimize jobs for efficiency and reduce expenses.
Google Cloud’s services like BigQuery, Dataflow, Pub/Sub, and Kubernetes provided significant technical and business advantages, enabling faster experimentation and broader team autonomy.
Etsy’s data science and ML teams now leverage cloud tools to build complex, scalable systems—like a streaming Lambda architecture—without needing deep infrastructure expertise.
While Google Cloud’s AI APIs (like Vision) are powerful, Etsy's unique item data requires deep customization, limiting their immediate effectiveness.
Close collaboration with Google Cloud engineers through meetings and Slack support helped shape the migration strategy and improve long-term platform stability.
Summary:
Etsy’s cloud migration journey highlights a strategic shift from on-premises Hadoop to Google Cloud, driven by the need for scalability, flexibility, and team autonomy. The process was carefully phased, using isolated sandboxes to ensure data integrity and prevent disruptions during the transition. Despite initial cost overruns, the move unlocked significant operational efficiencies, enabling teams to build complex data and machine learning systems more rapidly.
Key benefits include auto-scaling with Kubernetes, streamlined access to BigQuery and Dataflow, and reduced reliance on specialized infrastructure teams. While Google Cloud’s AI tools are impressive, Etsy’s unique marketplace data demands custom solutions. The migration also reinforced that cloud transitions are inherently complex and unpredictable, requiring patience and adaptive planning.
Ultimately, Etsy now enjoys faster experiment velocity, improved system resilience, and a more agile data platform—proving that thoughtful, incremental cloud adoption delivers long-term value. The episode underscores broader industry trends: cloud platforms are evolving beyond just infrastructure to become developer and machine learning ecosystems, with companies increasingly prioritizing platform evolution over simple migration.
FAQs
Moving to the cloud enables elastic scaling, unbounded storage, improved fault tolerance, and decoupling of teams to avoid system-wide failures. It also allows for greater flexibility and faster innovation in data processing and machine learning workflows.
Etsy used a phased, sandboxed approach where jobs were tested in isolated, secure environments. This ensured data integrity and prevented disruptions to on-premises systems, allowing gradual migration of processing pipelines without risk.
Cloud costs initially exceeded expectations due to the lack of resource constraints on-premises. This led to a stronger focus on job optimization and efficiency, turning cloud costs into a direct business driver for performance improvements.
Etsy uses BigQuery, Cloud Dataflow, and Pub/Sub for data processing and analytics. These services offer technical power, ease of use, and integration, enabling teams to build complex data pipelines without deep infrastructure expertise.
The cloud enables faster experiment velocity, auto-scaling, and access to advanced data processing tools. This allows Etsy to test and deploy machine learning models more frequently, improving search relevance and discovery for users.
Success is measured through increased experiment velocity, faster model deployment, and improved user engagement metrics like search conversion and marketplace growth rates, which are tracked over time.
Chat with AI
Loading...
Pro features
Go deeper with this episode
Unlock creator-grade tools that turn any transcript into show notes and subtitle files.