[Music] Atlin is the metadata hub for your data ecosystem. Instead of locking your metadata into a new silo, unleash its transformative potential with Atlin's active metadata capabilities. Push information about data freshness and quality to your business intelligence, automatically scale up and down your warehouse based on usage patterns, and let the bots answer those questions and slack so that the humans can focus on delivering real value. Go to dataengineeringpodcast.com/AtlinToday. That's ATLAN, to learn more about how Atlin's active metadata platform is helping pioneering data teams like Postman, Plad, WeWork and Unilever achieve extraordinary things with metadata. When you're ready to build your next pipeline or want to test out the projects you hear about on the show, you'll need somewhere to deploy it, so check out our friends at Linode. With their new managed database service, you can launch a production ready, mySQL, Postgres or MongoDB cluster in minutes, with automated backups, 40 gigabit connections from your application hosts and high throughput SSDs. Go to dataengineeringpodcast.com/Linode today and get a $100 credit to launch a database, create a Kubernetes cluster, or take advantage of all of their other services, and don't forget to thank them for their continued support of this show. Your host is Tobias Macy, and today I'm interviewing Joe Reese, and Matt Housley about their new book on the fundamentals of data engineering. So Joe, can you start by introducing yourself? Yeah, hi, I'm Joe Reese. I'm a recovering data scientist, co-founder of CEO of Turnery Data, author of fundamentals of data engineering. And Matt, how about yourself? I'm Matt Housley, CTO of Turnery Data, and Po-author of Fundamentals of Data Engineering. Also, recovering data scientist. And going back to you, Joe, do you remember I, Hey, for us, got started working in data? Yeah, I mean, I've always worked in data in some capacity or in others for over 20 years now. So I started by, I guess, pursuing the actuarial route, then moved into analytics, data science, and then data engineering. And Matt, how did you get started in data? You know, I would trace it back to my undergraduate degree in physics, and then actually getting a master's degree in physics. So I was doing a lot of experimental work back then, and so there was always a data component even back then, even though I wasn't using anything fancy in like a database, it tended to be a lot of excel and such. And then I took kind of a break and got a PhD in pure math, and then I came back as a data scientist later. And like Joe said, you know, recovering data scientist, meaning kind of recognize the need for data engineering to get my job done, and then just ended up becoming a data engineer that way. In terms of the topic at hand, can you just start by explaining what possessed you both to write such an ambitious book? When we surveyed the books on data engineering that were out there, we thought there were a lot of very useful titles, but they're typically geared towards very specific technologies, so maybe machine learning with a certain cloud platform or certain language, or the books were along the lines of designing data intensive applications, which is a fantastic book, one of my favorite books of all time. But we really felt like the books didn't provide a comprehensive ground level view of data engineering as it was practiced in the early 2020s. And so we really wanted to take a step back and understand, you know, what are the things that data engineers would need to know to get hired, to do their jobs, and to succeed as a data engineer today? I think we got a lot of inspiration from the data engineering community as well. I feel like the world of podcasting and like medium and sub stack posts is kind of way ahead of what exists in books in terms of defining what data engineering is all about. But at the same time, we felt like maybe it was time for someone to codify that in more of a formal way and just to gather those ideas and that information. And so for example, we took a lot of inspiration from Google Cloud Platform has their notion of the data lifecycle, and instead of just thinking about tools, or thinking about how data flows and why you care about data and what you're going to do with it. So I think that almost became a starting point for our ideas about the data engineering lifecycle plus again, a lot of ideas just from the data community. Plus it was also to answer a lot of questions that we'd get. Like how do I become a data engineer? What do I need to know to be a data engineer and so forth? And so you're at the book because everybody got angry when you said read designing data and sense of applications and they said, "I don't have three months to do that." That was definitely a response that we would get. Yeah, that book, I was still one of my favorite books in data, but it's definitely not the most approachable book. Yeah, it's a bit of a slog. Problem too, and I think Martin himself basically said this, is that you can read the whole book and you'll know a lot about designing distributed systems, but you're not necessarily going to know what it is to be a practicing data engineer, especially where we emphasize a lot of managed services. Like where you can use something off the shelf, don't reinvent the wheel. Go read Martin's books, he can understand how these systems work, but you're probably not going to be in most jobs, designing really complex systems day to day, in terms of like, you know, thousand nodes and managing all these nodes and that kind of thing. Absolutely. So as far as the objectives that you had, you said that you were tired of answering the same questions over and over again, but what are the overall goals that you had going into writing this book and how do you feel about your level of success in actually addressing those goals? The goals were really what is likely to be at least somewhat immutable in the next five to ten years. To actually invert the question, what were some of the goals we did not want with this book? We didn't want to write a book that was very transient, very ephemeral. So that would have been one of those data engineering with technology, X, Y, R, Z or Cloud Platform, ABC, right? So we really wanted to take a step back and assess what would be the things where Matt and I being incredibly lazy people wouldn't need to rewrite the book in a year or two years. What were the things that, if you were to pick up the book, say, you know, again, five, ten years from now, most of it would still be relevant, or still be useful. So I'm sure we have all those types of technology books, you know, in our libraries. And when I pick up those kinds of books and I reread them, there's a sense of gratitude that the author took the time to kind of take a step back and really understand the bigger picture that was happening in the field at the time that they wrote it. I think as well, there's this sense that, again, we intend this as a compliment to all the technical books, but it takes a long, long time to accumulate the knowledge of what it is to actually be a data engineer, especially if you start with one or two technologies. Like, wait, what's the big picture here? What are the goals? How do I actually be successful in this profession? And so that's the laziness idea. We hope that we've provided a shortcut to people that we've achieved that goal that's kind of scaffolding that they can plug technologies into and kind of accelerate their careers in that way. Yeah, that's exactly the framing that I was going to say is that this book kind of sets the groundwork for you to figure out what to plug into which of the pieces and what those slots are for you to even know what to start thinking about and what technologies to evaluate beyond just. Okay, I need to get data from A to B. I guess I'll write a bunch of bash scripts and Cron and hope that it works. I think in general, the problem with any really hot field that's trendy is that people tend to focus on the wrong things. And I think we've seen this happen in data science and then kind of the next iteration machine learning. And I think we saw it happen with the big data movement and subsequently with the modern notion of data engineering. Everyone wants to focus on like the shiny objects as opposed to the blue collar aspect of the job and the blue collar aspect of the job is what gets things done. It's not just grabbing the latest version of spark that's actually going to deliver quality data. It's understanding the whole life cycle. What it means to serve data and who your customers are and those kinds of questions. As far as the overall process of figuring out what are those evergreen topics, what are the foundational components that are necessary for somebody who is interested in working as a data engineer or at least understanding what it is the data engineers do. How did you go about figuring out what those pieces were and then boiling them down to their essence. This is one of the harder parts of writing the book. It took a lot of I think soul searching on our part to understand, okay, so in our day to day jobs working as data engineers. And knowing a lot of other data engineers, what were the subjects that they had most care about and not in a pandering sense. We wanted to write a book that would only appeal to certain nuances and areas and not others. But what were going to be the things that intersected with everybody's experiences as a data engineer. At the same time, we're looking at lots of data life cycle diagrams to see how is data life cycle described by various organizations. If you Google data life cycle and look at the number of images, there's countless of them. It's like you looked at the James Webb and saw how many galaxies are on the universe. It's about as many data life cycle diagrams exist. But there are certain commonalities to these. And the commonalities that we found were applicable to data engineering, which we're talking more about a bit. Those are the ones we kept. That represented the subject areas that we're probably going to be most relevant for this book. And I think a lot of it too was just again reading blog posts, listening to podcasts episodes by people were respected. And what we found at some point is there was this cultural shift happening in data engineering where people started talking a lot more about data ops and data management and data quality and data observability and like you know, other problems. And so those were the topics were like, okay, that's the stuff that really needs to go into this book because it's not only trendy, but it's actually trendy for a very compelling reason, which is this is the stuff that helps us to get the job done.
and that's what should go in the book. And there's also the interesting moving target of what the data engineer job role actually is because as data becomes more of a core consideration for more organizations, it starts to bleed into, okay, well, software engineers do data engineering or data engineers do some software engineering or data engineers are doing some of the analytical engineering or data science work. Curious how you thought about what those boundary conditions are for what do I actually need to put into this book and what the data engineer role really means and how much of that is going to stay true over the time span that you're hoping this book will last for. >> The way we thought about it was, the data engineers role done, I suppose, either properly or in a theoretical vacuum would be getting data from source systems, whether those are databases that reside in application, APIs or whatever other source system, making that data useful for downstream consumption by data scientists, analysts, maybe other processes like reverse ETL. And so really the data engineer, as we describe it today, sits in between those source systems and the downstream use and application of data. It's interesting seeing how this definition of data engineering as a practice has shifted over the years. This data engineering really wasn't a title until maybe a few years ago, right? But it always existed in some form or another, whether you're describing that as maybe even a software engineer, working with data or a BI engineer or a ETL developer or any number of other titles that are now sort of, kind of, a hodgepodge end with data engineer. So I mean, this practice has existed for quite a while actually, but data engineering as a practice, I think could really be boiled down to the steps to describe where you're getting data from source systems and making it useful for downstream use. I think in terms of defining our boundaries and our edges too, it was very helpful for us to think about roles that we were targeting. Basically say, this is the persona of our intended audience. Hopefully other people will find this book useful as well, but this is who we're really targeting. And I think a lot of data engineering books have tended to focus maybe on the fang audience to some extent, like people who work on various sophisticated systems companies with a lot of resources. And the problem with that is that that type of expertise is actually not necessarily very transferable out of that thing world. So for example, if you work at Google, you may just work on their colossus storage system. And that may be your job. You just fine tune that system for different applications, including big data, data engineering applications. So we define our target audience to be more the, let's say blue collar data engineer, who's not as interested in the low level fine grain details of tools, and much more interested in just getting the job done. And I think in terms of longevity, we hope that that role is here to stay, even if some of the details evolve over time. I think it's the first chapter. You address sort of what that evolution of data engineering and its responsibilities has gone through over the past few decades. And you say that you're focusing on what you term the data lifecycle engineer, which is a term that you mentioned earlier. And I'm curious if you can talk to what that means for somebody who's actually doing that work. And for somebody who is maybe working at those lower levels, how much of the topics that you're addressing in this book are really going to be relevant to them, or are worth knowing about as somebody who might be consuming the tools that you're building. So if you are the person who's responsible for keeping colossus up-to-date or adding new features to that, how much of the broader use case of data lifecycle engineering is necessary to have to be able to do your job effectively? I think there is utility there. And the utility is that it does help you to understand the big picture. And as you evolve in your career, and maybe move into leadership in other areas, hopefully this will be very useful to say, you know what, I'm no longer working on colossus. I'm now a lead engineer, and I'm supposed to define the direction of Google. How do I do that? And let's think about the big picture of how Google uses data and how they consume data and how they ingest data, as opposed to just one little slice of things. Yeah, I think there's applicability across companies with different sizes and data maturity. You say that you're working on a non-tech company, quote, unquote, right? Bigger companies are going to have teams dedicated to certain parts of the data lifecycle, right? So we're as a team that's in charge of managing the storage systems where all the data is stored or some aspect of it. Other teams are responsible for data pipelines. And we hit this-- hit on this in the book a lot. It's incumbent to know what your upstream and downholt stakeholders want, right? This isn't just a technology discussion. It's very much about team dynamics, about communication as well. Typically, when we see data projects fail, it's very seldom due to the technology. 90% of the time it has to do with the people and how they interact and communicate, or probably don't more to the point. So these are the things I would say was just as important as a technology to get across in the book, which were, I guess, empathy and ways to communicate and understanding the needs of, again, your upstream and downstream stakeholders, whether those are who work on source systems, whether that's the team that works on ingestion, storage, transformations, BI machine learning, and so forth. You really should understand what their job entails, what they want in order for you to do your job most effectively. And kind of back to your earlier question, kind of about how the definitions of data engineering and sort of who this book is for. There's this old trope that goes around about data scientists spending 80%, 90% of their time getting data, cleaning data, all the non-quote data science you stuff. The job of a data engineer at the end of the day really should be to, I guess, invert that percentage, right? So maybe the data scientist is spending 10% of their time getting a cleaning data, 90% of their time doing the kinds of work that they were professionally trying to do with this machine learning analytics, so forth. The data engineer should enable their stakeholders to do the best version of their jobs that they were hired to do. And so again, to bring it back to the question, just asked, data lifecycle engineer really does mean understanding the tools, practices, as well as the stakeholders involved in the data lifecycle and helping them do their job the best they can. Yeah, I mean, because of our backgrounds, maybe the stakeholders are actually your primary focus because you've done that job. And so you want people to be able to do that job as a data simulator. Yeah, and if you want to be really good at data engineering, you should go work as an analyst or a data scientist for a bit. Because it's only when you understand the outcomes and the outputs. So I think you really can work backwards and understand how best to get to those results. The other interesting part of working in the data engineering space is similar to the software engineer versus this admin space of the balance of rigor and strictness and repeatability with flexibility and iteration speed and ability to just get things done. And as you said, it's definitely necessary to have worked in both sides to figure out how do I manage that balance and how far in which direction to push things depending on what type of operating environment I'm in. Oh, for sure. Yeah, I mean, especially when you're doing stuff like sprint planning, for example, if your company happens to do that, knowing what's involved in the outcome, I cannot begin to tell you how important that is. Whereas if you just have a sort of passing notion or if you've ever talked to your stakeholders, how are you really going to know if you're best serving them or if they're getting the best outcomes that they should be getting? So as to the specific audience of the book, you said that it is oriented towards people who are looking to get into data engineering. But obviously, there's a broader audience that would benefit from the lessons here. And to your point of understanding, what are the things that go into all these pieces of work? I can definitely see the emerging role of the data product manager as being somebody who should really read something like this book, if not this book, specifically. And I'm curious how you thought about being able to balance focusing on the audience of somebody who wants to be a data engineer while making it approachable for people who just want to understand the context of data engineering. We tried to write it in such a way that while there is a very specific target persona, the applicability would be much broader. And even going beyond data product manager, we hope that people who are trying to undertake some kind of digital transformation-- you know, that's a huge cliche. I think we know what we mean by it to some extent-- we'll be able to read the book and make decisions about hiring and steps and sequencing in order to get from A to B. And so we hope it's written in such a way that it does have this much broader audience. Well, we've got a lot of requests for that. Data engineers are very excited about this book. We get a lot of questions from them. Data scientists and analysts will work product managers. The one we were most surprised by-- we've got a lot of feedback from product managers that want to pick up a book just to understand what's going on. I talk to data engineers every day. I have no idea what they do. Maybe your book will help me at least understand what they are doing or maybe should be doing. So-- and that's the other feedback we've gotten, too, is the book is really much a rubric. Not just from data engineers, especially. They're like, well, I've been working as a data engineer for a while, but I've never had a holistic context of the title I have or the job I was hired to do. And so this is something we've been talking to, team leads and execs about increasingly is leveling up the knowledge on data teams for data engineering in particular. Because as of now, I think without exception, there aren't really standardizations of practices or skills for data engineering teams. It's kind of like, well, you know about databases. You can type on a keyboard. Cool. Join our team. You know about a AWS. You'll be a great data engineer in our team. I mean, can you imagine trying to pick a sports team that way? So your girls will play the Super Bowl. You know, right now it's pretty random. It's kind of a. after school like football team, more than as a professional sports team. And I think hopefully, you know, stuff like this, you know, like this book, and I think just standardization of practices and expectations goes a long way towards, you know, making data teams, I think just a lot more focused, specialized, and at least playing on the same playbook. - Digging into the contents of the book specifically, I'm wondering if you can talk about how you thought about the overall structure and which concepts to address in which sections and just sort of the overall layout and organization of the book and how you thought about bringing people through that journey of, okay, this is what data engineers do and why you might be interested in reading this book through to, okay, now you've read this book, go out and, you know, be effective and, you know, these are the next steps for you to actually put this into practice. - I think we decided pretty early on to structure it around this idea of the data engineering lifecycle. And again, that was inspired to some extent by Google and other sources where we're like, okay, if you boiled data engineering down to its fundamentals, what are you really doing? Like, let's get down to the essentials. Once we had those sections, we realized that you probably need a discussion of actually how to choose technologies because that's actually something that isn't, we don't do a good job at as a profession, we talk a lot about why certain technologies are cool. We don't talk about how to make technology decisions and because so often that falls on data engineers, we ended up adding a section for that. And then we thought further about it, we're like, okay, as architecture really the same thing is choosing technologies, it seems like it's a bit different actually because you're making decisions about team structure and organization, about responsibilities, about data flows, and eventually about technologies too. And so we decided we needed to add a chapter on architecture for that reason, especially because in small organizations, data engineers often end up being kind of the de facto architects. And then originally we thought we would have a fairly strict hierarchy, like we'd have topics that would sit under ingestion, topics that would sit under generation, but then we realized that there were these ideas and data engineering that don't sit into a stage of the life cycle. So for example, security is everywhere, right? If you're not thinking about security at every stage, then you're just asking for breach. Same thing with orchestration, you know, like Apache Yearflow or many other technologies, orchestration cuts across all the stages because you're really managing data flows across all stages. That's this concept of the undercurrents where they cut across each chapter and we keep revisiting them. And we hope that that's like a very concrete pragmatic stuff that will help people to become practicing data engineers. >> Yeah, I mean it came to stuff like a data governance and data management for example, right? Like I'm sure if you thought the DM book of knowledge, the DM book of knowledge, I think it's huge. I think certain areas of that book that applied more than others to data engineers and data engineering life cycles so we just said, well, fine, we'll choose some of these undercurrents, we'll highlight these as being what you need to know. But for the reading please read these other books like DAMA and many other books that we listen in our book. So don't shortage a great resource out there. But the other day we really thought that the engineering life cycle, the undercurrents, architecture, choosing tools, again, in the end we highlight security. That's a huge one. We just wanted to re-emphasize it at the end. Like don't forget about security because if you do that, that things happen to good people like you. And we finished it off with the feature of data engineering. Sort of what is our take on where things are going. That was a fun one to write because you could be totally wrong and so have a lot of fun writing it. So the other chapters you had to be pretty correct were very correct actually. So that was fun in itself. I would say the part two of the book where we go through the data engineering life cycle in particular, it was great. Our panel of tech reviewers was, I would say, world class and just really beat the crap out of us and the drafts. So the end product is very happy with it. In that process of saying, okay, we're going to write this book. These are the topics we're going to cover. This is how we're going to lay it out. Then there's also the hard work of saying, okay, well, how do I apply this level of rigor to make sure that I'm getting things right? And I'm not leading people astray and not just writing this just for the sake of writing something because obviously it's going to make you rich, right? (laughing) That's why we did it. It's all for the royalties. Yeah. Matt has new Bentley on the way. Yeah, that's right. Yeah, he had a scold standard for. But just curious how you actually went through that process of saying, okay, we've created this framework but now we actually have to go and do the research and distill it and make it addressable for our end users because obviously nobody's going to be expert in everything. So I'm just curious, what were some of the pieces that you had to go and actually do research and do your own independent learning on to be able to reflect that to your readers? I know personally it was like 90% research and reading and like 10% writing. And the book took us about a year and a half to write so you can kind of understand how much research was involved in this. It was not trivial. If you look at our further reading section, for example, you can kind of get an idea of how much we read. So I'd say, you know, definitely going out podcasting helped a ton too. So we have podcasts. Yeah, we talk to guests and I think by doing that got a lot of opinions and insights that we just wouldn't have got if we were just reading, right? So in watching videos on YouTube helps, but at the end of the day, it's just sort of a lot of a pollusive effect. There's Charlie Munger calls it where you just get a hard project ideas synthesize them and something new comes out at the end of the day. You know, the outline of it, I think, was an easy part. And we came up with that because we had to do the book outline in the table of contents, but then researching stuff is like data is such a huge field. Data engineering is huge. And every time you think you got a subject and you start reading about something else, you know, like, well, crap, I got to add that into. So it just grew and grew and grew and grew and grew what happened. I think to some extent, some things you know that you don't know basically. So you go do a bunch of research and then sometimes you would start writing and you're like, there's this area here actually. I'm going to pull on this thread and you're like, ah, I actually don't know, you know, if about this topic. And then you would run off and do some research on that. That happened many times. And then just us questioning each other on stuff was very helpful. Yeah. Joe, question me. Question in his writing saying, hey, is this really complete or is this the correct view of this? Should we take into account other points of view? And then our reviewers were just invaluable in this respect too in terms of pushing back on certain things or just suggesting like, hey, have you taken this into account that was invaluable to? And then I would say also part of what we did, which was kind of painful. So Joe mentioned like DM Bach and some of these like really old school enterprise-y practice books that, frankly, people in the engineering space don't particularly like in many respects. We kind of realize that we probably need to give credit where credits do and also just borrow the best ideas from those books by reading them and then integrating them where possible and kind of saying, take these enterprise practices that you might view as stodgy and then make them your own like modernize them. They actually have a lot of really good ideas here about data management. There's a lot of nuggets. And I think we realize there's nothing really new out there per se. Even the innovations that you see from big data tools all the way to now, they really do stand on the shoulders of papers and journals and people that have come-- these ideas decades ago. I would say the best thing to do is just read a lot of history and a lot of the original writings. And I think that gives you the context to write a book like this. When we talked to O'Reilly about this book, they thought we were insane. So I remember Jess Haberman or acquisitions editor was like, guys, do you really want to have this as your first book? And we're like, yeah, well, why? And it's like, because it's hard. Nobody's done this for a reason. There's so much to pull on here. This is not your typical book. But she apparently didn't persuade us not to write it. I got greenlit really fast once we had the book proposal done. So she's like, OK, you guys think you want to do this? Go do it. So she did everything in her power, I think, to discourage us from writing this. She's like this skill of like 1 to 10. This is probably a 9 or 10 in terms of difficulty to write. This is not a trivial book, not at all. Because think about what you're trying to do. You're trying to come up with a definition of a field that I think has been sort of defined, but pretty vaguely. And then come up with the practices of a field that has been not really defined. We all do it. But it hasn't been comprehensively defined in this way before. And so that was very challenging. But along the way, what was pretty cool is calling up people like Bill Inman and asking him, does this section on data warehousing look correct to you? And he'd say, there's fixed a couple things here, but it looks pretty good. So just getting the expertise of people who were truly originators in this field, I think, was one of the great experiences of writing this. So everyone wanted to pitch in too, I think, because at least quite a few people were rooting for us, especially towards the end, where they're like, this is pretty cool. Yeah. Definitely not trivial to be trying to write a definitive guide in an evergreen format for such a moving target. No. [LAUGHTER] So because of the fact that you are trying to treat this broad subject area in an evergreen fashion and trying to remain technology agnostic, I'm curious how you balanced that with the need to actually provide useful examples for people to be able to crystallize and conceptualize the different topics that are being covered and just how you thought about where and when and how to introduce specific technologies or references to technologies as ways to illustrate the examples or the subjects that you're trying to cover. I think ultimately we had to choose a collection of what we considered to be best in class technologies to use as examples across the board. And so we always tried to cite several examples of technologies where possible or maybe illustrate very narrow examples with a single technology. We also tried to keep it pretty broad and to come up with applications where like here's where you would use Apache Druid, for example. Here's why it's useful. Digging into--
details there because yeah, otherwise you risk getting just pulled in to the direction of a single technology and sort of obsolete yourself and obsolete your book within a couple of years. I don't know what's here. Yeah, I mean, you try and find the technologies that will be around, hopefully, but it's kind of the books in print. It was kind of funny though. Originally, we did mention quite a few modern data startups. And I would say overwhelmingly a lot of those startups by the time we were about to finish the book, they either got acquired or something happened and they weren't around anymore. That was interesting. We tried to go with at least either Blue Chip or abstracting it away and saying, well, here's an example of X, Y, and Z technology. No, and caveatting it as much as possible saying maybe when you're talking about technology, we say maybe for example, such and such technology does this and so forth. We're cautioned by other authors of technology books to definitely keep the mentions of specific technologies down to a minimum. It's going to date your book automatically. The other thing I'd mention and we definitely took some inspiration from designing data intensive applications for this is that we tried to slice things into some big ideas about where data engineering was going. And so, for example, we have a pretty extensive discussion about separating compute and storage. And then I also go on to talk about how in reality, mostly systems are hybridizing, you know, separation and co-location in some ways. And then we just cite a whole litany of examples of how different systems do this. You can get the principles of how you can both separate compute and storage and yet improve performance through things like caching by just talking about how a lot of different technologies do this. And ideally, if you, as new technologies come out, some of them will do new things. In many cases, they'll do old things better. And so, hopefully, a lot of those approaches and ideas are already on that list, cited. Yeah. And to your point about the way that the designing data intensive applications book treats things, one of the things that's standing out in my mind is that a specific technology that is used as an example to illustrate a particular database internals concept is the React database, which is no longer a thing. It has gone defunct and it's not available anymore. But the example of what you're trying to illustrate is still valid. Like, regardless of the fact that React isn't a technology that's in use, it still does the job of showing you this is what this is doing and how. They think that that's definitely a kind of useful sort of firewall that you have where you're not trying to be instructive about a technology. You're just using the technology for illustrative purposes. So even if it does go out, you know, it seems to be maintained or the company goes out of business, the fact that it was there as a illustrative example is still valid even five to ten years from now or, you know, once that company gets acquired and goes in a different direction. Oh, for sure. I mean, you can pick up design data intensive applications today and it's still a great way. It still feels really fresh. It does. It feels really well. I believe Martin's working on a new addition of it. I can't confirm this. But even he's, you know, I think recognizing that, you know, perhaps the book needs, you know, some updating. So we'll see how that turns out. I'm very curious. To cite another example, I mean, we realized from our consulting practice that it's very important to explain to people how a columnar databases work. And it's actually shockingly often we see people just abusing columnar databases in various ways and then they get poor performance and high bills. But at the same time to your point, you need to illustrate that in concrete ways, right? So you need to talk about like park a files and snowflake and Bitcoin, other technologies redshift that are columnar so that people can get, feel like they're getting a bit more hands on because if you just talk about columnar databases abstractly, it becomes very hard to understand. Oh, that sounds boring. Yeah. And because of the fact that you were using some of these technologies to illustrate different points and you had to go through the exercises of being sufficiently knowledgeable about them yourselves. I'm curious how you approached that process of selecting which tools to use as an example, which examples to actually work through and just validating your assumptions and your instructions, yourselves, and how you managed the sort of iterative process of saying, okay, this is what I'm trying to cover. I'm going to have to actually set up this environment and do these operations to make sure that I'm not saying something that's not exactly true or that this isn't behaving the way that I'm saying it is. Yeah, it's definitely a mix of experience and just, it was a ruthlessly reading documentation over and over again, just to make sure you're not mistaking something, right? But again, I think it comes from a lot of experience with these tools as well. And observations too of, I would say both great patterns and anti-patterns. So it's a mix of everything. I think observations experience and what is it? RTFM. But yeah, all these things combined, I think, was how we approach all these examples. Because I mean, you know, Matt's got a lot of experience with certain systems. I have a lot of experience with the different systems. We often have the same experiences, the same systems. And so, you know, it's just really drawing on the body and knowledge and talking to engineers too. I think that's another big thing where we felt like talking with data engineers about their experiences and practices was invaluable. There's Matt and I say too, like if one of us misses something, the other misses it too. We hang out so often that we can finish each other's sentences basically. So the danger in writing a book like this with a co-author who's basically your clone is that, you know, again, it's great, but it also forces a certain myopia that you can only break out of by, you know, talking to a lot of other people. Prefect is the Dataflow Automation Platform for the Modern Data Stack, empowering data practitioners to build, run, and monitor robust pipelines at scale. Guided by the principle that the orchestrator shouldn't get in your way, prefect is the only tool of its kind to offer the flexibility to write workflows as code. Prefect specializes in gluing together the disparate pieces of a pipeline and integrating with modern distributed compute libraries to bring power where you need it when you need it. Trusted by thousands of organizations and supported by over 20,000 community members, prefect power is over 100 million business critical tasks a month. For more information on prefect, go to dataengineeringpodcast.com/prefecttoday. It's P-R-E-F-E-C-T. In terms of the final results and the shifts that you saw just in the process of writing the book over that year and a half, what are some of the things that you anticipate needing to revisit in a V2 over the next two to five years? And what are the pieces that you are confident will remain evergreen and you're not actually going to have to touch? The basic ideas are going to stay in place. In other words, these stages of the data engineering lifecycle, you data is generated in source systems, you ingest it, you store it and so on. I think those core ideas are not going away. I think what's likely to happen is that the next addition is going to emphasize streaming even more than we already do. The reason I believe that is that we're already seeing in managed services is that managed services have made stream processing and real time processing much easier than it was five years ago. The uptake of that is still kind of a slow process like your heavy hitter tech companies are doing a lot of real time processing. I think outside of that, it's certainly there, but not as much as it's going to be in five years. I think in the next five years what we'll see is a huge increase in, well, just massive simplification of streaming services even more than has already happened and then really big uptake. And so just the amount of ink that we spill on streaming would need to increase quite a bit and then talk about some things that really haven't even been defined yet. There's kind of this active debate in the data engineering community about how you model streaming data. So hopefully we as a community will kind of figure some of those things out and then we'll be able to write about that in the next edition someday. >>The periphery, right? So in one end, you have software engineering or the other, you have machine learning. I think the intersection between software engineering, data engineering, machine learning is going to be very fascinating to watch. As Matt points out, as real time data becomes more ubiquitous, what I see happening in the space is streaming is going to become commodity, just like data warehouses and data lakes were commodity in the 2010s. Prior to that, those are very expensive multi-million dollar on-prem installations. Now it's like, who buys that right now? That same simplification and accessibility and democratization is going to happen in streaming, machine learning. I think a lot of stuff is going to move back to the application layer for software engineers which means they may in fact become the data engineers and not the data engineers themselves. So managed whatever the data engineering lifecycle happens to be, I don't see that as a principle going away, but maybe the lifecycle shortened dramatically, the feedback loops become a lot shorter too. I think this seems like somewhat of an inevitability, although every time I say that word, I should punch myself because nothing's inevitable. But at least from where I sit, that seems to be more things are going. So the things that we probably need to revisit is probably exactly these assumptions. We're talking to Jordan Tagani a couple of weeks ago, one of the co-creators of BigQuery and he's talking about small data now, using DuckDp for everything. So every time you think you have a sense of where things are going, I think it pays to have an open mind and maybe goes the opposite way, but there's nuggets of truth to everything too. It's not like it's sort of binary outcomes either, right? So I think it could involve all the above to some extent. Yeah, it's interesting because the data ecosystem is very fractal and in some ways to your point of, oh, big data, now small data, both are true. It's somewhat sort of the redshift of the universe applied to the data ecosystem where everything is expanding and it's moving away at an accelerating pace. If you started at the point of inception of database systems in the 1970s and you have remained in the industry today, you have to pick a direction to move in because you're never going to be able to encompass the entirety of the ecosystem. And so for somebody who is coming into it today, you have to figure out what area of that universe you want to actually live in because you're never going to be able to
that go from one end to the other within your lifespan. - Oh yeah, for sure. And the other big thing I think is gonna happen is data modeling is, I mean, it's coming, I wouldn't say it ever went away, but it is coming back into Vogue to some extent. But the interesting thing is, like a holistic view of data modeling from like a streaming, incorporating things like graph database as machine learning and stuff, like something that I'm hugely interested in right now and whether or not that gets revised in the next edition of the book, or whether that's a standalone book, I think is still TBD, but that's narrow that it personally feels, that's gonna get a lot more attention as well pretty soon. So, 'cause a lot of the techniques we've been talking about, as you point, you know, talking about old stuff, it's like, it relates to relationships within things like structured data for the most part, right? But what about all the other data out there, images, audio, text, everything else? That's fascinating. - I think, Joan, I both believe that a new conversation on modeling needs to happen to your point, which is we have these kind of established versions of modeling, like Kimple and Imman, which are really fantastic for what they cover. But we need to figure out ways to update those for newer technologies like columnar databases, where you don't necessarily want to normalize everything, it actually doesn't make sense to do so, and also to cover more types of data and somehow be more holistic about your modeling and extend it beyond the boundaries of just the data warehouse. - In your work of writing this book and doing other research and working with your reviewers and the community, what are some of the most interesting or innovative or surprising elements of the space that you encountered or that you learned about? - There's so much to learn and to know. It was interesting I was talking to one of my friends, he was like the fourth user of Hadoop in the world, so pretty old school, and I would say he's like the data engineer's data engineer, and he reviewed our book and he said it's kind of crazy, the amount of stuff he needed to know as a data engineer. And so I think that that was actually a very interesting, some interesting feedback, 'cause for Matt and I, as I go, we're covering enough, so maybe we're covered more than we would, but I think that as you just kept unraveling the string sort of speak, it was like there's just more and more to it, and I don't even think that we covered to a level of detail what is possible out there or what's available, there's a lot. - Well, I'll say too, I mean that, yeah, the conversation with Jordan Tegani kind of genuinely surprised me, because after so many conversations, we saw a lot of iterations on existing ideas and various incremental improvements, but this idea of like having a hybrid sort of traditional database with an edge type web app that queries data, kind of like that's the standard for mobile apps now, for example, right? Back in the lives in the cloud, with a lot of local processing. I'm like, okay, that's a genuinely cool new idea. I mean, you're borrowing from the Domain of Web Development, but that's a really cool new thing, and so I'll be curious to see what comes out of that in terms of evolution. - Yeah, but once in a while, you talk to a lot of people and hear a lot of cool stuff in the once in a while, something just like really pops for you and you're like, huh, I hadn't thought of that before. - Yeah, that was the fun part. I was just talking to people, and we talked to hundreds of people, experts in the space, people that you know of, and I think that was probably the most fun, was just getting to know these people, and they're accomplishments and their thoughts in the field, and also just their backgrounds, kind of what makes them tick, I could think data engineering is as much about the technologies, is about the players and the personalities in the space, I think that's what makes it pretty cool, people like yourself, right? Listen to your podcasts for ages, and now we're talking on your podcasting, that's really cool. You know, last week, us doing our flip the script interview and understanding what makes you tick, I think that was a very, very awesome experience, I'll remember for a long time, and just, you know, understanding like, what is it that drives the space forward? That was the most surprising things too, is talking to people who have invented a lot of the technologies that we take for granted these days, right? But understanding, okay, so like, what was the motivation to create this? Like, why do this? Why not use something off the shelf, or why not take a different approach? But that was the approach that they took, and obviously it worked really well. So I think that's a cool thing, because it highlights, you know, just the important principles of technology in general, the field is never static. People are always trying to solve new problems, and to me, that was, I think the most interesting and inspiring part of writing this book, which is again, getting to the personalities that we all know, and, you know, if I've helped drive the space forward. So it was cool. And to the point of always solving new problems, and its intersection with the love, conservation of complexities, you know, how many of the problems that you're solving are ones that you created by the previous solution? (laughing) Conservation complexity, that's also-- - Yeah, that's interesting. - Yeah, because I feel like a lot was accomplished in the Hadoop era in terms of suddenly being able to scale and systems much larger without spending, you know, $100 million. But we seem to build a lot of complexity in that era too, and some of that wasn't necessary, and just ended up being a lot of tech debt in the long term. And so I guess the question is, what are we doing right now that's gonna cause, like, massive pain in two years? It's just-- - Well, I mean, you see people are complaining about the modern data stack causing that kind of pain. So I mean, I think people just like to complain too. (laughing) - Yeah, we love complaining. It's fun. - It's like it's not a thing's ever good enough. You always have to complain about it. - If you complain about it loud enough, it'll justify all the time you spend on trying to build the next new thing to solve the thing you're complaining about. (laughing) - Time you raise your next round of fun. - Exactly. (laughing) - And so in the process of writing the book and actually doing the authoring and the research, what are the most interesting or unexpected or challenging lessons that you each learned personally in the process? - We were talking with Justin Borgman today from Starburstone, one of our shows about this. And I think it was writing a book and running a business was probably a really dumb idea. So, and I think that, you know, that was just a lot of work, but we managed to get it done. And I think just how much you learn when writing a book is our first book, but it definitely pushes you to boundaries that I would say you will never get otherwise. And so by writing, it's really a good way of refining your thinking and I think refining your research as well around ideas. And then finally, I think it was also interesting 'cause as we were finishing the book, we started getting a lot of good feedback on it. And then we started noticing on places like Amazon, it was already the number one new release and this is like while we were writing it. And so you wanna talk about having pressure to finish strong. Like, it's hard enough finishing on its own, but when you get these external validations of your book when it's not even done, I think the keeping focused got a lot harder for me personally, just like it has to be really good. Like you can't screw this up. To hopefully the audience likes it. (laughing) But we'll see. - I would say too. I mean, it turned out to be more fun than I thought it would be to write this book. Certainly it was challenging. And you know, Joan, I had occasional disagreements and such. Just quite a few lot of pressure trying to get things done and hit deadlines. - It's like a lot of. (laughing) - But it turned out to be surprisingly fun. And I think Joe has been tempted into maybe starting another project sometimes. And we'll see if that happens. And I'm not quite ready for that yet. I need to let this work late for a bit before I start to finish. - Yeah, I might be like, when I had my first kid, it was kinda like, yeah, we'll probably not do that for a while. And you know, time heals all. And you're like, "Gah, that's happening to the kid." And I feel like that's what books are like. Where you're just like, you get out of it and you're just thinking, "Oh, I'm never gonna do that again." That's terrible. But what happens is you get into book mode, right? And you get into thinking mode. I think you start seeing things differently too. Like the person you are before the book is not the person you are after. Like you just, you see problems differently. You kinda see the industry differently. And then you wanna probably write more about it if you're a massacist. So that's what we are. - And so digging now into the predictions that you have for the future of data engineering, you wrote a bit about that in the book. And I'm wondering if you can talk to some of the themes that came out there and maybe some of the ways that you're thinking about it has evolved since you've hit save on that last version of that section of the book. - Yeah, I mean, kind of alluded to this with what we call the live data stack. So that's sort of the shortening of the feedback loop between applications and machine learning, embedded analytics and so forth. I think that's gonna be a real thing. Some variation of that, maybe it's not called the live data stack, but something resembling that I think is gonna happen. We also talk about data engineering is becoming more quote, enter pricey. And what we mean by this is, as tooling becomes more abstract, data engineers focus is gonna shift from maintaining tools to maintaining processes, management, all the stuff that was once considered sort of, reserved for large enterprises and so forth, that's trickling down. You can see it, observability that was in talk about a few years ago, data ops, quality, all these quote boring subjects are now like, the hottest thing that's going on, data engineering. I think finally, it's kind of weird saying this, but I think there's gonna be a lot of attention put on spreadsheets, like spreadsheets or sort of the dark matter of the data universe. That's amazing that we spend all this time talking about tooling around data warehouses and all this other stuff. When you just go out into the world, like people are using spreadsheets way, way, way, way more than these fancy data systems that we all use on the show. To me, that's one of the uncharted areas for data is actually tooling and better practices around spreadsheets and incorporating those probably back into some sort of infrastructure, but well, there's two billion, two point, something billion spreadsheet users out in the world. Oh, yeah, I mean, we talk about that specific theme in the book and it may not actually be spreadsheets that are the future of this area, but what I'll call it is like a new way for people to interface with data. If you think about it, when did the spreadsheet revolution happen? Will it happen with FISAC-OC on the Apple too? Not alone was enough to boost the Apple too into success. Like, that alone became a huge driver of the Apple too as a business platform. And spreadsheets have evolved a lot since then, but I think there needs to be a next step of some sort. Maybe we completely rethink the way that people interface with data, but it feels like with modern spreadsheets and with dashboarding tools, we're not quite there yet. There's another potential step we can take as data engineers, analytic engineers, to make data much more successful.
accessible to people who actually need to use it like CFOs and like business management. >> Yeah, because it's just connect. >> Every time we see exact teams or all these spreadsheets, that may have been populated from the data warehouse. I think that's the entire point is there's just a huge data divide right now between where data and decisions are getting made and the type of work that data professionals do. Oftentimes these are not the same things. So that's the other prediction for the future of data engineering. Is it becomes less exciting, I suppose, for some data engineers. But yeah, spreadsheets are kind of like the business mallet of the world. So going back to what Joe was saying about the live data stack, I think again, that's going to be driven by improvements in managed services. I think what we've seen right now is that if you're on like AWS or Google Cloud Platform, they have really, really nice tooling for managing real-time data that was not mature like five to 10 years ago. So for example, you just go in and you turn on Amazon, Kinesis data streams, and then you can connect that to what is it Kinesis Analytics. And there you go, you have like a real-time data analytics platform. But there's still this last mile problem. We're getting the data from your application database streaming into Kinesis. It's still tough and the same is true on Google Cloud Platform. And so I think in that managed service domain, something is going to happen where in the future will be almost turnkey where I can turn on Postgres and then I have streaming changed data capture right there. And it's just that means that as a product manager, whether I'm an application product manager or specifically a data product manager, suddenly I can start thinking from the get go about what my analytics are going to look like for this application and like, hey, I have an application, but also I can have embedded analytics for my SaaS platform users from the get go. And maybe I can have one or two data engineers on my team to get that up and running rather than having to have like 10 to run Kafka, you know, spin it up and manage it myself. So it's going to be really interesting to watch that evolution. And another evolution I'll say is that what we've seen in the data engineering space, even though we complain a lot about people getting centered on technologies and products, certain technologies of just like revolutionize the space very quickly. And that's hard to predict, but like something new will come down the pipe in a year or two. And it's going to be the hot new thing and it's going to change the way that people look at data in a way that we couldn't predict. So curious to see what that is. Excel. Excel 365. To your point about spreadsheets, we're already starting to see that. So I had the founder of the company Canvas on a little while ago, where that's their whole product is spreadsheets for the quote unquote modern data stack, where it does hook into your DBT workload hooks into your data warehouse. And it's just the spreadsheet interface for product owners and business users to be able to actually explore the data that you're working with in an interface that is familiar to them. But still having that escape hatch for data engineers and analytics engineers to be able to actually provide some useful guard rails so that you don't end up with this, gross spreadsheet with 5,000 formulas that don't necessarily tell you what you think they're telling you. Oh yeah, for sure. It's writing in our newsletter last Thursday. Predicting the future of technology is more about analyzing where the pendulum is sort of along the continuum. It seems to go from one extreme to another, right? And so whereas I would say several years ago, the pendulum was more on the end of kind of a free-for-all, no-seek wall. This kind of issue, a lot of the formal practices at the past, it's swinging back towards the formal practices of the past. And as you point out, some of the interfaces that we've used in the past. And so really view technology not as a. There's obviously innovation and progression that occurs with each swing of the pendulum. But really each time you get new innovations, it's more of a reaction to the extreme on the other side of the pendulum. Dear point to about the kind of live data systems and some of the other conversations I've been having a lot recently is that I think one of the things that's happening slowly and will probably start to pick up speed is bringing the software developers and application engineers more in line with the data engineers where their systems and their tooling will more natively integrate with these downstream data storage and consumption platforms so that you don't have to worry about change data capture out of the database. You're actually going to have an easy way to build these event capture systems where you don't have to worry about pulling the stream of changes out of the database and trying to reconstruct what that actually means. You're actually just going to have the application engineers in the right path to the database where they're trying to serialize these objects for recovery and the application. They're also going to serialize that actual domain object into that Kinesis stream or into that Kafka queue so that as a data engineer you can say, "Okay, I understand what the context and utility is of this piece of data without having to reconstitute it." It's not the meal that you're taking with you to hike the appellation trail that you have to try and turn into something that's somewhat palatable. It's the gourmet dinner that you're getting handed at a restaurant. My friends at Maroxa, I think they had a video on a conduit, which is a way of basically for application developers to create events streams from an SDK. When I saw that, I was watching the old Ruby on Rails video from DHH back in the day. For a split second, I thought, "Okay, this is inspiring. This is really cool to watch. Let's see where this goes." If you can just create a data pipeline programmatically at the application layer, I suppose it cuts out a lot of steps. Doesn't it? Pretty cool. Yeah. Data engineers also spend a lot of time whining about application engineers and how the data is not in good shape when they get it to do analytics. Rightfully so, some of those complaints are legitimate, but I think there's both technology change happening and a cultural change. The cultural change is to actually give the application developers a part of the glory for analytics to say, "Hey, your stakeholders in this, you should care about analytics. Help us out here by designing a schema that's suitable both for your application and for analytics." To your point, Tobias, the technology changes are going to make this much more seamless. Where from the get-go, they can have what appears to be a database that serves both analytics and the application back in. Now, the cap theorem tells you that you can't really do that, and that's true, but we used to call this the Lambda architecture, and now all those details are hidden behind the scenes, you still have that stuff going on. But yeah, it's so managed that it's just like magic and it's like a gourmet meal instead of like an MRT. Well, for anybody who wants to get in touch with each of you and follow along with the work that you're doing, I'll have you add your preferred contact information to the show notes. And as the final question, I'd like to get each of your perspectives on what you see as being the biggest gap and the tooling or technology that's available for data management today. So this is not necessarily a gap per se, but I think a lot of consolidation probably needs to happen in the data observability space. There's just such a pro-movuration of tooling that the pro-movuration is harming the space, right? Like there are just so many different options, many, many great options, but it's just become this like an overwhelming pallet of things to choose from. And so I think at some point we will see a consolidation and a more clear picture of what data observability looks like. I think I expect to see some other maturation in data observability as well, where we start doing a better job of understanding data as it changes in the real world, in the outside world. So in other words, I think in observability there's a big focus right now on operations, on making sure my systems are running, and what I'll call expectations. So saying, okay, this should be null, this should not be null, these values are correlated even though they're in different fields in the table. They value A should not coincide with value 50 in this other field. So basic business logic type observability. I think we need better tools for almost like borderline machine learning to say, "Hey, this data looks really weird." Is it because we had a recession or COVID or is it because the data is bad? And start being able to raise alarms when we see weird data, so that data engineers and data stakeholders can start investigating and discover if there's a change in the real world, or some bad data coming into the system. So sometimes I call that data entropy basically, but the ways, whenever you're handling data, the world can do weird stuff or any system can do weird stuff, and you have to be able to account for that in your ops, basically. I think last time we were chatting to bias, I talked about how data modeling is a big gap in the tooling and technology space. I also think that practices on data teams need to be standardized. As we kind of talk about it earlier, right now with data engineering teams, for example, it really feels like the teams are very ad hoc. You kind of find whoever you can get. There's not a lot of rhyme or reason as to the skill sets. They sort of make sense, but there really hasn't been a standardization of skills or knowledge. I would like to see that gap filled as well. I just think it would make for better performing data teams across the board. Say that they read a book like ours, for example, right? I mean, that at least gives them a certain playbook against which they can make decisions. You know, obviously other books as well, right? I'm just going out as an example, but I wish that there was more standardization of knowledge and skills in the data engineering space. Because like I said right now, it seems like that knowledge is spread across whatever random blog posts you happen to be reading. And hopefully that people listen to your podcast as well to stay on top of the cutting edge of the space. Between these spectrums, hopefully data engineering teams over the next few years can I think perform at a much higher level. Yeah, it's definitely necessary to have that shared vocabulary so that people who are trying to collaborate on building these systems understand what the other person is talking about. Because if you don't have that shared vocabulary, you can be using the same words, but talking about completely different things happens all the time. All right. Well, thank you both very much for taking the time today to join me. And for sharing the work that you've been doing on the book and just for writing the book in the first place, definitely something that I've been enjoying reading and will be recommending to
Lots of other people, working on getting everybody on my team to read it. So thank you again for all of the time and energy you've put into this, and I hope you enjoy the rest of your day. Yeah, thanks to my ask. Thank you, this has been fun. Thank you for listening. Don't forget to check out our other shows. The Data Engineering Podcast, which covers the latest on modern data management, and the Machine Learning Podcast, which helps you go from idea to production with machine learning. Visit the site at pythonpodcast.com to subscribe to the show, sign up for the mailing list, and read the show notes. And if you learned something or tried out a project from the show, then tell us about it. Email
[email protected] with your story. And to help other people find the show, please leave a review on Apple Podcasts and tell your friends and coworkers. (upbeat music)
PodcastTranscript AI