Data Science and Analytics

Hi, I'm Dave Bazooki, founder and CEO at Roblox, and you're listening to Tech Talks, a podcast about the people and ideas that are shaping the future of the Metaverse. In this series, we'll be exploring some of the most innovative technologies that have emerged in this new category and sharing stories with the Robloxians that are building them. Today I'm joined by Todd Rudak, director of Data Science and Analytics at Roblox. Todd leads our Data Science and Analytics team, which supports decision-making and building the Roblox platform. Today we'll discuss the unique qualities of data science and analytics at Roblox and the work they do to support innovation. Let's get started, Todd. So great to see you in person, your two rooms away. Welcome. I miss you. Hi, Dave. Thanks for having me here. Yeah. It's wonderful. How was your holiday break, Todd? Absolutely. Wonderful. Got to spend some time with family. Got to recharge. Loving it. Okay. So this is going to be really fun because I think a lot of our listeners are familiar with data science and analytics, but to a lot of our listeners, there's engineers and designers and product people and what is data science. So we're going to kind of get into it and maybe start first with your background. If you could, could you share a little how you got into this field and how you came to work at Roblox? That's a great question. I would say that I'd never actually intended to be a data scientist in my field. It was one of those cases where even back as a teenager, I would be playing around in Microsoft Word, while creating an essay in high school. And I would always ask myself a question of why do I have to go forward windows in to find the feature I want. I've already picked that up five times. And I didn't realize until 15 or 20 years later when an entire industry started to build around this idea of understanding personal behaviors and making products towards them. So my background started in computer science. I also spent a lot of time studying statistics and econometrics. In early parts of my career was more focused on marketing. That was the first use case for analytics and a little bit of a science side. That grew over time, spent more time from big data in the banks in a financial world to even big data when it came to media platforms, whether they be social or deep content. So question Todd, when you said you were you were very into computer science and physics and you went into marketing. Was it the analytical part of marketing or marketing general marketing? It was the end of the side. It was the one that was creating loyalty programs, understanding marketing campaigns, understanding behavior of what people did on various websites and various tools to really understand what we need to build. And that was the early, early days of really what analytics was. Yeah, well, this was, I mean, marketing is really interesting because there's this wide range of things that you can measure and attribute versus things that are very, very fuzzy. You know, when I watch madmen, for example, that seems like a day when it was really hard to measure whether things were working. So were you in a more new area where you could measure these things? Yeah, you can think of the classic case of marketing campaigns, which you'd actually send out snail mail to people. You want to open up a count? We think you're pretty good with your credit. So why don't you open up a credit card here or open up a checking account and we could actually determine who were the right people to target, send some mail out, track them into the business and help them grow, but also meet their financial needs. That sounds really cool. Okay. So you're in marketing, but you're already doing analytics and how did that evolve? So outside of marketing, I moved to a couple of different companies and through the course of that, the company started to want to understand how people change their behavior on the website. So, you know, which page they click on, which product they choose, what happens when they were no longer interested. And that starts to build. It was also very the early days of.com. I started my career around 97, 98 back in college and then came out to Silicon Valley around 2001, 2002 at the tail end of it. And this is the time where people started to log a lot of data. And we didn't exactly know all of the worlds that would open up for us, but it was fun. And we got to determine what were some of the interesting things that people did on various websites just as they were in their very early stages. Yeah. So this, I mean, this was an amazing time because the feedback loops were so short with traditional marketing programs. As you mentioned, TV advertising or sending out mailers, you know, your week waiting days and weeks and months, whereas you had moved into a real time domain almost, was that how did that feel like was it? It must have been very exciting in those early years to be doing real time type of marketing. Oh, yeah. Absolutely. It's a feedback and that goes through a lot of my career, but you don't want to wait three months to get a signal on whether you did a good job. You want to wait maybe a couple of days at most a week. And so getting that direct feedback and seeing what the potential was for the web at the time was really just like generally exciting for anybody in their career in this type of field. Cool. And then so from there along the way, can you share a bit of the trajectory towards where you've we finally came together and you started running analytics at Roblox? Yeah. I would say that my time in the finance world eventually started to weigh my interest and I wanted to explore a couple of different areas plus I realized that the amount of data was getting bigger and bigger and bigger by the moment. Banks used to be known as the one with the big data and then income, social media platforms and they completely trounced it in the orders of magnitude. And I want to make sure I stayed ahead of my own skill set as well as the interest of the new questions that we can ask ourselves. And so I moved along to a big social media platform which the treasure throws of things that you could ask and answer in a lot of different ways was a new world. And then from that point, I spun off of that in order to understand a little bit more in deeper content content you spend two to three hours watching in video streaming over a platform and that just added up a whole another color to my past experience. What was interesting about Roblox is we added a third dimension. You're not talking about 2D content on a TV or on a website. You're talking about 3D immersive content and as a whole level complexity and obviously difficulty in understanding what's going on in these worlds. Yeah. And maybe a big responsibility as well in that as we have the opportunity to gather this information, we have to do it in a civil way which we do wonderfully. Also I'm very proud of the fact that because we're powered by a virtual economy, there's not a lot of pull for this information for advertising feedback or anything like that. It's primarily around making experiences better. But as you highlight it in 3D, it's almost as if we there's interesting data around what people are doing, what kind of friends they have all of that. So it's a super rich playing field. Absolutely. It's only going to continue if we're all in here. Yeah, that's wonderful. So you came on board at Roblox and we started by the way, thank you. We are actually building out the data science team. I think it was pretty small when you joined. Can you tell us a bit about what the team was like when you joined and what your team does and what its purpose is? Yeah, that was part of the exciting thing of joining Roblox is I was given the opportunity to grow a small team. I think in those days we were probably five or six people and a lot of the understanding of the metaverse, we didn't really know. We had a team that started to pick apart some of the data and understand some general directions, but we didn't have a strong cultural experimentation to really keep us honest around whether we're making a positive impact on the community. So one thing about the data science and Alex team here at Roblox, our internal mission is actually to increase our speed, frequency, and acumen of our decision making as we build the Roblox platform. And we do this while adhering to all legal requirements for data collection, storage and privacy. Cool. So if we dive into that and maybe we think about an individual team with engineers on it, building some area of functionality that everyone is using and if we imagine that team of 10 or 12 engineers and a product leader and maybe a designer and things like that, would they have a data science or analytics person on that team possibly? They would typically and we've been growing our footprint into each of the product teams. So we have dedicated partners. We put data scientists to embody those problems. We don't want to be centralized or we don't want to throw problems or throw numbers over the wall. We want to actually embody those problems and help to make holistic solutions to make a better platform for everybody. Yeah, that is super cool. And so you've built out the org that embeds them a bit. Can you share a bit around the ranges of the types of things they're analyzing and types of data and the types of teams they're working on? Yeah, absolutely. So it's a wide range of these complex problems I talked about. You have some interesting areas around multiple two-sided supply and demand marketplaces. You have general experiences and understanding search and discovery. You have assets with those experiences as developers are sharing these with each other. You have avatars in which we're moving towards user-generated content of those avatars. But underneath the hood, there's a lot of other things. You have general economic health, as we use our currency of real bucks. You have the content of 3D immersive nature that I talked about, which generate by the community, our networking and streaming of this content, so a lot of infrastructure and platform problems we want to help solve. And then you have wide-raging demographics across the globe, including age, gender, and device. And so most importantly, keeping our community safe at all these perspectives is the number and priority. Maybe one of the basics as part of this is I know that there are teams that are rolling out improvements to their functionality all the time. And I'm assuming one of the things, like I know one of the things is actually we do, is it working or not? I'm assuming you're seeing both sides of that, because I see some of the numbers where it's like, "Oh my gosh, it's really working." And then I also love the case where data science says, "We haven't really seen any change, and this is adding complexities to let's not add this functionality at all." Can you share a bit about those two scenarios? Yeah, the latter one is my most favorite, because a lot of people forget about those. I recall a time back in early 2019, probably I was at the company for just a few months. And we had a couple of people sitting in the room debating over whether we wanted to build a certain feature for developers. And in concept, it sounded great. We had a lot of people that were very interested in it, very excited about that. But what slowed us down is it would take an entire year's worth, an engineering team, to push it out. That's a lot of investment. And I think if you would look back at the company a few years ago, we probably would have just done it as we thought Phil Soppley was the best thing to do. But instead, we used the opportunity to debate, to say, "Let's set up a lightweight experiment to actually prove this out." And so through a few different iterations of this, we actually found that we weren't making a positive impact for the community. We were moving around a couple of metrics, but those were dissipated over time. We didn't feel like we were improving the health of the community at all. We weren't thinking that we were making it better for the platform. And I just kind of fell flat. And what that shows for us is we could debate all day about whether something's good, but the community is actually going to tell us if it's good in the long run. So we got a strong signal that saved a lot of time from an entire engineering team. They could work on something better. It was more impactful and better for our community. Here's another fun one. I love the case where I think there's a host of platform things that could be done that can improve short-term metrics like an hour in or a day in based on novelty. But 30 days out actually degrade the platform because this was exciting for the first day. And now it's not really useful. Have we found any of those in either direction? I love the notion of those. Oh, absolutely. And that was part of the example I was showcasing where we might see short-term novelty effects. You put a shiny button out there, people are going to click on their button. It doesn't mean it's a good button necessarily. But you have to work through some of the seasonality we've seen in people's behavior. But even then, we might run an experiment for a few weeks, maybe a month. What we're currently working on is having longer-term holdouts from some of these features. We can understand, does that interest kind of wane over time? And do we still want to support that feature in the long run? Or is it just a longer path that we had for success of that feature? OK. Now just for the mathematicians out there, I'm going to roll the clock back seven or eight years before you're right here, Todd. And I can remember, you know, running some experiments where we would calculate it. And there weren't that many people on the Roblox platform. And we would go, oh my gosh, we might have to run this experiment for weeks and weeks and weeks and weeks. Because the statisticians are telling us to get some level of confidence. We need to run this many users through the experiment. Can you share a little about why that is? And maybe today at Roblox, how long we have to run things to get statistical significance? Yeah, absolutely. It's a lot easier in today's terms. When you have millions and millions of people on the Roblox platform on a daily basis, you don't need to test all of them. In fact, we have a couple of different time frames in which we like to test them our features. If it is something that's user-facing for our community, we typically like to work through one to two weeks of the experiment. So we can understand what are the effects from a Monday versus a Saturday or as new users join the platform. We need to understand those different areas. Plus, there's also just generally sensitivity of metrics, whether we can get some kind of signal from that. And then there's other cases where it might be behind the scenes, something in the back end, which case our experimentation can essentially be much quicker. Maybe we want to get to the point where we're actually releasing things mid-day because maybe we're just improving an algorithm or some kind of network protocol. Cool. So you've been at a few well-known places in Silicon Valley that all have awesome data analytics platforms and personnel. Is there, can you share anything that you think at Roblox is unique compared to those functions elsewhere? Absolutely. And part of the culture I've tried to set forth in the data science analytics team is I take all of the good that I've seen in some of those past organizations. But I also reflect on side effects of the way some things were structured and tried to course correct for them. And so what I think it adheres itself into our culture is I've actually learned to hate the term data driven. And I talk about candidates for that. And that's because when you talk about things that data driven, you tend to get pulled by the data you're getting pulled by the metric. And oftentimes if you do that multiple times over and you iterate, you tend to lose the context of what you're trying to do in the first place. And what I love about data science and analytics here at Roblox is we meld together a strong vision from our product managers. We bring in quantitative input from my team. We bring in qualitative input from our user research team in order to make that final decision. It's not one or another team that makes the final call. It's really a great collective decision. Maybe a metaphor would be and you can correct you might know more historically when Disney was creating Disney land. I have the feeling that was not data driven like there was some rough high level data like number of people in the US and who could visit it. But I don't think the haunted mansion was optimized with data. Do you think that's an example of a non data driven creation. I think there's a lot of vision that goes with that and that's part of the metaverse to I don't think data will be able to necessarily tell you 10 years ago what the metaverse was going to be. But we can certainly say what's working now was not working and we can course correct in the next 10 to 20 years as we continue to build. What's interesting is there's always a great question of well, how can we use data to actually measure our own innovation. There's no great answer to the question, other than if we have instituted a culture of experimentation, that means that we have an idea of fail first and fail fast innovation. And if we can do more of that, then we're going to focus on the things that are beneficial, the big ticket items. And that means if people are bringing in some large product features that we want to be able to test something that's completely game changing sure we can test if that's going to work both in short term and long term, but it's not going to necessarily bring up those ideas from scratch. If you if you had to think of then if I kind of like the radicalness of saying not data driven or however you're saying it, but at the same time, radically supported by data science and analytics, if you had to think of it, that almost feels to me more like thesis driven, but you know validated like is there a way you would say that. i'm still looking for that that perfect term for us, a lot of times terms up and using thrown around our culture is data centric, you have to make sure that you're guided in the right way. And you have there either have data that's validating your point or invalid any point, not to the point that you're causing biases, but actually removing biases from our decision making. I think the the antithesis of maybe creating Disneyland is I think there was a time 10 or 15 years ago early in social gaming where there's a lot of creativity and then in the midst of the fray there was some hope that you would just throw up something, you know, get it live and purely optimize through data science, the creation of an actual immersive. You know gaming experience, did you ever come across that I think those all failed essentially but I was wondering if you ever saw that going on yeah the term of gamification in a lot of things that we do. And while it works in some manners if you want to help exercise and spend more time in healthy matters, sure gamification of your own habits helps, but to the point where it may cause. Undo amount of resources on you and not in a good way that's where we don't want to go with roblox and the better verse. Yeah, so it's I always like to think there's a healthy tension where we're trying to have both innovation and come up with these topological changes like oh my gosh. We have the instinct this is big and we're going to do it with all of the great things of data science and optimization can you kind of talk about the balance of those two things like innovation topological change versus constant iterative growth and improvement. Yeah, it is interesting because it also comes down with our partnership with product teams. There's a portion of us in which we have to be awesome partners, be able to to shepherd through what we think is going to be that future vision and what kind of features we have. But we also have to have a little bit of policing of our own behavior to make sure we're not going down the wrong path and that also leans well with innovation versus optimization. I always use an analogy that I put in a blog post around rally car, I see a product manager as being the driver they're actually turning left or turning right. The data scientist in a lot of ways is that an navigational person sitting in the site next to him. We're going to tell you up ahead what the map the map looks like, whether we want to turn left turn right or throw that map out the window and start with the whole new map because of what we're finding. So I mean, I know there was a time when we were smaller and you could ask just a group of devs or players what we wanted to do and I know this worked at small scale, but can you share a bit how that might have to change as we take the long view here. Yeah, I think there's some obviously we can't put a million people in a room and ask them what they feel. If we could people are going to have louder voices than others, so what we want to do by removing all these biases we're using data as the voice of the customer at scale. We can understand what local developers maybe in the US might want versus those in other parts of the world, as well as on the the experiencing side of it and whether people want to have certain features or how they react to those various experiences. Okay, so let's roll up our sleeves without giving away anything that's shipping or you know that we haven't kind of shared at the same time it's fun to talk about some of the stuff you're working on to the extent we can. So is there anything that we can share Todd that you're working on kind of as a hint or a generalization. So sure put me on the fire here, I would say that some of the interesting things in data science that we're currently working through you know we talk a lot about experimentation with that being said though there's certain features that we might want to test that are going to have social network effects. For example, if we want to upgrade our chat system, if we set it up poorly, some of those effects are going to bleed into people as we communicate with our friends and some of the innovation that we're pushing forward is to have the idea of social clustering in our experiments. So that you and your friends and your your sub network within the social space are all getting that same feature and we can really understand both individual effects and those effects as a community. And that's going to move us to the next realm. So I also see a future in which I would love to be able to expose our experimentation platform to developers and they have access and be able to ask the same kind of questions that we're asking of our platform and be critical thinkers for the experiences they build. Yeah, I think this is a common theme on Roblox and behind the scenes as we go up and down our stack. It would be a wonderful world at some point in the future whether it's our persistence framework or event framework or analytics framework or a testing framework. It would be wonderful to imagine that same infra being used by us internally and the developers on Roblox at the same time. Absolutely. And I want to extend some of that thought here because if you would have asked the community 10 years ago, could a 10 year old build a online immersive multiplayer game. The answer would have been no, but today you can. I want to do the same thing for the community when it comes to data science. I want a 10 year old to be able to understand how to transform data, get some interesting questions and answers and then be able to put that back their thoughts and really be a truly critical thinker. Okay, I'm rolling forward historically and then rolling forward historically, I think at Roblox, we also had this thing where we would never run two experiments at the same time with the same user. And that just doesn't scale if we have a lot of product teams doing that. So can, so then various concepts came along. One is we slice the user and one experiment gets this and then one gets to that. And then I think there's a higher level concept where every user is in all the experiments at the same time. And then there's maybe even a higher level concept where, well, every user is in all the experiments at the same time and we can actually see some features affecting other features, which would be really interesting. We can kind of go through that swim lane stack versus matrix stack versus matrix correlation stack. Yeah, a lot of the early work in 2019, 2020 was revamping our experimentation platform. And so we can answer the micro decisions, whether a feature was good on its own. Now we're at the state of starting to roll out, how do we bundle features together or a suite of features over a longer time period to see. We can't apply to each other and maybe we only have one versus the next, or do they work together. I think a good analogy here would be thinking about utensils over the last few centuries, a fork, a spoon and a knife. You can pair some of this together and they work wonders or they don't do anything together. And we kind of look at that from a feature perspective as well in the way innovate what's different is, you know, we've had centuries to be able to work through the problem of utensils and where they benefit us. And we're still just getting into the early work of the metaverse and some of these observations are not going to be as obvious as we think guard, but hopefully in 10 years they will look obvious. Okay, here's the future prediction. Let's say we have two very unrelated teams working on new functionality. One is maybe over in the 3D avatar team and then one is on somewhere else search and discovery team. The same experiments at the same time and both of those experiments independently maybe improve our long term user satisfaction or retention. Do you think we will ever find a situation where those two functionalities at the same time. Do even better than the sum of the individual functionalities kind of like an accelerator effect rather than a degradation, you know overlap effect. That's a decent measure of innovation today is already producing product teams that are choosing these features or cross product teams. And when you bundle them together over time. Are they actually more than the summer their parts and that that's really a good. A good cycle for us to measure ourselves and be honest to ourselves. So looking forward a bit for both people in the industry maybe people who are studying right now comp sigh and thinking where they want to go. Can you share a bit about the technology and expertise going forward to help solve some of these challenges. So we have a huge plethora of data and it's only going to get bigger as we continue to expand knowing big data tools is going to be absolutely important and those are constantly transforming themselves. You know we have to log we have the transformative aggregate data and we have to analyze behavior on the platform for millions of users. Obviously it's timely but it's also costly so while tools exist in the industry today and they're effective they still need to continue to innovate and be able to process order of magnitude more data in upcoming years. Another aspect of this is no the sexy side of it is data science the actual output of what we produce. But what people miss is data engineers are just important or even more important in a lot of cases. We simply need people that know how to effectively use these tools the ones that both exist today, but also know the shortcomings and how do we build the next generation of tools tools assigned for distributed science analytics of the metaverse at the edge. Hey, do you think you know one thing we talk about is the notion that over time things move from millions to billions to trillions of events, you know millions of people on our platform are sending millions of events that the creators on our platform. You have these events and are trying to improve the how wonderful their experiences are how civil they are have how people enjoy them. And at the same time a lot of the creators on Roblox are just learning data science and the notion of how you would query all of these events is somewhat new. What I think we might have is that someday people don't even have to worry about it and I think the data scientists and you and a little bit of the engineer in me knows that some queries are very easy and can scan a lot of data and give results back really quickly. As some queries you know how many people who were wearing blue shirts three days ago who drove in the car made a friend five hours later and then came back and bought something you know two days later that's probably a more complex query. How fast do you think the tools over time are going to support those types of queries for your average new data scientists. I would like to see the tools conversion no more than five years because if you look at it 20 years ago terabytes was the size of data we're talking about. Now we're having conversations around exabytes of data that we're going to have to the weed through to really get to the insights. And so with that there's a piece of the tools but they're also piece of are we asking the right question. And are we building the tools to be able to ask those questions part of that is data scientists have to start thinking about how to abstract their work. There's a lot of common questions that we ask ourselves on a daily basis are you somebody adopting a feature for example. We don't want to get into the mode of the lack of scaling because we're asking those same questions of a person every time we need to be able to automate those and essentially abstract a data scientist mind in working through those into the tool set itself. And that's the only way that we will build a strong foundation we will ask the more novel questions as the metaverse starts to evolve. So so is it fair to say today when data science teams ask the worries I just shared. There might be a little work there like intermediate representations or things that maybe in the future we don't have to have exactly and we're still relying storytelling from a human perspective. You know somebody on my team would have to try to show the data look at all of the different ways that's pointing to and come up with a narrative and go off and commit somebody that narratives correct. I want to get to the world where we actually auto generate those narratives and not just that their narratives are for us to understand. We can also essentially take decisions on them in real time and move past those things that we commonly agree on into those more difficult questions that we know are going to come up with us in the next few years. All right. And then back to those candidates who right now are maybe have aspirations are attracted to data in this area. Is there can you share a bit about what we're looking for in those candidates that helps make them successful at Roblox. Yeah, that's a good question. I would say over the past few years of growing this team. We generally look for for forced traits. The first trait is around personal drive. We're a team that doesn't sit on our hands. We're not waiting for questions from partners are the product team. We're essentially leaders that are asking those big questions and building a roadmap to answer them. So I look for people that can push themselves. And the first trait I look for is general curiosity, which I think anybody in the world needs to have, you know, feeding off the previous quality of drive, you have to have this sense because a great analysis often generates more questions and answers. So you have to have that curiosity to use the product, try to break the product contextualize and embody the problems. Third, I look for betterment. Don't stop at identifying problems to solve, have a sense to be able to solve those problems and make the product better for the community. Don't throw numbers over the wall. That's not the type of team that we're trying to bring. And the last strong communication data science is still a new field. It's an effective data scientist will be able to communicate complex topics nuances in the way that we look at the data or simplify the information being shared. And we also have to influence our partners around to take some action. You know, like I said, throwing numbers of the wall without taking action. It's ineffective and adds no value. So a lot of this job is actually just prevents people to change their minds. Yeah, well, I'm really impressed with the team you've brought on so far. So I think something's working and I think the all of these qualities interlock really well with our Roblox values. You know, one of our values is taking the long view, which is highly innovation based and taking the long view and innovation, as we said, can be have a little tension. I think there's like the right balance of innovation culture, which is creating Disneyland with pure analytics driven culture, which we talked a bit. Can you maybe talk about this long view value and how it guides the data science side of things like is that unique? Does it make it harder for you to hire people because we're trying to do a lot of innovation at the same time or or any other color on that. There's a lot of data scientists in the field because we have a very wide definition of data science today, which covers product analytics modeling and a few other areas. The reason is because you want to give a very long runway for data scientists to think through holistic solutions. In a lot of roles within the industry, there's a very small swim lane in one or or another dimension here. And what that means is you typically try to locally optimize you're only looking at opportunities or you're only looking to optimize a particular model. But we have to know when is the right time to jump out a model because we're trying to squeeze water of a rock and look for the next big opportunities in right next to us that we're essentially being blind to. And that's part of the taking a long view data science here. Oh, hey, thank you for that because we don't want local, you know, we want to pop topologically into those next big things. I guess as we kind of start zeroing in at the end of our time here, if we look to the future. And you know, some of the big challenges we're going to be tackling in the future at Roblox, you shared a bit about the evolution from early stuff can you give us a little vision of some of the things we might have to be dealing with in five years here. And the five years time is I would love to be able to say that the data science community externally of Roblox is just a strong internally. And that means that we're providing strong set of tools, whether it's metrics and analytical tools, big data tools, experimentation platforms, machine learning platforms. I want to be able to democratize that knowledge at a very young age and get people to build a lot of things on their own that will never even dream of because they'll be able to innovate in ways that will never think of. That's really cool because I think right now when we think of Roblox in education, we think of design, art, creation, computer science, coding, production, community, we're not as much thinking of data science, but I can imagine a world where in the future that will be just as common. And learning to, you know, a classroom has created a Roblox experience there together looking at the data learning about the data science thing just side by side with learning to code or learning to create or learning art or audio. The thing is setting framing of the problems that I'm excited because I also have young kids and I want to be able to use Roblox to help them educate in a lot of the STEM aspects. Well, we do too and I'm very optimistic about Roblox's role in STEM education around the world and data science is a big part of that. So Todd, I really appreciate you sharing more about this exciting discipline and joining the conversation today. I'm very excited to see you in person today as well. Well, thanks for having me, Dave. It's been great chat with you. Alright, thank you, Todd. That's all for this episode of Tech Talks. Thank you all for listening. To learn more about careers at Roblox, visit roblox.com/careers. I'm Dave Bazuki. See you again next time.