Useable and meaningful data in the design process

There is more nuances between behavior change and conversion performances. So actually, that's something that makes our design works even more interesting than before. We remove the guesswork in anything that we are designing or even before we design some solution. It sort of helps us like to prioritize which problems to tackle first, right? Hello and welcome to the Design Explorers, a podcast by the Agoda Design Team. Agoda.com is a global digital travel platform where you can book hotels, vacation rentals, flights, and airport transfer. In this podcast, we will be sharing the awesome work of our design team. Discuss interesting trends in relation to design and travel and talk about product design in general. My name is Nacho Miamin and I will be your host for the show. Today's episode is all about the relationship between data and design. Here at Agoda, we trust data to inform our design decisions and provide the best experience for our users. I have set down for a conversation with Joshua Jo, a design manager and Javier Law, a senior UX designer to talk about how designers in our team are using data. Have a nice listening and I hope you will enjoy the show. Welcome to our first episode of the Design Explorers. I am very happy to have Joshua and Javier here with me today to discuss about how we are using data in the design process and how they have started its initiative of making data more accessible and meaningful for our design team. Joshua and Javier, welcome to our show. Thanks Nacho. I'm very excited to be here. Yeah, it's a pleasure to be here. Before we start with our topic for today, let's give our listeners some background about who you are and what is your role here at Agoda. Joshua, maybe we can start with you. You've been at Agoda for almost five years now, starting as a UX designer and growing to becoming a design manager. Yeah, you're right. I've been here for almost five years and I witnessed the design team's growth over these almost five years. So for the first three years, I worked on the consumer facing product. You can understand as a web and also mobile app where the users can use to book their hotels for their trip. We practice a lot of A/B testing at that time because Agoda is a data driven, so we rely heavily on that. Now, thanks for the opportunities that I'm managing a team working on at the enterprise site. So basically, how you understand the enterprise site, so you can simply understand that, you know, the product that are the ones that we allowed our hotel partners to manage their inventory on the platform and also the product for consumer experience specialist to handle the customer support. So in these particular scenarios, we define the term of user or say customers are actually the hotels or our partners service specialist, but not the travelers who, you know, used app to book the trips. And Xavier, you are a senior UX designer that has been working on the experience of both the Agoda Web and app for almost four years now, right? Yeah, you're right. I've been here for four years now and mostly working on the travelers site, which is involving the both platforms, which is app and also web. And I sort of started back with actually. So when I first joined, I worked on booking form and then moved back to the funnel, which is property page and then also search page basically. So yeah, it's pretty cool. And then I've learned a lot throughout these years and then the thing about like being able to work in full funnels is that it sort of helped me to understand what type of data that is available in each funnel in each page and then also like to understand the business as a whole. So yeah, it's been pretty long journey and also been full journey like it's quite fun actually. And luckily, I still haven't feel bored yet. Yeah, still have a lot of stuff to get great learnings here. I think that's why, you know, we are still working, you know, I got to hear. Yeah. In our first episode today, we are talking about data and here at Agoda, you know, we as designer have a lot of data at our disposal, but it is not always easy for us to include it as part of our design process. And you guys are actually working on making it easier. But before we go into that, let's begin with the basic question of what does data even means. Okay, so just for this podcast purposes, actually, let's all agree that when we talk about data, we talk about we are talking about quantitative data, right. So within Agoda itself, like the types of quantitative data can be anything that we are tracking on our product. It can be like number of clicks, number of visits, like how many people are have seen this this element or UI element and things like that. So yeah, so basically those are the type of data that we are reused often, right. Mostly in the front end facing world. Yeah, just not have you mentioned about quantitative data. So I think there is a very simple way for us to differentiate quantitative versus qualitative. For us, you can if you go to Agoda.com, you are trying to do a destination search, you're typing the city in the click the search button and when you go to the search result page. So if you if you notice that we have a filter on the top, the filters can the users can use to filter for the price range for a special offers or something like that. So the filter on the top is something we sorry, okay, so for the filters, if the users use this filter to get the result and make the bookings are not and this actually is the insights of qualitative insights. So anything that you observe, right, anything that you observe that if the if what is the user behavior look like, that's how the user behavior on the on the site, that's the qualitative side that that's the quality of insights. And so if we if the user actually use the filter and we know this is a user pattern and how to, you know, how to understand the pattern is is, okay, what is the percentage of the user actually use the filter to make the booking, then that's then this is a quantitative insights. Yeah, actually the actual number like, for example, and how many users are actually using free breakfast, for example, and in book. So that's quite qualitative. And also to expand on the qualitative side, like the easiest way to to look at it is also like any type of data that you get from user testing, for example, like user lab, right, or yeah, I think that's the easiest way to distinguish within those two types of data. So quantitative help us understand what is happening. Yeah. And qualitative help us understand why it's happening. Yes, exactly. What is happening and also to which extent that would that happen. Right. Yeah. Okay, so let's talk about your ongoing initiative, which is basically trying to make it easier for designers in our team to access and understand data. What and how, you know, made you start this initiative. Okay. So do you want an official answer or a personal answer? I want your answer. Okay, cool. So yeah, officially, I think everyone's thinking about what is the next step for UX designer to succeed to succeed in the internet company like Dakota. So I think for data is something that we can use as a domain to upscale self. But actually, for personal story, actually, that that was a very funny. At first, you know, I when I was working on the product that, you know, specifically targeting a certain user segment. At that time, I bought a, you know, our data team a lot by sending them a lot of emails to asking some stupid questions sometimes. But most of our questions are, you know, data related. So what, what I, what I experienced is, okay, so I send the question to them. I usually takes is usually, is usually took like two weeks before they actually, you know, get the, get the questions answered. And sometimes I need to, you know, rephrase the question again and again. So get so so that I can get the, the similar data points result based on the same theme that I asked for. So and at that, at each times, I think the question is very easy because of the waiting queue is very long. I need to wait another, you know, months to get my question answered. Which means I missed a lot of good opportunity opportunity for me to pitch a very good design initiative to the, to the key stakeholders. So basically you're frustrated with this. Wow. Right. Not really frustrated because we have an official procedure for, you know, managing those data requests. I add it to all of the teams. And design team is also part of it. So actually, the data team is very supportive. But because I go, that's a very big company. We have thousands right in employee here. Yeah. So they cannot, they cannot really answer every, you know, person's data question every request. Every request that they have like probably thousands of requests per week basically. Yeah. Yeah. Yeah. You can see that workload is very, it's very high. So I think instead of, you know, suffering from waiting, I think, hey, what if I learn how to query by myself? It's the idea at the beginning because I'm seeing some, you know, like crazy coatings. In the SQL query that, you know, the data team is working on. But I think maybe for some simple question that I can start with. So that's why I started to learn the SQL query. You know, but that, you know, everything was changed at that time. So I think learning SQL query actually opened another new door for me to, to step into the business world of, because previously we were going to design. And we, we are not really having a very strong standpoint in the organization. And sometimes if you want to drive something from design standpoint, the flow that we went through is we need to, you know, be well prepared for the rationale of why we should do this because, you know, is a performance driven. But now we have, you know, capability of, you know, data. Now we can ask the data, you know, the, the right data question. And then we can, you know, back up our design decisions. So I think that's, yeah, that's, that's simply how, how things started from, from my side. And actually, because we are working as a team and, and, you know, we, we have been here for, for a while. And the team actually grow for, you know, grew for a while for the, for the past couple of years. And we also need to think about what is the next step for the team as well. So team material maturity level are related to the design quality, but also could be related to the data driven culture, because Agoda is data driven company. So that's why that's the initiative come out. And Havir, how did this initiative started for you? So, to me, it's very natural in a way, I think. So basically, it all started with like Joshua mentioned is like the need, there's a need for the team to, to level up. And also working with all the product managers, makes me realize that we also need to, to speak the same language as them. And because all the PMs here are basically only speaking one language, which is data, right. So for us to work smoothly with them, I think it's, it makes sense for us to understand data, right. And, yes, starting from that, we, it's sort of like naturally goes into this world, right. Similar to Joshua, and then Joshua also mentioned that he learned SQL, and has also started learning SQL by myself, right. And then which is not, I had a background before, but it just like, need practicing, right. Yeah. So, yeah, so basically that's how he started. Yeah, yeah. Cool. So let's talk more about this point. Why should we as designers, you know, actually care about data and why should we consider including it as part of our design process. Shouldn't we just leave it for the experts? To me, I think the best way to think about data is based so why, right. The why is basically it's quite simple, right. It helps us validate our design decisions more, right. So we remove the guesswork in anything that we are designing or even before we design some solution. So it gives you like a hard proof to what you should design or how you should design it basically. You know, what I think about is you know, designing data relationship more like the two components in the whole loop of the product development cycle. You know, we are designers, we are not artists and we propose our design to solve both user problems and also business problems. So instead of being artistic, I think we rather need to be more rational to include data into the design process to get more buying from stakeholders. Like I said, you know, we solve, we solve both user problems and business problems. You know, the definition from myself like business problems are usually quantified as conversion rate problems by our product managers. So basically they focus on the conversion rate. And to solve these conversion rate problems, I think product managers will form high level product hypothesis. For example, you know, a very typical example for the product hypothesis could be like based on our prior data insights or data signal or competitive benchmarking. If we build up a new feature, then this new feature will bring us acts amount of conversion rate increase. So this sounds like you know, the business side need to be achieved. So product hypothesis is more about you know, how will go like I mentioned and high level measurement. And to designers, the ways to form design hypothesis is more about interpretating the factors that lead to the high level product hypothesis through observing the user behavior changes and the psychological influences that behind these behavior changes. Right. So though psychological influence might be a qualitative term, but you know, mass majority is, but it is very much reflected in the result of the quantity. Right. For example, how many users changed their behavior and the good thing is we have the luxury of improving those behavior patterns by AB tests. So I go that we have thousands of tests running for each month. So you know, study and get in the getting familiar with this kind of data actually helps us sharpen our instincts. And you know, for long term wise, I think makes us propose design faster and with less wrong design hypothesis possible. So that's why I think data is important and should be included in our design process. And one thing I think I might need to be, I think one thing I might need to point out is, you know, design hypothesis doesn't really necessary positively impact the product hypothesis every time. Since the behavior change might happen because of the design, but the conversion performance might not be predictable, even though we have very strong confidence on the possible behavior change. So I think that is so there is more nuances between behavior change and conversion performances. So actually that that's something that make our design works even more interesting than before. Okay, so we understand and agree now that data is important and can be very helpful in the design process. But, you know, data can mistakenly be considered as something boring and a subject that deals only with numbers, especially for designers who are sometimes perceived as visual creatures. What are we doing to make it more accessible and less intimidating for them? So there are a couple of initiatives to make it less intimidating, right? And then it's all in the form of learnings, right? So what Joshua did before is to organize classes with partnership with data science team, right? So which is very helpful. And the goal for those classes is to basically tell our designers what are the types of data that is available and what are the tools that is currently being used by everyone here, right? And then there's another initiative is also form of learning, which is the data guild that me and Joshua is currently running where designers, so we divided into two sessions where the first session is basically to learn how to form a design hypothesis better, right? And then the second session is more of a technical side where like learning about the tools, like learning about all the. So you guys are breaking down into different components to help designer better understand. Yeah, yeah, exactly. And yeah, I think I think I have pretty much mentioned all the things that we are trying to do right now. But I think for a simple start, it's very easy actually, maybe reach out to whatever the people from the data team or the business intelligence team reach out to them and see because we are UX designers, we have the strength of making their tools easy. So let's do a quick audit for their tools and start to build up the connection between them, start to do something for them. And actually we are asking something back from them is, okay, so we should we should build up very long term relationship and then we can actually make the help me to happen for all of those activities, like have you mentioned about. Do like a data guild, data trainings and also the follow up like learning clubs that we can to do together with them. Yeah, yeah, actually, Joshua made a good point like I cannot stress enough that building a relationship earlier is the most important thing, right? So just looking back, what I did with the design experimentation team is that they need a designer and then back then we sort of don't have enough resources for it. We still do stuff for them designing like the online analytics tool for them and on the side, like outside of our current scope, which in turn makes them wants to help us more, right? Which is pretty cool. Just be positive and reach out to them first and try to do something for them or maybe even organize a launch with them and talk about, hey, what are the, so if like have you mentioned about, I'm working on the booking form. So what are the data points that we are tracking on the booking form so that you can start from those kind of data point conversation and after that, you can, you know, ask them to navigate navigate you to the database and show you, okay, so for the booking form, these are the data that we're tracking and what are the data is actually related to the design related to the user behavior that you can start a seven second conversation to follow those, you know, conversation that actually you need. So in doing this process, what are some of your biggest challenges the ones you are facing or faced in the past by trying to make it easier for designers to understand data. The most challenging thing for me is basically including this extra component into my design process and then also sharing it to other people, right? I guess when we started working on data, I think there's more questions like pops out here and there, right? And especially when other designers are relying on you to sort of have all the answers, like those are the toughest part because I think even now, even me and Joshua probably wouldn't have all the answers, right? And then including that into our process and then makes things a bit slower in terms of executing design, right? And yeah, basically those are the main challenges that I am facing right now. So it's an ongoing process that you guys are still learning by doing. Yeah, I think the cultural shift is actually the biggest challenge. At the first time, we initiate that data-driven design or it's not really a big topic like that, but at first we're thinking about we should include more data thinking to the design process. And we actually advocate this concept to different people. For example, sometimes the PMs actually don't understand, hey, why you guys would like to learn these data tools? For example, beginning, they don't really understand why some of the designers might like myself and have you learn SQL. So I think for learning SQL for myself even for Harvey, I think it's not about learning the SQL clause themselves, but it's more about learning how we track and store the data so that we understand the business components better. Because what we open the door is we open with the learning execute or actually learning the open the door for us to step into the data world and also trying to look into the business components instead of the high level ones. Because I think for designers, we have tried so long time just focusing on the high level goals, but what are the actual components waiting that something that we should break down and do a deeper look at it? So that's why I think we should we should shift the culture and actually advocate that data is a very important process for the designers. And after we have tried to learn those tools and try to learn a data domain, I think for the bare minimum, I'm seeing a lot of good designers actually asking good data questions from now on, which is very visible. Like before, we don't have this capability. Speaking about learning, what are some of your biggest lessons learned that you kind of wish you knew before you starting this process? So to me, one of the biggest lessons, and I also think it's one of the most important lessons is when you look into data, you are more prone to confirmation bias. Say for example, if you have, especially when you have a design solution ready, at all, especially if you have like a strong belief in one ideas, then you sort of can be like sort of biased in terms of like what type of data that you're looking at. So that can be dangerous basically. So confirmation bias, you mean you're trying to prove your hypothesis, you try to prove your design belief basically. So those things can you can easily do it with data because any type of data that you're looking at, you can sort of twist the type of data that you're looking at to be confirming your initial belief design belief basically. So those are the things that it's quite important and quite dangerous, right? Yeah, totally agree. So I think for my personal experience. Like you mentioned about the confirmation bias, right? And also I made a lot of mistake of selection bias. That's that's the beginning mistake, but I think it's okay. There's a very good learning for me as well. During the process, I've been talking to different PMs and also data guys and also developers and trying to avoid that happen again. So basically the mistake was, you know, from a user standpoint, when we look at the, I got a website search result. It was the search results, the search result page looks a little bit busy for you. We display a lot of a lot of information like different badges of benefits, like breath for us, breath for us. It is important, by the way. But we have different, so many different badges, color for graphics visuals happening on the cards, which made the card, which makes the card very big and very busy. So at that time, my hypothesis is, if I am, you know, from a user standpoint, and also I, you know, from my own experience of booking, make bookings on the go to actually not really looking at those components on the card. So that's why I have formed hypothesis. If we, you know, if we simplify the UI that maybe can make the user experience better, so because we can display a core messages to the user so that it makes it easier for them to make decision. But how do we actually quantify the term of simplify? Because we simplify the UI, you can simplify that color, or you can simplify the information, or maybe you can shorten the language that we used on the card. But my approach at that time is make the card smaller. That was the, that was my personal interpretation of simplify it. So because my hypothesis is simplify it, which means we make the card smaller, can help the user, can actually help help the conversion. That's my hypothesis. So that's why I go back and take a look at, you know, the past experiments, the past AB test that, you know, could possibly make the cards smaller. For example, like if someone removes some components in the card, if someone changed maybe the height of the card. So I was trying to find out those experiment to prove my hypothesis and actually did. A lot of query, by the way, at the time I already knew how to do query. So that's why I pull out a lot of experimentation and trying to use that to support my hypothesis. And when I show it to our senior PM, he's, he's, he's very nice. And he, he told me that hey Joshua, you are going in a wrong direction, because you cannot avoid the selection bias. Now the time I asked him, what is the selection bias? Basically, you know the result first. And then you're trying to find the text or the find the AB tests supporting your hypothesis by those results. But you are by purposely you ignore the rest of ones. For example, if, if the rest ones by simplifying the cards actually do hurting on our conversion, but I ignore that. So that's the only select the one that was winning. Yes, yes, yes. That's that's a typical selection bias. So, okay, I think that's a very good feedback for me. So that's why I follow up and, okay, this time I, I use the same query and I, and I pull out pretty much the, pretty much the same experimentation, but, but more with experimentation. I have more, you know, results, not just for a positive impact on the bookings, but also negatively or maybe a flat test. So I put together or every experimentation that I can found that physically that I think physically can make the card bigger or smaller, because you cannot really look at make it smaller, you need to, you know, compare the both ways. So that's why I pull out experimentation. I could potentially make the cars, both smaller and bigger, and I don't put the conversion numbers. I just put the list of the experimentation and I sent to the developers and asked them to, you know, to tell me technically, is this experimentation, make the car actually smaller or bigger. And then after the reading, I get back there is the list from the developer teams and I put the numbers for each experimentation. And after that, I feel more comfortable and more confident to propose the same concept again, because I'm still seeing the parent, but that parent is more solid than before, because you back up with the developers, you know, how to say developers measurement. Yeah, and developers evaluation on the, you know, if the test can physically increase or decrease the size of the card, and that actually makes me more confident. So essentially what both of you are saying is don't fall in love with your solution and try to be objective when you are working with data. Correct, that's correct. That's the overall suggestion and trying to avoid that. I'm pretty sure that people could, especially for designers who step into this world, they could, you know, make the same mistakes that like we did. So let's talk about this new designers and designers that are very new to the idea of data. And also, you know, here at Agoda, we have the luxury of being empowered to access and use data, but some of our listeners might be working in smaller companies and with less resources. What would you recommend those guys? How can they start and approach this subject of data in that case? Yeah, this is a, this is an interesting question, but we, I don't think there is a very good answer to that. I think the concept data, you know, data is all about measurement or say assessment. So I think they could be reflected in the form of number, but also could be reflected in the form of certain customer behavior. Right. So for designers who don't have the luxury of accessing data team, I would recommend them to start from the business result. I mean, you know, what the business is going to achieve. And trying to reverse engineer the business result to see what other components in it. I know a lot of times that the result was achieved by cross functional teams. So it could be joint efforts, but what are the parts that mostly relevant to design. So thinking about that, I think that's something that we as designer can do. And also trying to split them out from the expected business outcome and trying to figure out how you can make sure you're doing good job, you know, doing good job of influencing the user behavior. So basically, I think you know, get yourself closer to the data thinking, even though for the short term, maybe data, data support is not there is not very feasible to you, but I believe running every business, there is no escape for anyone to touch data. So there must be data here and there for you to discover. So start from the result, follow the path leading to the result and you won't miss the data. That's that that's what I thinking about it. Yeah, to me, the other way to utilize to sort of start into this world is basically to utilize whatever you have right now within your company. Like Joshua mentioned about start with the end results, right, like I'm pretty sure the business intelligence people, if you have any or even your CEO might have an end goal in mind, which means that obviously for some company, they might be focusing on growth and some company, they might be focusing on on any other metrics, right. So I think start by asking them like, okay, so what are the things that you want to achieve and then have we track this or not and if you haven't track anything at all, then I would suggest start by doing that, like at least talk to developer and ask them like whether you guys are tracking the your product or not, right. Because I think right now nowadays it's very hard for business to survive without like any data at all, actually. So better understand the objectives and the goals. Yeah, better understand the objectives and goals and also better understand like what are your current resources basically, like you said, like not not everyone have data team, right. When you can sort of start with your developer and then if they don't have tracking and then you sort of can as a designer that you need to be proactive enough and then ask them to sort of add the tracking basically. Okay, we're coming to the end of this episode and Joshua and Javier, I want to thank you for taking the time today and discussing with us about this interesting subject of data. Before we close, do you have some any recommendations and books or some advice for our listeners? I think, yeah. Okay, my recommendation for the designers who like to step into the data world, it will be, you know, start from asking data related questions. So, you know, for ourselves, we ran the sessions called question burst on data points. By the way, question burst is a very useful brainstorm tool that invented by a whole Gregerson who is the executive director of the MIT Leadership Center. So what we did was based on the product hypothesis that came out from our product managers. So we ran the excess anthropo question burst on trying to, you know, ask the question that round two topic. One is what are the user behavior data points that we should look at before and after the new feature is built. And the second one will be, and if the product hypothesis is true, what will be expected is the behavior pattern change should be reflected on those data points. That's something that we start with. Nothing could be a very good exercise for designers who like to stepping to the data world to try as a big, as a first step. Think to me, if you talk about books, there are books that I think is quite important. To me, there is one book called a practical guide to designing with data by Brian Suda. And then I think start by reading a lot of books is quite important. I mean, of course, like you wouldn't be able to apply everything that you read. At the start, but trust me, like somehow, while you're doing your day to day job, you sort of can receive an idea of like from the books itself. So I think it's pretty helpful also. All right. Thank you guys. Thank you very much. Thanks, Nahum. Very happy to be here today. Thank you so much for listening. If you enjoyed this podcast, please leave us a review on Apple Podcasts, Spotify or anywhere you are listening and share it with your friends and colleagues. Don't forget to subscribe to our show to get notified when we are releasing a new episode. And if you want to learn more about the work of the design team at Agoda, visit agoda.design. Thanks again for listening and hope to see you in our next episode.