Design Research

This is the Clear Left Podcast. Earlier this year, Clear Left ran an event called UX Fest. It was the online version of our regular UX London conference. One of the speakers was Theresa Torres. She began by differentiating between discovery and delivery. So all product teams have to discover what to build and they also have to deliver that product. If you're familiar with the double diamond model of the design process, this should resonate with you. So we look at discovery as all the activities that we're doing to decide what to build. So think about all the decisions that we're making, how we making those decisions, what data are we collecting, who we interacting with. That's the first diamond in the double diamond diagram. Whereas delivery is the work that we're doing to build and ship scalable production quality products. That's the second diamond. Theresa's focus is on that first diamond. She's written a book called Continuous Discovery. I define continuous discovery as at a minimum weekly touch points with your customers. By the team building the product, where that team is conducting small research activities, in pursuit of a desired product outcome. Sounds great. So what does the team look like for continuous discovery? It starts with this idea of a product trio. So this is becoming a more common model. Some people call it a triad, some people call it a three-legged stool. It's this idea of the product manager, the design lead and the tech lead, being jointly responsible for those discovery decisions. With a trio-based model, we're basically saying, let's have these three rules work together from the beginning. Let's have them drive our discovery decisions and jointly decide what to build. Wait a minute. A project manager, a design lead and a tech lead. That sounds great, but what Theresa has described, it sounds a lot like research. After her talk, I asked her, where are the researchers? So a lot of people interpret the trio as, we're leaving user researchers out. So I want to be really clear, we're not. The vast majority of companies don't have user researchers and they don't have enough user researchers. It may be 30 years from now, we'll get there. Hopefully it doesn't take that long. If we worked in an environment where most teams had a user researcher on their team, I would be talking about a quad. But that's not our reality, right? So the idea of a trio is trying to reflect the resources that are available to most teams. Ouch. So in an ideal world, researchers would be involved in the continuous discovery process. But when it comes to research, we are not living in an ideal world. Greg Bernstein, a researcher, also spoke at UXFest. On LinkedIn right now, there are about 127,000 opened front end designer positions worldwide. There are roughly 53,000 open product designer positions. There are roughly 49,000 open front end engineer positions. I think you know where I'm going with this. There are roughly 2400 open user research roles on LinkedIn right now. And here's the thing, most every organization knows it should be doing research. We know that we should be gathering the information we need to do our best work. And we have these models that tell us to, for example, empathize and prototype and test with our users. Or we need to analyze and then test and then prototype or we need to enter a discovery process where we focus group and then interview and then survey. My point is we know we need to put users at the center of our work. And despite knowing all this and despite knowing that there's these models that we've made that demonstrate this, the deck is stacked against research. So both Theresa and Greg gave the same message. Researchers are thin on the ground. But even if research isn't a role at your organization, perhaps it can be a skill. So if you aren't a researcher and I'm guessing you're not because again, there are very few of us, but there's a lot of everybody else. But if you're not a researcher and you want to bring the practice of research into your work, what can you do to get those missing perspectives? You can start within your own organization. You can start with support. That's usually the best place to start. And it starts with one Slack message or one single email. In the early days of MailChimp, when I was a researcher there, I would send instant messages to the people on the support team. Whether I knew them or not, I was just pinging them out of the blue. Let's say I'm working on a project that say, "Hey, what can you tell me about our building page?" Or, "I'm just curious, what is the single biggest complaint? And what might we do to fix it?" You know, what's the thing that would reduce support tickets by 10%. I wanted an additional perspective in my work. I wanted that context. Another researcher who used to work at MailChimp is Stephanie Troth. She's also worked as a researcher at Clear Left. Hi, I'm Steph. I am a strategist and researcher as well as a writer. I asked Steph if research could be a skill rather than a role. Shouldn't the well-rounded designer be able to do research? It is definitely a different mindset. I've won all these hats. And I know that if I have a designer hat on, it's not the same as a research hat. The thing is, you can't wear both at the same time. I cannot go into a room and be a researcher and a designer simultaneously. Because when you have a researcher hat on the research mindset, your goal is to put aside your own ideas of what is right, wrong, true or false, or what is good or what is bad. When you're a researcher, you're going to space and I'm going to learn everything I can. I've got some assumptions. Yeah, I got these. I can throw them out. They're just assumptions. I've got some hypotheses. Yeah, I can throw them out. They're here. So when you're a researcher, you are not solutionizing and that is really difficult for most designers to do for my press experience. It takes practice. But on the other hand, when you're coming into a situation as a designer, you're thinking like, "Oh, right. How do I come up with potential solutions?" So the mindset's completely different. And I know I cannot do both at the same time. Here's another former clear left researcher, Mite Atondo. So you can be a generalist, but I guess I is only going to take you so far. The thing is, you need probably different sets of skills to conduct research and then to create design concepts and then to actually create the final design. So I'm of the opinion that I is better to have teams with people from different skills that can work together across that process. I asked Mite to define some terms for me. Like, what's the difference between qualitative research and quantitative research? A very simple times qualitative research is about understanding people's perception about something. So understanding their stories in their own words and quantitative information is about understanding volume. So how many times something happens or how many people say something. So I guess the difference is one is about stories told by people and the other is about volume. But this categorization of qualitative and quantitative research isn't the only dichotomy. There's also generative research and evaluative research. Mite describes the difference using the familiar double diamond model. I guess you could say the first diamond, you would explore people's context and then potentially what people's needs are and what people's behaviors are to start eventually thinking of a solution that could actually help them. So some people call it generative research. The one you would conduct on the first diamond, so the more exploratory one. And then the one you would conduct on the second part of the diamond would be called evaluative research. That's a term that's used in the design industry. I don't think that comes from researchers, but that comes mostly from designers, I think. And then on the second part, it's more about people evaluating something that you have created. So the first one probably is less product focus. This is more context focus, whereas evaluative research tends to be more focused on the product or the design that you're working on. Here's Clear Left's Strategy Director, Andy Thornton. I think generative research is about exploring problem space. It's much more open-ended and evaluative research is much more about testing a thing, testing a prototype, testing a hypothesis. And that's how I describe the difference between those two is if you're talking about the scientific method, which I know makes some researchers nervous because especially design researchers, it's not entirely a rigorous scientific process. But evaluative research is when you've already formed a hypothesis, when you want to validate or invalidate a hypothesis to see if it's got some truth in a variety of scenarios, whereas I think generative is less about getting as concrete an answer as possible and much more about finding the right questions to ask, then need to be solved through a creative design process. So the first one is about exploring the problem and the second one is about exploring the solution. So I guess in the first step it's a bit more general and it's a more broader view. And then once you have that understanding of the context and the thing that I'm talking about, that people are trying to achieve and people's needs, then you can go and evaluate a specific solution. Genre of research is the type of research that's going to bring you more questions. It's going to broaden up your perspective. Evaluative research tends to be more about, is this good, is this working like it should, so it's quite narrow in terms of outcomes and what you're trying to establish. I see it as a spectrum and the exciting place tends to be sort of like halfway in the middle. I love genitive research but often you can get back a lot of data that you may not know what to do with. So sometimes an interesting space to be is in the middle where you bounce between the two based on the data that you get back. If we have something, like prototype or something that you're ready built and you're trying to decide is working as well as it should, you might actually get and be open-minded that you might get answers, you are not expecting at all and let that guide you to say actually we need to understand the landscape far more. Then you sort of swing back on one side and say okay let's try and understand the broader questions. You can also do the other way, you can start out with broad questions trying to understand the context and then you see the specific problems that people are struggling with and you go hey here's where we can try out something and see if it helps us understand a bigger picture better. So it is, for me it's not too separate cats, it's a spectrum and I feel like when you're doing effective pragmatic research you're bouncing between the two all the time. Research is about the present and design is about the future so I guess it depends on how far you want to go in the future. It depends on what you're trying to achieve. So if you're thinking I don't know if you want to explore the future of something in 50 years, maybe it's enough to know general trends about things whereas if you want to launch something next year maybe the level of detail that you're going to need to inform those decisions is going to be a bit bigger. So I think it depends. Obviously research can reduce the risk because at least you understand the context and the people you are designing for. So that helps you frame decisions because you understand what is going on. You have a better idea of how something that you create is going to change the future based on what you understand about the present. So research can help reduce risk which is great but there's a danger in pushing that too far. I've seen organizations where everything needs to be tested before being launched for example. So you have a bunch of researchers just conducting research about very small things about like very small interface decisions. That is the least impactful kind of research you can actually conduct. You're missing the value that a bunch of researchers could add to an organization just by you know asking them to conduct research that the only aim of that research is to reduce risk. So I think it's a bit problematic to think that is the only role of research but at the same time obviously it can help you reduce risk. Research can be a form of validation right? That's Fawns Morris another speaker at UX Fest. He works at Netflix where they do a huge amount of evaluative research. But I think research is really necessary because if we're coming up with these decisions and our team here in California how are we really going to know what people outside of our team think about that without doing the research. So I think the research is really more of education first than just validation. It's us being able to learn from these ideas that we have does this really work for our users. The validation comes second to me as opposed to the education where I need to know what my users think about this whether it's good or bad. I really value when I get feedback that may not be this is the best design or this is completely opposite of what I thought because that's allowing me and my team to now think about this from another perspective and having the more empathy for our users. So I'm a super super supporter of research. So a lot of us have adopted what I call a validation mindset where we do most of the work up front when we're all done and we've got the prototype ready to go we've designed everything. We put it in front of customers and we validate did we get it right. The challenge with the validation mindset we do need to do that validation research but the challenges we're often getting feedback too late right at the point that we're getting feedback our engineers are waiting for it to go into the next sprint. We have very little time to make changes and we tend to just make the small cosmetic changes and we don't have time to make the big changes that would dramatically improve the product based on our customers feedback. I do have a problem with using the word validation. I have a serious problem and I know I've clearly left we use that. I've had conversations about this with people in the studio because I think it's not about validating. I don't think the role of research is validating design. I think the role of research is explaining a context. So describing a context from the point of view of the people that experienced it in a way that inspire us action in a way that inspires design in a way that helps us frame, imagine and facilitate a better future for the people in this context. When it comes to evaluating solutions I think it's about understanding what we think works well and what we think doesn't work well but that doesn't mean validating. I think if you go to for example a usability testing session with the mindset of validation that is going to influence the ways in which you ask questions the tasks you create. We know we're biased already so I think if we go with that validation mindset we might end up maybe increasing our chances of confirmation bias for example. So we are just going to be looking for the things that are useful for us to validate whereas we should be I guess challenging the solution and capturing the data that maybe supports some design decisions and capturing the data that goes against some design decisions and we use that to assess the solution by it's not that we validate. I think sometimes designers maybe use these stems to increase the confidence of the stakeholders so we're going to validate something and that sounds very like people are going to trust me and we're going to validate this and then we're going to go within but researchers in other contexts would say for example instead of validate hypothesis they would say they are going to falsify hypotheses and that is a different approach so I don't think it's about validation I think it's about gathering information to assess design decisions and maybe as a result to that process say yes we think that was the right decision to make and in a way you validate that but that is a second step I don't think you should go to any engagement with people to assess and a design solution thinking that your role is to validate that. The trick I think in generative research is to understand the assumptions that you're coming in with and then be very very open-minded and think actually I got my assumptions completely wrong, throw them up the window and let's reform our assumptions because when you're doing generative research it's the wrong stage to have hypotheses, hypothesis up towards the more evolved with end of that spectrum. If I only had my assumptions reinforced I might have done something wrong, to be honest with you, for me a measure of how good of a researcher I am is do I find something I do not expect to find. If I found everything that I expected to find I might have done my job quite so well. From everything I've heard it sounds to me like there's a narrative arc to how researchers treated in an organization. To start with an organization might not have any researchers even if they say they get the value of research and then when they do higher researchers those researchers only get to do a evaluative research. What's the next chapter in that story? Is it even possible to shift from a evaluative research to the bigger picture more open-ended generative research? It is possible to shift a team's outlook from starting with evaluative to generative. It takes about 18 months or two years to do that process. The idea of evaluative research tends to be either a checkbox, it's not working, let's fix whatever is not working. It's the band-aid and you always start with that band-aid. But the more research you do you start making people think about the broader questions little by little and that's how you get some bigger strategic or generative piece. The other option sometimes is for me I try to work with the highest level this doesn't make it that I can because the other questions tend to be the broader ones and the more fundamental ones. For example in your episode on services I and we talk about all the pieces to make a good experience. In the same way research has the same mindset where for me it's like where are other decisions being made that is causing a better bad customer experience or a bad user experience. And often it's at the business level. And so if I want to change the user outcome, I almost have to figure out where that decision point is. So very similar to if you're coming in as a stretchist for user experience, but I'm also a stretchist. And maybe that's why I think that way. For me, it's about where the decisions get made. Because you can do all the research you want. And if nobody listens to it or acts on it, why do you spend the money? There is such a thing as research waste or research debt, if you like, where you have a lot of research, but nothing gets done. And often the conversations do have to happen at this stakeholder level in order for the research to make an impact. For research to make an impact, there might be some uncomfortable truths to be told along the way. I think we want to tell the truth. Sometimes the truth is uncomfortable because it would actually say the market that you've been trying for the last few years is not ever going to work because people are not able to buy it. That has happened to me several times in the research work that I've done. But then obviously you spin around and say, I've just said a few millions. The challenges people don't like ideas being killed. I know that designer struggled with this too, the whole concept of someone has an idea and they're stuck with it and they're going to go with it. And when you've got something like research coming in and say, actually, we don't think it's going to work. Nobody wants to hear that. Actually, one of the interesting issues we have in the growing and the careers of researchers is we are not always taught how to frame the truth, which is again why, for me, speaking to stakeholders in the early days is important because it helps you frame it the way they need to hear it. I think by nature of research, as would want to tell the truth, I think often we are constrained. One of the things that got me into research in the first place was some really hard knocks through a startup where I had somebody else help me with the usability. However, it didn't matter what we did. The CEO wasn't ever going to listen and this happens quite a lot. You get people in positions of power who think they know everything already, there is to know and they don't need to hear anything new. And if the narrative internally in an organisation is that way, any research will not really have real impact. What we're up against is the fact that people think they know what there is to know, that they are geniuses and they are going to succeed. What we're up against is the lack of humility in the industry. Generally, you get two camps. You get the people who believe in research and come from the design background and understand the value of it. And then you get the other camp where it's. Yeah, I know this industry. I've been doing this for 50 years and saw it any research. So you get both. It's just a lot easier to find the latter than the former, sadly. Thinking back to what Theresa and Greg said, I had one last question for Steph. Why aren't companies hiring more researchers? I think it's because the truth hurts. My thanks to Steph Troeth and my Teodondo, both brilliant researchers I've had the pleasure to work with at Clear Left. Now that I know how researchers underserved and most organizations, I feel very lucky that at Clear Left, research and researchers are an integral part of our process. In fact, we're looking for another design researcher right now. Could that be you? Head along to ClearLeft.com/About/Jobs. And thanks for listening.