How to Remove Emotion from Data Analysis ft. Gerda Thomas

anything you do as a human being has an emotional charge behind it because you are a human being with emotions. What is going on Certified Homies? Welcome to another episode of For May to Be. You have Shiva and I'm joined by a repeat guest, Gerta the Koala. Oh my God, I almost said Gerta the Qualitative. Whatever. Gerta the Koala Voktownis, who is the co-founder of qualitative, qualitative.com, all around amazing human being. Gerta, welcome back. Hello, hello. So nice to be back. And that's why that's a terrible transition to the topic for today, which is taking out the emotion in experimentation. Let me ask you my first question here. I should have done plugs. By the way, we have a sub-stack, not a sub-stack. Don't go to sub-stack. We have a beehive for maydb.bhav.com. I follow a lot of YouTubers and stuff. I'm like, they always plug stuff before they start. And I just like, I don't remember 'cause I'm excited to do it. Got a self-promote. Yeah, cool. Well, I just assume everyone just use the timestamps and skips to like the first part, ignore all the bullshit in the beginning anyway. If you're listening, get a thumbs up. If you heard this. All right, I did the part. Okay, first question for you, Gerda. What's the most emotional you've ever been in the experimentation process? Like giving an example of you having emotions in the process? I'm actually pretty surprised that you chose this topic because you just mentioned earlier, too, that you're like one of the most emotional guys about this in this tree, right? You're very excited about what you're doing, which is good. But for me, I think more of the emotionality was probably in the beginning of my career when I was still trying to find a steady ground to stand on, not having enough experience, just kind of trying to put things together. And then somebody comes in and says like, no, this is shit, what you're doing. You like, stop and try again. And then that hurts. But then as you do more and more of this job, you realize like, yeah, it was pretty bad. Like I needed to learn and get better. - It's interesting because I think that's what drew me to this job. And I think that's what draws us to this job is like there is some emotion to it. But there's like that dopamine hit of winning that people like to chase. I don't know, like that first test win that I had, that was actually a test win, not just like one for an hour after I launched the test. That was such a cool feeling. Like you did something and you measure that impact and you're like it won. And that was the thing that I did. Same for me as like, I was a junior in my career. That was a cool moment. I think I'm probably perpetually chasing that dragon where I don't think I'll ever get that high that first test win. But I will do my best to try and chase that dragon over and over and over again. - Yeah, even, I mean, but stuff like that still happens. Obviously everybody messes up like just today, I had a client call where like we launched the test, I think yesterday or something and overnight, it had only been alive for like 12 hours or something and it just completely broke down for unknown reasons that we now have to figure out. And you know, everybody panicked and like everybody knew it was our fault. And we kind of like had to deal with it on the spot, right? And that just, it feels bad. Like I don't want stuff like that to happen and I'm doing my best to avoid stuff like that, but it does happen. So you're going into the test, wanting it to win, and wanting it to do the best you can with preparation of like running the research, doing the test build, QAing it. Like you're doing everything you can to set it up and the best way you believe possible to set it up for the most amount of success. And I think maybe that's where sometimes a motion plays into it where it's like you're so invested in this thing. You spent so much time, money, effort, brain power. When it loses, you're like, well that was a fucking waste. Like that sucks. Yeah, like the analogy just just popped into my head. That would be like an equivalent situation with, you know, like buying a house, for example. It's like, oh, you need to imagine your entire rest of your life at this random house that you're viewing right now, but also you're probably not going to get it. So don't get attached. And then over and over and over again, until you find something where you can live. It's kind of the same with my Navy test, it's like. Especially in this housing market. Yeah, or even the ones that you're like, this is amazing. You're like, oh, I can't afford a $5,000 a month mortgage. That's cool. I guess this is never going to be a dream for me. Yeah, but with A/B testing too, it's like, you need to get people excited about it enough to push it through to make sure you can even launch anything. And then also try to mitigate biases and personal opinions about like, what do these results actually mean and these results are just, it's just like a point in time. It's not forever. It's not attached to anybody's identity or anybody's worth as a person or anything like that. Let's just try again, right? That's really interesting you brought that up. It's not a reflection of your identity. But I think that's where when I was a junior, I did feel that way because it was like, it was my idea. Like, it was my brain that came up with the idea and I spent so much time building it. And then I was invested in it. And that's why I think that's where early in my career, I felt like my ideas had to win. Otherwise, it would be a reflection of how dumb I was. Yeah, otherwise you're a bad person. Otherwise, I'm not only am I a terrible person for screwing the business over, I am a dumb ass person because my ideas were terrible, qualitatively and quantitatively speaking. And I think that's where it did affect me. Like, test losses early on my career. Did hurt me like, I felt bad about it. I felt like I was an idiot because my tests weren't win. Yeah, I think that's kind of like maybe even a bigger can of worms in terms of corporate culture where people's identities are so attached to their job. Even descriptions and just like random things that they have a power over in an organization and so many random problems stem from that. Like trying to get access to random tools sometimes is like an uphill battle because people feel this weird like ownership over like, you know, their analytics set up. I don't want anyone else to look at it to point out flaws in it and you know, because otherwise everything will come crashing down and they'll see that I did something wrong. When it's like, no, we're just trying to figure it out together and make it better, right? Do you think being emotional is inherently a bad thing 'cause I'm sure there's some nuance to it where I'm sure there's some scenarios where like being emotional and experimentation process can actually be a boon and not necessarily a super big detriment. - I mean, it's not possible to be not emotional. I mean, then you would have to be like a psychopath, right, or a robot, your roommate. - Or hey, right, how else could I say that? - Or an AI, yeah, just hire an AI to do your experimentation. If you don't want emotions in your experimentation product, yeah, I don't think it is possible and it is good to an extent like that. That's what makes us, you know, work better with other human beings, be considerate of variables and just like not try to force anything that's unnecessary, I guess, because at the end of the day who are we optimizing these things for as well while other human beings, right, so. - I'm gonna get on my anti-AI soap box for a second. - Nice. - I agree, and that's where I think AI is really good at optimizing for metrics, not for humans, right? And sometimes there's overlap. Sometimes the metrics are the things that are human behavior that are great, but having empathy for your users and being able to bring the human emotion into your decision-making and your hypothesis into your analysis, you can start contextualizing some of the metrics that like the whole, and that's where dark patterns emerge, right? That's where you take the emotion out of that and you're like, how can I squeeze as much juice out of this person, this number? And not like, wow, yeah, I guess your numbers might go up if you make a 50 step process to cancel your gym membership, but there's a human part of that that's like, this thing fucking sucks. And if you're a human and you have emotion and you can empathize, you realize that's stupid and it just ends up creating more friction. And the LTV metrics probably going down if you do that kind of stuff, but people aren't focused on that. So I think that's where empathy is an emotion, I think. To actually get into this. It is, okay, all right. I don't know if that. 'Cause it's just stealing a mug. I guess it's, yeah, okay. Yeah, and that's where the whole like, data versus intuition stuff comes in too, I guess, because like, especially when you start out and I think so much of the narrative around CRO and experimentation is that you're not supposed to trust your opinion, only hard data, you need to make decisions based on data only. That's what leads you down a dark path sometimes because sometimes the data shows that when we screw customers over, conversion goes up. But does that mean that we should be doing that? Probably not, if you want to be a decent human being and make sure your brand is sustainable in the long term and useful for people, so. And I think that empathy extends beyond the users that come to your site. And there's also empathy with their stakeholders too, right? Understanding that maybe some decisions that they're making, maybe it's not even decisions they're making, right? Maybe there's someone else who's forcing them to do something a specific way. And instead of being frustrated 'cause they're not agreeing with the decision you want, trying to understand where they're coming from, might create a partnership with that person, and then you and that person can partner together and create a better scenario that maybe you guys are aligned inherently, but it's just they're getting pressure from something else. Should you leadership, bad decisions? They need to do something else. I think that's where having a motion in the experimentation process is a boon and it's really great. The other side of the coin or the reality is that often experimentation does sort of bring things into light that otherwise wouldn't have come out. Like, oh, there are holes in this process or this UX flow or whatever that we should be doing differently. And then people who have worked years on putting those things together, as you know, we mentioned earlier, their identities kind of intertwined with that stuff that they've been working on, that ego gets scarred, obviously. And they think that that's like a massively bad thing and change is hard, let's just say. - Yeah, and the empathy to know that like, that is their baby, right, especially when you're talking like, you know, product people are founders, like they're sinking so much of their blood sweat and tears. They have the same feelings that we do with our tests, right? They have that with that product. So someone else is coming in challenging that. It's not a great feeling. And even with the best of intention, it could still not feel good that someone tells you some information that you didn't want to hear. Or you didn't like to hear. And you want to assume that everything's good. And when things aren't good, you're like, well, that's not good. So I'm very good at words. - Yeah. (laughs) - Words are hard. Also sort of like related to that, did you see that new viral tweet from the product guy from X that went live like yesterday? - I did. Do you want to explain it though? So basically he posted that he can't do AB tests anymore. If it's a good feature, it goes into production. I will not wait for results in quotes. The only results is when DAU go up and DAU, there's too many marketing collaborations. Daily active users. So that's like the only metric that matters to him, I guess. And then he finishes it off with. And if it breaks something, you guys, as end users of the product, are responsible for complaining about it. And then they will like us take it into production to fix it. - So I thought that was a joke, is it serious? - I don't know. I've seen it like people reposting it everywhere and stuff. And like I look forward to the next three months when everybody just says that AB testing is dead again. But so I read that and I legitimately thought it was a joke. I was like, he complains, he says the users are the problem. If a good feature doesn't work, I was like, that's the shittiest thing a product person could say. And that has to be a joke. But I guess in 2025, there are some things that are very serious or people's-- - I mean, I wouldn't be surprised at this point, yeah. And like if I doubt that like such a person that high up in a company like that can afford to post chunks like that. I don't know, maybe it's just for engagement. - I mean, it is an extra calm, right? - True. (laughing) - But yeah, but people will nonetheless take it seriously. And especially in our industry, we'll kind of try to piggyback off of it and explain other narratives away of why AB testing should be not being your vocabulary anymore. - We would say in this case, that's on the less emotional side of things, right? He's being trying to be matter-of-fact and basically be like, we never test because we'll do things that work. And then if it doesn't work, it's your fault, right? I mean, that's basically-- I think I'm summarizing that properly. - I mean, I guess it's more like just putting the responsibility of key way almost on the customer. - Right, which seems-- - Is it just a break? - It's true if it breaks. But I guess like what happens if there's a feature that's shipped and it's not-- it works by design, but it's the shitty design. Or it's stuff that decreases DAU. What's the feedback? What is their intended feedback mechanism for that? - I have no idea. And the other question that I have is like, okay, then what constitutes a good feature in the first place? Like you still need some type of framework to prioritize things, right? - I know what it is. A good idea is when it comes from my brain. And then that's how some people operate, right? I mean, that's how I operate it. Well, I didn't operate with the cockiness of that when I was younger in my career. But I did operate from the-- I wouldn't recommend ideas unless they were good ideas, right? I wouldn't say anything or suggest to do something unless I knew it was a good idea. And I put that stamp on it. - Yeah, it kind of works the other way too, where like we've had clients that have such like faith in us, which is obviously like great and that they're like, oh, but if you suggested and you've seen it work in another project, then why are we even testing it? And then we almost have to like undermine ourselves in a way where like no, no, no, no, like we still should, like we don't know what's it gonna do here. Like we need a way to measure it properly, right? So-- - Well, you should just ship it according to x.com. And then if it doesn't work, blame the users. - Yeah. - That's a great client model, right? Like that's the perfect way for you to just never have any accountability for anything. - Yeah, and like at this point, isn't there like studies and stuff already that show that companies that do way be testing grow way more and have better stuff going on than the companies that don't do it? Like it's kind of odd that everyone's still debating about that. - You know, it's funny. There's some selection bias in there. I think that's the right word. People who don't believe data won't believe the data about A/B testing working, you know what I mean? Like if you believe data, you don't need an A/B test equals profit chart to tell you A/B testing is good. But if you don't believe data and you're like, I don't need to A/B test to have good ideas, you will then come at that chart with saying, that's a shady chart because I don't believe data anyway. You know what I mean? - Mm-hmm. Yeah, I guess so the overarching theme of that is more about interpreting data because like two people can interpret the same set completely differently, right? Actually, you know what, that's a good transition. I know it kind of went off a little bit with the X stuff. Is there a motion and analysis of data? Like two people can interpret a data set differently. And the difference might be skill-based. It might, and it might be background-based, right? Like different mental models, people have different ways of analyzing their world's use and stuff, so like that might create some slightly different analysis for how people look at it. It's part of it a motion. Early on in my career, I tried my best to look for the metrics that made it look good because I wanted wins early on. But as I grew in my career, I was like, I need to be honest about when a test wins are loses 'cause the credibility that what I say matters, matters more than finding winners. And that's just something that I try and live by. But if it's your test idea, do you like look at it differently than someone else's test idea? And maybe that's something a motion related that you're like, I want my test to win, so I'll have a lower barrier. There's some emotional aspects to data analysis, isn't there? - Mm-hmm. I mean, as I said before, I think anything you do as a human being has an emotional charge behind it because you are a human being with emotions. Even if you think that you're being fully logical and unbiased, you're probably not because that's what biases are, right? Like you're unaware of what you're subconsciously doing often. And yeah, especially interpreting like test results, it can get really tricky because, especially if you're working with like lower data sets and you don't have that much sample and you still need to make decisions. That's something that I struggle with all the time and it's like, there are so many variables that influence these things too, right? Like we can't sort of claim that we make one little change on a website and then it will be projected into the future forever and ever if it won and all this kind of stuff, it gets so muddy and complicated. Part of the answer at least is that you need people in your team that are really well first in statistics and really like understand the topic, like I don't think that everybody needs to understand it and it's because it's not feasible anyway but try to find people to work with who are just really critical thinkers about those things specifically. - You mentioned something about decision making. And I think that's another part where, like perhaps it shouldn't be emotional but it's hard to ignore that there is emotion in decision making process. Like in a normal day when you're not under the gun with layoffs happening in the background, is the economy doing whatever the hell is doing? Like in a bubble, theoretically you take out emotions with decision making and you just like, here's the results, good or bad, okay, move on. - But even that is so hard like to interpret sometimes, is it good or bad, if it's inconclusive? - Like what is good or bad, that's objective too, right? And that's actually, it's interesting you brought that up. There is a motion in determining what's good or bad, right? I mean there has to be. - Yeah, and when it's bad especially then it's like, okay, why did we waste all this resource into developing these tests and what's the point of this program in the first place, right? And then on the opposite side, I've also seen when a test wins, people are like, okay, we already knew it was gonna win. Why did we even test it and waste all these resources? So it goes, it goes always both ways and that's so. - And how does that make you feel, Gerda? How, what are the emotions that go through your mind for me? - I guess I feel frustration that people fail to grasp the bigger picture of what the point of doing experimentation is and how important the mindset is. And it's just like so easy to get stuck in the weeds and just spend hours and hours arguing over, oh, what should this button say? What should this test do specifically where as an external consultant, especially steer people in the direction of thinking long term, bigger picture, why are we doing this? What's the benefit over a year, two years, the learnings, the like savings by just not implementing stuff blindly, right? It's not ever about just that one test. - And I think that's something that's important that I was bad at this early in my career where I had a lot of emotions and I didn't channel it appropriately or I guess productively, appropriate subjective. I didn't channel it productively into the avenues where it made sense and then pull out emotion and be more objective when I need it to be. I would get very frustrated when stakeholders would just reject my test ideas. I'd get very frustrated when just like what the people were like, well, why'd we run the test? We knew it was gonna win. It's like, we didn't know it was gonna win. That's why we ran the test. We talked about that before. And similarly, I'd get frustrated when people were like, why are we testing this when I was gonna win pre-test? I'm like, we don't, that's the point of testing. I'm curious if you agree with this. It's not a, I almost said it's not appropriate. I think appropriate subjective, it's not productive to channel negative emotions when people push back. I think part of this is just trying to understand where are they coming from. Why might they push back? Where is their mental model? 'Cause you could like, you could complain about it. I'm LinkedIn and I support this. Talk about how shitty stakeholders can be and how it's frustrating to deal with that process. Great therapy, I support it. Sierra around table, go to like experimentation conferences and bitch about it. That's a great use of your money. Talk to your therapist about it, I do all the time. He doesn't know what to Sierra is. He's like, I'm not getting paid enough for this bullshit. - This episode is sponsored by BetterHelp. - No, dude, if anyone is sponsored for me, I would love it 'cause my co-pay sucks with them. - Anyway, yes, speaking of sponsors, I'm doing this shit entirely for free, guys. I'm editing this on a Saturday morning just so your ears can hear this beautiful podcast, okay? So maybe go like, go subscribe, throw some comments, get some engagement, share this on LinkedIn. Maybe go to our for me to be.be-hive.com and subscribe to our newsletter. Be-e-e-h-i-a-v.com, yeah, it's free. All right, let's get back to the show. - I think those are super appropriate ways because it's not, don't run from the emotion. It's right, it's logical to feel frustrated in those times. But man, like, if you wear that on your sleeve where a stakeholder is like, I think that's a shitty idea. It escalates the temperature and it creates a cycle of maybe they don't wanna work with you again. And they're like, why would I keep on working? He just gets mad every time I try and all for my opinion. And they might create their own narrative. Well, you're trying to push what you think is right. It escalates and it creates a cycle of them not wanting to work with you most likely. And that's not productive. Where, how would it be productive to channel your emotion when you're getting frustrated with stakeholders like that where they're not approving your test ideas? They don't wanna run tests. They don't wanna run your test ideas. They don't wanna, like how might I channel that appropriately? Or again, productively, not appropriately? - Yeah, I mean, sometimes the road just is blocked and you have to find other ways or other areas that you can focus on. And then meanwhile, it might be research or whatever, but I think it's also good to think about testing as like, there are three distinct, once the word that I'm looking for. - Periods of running a test, I guess, or you know, you have the pre-test era where you're trying to figure out what a good hype is, how to support it with data, get buy-in from all your stakeholders and just like get the thing live. Then you have the era where your test is running where you need to like monitor it, but also not do too much peaking and too much deciding on things based on low sample sizes and also mitigating a lot of expectations about why is it taking so long and all this stuff? And then you have the post-test era where you need to interpret the results properly and also just distribute the results across organizations and like make sure everyone understands what the next steps are and are we iterating or are we not like all these things. These three different periods of running the test, like different levels of emotions are needed, I guess, like you know, when you're trying to get the thing live, you need to be more of a champion and get everyone excited and you know, want to participate in it. And then you know, there's the middle part, as I said, kind of like everyone's impatient, like okay, like let's conclude this already, what's the results and then, you know, interpreting everything. So yeah, I feel like your personality probably is good for the first part, I've just championing the ideas and getting things live because, you know, your excitement probably infects other people and they get excited as you said about it too. So that's really important. But then it's like, okay, if it loses and we were all excited, oh no. Now we got to do a post-mortem and see what went wrong and I'll feel that together and then somehow pick ourselves up and move on. - You don't want to fake excitement 'cause I think people could read that and they're like, you're just cheerleading bullshit and that reduces the trust. But I think that's also okay to be like, it sucks that it lost and not be happy that it lost. But like, I know this is like test to learn sometimes feels like it said too much and it's like, all right, well, we learned that 11 tests lost this month, like a client who's paying you like thousands of dollars. I don't know how excited they're gonna be like, wow, we paid you like $10,000 this month and you just gave me 11 tests that lost. Like, there's an emotion on their side that's like to fuck this, what the hell? Winwin once, this isn't paying for the program. Like what are we doing about it? The excitement isn't the right emotion for when a test loses, but it's security as to why and focusing on that part, like it lost. And yeah, that sucks. But there's some really curious things that we figured out here. It's your point. Let's focus on what we're gonna do about it. But that curiosity pieces where I think you lean into it. And if it wins, pop the champagne, right? Like, let's have some fun party, especially if it's not your test idea. If someone else who's like not on your team or not even like Webb, like sales or customer service or whatever, they give a test idea and it killed it, throw them all the flowers and be super jazzed about that. 'Cause I think that gets their dopamine and they're like, hell yeah. And then other people see it and they're like, I got ideas too with the hell and then they get. So I think that's great. The curiosity piece of when a test loses, I think that's the emotion we have to lean into. So I think that's a positive emotion that will, I guess slightly redirect a ray from the negatives of the loss while still being emotional maybe. Maybe I'm talking myself into like, actually emotions totally find an experimentation. It's just where it needs emotion, where is and the appropriateness of it is like more important than taking it away entirely. It's just like life. I mean, yeah, also obviously winds are important in a program because if you have like 100% losing tests and yeah, like then we probably need to, we need to hire different people to try to do this. But also, I think for anybody, it's really hard to retroactively try to think about like, you know, let's say we have a losing test and we lost a certain amount of revenue for the business then trying to conceptualize, okay, but what would have happened if we would have implemented it completely, how much would have been lost then without even knowing or getting any learnings about it. Like that's just, you know, it's all like a hypothetical situation, but it does matter still. But in it's really hard to just put yourself in that mindset. So I think as a CRO manager or program manager, whatever, that's part of our job is to kind of highlight those nuances. Like, okay, there's kind of more beneath the surface here to think about, it's not so black and white. - So I know what Tyler, we bookended like pre-test, be excited about launch, post-test, excited for winds and maybe curious about if it loses, what are you doing about it, why did it lose? What about that middle part during the test, right? There's going to be anxiety about, probably anxiety, the first couple hours of like, man, I hope this thing doesn't totally tank and you wait for a couple days and anxiety of like, we need to call it soon, we need to make a decision. Like, come on, it looks like it's green, the test will tell me it's green. We had to launch it like, what emotions are productive during the test launch? I'm sorry, like when the test is actually live. - Yeah, just trying to stay like detached, I guess. - So no emotion then. - No, it's like like hopeful, but cautious, I guess. You know, prepare for the worst, hope for the best kind of thing. Again, take context into account where, I think it was maybe even on your podcast, I heard somebody talk about it or, you know, it's obviously a well-known thing where you have to consider how long the sales cycle is. We can't expect any results from a shorter time period than your actual sales cycle is. So that's like a really good point to always bring up while managing expectations. Like, okay, like we want these tests to be concluded as fast as possible, but then your product takes like months for customers to consider and buy, right? So it's not realistic. - I think that was the episode with Toss. - Oh yeah, maybe, yeah, I think that was it. - I think that was her. For that's the plug. Good job podcast, good job podcast Shiva. Editor Shiva will be happy to have that. Okay, so I think that's great. I think that's where we've kind of broken out emotion in the testing process. What are some ways to manage the emotion? How I found it productive to manage my emotions, one, lots of therapy, betterhelp.com, promo code Shiva.com, not really, and I'm not sponsored by them. But therapy actually does help because there's some really great tips and tricks to actually manage those emotions. But I've actually learned a lot about psychology and therapy that helps me work with other stakeholders appropriately. And the biggest thing I've learned in my career but also in therapy is just like focus on the problems. If someone is rejecting your test idea and they're being butt heads about it, it's probably not personal. And it's probably because they have other problems they need to focus on. And your problem you're solving is data to ship a thing. Their problem is I need to ship a thing. Otherwise someone will get mad. I mean, I'll lose my job as an example. Understanding and I guess part of it's actually just being able to let go, which is an incredibly hard skill for me. I rarely let things go. I'm very petty. So like, I think that's something that is important too is like you're not gonna win every battle. You shouldn't expect to win every battle. By the way, have you seen that? Like I've actually noticed a lot of junior CROs get like super mad when tests are not run and they're like I'm this a year old guy and you're not running into my tests. Or I guess any of the tests is bad but you're not running this one specific test that I super want you to run and I'm pissed that you didn't do it, fuck you. (laughing) I guess that's not productive. - Yeah, so two things that I wanna touch on here. One is that a lot of the collective problems that we have are actually personal problems. And yeah, as you said, everyone sort of has to do the work on themselves. - When you say we, you mean like CROs, right? - I think generally just like corporate ones. - Culture as well, you know? Because you know, whatever we're dealing with in our personal lives, we do end up bringing to work and to an extent we can't control what our co-workers do in their personal lives and that leads me to the second point is that it truly starts with hiring for your culture fit. And I've seen it so many times with like countless companies that I've worked with where they bring in people just based on hard skills and resumes and like kind of, you know, build them up as this rock star human being who's going to come and save us and do all these amazing things. And then they come into the company and they're a straight up asshole to everybody. They can't get along. They just like are refusing to communicate, play ball, whatever. And in six months they end up getting fired. And then we're back to square one and the cycle continues. I don't know, I just can't emphasize that enough. How important it is to hire people based on like mindset as well because in experimentation, especially if you don't have an open mind, you're close-minded about like what you're trying to do. You can't succeed in this job. You just can't. - It feels like a lot of problems. It just with jobs, at least from what I've seen, a lot of problems end up being like people and it's not skill related. It's like, I don't know how to do a thing. It's people related like there's conflicts of the way people try to run businesses or do things. And it's less like, I don't know how to do Bayesian stats. And that's the reason why our company's taking away like this is never in the case. - That stuff seems hard and it is hard to an extent. But I truly believe that everyone can learn it if they want to. And as I said earlier, I don't think everyone needs to understand stats like that granularly if you have team members who are dedicated to that. So like you can learn all those things, how to do strategy and how to analyze tests and do research like literally everybody can learn it. But if you cannot like communicate properly and work in the team and sort of understand how you fit into that organization, that's when things fall apart. - Yeah, I mean, you could have the best test ideas in the world if no one wants to run it 'cause you're a pain in the ass to work with. Good luck. Okay, so I think that's, I agree with all that. Emotion is appropriate. I think when getting people excited about running tests and then when they conclude, when they win, I think that's great. That's not emotion to remove. I think that's emotion to double down on. - A lot of the education, especially if you're building a new program, what I kind of overlook often because it is so, you know, like we think that everyone knows what we know, right? Like what's basic to us might not be basic to others. And especially what even is possible to achieve with a testing tool. Like I find myself in these conversations often where we're just proposing the test and it gets perceived as this massive like effort and lift and we're like, no, no, it's actually like a couple of hours of work. Like it's super easy lift with a testing tool. Like we can manipulate the, you know, the front end of the website and it's like people like, oh, you can do that. It's not like a full on development project and we're like, no, like that's kind of one of the benefits of what we're doing here. And just, you know, reminding yourself that there are, yeah, a lot of those things as you said to constantly remind people of which may get them excited as well because it eventually makes their job easier. - Is that an emotion like making someone's life easier? Is that an emotion? I don't think it's an emotion. It's more of a task, but like there are feelings associated with making someone's life easier, right? - Mm-hmm. - That's one that you quadruple the hell on. You quadruple down on all the, whatever I can't, just mixing it out, just do a lot of that. Shiva, to do that, yes. Because I think like the more you're seen as someone who reduces friction makes their life easier. For me, when I'm presenting test ideas, it's like, here's the data. Here's the reason why I'm running it. Here's the design. It's like a very quick packet where I just need someone to just check it and be like, good to go. Rather than like, hey, let's collaborate a bunch 'cause sometimes people don't have the time to offer input and stuff. And if they want to offer input, I'm like, great. I'll get the feedback incorporated and we'll launch, and I'll tell you how it did. Like making people's lives easier on that process. It's interesting 'cause there's a balance of like collaboration versus make your life simple and easier. That's a balance you just have to figure out for your company because it's not gonna be a simple rule. Some people prefer to be involved as early in the process and be if involved in making the sausage. Some people are like, dog, I don't care. Just tell me where to sign. If this is a good idea, cosine, great. Then the less friction you give me, the better. So there is a feeling associated with like, make my life easier that people gravitate towards. So it's not necessarily emotion and experimentation, but it's like, make people feel good about working with you, I guess. Yeah, and it's almost like to achieve that, you sort of have to be a little bit sneaky and sneak it into their workflows because especially if people are not used to doing testing, they will perceive it as this like extra work that is going to be slammed on top of their already very busy schedule. And it's like, oh, it's like extra thing we have to do and what's the point of it and whatever. So it's like, you have to be very tactful, I guess, to try to be like, no, like this is part of the mindset of how we're doing things now. We're just not launching things willy-nilly. We're kind of trying to test it out first and it should be just part of our integrated process. It's not like an extra thing on top of your regular product feature launches. So manipulate people, two thumbs up from Gerda. Mm-hmm. Yeah. It's just like, just no emotion. You're like, yeah, that's what I said. Do you hear me? No, but use your powers for good, you know? Don't be fat. Do it for the greater good. It is a really great point, too, because like, I think that gets, there's some flex and to like, understand their problems because like, if you understand where their problems are and you almost pre-solve their problems, or you understand their problems and pre-solve their pushback, they're gonna be way more likely to do it. So like, okay, I have this one stakeholder. We need to run tests, they hate running tests because they say it takes too long. The designing takes too long and they're like, the data, I'd rather just go live 'cause I know what works. And it's like, all right, well, in order to address this person, why don't I have the test ready to go always, or at least in the short term, have it ready to go, not include any big disruption to timeline for their internal timelines. If they're like, I need it to be live by November and by the time the test concludes, and we have a decision, it'll be mid-October, so that doesn't compromise their timelines. I have the design ready, I have the test ready, they just have to say, yes. And you talk to them and they're like, all right, whatever, do your fucking bullshit, whatever, it's not gonna compromise my shit, right? All right, do whatever, right? Like, maybe that gives them the passive just like, all right, whatever, do whatever, you're testing thing you wanna do, you fucking nerd, right? But then like, when the test wins, and they're like, good job, this thing's amazing, plus 16% lift to conversion rate, put that on your resume, that's an amazing. Then like, wait, I didn't know. Wait, I made a lot of way, how much money is that? Oh, that's like plus $40,000 a month. And they're like, huh, maybe I wanna work with you again. Right, like, it's a little light bulb moment. Part, I guess, part one's like, make it easy. And then, lead them to the light bulb moments without, I'm sorry, we've agreed them to water, without, whatever, whatever, I'm sorry, I can't, I can't think of an outage. Make them feel-- - Yeah, I mean, I get what you mean. Yeah, we totally take that approach with all of our clients too. It's like, none of the stuff that gets achieved, like we take some credit, but it's like, it's a team effort and we always highlight like where the idea came from, like internally and they're input on it because it does matter so much, especially if you're working on a more complicated product or something and there are a lot of conflicting features which write a launch something and then somebody's like, "Oh, no, but like this will actually mess this other thing up and there's a contingency and all this." Like, it does take a village, so to say. So, yeah, I think that's kind of like what a lot of agencies do. I don't know if wrong, but there's a specific mindset where agencies are like, "Oh, look at all the stuff we did for you." And I don't know, we're just kind of like going in a different direction like, no, it's like, we're here to support you and help you like get better, right? - Amazing, all right. So, I think the episode title is gonna be "Makers, take holders, feel good." - That sounds wrong. - Sounds accurate to me. I don't know what you're talking about. I have a clean mind. Gerda, I know you are qualitative with Ryan, our audience are really cool stuff. I know one of the things they're working on is an AB testing analysis tool. Excited to see where you guys land with that. I think that'd be really cool to pull it out of testing tools and be centralized rather than being married to whatever tool you have to use. I think that's really cool. - Yeah, we live in a time of data warehouses and complicated analytics products. So, we're trying to simplify all that and not force people to write SQL. So, that's kind of our tagline. But, yeah, connect with me if you're interested in many of that. - Cool, so that's Gerda, Vogue, Thomas, on LinkedIn, qualitative.com. Anything else you want to plug? - But that's pretty much it. - Cool. - All right, Gerda, thank you so much for hopping in. I hope you feel great. I hope you have positive emotion. - Yeah, you too. - I don't know, I guess that sounds a little weird. - Thank you, all right, we're just gonna end here. All right, great, good to talk to you. - Okay, thanks. (upbeat music) (upbeat music)