Episode 178: Testing vs Quality

That doesn't translate into quality software, which is what I care about far more. Who cares if a crummy product that nobody uses is well-tested? Welcome to A.V. Testing Podcast, your modern testing podcast. Your host, Alan, and Brent will be here to guide you through topics on testing, leadership, agile, and anything else that comes to mind. Now, on with the show. Welcome, everybody. This is Alan, and once again, I am pissed off at Zencaster. But Zencaster, Alan, is the easiest way to record your boy podcast interviews in studio quality. I gotta tell you, when it works, it is awesome. And honestly, I would pay money for it if you guarantee me it would work, but I have none such confidence. Here's what happened. I'm happy all. And then the Zencaster algorithm is going to sneak in, hear my voice, and blow the whole thing up. But Brent was late. What's new? And then I started, I just as I was starting to record without him, and it was good. I had a speech going, and it was awesome. Brent showed up. And then Brent was coming out the wrong speaker because Zencaster can't read its settings from a hole in the wall. That's a horrible metaphor, but you know what I mean? What I have to do is I have to change the speaker from the one that's correct to something else, then back to the one that's correct, and then it works fine. And then I pressed record, and then Zencaster restarted for both of us, and it went back to the broken way, had to do it again. Now here we are. Welcome to the Zenbcaster or something. Anyway, it's, when it's good, it's great. So when it's annoying, it's really, really annoying. But I'm happy to have someone from Zencaster on the podcast to talk us through it. And honestly, we're getting value from it. So maybe it's all right. How are you doing, Brent? I'm doing all right. I'm currently looking up to see how we can get someone on the podcast. Oh, how brilliant. Like they publish their, looks like they publish their Twitter handles for. Good. We'll ask them. We'll ask them to come on, and they can, because you know, we have had, to be clear, Zencaster has worked perfectly for us a few times. And meanwhile, over in the left pane, I see one recording of 31 seconds and one recording of 0, 0 seconds. Neither of the ones which will be published, but we appear to be recording now. So, okay. Enough with Zencaster. We just talked last week. And we talked a lot about AI. I talked about a lot of the reasons why I left my old job. And maybe this time we'll get the talking about what's next. But first, can we talk about how freaking stupid I am? Don't, like we've beaten that horse to death. No, we talk about how stupid you are, Brent. Oh, oh, I misinterpreted. So every year, every year, you don't know what this is and stop typing while I'm talking. Well, actually, it's better for you to type when I'm talking them when you're talking. Because if you're, I want I can edit one. I just poke myself in the eyes around. Every year, every year, I have a reminder of my calendar in March to sign up for the lottery. Brent is banging on a fucking can on his side. Let me finish that thought before people think I'm all about buying lottery tickets. Every year, I sign up for the lottery to get early access to the Wonderland Trail. 99.9% of our three listeners, including Brent, do not know what the Wonderland Trail is. The Wonderland Trail is this trail that goes all the way around Matt right here. It is freaking amazing and beautiful and awesome. And every year, I want to go do it. And every year, I do not get selected for the lottery. And if you don't get the lottery, you can try and figure out campsites, but it's painful. And just it's, it's really difficult to make work. So this year, for the first time ever, I got accepted into the Wonderland lottery. And Brent's applauding and Tuesday morning was the day that you, I could begin to go get reservations because we'll open up for just a lottery winners. And I completely forgot. So I remembered today as I was one of the things I did on Fridays, was clean out my email. I was going through and I saw that and my stomach dropped to the ground. And I just, I went ahead and went. Was it Tuesday and Tuesday only? Or are you competing? No, it opened up Tuesday. So what happened is everybody else had lottery access, which is not a small number. And there's a limited number of campsites. There are specific places you can camp on the Wonderland Trail because of the bear issues. And they have bear precautions for food at those camps. And also just trying to limit the people so it stays awesome and beautiful. And so open Tuesday. So Tuesday, Wednesday, Thursday, most of the day for I to hear people have been filling out the right interiors and I go. And of course, the days I wanted to go, there's no campsites available. And luckily I was able to do some, and here's the deal you're trying to, it's 95 miles. So you're trying to figure out lots of ups and downs, trying to figure out a stainless pace to get around this thing. And I had it. It had it all worked out. It was so, it was nice. It was like nine to 12 miles a day, which is a lot when you're going up and down steeply. Or anyway, it was, it was perfect. It was perfect. It was perfect. None of those campsites are available. So I am doing some weird like 16 mile days and a week later than I wanted to go, but it should be all right. And I'm a little bummed about it. That just happened like a half hour ago. It's like, oh no. So that was, that's me being stupid. But then I was smart today too. But we can talk about that in a minute. Any comments? Do you want to go on the one? I don't know. So it's, it's only book one seat. So you have, you have. Oh, so you're just going to go off by yourself? Yeah. Okay. So this one, is this the big like bucket list trip you've been talking about? No, no, the bucket, the bucket someday is the Pacificrest Trail. That'll take about four months. Ah, okay. Okay. This is 93 miles 16. So what is that? Two weeks? Nine days. The PCT is interesting and I've always wanted to do it. But ever since that dumb movie came out about it around eight years ago, it's gotten super popular. And as you know, and as our listeners know, all three of you, I do not like people. That's why he's going camping for nine days by himself. Yeah. That's okay. It sounds pretty limited. Most of the campsites only hold like five to ten people. So it stays, once you spread out, it's pretty, you don't see people much on the wonderland is what, what I'm led to believe. PCT is also a permit system. Pacificrest Trail. But there's so many people on it. And what happens there is a whole bunch of people get excited. They watch the movie. They, they think, oh, I'm going to go do this thing. And they somehow managed to get one of their 50 spots a day going from Mexico. And I would guess half of them don't make it the first 300 miles. But for those first 300 miles, there's a shit ton of people around. So if you can make it to 300, you're good. Then it spreads out. Yeah. And unfortunately, because I move kind of fast, I would, I would, if I did it from Mexico, I would shoot, we're way off in the weeds here, but I'm happy to share this. I don't have to find the time I can get far more on stuff as a whole other story. But if I would start late and be passing people, so I'd see more people until, you know, Northern California or so and they're all eventually gone. I've been thinking really hard, you know, the vast majority of people do it from Mexico to Canada. I've been thinking about going the other way just to avoid seeing, starting at Canada and go into Mexico, just to avoid seeing as many people. There's advantages and disadvantages. The disadvantage is you're starting off with some huge up and downs in Washington going south. Right. But once you get over that, it's all downhill from here. Yeah, yeah, yeah. But the thing is though, then you end your hike dealing with the zombie hoard. No, because you're starting much later. So if you start from the, you're trying to dodge the PCT these days between getting the snow the hell out of your way before the forest fires come. So if you're starting from the south and you usually start like an April, some, some may some March, but April ish. But if you're starting, if you're starting from Washington, get away from the snow to melt. So you don't even leave till July mid July. So if you go south and you're going like from July to November and you're ending up in the desert in November, which is cold at night, but doable. And if you're starting south, then you start an April and end up in August. Anyway, something for me to figure out. I don't know. This is just something to get away from people and get some exercise, take some good pictures, try and eat and buy a bear. It's fun for me. So yeah, but I totally blew the reservation system. I feel really dumb. But I did something else today that was something I don't normally do, but it's actually maybe even a topic for the podcast. So let's pause on that. Anything you want to touch on from your life. from your dreams and fantasies or no, but never mind. - Now I had a weird dream last night. I fell asleep watching, watching "Turner of Mutt of Champions" which is a, it's a up-leveled chef competition show. - Love chef competition shows. - Yeah, it's basically they have like the winners of Iron Chef competing against the winners of Top Chef, right? It's like a super high bar. And I fell asleep watching it. And I had this weird dream where suddenly Guy Fieri was cheering me on and I was one of the chefs. And it's rare for me to remember my dreams lately. But this one, I'm like, you know, it reminded me, I used to always say that whenever I retired from Microsoft, I would go to culinary school. You know, I always thought about being a smelly A or a chef as well. - Maybe I can see you being a smelly A. My son, like just to give you a sense, my son, my interest in desire to around cooking is so strong, he was seriously considering going to into that as a career. 'Cause his bar was essentially, let me just do the things that Dad seems to be interested in. The quite honestly, he hadn't come up with his own goal himself. He's now almost graduated and I was, I'm positive to say, he's realized he and I had a conversation about a month ago. And he's like, Dad, I can't stand coding. I cannot stand it. Hatred, but I want the CS and I want, and I said, well, talk to me about the things that you kind of want to do. He's like, without boring our audience, long story short, he wants to be a PM. And I'm like, okay, well, let me introduce you to a few folks and then start directing it. And the other thing, he's very fascinated about his cybersecurity. And I'm like, oh, yeah, that one will be good for you. He's right now at Walt Disney World. Did I tell you this? No. He and his fiancee are part of the Walt Disney World College program. So like, you've been to Walt Disney World, I presume. I have not. You have not. I don't go to Florida. Okay. Walt Disney World, a large portion of their staff are essentially college level people from all around the world. And they accepted both of them. So they're working Walt Disney World for the next six months. Like he was working one of the glow carts. I don't know if you've seen this, but most of these amusement parks, like they have these carts with shiny, spinny things, particularly when it's parade time. And he he talked me this today. And he's like, oh, yeah, me and my partner, we sold a bunch of these things. And but anyway, he got an opportunity. Their IT department connected him, are connected with him, just out of the blue. They're like, hey, we see that you have a CS degree or you're working on it. We have this opportunity. Do you want it? He's like, yeah, I don't fully understand the opportunity, but apparently it's combined CS, that engineering data science in customer connection. And he's gentle of everything. He's super pumped about it. He wants to be a generalist. That and I'm supportive of that. Me too. Me too. Next week, spring break. So you asked my news. My daughter is doing a class trip. Both I and my wife will be. Yeah, you're going to Philadelphia and Washington DC. Yeah. So we talked about this before, drink. That happens starting Monday. Nice. Nice. I'm not not been to that part of the world. I'll say right now, I don't know what I'll be most excited about retroactively. I'm sure this is one of the things that I have picked up from Fallout, right? Because Fallout does such a great job sort of modeling. What the area of the world would look like in an apocalypse? You then go and look, you then go to the real place and you go, damn, those Fallout guys, they did a really exceptional job. And with Fallout 4 being all about the Boston area, I'm like, you know, I would love to go to Boston even more now just because of the real historical context. So I've done the opposite. I've done extensive walking self-guided walking tours of Boston before playing Fallout 4. Okay. And when you find it, it'll still be fun to go back. I have, I have definitely walked more of Boston in Fallout 4 than I did over days and days of it in person. So the question though, though, when you played Fallout 4, right, did you go like, I have totally been there. Like did you? Yes. And a couple of places, yes. Yeah. So for Vegas, that's what happened with me. Anyway, so part of my bucket list is certainly to do historical, visit historical sites. Excited about going to DC. I don't know that we're gonna see the White House, but we're definitely gonna see the Lincoln Memorial. So that should be cool. I will actually have a Philly Chees stick from Philly. Right? Yeah. Yeah. But I was a fan. You are far more troubled than I am. I still have not been to Europe. All right. I'm gonna push off the, we have to have an AI discussion chat, GPT discussion. We do? I thought we were gonna avoid it this time. We can we ever really? All right. So it was interesting. There was, I don't normally respond to random posts on LinkedIn. I shouldn't even say random. I don't normally respond to posts on LinkedIn. Often it is not worth the effort. Okay. So you, do you find, as an aside, do you find responding to posts on LinkedIn have has less ROI in general than on Twitter? Oh, that's a good question. People want longer answers, so they're, it's different. There's a lot more posturing on LinkedIn these days. And Twitter's kind of turning into a ghost town. Yeah. So, but there was one struck something with me. And it was from, and you, you, you've still, he still works at Microsoft. You've crossed paths with Wayne Roseberry before. And he has a lot of good insightful posts. And he's someone who's still doing at Microsoft, still doing mostly testing as his job. So I'm, I know there's very little of those roles like that. So it's interesting, but good for him. And he's still still doing that? Yeah, yeah, yeah. And he's telling a story about where he wrote some automation to open and close word a bunch of times and found a memory leak. And reminding me of a story I told in how we test software at Microsoft. And I promised someone a while back. I can't remember who that I would write the 15 years later readers guide to that book. And I'll do that this year sometime. But it reminded me there's a story I wrote on Windows CE. If you remember the Mira client, which was a stupid thing, there was this idea. Remember this is 19, it might have been 2000, 2001, but the idea was the idea for a device was, we sold a couple of these. It's like a computer monitor, like sitting on your desk, like you have a pad, like imagine using an iPad as your, as your monitor. Anyway, the idea was when you undocked it, you could walk away with it. And it would immediately become a terminal server client logging into your machine. So basically you're taking your computer with you on a monitor. This was before we had the idea of what a tablet was. And it worked, but it was dumb. It was heavy, it was expensive. It was too hard to use on the toilet, I think, was the final review that it wasn't going to happen. Is that what killed it? Maybe. So I added a story, the paraphrased short version of what I put in the book saying, that brute force automation sometimes is interesting because I did the same thing there. I wrote a little script, not in Selenium, not in Appium, not in any of that crap. I just wrote a little win32 app that found the control, clicked on it, and then found the control in the frame for the terminal server and clicked disconnect and just looped. I don't know why it felt fun to do, but after about 20 minutes my test crashed. And I thought immediately the thing you think, wow, I must have wrote some crummy win32 code. So I looked at it, it looked fine. I cleaned it up a little bit, added some logging, ran it again. And in 20 minutes it crashed again. That's okay, this time I'll look at it in the debugger and sure enough, leaking GDI objects and it crashed because when the seeded, now that's that much memory. That's all just the same story told a different way from a Wayne tool, but then I said the thing I always say, which is, I'm gonna say it again, I wish more testers and developers used automation as a discovery mechanism rather than stupidly trying to automate a bunch of user workflows. which I have been saying for at least a decade, at least a decade, stop it. In fact, it was like over 10 years ago, I gave my talk on stop writing automation. But nobody listens to me, Brent. Nobody listens. There are hundreds of thousands of people out there writing user workflow automation and Selenium, Appium, TestCafet, Cypress, this and that. Spending all their time writing user workflow automation, Brent, why, why, why, why, why? Yeah, it's a big test auto. Okay. Yeah. You just want small voice and you know the big test automation, you know, and they're lobbyists, they're out to get, you're just a snowball in their heat wave. Sorry. I write that and I go to the gym. It's time this morning. And I'm at the gym and I get a little notification. It says Michael Bolton has replied to your comment. Yeah, I'm staring at it. Your BFF was just like, yeah. And wait, wait, wait a f*** the story. Wait a f*** nice. Oh my god, this is why I don't reply on LinkedIn. But I checked and he says here here and, he gave a like to my post because the idea is that one thing Michael Bolton and I can agree on is that writing a bunch of user workflow UI automation is not that helpful. No, it's awful. It's, it's just terrible. As I mentioned before, James and Michael and I have this written truth where we don't really, we just kind of stay in our own lanes. Even when I wrote that post, remember the one where I said, not all teams need dedicated testers, which seems so brain dead simple to me like, yeah, of course, not all teams dedicated testers yet a whole bunch of testers got really mad. So I didn't know anything about testing and blah, blah, blah. And then tagged in Michael and James, that's what they do. They bring in the, bring in the grownups to come yell at me. They didn't yell at me. As fast as they got was Bolton to something about, well, this author doesn't do blah, blah, blah, blah, which was wrong, but I didn't take it like, yeah, yeah. And I don't know if you saw it in that same post just before it. The issue is you validated, you accidentally went down the path of validating the whole check versus testing thing. That's the only reason why Michael here, here you. Did I? So I didn't, you did I post, I didn't, I didn't validate any checking versus testing. No, but just before, I just before you posted, Bolton replied to Wayne's thing and said, that was not an automated test. That was a test, right? I didn't read anybody else's stuff, whatever. Okay, no, but you were, that was you using your intellect, your skills and a bit of code to say it was an automated test is like saying that a hammer built the house, right? And then he goes off into his weird checking versus testing metaphor. It was a test completely agree. And it was automated. I do not understand why it's not that simple. And then you said, so the way I think Bolton was translating it, the term automation is now a tool. I wish, wish more developers and testers used tools as a discovery mechanism rather than trying to do a bunch of work. I do, I do. And if you think about, and this is a big, big point of mind, I've never understood the interest other than a cool little demo to watch of trying to automate a user workflow. You use, you want to use, I'll call it test automation test tools, whatever the hell ever, you want to use the power of programming to do the testing that is too difficult to do by hand. But if I want to create a thousand entries in an address book, I'm not going to go create them one by one. I'm going to write a freaking loop. If I need to create any sort, I want, I can only use automation to do test data. If I want to understand, well, what happens if I do this action a million times? I'm going to write myself a little program to do that, understand that. I am not going to write automation to put an item in the cart, open the cart, click on payment, enter a fake credit card, and make sure I get the sale done. I don't know how far to ridiculous you want to go with this conversation. But another blatant example is, okay, can my website handle a thousand simultaneous users? Sure. Right, do that as a manual test case. All your friends and you have them log into your stupid website. Even there, you can do some cool stuff using Postman. If it's out of your website, but it's an API and you want to simulate some requests, you can do it there or. My point is doing it with. Doing it in the manual way is dumb AF. It's all I'm saying. Yes, yes. Right. But that curi. Here's what's wrong. And here's where maybe we agree, but I don't want to go in. I mean with Michael Bolton is, I want to use automation to let me be curious about what happens, not to satisfy some requirements. I know. There's somebody. I can't remember their name, but they're in my feed, never removed them yet. But they just post these esoteric comments about testing that mean nothing. But they always at the top of my feed because they post so often. I looked at one the other day and it was something around. Oh God, now I just. It left my head because it was so stupid. I don't know. It doesn't matter. It'll come back to me later and I'll blur it out. But I just want testers and that are testing. People who do testing and once again, research has shown that automated tests owned by the dev team have a high correlation with quality and those written by the QA team do not have that correlation. So just want to throw that out there for those of you that have forgotten every time I bring it up. I used to have the page memorized. Yeah, so here is. I don't know how. Look at AB Tester talking about testing. Welcome to the podcast. I'm not certain how you manage to successfully trigger me, but you have. It's my job. Testing? I'm good at my job. Testing at a high level, this is the thing that bothers me. And I think it's an alignment. There are a lot of people that we have spoken to in the present as well as in the past that view testing as the goal. Yes. It's F and not. Testing is a means, not an ints. Okay, number one. Number two, when you are going through an understanding what the hell you're trying to achieve, that's, I guess, that's my other problem is that there's so many people who don't even realize it's a means on an ints. So they don't even have a sense of what's the end, the goal you're trying to achieve. This is why principle number one is principle number one for MTP. Now, to generalize what you were saying just a second ago is, okay, if testing is a means and not an ints, once I understand the ints, then I should be looking for the high ROI ways to achieve those ends. To me, it's like, you know, we want to take a trip to Hawaii. And, you know, we can, we can start hiking down the PCT. See what I did there. Nice, nice, nice, yeah. You're not going to get there. You're not going to get there anytime soon and you're going to have a big bottleneck at some point in time. Well, and let me just loop this into something you and I have both said before. Yeah, because this is exactly the difference. And I was trying to find the article real quick that where James wrote this, but I'm 99% positive I read this and didn't, didn't dream it. Because why would I dream about James Block? That would be weird. And again, I will not say anything here if he wants to listen to the podcast, he won't, but nothing I wouldn't say to his face is you and I have said, Brent, you and I have said many times we care about quality far, far more than we care about testing. Yes. James has come out and said the opposite. It wasn't like a premise or the thesis of his, but he cares a lot about testing. That's fine. That's fine. I disagree with that as the thing to care about. But when I look at these folks who are doing what all, if he's expressing his passion, his passion is testing. Yes. Yeah. And that's and a lot of people like there's a school of we care about testing, testing is super important. That we have skilled people doing great testing. And while I can't disagree with the passion to me that doesn't translate into quality software, which is what I care about far more, who cares if a crummy product that nobody uses is well tested. The testers maybe great, great. Yeah, it is fantastic. You're 100% right. That's the best kind of right. To me, he's basically saying and good for him, right? His talent is going deep. That has helped society. Yeah, no, but but his is. Did I just say yeah? No, I'll explain that later. Go on. But the acolytes that follow him and here I specifically use the term acolytes, right? It's the blind faith that then take his statements and misunderstanding in my humble opinion. I don't know if it's that actual misunderstanding, but misunderstanding that that he is stating this is what everyone should do. And it's not. He's saying he's going deep on this specific means. That's where his passion is and great, great. Go for it. But from a business context, I'm going to stand by the side of it's a means not an ends and I want to understand the ROI of alternatives. If there are multiple ways for me to achieve the ends, why would I do the one. Why would I do this one versus the alternatives? Yeah, and I'll be very, very clear here because this is. I don't want to say just preference. I'm not saying what James is doing is bad-baiting. I think it's because it's not. He's very good at what he does. Sometimes he's a jerk to people and I don't want to go into that right now. But what we really are arguing about is given the preference of. Could we make a product that is our customers view as a quality product? Or could we make a product that we feel is well tested? There's one that you and I would choose turns out at 10 times out of 10. And another one that people like James would choose 10 out of 10. The philosophical principle difference is that I believe that the customer determines quality. So if they like it, it doesn't matter what kind of testing we did. Yeah. Yeah. And if I do, if I have the best testers in the world applying great testing to something and doing things in a very thoughtful and using all the things from RST, and customers think the software is a piece of shit, I'm not proud of that work. No, I was just thinking because we just had this example of where we did a bunch of testing. And to be clear, there's an overlap. There's a van over here. I'm just saying if I can choose one, I choose quality. Oh, no, no. I mean, you of course can choose one. Damn it. Page, maybe lose my track of thought. That's what I do, man. That's my job. That's my job. Your job, your job is to get me what we learned is you have two jobs. Yeah, wrap it up. We got an early thing. We started it. It's going to be a short podcast, which is much easier to listen to on your commute, because you can have a shorter commute or welcome. What were you going to say? Oh, so you said, if the customers love the product, I think you're paraphrasing. Customers love the product, then you said something like, how does the testing matter? I don't care. Yeah, I don't care how it was. Okay, if customers love the product, I don't care how it was tested. Let me ask you this. If customers don't love the product. Oh, customers don't love the product. It's a testing problem. I'm low on lying. It's not because all, you know, what? Because that happens from often from the same crowd. Customers don't like it. It'll must be a testing issue. It's like, choose one side of the other. No, it means we built the wrong product. To me, it's quality. And I wrote a very abstract and painful to read blog posts on this topic. So I'm not going to recommend it. But, but, dead don't. Quality is first and foremost. How well is this, this thing solved my problem? Okay. Then it's okay. Now, what additional pain in the ass things is it bring to it? It reminds me of systems theory, where one of one of the, particularly, systematics, there is a law in systematics, which says the number of problems in the universe is constant. So oh, yeah, we talked about this like two times ago. I'll pound my look right. It's essentially okay. Great. If it's solving my problem and if it's creating bigger problems for me, then then then my original problem, right, without overusing the word problem, that then becomes the problem. Yes. And that's where bugs and such can come into play. But if, if the initial hypothesis for the purpose of the thing is wrong, it doesn't matter if you tested it well or didn't test it at all. Right. Right. I love the part where I tell you to wrap it up and you go into some big long thing. Well, that's your job. That's awesome. Yeah, that's my job. Okay. Well, we're going to call it good here. I got a little something to play here to work here. All right. And we're going to leave you with that. I'm Alan. I'm Brent. Talk to you soon. Bye.