How Generative AI is Elevating Testing Quality in Healthcare with Richard Kedziora

In a land of testers born by the journey Seeking answers, seeking skills Seeking a better way Oh the test killer of Asian testing podcasts Gliding testers with automation or some nest From ancient realms to modern days they lead the way Oh the test killer of Asian testing podcasts Hey do you want to learn more about how generative AI can really elevate your testing quality in healthcare and AI in general? Well you know for special tricks we have RJ joining us who's the co-founder of Ascendos Solutions, a leader in custom software and data analysis for healthcare with over 30 years of experience in designing and deploying software software projects he's passionate about creating cost-effective solutions that really improve patient outcomes. He also holds multiple degrees and is a frequent speaker and publish author as you could tell Richard brings a bunch of wealth of information and knowledge and healthcare tech especially to the show so you don't want to miss it listen up check it out hey RJ welcome to the guild. Thanks for having me I have to to live up to that now for sure the challenge is on challenge accepted. All right so it seems like AI really is just sweet spot like at a whole at a high level so how do you get into healthcare and then how do you get so passionate seems about AI in healthcare. The thing I always find interesting is people seem to forget that AI has been around for a long time or maybe just Gen AI sort of brought it into the common culture. I was fortunate in high school through the you know 80s that I was involved in technology had the computer was going to go to college for that to get that PhD in artificial intelligence but then took a job and you know started earning 11 but always had my hands in it and developing expert systems over the years and dabbled in multiple fields railroad car scheduling was was intriguing I used to work for us steel weight weight back when in the 90s but I got to healthcare through some consulting that I was doing pharmaceuticals work for a lot of the pharmaceuticals I live in the Philadelphia area and it's called the two or two Carter all their headquarters are right there so it was very easy to work for a bunch of those different companies over the years as a consultant and then trying to figure out where I wanted to go in life and career and make an impact in the world and and healthcare really was it you know I'm not a doctor I don't play one on TV but I understand technology and that technology is about people and so I help now you know stand up my company which we founded though 20 odd years ago with a good friend or Lewis we can develop systems that help lots of people you know a doctor nurses they do a credible things but it's end of one it's one person making a difference and they do make a difference but I can help develop a system then that are impacting more and more people which is just an amazing feeling when you go to sleep at night I love it so as you develop a software for these type of companies do you keep quality in mind or testing is that part of a big part of what you all do or is that they how does that work and how does it specific to healthcare how maybe it's more critical than maybe other like a simple application that's just a web website or something. It testing and quality is at the heart of everything we do for just what you the reason you just said that it is about people it's about their health and wellness and our typical project is not billing so it is important to get the billing right but that's not our typical project we do focus more on what I think of the innovative patient side of things dealing with data particularly as it comes off of the medical device so blood pressure blood glucose information your heart rate taking that information marrying it together with the medical record information your labs and medication to be able to make recommendations you know as we work with startups and it's like oh we have all this data that's fine patterns and show them to the doctor well you need to do a little bit more than that these days because they don't have a lot of time so testing and the quality of those applications is power out and so Estenda is I say 13485 certified which is a standard that says we have a good software development process that it's well documented SOP driven template driven checklist driven and as we do code reviews it's like let's make sure we were asking the questions each time to make sure that we are putting out a quality product and then even beyond the actual software testing the algorithm testing then things usually go into a clinical trial and so we have PhDs on staff that can help design those clinical trials to make sure that they do work in the real world so I used to work for a huge healthcare company and I always thought like this is awful because it's really hard to innovate I don't know if it's because it was an enterprise or because it's so regulated like why even like how hard is it to then create software that is innovative and fills everyone's concerns about getting audited and things like that and we get audited all the time we do self audits and we do we have independent third parties that come in and audit us customers audit us so it always makes it interesting but that keeps us honest and it's like okay we're going to go through this process we're going to do a write and we do have these conversations with various different particularly with the startups that are trying to just get into healthcare and they're like oh wait this is what we need to do and it's like yes you know you need to have this documented process you can't just start writing software and see what happens because there is that level of scrutiny and that it can impact the health of people so there are different levels of that obviously some systems that are you know not directly impacting patient health there's less of a threshold there yeah it does make a difference and it doesn't squash innovation you just have to be aware of this upfront as you're developing these solutions and thinking about it the FDA that if you need to go that far and you're going to get need FDA approval as you sell your particular product whatever you're putting on a market they're your front they're there to help you to make sure that you're doing this right that you're not surprised so if you do think you're going down this road you know talk to the FDA talk to you know there's lots of consultants off there obviously that can help you understand that journey us included and so just make sure that you're aware of what you need to do but yeah it's I wouldn't say it's squashing innovation but any means you just have to be aware absolutely I always think this area is also right for disruption especially with AI so do you see AI being almost like taking us to a new level a new era of healthcare maybe where we can get insights more cost effective healthcare or is that pie in the sky it's still a bureaucracy and you know roadblocks I we're starting to see it already so healthcare is the last industry to really embrace data there's a lot of technology in healthcare the MRIs the image of machines X-rays incredible technology from that perspective but it's the last industry to really embrace data and now that we have the medical record systems out there that are starting to capture more and more information and all of the wearables that are on the market that there's just the amount of them yet you go go on up the amount of information that's available now to the professionals is just opening up the world of possibilities and in that note of being cautious in healthcare in the early use cases are around note taking and summarizing content so low risk you know as you go into any project it we're going straight okay what is the risk of this project what is going to be the impact of it and then that drives your your overall strategy so you know if I'm making you know clinical decision algorithms to you know treat cancer that's one thing or you know if you're just okay here's your steps for the day you know there it's different thresholds of what you have to worry about and think about right so I think specific use cases that you've seen gender VI being applied to healthcare for example back in the day we used to have to do translations into different language we had an excel sheet we'd go to an offshore company I knew that the the language and say this how it should look and then we go look at the application make sure I matches seems like AI would be very easy you know I just say take the this UI translate into this language and boom I have I have the correct translation is that or is that easy and that's already been done for a while so the biggest earliest use case that's getting quickly picked up I I referenced the second I go but it the idea of ambient listening so doctors nurses healthcare professionals don't want to sit there and type on a keyboard and end up their notes it's let the computer technology listen to the conversation in the room to be able to take those notes to enter the information in the medical record system so that's quickly becoming you know I would think a top use case in medical that's being rolled out faster and faster these days to free up doctors to do what they want to do there's I've heard it called pajama time it's like when you're done seeing patients and you got to go home after dinner and to the night now you're entering all your notes into the system and it's like oh you know go have fun with the family. Go out, you know, have fun with friends instead of taking all those notes, let the power of the technology take that on for you. Low risk again. And then likewise, it's also summary of that information in the EMR, which is interesting. You referenced this, the idea of language translation. And I think that is a powerful use case. But as more and more people start looking at what these systems are capable of, you have to think about the training data. And so these systems have grown out and just vacuumed up everything on the internet. The vast amount of information on the internet is in English. So there are been various research reports that talk about that language translation into Spanish or German Japanese is not as good as it is in English because of that smaller training data set. People are taking this challenge on and creating better and better systems. But that was one of my early assignments. My wife happens to be a nurse, manager at a practice and they do bring in language translators. I was like, oh my god, Gen AI, this is going to like solve your problem. You're not, you know, you're able to do everything. But that's where you can't rely on it because especially if you don't know Spanish, it's going to sound good to you because you don't understand Spanish. But then that Spanish speaker is going to be like, what are you talking about? Is this going to lose those nuances unless you that native speaker or have the ability to translate between the languages? I'm not going to pick them up. I'm not going to know Spanish. I'm like, hey, it's doing a great job. But you know, my not be. So you do have to be cautious of that. We're an early day still. It's funny to think of, you know, but we are in early days and maybe two years. Since Gen AI sort of hit the market, but we're still testing and still trying out and figuring out what it's good at. How do you avoid over reliance then, an AI? Oh, it's all powerful Oracle. I'm going to apply it. It's translation beautiful. I'm sure it's an AI to an L.M. It's been trained. I'm rolling with it. Yeah. It is. It's I think of it as an intern. Think of it as, you know, that first year person, you know, it's very capable, but it also sounds very trustworthy. Sounds very authoritative when it just spits out something. It's very easy to be like, okay, it's right. Now you have to, you know, go back and double check it. You know, we do use it a lot. I, you know, huge advocate for it, but it's an 80% solution. Never just sit here and be like, oh, yeah, okay. That's X, Y and Z. There's a various stories where lawyers have gotten in trouble, you know, different professionals have gotten in trouble because, you know, they just took it at face value and didn't cross check the information. Because you know, even, you know, in the case of the lawyer, like made up case references. And the lawyer didn't go, you know, check that case reference and to make sure it was real. So you have to be careful of it from that perspective. Same thing in the field of medicine. There's a story of a couple months ago. It's like, how many rocks a day do you eat? Three rocks a day to be healthy. You would I know, you know, you're not going to eat rocks, but, you know, insert other food item that's not as good for you. You know, stickers, candy bars and, you know, maybe it doesn't give you quite the right advice. So you do have to be cautious of it. And it it uses it because it's taking over the world. When Oprah had a special on in a couple of weeks ago, that's when I knew, okay, we had a flash point now. Oprah's talking about it. So you make a good point though. God, imagine garbage out like, how many rocks I like so how do you is any suggestions you had for testers to make sure that when they are interacting with Jenny, they are using good prompting techniques to make sure they're getting good output. Yeah. There is a couple things to think about when you're in Jenny, I systems one, the hallucinations. Is it just making stuff up? So you got to be aware of that as you're testing and how do you mitigate against that? Is there bias in the data that was used? So if you start out with a generic large language model and then you tune it and train it, add in your specific information material. So you work for a major health system, what information do you have available? You're going to feed that into the model to make it more specific to you. So is it accurately representing that information? So you got to be careful of that as well. So it is, it's the challenge here is it's non-deterministic. So I can answer that ask the same question and five minutes later you can ask the same question again and you're going to get it due for an answer. So it's not just testing, it's also monitoring over time to make sure that it is performing as you expect. Yes. So in the pre-show you mentioned health care is very high level. There's a lot of different verticals within health care. So I assume each other vertical has its own standards, its own regulations. Can we use AI to say, hey, we tested this wearing clinical trials? Does this meet all the standards that we have to adhere to? I don't know, is that that's probably too vague but could it help you do something like that? This is where that bias in the training data and where you mentioned garbage and garbage out, but it's even how do you code, you medical information at your entity. So we've seen examples of this and the FDA is even called this out that the industry has to be somewhat self accountable for this. You can't just take an algorithm that was generated in one hospital and move it into another hospital without testing it. What are the specifics of your population? So if I live in the suburbs of Philadelphia and it's like there's a very specific demographic of people that live here. So now I take that and move it down south and there's a different demographic or remove it out west or Canada or Mexico or Japan. There's different demographics of people. So there's the same algorithms that were developed here, work for those populations, probably not, which is why you need to test and as you're developing them, make sure that you are using data set that is represented of a wider population. So do you get pushed back about using AI, especially with data because a hippocopliance like leaking real patient information for test data somehow or having these LMS being trained on it and all of a sudden it could maybe spit out a real patient data when you're prompting it. Who knows? That is in my opinion one of the overblown concerns. Really? Okay. That actually happening. I'm not saying it's never going to happen but it's an overblown concern that if I'm a doctor and I put in some information that somebody else somehow prompts us and RJ Kedsiora, male of in the 50s comes up with some specific information. And then that person knows who I am and can be able to do anything with that. You know, I would not enter that specific information so you can use the systems, you know, for physician and you have a specific question, you can, you don't have to put my name in. You know, to be able to do that, you don't have to put an identifier in there to be able to get meaningful feedback from it with HIPAA, you know, if you're a healthcare provider, you need, you know, we sign business associate agreements all the time. Then that makes us accountable for the use of this information as well. The large language models out there, these corporations are now signing the business associate agreements. They weren't early on but they are signing them now and be able to like hold your data separately so that's not incorporated into the model to address these concerns. So, you know, as they're rolling out new models, you know, make sure, you know, what is your business associate agreement actually cover? Does it cover, you know, the use of, you know, the current model, the model that just came out last week, probably not, you do have to be aware of it. And think of these, it's all about risk is what it is. So, when you develop yourself or yourself for all these different clients, do you actually use Gen AI maybe to create user stories or test Nero's from code or like how do you actually use AI in your day to day type of delivery? I do. It's even starting as early on. So, I've been doing, you know, you think about trying to figure out what your system is going to do. What are the user requirements and you're going to do interviewing, you know, the experts or just the user, your system. And I've been doing these types of interviews for decades. And so I can, you know, I know I can do them. But by entering the input and my parameters into say chat you be to your cloud, it's spitting it out a lot faster for me. And getting the going, it's that 80% it's like that I can go and tweak those questions and and make them different, but it's an efficiency thing. So starting very early on in user interviews, it's like I'm developing a system that's addressing prostate cancer, you know, at a specific institution. If the doctor with 30 years of experience, he's written these articles, you can tell it like you journal articles, you know, what questions I need to ask him to, you know, start getting to the core of the system that we're going to develop. So doing that, writing requirements. It's the same way as you were, you know, record those interviews with all of the different people. And then the AI systems are able to extract the meaningful pieces of information from those again, it's an efficiency thing making life a lot easier. So lots of opportunities to use it from that perspective. And then once we do have requirements created, you know, we do a lot of human review. But you can also plug them into the system. And I always love asking the question, what did I forget? What is that other thing that I forget, you know, I referenced our process. It's very SOP template driven, checklist driven, to make sure that we're not forgetting all the little things that you need to think about when you're developing a system. But use the AIs to help you with that. There's a standard ISO 25,000, I think it's 25,000, where it talks about look at scalability and usability and stuff like that. So, as I'm prompting writing those prompts, it's like, "Okay, based on this ISO standard, which covers all these things, what am I not remembering to ask?" Kind of thing. And it can put those things together. Well, it's a huge time saver. Which is really nice. But then we can continue down the chain. Now I have requirements. Now I need to write tests. You from the software developer perspective, from a QA perspective, write those tests. I've done an example a couple times now where you have a screenshot. Take the screenshot, drop it into one of the models that has a vision capability. Can ingest that image. And it's like, "Right, test cases for this." And that's where it truly amazed me months ago when I first did this. Like, typically, if you have a data input field and you put a little asterisk next to it, which means it's a required field, it knew that. And so, writing test cases that, "Okay, user name is a required field." But then it even helps with a security testing. Sequel injection. You have to make sure that SQL injection isn't a problem. If you're newer to the field, maybe you don't remember to test SQL injection. Or you're not familiar with it. So you can very easily then just be like, "Oh, wait, what is SQL injection? How does it work?" To really help you out from that perspective. And then I'll say probably last is test generation, particularly in healthcare, with the wearables, with the electronic and medical record systems. There's so much data that you need to be able to test these systems. It's beautiful for creating data for you. A hundred percent agree. And I love the concept of doing these interviews. It's almost like if you've ever heard of behavior-driven development, you can create personas of people based on those interviews and then be able to say, "Okay, as John, I want to do this, this, and it could probably come up with some really good ideas." So you're not being replaced. It's almost like it's getting you more ideas of like, "Okay, how can I make sure I'm creating test cases for all these different things that persona, John is going to do?" Oh, absolutely. And that's the beauty of it because it does help build that creativity, that brain storming. It's like, "Okay, talking to John who's a surgeon generate me 10 questions." "Okay, generate me 10 more questions." "Generate 10 more questions." "Generate 10 more. " And it will keep Jack hurting more questions for you. You know, there will be repetition in that, but it just gets the creative juices flowing and produces your time to get to that point where you have a good set of questions. And also, I like the point about security testing and things. You probably could say, "Hey, what kind of tests am I missing?" And it may say, "Hey, you missing security testing." And if you don't know, like you said, you could say, "Okay, give me some examples of some good security tests that I can run against this specific application." For sure. Yeah. And it's fascinating. I did because we do, you know, healthcare and work in the wearable data kind of thing. Yeah. I happen to do triathlon so much, very much focused on heart rate and different things like that. And I have been able to say, "Okay, I have a person that's out of shape and 60 years old and you know, generate me a heart rate profile of a person running a mile." You know, this. And it's like, "Okay." And it generates that data. It's like for every minute, here's, you know, specific it and it starts out, you know, say, 90 beats per minute and it goes up and then, you know, towards the end of the run, it comes back down. And now I have a nice profile of this person's heart rate over time. And I was like, "Okay, generate me the same data for a 30-year-old athlete who's in really good shape and it just amazes me because then it drops the heart rate down." And it's like, it doesn't, because it doesn't understand heart rate, it doesn't understand exercise parameter or age or anything like that. But it was able to generate a good representative data set that differentiated between an out of shape 60-year-old and a good, good in shape athlete in their 20s or 30s. And it's like, it differentiated those and created it. So it was very meaningful to be able to help test systems. No, it's interesting. I know someone that wants to get into running and he had a coach that slowly is then to him with the plan, like, this we do this day one, day two, day three. Almost sounds like he can use this as a coach as well for pretty much. Yeah, it's a good idea. Right. Nice. So does it, when multimodal coming out, do you find it like, sometimes you could feed it like an image of a screen and say, "Hey, can you help me?" From a user's perspective, can you tell me how my user would interact with this or anything I should test to make sure my user is going to have a good experience. Can you get that for a road? Yeah, we do. Because it's partially, it started out as just as curiosity. It was like, "Well, this actually work." And it does. It's fascinating. You know, it'll be able to differentiate the fields. And I think it's a key thing that if it's not interpreting it correctly, what is that telling you? Is it well designed? Do we have to reconsider how we're using or positioning it to your element? If it didn't recognize it. Maybe it is still a good design and the AA just couldn't interpret it properly, but it should cause you to ask that question. They're like, "Okay. The AA system can handle this. What is a human going to do with it?" 100%. So, Argy, for your company itself, maybe a little background, what you all do. So if someone's listening, they're like, "Oh, I'm trying to develop a healthcare application. Maybe I need a little help here with my AI strategy or development strategy." Is that something you would help them with or could help them with? Yeah. A standard is a professional services company, so we don't sell products. We help other companies develop their products, develop strategies. And we do. We work with a mix of large corporations, government agencies, particularly the R&D department. If we work with health systems, it's going to be with the PhD researchers. It's usually very cutting edge, which is where our sweet spot is. We might not quite know what you want to do. We're going to help you develop this system and how it's going to operate. We've had a couple clients over the years put our name on patents, because we've been so much part of that process of creation. They're like, "Okay. Yeah, let's put their name on it." We never own anything. You as the client always own everything, but it lets to explore lots of things. As we have those AI conversations, we're today, a lot of conversations start. One of the first early questions is always around what's your data strategy? What's your quality strategy? Like, in some companies, particularly more on the startup sides, they don't have that figured out yet. And you're not going to get very far if you don't have your data strategy. If you don't have a quality strategy in place, as you start thinking about the AI, it starts with the data. Garbage, you can garbage out. You said it. Yeah, for sure. So what's cutting edge? You could sort of cutting edges. Anything you see on the horizon, like, yeah, Gen AI, I get it, blah, blah, blah, blah. Is it something you're like, okay, people need to be aware of this because this is about to take off or anything you've been messing around with in the lab. The big aspects of where all this is going, I think is one, the intersection of wearables, because more and more of these devices are coming out. They're more and more capable, generating more and more data. So it's the intersection of the wearables and the AI to interpret that data and give it back to the person. It's getting care out of the four walls of the hospital. So me as an individual, or even as the population ages, this is a concept of care at home. So as my parents are getting elderly, fortunately, they're still doing pretty good. But you don't want to have to move them into an assisted living home. They're very comfortable in the home where they live. So how can we apply these tools, the technology, to understand their living environment? That's where a lot of this is going beyond the four walls of the hospital. So if you know your mother got out of bed this morning that she used the facilities, she brushed her teeth, she opened the refrigerator, she turned the coffee on, she turned the TV on. Okay, she's having a good normal day. But and you're not sitting there with cameras in the room like observing her. And so she doesn't feel like she's being observed. Or you're protecting her privacy a little bit more at home where she's comfortable. And then one day, it's like, "Oh, okay, wait a minute. It's not a clock. She's usually up at six in the morning." She didn't get out of bed. She didn't turn on the clock. She didn't turn them. You know, let me pick up the phone and they're like, "Hey mom, how you doing?" Very simple use case of that technology, which is where this is going. All right, so this might be a little wild. But have you seen the clips of the Tesla robot? How close are we that to like care-given where not only can we wear wearables, but also we can use some sort of robotic assistant to almost, I don't have kids, so I'm betting on or hoping on these robots to really do that thing in 20, 25 years. The technology can help you. I got deeply fascinated, particularly by what, you know, Alon did recently kind of thing. It does turn out there. There was a little sort of magic behind the scenes in what he was showing. And it wasn't as up front as it looked like on those videos. But to his point, the reason he was doing it and the reason he was comfortable, he's like, "Look at how far we've already come. This is around the corner." So it really is. And Japan as a culture has really embraced this. So they are at the point where they do have like robots that are in the home to help care. I think you know, it's not totally autonomous. I miss kind of thing just running around like the Jetsons or anything like that at this point. But yeah, look to Japan, if you're really curious to where this tech is going and what it's becoming capable of, Japan is really embracing technology. But yeah, it's coming. It's not quite right around the corner, but it's not that far down the road to having that robot to provide care at home. And you have to think about it, you know, self-driving cars. It's like, okay, a lot of people are self-scared or apprehensive of self-driving cars. You don't have control anymore. But what's better driving in the car? You or where that computer that's in the car and it's the computer that's better driving. 100%. But Karajay, before we go, is there one piece of actual voice you can give to someone to help them with their AI testing efforts? And what's best way to find contact you'll learn more about Standa? So to get a hold of us, you know, our website, thestanda.com, my LinkedIn always works at these RJ kids of your, love to have these conversations. It just always fascinates me what people are coming up with and thoughts. My one piece of advice is use it. And it's that simple, but you need to do a little learning and education. And there was this idea around crafting the right prompt. And okay, how do I write a better prompt? Well, one of the fascinating things is just ask, ChengchipuTi or Klau, or any of these solutions to write me a better prompt. You don't need to become a prompt expert anymore. Ask it to help you. Ask it to ask you questions to get to, you know, a better answer. If you're not sure what to ask it or how to use it, ask it to help you and it will, which is one of those things that's amazing. And lastly, I would say it's not gonna replace you, but people using it are gonna replace you. So get to know it. Thanks again for your automation awesomest. The links to every other value we covered in this episode hand on over to testgild.com/a521. And if the show has helped you in any way, why not rate it and review it in iTunes? Reviews really help in the rankings of the show when I read each and every one of them. So that's it for this episode of the Testgild automation podcast. Joe, my mission is to help you succeed creating end to end full stack automation awesomest. As always, test everything and keep the good. Cheers. Hey, thank you for tuning in. It's incredible to connect with close to 400,000 followers across all our platforms and over 40,000 email subscribers who are at the forefront of automation, testing, and DevOps. If you haven't yet, join our vibrant community at testgild.com where you become part of our elite circle, driving innovation, software testing, and automation. And if you're a tube provider or have a service looking to empower our guild, what solutions that elevate skills and tackle real world challenges, we're excited to collaborate. Visit testgild.info to explore how we can create transformative experiences together. Let's push the boundaries of what we can achieve. ♪ Over testgild automation testing podcast ♪ ♪ With lutes and liars the bugs began their song ♪ ♪ A tune of knowledge, a melody of code ♪ ♪ Through the air it spread like wildfire through the land ♪ ♪ Guiding testers showing the secrets to behold ♪