#458: What the FDA Actually Says About AI in Medical Devices

Welcome to the Global Medical Device Podcast, where today's brightest minds in the medical device industry go to get their most useful and actionable insider knowledge, direct from some of the world's leading medical device experts and companies. Hey guys, a few years ago, when I first started paying attention to AI, it was really interesting. It was a novelty. I was fascinated by it, but also mostly amused by it. It was very interesting. It was fun. You can make all kinds of different videos or pictures mostly. I think I made my first AI video in 2020, late 2022, maybe early 2023, and it was hilarious. It was a picture of my face and my mouth moved to read a script that I've provided with it. It provided it. Well, pretty quickly after that, people started feeling kind of AI fatigue. They weren't really, I mean, they were pretty much dismissed as a toy after a while. People were so sick about hearing about AI that just mentioning the word and a headline was enough to make everybody scroll past it. We've probably gotten back to that point again as well. I got pretty sick of hearing about it myself as well. But something has happened that really we can't afford to ignore. When the FDA started incorporating AI into their own scientific review process, they started using it internally, if you remember, it's called ELSA. The moment that happened, the moment the regulators started using that binge-worthy word, the industry couldn't pretend that it wasn't real anymore. Suddenly, AI was everywhere in Medtech. Companies are building it into their devices. QMS platforms are adding AI features. Every submissions are all being built by AI. People are talking about this in all different ways. I kept hearing the same questions from people in the industry. What does this actually mean for us? What does the FDA actually think about AI in medical devices? What are the rules? That's what today is about. Not the hype, not the things that, you know, but I want to talk about the actual rules related to AI in Medtech. Before we get into the actual guidance documents, I want to give you an idea or a framework or a way to think about this because I think most people are reaching for the wrong comparison when they try to understand what AI is or what AI is doing to our industry, the medical device industry. People say AI is like the internet, and I get why the internet changed everything about how business worked, but I don't think that's the right analogy. In fact, if you think about AI from maybe if you're in marketing or something like that, it does feel very similar with the internet and how business changed, but I think it's deeper than that. I think AI is more like electricity because, well, I'll just tell you what I mean. Before electricity, people used candles and lanterns to light their homes, and those people understood the light sources completely. If your flame was weak, you maybe you trimmed the wick. If the lantern was running low, maybe you needed to add some kerosene or fuel. You could diagnose every problem yourself. You were the expert. Now, I've built an off-grid house that I live in. I wired it all myself, so that's why this analogy came to me and it's so poignant for me personally, because after the candles and the lanterns and that era of life, electricity came along, and it was better in almost every way. It was easier, faster, cleaner, brighter. There was way less risk of a fire. You didn't have to trim those wicks before you went to bed or when you were waking up from bed. But here's the thing. Most people had no idea what was going on inside their walls, those wires, the breaker box outside, the transformer down the street, the whole network that had to work exactly right before that switch in your bedroom did anything. And if something went wrong, you didn't troubleshoot it yourself. You didn't trim the wick on your wires. You called an electrician. You had the national electrical code, the NEC, which I've actually now read quite a bit of in wiring my own house. But most people are not doing that, most people are hiring that electrician. So now we have inspectors, we have permits, we have certifications. We have journeymen who have studied under other people for years before they start doing this for other people's house. But we didn't give up electricity because it was dangerous. We did it because electricity, we stuck with it because electricity was powerful enough to matter. And when something matters that much, you build standards around it. And so I would submit to you that that is where AI is right now. We're on the precipice of a mountain of standards and maybe a, a, a centralizing that look, that information into that subset of people, the electricians of AI, if you will. So here's the other part of the analogy that I think it's missed. Think about it this way, if you were a toaster company in the early days of electricity, you used to make toast by, by heating it over a grill, perhaps some kind of direct flame situation. And then you switched to electric coils. You didn't go around telling everyone, we're an electric company now. We're an electric toaster company. That wasn't the point. The point was that you can make better toast. You can make it faster. You can make it more consistent. You could plug it in and go. The electricity was just the thing that made it work. Your customers didn't care about the electricity, they only cared about the toast. So that's the other thing you need to think about AI, artificial intelligence and medical devices is the same. The device companies that are going to win aren't the ones making the biggest deal out of having AI. They're the ones whose devices actually work better because of it and whose quality systems can prove that to the FDA. So that brings us to what the FDA is actually asking for. That's what we care about, right? Well, let's start at the foundation because if you don't understand the framework, the FDA is thinking in the specific guidance documents, they're not going to make a lot of sense. So first thing I tend to understand is that the FDA has a category called the Software as a Medical Device, SAMD, not SAMDA or a SAMD. Let's just call it SAMD, okay? If you have a different way of pronouncing it, hit me up. And I know this is a little bit simplistic for some of you, stay with me here for a minute. But if your AI is doing anything that falls under the definition of a medical device, whether it's analyzing patient data, supporting clinical decisions. And I know we have clinical support decisions or clinical decision support software and a recent guidance about that. We talk about that at another time, but detecting conditions. A lot of these are probably SAMD. That matters because SAMD has a full regulatory pathway. It's not an accessory. It's not a feature. It is a device. All right. Let's talk about locked versus adaptive algorithms. This is another distinction that's, I think it's important for everything else we're going to talk about today. A locked algorithm doesn't change once it's deployed. You trained it, you validated it. It performs a defined function and it stays that way until you deliberately update it through a controlled process. You decide how it's going to be able to change. An adaptive algorithm continues to learn from new data after the deployment. It changes on its own, which makes total sense and it's really where the power of AI can actually be harnessed. The FDA has been pretty clear. Adaptive algorithms are fundamentally, they are a different regulatory challenge because the thing you validated is not going to be the thing that's running in the field six months later. So, if you're thinking, well, wait, that means adaptive AI is impossible to clear. Stay with me because that's where the next thing comes in. That's where PCCP predetermined change control planning comes in. All right. So, let's think about the total product life cycle approach. FDA's approach to AI isn't submitted once and you're done. It's more of a life cycle model. They want to see that you're monitoring the algorithm's performance after market release. That you have a plan for what happens when it drifts. If it drifts, that your quality system connects the dots from the development all the way through to the post-market surveillance. This is the biggest mental shift for companies that are used to thinking about FDA clearance as a finish line. With AI clearances, more of a checkpoint, you're going to have multiple of these checkpoints along the way. Okay. So, what are some of the guidances that are important to read? We can walk through some of those actual documents. The 2021 AI ML Action Plan is one we're going to consider. Now, this is a little bit older, but the FDA published an action plan in 2021 that laid out five specific areas they were going to develop guidance on. It's worth reading even now because it tells you where the FDA's head was and it explains why the guidance that came after it looks the way that it does. So, the five areas were software modification, the pre-determined change control plan, good machine learning practices, patient-centered approaches, transparency, and real-world performance monitoring. Notice that most of these are about the process, not the algorithm itself, which I think is intentional. The FDA wasn't trying to become an AI engineer or a source of AI engineering. They're trying to make sure your engineering process is sound. Now, if you remember, in 2023, there was a guidance on pre-determined change control plans. PCCP. It was originally just for AI ML, later on, it came out just for a more of a generic PCCP. And if you, if you, the reason this came out the way that it did, well, I don't want to get into the omnibus act necessarily, but it was actually in the code of federal regulations. So there was an update. in the Omnibus Act, I think it was December of 2022. But this is one that most companies haven't really, I would submit to you that most companies probably have not fully leveraged the PCCP. And it's one that really is important for AIML. So a predetermined change control plan is exactly what it sounds like. It's a document you submit to the FDA that says, "Here are the types of changes we anticipate making to this algorithm after clearance. Here's our protocol for making these changes. Here's how we'll know if the performance is still acceptable." So the FDA's guidance on the PCCP's cover three main required components. One is the description of the plan and modifications. What kind of changes do you actually expect? Is it going to be more training data? Is it an expanded and tended use? Are there going to be algorithm architecture updates? You have to be specific about the type of changes you're planning, not every possible change in the universe, but the actual categories of modification to your development team is likely to make. Second is the modification protocol. How are these changes going to be implemented? What are the steps the validation is going to require? Who's going to approve those? This is basically a change control procedure specifically for your algorithm. And then third is the impact assessment. For each plan modification type, what's the expected impact on device performance and safety? What testing would you do to confirm? It's still safe and effective. What is the impact? I mean, it says straightforward as that. And the failure mode that you'll probably see more often than not is that companies are going to treat that PCCP as a submission document. Something you write right before you file. But the PCCP actually has to reflect your actual development process. If your development process is built around it from the beginning, the FDA is going to be able to tell. You almost need to treat this like its own standalone submission that is a part of the submission. So you need a full understanding of all these different processes. This is the 2021 draft guidance that is called the 510K considerations for AI ML-based SAMD. Few things worth highlighting. Training, validation, and testing data. The FDA wanted to understand where is your data coming from? What population does it represent? How was it split between training and validation and testing? Is it representative of your intended use population, whether it's geographically, demographically, clinically? The last one's important as well. There have been additional guidance documents that have come out about the demographics and the clinical importance of your medical device. It's a safety issue. If your algorithm performs well on one demographic and doesn't on another, and you don't know that, you don't have any-- somebody else is going to have to tell you that. That's a problem. Also, there's something called algorithm transparency. The FDA wants enough detail to evaluate the algorithms performance and understand its failure modes. You can't have a proprietary blend, all right? They're not asking for your source code, but they are asking for you to explain how it works well enough so that a reviewer can assess whether or not that is safe. Here's where I want to bring back the electricity analogy. So when electricity became the standard, every house needed a certain kind of wiring, right? A breaker box, grounded outlets. Now you actually have to have tamperproof outlets where they're very difficult to stick the-- what is the thing called the end of the wire. It's hard to stick them to the outlet now. I've learned this the hard way. The infrastructure has to be there before any of the appliances could do what they're actually supposed to do, right? It's all has to be a part of your house. So your quality system is that infrastructure for AI. The companies are going to sail through an FDA review on an AI-enabled device. Those are the ones that are-- they've built their QMS with AI in mind from the start. They're not the ones trying to retrofit their existing system where it's time to file. Specifically, there are a few things your QMS has to be able to handle that most traditional QMS frameworks aren't going to be built for, OK? Number one, design controls that account for the algorithm change before it happens, not just after the fact in your change management process. Risk management under ISO 14971 applied to AI-specific failure modes. What does a false positive mean for patient safety? What about a false negative? What if your algorithm performs well and aggregate but fails in a specific subpopulation? Those are some failure modes you're going to be documenting under 14971. But they're different from a mechanical component failure. And if you're struggling with ISO 14971 and how to do all of those, I believe it's 34971. Let me look that up actually really quick. ISO 34971 that talks about applying this to an AI or ML device. Yes, application of ISO 14971 to machine learning and artificial intelligence. So this is going to be incredibly important as you're building your QMS. Third thing that I'll say is the post-market surveillance. That's actually connected to algorithm performance, not just complaint handling, which historically is what the medical device industry use post-market surveillance. So that's where they got-- they waited for people to complain about it. That's not going to be acceptable if you have an adaptive algorithm, active monitoring of how that algorithm is performing in the real world with a defined process for what happens when, not if it drifts outside those acceptable bounds. So that's going to be important. Post-market surveillance is going to be incredibly important, not reactive, but being proactive. And it's a different kind of quality system than most companies have built. And the earlier you build it, the less painful that whole process is going to be. All right, here's what I want you to leave and walk away with today. AI is not going away for medical devices. It does not matter if you have AI fatigue. You and I both were in the same boat there, FDA is not trying to stop this. They're trying to figure out how to let it happen safely. And so the guy in the stockings we talked about today-- and I'm sure they're going to be more that are coming-- those are-- they're attempt to build the National Electrical Code for AI. Your job, as a QARA professional or as a device company or product development engineer, is to understand those rules well enough that you build your device and your quality system around that national, that NEC, National Electrical Code for AI, whatever you want to call it. Before you're in the middle of a submission, wondering why you're getting deficiency letters. When I was building my house, this off-grid house that I've built, I have, I don't know, I think a 20 or 30 breakers in the breaker box, I had to have an inspector come and inspect it, had I not done a little bit of research and recognized what kind of breaker I had, he would have made me change every single one of those. And he actually told me, he said, this latest National Electrical Code change has actually increased the cost of homes by $1,500 to $2,000 just in breakers and wiring alone. It's important to stay up to date on these things. So next week, I want to talk about the mistakes a lot of medical device companies are coming or are incurring, I suppose, the patterns I see in AI submissions that trip companies up, the things that I've heard, not because they have bad AI, but because they didn't build the right foundation around it. So hopefully we'll see you then. If you found this useful, send it to one person on your team who's working on an AI-enabled device right now. Send this to someone who's interested in understanding AI in the medical device industry because there's more than just the AI-enabled devices. We're going to get to talking in the third part of this series about the processes you can use when you're using AI, not necessarily for an AI-enabled device, but for AI in your company itself. So if you're interested in learning more about that, hit me up on LinkedIn, Etienne Nichols, I'm very active there. I'd love to hear any thoughts or questions you might have, but stay tuned for the next two episodes. We'll see you then. (upbeat music) Thanks for tuning into the Global Medical Device podcast. If you found value in today's conversation, please take a moment to rate, review, and subscribe on your favorite podcast platform. If you've got thoughts or questions, we'd love to hear from you, email us at [email protected]. Stay connected for more insights into the future of MedTech Innovation. And if you're ready to take your product development to the next level, visit us at www.greenlight.guru. Until next time, keep innovating and improving the quality of life.