Interpretability and Data Challenges with Duke University's Dr. Cynthia Rudin

(upbeat music) - This is the AI Health Podcast, where we explore the ways in which AI will transform healthcare, biotech, and medicine through conversations with entrepreneurs, investors, and scientists. - I'm your co-host, Pranav Rajwakar. - And I'm Adriel Seporta. - And you're listening to the AI Health Podcast. - Today, we'll talk a little bit about machine learning for prognosis, the use of black box models, and the importance of interpretability. - I'm excited for that. Let's start with prognosis. Prognosis is a branch of medicine that specializes in predicting the future of patients. Doctors make medical prognoses on a routine basis, and both the methods and the tools have been evolving to help doctors make more accurate and individualized predictions about a patient's future health. AI is a powerful toolkit for prognosis, and can provide a tremendous boost to this field of medicine by using many different types of medical data to make accurate predictions about a patient's health in the future. - That's awesome. I'd love to chat about what prognostic models are. - We can think of prognostic models as a system that takes in a profile of a patient as input and outputs a risk score for that patient. The patient profile can include the clinical history, which includes major illnesses, any previous procedures. The profile might also include physical exam findings, such as temperature and blood pressure. A prognostic model then takes in one or more of these pieces and outputs a risk score for that patient, like the risk of having a heart attack. - I know you dive deep into prognostic models in course two of your AI for Medicine Coursera specialization, and talk about how one can develop and evaluate machine learning models for making these prognostic predictions. - That's right. It was a ton of fun building that. Now, we know we can use machine learning to learn from data to build these prognostic models, and sometimes they're simple and sometimes they're complex. Often machine learning models are considered to be black boxes due to their complex inner workings. - Yeah, we've heard this association of AI with black box models a lot. - That's right. A black box model could be either a function that is too complicated for any human to comprehend, or a function that is proprietary. Deep learning models, for instance, tend to be black boxes of the first kind because they're highly recursive. In medicine, the ability to explain an interpretive model may be critical for human acceptance and trust. - I totally agree. And it's good and important to think about AI in terms of the people that are actually using it. So how do we get models to be considered interpretable? - Usually an interpretable machine learning model is constrained in model form so that it's useful to someone. Let's take it back to risk scores here and think about what might an interpretable model look like. For this, I think it's useful to take a look at an existing risk score. We'll look at the Chad's VAS score, for example. This is a score that is used for patients with atrial fibrillation. Atrial fibrillation is a common abnormal heart rhythm that puts patients at the risk of stroke, which is when blood flow to an area of the brain is cut off. Now, for patients with atrial fibrillation, also called AFib, we use the Chad's VAS score model to predict the one-year risk of stroke. - So let's say I'm a patient with AFib and I wanna know my risk of having a stroke. How can I do that? - Well, let's say you were a 70-year-old male diagnosed with atrial fibrillation who had hypertension and diabetes. We could calculate the one-year risk of stroke as following. This model will say that you get one point for your age bracket of 65 to 74, zero points for being male, one point for hypertension and one point for diabetes. You sum all those points to get to three points in total and now there's a guidance out there that says a score of two or greater is moderate to high risk and should be considered for an anticoagulant or blood-tenor treatment. - Huh, so it seems like part of what makes this model interpretable is that this point system and the fairly simple addition of the scores is easy to understand. And it's also useful for a physician to know that a model is relying on features that are definitely medically relevant. - That's right. Now that's interpretability. Now, other than trying to create models that are inherently interpretable, there's also been work on explainable machine learning where a second model is created to explain the first black box model. So for example, you might be able to explain a model prediction by saying that it classifies someone as having high risk because the model is giving high importance to them being old and having a particular condition. - Awesome. So we've talked about prognosis, which is predicting the future health of patients. We've talked about interpretability, which is when the form of a machine learning model is constrained so that it's understandable and useful. And we've seen what this could look like in a medical setting. - One other concept we should chat about before our interview today is data privacy. - Yeah, especially in the medical context, data privacy often involves a balance of competing interests. Making data available for analysis is critical for conducting medical research, but on the other hand, we need to make sure that that data is safeguarded because protecting patients' health data is critical. - Right. For example, if a life insurance company got their hands on certain parts of a person's health data, their life insurance premiums could skyrocket and we don't want that. - Right. And a lot of people and companies will argue that if the data is anonymized, then you're fine, we're safe. But research has shown how little information is required to take that supposedly anonymized data and de-anonymize it. So one of my favorite examples of this is back in the mid '90s, a government agency called the Massachusetts Group Insurance Commission released supposedly anonymized data of state employee hospital visits. So we're talking 135,000 data records. At the time, then governor of Massachusetts, William Weld, assured the public that they had protected patient privacy by deleting identifiers. Professor Latanya Sweeney, who is now a professor at Harvard, but at the time was a graduate student at MIT, decided to do a little sleuthing. She knew that Weld lived in Cambridge. So for $20, she bought the Cambridge voter list, which contained the names, addresses, birth dates, and genders of everyone in the city. She joined this voter data with the supposedly anonymized data on state employee hospital visits and found that only six people in Cambridge shared Weld's birth date. Only three of those six people were men and only Weld lived in that zip code. So Sweeney found the hospital visit records for Weld and then sent his health records to his office. - That's a fascinating story. So the moral here is that it's actually not as hard as one may think to de-anonymize the supposedly anonymized data. - Right. So Sweeney found that 87% of Americans could be uniquely identified using only zip code, birth date, and gender. - That's fascinating. And there are many other examples of this. I remember reading another study that showed that it was possible to create reconstructions of patients' faces using these de-identified MRI images. And then those face reconstructions could then be identified using facial recognition systems. - It's scary. So there's a lawsuit, Dinerstein versus Google, where the University of Chicago Medical Center sold patient records to Google so that it could develop AI products, allegedly violating HIPAA. Just this past September, a judge dismissed the lawsuit, ruling that the patient who filed the suit failed to adequately demonstrate the damages suffered as a result of the partnership. But there's an important line in the lawsuit that reads, Google is uniquely able to determine the identity of almost every medical record the University released due to its expertise and resources in AI. - And Google's not alone there. There have been other tech companies that have made data partnership agreements with healthcare organizations like university research hospitals and insurance companies to gain access to health data for building these AI systems. Okay, so we've outlined all the possibly scary ways that patient data may not be properly shared or anonymized. Have there been any attempts to regulate this? - So our listeners may have heard of GDPR, and that's probably the most important attempt made thus far. GDPR stands for General Data Protection Regulation, and it was passed by the European Union and represented a real sea change in how individuals' data and privacy would be regulated. - And this was intended to protect all sorts of user data on the internet, not just limited to health data, right? - Right, so with GDPR, the EU essentially planted a stake in the ground and said, this is the new standard for protecting all kinds of consumer data and privacy. GDPR has a whole ton of mandates, but in a nutshell, it sets guidelines and limits on how companies can collect and process the personal information of people who live in the European Union. And if companies don't abide by those rules, they can be hit with huge fines. GDPR came into full effect in 2018. So we've been living in a GDPR world for about two and a half years now. And I say that because even if you're not living in the EU, chances are that you've been affected by GDPR. It doesn't matter where a company or a website is based. If the company has European consumers or visitors, then GDPR applies to them too. So Facebook and other big companies all need to comply with GDPR, which means that consumers outside of Europe often benefit from whatever changes those companies make. - And this explains the increase in pop-ups we've seen recently on websites asking us to agree and consent to them collecting information and storing cookies. - Yeah, you can thank GDPR for that. GDPR requires that visitors be notified of whatever data the site collects from you. And to bring this back to our earlier conversation on anonymized data, GDPR also requires that any personally identifiable information be anonymized or pseudonymized. So in other words, the user's identity is replaced with a pseudonym. And personally identifiable information includes age, race, gender, sexual orientation, religion and health information, among other categories. - That's right. And GDPR also governs the quote "right to an explanation" where for any prediction, an explanation of that prediction is required. So the EU has really led the way in establishing regulatory standards for data privacy. - Yeah, generally speaking, Europe has adopted a more assertive regulatory approach than the US. But it's important to also mention the California Consumer Privacy Act, also known as the CCPA, which took effect at the beginning of this year, 2020. And the purpose of the CCPA is to protect consumers in California in much the same way that GDPR did for consumers in the EU. - And given that California alone is the fifth largest global economy, I'm assuming that just like with GDPR, the CCPA will have an influence beyond just California. - Yeah, I think so. And one last point I'll mention on GDPR and CCPA. While there's a lot of overlap between the two, there are differences. And maybe the most important for our world of AI and health is that unlike the GDPR, the CCPA specifically excludes from its scope the processing of medical information. And the idea here is that this type of health data is already covered by the Confidentiality of Medical Information Act and the Health Insurance Portability and Accountability Act, otherwise known as HIPAA. - So we've now chatted about data privacy and regulations surrounding that. Without further ado, let's introduce our guest today, Dr. Cynthia Rudin. Dr. Rudin is a professor of computer science, electrical and computer engineering and statistical science at Duke University and directs a prediction analysis lab, whose main focus is an interpretable machine learning. - We'll be chatting about risk scores, interpretable machine learning and data privacy. - Welcome to the AI Health Podcast. - Happy to be here. - I wanna start by asking about some of the ways in which you think about building statistical models or machine learning models for making personalized predictions in medicine. So for example, you've introduced machine learning models to build risk scores. - Yeah, so risk scores, by the way, just to give you a little history on them, they've been around for a really, really long time, like way before machine learning, way before databases, before all that stuff. These are the kind of models that doctors design before they have data. So these are very simple point scores. You get one point for this, two points for that, another point for this, another point if you have that. And then the doctor adds up the points and then the total number of points translates into a risk. And they've been using criminal justice for about 100 years, these models. So they're very traditional. And the reason that people like them is because they allow you to really understand how the variables work jointly together to form a prediction. You got six points because you got three points for this, two points for that, and another point for this other thing. And that's why your score is six points and so that's why your risk is 83% or whatever. So these scores also allow you to reason about counterfactual. So in other words, you could say, well, you know, my score is six points, but if I was three years older, then my score would be seven points and then my risk would be even higher. So just being able to do that kind of reasoning in your head is why people really like these kind of models. The traditional way to build them is just to have a bunch of people in a room kind of decide on what the model is, right? It's just to have domain experts create these models. And so we said, look, we have a database. We're gonna try and create these models from data. And then what we found was that you could create these models without losing any accuracy over really, really complicated machine learning methods like neural networks or boosted decision trees or whatever you want. And for many of the problems that I work on, for many of the databases that I have, you can create these interpretable models that have the same level of accuracy as the more complicated black boxes. And so that's what's allowed us to build risk scores for problems like seizure predictions in patients that are at risk for brain seizures and are in the intensive care units of hospitals. I mean, there's not too many machine learning researchers who can claim that their work is being used in intensive care units of hospitals because these are really high stakes decisions. And so in order to actually get these models implemented, well, the fact that they were so interpretable means that doctors can take a look at the models. - Got it, so it's just like the old models from a doctor's perspective. The only difference is that you guys are using data to actually come up with those numbers instead of using a group of domain experts sitting around a room. - I'd love to understand what are some of the technical considerations that you've found to be useful to understand while building these models? - Well, so you're trying to optimize for accuracy, but at the same time, you want to constrain it to be interpretable, to be something that you could hand to a doctor and say, "Hey, what do you think of this model?" And the doctor could actually look at the model and say, "Hey, I think that's a bad model." That's what we want. We want the doctor to look at it and be able to judge it. And so the constraints really are, they're very heavy constraints. Like the model has to only have five terms in it or the model has to have integer coefficients or something like that. You know, when you impose those kinds of constraints, you end up with very, very hard optimization problems, mathematical optimization problems where you say, "Okay, maximize accuracy subject to constraint." We found that you need to know quite a lot of optimization in order to actually solve some of these problems. Like the way that we tend to work in my lab is that we sort of say, "Look, let's pretend optimization is not an issue. Let's pretend we can optimize anything in the world. Let's just form our dream problem, right? Let's just write down the problem we want to solve and we'll figure out a way." - That's fantastic. You mentioned a particular application which was seizure prediction ICU patients. I'd love to understand a little bit more about the problem you were solving here. - Sure. So this is work with Bridget Westover and Erin Struck and Burke Biston. This problem is faced by neurologists every day. So they have these patients. They might be stroke patients. They might have a brain hemorrhage. Like they might have a burst aneurysm and there's blood in their brain and then they get surgery and then they're being monitored, right? They're being monitored by continuous EEG monitors and the monitors are like physical monitors or they're on their heads and the patients are being monitored for seizures, brain seizures. So these are seizures that are just in the patient's brain. You can't see the patient shake or anything like that. It's not that kind of seizure. It's only in their brain. And these seizures can be really, really dangerous to the point where they cause brain damage and they can cause death. And so the neurologists do everything in their power to try to prevent these seizures. They might even turn off parts of the patient's brain so that the patient won't suffer these horrible seizures. And so it's really important to be able to predict these seizures in advance, both for the purpose of helping the patient but also for determining whether the patient is out of the woods and they can take the monitors off that patient and give it to another patient that they might need to monitor. 'Cause sometimes you can't tell which patients actually need the monitors until you actually put them on. And so we worked with data from several different hospitals and created this model and the idea is that as the patients are being monitored with these continuous EEG monitors, then the doctors are reading, they're like, oh, they're saying, oh, the two helps, two B scores, three. We need to keep the patient on these monitors for a really long time, for like 72 hours at least. Or, oh, the patient's two helps, two B score is one. Maybe we only need to keep the patient on for a little while longer and then we can hand the monitors off to someone who doesn't have them. It's really helped quite a lot. The doctors that we worked with did a validation study that I wasn't involved with where they looked at the benefit of having this extra model. And they found that it allowed them to monitor over 2.8 times more patients than they were monitoring before, which really leads to substantial reductions in brain damage and death overall for these patients. - And just so I understand sort of the setup, the idea is that the doctors are normally monitoring the monitors that are sitting on patients' heads, but now in addition to that information, they also have access to these risk scores. It's just another piece of information that they can use when making decisions about how to treat the patient. - Yeah, that's right. - Okay, how is it possible that they can monitor more patients this way? Is it just because they can go through them more quickly? - Well, they can take the monitors off the low-risk patients 'cause the monitors would tend to sit on patients way too long, right? You have these patients who actually need the monitors and then this patient who's low risk, but you're not sure and you don't wanna take the patient off the monitor and need some justification. So now if we can allocate the monitors better, then we can monitor more patients. - And this is going back a little bit to the constraint problem that you were talking about, but when you're building these constraints into the models, I'm assuming you're also working with domain experts as well to build those constraints in or not at all. Is it just separate from the domain entirely? - So it always needs to involve the domain experts. For the models that we've built for healthcare, we generally try to build risk scores, to be honest, because those are very familiar to doctors. Like they have many websites with different risk scores that are used for different specialties of medicine. And so we try to stick to those, but often doctors have more constraints that they're not able to articulate. So what happens is you can show the doctors a bunch of scores that you've produced, right? So let's say we produce 15 different risk scores and then we show the doctors all of the risk scores and then they say, well, that one doesn't make any sense because of blah. And we were like, oh, well, we didn't know about that. Let's just put that as a constraint. So we'll never do that again. - And so now you developed this model a few years back. Have you found it being adopted by clinics? Are you validating it further? - Yeah, so my collaborators did that validation study that I mentioned, but you can go to like websites like medscore.com and you can actually see it on there in the neurology listing. So if people are looking for a score of that kind, they can go and use it. The nice thing about these scores is that you don't have to trust them. They don't force you to trust them. - I'd love to dive a little bit into the technical details of the development of this. So you mentioned you cast it as an optimization problem and having read this paper, you chose to formulate this as a mixed integer programming problem. I'd just love for you to be able to describe to the listeners what this formulation is and why you chose this formulation for the problem. - Oh, fun math with words. So it's actually a mixed integer non-linear program which makes it even harder than a mixed integer program. It's actually the kind of problem that if you put it into a generic, they have these solvers, right? If you put the math problem into a solver and say, go solve it, the solver just chokes. It just says, okay, come back at the end of the universe. I'll solve it at that point. And yeah, it's a really, really hard math problem. So the reason it's hard is because you're optimizing, let's see, so you want to make sure that the accuracy is high, right? And for accuracy, we use a common logistic loss, okay? So logistic loss tries to make sure that the risks that we get out are calibrated to the data and that the model is as accurate as possible. The next thing we want to optimize is the number of terms in the model. We want to keep the number of terms really small so that we don't have these risk scores that involve calculating 20 different things. 'Cause like these are doctors, they're working very quickly. They have to analyze the EEG score very quickly and they want just a few reliable measurements, okay? So we told it, okay, please keep the model nice and sparse. And then we constrained the results to be, again, point scores or integer coefficients if you want to say it that way. So that whatever the model spits out, it would be you get two points for this. You get one point for that. You get one point for this. You get one point for that. You get 10 points for this. So all of the point scores had to be between negative 10 and 10. And then there were some additional constraints that some of the points had to be positive just based on what the doctors told us. These were the kinds of constraints that we had. So when you put all that together, then you get a mixed integer and nonlinear program whose feasible region is the integer lattice. And yeah, then it's a really hard math problem. But luckily, our lab specializes in solving these kind of interesting math problems. - And if I just contrast that to what might be a traditional approach, you would take your data about many different input features and then put that into let's say a logistic regression model or one of these decision trees, which would say here is the complicated relationship between all these variables where we would not be able to control how this relationship maps to the outcome. Is that a fair way of describing the difference between a traditional approach versus the approach you describe where you have control over all these components? - Yeah, that's right. Like I said, we create our dream problem and we find a way to solve it. What people very often do is they just run logistic regression and then they would try to round the coefficients to integers because if you just run logistic regression, you get like 3.253 points and 49.14 points. And so they would try to round them. But the problem with rounding things is that you only want a few coefficients that are not zero. So you have to round a whole bunch of stuff to zero. And then everything else, you have to round up or down. And so you can actually get really bad performance by playing around with the rounding and then kind of manually eliminating variables. So you have to fiddle around with all kinds of stuff. And it was to the point where it was just like, yeah, now I know why doctors don't really like doing this. This is a math problem. And it's kind of sad that for so many years, we sort of dumped that math problem on the doctors, right? That's something that a data scientist should be working on, something that a mathematician should be working on, a computer scientist, right? So yeah, it's a good problem where our skills are best used to relieve the doctors of that problem. - And I'd love to actually dig in a little bit more on the human interaction component here because you mentioned earlier that the doctors don't have to trust the model. I'm assuming that maybe if I saw a risk score, I would have a hard time not having it impact my decision, even if I felt like I was an expert in the space. If I'm a doctor and I feel really strongly about a certain diagnosis, but then I see a risk score, maybe I start to doubt myself. And even though I know I don't have to trust the model, maybe it does have impacts on the types of diagnoses I make. And I'm curious if there were any issues or challenges that surprised you once you actually implemented these risk scores in the clinical setting. - So the doctors that I work with, I think, I can't speak for them, but I think they only chose this particular risk score because it was similar to what they were doing anyway. It just allowed them to sort of quantify and be more consistent. And obviously it'll lead to other doctors across different hospitals being able to be as consistent and high quality as our MGH doctors, right? Wouldn't you want a doctor from MGH no matter where you had, you know, what hospital you were at? So even in third world countries and so on, right? Now we have the expertise of the MGH doctors encoded into these models, right? But I think that there are other types of problems where the question that you've asked have been studied more heavily. So for instance, risk scores being used in criminal justice have been studied. Obviously my risk scores are not used in the justice system, but risk scores that are used in the justice system, they have been studied and they found that judges tend to be more consistent, tend to be less racially biased and tend to release people more often when they're using the risk scores and when they don't. Because frankly, most of the people coming through the justice system should be granted parole. And there are a lot of judges who grant, who don't grant parole that often. And then when they're seeing the result of these risk scores, they start to actually change what they're doing. They grant parole more often. You know, I didn't, I wasn't an author in any of these studies. Like I said, it's just what I've read. But yes, people do change their behavior based on these statistics. I think they recognize that no human can realistically compute risks in their head from large databases, right? Have you ever seen a judge try to take a whole database and process it in their heads and get, come up with a risk score? So I think the statistics do provide value. Obviously with criminal justice, there's a lot to argue about. But so far the science has said that, at least that's something that comes out of these risks or the criminal justice that judges do change their behavior. - I'm curious how you actually convince doctors to work with a new risk score in the first place. Was it hard to convince them to embrace something new like this or are they actually pretty willing to take a step forward? - Well, my collaborators happen to be absolutely excellent. I think it really just depends on what collaborators you have. It doesn't always have to be like that. Sometimes you can get people who are resistant. And I think another issue is data, right? Data is a major problem. You know, I know we're going a little tangential from the original topic, but I actually think this is central to all work on healthcare analytics. You know, I just happened to have a collaborator who is designing his own databases, is that kind of collaborator. But I had other collaborators where it was very, very difficult to get access to the data. And that made it very difficult to succeed in our projects. And I think that's a major, major problem with healthcare analytics that needs to be highlighted. I even wrote a little article in the Washington Post about that. It's just kind of sad that like, I'm glad we came up with a COVID-19 vaccine so quickly, but we came up with that vaccine before we got access to COVID data. And that's ridiculous. I mean, I thought when this whole thing started that it would be a matter of like two weeks before there would be a whole bunch of databases posted on the internet and everybody was begging all the machine learning people to use them, but no, that's not what happened. I mean, there were databases that were generated, but they were generated after several months and getting access to the databases was a big pain. Like you had to write an application for exactly what you wanted to do, which basically meant that you couldn't really do exploratory data analysis, 'cause you had to go in there and know what you were looking for. And I was just thinking like, this is not how I can truly help the human condition. And by the way, how long is this database gonna exist? If I write this paper five years down the road, is this database gonna exist so that someone can reproduce my work? And if it did exist, could that person get access to it? I mean, I understand there's all these privacy concerns, but there are also, there's an entire field of data privacy. And why don't we trust them to help us out? Why don't we get them to help us anonymize data, simulate data with similar distribution? - Totally, totally. And especially with some of these risk scores, given that they only use a few variables, it might be much easier to anonymize that and the outcome data than it might be for a whole health record to be anonymized. - Yeah, you often don't need so much detailed information. Most of these healthcare studies, if you wanna reproduce them, they're not using very detailed information. They don't need to know what zip code people are hanging out in, what hospital they were at. We don't need to know any of that. The fact is we're generating this data. Our insurance companies have it and they're not willing to share it. Our hospitals have it. Yeah, it's just such a pity. - I do think that's fascinating. And I'm curious if you think that at least in the U.S., where our healthcare system is as siloed and fragmented as it is, if you think there are some solutions that we can look towards or things that you hope to see maybe in the next five or 10 years that might make it easier to access data in a safe way? - Oh, I wish. I mean, I'm working on a project on trying to get data access to be better through the National Academy where I'm on a committee there. So I'm trying to start a project on this exact topic. I think there are solutions, but I don't know exactly what they are without. So right now, you don't choose what happens to your data. The people who are in control of the data have no incentive to release it, right? Insurance companies are very cagey about releasing data 'cause that's their secret sauce, right? That's how they calculate risks. And yet, if you want your data to be used for public benefit, that's not something that you can really volunteer for. I think the best option, to be honest, is to create these simulators that really make it, they're really guaranteed privacy, and then you can release the simulated data. I think that every study should be releasing simulated data. It won't solve the whole problem, but at least if you want to sort of verify what people are doing. - It's a step in the right direction. - It's at least a step in the right direction, yeah. 'Cause there's no privacy issues. We know there's no privacy issues, the data's simulated. - Do you see this being done better in other countries? Are there places in Europe where you think this is easier or has this not been solved yet? - I'm not an expert in this, I don't work in Europe. I think there are countries that give people access to the entire country's data. For instance, all those wonderful studies on vaccinations and they had all of Denmark, and then they were able to figure out that like, the MMR vaccine doesn't cause autism or there's no clear link between autism and vaccines, right? And the only reason that they were able to do that study is because they had access to all of Denmark's data. Can you imagine them doing that study for all of the US? Were all the data's in the insurance companies? - I mean, it just wouldn't happen. - No, it wouldn't, it wouldn't. You can get data on veterans if you're part of that hospital system. Yeah, the VA, I'm not part of that system. I've just had so much trouble getting data access and you know what? I've had data removed from underneath me in the middle of a project too. Like where you're working on a project, you're about ready to publish it or you've sent it in and the paper was rejected and you need to revise it and then all of a sudden your data access gets revoked because the data owner no longer decides that you are relevant to what they're doing. And it's like, well, thanks a lot. Now I can't even get access to my own data and I can't get my paper published. And now I know nobody's gonna be able to reproduce it. And it just goes to show you that data analysts are really considered second-class citizens when it comes to these kinds of studies. And I don't see why we're always second-class citizens. It's just so sad. - Yeah, for sure. And I hope that'll change. - Yeah, I can't see a way that it would change unless there's actual making something public or unless you're the one collecting the data. But how are you the one collecting the data? You're not the doctor collecting the data. You're not the insurance company collecting the information from the patient. You're sort of begging someone for data access, which, you know, or else you're applying for it and you have to be approved. So there's always this gate that you have to unlock to get, oh, there's the data. - I'd love to switch gears and talk about interpretability. So interpretability has been a core topic of your research. Now, interpretability is a hard idea to define across domains. So I'd love to ask you for the definition of interpretability that you find most useful, especially in the context of health care. - The way I define an interpretable machine learning model is that it's a machine learning model that obeys a domain-specific set of constraints so that a human can better understand it. And so, yeah, it's totally dependent on the domain. One topic or one concept that comes up a lot in the discussion of interpretability is black box models. Could you describe what a black box model is? - Sure. The way I define it, a black box model is a model that's either, it's a formula that's either too complicated for any human to comprehend or it's proprietary so that it's someone's secret sauce and you can't get access to it. Because if it's proprietary, you don't know how complicated it really is. So, yeah, that's what a black box is. - Got it, so it's either complicated or I have no idea how complicated it is because it's hidden from me. - Yeah, it's something you can't, you know, you can't put on a PowerPoint slide. - Got it, cool. Now, you've mentioned in your work that there are many examples in and outside of health care where black box models can hurt. Could you describe maybe one or two examples of this in the health care context? - Well, there was an interesting study a couple of years ago. I think they were at Stanford, actually, where you guys are, I can't remember exactly, but there's a study on x-rays and they found that neural networks for leveraging confounding information rather than paying attention to the actual medical content of the x-ray. Where the data were collected, there were two different types of x-ray machines. There was a portable machine and a regular machine and they were trying to predict health outcomes. But you can predict health outcomes really well from just knowing which machine the patients were getting their x-rays taken from because who's getting the portable x-rays, right? They're the patients who can't make it to the regular x-ray machine 'cause they're not doing so well, so their health outcomes are probably not gonna be very good. And so the neural network that they were using was paying attention to the type of x-ray machine that was written on the x-ray slide rather than the medical content of the x-ray. 'Cause it was just saying, oh, this person's on a portable machine. Let's predict that this person won't do so well. And it was very accurate. We've been working on mammograms too. And so what we found is that we can use a neural network and get some level of accuracy and then look at what information it's paying attention to and realize that it's paying attention to all the wrong information, which basically means that it won't generalize beyond that data set. We've been working on trying to make sure that the network is actually doing the right thing and using the right bits of information. And we also are checking to make sure it's using the bits of information in a way that makes sense to humans, right? That it's doing the right kind of comparison from this image to other images to see that, yeah, okay, it's reasoning about this image the correct way. Yeah, it's very, very easy for confounding to creep into complicated models. Not just neural networks, but any of these complicated models, they leverage whatever information they can. There's also been a lot of studies where they used huge databases and some of the data of images and some of the images had like green markings on them. And so the neural network picked up on, oh, there's a green marking, I'll use that information. But that wouldn't generalize beyond the database, right? - Those are great examples. And it seems like a very hard problem to solve in general because unless I had some understanding of how chess X-ray interpretation worked, I wouldn't be able to tell, let's say that a reliance and a confounder. I think there are other examples where there are skin lesions and there are markers that are being picked up to determine whether there are skin lesions. So confounders seem very difficult unless I know the domain or I know what's generating the data, the process. So how do you think about how we as a community might go about tackling this problem? - Well, I've never seen a clean database in my life, okay? So anytime you're using machine learning on a big database, beware 'cause, yeah. So since I don't trust databases, I don't really trust the models built from those databases unless I can actually understand what they're doing. And so that's why I think machine learning models for healthcare, especially for high stakes decisions need to be interpretable. Yeah, so again, interpretability is very dependent on the domain. So if it's not a risk score, like let's say you're reading images, right? Images you can't really do a risk score for images, right? So for images, my lab has tried to design a bunch of different machine learning methods that help give you insight into what the network is looking at and the way it's looking at it and the comparisons it's making when it does look at an image. So those are the kinds of things I think we should be building into our models. There are some cases where you don't need interpretability. One example is there is a model for segmenting the chambers of the heart. And in that case, the machine learning model's job is only to segment the chambers of the heart. And then the doctor can go back out and like fix it if it did it wrong. And because the human can check it and edit it, then it doesn't need to be interpretable. I don't need to know how exactly it did it 'cause the human can fix it. For these other cases where a human actually cannot go and fix it, they can't process all that data in their head. That's where we need the computer to explain itself. - I'm curious to hear from you when we ask people why they made a particular decision, their communication is in natural language. Should we have the same expectation from a machine? - This is again, a question that is domain specific. So what do you consider a risk score? Do you consider that natural language? That's what doctors use, right? That's what they use is risk scores. That's a language they feel comfortable communicating in for a lot of these different problems, right? I think it gets more interesting when you talk about images because the old CAD programs, the computer aided diagnosis, they were terrible. They were so bad that either they would just tell every patient to get a biopsy, in which case there were a lot of patients who had unnecessary biopsies. I mean, and I'm talking huge numbers of patients that got unnecessary biopsies, like the CAD program would just, because they didn't want to be responsible if they said, oh yeah, you don't, patient doesn't need a biopsy and then the doctor doesn't get the patient a biopsy and they have cancer, right? That'd be really bad. So all the CAD programs were designed to just tell everybody to get a biopsy. And so the doctors were like, well, I have to do what the CAD program says because otherwise I'd never be responsible for this. So, yeah, so vision is more difficult, right? So for vision, we have to be very, very careful about how we explain things to people. And I don't think we need it in natural language. I think we need it visually, right? I think we need to say, okay, here's the visual reason why this patient needs a biopsy. It's because this lesion has these characteristics. Bits of this lesion look like bits of this lesion and this lesion and this lesion. And these lesions are prototypical malignant lesions. And then the doctor can think for themselves whether or not these visual comparisons make sense. - So in the context of image explainability, you would say that maybe a direction that we should be working towards is a model being able to pull in other examples of similar-looking lesions or similar-looking tumors to try to explain why it's made the decision that it's made as opposed to just segmenting the tumor and saying, well, here it is. Is that a fair representation of what you're suggesting? - Well, that's one of the things we're doing. We're doing, it's called case-based reasoning. Case-based reasoning is very, very often used by doctors. They find similar cases that happened in the past. Dr. House is a great example. I don't know if you guys like to watch that TV show. I love that TV show. And Dr. House very, very often does case-based reasoning. He says, these aspects of this patient remind me of this other patient I saw 20 years ago. And that patient died and I feel terrible about it. And I really wanna get this patient right this time. Oh, but these other aspects of this patient look like this other thing. And so that other patient responded to blah, blah, blah, drugs, so maybe we should try that. - Got it. I think a lot of people like to think of sort of these deep neural networks as just totally replacing radiologists entirely and we're gonna move on with just machines diagnosing us. And I like that you sort of have places emphasis on how do doctors think and how can we understand how models are thinking or not thinking like doctors are thinking. - Yeah, there's different contexts in which you wanted to use different models. So I do think that there's some room for a completely automated work. For instance, if there is simply no lesion anywhere in the image, it's not clear that that image really needs to be read by a radiologist. So it would help the workflow in that way. But if there is a lesion or if there's even anything in the image that is a little bit tricky, at this point it probably does need to be read by radiologists. But you can use the machine learning to help organize the radiologists' work day so that the most tricky cases are first. And then they can get those done first and the more high pressure difficult decisions are made and then the easier decisions come for the rest of the day. And so if the radiologist doesn't quite finish their work for that day, it's probably okay. They can do it the next day. - I would love to end on a policy note, if you don't mind. So the European Union's GDPR, the General Data Protection Regulation and other AI regulation plans govern what they call the right to an explanation. And they say that an explanation is required but it's not an interpretable model. I'm kind of curious what you think about this, about this nuance and this policy. - Well, I think it allows companies to get away with almost anything they want to. As long as there's an explanation, whatever that means, even if it's incorrect, even if it's incorrect a large portion of the time. As we know, there are serious problems with explaining black box models because the explanations are very often wrong. People have called all kinds of things explanations. There are people who've tried to label individual neurons in a neural network as to what they actually do. So the quintessential example I like to give is a lamp node. Like if you have a model that's trying to detect why there a room is a bedroom. And the question is, how much is it depending on a lamp? Is it detecting a lamp? People have done this. They send a whole bunch of images through the neural network. They see which neurons are activated and then they ask a human, okay, what is this node doing? It's activating on this image, this image, this image and that image. And the human can go, oh, there's a lamp in all of those images. I think this is a lamp node. The problem with that is that this may not be what the network is actually doing. The lamp information might be spread all over the network. It just happens to be that this one neuron is activating on lamps more often. So there's a lot of like wacky stuff that people are doing right now to try to explain what these neural networks are doing. And it just almost seems like they're going to so much trouble to explain what the networks are doing that they're not even thinking that there might be a very simple possibility that you could just ask the network to just explain itself by just constraining it or just encouraging the network to be self-explainable. - And this idea of understanding what a neuron is doing or trying to explain it has similarities in neuroscience as well. The grandmother cell or the Jennifer Aniston neuron, the idea that there's this neuron in the brain that activates when shown a photo of Jennifer Aniston. So I feel like there have been quite a few similarities there. But I guess the power we have for neural networks or just machine learning models that we can design is we can make them interpretable rather than have them be black boxes that are tried to be explained later, maybe in not so accurate ways. - Yeah, for all of these different problems we've found with explaining black boxes, there are ways of constraining neural networks to be interpretable that don't lose accuracy. So in other words, let's say you wanted a neural network with a lamp node, right? So a neural network that all the information about lamp goes through one node. We can construct that now and we can do it without any loss of accuracy on the benchmark data sets for computer vision. So I think that these people who are trying to explain neural networks are valuable in the sense that they're telling you what they would like a neural network to actually do. But of course their claims that they actually understand the neural network are not correct. - Cynthia, thank you so much for being on the show with us. It's been so fantastic having you. - My pleasure. - And that's all folks. A big thank you to Dr. Cynthia Rudin for talking to us today. And thank you for listening. We're your hosts, Pranav and Adriel. And until next time, stay safe and stay healthy. - The AI Health podcast is produced and edited by Oishi Banerjee, music by Ethan Agee. Many thanks to Stanford Amy for their appreciation and support and to consulting producer, Margaret Catcher. If you like what you just heard, let a friend know. Subscribe to the show and give us a five star review on Apple podcasts. Follow us on Spotify or connect with us on Twitter at AI Health podcast.