Go back

3. Computer says no: is AI making healthcare worse for women?

from Visible Women with Caroline Criado Perez

45m 9s

3. Computer says no: is AI making healthcare worse for women?

Sally B’s story reveals a systemic failure in diagnosing heart attacks in women, where a 36-year-old woman was initially dismissed as having a panic attack despite suffering a life-threatening heart attack caused by Spontaneous Coronary Artery Dissection (SCAT). SCAT is the leading cause of heart attacks in young women and is now recognized as historically underdiagnosed due to a male bias in medical research and clinical practice. Women are disproportionately misdiagnosed and die at higher rates because medical data has long relied on male bodies, leading to a gender gap in healthcare. This bias is amplified by AI systems trained on non-diverse datasets, which produce inaccurate and unfair outcomes—such as underdiagnosing women in heart attacks or domestic violence—despite appearing technically accurate. The problem is systemic, rooted in lack of gender-disaggregated data, poor training, and a culture where speed and efficiency override equity. While AI holds promise, it risks worsening existing disparities unless rigorously audited and redesigned with inclusive data. Solutions include mandatory gender breakdowns in research, interdisciplinary collaboration between clinicians, researchers, and ethicists, and regulatory oversight to ensure fairness. A promising example is a study using electronic health records to predict domestic violence years in advance, showing how AI can be used to empower early detection in underserved areas. Though progress is slow, the episode underscores that true innovation in healthcare must center on equity—ensuring that women and marginalized groups are not just included, but fundamentally represented and protected.

Transcription

7354 Words, 41228 Characters

English
I was actually at a kid's birthday party. I was holding my daughter, the boys were running around and I just felt really poorly, really suddenly. So I handed my little girl to a friend and I went to the toilet and I don't know whether I thought I was going to be sick or whatever. I just knew something was really wrong. Sally B is telling me about a summer's day in 2004 when she was 36 years old. When I came back out and collapsed on the floor and all my friends tried to help me but they didn't know what was going on. I had recently watched an episode of ER where someone was having a panic attack and they bought them a bag to breathe into. Someone actually brought me a packet of cheese and onion crisps which didn't really have the desired effect. Sally is an author, a coach, a motivational speaker and I do TV, presenting and cooking and you know a bit of everything really, jack of all trades. But first and foremost, she calls herself a survivor. I just remember the pain absolutely tearing through my chest and the ambulance came and by then because my breathing was so forced and I was panicking, I suppose. My hands went all crunched up and I couldn't open my hands. And so the ambulance guys presumed I was having a panic attack and I was trying to explain well I am panicking now but it didn't start that way. Sally was taken to Warwick Hospital and sent home that same evening with medication to treat indigestion then a couple of days later ended up back in hospital with the same chest pain was given an ECG and a student nurse saw the ECG and she was like oh my goodness this is saying you're having a heart attack so she called the cardiologist down. So I had three cardiologists standing around my bed. All of them saying now look this ECG is telling us you're having a heart attack but we actually don't believe it because you're 36 you don't drink you don't smoke you have you're not overweight and you don't have any family history of heart attacks. Sally tells me one of the cardiologists they were all men by the way said he would bet her a million pounds that she wasn't having a heart attack but she was and if Sally had been diagnosed sooner her heart might have ended up being less damaged. I'm Caroline Criado-Peras and this is Visible Women my new weekly podcast from Tortoise investigating how we finally fix a world designed for men. In this episode I'm actually not just looking at heart attacks I'm looking at bias and healthcare and whether the emerging use of AI means it's about to get a lot worse. I start off this episode with all sorts of ideas about how to fix bias in AI but by the end as you'll hear I realized that maybe I was approaching this all wrong. This was an episode that really forced me to re-evaluate my thinking oh and if you're the cardiologist that owes Sally a million pounds get in touch I'll get you her bank details. It turns out that Sally has a condition called Spontaneous Coronary Artory Dissection or SCAT. SCAT has until fairly recently been thought of as a rare condition but some cardiologists now think it's actually been historically underdiagnosed. It's the most common cause of heart attacks in women under the age of 40 as well as in women who are either pregnant or have recently given birth and more than 90% of SCAT patients are female. The thing that's so shocking about Sally's story is that it's about more than just the fact that her symptoms weren't taken seriously although that's bad enough. It's that 12 years later it happened all over again. So I was older I knew all about my condition by then. I had my ECG laminated up on the fridge, I had a cardiologist letter up on the fridge, I knew what I had to do in an emergency, I understood how dangerous it was and then I woke up in the morning and that pain just hit me again and I knew exactly what it was. This was in 2016 by the way. We phoned the ambulance, they came and they wouldn't take me seriously at all. So the paramedic that came said it could just be angina, it could be menopause, lots of women in your age get chest pain and he wanted to take me to the little local hospital where I live and I said no I have to go to the big hospital with cardiology. He wouldn't listen and he wouldn't take me. I was crying, I was sobbing, not from pain although I was in pain but from frustration. The paramedic did an ECG to check Sally's heartbeat but she knew the equipment he was using wouldn't pick up her kind of heart attack. Sally explained this to him but it didn't help. Her husband ended up phoning her cardiologist. And even my cardiologist was almost begging this paramedic no please you must take her to the bigger hospital and he still was arguing so it took 20 minutes for this paramedic to say all right then we'll take you to the further hospital. And that could have been the last 20 minutes of my life. Sally's story is shocking but I didn't find it surprising. Women who have a heart attack in the UK are 50% more likely to be misdiagnosed than men. They are also more likely to die and it's basically because the vast majority of medical data we have collected historically and continue to collect today including in cardiovascular research has been in the male body, male humans, male animals, even male cells. The gender data gap in healthcare was actually what prompted me to write my book Invisible Women. It was all about the systematic underrepresentation of women in data. I was used to men being the default humans in politics, in the media, in books and films and even banknotes, long time fans might remember that as one of my greatest hits. A new Jane Austen 10 pound note announced today. But to find that this default male bias existed in science, in medicine, that the male body was being treated as an essentially gender neutral body in a field that is and I cannot stress this enough, literally focused on studying actual human bodies and that the result of this default male bias was that women were dying, well that was a shock. To put it in context, under the age of 65, double the number of women die of cardiovascular disease than from breast cancer, it's the leading cause of death for women in the world. This is Dr. Nishat Sediki. She's a consultant cardiologist at a busy teaching hospital in South Wales and 50% of her workload is women. Nishat tells me that the most common symptom of a heart attack for both men and women is chest pain. For some women the chest pain may feel more like a tightness across the chest. Research also shows that some women will experience different symptoms like breathlessness, nausea, fatigue. But Nishat says the symptoms aren't really the problem. The problem is that even when women do present with the classic symptoms of a heart attack, like Sally did, no one believes that they are in fact having a heart attack. Women themselves present late on the whole and when they do present their symptoms are often dismissed as stress and anxiety related. It's a bit of a vicious circle. The lack of research on women means doctors know less about what a heart attack looks like in women. This leads to under diagnosis, which in turn fuels the perception that heart attacks don't happen in women and so the cycle continues. Nishat tells me that doctors are often not thinking about female specific risk factors when they're assessing patients. So for example, if a woman has been pregnant and she has had a preterm baby or if she's had preclampsia, hypertension or gestational diabetes, that automatically is like a warning wreck flag that she's at a higher risk of developing cardiovascular disease. And yet when I see patients who've come to me from my other colleagues quite often, nobody has bothered checking their past history. I think about Sally. Her cardiologists were so busy ignoring her ECG results because a 36 year old female couldn't possibly be having a heart attack that they never thought to ask about her pregnancies. In fact, recent data shows that an increasing proportion of people hospitalized with a heart attack in the US are under 55 and the largest increase is in young women. But because we have for years relied on medical data collected in men to treat women, doctors simply have not been sufficiently educated on female dominated risk factors, which aren't exclusively related to pregnancy by the way. Other risk factors include starting your periods before the age of 11, having irregular cycles, and suffering from auto-improving. immune disorders which themselves disproportionately affect women. The result of this male bias is that when women turn up at a hospital like Sally did, they often don't get properly examined. Across the board, men were more likely to receive all of the standard treatment compared to women. Nisha tells me about one study that compares men and women's outcomes after having had a heart attack. The only thing when women had a higher incidence of getting something more than men in that study was that women were more likely to die from a heart attack than men. In fact, according to research funded by the British Heart Foundation, more than 8,000 women died between 2002 and 2013 in England and Wales because they did not receive the same standard of care as men. I knew about the poor outcomes for women who have heart attacks from researching my book, "Invisible Women." I also knew that these poor outcomes are intrinsically related to the over-representation of men in cardiovascular research. There is a huge data gap here and it's one that I've been worrying about for years now. But there's a related issue I mentioned in "Invisible Women" that's been making me even more worried. Can artificial intelligence make health care better for all of us and save the NHS? People should stop training radiologists now. It's just completely obvious that within five years deep learning is going to do better than radiologists because it's going to be able to get a lot more experience. Just to start with the basics, AI refers to artificial intelligence, computers which have been taught to think for themselves. They use algorithms, basically a set of automated rules to analyse data and find answers to whatever questions are being posed by their human overlords or over-ladies. We're all used to thinking of AI as neutral and objective, and sure, AI doesn't literally hate women and want us to die, as far as I'm aware, AI has no strong feelings on the matter. But AI is only as good as the data we feed it. And when it comes to medical data on women, the data is pretty non-existent. Still though, you might ask, why does this make AI any worse than human doctors? They are also only as good as the data they are fed. And yes, that's true, but humans also aren't on the whole any worse than the data they're fed. AI, on the other hand, suffers from something called bias amplification. Readers of invisible women may remember a study I cited where an algorithm was trained on a very commonly used open source image data set. The data set had a bias, whereby pictures of cooking were 33% more likely to include women than men. But after the algorithm was trained on this data set, it was connecting pictures of cooking with women 68% of the time. That is, it was labeling pictures of men as female just because they were standing in a kitchen. This example might not sound very life and death, but imagine an algorithm this biased, let loose in a hospital. One of the things that we're starting to really uncover is that, yeah, the algorithm can magnify bias, but ultimately the healthcare system in itself before AI ever entered the picture is in fact inequitable. This is Irene Chen. She works on machine learning for equitable healthcare, and is currently a PhD student at MIT. I wanted to ask Irene about my concerns that adding AI to the mix risks making everything worse. The systematic undiagnosis of women already leads to a perception among human doctors that women don't have heart attacks, that vicious cycle we mentioned. But what about an algorithm trained on a data set riddled with these mist heart attacks? Now, when an algorithm learns it, if they want to sort of scale doctors, they want to help doctors out, they'll just magnify what happened in the existing setting. And so I think that kind of acknowledgement of the existing system being inequitable and that scaling that blindly would be bad is incredibly important. I think listening to Irene made me think of an article that I came across at the end of 2019, about eight months after invisible women was published. It was the headline that first sparked my interest. AI, to predict heart attacks, at least five years before they occur. The article was illustrated with a picture of a woman, which pleased me. But the study the article was based on pleased me a little less. The studies the algorithm had been trained on were heavily male dominated, and the paper provided barely any sex-desaggregated data. The lead author of the paper was quoted in the article saying, "We genuinely believe this technology could be saving lives within the next year." Men's lives, maybe. Women's lives? Well, that might have to wait. I was so frustrated reading this study. I knew what the research said about bias in healthcare and bias in AI, and it made me so angry that algorithms like this were still being produced, their gender-neutral claims credulously reported by the media. Why wasn't anyone paying attention, and more importantly, what needed to be done to change this? Healthcare is one area where AI potentially have some of the greatest impacts if there's a human welfare. James, though, is a professor at Stanford University, where he leads a group developing AI, and his speciality is healthcare. I first came across James's work when I was researching invisible women. I cited a study of his which looked at a language-learning AI. They found that the top occupation the algorithm linked to women was homemaker, while the top occupation linked to men was Maestro, which is, of course, fine. James is pretty excited about what AI could do for healthcare, especially its capacity to diagnose quickly and cheaply. But he also says there are a lot of issues to be ironed out before it can be safely deployed. I've been very interested in thinking about how do we systematically evaluate and how do we audit these AI algorithms, or similarly to how you might audit someone's tax return, or how might you evaluate how well your car's performing every couple of years. We also want to have systematic ways to evaluate these AI models. But that's not really happening right now. And yet, over in the US, James tells me that the FDA has already approved 130 AI medical devices to be used on patients. I wanted to know how many AI's were currently in use in the NHS. So I called on my fellow data enthusiast Patricia Clark, a data journalist at Tortoise. You know, do we have any in-use in hospitals right now? Yeah, yeah, I'll see if I can reach out to the NHS and see if they've got a sort of list of companies that they're working with and the kind of technologies that they're using. Yeah, and how does they decide what's the quality control? After several weeks of trying, Patricia hadn't really got anywhere. Which is really unusual because she's very good at her job. It's an incredibly kind of murky area, I think. Also, the NHS have the AI lab that they use and they've got case studies on their website where they talk about ways in which they've trialed certain kinds of AI. And I spoke to someone with very good knowledge of that area and they said, look, it's impossible to know how much AI has been rolled out in the NHS for a number of reasons. One being that the NHS, despite it being thought of as one singular body, is actually a decentralized kind of series of independent trusts and so on and we could go. Can I just add, this person works very high up in AI for the NHS, they should know. Even someone who is very, very close and has worked in the NHS in that kind of area doesn't know how often it's being rolled out. We rely on the case studies that they put on their website and that they share with us and when it gets into the partnerships with private companies and so on, then for a whole bunch of reasons, we don't know exactly how much it's being rolled out. Just to interact with one second, I mean, that's really poor practice that they put these sort of agreements with private companies who are in it to make profit and want to defend their proprietary software above transparency with the public, which is I think really shocking, that part of the reason we can't find out to what extent these algorithms and AI are being used in the NHS seems to be because of non-disclosure agreements with private companies, which just seems like a little bit problematic. The lack of transparency and refusal to engage felt like PPE all over again. That was in episode one, by the way, in case you missed it. Most of the FDA-approved AIs are computer vision type AI systems. They look at things like chest X-rays and make diagnoses. This worries me because I don't see any evidence that we're ready for these kind of algorithms to be deployed anywhere near patients. Here's Irene again, telling me about an algorithm that had been trained on three prominent chest X-ray data sets. She wanted to know how well the algorithm would diagnose different subgroups such as race, age and sex. If we were to find, for example, which we did, that there is sort of pervasive underdiagnosis for different groups, then that would be incredibly concerning, especially as these large-scale chest X-ray classifiers get rolled out. The subgroup that the algorithm did worst on, by the way, it was women. Irene tells me that looking for bias in AI, is kind of like a detective mystery. At the very end, you see the dead body. You see, oh, the graph says, this group, you know, black patients are getting an accuracy of 70 and white patients are getting accuracy of 90. Uh-oh, what's going on? And so then you start to take a look at all the clues that you have. The clue Irene had in her chest X-ray study was that algorithms that are literally called state of the art are systematically underdiagnosing female patients. When I read Irene's paper, there was one bit that really hit me. She pointed out that previous work on these algorithms has basically just raved about their radiologist level performance. But until her paper, nobody was even thinking about whether or not these algorithms were biased. On paper, the performance is great. Like, the reason it's so concerning is because on paper, these algorithms can go faster, work consistent, and in some cases, even better than specialists who've trained in that specific field. But bodies don't exist on paper. And in bodies, things look considerably less rosy. James, our Stanford professor, tells me about an algorithm that was analyzing skin lesions to determine whether or not the patient had cancer. What we discovered is that those algorithms, which could be very powerful, they are actually systematically much worse, once applied on images from specific populations. So as for example, if you have images from darker skin individuals, those algorithms are being much less reliable than they've made a lot more mistakes. This is basically because the data fed into those algorithms, as in the pictures it was trained on, was mostly of white patients. To be fair to the AI community, not everyone is like these researchers, or the authors of that 2019 heart attack prediction paper. Plenty of people are actively trying to fix these problems. Yes, I think the view in the community has definitely changed quite a lot over the last five years. I think now, in 2022, there is much broader acceptance and recognition that it's really important to build algorithms, AI algorithms that are trustworthy. And by being trustworthy, certainly reducing and evaluating and mitigating these biases is one big component of that. The trouble is, it's just not that easy. Now, there's still a very much an open question about what are the best approaches for doing that in practice? How do we actually evaluate an algorithm for potential biases? And how do we mitigate those biases after we identify them? First start, we've got a big data problem. Sometimes, and this is like the most insidious part, is that we don't measure race or gender in the data set at all. So we'll have all these patients and we don't know some crucial information that would allow us to present all of these broken down metrics. And that's sort of the most crazy thing to me. That's like, in that drives me up the wall, is that then you can't even do anything on these algorithms because then you don't know what the actual groups that these people belong to. And it's not just that the data isn't great, or it's missing, or it's not sufficiently diverse, or it's not disaggregated. It's also that it's completely unstandardized. Every hospital, every university has different ways of labeling, formatting and organising the data. And until we have a single standard. Standards. Our standards, my favourite thing. Anyway, without a standard, all these diverse data sets will remain in their own silos. It's a huge waste of knowledge. The obvious solution here is to collect more and better data. But as Irene explains, this sounds easier than it actually is. First of all, we would have to identify what data we need. And this would require a huge data analysis project to identify all the gaps and biases that exist in the data we have. Only then could we go out and collect our new data. This could and arguably should be done, but it's really time consuming and expensive, so it's not a quick or easy fix. I discussed loads of other potential solutions to the bias problem in AI with both Irene and James. I got really excited by an idea I came across in a couple of papers that suggested closing the data gap by synthesising data, basically making it up. But Irene isn't convinced. She tells me about an algorithm that was meant to be generating new data. The developers ran a little test to see if it was working as intended. So they'll have like a fill in the blank question. It's like this patient should go to blank. They have these symptoms. This patient should go to blank. And if you ask the algorithm to fill in the blank here for a white patient, it might say the hospital. They should go to the hospital. And then for a black patient, it would say this patient should go to jail. So if more data isn't going to fix everything, what else are AI researchers trying? There's been a lot of discussion about what are called black box algorithms. This is basically an algorithm where we really don't know what's going on inside it. And this makes it pretty much impossible to explain why the algorithm has come to any decision. Some people think the solution is to make algorithms simpler, that people should have a right to explanation. But James explains that the problem with this idea is that in order to be explainable, algorithms have to be simpler. And simpler algorithms really aren't sophisticated enough for a field as complex as healthcare. He compares our expectations of understanding algorithms to our expectations of understanding how the pill you took for your headache works. It's a good point. I haven't got to clue what the paracetamol is doing in my body once I've swallowed it. If I'm trying to take like a drug, right, and explain that to a user who's not a chemist or who's not a biologist, then I have to make certain of some simplifications and say, okay, so maybe this is a drug, right. I don't have to explain the exact molecular details of the drug, but I can say, okay, this is meant to help to reduce your headache. So collecting better data isn't a simple fix and neither is creating explainable algorithms. What about if we tried asking the algorithm different questions? So instead of asking the algorithm to prioritize overall accuracy as in being right 90% of the time, we instead asked it to prioritize fairness as in not being more accurate for men than women. But without better data, the algorithm tends to do this by just being less accurate for men, rather than by improving accuracy for women. So it's fairer, but it's not actually as good. That's often not a conversation that doctors want to have. If my grandmother is being treated, I wanted to have the best possible algorithm. But at a sort of a high level, it's a non-starter to go to a clinician and say, oh, I want you to use this lovely AI algorithm. There is actually a better one out there, but use this slightly worse one at a top level overall performance. But even if we manage to fix these coding issues, I really explain that it's not likely to happen anytime soon. A lot of it is about incentives right now. The truth is that there is not really a great incentive structure to report, you know, you would never be able to get a paper published without saying what the top line performances. There should be an expectation that just as you should have the top line metric, maybe you should also have to break it down by different well known groups. Finally, a tangible solution, journals and machine learning conferences could simply refuse to publish papers that don't break down their results. Although Irene says a lot of it is about the availability of data. Like it is no fun to go door to door asking for data is no fun sitting around saying, well, if only we had this data, then we could really ask the questions. It is much more fun as a machine learning person to say, oh, we have this giant data set like let's see what happens. So who is going to actually change these incentives and make tech developers want to focus on fixing bias instead of just building shiny new algorithms. This was all starting to remind me of Dr. Katrina Hutchison. She's the bioethicist we met back in the PPE episode who told me about moral aggregation problems. The moral aggregation is the idea that you have small on their own sort of harmless or morally not not worthy kinds of things. But when you get a lot of them or you get different types of them, they have a kind of cumulative or aggregate effect. This really feels like another one of those problems. There's no one obvious person or institution who can fix it. Fixing bias in AI is going to take changes in all sorts of different places, journals, universities, individual researchers, funders, regulators. When I spoke to Katrina, she also told me about another concept that I thought was really helpful for thinking about how we might fix AI. It's called the tragedy of the commons and it's kind of about cows. It's the kind of problem that arises when you have a common pull resource or a kind of shared resource. And it's in the interest of each individual to take more than their share and if just one individual does, that won't cause any problem for the group as a whole. But if everybody does, it will. one of their kind of most familiar. familiar framings of the problem by Garrett Harden is you have kind of a common land with grazeeers who are grazing on stock on that land. So imagine you've got a herd of cows. It would be in your interest to graze as many cows as you could. You might get more milk that way or more calves. But cumulatively the commons may become over grazed and might suffer a kind of catastrophic environmental collapse. It's perhaps not a direct one-to-one since AI developers aren't exactly working from a common resource. But the principle of the incentive for the individual researcher or company versus what works for the group, i.e. humanity as a whole, feels very relevant. Katrina has done some research applying the idea of the tragedy of the commons to the development of medical devices. And she's found similar problems to what I found with AI and PPE for that matter. And then there's this expediency issue. There's just so many forces that are saying do things more quickly, do things more cheaply, you know, make medical devices that suit everybody, not different devices for younger people, older people, women, men, people who have. For an individual AI developer, it might make sense to create an algorithm that doesn't consider sex, because it will be quicker and cheaper to develop. But when that algorithm turns out to discriminate against women, well, that's a tragedy of the commons, because the end result is women dying from preventable causes. So how can we make the individual AI developer care about the commons? Katrina says there are two approaches. The first would be for members of the AI community to come together as a group and agree a shared set of rules. I stand it, if you will. There has to be a kind of agreement that everybody will do the right thing in the same way, otherwise some people will kind of exploit the goodwill of other people. The other approach is regulation. If you're going to be punished or you're not going to be able to have your device on the market, unless you can form with certain kinds of policies, you'll do it even if otherwise, if the policy wasn't there or the rule wasn't there, it wouldn't have been in your interest to do so. So even if it's a burdensome or costly, if that's the rule and that's the only way you're going to be able to kind of play the game or be able to have your device on the market, you will follow that rule. There are other tangible solutions too, like increasing diversity in the AI workforce, which is incredibly male dominated. I think it's one of the factors that we definitely want to be very thoughtful about, right? So as you know, I think that AI workforce on AI researchers and AI developers is not very diverse. It's certainly not balanced across gender. You can say that again, James. In fact, a 2019 study reported that only 18% of authors at leading AI conferences are women and more than four out of five AI professors are men. The same study revealed that women made up only 15% of AI research staff at Facebook and only 10% at Google. Research shows that scientists from underrepresented groups often study topics that have traditionally received less research effort. They also tend to produce more novel research, but they get less funding and their innovations are taken up at lower rates. A lack of diversity in innovation can also produce some embarrassing mistakes, as Apple found out when they launched their comprehensive health tracker app back in 2014. The app could track your intake of copper and millivitanum, yeah, I don't know what that is either, but it couldn't track your period. James also thinks progress could be made further up the pipeline, for example, at universities. And I would love for these topics to be more deeply integrated into the curriculum in computer science and data science and statistics. So oftentimes, you know, you maybe have some core classes in machine learning and AI, and then there would be some separate classes that are more specific topics on ethics or policy, but I think it would be more effective if those classes are ends up integrated. And remember James' whole thing about auditing AI? Well, it turns out some people are working on creating AI that can audit AI. And I think that's going to become increasingly common, whereas like that basically auditors, it's going to be some combination of human plus AI to systematically check and tell you all these models. And who's going to audit the auditors? Come on, that was a great line. So anyway, there are encouraging noises, but I still felt unsatisfied. It didn't feel like we were getting to the bottom of a slam dunk fix. It all felt really messy. But then something Irene said made me wonder if I was looking at this all wrong. AI right now is about predicting the future if nothing changes, and if we could switch that thinking into AI should be about empowering people who haven't had power in the system before, or understanding what would happen if we make changes in the system, and how would we go about using AI to get there. So this is my sort of glass half full pitch about how health inequities could be addressed. If we start from the very beginning, and we say what are known health inequities that we want to address, and how would we go about doing that? What Irene's saying kind of reminds me of that thing that Henry Ford probably didn't actually say. If I'd asked people what they wanted, they'd have said faster horses. Instead, of course, he built a car. By the way, I know Henry Ford didn't come from Mississippi, but my husband does, and it's the best we could do at short notice. So instead of using AI to try and replace doctors, maybe we should be using AI to fill in the gaps, doctors, and researchers have left open for decades. Maybe we're being too unambitious. There are diseases that affect women that are understudied and underdiagnosed endometriosis. Is one of those, for example, on average, I believe it's seven years for diagnosis. There are researchers at Columbia University, for example, who have made it their life's work to say, can we use machine learning to diagnose endometriosis earlier, and can we better understand this disease that we don't know that much about because it's underfunded and it's diagnosed so late. OK, now we're getting somewhere. Now I'm actually starting to feel excited. And then Irene tells me about this amazing project she's working on. So this work is with Dr. Barti Karana, who is at Brigham & Women's Hospital, and so together we've been looking at this question of how we can use all of these observational electronic health records to better understand and perhaps do early detection for intimate partner of violence. By the way, intimate partner violence is a form of domestic violence. Intimate partner of violence itself is tricky because there's an underdiagnosis problem. Vaccinations are reluctant to come forward because of the stigma. They don't know what the consequences will be. They might be reliant on their abusers. They're sort of mistrustful of health care professionals. And also clinicians and health care professionals, practitioners are often not on the lookout for these things. They're not trained in the right way. They might be busy. They might be resource constrained. They might only have a few minutes with each patient at a time. And so it's very difficult to get the diagnosis to talk about later. Naturally, this lack of data didn't deter Irene. Together with her research partner, Dr. Barti Karana, she started to aggregate what data was available, things like health insurance billing codes, patient cell reports, radiology reports, and built a data set from scratch. Using that our preliminary data that we looked at for almost a thousand patients overall is that we were able to predict about three years in advance of them entering a violence prevention program. We could actually see these patterns of what's going on. OK, can we just stop here to appreciate how amazing this is? This could be a total game changer for domestic violence prevention. This could save women's lives. I was so excited about this. I had to tell Patricia. Three years is incredible. Yeah, right. Three years. I mean, scary as well. The number of lives that could be saved by early detection like this. So that must mean that there's just potential for lots more research in this area. I mean, this is just the first study of its kind, right? Yeah, for sure, for sure. You know, and that's what's so exciting about it, right? That's why I felt suddenly, OK, right? So there is something here, you know, yeah, there is hope. And let's more, we can find what are sort of risk factors, what are smoking guns that we might say. Oh, this is like a huge, if we see this on a radiology report, this would be a huge thing that we'd be concerned with, one of which is an ulna fracture. So the ulna is the bone in your forearm. And essentially, people might come in and say, oh, I fell. And that's why I fractured my ulna. But if you fall, you fracture your wrist. And the ulna is much more likely to be a defensive wound from above. And so the ulna fracture can then become a almost like very high, high predictor that something is going on. And so if we were able to find even a few more of these predictors, we could then empower clinicians, you know, we don't need to roll out a full on AI model just yet. We could even just find these patterns and better understand what's going on. I'm just blown away by this already. Irene's work has produced tangible results that can be used to help patients right now, even without rolling out an algorithm. And as Irene says, what makes this research so important is that it's filling in a data gap, no clinical trials. would study this. This is definitely not something that is prioritized, funding-wise. So it's a very exciting project. It's definitely ongoing, and I'm excited to see what else we can find. It's incredibly exciting. Oh, and yes, Irene did sex disaggregate her data. But her approach is not very popular yet. She tells me that aren't many people working on health inequities in machine learning. Still, the work is being done. When I was researching this topic, I read about an algorithm that looked through studies on 11 different diseases going back 52 years to determine the extent of the data gap. This is hugely important work. It's also hugely costly and time-consuming to do it by hand. Being able to automate this is a total game changer. We can finally stop having the argument over whether the data gap in health research even exists and just focus on fixing it. Okay, final question. Are you optimistic that we can fix these bias issues before AI becomes widespread in healthcare? I am. Even though we talk about a lot of these challenges, I'm still very excited about the potential of AI in healthcare. I think it could really become transformative and improve the health and well-being of really broad populations. Irene thinks we should get Batman on the case. I definitely am always for more collaboration. Even between machine learning researchers or machine learning researchers and clinical practitioners and then even bringing in ethicists and anthropologists and people who, economists, public health people, there's a whole branch of people who could all work on similar problems from slightly different angles. You think of it as the justice league coming in. You have the superpower, but I have the superpower. I guess nowadays it's marvel, but I have no idea. You don't need to worry about that. If we could bring all of this together, then that would be great and we could leverage each other's strengths. For now, though, we're just not ready to replace human doctors with hell-9,000. You remember how the AI that tried to kill off his entire spaceship crew in 2001, a space odyssey? At the moment, humans have a major advantage over AI, and it's simply that they're human, they're flexible, and able to listen, and understand when they're wrong. Even if, like Sally's paramedic, it sometimes takes a full 20 minutes and an intervention from a senior cardiologist. You know, why do you think he was so reluctant to believe you? All I know was the situation at the time. He'd had a directive to say that you can't have a heart attack without something called ST elevation. An ST elevation is a kind of elevation on your ECG. I knew that their ECG machine wasn't picking it up, but I knew I was having a heart attack, but he was following his protocol, which told him no ST elevation, no heart attack, and that's why I think it's got to be a human response as well as a protocol generic response. We can make people better, and in fact, we are, even if it's slow work. I have to say that when I got to the hospital, the care was absolutely amazing, and they knew about my condition because I'd been raising awareness, so had everyone else. So actually, when I went in, we know exactly what to do with you. We've had the SCAD research team in telling us about this, and they knew exactly what to do. The treatment was amazing. But that wouldn't have mattered if I hadn't made it that far. Sally was lucky that those 20 minutes didn't cost her her life, but women shouldn't have to rely on luck. Women should be able to rely on data. Better data means better doctors, both of the human and the AI persuasion. An AI could have an important role to play in collecting and auditing that data. Meanwhile, I'm excited to see what Irene does next, and to hear about the next algorithm that we didn't even know we needed. Thanks for listening to this episode of Visible Women from Tortoise. If you're a Tortoise Plus listener on Apple podcasts or a Tortoise member, listen out for a bonus episode coming on Friday. As for the rest of you, if you're hungry for more, go to tortoisesmedia.com/caralline and use my code Caroline50. This episode was written and produced by me, Caroline Criado Perez, alongside Hannah Varrell and Patricia Clark. The executive producer is Basha Cummings. It features original music by Tom Kinsella and sound design from Studio Clong.

Podcast Summary

Key Points:

  1. Sally B experienced a heart attack at 36, initially misdiagnosed as a panic attack due to gender bias in medical assumptions.
  2. Her condition, Spontaneous Coronary Artery Dissection (SCAT), is the most common cause of heart attacks in women under 40 and is frequently underdiagnosed.
  3. Women are 50% more likely than men to be misdiagnosed with heart attacks, largely because medical research has historically used male bodies as the default.
  4. AI in healthcare is only as reliable as the data it is trained on, and existing datasets are overwhelmingly male-dominated, leading to biased outcomes for women.
  5. Algorithms trained on incomplete or ungender-disaggregated data systematically underdiagnose women, especially in heart attack and domestic violence prediction.
  6. Systemic bias in healthcare persists due to lack of education on female-specific risk factors, such as pregnancy history and autoimmune disorders.
  7. Despite growing awareness, AI development lacks transparency, standardization, and accountability, with few mechanisms to audit bias or enforce fairness.
  8. Positive change is possible through interdisciplinary collaboration, better data collection, and regulation that mandates gender-disaggregated analysis.

Summary:

Sally B’s story reveals a systemic failure in diagnosing heart attacks in women, where a 36-year-old woman was initially dismissed as having a panic attack despite suffering a life-threatening heart attack caused by Spontaneous Coronary Artery Dissection (SCAT). SCAT is the leading cause of heart attacks in young women and is now recognized as historically underdiagnosed due to a male bias in medical research and clinical practice. Women are disproportionately misdiagnosed and die at higher rates because medical data has long relied on male bodies, leading to a gender gap in healthcare.

This bias is amplified by AI systems trained on non-diverse datasets, which produce inaccurate and unfair outcomes—such as underdiagnosing women in heart attacks or domestic violence—despite appearing technically accurate. The problem is systemic, rooted in lack of gender-disaggregated data, poor training, and a culture where speed and efficiency override equity. While AI holds promise, it risks worsening existing disparities unless rigorously audited and redesigned with inclusive data.

Solutions include mandatory gender breakdowns in research, interdisciplinary collaboration between clinicians, researchers, and ethicists, and regulatory oversight to ensure fairness. A promising example is a study using electronic health records to predict domestic violence years in advance, showing how AI can be used to empower early detection in underserved areas. Though progress is slow, the episode underscores that true innovation in healthcare must center on equity—ensuring that women and marginalized groups are not just included, but fundamentally represented and protected.

FAQs

SCAT is a rare condition where a tear occurs in the coronary artery, causing a heart attack. It is now recognized as the most common cause of heart attacks in women under 40, especially those who have recently given birth or have a history of pregnancy complications.

Women are more likely to be misdiagnosed because historical medical data has primarily been based on male bodies. This leads to a lack of awareness about how heart attacks present in women, resulting in symptoms like chest tightness, nausea, or fatigue being dismissed as stress or anxiety.

Gender bias in research means that medical data and protocols are often designed based on men, leading to underdiagnosis and delayed treatment for women. This results in women being less likely to receive standard care and facing higher mortality rates after a heart attack.

AI can worsen gender bias because it learns from historical data that is predominantly male-dominated. This leads to algorithms that underdiagnose conditions like heart attacks in women, even when trained to be accurate overall.

Researchers are advocating for better, sex-disaggregated data, audits of AI models for bias, and mandatory transparency in results. Some are also developing AI systems that prioritize fairness over overall accuracy, and pushing for inclusive data collection across genders and demographics.

The paramedic dismissed her symptoms because he followed a protocol that required ST elevation on an ECG to confirm a heart attack. He didn’t recognize that Sally’s condition—SCAT—would not show ST elevation, and he lacked awareness of how heart attacks manifest in women.

Chat with AI

Loading...

Pro features

Go deeper with this episode

Unlock creator-grade tools that turn any transcript into show notes and subtitle files.