Thinking About Gender Bias and Patient Participation in AI Development

This podcast is created on the current and traditional territories of the peoples of the Treaty 7 region, which includes the Blackfoot Confederacy, comprised of the Siksika, the Bighani, and the Kainai First Nations, the Soutina First Nation, and the Stony Nakoda, including Chiniki, Barespa, and Good Stony First Nations. The City of Calgary is also home to the matey nation of Alberta, Districts 5 and 6. Traditional knowledge systems and approaches to understanding our world have sustained communities for thousands of years. As we explore new technologies and ways of knowing through AI, we recognize the importance of diverse perspectives, and the wisdom embedded in established knowledge systems that have long preceded any digital innovations. Welcome to AI Rounds, a podcast designed specifically for faculty navigating the evolving landscape of artificial intelligence in health care, research, and education. I'm your host, Dr. Jesslin-Holodinski, a data scientist from the Department of Emergency Medicine and Director of AI and Data, Education and Ethics, in the Office of Faculty Development at the coming School of Medicine. Each episode will break down complex AI concepts into digestible insights, explore practical applications in clinical and classroom settings, and discuss how these technologies are reshaping medical education and practice. Whether you're AI curious or looking to deepen your understanding, think of this podcast as your grand rounds for artificial intelligence, where we will examine, learn, and grow together. So welcome back everyone, and we have a guest on AI Rounds with us today. It is my pleasure to introduce my colleague, my collaborator, and my friend, Dr. Zach Marshall. Dr. Marshall is an associate professor in the Department of Community Health Sciences, and he has interdisciplinary training in psychology, social work, and community health. He's an expert in community-based participatory research, research ethics, knowledge synthesis, 2SLGBTQ+ health and wellness, and ethical and responsible AI. Dr. Marshall also keeps very busy as the graduate program director for community health sciences here at the coming School of Medicine, and he is passionate about how we can leverage data and research for transformative social change, and I can't wait to hear more about it. So welcome, Zach, thanks for coming on the podcast today. Thank you for having me, that was a bit of a daunting introduction, but yes, very exciting and diverse career paths. So I am really excited for our listeners to hear from you. I know, I feel like if I told you how you might be thinking how the heck did you get into this responsible AI world. So that is a story which we may or may not discuss. Well, you brought it up, so that's the story I wanted to discuss, because I've been saying on this podcast that we are all first-generation AI users. So I think everybody is finding their way into AI in new and interesting and different ways, so tell me your story. How did you get into responsible AI? And maybe tell everybody what that even means. What is responsible AI? Well, maybe that's part of why I'm interested in it. So responsible AI is technically AI that is ethical, transparent, and fair, and ensures that it does not cause harm or perpetuate biases, ideally. Almost like AI for good is how I would think of it. AI for good, I like that. So how I got into this. Yeah, tell us a little bit about your journey. Who I, one of my mentors, was invited to a conference in the Hague on automation and systematic reviews. And she couldn't go. So she asked me if I wanted to go and I was like, "The Hague." And also automation and systematic reviews, signed me up. It was quite an amazing meeting. It was about 40 people, therefore I think three days. And it was basically all of these people that are obsessed with evidence synthesis and reviews. And all of these people who are obsessed with automation, machine learning, and all in one room. And having like the conversations were quite amazing because it was like kind of high extreme geek from two different geek areas. And so when I was there, I met someone who was from Serbia. Really great computer science faculty member. And she was telling me, "Oh, you know, I work with this really great team from Toronto." And so she starts talking about, "Oh, and there's this person, Everhim Begeri. He's so awesome. I'm going to introduce you to him." So I get back. She did introduce me to him. And I had been looking for computer scientists to collaborate with. I had tried to find some partners for a project we were working on. And I met Everhim Begeri, who's computer science, who at the time was at TMU, and is now at University of Toronto. And we just hit it off. We just knew that we wanted to work together on something. And he came to me with this idea of an answer-create grant focused on responsible AI. He had a lot of work on research ethics and community engagement. And he knew I was interested in automation with reviews, which was kind of a side thing at this point. And so he said, "Do you want to go in on this review because we're allowed 30% non-NSE." So non-science and engineering people. So we're allowed 30% social science kind of humanities folks. And so I ended up working with him on that project. And that's how he ended up here. Our project was successful. So yeah. Amazing. And I think it's so responsible AI is such important work. And I think that it's something that our listeners, faculty and CSM and whether they might be involved in teaching in a classroom or maybe they're doing a bedside teaching or maybe they're seeing patients. Thinking about all the tools we have in using them responsibly, I think that that's something that's really top of mind for everybody right now. So I'm so glad that there's folks like you and your large team that are doing lots of this work in it. I want to ask a couple of pointed questions about where maybe AI can go wrong and how do we think about this. So I really want to talk about gender bias in AI. I know that you do some work in that space. Can you tell your listeners what is gender bias in AI? Why is AI sometimes gender bias? How do we look out for that? What does it mean? I know that's a big topic. It is big. I'm like, "Okay. Well, we recently worked on a book/special issue about gender bias in AI. And so with my team, we wrote the section on thinking about sex gender and gender diversity. So it gave me a chance along with Batool Masawi and my co-author to really deep dive into some of this. So I'm really like to share with you. I also learned a lot in the process. So I think one of the key things we have to think about is when we're talking about AI and bias is both. It's sort of like what we might call a two-sided problem in that we have to think about both the data set that is being used to train AI, but also the developers themselves. If we leave them out, then all we have is, "Okay, well, yes, we can have a bias data set." But not only that, we have to think about where is bias being reflected through who the developers are. So we always try to think about both of those because if you think about gender bias and you think about developers, that's a really easy place to start. Because we know that in the AI field, some recent data is that about 30% of people who are working in AI on the tech side are women. And then if we talked about sort of more executive level, I think it's like 18% are women and about 25% of AI researchers are women. So already there, we see like dominance of men and also known to be a dominance of white men. So that piece sort of cis white men, et cetera, et cetera. So we then know that whatever experiences they're bringing into their work is also being reflected. And we've seen that in so many layers, right, with AI, whether it's around like facial recognition technology, would be one thing, what people are thinking about, say around employment apps, any of the structural stereotypes stigma and oppressions are automatically going to be built in to our AI systems. And so then we have to be deliberate in trying to mitigate those biases. So how do we be deliberate in trying to mitigate that? Because I understand that we have to train AI on data, and I think it's really important for everybody to step back and remember that it was humans that created that data in the first place. So we know that any human biases that exist are going to be reflected in that data. So how do we be really thoughtful about mitigating bias? Yeah, it's such a good question because sometimes the bias is also a preference. Like I think of this, oh, there's a whole area of research around robots and kind of like, you know, the kind of Alexis, you know, Siri. Yes. All of these tools are actually very gendered as well. And it turns out that in many cases, humans prefer women robots. Interesting. And so this sort of, I would think of it as the feminization of especially emotional labor is then reflected in people's preferences for these, what might be called tools slash services. So for your voice assistant to be active, you a woman's voice. Yeah, yeah, or you know, even probably something as simple as when you're driving, you know what I mean, and you have your instructions for a map, you know, what are we picking? So there's, I think, an additional layer of complication, especially if you add in the capitalist preference on top of that, it's a business. So if your customer's preference is for a woman robot or women robot service, then what are you supposed to do about that? That's a really interesting, that's an interesting thought. I hadn't really thought about it from that, from that perspective, that's very interesting. I mean, that's just a piece of it, but it is a kind of complication of the fact that there are human preferences also layered on top of it when it becomes offered as a paid service where someone has a choice. Yes. If it's more like, okay, on the back end, for sure, we can think of what we could do. And in that sense, I would be thinking about diversity of the developer workforce. But that might be a slower, I mean, there's many things we could do on pipeline than about training. And when do we engage people? And even in like high school, junior high, you know, from early stages, who are we excluding? Who are we including in opportunities to get involved in, say, computer science training and programming, coding camps, all of those kind of things, right? So thinking about gender and STEM, I think is a key piece and also other intersectionalities. But then for me, I keep wondering about involving the public. Yes. Public and/or patients, depending on the language we might be using, but if we're involving people who might be impacted and/or using the tools, that makes a huge difference. Because for sure we know in computer science, world, and tech in general, it's not uncommon to have people develop the entire tool, you know what I mean? Sure, yeah. I guess it's not a lab, but I'm like in the lab at their desk and then only later be like, well, why doesn't it work? You know, well, we talked to a doctor or we talked to a one person, you know, who works in that area, so what's the issue? And I mean, this is a really long standing challenge, right, which has been studied for years. So that's a piece of it too. So I just keep thinking, oh, if we engage members of the public earlier, those biases, the challenges, and even just the end user should be engaged earlier, right? So that they can highlight, like, well, that's not going to work because of this reason. It's evident. Yeah. Okay. I like that a lot. So thinking about, there's lots of levels of engagement there, or just, you know, maybe we'll unpack each of them. So let's talk about maybe, you know, the patient or the public. So you know, as we're thinking about maybe building or modifying, you know, AI tools for use in medicine, you know, what's the right way to go about, you know, involving patients or the public in that process? How do we make sure that we're getting maybe those traditionally marginalized voices from the start, you know, and not as an afterthought? How do we go about doing that? I mean, that part to me, I feel we already have a lot of knowledge about how to engage patients, right? Like if we were to think about service offer through apps where you or pace or, you know, we really know how to engage people early. And people who are most impacted and having clarity in terms of, okay, like what age group might we need or what racial ethnic groups might we need? Obviously thinking about gender. So, and I think age is another component. So I think when we think through those pieces, sometimes people say, oh, we can't involve the public because it's too confusing, you know. We wouldn't be able to explain what's happening, plus it's it's a sort of like quote unquote black box. We don't even really know what's happening. But I'm like, oh, it feels like a cop out. I agree. You know, we can figure out how to explain as long as we can figure out how to explain it. That's maybe then the key challenge. Well, then let's work with someone who can help us figure out how to explain it if we don't quite know. Yeah. And actually, I think I think AI can be a place that can help us explain AI. So I've used before I in some patient engagement work that I'm doing with patients with chronic pain. I actually used notebook LM to help me synthesize my, you know, technical research document. And then actually use their podcast feature to make a podcast out of that technical research document. And my patient partners loved it. I got such good feedback from them that, you know, they could listen to that podcast and get a better understanding of some of the methodologies we were using and they felt, you know, better able to participate in our group meetings. So actually, there's a place for AI to help the patient engagement too. I love that. I absolutely love it. I think that's a fantastic idea. I have not thought of it. And I think it's, I mean, it's great. And you have direct experience of positive feedback from it, right? Yeah. So the piece about, oh, people wouldn't understand. I just feel like that is also not giving enough credit to members of the public. I agreed. I mean, I've worked with junior high students before around privacy and medical images on Google. And they were way ahead. You know, they're way their concerns were also way ahead of what I anticipated they would be. So I think it's always just we need to know our audience. And if we're not totally sure, well, then like anything else, we would probably pilot. You know what I mean? Have a conversation with a couple of people see if it's clear of that are representative and then evolve our message from there. So we have to learn too. I think that's it. We have to learn how to communicate. And I mean, the truth is when you're working on transdisciplinary teams, it's the similar problem, right? Sure. I don't understand how it works either. I don't actually have a clue. I have to be like, so can you tell me how this might. I mean, I went to a meeting the other day and I was talking with the summer students, right? About, okay, how are we going to deal with the duplication in this data set? And there was like the raw data was kind of it's, it's a little messy, you would say, right? And I said, so well, I guess we're going to take three months and manually go through it. I get this email after right from the third year undergrad student who's like, Dr. Partial, I have a proposal to make. We could write some code that could easily do this, probably in like, I don't even know an afternoon. You know, so probably less. It's just an example of we can be very limited by our current ways of knowing. And when we bring together these transdisciplinary groups, which is what I love of one thing I love about this this integrated team is that then we, we can really have new ideas, brand new possibilities. And I guess AIs is a component of that for sure, sure. So I was asking, where do you see some of the biggest opportunities for that transformative change, like you said, you like to use, you know, data for transformative change, where do you see some big opportunities for transformative change in the medical AIs space? Like, what do you see that we, you know, we could be doing or maybe we should be doing. I guess because my world is so much around engaging people with lived experience and public engagement, I would see two areas. One is really having the public much more a part of these conversations. So far, when it comes to AI, the level of engagement from communities has been fairly limited, at least in the research sphere in the universities, when we look at the projects, you know, there are definitely some examples at University Calgary right now. But overall, it's under explored area. And so for me, that's one piece. And there are some people who started to work on it in states that I've seen some papers about like, well, this is a good idea and why it could be important. But for me, that is probably one of the biggest opportunities. The other is these trans disciplinary conversations, because if we actually bring, you know, instead of having no offense to our computer science and engineering folks. But if someone is identifying a problem that is not actually working in the setting. It doesn't work. Like, we have to have them, it becomes an implementation problem, right. So if we have implementation science people, people who are say clinicians who've identified a challenge and/or patients who identified a challenge and then computer science and engineering and other design folks together from a much earlier stage, I think we can be much more successful, including this implementation piece, because I think some people do not know that there is an entire field. People that study implementation, they would be like, okay, here's the results, we're good to go. And that's just not realistic, right. Well, and I think that speaks to, you know, some of the things that we're trying to do, you know, in our learning health system is get that kind of like knowledge to action cycle going faster, because we know it can take years for knowledge to be uptaken after, you know, something has been proven effective. And so I wonder, is there a space for AI in that as well? It can AI help us uptake findings faster or get things into the healthcare system faster? Is there a space for AI to enable the learning health system? Oh, definitely, I mean, even what you were talking about about making the tools more accessible for multiple audiences, that would be one way for sure, because we all know, you know, that you can write a message and say, oh, could you make this more formal? Sure. Or could you make this, my audience is funders, or my audience is these type of researchers or clinicians, like definitely in that sense. I mean, we know that we also have to check all these things. It's not always right, but I think it can be a really great tool in that sense. I think the other piece, I love your idea of almost like turning it back on itself, because then you're like, okay, well, we want to make AI more responsible and more potentially even useful. And then we could use AI to think about that. I'm so intrigued by this idea. I wanted to go back one moment just to the part when you were saying, what else can we do say about gender bias? Yeah. There's this really cool idea, which they call the data nutrition labels. I don't know, have you heard of this? No, no, it's really, really cool. Okay, so data nutrition project. So it's kind of like, you know, when you buy something in the store, like a box of cookies, right? And coming to mind. So on the back, it'll have like all the ingredients and then it has like the value of each, you know, how much protein, how much carbs, blah, blah, blah, how much fat? So in this thing, I like that you're having cookies that have protein as the first is the first. You're telling more about that later, but yeah, the kind of cookies they can make nowadays. It's kind of shocking. But anyway, on that note, so what they do is in this case in the data nutrition project, they have three parts. So one is like the overview of the data set, then there's like use cases and alerts and then information about the data set. And so this is in line with sort of like increased transparency. Oh, I like this a lot. Yeah, so it just makes much, much clearer what is involved with the data set itself. And so they've just been advocating for things like this, that piece. And then there's also this idea of having data sheets that talk about even the data set development process and maybe what prompts were used. So those type of pieces about being transparent and almost like attaching them as a component of the data set, kind of like in the wrapping of it, then that is like something that is being advocated by some. I think that's a great idea is one of the things that sometimes people worry about in this space is, you know, we are creating data at a volume that we never have before. Which is a great thing for machine learning and AI, but also the potential to reuse or misuse a data set just because it happens to be available to you. So understanding those nutrition facts of the data set to maybe help you make an informed decision of, you know, is this the right data set for this problem? That could be super helpful. So we've talked about some like, you know, big opportunities maybe for transformative change. And you know, some of these come through like really interesting tools that people might be developing. But are there any ways this could go wrong? And I'm thinking back to some conversations we've had about gender bias in AI. And you know, sometimes that gets me a little bit worried. So is there anywhere this could go wrong? What should we be concerned about? Unfortunately, there's so many ways it can all go wrong. So yes, we should all I feel like my default is yes, we should always be concerned. And then if we're fortunate and we put enough care into it, hopefully it will go more smoothly, but we have to be, I feel like we have to be sort of cautious as a first response. Yeah, approaching this with a little bit of caution. I think that's actually really, you know, healthy. And I've talked a lot on this podcast about how I kind of you AI as like racing a child, like training your AI is like racing a child. And you know, you're cautious with the raising of your child, right? So I think, you know, you could be cautious with the raising of your AI. Yeah, yeah. But when you asked me before about what are the possibilities, you know, all of a sudden we were, we went down the robots a little bit. We were talking a bit about robots and there's a whole piece around that. But also I was thinking, oh, the automated facial recognition technology could easily be used say as an intake tool. Sure. To see, you know, who's coming in the emergency department and now even before they, you know, sometimes we've got a lineup out the door even to hit triage. But what if, you know, the moment you walked in the door, you were recognized and, you know, and your, your health history was available. I can see places that that could be really useful. But maybe I've got my rose colored glasses on a little bit. Well, no, like I, I agree with you. I think it could be useful. But there's a bit of a challenge. I guess I would say around the challenge. Yes. Okay. So part of how I think of it relates to, okay, so how can I put this? I guess I should start by saying that faces are typically marked by what we call sexual dimorphism. So typically people and also humans can tell with a high degree of accuracy. I think it's like almost 96% whether or not a face that's shown to them is male or female. But then when I start thinking about this in relation to gender diversity and also cultural diversity, it becomes more complicated. So say for trans folks. Yes. If they were to come in to an intake and sort of go through the facial recognition scanner and then are interpreted as male by sex, even though they are women, this could be a problem. Absolutely. Because this mismatch, they might say, oh, wrong person story, you can't come in or you don't match what you're supposed to match according to your identification. Yes. According to the say sex marker on your health card. So there's definitely pieces there and there's been a whole thing about people sort of like trying to game the system in terms of how their faces are perceived. Right now we see it probably most often in air travel. This is where they're using the facial recognition technology at a very high level. I don't know about other borders, but that's definitely one place we're seeing it a lot. So when I think about it in the healthcare context, knowing some of the challenges people are having in the border sort of security area, I can see there being, I can see there being some potential difficulties. And then as I said, when we bring in some of the research that's been done around say in relation to black communities and facial recognition and or misrecognition. It might be a better way of putting it failure to detect people's faces and their genders has particularly impacted black women say. So all those pieces, I'm like, it could be a useful tool. And how do we build it better? I like that. How do we build it better? I think we go into lots of these things, you know, with the best intentions for our patients or for our providers. And then there's unintended consequences. And so I think what you've highlighted for me today in this entire conversation is that transdisciplinary work matters and community engagement, public engagement, patient engagement, clinician engagement, this all matters so that we have a lot of voices at the table. When you are developing or testing or trying to implement these different tools, because we all have our own blind spots, whether that be because, you know, you're the developer, but you're not the implementer or whether that be based on your lived experience or we all have this different lens to bring to this. So what I'm hearing from this conversation is the more people at the table, the better. Oh, absolutely. I totally agree with that. And maybe also the feedback loop. Feedback, okay. Tell me more. Well, once we do implement, then where are the opportunities for people to share their experiences? Because I think sometimes it's like, okay, well, it's in place and we're good now. But I know like with the algorithmic justice league, right? They've been talking about how to, say, resist, you know what I mean, or report challenges. And if people don't report the challenges, I think it can become a more invisible sort of insidious form of discrimination or challenges. People might experiences, but if we have good mechanisms for reporting those challenges, then I think and adapting, then we're in a much better place. Right? Because they're like, okay, wow, that didn't go well. This portion of our people who are using the service, it's not working for them or the experience. They had a negative experience. Then we could, we have, we have to have that feedback loop. I like that. Well, we could sit here and talk about this probably forever because I love digging into these, you know, deep topics, especially with someone that's so steeped in them, like yourself. But for the sake of our listeners, I'm going to cut it off sadly, but tell our listeners, you know, where can they find you if our listeners are interested in your research, interested in getting engaged, how can they get in touch? Oh, yes, definitely community health sciences in coming school of medicine, University of Calgary. I'd love to hear from people. And also, of course, we have the Responsible AI Initiative. Our website is through Toronto Metropolitan University. And yeah, we still have another three years left on that project, so lots to do. Amazing. Well, thank you so much, Zach, for spending some time with me today, spending some time with our listeners today. I know that I learned a lot, so I hope that you did as well listeners, and I can't wait to see you on the next episode. I hope these insights help you better understand how AI is shaping the future of medicine and your practice. If you have questions or suggestions for future topics, please reach out to us at [email protected]. I'm Jostelyn Holodinski, and we'll be back with more AI insights on our next episode of AI rounds. Thank you for joining us. Hi, I'm Luigi Riscaldino, producer of AI rounds. This podcast is brought to you by the Office of Faculty Development at the coming school of medicine. For additional resources on AI and healthcare, visit our website. You can find the link in our show notes. Thanks for listening.