The Hidden Biases Built Into Every AI System | Dr. Joy Buolamwini

You were on stage with Sam Altman a few years ago. You stated that you're worried about the ways that AI could kill us slowly. What are we missing when we talk about the ways AI could lead to different harm? How do we train machines to see when women were using AI tools to negotiate? They were being advised to negotiate lower. The data set that's supposed to represent all of humanity might be 70% male label faces and 80% lighter skin faces. When I got the software that's meant to detect people's faces, it literally could not see my face as a human face. What are all of the ways somebody may be interacting with AI systems that could be changing the course of their life and they don't even realize it? One of the biggest ways is AI scoring. You are having data profiles made of you by many different data brokers. Face data sets are extremely valuable. We now have a biometric surveillance state that would be the dream of many authoritarian governments. It's a misconception that I have nothing to hide, so it doesn't really matter that's not what's actually at play here. No one is immune to being ex-coded. Dr. Joy Bowen-Weeney is a computer scientist and researcher and one of the most impactful voices shining a light on the societal implications artificial intelligence. She holds degrees from MIT, Oxford, and Georgia Tech. Her research has been covered in over 40 countries. She advises world leaders and tech executives on the societal harm related to AI and she also uses poetry and art to shine a light on some of these harms. She's the author of the best-selling book on masking AI, "My mission to protect what is human in a world with machines." I'm Shnebovel and this is "I've Got Questions." Can you take us back to that first moment that you realized that the AI algorithms, tech companies were building and we're about to flood our world with. Had all of these biases that were working against us, what you ended up calling the coded gaze? Can you take us through that story? Absolutely. I mean, it starts innocently enough. I had made it to MIT, this mecca of tech innovation as a grad student and I was at the Media Lab excited to dive into creating cool futuristic technologies. So I decided I wanted to create a mirror that would allow me to shapeshift into anything I wanted to be inspired by a Nazi story, a shapeshifting spider from my culture, I'm from Ghana. So I thought, "Okay, let's see what we can do." And as I was making this exploration, the idea was to use a certain kind of mirror that's called half-silvered mirror, where if you have light in the background, it'll shine through. But if you have a black background, it'll look just like a mirror. So this meant I could put a digital filter on my face, but just look like I'm in a regular mirror. Again, I'm at the Media Lab. We're having fun. Why not? So I figured out a way to actually have a face come onto mine in a real mirror. So if I wanted to look like the greatest of all time my hero, Serena Williams, I could do it. Once I figured out how to make that work, I thought, "Oh, let me take it another level." So I wanted to see if this could actually, this mask in the mirror, could move with me, right? Of someone's actual face. The problem happened when I got the computer vision software that's meant to detect people's faces. It literally could not see my face, did not detect my face. And here's the thing, I start experimenting. So I drew a smiley face on my palm, held that to the camera, and when it got the smiley face on my palm, I was like, "Okay, anything's up for grabs." It happened to be Halloween time, so I had a white mask next to me in my office, and I decided, "Okay, let me just put on a white mask." Before I even had the white mask all over my face, it was detecting the white mask as a human face, as opposed to my dark skin face. And that was a moment for me who I'm thinking, "Wait a minute, is this just a technical glitch?" Because of the lighting, the angle, or is there something more at play? So it was really this experience as a grad student at MIT, working on an art project for fun that I encountered what I now call the coded gaze, the ways in which AI systems, machines read our faces, but are influenced by the people behind it. And then what was your next step? So you make this discovery, and then how did it influence your research? What did you go on to do as a result of that IOPian experience? Well, to be honest, I didn't want to do anything because it seemed like it could be a can of worms. At that time, you had, you know, letters circulating about lethal autonomous weapon systems. I was trying to get my degree and go about my business, so I was hoping it would somebody else would take care of these problems, somebody smarter with more power. I'm a grad student, let me do my thing. But what happened as I was trying to run away from it is Georgetown Law released a really important paper called the perpetual lineup. And at that time, they were showing that over 117 million adults had their faces and facial recognition networks, databases that could be searched by law enforcement using algorithms that hadn't been audited for accuracy at all without any warrants. And so I started to make the connection that what I was experiencing with my art project actually had much larger implications, especially when thinking about the use of facial recognition in law enforcement. But also going beyond that, it wasn't just facial recognition. It was just thinking in general about pattern recognition, which is what we know AI systems are essentially right. Great pattern recognition technologies. And so what would happen in other areas, maybe you're trying to diagnose a disease. Well, certain groups be left out of that, right? Maybe you're trying to determine who gets into college, who gets hired, all of these decisions. I started to wonder if the same kind of issue I was experiencing by putting on a white mass to have my face detected might go further. And so that coded gaze concept became something I realized wasn't just about AI fuel facial detection technologies or broader facial recognition technologies. And for someone who's hearing about bias and algorithm for the first time, and they're not sure how do these biases even end up in AI systems. So when AI doesn't work for certain people or maybe exclude you out of certain systems, why is that? Where is that happening in the AI system? No, that's such a great question, right? We're thinking about computers, AI systems, these are machines. They should be neutral. That was my hope. I didn't want to get I studied computer science to focus on the math, not getting some messy human situations. So I was the person who most wanted AI to be neutral, but my own experiences were giving me a different angle on what was going on. And so essentially I started looking at how do we train machines to see how is AI applied to something like facial recognition and it turned out like many other applications of AI, it was using machine learning and the machine was learning from data. So I started to look at the data sets and what I found where there were so many pale male data sets. So a data set that's supposed to represent all of humanity might be 70% male label faces and maybe 80% lighter skin faces. And so I started to realize that the very data fueling the vision of the world for AI systems didn't actually reflect the world as is and that's where the data bias starts to creep in and then you start to see it show up in terms of the outputs of different AI systems. That was back maybe in 2017 when I started this research before we got to generative AI systems. So now we've seen that's gone even further right because then what is created can also inherit that bias. So essentially any bias that society has held at some point in time if that is going to be if that's appears in literature if that's in movies if that's in any form of art anything on the internet and AI is training from that data that is going to appear or even be exacerbated in how that AI system performs or in the output of those AI systems. Absolutely, I like to say the past dwells within our data. I love that and I think it can be easy for somebody to say well you know maybe I don't pass through facial recognition algorithms every day or maybe this doesn't actually impact me. But if you were to think about the average week of somebody it's a here in North America, what are all of the ways they may be interacting with AI systems that could be changing the course of their life and they don't even realize it. Yes, I think one of the biggest ways is AI scoring right so based on your pattern of use of different apps it could be social media and so forth you are having data profiles made of you by many different data brokers that can be sold to the government right it could be also assessed by people who have something against you. And so to me I think about the trajectory of where AI systems are going because oftentimes I'll hear well I have nothing to hide there's nothing going on why do I. care, right? And I'm thinking through, where is this going? Now that we have cameras everywhere, not just cameras that used to be on, let's say, a government surveillance poll. But now people are adding cameras to their doorbells, right? Amazon Ring, more than 10 million people have these installed in their homes or in their front porches. Then you have more than 2 million meta-glasses that have been sold and more are coming. And so even if you say, "I'm not involved, I'm just walking around," you now are literally part of the surveillance state that has been created. It can lead to false identifications arrest. We've already seen this happen actually just before I got on this call. I was reading through an article about Angela Lips. She's a 50-year-old grandmother from Tennessee who was held in jail for over five months for a crime she did not commit in North Dakota. Why? Because she's never been to North Dakota. But they ran served AI surveillance technology, right? They scanned a match to her. And because of that, she had US marshals come to her in front of four young children that she was babysitting, arrested her at gunpoint, and then put her in jail, right? And then she was eventually taken to North Dakota. People checked the evidence, turned out it wasn't her, couldn't have been her. She wasn't in the state. But I bring this sort of example up because I think sometimes it's easy to think this is other people's problem. One of the things I talk a lot about in terms in addition to the idea of the coded gaze is the idea of the x-coded. Those who are exploited, evicted, condemned, convicted, you know, because of AI. And for a long time, I think there's been this assumption that, oh, that's other people's problem. If I have some kind of privilege in society, maybe it won't impact me. But when we're seeing AI deployed, particularly in a surveillance context, no one is immune to being x-coded. And so that's why I would caution anybody who thinks, oh, I can just exist in an AI world. And as long as I have no bad intentions, I'm fine. Right. The misconception that I have nothing to hide. So it doesn't really matter how I go about my day, because that's not what's actually at play here. And what you were describing, it's essentially this penopticon by the private sector. And there's billions of cameras everywhere. And it's not just our faces now that AI can detect. Right. It's our voice print. So how you speak, how you walk. So even if you think, okay, well, maybe I have certain privileges that AI might work better for me than my neighbor, how you walk is very specific to you. And if AI misidentifies that, or even if that's captured, I mean, it's one thing to have your social security number stolen, but imagine your face. Right. So there are all of these different ways that we are now getting captured by these systems. And once we are in these AI databases, I mean, can we get out of that? Or we don't even really know where our data is at this point. I think it's important to understand that one, your biometrics are always changing. And so sometimes people ask when we've done campaigns such as our Freedom Flyers campaign, where we let people know in the US at TSA checkpoints, everybody has the right to opt out. Right. And supposedly they will delete your data if you ask or if you don't. But what I have found is it's important to even put, I would say, a wrench in the system because oftentimes what we hear is this is what people want. It's friction-free travel. And these are what I start to call convenience shackles. Right. And so it just becomes, oh, let me take an extra five seconds off. My experience has been these things actually take longer. I've, you know, I've stood behind people where by the time they turn around and take the hat off and all of that, they really could have just looked at your ID in the first place. The other thing we found with some of our research with the complied-of-fly report was that oftentimes when you said no, when you refused, when you joined the off-doubt club, right. We actually do have a club. People behind them who didn't know it was an option then felt involved in to exercise it because it's so easy to have this sort of herd mentality situation. So going back to the data deletion and herd mentality, there's also the assumption that we just have to go along with it. There's no way to get these, this data purged. And we've seen when people actually push, you can get data purges. We saw Facebook met a delete a billion face prints, right. And so just like you have fingerprints, they can create a face print of you as a biometric. And that's because there was pushback. There were billions of dollars of settlement. There was a real cost to having this kind of data. So I do believe that data purges can and should happen, but you have to push for them. And what do you mean by shackles of convenience? Well, if you've traveled in the US by flight or through the airport, you have now probably interacted or had the option to use a facial recognition system. So if you didn't have to show your passport, that's because your face was being scanned by an AI system. I personally always opt out and exercise my agency to do that. Many people may be, as you said, don't know that you can do that. But why should somebody consider opting out of that? What are the risks of using the facial recognition system at the airport under TSA? What could go wrong? Well, one thing is data travels. So sometimes what we hear is, oh, it's immediately deleted. And then we look at the fine print. Oh, it's stored for 30 days or oh, it's stored for multiple years. Well, it depends on your citizenship. Well, we have a new administration and that's out the window. I was literally just looking at the AI use cases for the Department of Homeland Security over 200 active use cases. And for an app called mobile fortify, where ICE agents are able to use this to try to see, can we identify who this person is? They actually link it back to the traveler verification system, which is what's being used in the airport. And that means you could be falsely identified as somebody else, right? And so it's not even just what happens at that one checkpoint in the airport. It's what's happening with this entire data network. And so understanding that your data travels means also that your face data could travel. And it could be collected and connected to other data about you. And so in this case, for example, with mobile fortify, it can scan your face and then give a guess of who you are. It might guess and say that you're not a US citizen, even if you have your passport. And there's already been a case of this, right? Where that system was used to justify saying somebody should be deported. So then this is why I keep saying everybody's at risk of being escoded. You're like, well, I'm a citizen. I'm good. Not necessarily. So I do think keeping in mind that data travels is a really important piece when it comes to this. The other thing I think is important is resisting surveillance culture, right? And also increasing consent culture. Oftentimes, let's be honest in the airport, you are not asked. You are hurted. Come through, scan, step up here, and you are being giving a directive from somebody in power. It takes a lot with that kind of power imbalance to speak up. I have to admit, sometimes when I'm traveling and I want to get to my destination, it's a, I don't know, maybe it's an award ceremony or something like that. A vacation I really need. I do, I want to risk potentially not making it. You know, and then I remember I'm the founder of the algorithmic justice league. Let's walk the talk. But that's still a part of the calculus. Am I going to make my life harder? I get to the airport a little bit earlier, right? Knowing that I'm going to go through that process and understand what it is like for many travelers going through. So I don't say that lightly, but again, I think one of the biggest findings from our report was seeing that when somebody makes that choice, how it impacts everybody else around them. And so I would not underestimate it, especially if you are in a more privileged position, right? I think about people like Robert Williams or Portia Woodruff who've been falsely arrested due to face recognition, misidentifications. I think about people who are concerned about immigration officials, they might not feel they can use their voice in that moment. And that's completely understandable, right? But if you're in a position where you can, it becomes an act of collective resistance when many of us are doing it together. And it pushes back against this narrative that this is what travelers want. They want efficiency, right? This is the kind of narrative you'll see when you read the planning documents around this future of friction-free travel, where you just walk into the airport and you can go through anywhere. Sometimes friction is good, right? Friction means there's an opportunity to contest, right? In case something goes wrong. And it's like the slow, I love the term, the shackles of convenience because it's like the slow walk toward disempowerment or the surveillance state. We use the facial recognition to unlock our phones and now we see it at the airport and it's the slow continuum. I am the person that off-site at the airport. I am the person that doesn't have facial wreck on my iPhone unlock. And it's also cyber security. I remember reading in 2019 about the Department of Homeland Securities massive cyber breach where I think it was over 180,000 photos from airport. I think it was in Texas going through TSA were hacked. Some of them ended up on the dark web and then to go on to have their own life and something, you know, people can steal your identity and all of it. They're all of these second-order, third-order consequences of normalizing our face and our biometric, the thing that's really hard for us to change over time, unless you want to maybe go to LA and try to look different by the end of it. But the more we normalize that, the more we just end up in this world where it's really hard to reclaim that power. Absolutely. And something you just said that is so important, especially with the data breach that you are mentioning. Oftentimes, what you'll see when they go and investigate those data breaches, you'll say, well, we use the third party vendor and that third party vendor either violated XYZ or didn't have the right protections in place. So I think that's something else to keep in mind, particularly when it comes to government interactions. I do remember a few years ago, the algorithmic justice league, we were doing a campaign with a company called IDME that was being adopted by the US government as a way that people would scan their faces before having access to their tax documents. And also before having access to benefits, right? So you had the Department of Veteran Affairs, for example, using this. And we were seeing veterans who might have low quality cameras, right? Of all backgrounds, all skin types actually being blocked from their benefits. And this is sometimes where the so-called convenience isn't even true for it. And in this case, it was a situation where if you allow biometric identification to be the tool or to be the access point to crucial services, it's very easy to have a whole range of people blocked either unintentionally, which I believe is what was happening in that case, but also intentionally, if you start to have the enemies as well, or you have groups of people you don't want to participate in certain segments of government services or other parts of society. So these are the harms that are happening right now that we can see in the world of AI. And when it comes to areas of AI risk, people might be more familiar with the existential risk that tends to make news a lot. The idea that AI may one day pose a great threat to humanity. Maybe it becomes super intelligent. And then we lose control of it. We hear about that sort of risk a lot. And that topic was being discussed, existential risk, the potential for AI takeover, or losing control of this technology. So Sam answered the question. And then when it came to you, your response. And I want to make sure that I don't misquote you here. You stated that I get worried if we're not able to specify what the landscape of harms looks like across the entire AI cycle from design development and deployment. And then you said, a couple sentences later, and I quote, that you're worried about the ways that AI could kill us slowly. So what do you mean about the entire cycle of development? So where are we over indexing on the risks that I can present? And what are we missing when we talk about the ways that I could lead to different harm and kill us slowly? With that notion of the ways in which AI can kill you slowly, I was thinking about how we think about acute violence, right? You have the guns you're done, you know, but then there's also systemic violence. What happens when you don't have adequate access to healthcare? What happens when your home is next to a data center that's keeping you up, right? Or putting up your bills. And so now you don't actually have resources to better your life. And so in this way, you kill people's economic opportunities. You kill their health prospects. And this is what I mean in terms of ways in which AI can kill people slowly in terms of robbing people of opportunity or even in terms of the infrastructure used to create AI polluting the environment over time in ways that accumulate years, decades after where you see generational impacts. And so my point wasn't to say that we shouldn't think about the real harms of AI applied to the military of which I have many grave concerns about lethal autonomous weapons systems. But I was saying that that is not the only threat. And I think we have to have a conversation that goes beyond the God mode version of AI and think about the dog mode version of AI. And what do I mean by that? So some people might have seen a super bowl commercial. It was a commercial for Amazon ring. And what they were saying is we can now use ring cameras to help find a lost dog, right? And they were happy to share that they've been returning a dog a day. And it did not receive the public reception I think they were hoping for. People immediately saw that, okay, if we set up this infrastructure cameras everywhere that we purchase now AI enabled using computer vision. If I can find your lost dog, who else can I find? Maybe it's your neighbor, right? Maybe somebody who's a person of interest for some reason or another. And so I do think understanding this kind of surveillance inner working that isn't being just created by the government is being created by consumer devices. And so this is a different kind of infrastructure where we are paying in for the glasses, right? We're paying in for the doorbells for the earrings and the ear pods and other wearables that will soon be here that are always recording devices. And so that's what I mean when I think about dog mode of the AI surveillance apparatus that consumer devices are now feeding into because why was there pushback? When people saw this ad, they started asking, okay, what else is ring doing? And they found that there was a potential agreement to move forward with flock. Flock has 80,000 cameras across the United States and it's meant to scan license plates. And so if you continuously stick scan license plates at specific locations, you get a good sense of how people are moving, right? And so the question was, wait, are they now going to bring in the more than 10 million ring doorbells that are out here into the 80,000 flock network? And because there was public pushback, flock and ring came back to say, we know people are concerned. The dog situation doesn't have anything to do with this, but market pressure makes a difference. People were threatening and some people already had said, I'm no longer using this product. This product could get into the hands of officials that might be doing harm in my community. I don't want to be a part of it. And so I think it was a really good reminder also thinking about the dog mode version that we can push back, where I think the god mode version of the storytelling is it's all knowing it's all powerful. What can we do in the face of AI? Yep. I mean, I would argue that the corporate surveillance state right now, the infrastructure is much stronger in some ways than the public sector surveillance. And as you said, we are the ones unknowing or maybe unknowingly funding it. And when you think about that big picture, right, because you may think, okay, well, an Amazon ring camera capturing me or my golden retriever walking maybe isn't the end of the world, but it's not that an isolation. It's also, as you said, maybe it connects to a license plate data set and then everything you've ever purchased. And then all of that data that you give off when you're swiping or scrolling, all of the spelling mistakes you make while you're typing, that is the entire surveillance state, right, whether the corporate one, that paints a picture of who you are, who people like you are, and how well you do or don't pay back your loans and all of these different ways that it can get used against you. I mean, even in education, so you may think, okay, Again, I don't really have anything to hide, but more and more schools and colleges are now starting to use predictive algorithms to forecast things like likely to succeed or at risk and they use data such as your demographics or the demographics of people like you, your behavior data. So now your child could be predicted to maybe be less likely to succeed or maybe not actually need support when they do and it could be things like they're maybe they're narrow and typical and how they learn or maybe they've English as a second language or maybe it's just that they are a female. And we know there's more biases in these data sets against gender or or non conforming. So all of these different ways that the more data that gets collected on us can be used against us in ways that I think people don't really realize. Yeah, absolutely. And when I think about schools as well, oftentimes we see AI being adopted within the school, yes, for educational purposes, but also at times for security purposes, right. There have been school shootings. It's understandable. People want to say, whatever we can do to prevent the next one, let's go for a text solution as opposed to maybe a policy solution that could go further. And so I'm actually thinking about a teenager in Baltimore, who was a costed and handcuffed by police at his school because he pulled out a bag of chips, Doritos, but the system they were using detected those Doritos as a gun. And now when you have a situation where supposedly there's a weapon involved, then it justifies the use of force. So I think it's also important when we're thinking about these spaces that are meant to be for learning or you're even seeing the introduction of AI surveillance and spaces like hospitals. I was very surprised to learn in some instances, people use facial recognition and ambulances, and I'm thinking about these are some of the places where you're most vulnerable, right. Like what opportunity is there for consent and so forth. And for someone that's listening that thinks, how could the AI make that mistake? So how could AI confuse or falsely predict a bag of Doritos for a gun? What's happening on the technical side that AI is making those inaccurate predictions? You know, their glitches in the system and because these are pattern recognition systems, sometimes it gets the pattern wrong, very similar to was my face, the pattern that expected versus the white mask. And then you add things like if you're doing surveillance cameras, you can have grainy footage, the lighting conditions can all change. But there's so many variables that go into it. As a computer scientist, when you're thinking about computer vision, you think about illumination. What's the lighting looking like? When we test it for our own benchmarks, sometimes we do the best possible conditions, right. You and I in a studio looking directly at the camera. This is optimal. We would call it a constrained environment, AI is in the real world, bring in the rain, bring in the shadows, bring in the grainy footage. So when you have AI systems that have been trained in more pristine environments, now actually hitting the real world with edge cases, they that were never considered in the training data set and so forth, you can come up with these errors. I think it's so important to also realize it's not just about the ways in which AI makes mistakes because accurate AI systems can also be abused, right. Accurate AI systems can be used to track people that you might want to do harm to as well. So you have problems when they're misidentifications and even if we had the perfect biometric surveillance, which we do not, right. We now then have a biometric surveillance state that would be the dream of many authoritarian governments. And even when you think about trading off these conveniences for our own privacy and we hear about a future of wearables and some of this does sound great rate if we can avoid going to the hospital together or needing to call that ambulance because we have a device that lets us know your blood oxygen is dropping or your heart rate is beating abnormally. We don't have to head to the hospital in the first place, that is a win, that is an exciting future, except for when those types of wearables malfunction because of the color of your skin or because of your, because of your gender, I'm sure you read that study too with the blood oxygen wearable devices that weren't working properly for people with darker skin in particular black skin. So it was showing that black skin patients had normal blood oxygen levels when they didn't and then people weren't getting passed to the ICU that needed to go to the ICU, people weren't given oxygen when they needed oxygen. All of these insidious ways where bias can show up and this is why I always say if there's AI in your workplace, you are an AI company. So if you are a doctor now that has AI in their operating room, you have to understand all of the different ways that AI may not work for people because now you're at that touch point. Of course it has to also go upstream who is building these devices as you talked about from the design standpoint what data went into train them and clearly not enough data that's representative of the world that we actually live in, but then all the way downstream and I think the correction for when that discovery was made was even worse. The doctors just added a few basis points and said okay anytime somebody has darker skin just add a higher oxygen number to whatever it is that the reading is again completely misreading with the situation. And a few things there right because you know you and I are both black our skin type differs right and then if we get blessed by the sun you know that might differ as well and so with my research it was also really important that we were really nuance. Right when we were talking about the actual skin color skin reflectance and all of that you bring up this story and I don't think I've really shared this public we get but I want to share it with you and your audience as well. When I came to MIT in addition to building that mirror project I also took a health course where we had to build a pulse oximeter. According to my pulse oximeter I was dead right you know and it kind of reminded me of the white mask incident so to actually test out the homework project we realize it wasn't going to work on my skin right and then we had some Indian team member so we ended up finding the person with the light of skin. So we can get that project done little did I know right it would have all of these truly life or death implications when you're thinking about triage for a pandemic and COVID happening in that same class I met somebody named Alicia Chong Rodriguez and she was actually building on health tech company focused on women's heart health right so just going back to the wearables again. And as I was working with her I learned heart diseases the number one killer of everybody basically women of course and then in the US less than a quarter of research participants when it comes to clinical trials around women's heart health you know around heart disease less than a quarter of research participants when it comes to clinical trials around heart disease are women. You know wasn't even until 1993 women had to be included so given all of that going back to the past Wells within our data it was also the case that you have these consumer wearables that are trained on a mainly male data set so her intervention with Bloomer tech is the smart bra right to actually capture intimate details about women's heart health. Moving beyond the halter monitors where you go to the hospital stress then they're like calm down and we're going to see how your heart reacts versus okay let's actually get you in the real world where you're calm and so she had identified this major data gap and she's now you know built this incredible organization that I'm proud to support but I don't know that we would have even gotten to that if not for that class we're according to the Paul Sox similar I'm building from not having my face detected to being read as dead right I'm talking about right now it goes from MIT. It goes from MIT it's wild so no those I'm glad there has been more attention on this but not enough which is to say that I think sometimes some of the most dangerous applications of AI in our imagination are the good uses because they're like we've been talking about surveillance and all of this with health we want you to get better right you know and that story sounds great and sometimes we don't question the assumptions so with the same scrutiny we would want to look at surveillance AI applied to surveillance we want that scrutiny applied to help so we can actually realize the promise of these systems not just for a select few right I'm I totally I think most people would love to be in a future where before they don't have to go to the hospital because 10 years before something's gone wrong they had a ring that flagged something while they were sleeping that is the great state for everyone and that ring was 50 bucks but we can't get there if we don't acknowledge all of the way those types of devices will not work for certain people. And I think it's so imperative that we confront any societal bias, any stereotype that exists. If it can be found in the data, if it can be found in a movie script, if it can be found in how people are cast, it can get automated into the future. I was reading a study recently from, I think it was Stanford, Berkeley, and Oxford. And it was showing how the more experienced a woman is in her career, and therefore the older she would be, AI could be working against her. Because if you think about all of the images that we would see on the internet, so if you do a Google search of women in a certain profession, a lot of stock photos would come up, more a lot of different images from past movies that were really successful, and who gets cast in those roles, or who gets hired for the stock images of the female lawyer or the female doctor, she's probably much younger. So AI systems for getting trained on all of this type of imagery. And so when they asked AI systems, and these types of systems now get used in recruiting, and it worked even worse for the more professional, or prestigious a job, the worse it scored, women that were older and more experienced, when it asked the AI system, okay, generate a resume for a female physician or a female lawyer, it always had her age much younger, and as a recent grad. And so that means if you're an older woman with tons of experience, and you apply for a role that you're well qualified for, because you are older, the AI could flag you as an outlier, and then you don't even, you are completely dismissed from that role without even knowing it, when the resume says you are a perfect fit for that job. - There was another study that came out that was showing when women were using certain AI tools to help them negotiate. They were being advised to negotiate lower in terms of how much to be compensated, then compared to their male counterparts as well. So, following on from what you're saying, right, not being given your due despite experience and value. - And would that be because if an AI system is trained on historical data of salaries and pay, women have historically, because of the pay gap, women have been paid lower, so an AI system would see that, and then anchor down in a salary for a woman as that could be one of the reasons why AI would tell if it's a female negotiating for a salary, actually aim low. - It could be, right? And that's where you actually have to do the analysis and see what's going on. I remember when we were doing the gender shades paper, which was my MIT research that showed the skin type bias, map to racial bias and the gender bias. When we just looked at gender and when we just looked at skin type, right, we assumed that, okay, it's gonna overall work better for male faces and it's gonna overall work better for lighter skin faces. Then we took it a step further doing a subgroup analysis and intersectional analysis, and we found that for different systems, different groups had various types of performance. And so in one case, for example, the best performance was on darker skin males. So this is why I'm always very curious to dig into a particular system to find out what's going on because we might find an even richer story, right? It might be, oh, it's because of this particular profession, or this was the database that was more available when they crawled the websites. I know in Unmasking AI, I talk a lot about power shadows and I think what you just described is a great example of a potential power shadow. As we were talking about how does bias get into AI systems and we identified skewed data sets, my question is, why are these data sets so skewed anyways? So I started to look at the mechanism behind how that data was collected. And at that time, it was, let's use publicly available photos. Let's use elected officials. So if you're going to use elected officials as your source for your data set, then it wasn't so surprising to see that these data sets were 70% or more male. And then if you look at the global representation of who's empowered actually is around there, right? Similarly, when we started to ask, okay, there are more darker skin people in the world than lighter skin people. Why are these data sets overwhelmingly 80% right lighter skinned individuals? So we got even further to say, how do you detect the face in the first place online? And they were using face detection systems kind of like the face detection system where I literally had to put on a white mask. So it meant that even if you had images of dark skinned individuals, the automated face detector wasn't going to include them in the data set in the first place. And then you add the added bias of, you know, there is colorism, all kinds of isms in terms of who gets media attention. And when you put all of that together then the power shadow where you had much lighter skin representation isn't so surprising, right? So going back to your example, when we look at who's encouraged to negotiate, maybe if you actually did break down salaries by gender and the AI is reflecting that data, that's a power shadow embedded, right? That then makes the problem worse. - And we're gonna link to your gender shades paper because this was, I remember reading it when it first came out. And this paper made shock waves in the world. This really was a huge wake up moment for the private sector too, the response. So we're gonna definitely link to that in the comments or in the episode description. And what about all of those kind of AI memes and AI, I guess they would be AI memes, those trends that happen where you can use AI to show you and to pick to you in 20 years or you could use AI to depict you in what you would have looked like in the 70s or what you may look like if you had a family. All of these different trends that go viral and they seem really fun and they're usually really funny. Are there any impacts of participating in AI systems that are scanning our face to make those predictions? - Absolutely, I always say follow the data and we know that data travels. And on masking AI, I was talking about one of these trends that was going on, you know, to, I think this one was to see how much older what your face will look like a few years down the line. And then there was another trend that was show me in a new profession, right? And so the males were getting profession like, astronaut, doctor, women's photos and we continue to see this, they were having their clothes taken off or being put into a skimpy wear and so forth. And part of that again goes back to how these data sets are composed. If you look at a lot of the imagery of women online, if we look at the entire web, it's going to be hyper sexualized images and then you also get, I think what I found, all of this is disturbing but even more disturbing. This was happening with underaged individuals, right? And so now you have somebody where they might upload an image of their 14 year old self and that image then gets created in another form and then that can also get circulated as well. And so what should people on that note be aware of when it comes to sharing photos of your family and sharing photos of your kids? And I know that so many people knit out in different ways. Some people will blur the child's face. Some people don't really know and they'll just share the photos because you just want to update your friends and family about what your kids are up to. And then some people have very strict rules around that. What should parents be aware of when they're sharing photos of their kids online just so they have some more information about it? Yes, that data travels and so it can end up in a database somewhere else that's actually used for surveillance. And I think you even had this with real school photos being taken where the photographer at the school ended up uploading those faces to another type of data set because face data sets are extremely valuable. Biometric data, all of these examples we were talking about that enable surveillance. They need high quality face data. So an easy way to get high quality face data is to do it under the guise of a trend. That's fun. And so now I don't have to go scraping somebody's ring doorbell. I can invite you in and you might volunteer because you don't know that's what's actually happening because oftentimes the terms of use say that they can do basically what they wish or what they will with your face data. The only reason Facebook had to pay 1.4 billion in fines and then some more is because they deployed their facial recognition technologies in states that had biometric rights protections. And in most areas you don't necessarily have those protections. So when I think about my family members and they all ask me like, should I post the kids all of this? They already know Auntie Joy's going to say, but I'll cover on it that kind of thing. And also, who do you really want it to be shared with in the first place? Because maybe there's a smaller way to share it versus broadcasting on the internet where it can easily be scraped. For example, ClearView AI has 20 billion face photos that they have scraped online. Government agencies use ClearView AI. I was just looking at the inventory of AI use cases. And right now, ClearView AI is listed as a vendor to help with facial recognition for national security reasons. And I was looking at the part within this disclosure where are you able to challenge it, right? And the entry said not applicable. So knowing how much is out there in the first place, if I were a parent, I would very, even not a parent, right? I would limit, you know, the amount of exposure in terms of posting images of your kids and also thinking about consent, right? When they're older, this might not be something that they actually wanted out there for the entire web. So I think thinking about a smaller ways, you know, of sharing the photos with the people you love ultimately. And then there's also risks of identity theft and voice cloning and all of that to the more data that floods the web as the child gets older and you could use in AI, the better AI gets at making those predictions to predict what they may look like and then it creates all sorts of cyber security and identity risk as well. But you had written an op-ed and wired and I think it was about two years ago. And you wrote that you can envision a future where there's two groups of people who emerge. There's the one group who essentially just accept that our biometric data is captured, how we walk, how we talk, how we look, there's cameras everywhere and they've just accepted that. And then there's the second group you call the faceless. And the faceless resist and essentially try to opt out of this society. Can you paint a picture of what that future is that you're envisioning and what would be the benefits and risks of being on either side? - Yes, no, so in that wired op-ed, they asked me to imagine a future, right? Where I saw some of this ubiquitous AI surveillance tech going and it could look at facial recognition. So I thought about these AI field surveillance states, and then I also thought about free state or free face societies, which is where I would like for us to live, where you can go out and not be worried that someone's gonna capture your biometric print. But for those who don't live in the free face society and their cameras everywhere, I started thinking of what would it be like to go faceless? Maybe this means you get certain kinds of prosthetics that are attached to your glasses. Maybe it means the glasses have certain kinds of coating. Maybe we start to learn from other cultures where there's already a tradition of face covering. Even when I go to the airport right now, I usually have something over my face and dark glasses and things of that nature. And so I start to think the ways in which people might wanna make themselves invisible to a surveillance state because the government has not protected them. And we don't have to actually go too far out into the future. We can actually go back to the past. You know, when we think about protesters in Hong Kong, right? Where they were like, be like water, you know? And it got to the point where people were covering their faces and also smashing surveillance cameras that mask were banned, right? And so I think to see that there have been government actions on whether or not your face is visible tells you how important it is to actually have power over it. And I do believe I also got into what that might mean for children in the future. To be biometrics naive might be a luxury, right? That your child has never been scanned in certain ways or put into particular systems as well. And then I thought further, right, if you don't live in a free-face society, maybe one of the most intimate things that will be in the future is for somebody to actually see your true face. And there's so much there. You could, yeah, imagine a world where your wearables skew how the world sees you. And yeah, there's this economic divide in families based on who gets to opt out, who gets to buy the wearables that allow you to remain anonymous or to give you full autonomy over your own face. And I mean, I suppose the ideal scenario is that we don't have to go there. And we, with the truth is we don't, right? We're not necessarily there yet. We're starting to walk down a path where there's surveillance everywhere on the corporate level, on the public sector level. But that doesn't have to be the end dystopia that we end up in. First and foremost, what are the things that the average person can do today in their own life to try to be a part of a movement that gives us more agency and autonomy over our biometrics, over where we're surveilled, and gives us more rights in this environment? I think it's always important to remember if you have a face, you have a place in the conversation about AI and the conversation about surveillance, and also if you have a voice, you have a choice. So I think it's important to remember the power of our voice, sharing stories, letting people even know that opt out as an option. So often, even when I share, you know that it's expanding to 430 airports, it might be the first time that somebody has even learned that they have the option to opt out. So I would say where it is not mandatory, just think twice before automatically submitting your face to something, ask if there is an alternative, just as your default way of being. Because right now, the norm that's being established is if you're asked to submit your biometrics, your face data, you just go along. I think another thing that I've learned in terms of storytelling, and I'm a poet, you know, all of these various things, was when we did the documentary, "Coded bias." I honestly didn't want to share my experiences about coding in a white mask. I found it kind of embarrassing, but I decided it was worth speaking up, not because I necessarily knew the algorithmic justice league would form, or there would be this impact that there has been, but because I thought about the next person, right, who would be in a similar situation, and I wanted to say I at least did my part. And in speaking up, I realized that others realize that, oh, these experiences I was having, that I thought were just happening to me, or were in isolation, or maybe I did something wrong, or there's nothing I could do about it. That wasn't true, because others saw that story. And so when we literally just went on a world tour with "Coded bias," I remember being in Nairobi, and a young software developer coming up to me, and she was almost in tears, and she said, "When I saw you in "Coded bias," I knew I could be in this space, right? I remember when we were showing it in the UK, Rhodes House Oxford University, a student stood up and said that they study what they study now, because of unmasking AI, because of "Coded bias," because of story telling. And that continues to remind me of the power of our voice, not just because we have all the answers, or we have some big technical degree or something like this. I was doing this as a student. People see, you know, the doctor title now, but I like to remind them that this was when there was not that kind of credential behind my name, but saying this is worth saying something about right now. And so I think it's so important that people don't feel overwhelmed. Oh, I don't have a technical background. This is in a conversation for me. If you have a face, you have a place in the conversation because AI is shaping society for all of us, and you're the expert in your lived experience. And I think that is something to keep in mind, and exercise your voice when there are new AI systems being adopted at your school in your workplace. So for example, the New York public schools, they just recently released an AI policy that is available for comments. Go read through that, right? And let your voice be heard in terms of the questions that you have. I think these are all great opportunities to use your voice that isn't predicated on your prior knowledge is based on the fact that you're a valuable human being. And these technologies are being deployed in ways that impact you, that impact your neighbors, that impact people you care about. So I think it's so important to remember that power of the voice. Also the power of no. Oftentimes how this happens is there wasn't resistance. We saw when there was resistance to Amazon ring, they cut, you know, a whole collaboration because there was a larger risk. And I think sometimes people forget that they have the power to say no, and that power of refusal actually shifts markets. because ultimately, you do vote with the. your dollars and they do want your cash. So I think that's also important to keep in mind as well. I love that. Yeah, you've both your dollars. You can say no. You can not participate in some of those AI trends. We talked about online telling your neighbor about the impact of uploading photos of kids online where that could go. And then just asking for the school policy on AI, you know, what predictive, do we use predictive algorithms or do we use algorithms at the school with kids? And how are they used? What data is captured? All of those little acts are actually acts of resistance. But then when you, you know, you could be somebody listening to this right now that thinks, okay, I could do that. But they say now over what 70% of companies use AI in hiring, AI is now used in medicine. It's used in our vehicles. So AI in some ways, there's a lot of it that already surrounds our world. What can be done about the AI that's here today evaluating people is biased AI to something that we have to accept. And there's a line that you wrote in your book and I quote it all the time. And you say data isn't destiny. So what do you mean by that? Is there, is there more that can be done when it comes to the bias in these systems? I think it pushes to, you know, collective action where yes, you can do what you can in your local vicinity, your sphere of influence. But we also do have to push our lawmakers and our policy makers and our regulators. Well, we're talking about Facebook deleting a billion face prints. It's because there was a biometrics privacy act, you know, that was passed. And so I don't think we can step away from the importance of actually having laws. We also see that litigation helps to move move companies and move decisions. We recently gave Megan Elgarcia our Global AI Justice Award. She was the mother of Soulsetzer the third who unfortunately lost his life to chatbot a suicide, you know, he was encouraged to to no longer be here after interactions with the character he made on character dot AI. And actually she has a legal background. She decided to use that to push forward a case, you know, in her son's honor. And I do think more is going to happen on the litigation side. And we need more on the legislative side as well, right? If we're thinking about the defiance acts or all of these examples of AI harm, right? Where children are being, you know, stripped of their dignity in various ways online. It is important that we have laws in place. And that does mean having the political will. And I think something that encouraged me so much, you're talking about the reach of gender shades. I know as a grad student when I was working on it, I'm thinking, I don't know how far this will go. But it actually got IBM, you know, Microsoft later on Amazon to no longer sell facial recognition to law enforcement. You had laws be put on the books, you know, I remember when San Francisco ban facial recognition and the police said, I landed there that day. They were call get banned from Cisco and so forth. By bring up these examples to say when people resist when there is push back, then there is response. What happens is when we're complacent, it becomes business as usual. And it's up to us to say, this is the kind of society we want. We are demanding no to these actions, even if you have some of the most powerful foes. I'm so privileged to have recently joined the board of the NAACP Legal Defense Fund, right? And I think about the historic cases that they've argued in the past that made my educational path even to be at MIT to do this research possible when I think about Brown versus Board of Education or even their ongoing fights with birthright citizenship right now and 85 years of saying these communities matter. We believe in justice and we are going to strategize, we're going to organize, we're going to play the long game. Knowing that their good martial was fighting for decades to make some of these changes happen. Understanding that the doll test itself was a story to be told, I realized the white mask story was following in that legacy even without my knowing it. And so when I look at the history of sustained resistance, I see that it actually does change the course of history. And we are right in that history flow where we get to make these decisions. And I can see myself telling my children, my grandchildren, to come, right? What it was to be a member of the off-doubt club, what we were doing at the borders and so forth. Why we have a free-face society now or some free-face zones and things like that. I want to be able to tell the next generation that we thought we used our voices well. And I love that a paper led companies and some of the most powerful, some of the biggest companies in the world to change the products that they put on their shelves, right? So all of these ways that wherever you are in your journey, maybe you're a student, maybe you're a voter, maybe you can change what products you choose to buy. All of that are, all of that is part of the act of resistance and part of using your voice or using your wallet to participate in building towards a future that we actually want to live in. And that's the thing about the future, right? The future isn't some far out state. It's the decisions that happened today. That is how the future gets made. And what about when it comes to tech companies themselves or there could be somebody that's listening, they're building an HR resume, hiring tool, using artificial intelligence, what should they be aware of? So let's say that they've procured ethical data sets and everything has been lawfully procured together. But what should they know about bias? What needs to happen in these coding rooms? So we stop getting these types of AI products on the shelf that only work for every ninth person. Yes. Well, one thing I think about is algorithmic hygiene. So you wouldn't just floss once and think you're done with your dental care for the rest of the year. I floss January 1st. I'm good. And so I think sometimes when it comes to thinking about issues of bias or AI horrors, people just want to floss hard once and think they're set. And so this leads me to this idea of continual checking, right? Making sure you're exercising that algorithmic hygiene. I think the other part that's so important is affirmative consent. And to let people actually decide what they're using with informed information. So I think sometimes people are scared to share the limitation of their system because you want to hear the AI promise. But I think it's really important for developers to say this is who it was trained on, right? Data sheets for data sets, responsible AI cards. There are many ways of doing this where the whole idea is this is how it was trained. This is what it was made for. And these are the known limitations, right? And also in doing that exercise, you might realize you might not want to put it out if these are some of the known limitations as well. This actually reminds me of when I became a spokesmodel for OLA. We did this big decode the bias campaign talking about AI bias as it pertained to beauty, right? And so as I was getting ready for the campaign because it was invoked and there was a commercial all of this, I did all the right things. I went to bed on time. I drank water. I was exercising. Of course, I used the products, right? But this was knowing, you know, that there was an evaluation that was coming of me. And so in that way, I think it's really helpful, right, to think of yourself being evaluated, showing your work so people can make an informed decision about using your product. But so you also have an informed understanding of where it works and where it doesn't. I think the other thing that I've seen is the importance of speaking up internally, even though it can be hard. One way people aren't as aware of for how the gender shades paper was used, where people inside corporations saying, let's learn from that. So we don't necessarily make that mistake. Or maybe we can build more inclusive data sets. I remember hearing from some of the team members from Project Douglas, which ended up leaning Google to create more inclusive camera software, saying that they actually used the gender shades paper internally, as a way of gaining resources. You know, people money dollars to go behind this to create a different version or something that actually reflected the communities that they cared about. And so I do think it can be definitely difficult. And there is a cost. I think of my dear colleague, Dr. Timnick Abrue. She used to be the head. I say used to be the head. of AI ethics at Google. And when she spoke up, there was a cost to her job, even though she was literally warning of things that are now taken for granted as AI harms, like the environmental risk. And so I also had I talk about this in the book and masking AI kind of grappling with when you speak up versus not and also understanding what's that stake. I remember finishing my MIT PhD and saying, and just so you know, we had a legal team, right? To also understand that you are weighing different risk and sometimes you are in more of a position to speak up. Right. And sometimes you're in a position to influence somebody else who has more power and who can do a bit more of a cover to speak up. So that's why we invite people to share their stories, report.ajl.org. Maybe you're seeing something, but you're not in a position where you can say something out loud right now, but you know that this is important for other people to know. So I think it's also important to increase that surface area of how you can make a difference. But asking better questions, like who might be missing, right? From this data set, how have we tested and evaluated this? Yep. On all the products whether you're in HR or you're in marketing, who was probably in this data set, have we done, have we double checked various different profiles to ensure that our product works. And again, you don't necessarily have to have technical expertise just to present those questions to the builders in the room. You can just raise them. And so for people who want to be a part of your mission or just keep up with it to follow what it is that you're doing to follow the algorithmic justice league. How can they get involved? Where should they follow you? I've already I shout out your book. We did a live the other day or a couple weeks ago. And when people ask me what are the books that you read? I always recommend on Masking AI. It's such a great book, especially if you're you don't have to be technical to read it, right? It's much more the lived experiences of living alongside these systems. So that's a great place to start. But where can people follow you get involved? Where should they find you? I would definitely say become a member of the algorithmic justice league newsletter dot AJL dot org. You can be part of the broader movement to make sure that technology works well for all of us, not just the privileged view. I think also follow us on our social medias and things of that. But I think more importantly, to me, when I think through ways of supporting is truly educating yourselves and the community around you. So that might be watching a documentary like coded biases now on YouTube, reading and masking AI or maybe a reading empire of AI by, you know, Karen Howe. They're just so many great people and we've created a library on AJL dot org where you can find many more voices to have a better sense of what's going on as well. We're forever asking people to be part of the off doubt club. So if you've gone through TSA recently, you can go to TSA dot AJL dot org and literally share your lived experience. So we can continue to create research that keeps organizations accountable. And that's one thing I've been proud of the algorithmic justice league to be able to do because sometimes it's like, oh, I'm not a researcher. I don't have this background. I don't have that. You have your lived experience and that is so valuable for sure. And then I'd say keep supporting artists. I do have a point that I had wanted to share. They're actually two poems, but I'm thinking maybe the X coded poem would be more apt. I wrote this poem precisely who will die as I was thinking about the use of AI by the military. So here we are. Some say AI is an existential risk. We see AI is an exterminating reality, accelerating annihilation, augmenting destruction. I heard of a new gospel delivering death with the promise of precision, the code name for an old aim to target your enemy, the other reduced to rubble, face erased, named displaced as drones carry out in a formation that spells last shadow. AI wars first fought at the doors of our neighbors. Next the bombs drop on your private chambers. Seize to believe that fire on fire will deliver peace. Precisely who will die, zeroed out by ones and guns. And then take you from the low. I'll go I'll leave with a bit of a palette cleanse after that one. Okay. All right. This one to the X coded to the X coded resisting and revealing the lie that we must accept the surrender of our faces, the harvesting of our data, the plunder of our traces. We celebrate your courage, no silence, no consent. You show the pathway to algorithmic justice, requires a league, a sisterhood, a neighborhood, hallway gathering, sharpies and posters, coalitions, petitions, testimonies, letters, podcasts, research and potlucks. People asking questions like Chenade, dancing and music, everyone playing a role to orchestrate change to the X coded and freedom fighters around the world, persisting and prevailing against algorithms of oppression, automating inequality through weapons of math destruction. We stand with you in gratitude. You demonstrate the people have a voice and a choice. When defiant melodies harmonize to elevate human life, dignity and rights, the victory is ours. Thank you. I can't think of a better note to end this podcast on Dr. Joy Bolognini. It has been a pleasure. Thank you so much. You shared so much with us today. I learned a lot beyond the other end of this conversation and I've known you for years. So thank you so much for making the time and I know we're going to have you back on again soon.