The Ethical Algorithm with Michael Kearns

(upbeat music) - I'm Martin Reves, Chairman of the BCG Henderson Institute, and I'm joined today for our podcast by Michael Cairns, who is a professor of computer and information science at the University of Pennsylvania. Michael is an expert on machine learning, artificial intelligence, game theory, pointed to finance, but also a very topical subject, algorithmic ethics. He's a recent book with Aaron Roth on the ethical algorithm, the science of socially aware algorithm design, published at the end of last year, is what we'll be talking about today. So congratulations on the book, Michael, and welcome. - Thanks for having me here. - First of all, I wonder whether you could give us a sense for the pervasiveness and influence of algorithms in our lives. I think we've probably all got a general sense that this is quite an important topic, but could you give us a sense of just how pervas if these algorithms know all? - I'd say extremely pervasive, almost ubiquitous, I think really the rise of the consumer internet over the last two decades, and all of the behavioral data that's given has kind of morphed algorithmic decision-making into something fairly unusual and rare, or to something that influences all kinds of important decisions, like college admissions, medical decision-making, criminal sensing, consumer lending, and the like. So I think it's really everywhere, including areas where we don't even realize it, the most part. - Your book essentially tells us, you know, take heed to the ethical dimension of these increasingly effective algorithms, and I think it's widely known that privacy is an issue, and fairness, for instance, in hiring is an issue, but your book details a number of other ethical dimensions that we should pay attention to, often privacy and fairness. What are some of the other things that we should think about? When we think about the broad social impact of algorithms? - Well, certainly another one that's received quite a bit of attention is explainability and accountability, so not only having models and algorithms that behave fairly in some sense or preserve privacy, but can explain their decisions and their actions, perhaps even to ordinary people. So concrete example would be, if an algorithm has decided to deny you alone, you might like to have an explanation, and that explanation might take the form of the smallest change to your application that could have resulted in a different outcome. - I see, and others beyond even explainability? - Well, our book in particular, and the research that it describes focuses on sort of social norms where it feels like we have some kind of technical handle on what they mean, but of course, many people talk about the morality of algorithmic decision-making as well, on which there's at least at this point less to say that it's scientifically precise, and so a common type of discussion, maybe not a scientific one yet, is whether algorithms should be used for certain types of applications at all, for example, very topical these days is the debate over whether face recognition should even be a technology that's deployed against ordinary or upon ordinary civilians. - So let's dig into some of these problems then, and maybe try to say what the problem is, and what we should, at least in principle, 'cause I think many of these things don't even have solutions, at least in principle, how would begin to solve for some of these problems? So the first one is privacy. I think you're famous for saying that anonymized data isn't. So what's the problem, and what can we do about the fact that these machine learning algorithms operate on vast amounts of very private and personal data in some cases? - I wish I was famous for having said that. That quote is actually due to Cynthia Dwarf, who is a well-known privacy researcher and expert on the faculty at Harvard University. But what she meant by the term anonymized data isn't is kind of twofold. It means that either you haven't done enough anonymization of your data for it to truly prevent the re-identification of individual citizens in your data sets, or you've had to do so much, the data isn't really anonymous, or you've done so much modification to the data that it's not really useful data anymore. And so one of the points we make in our book, not just about privacy, but about other social values as well, is that it's important to have rigorous definitions around these issues, but it's also important to have the right definitions. And it is unfortunately the case that to the extent that we have laws or regulations that are precise about what they mean by data privacy, for example, they usually adopt some notion of anonymity or eradicating personally identifiable information and the like. And it's now well understood that these notions of anonymity are really provide no privacy guarantees whatsoever. They are easily hacked in the sense that you give me an allegedly anonymized database. And I combine it with other publicly available sources of data, and suddenly I've managed to figure out which of these medical records belongs to you. - So I believe one strategy you talk about in the book to deal with this is the idea of adding some random data to the data set without deteriorating the performance of the algorithm too much. And you also talk about the idea of differential privacy. Could you explain that to us a little bit? - Yeah, these are really the same idea. And it's not really adding additional random data. It's actually adding noise to the original data or more generally adding noise to a computation in order to preserve privacy. So the high level idea behind differential privacy is take a very concrete example. Suppose I want to compute the average of a very long list of numbers. Maybe it's the salaries of everybody in a large corporation. And I'd like to publish to the world what the average salary is. But I'd like to do so in a way that assures every employee in the company that their specific salary won't be reverse engineered from that average. And it turns out if you just straight up compute the real average to numerical precision and publish that to the world, it will be possible under certain circumstances for people to be able to reverse engineer specific people's salary. But if instead of releasing the exact average, I compute the exact average and then add a little bit of random noise to it, maybe a small random positive or negative number, then you can actually prove that there are limits to how much anyone can infer about any specific individual's salary. And differential privacy is basically a very rich body of algorithmic techniques that provides privacy guarantees by injecting noise into computations. - So let's go on to the next one, fairness. Is the main problem here bias in the real world training data sets? Or can fairness problems rise in other ways? In other words, if we code today's bias as real world bias is in the data then any algorithmic calculation is gonna be bias. Is that the main mechanism or are the other ways in which we can go into trouble? - Yeah, that is certainly one mechanism, but it is far from the only mechanism. So what needs to be a little bit more precise about what we mean by bias in the data set. If by bias in the data set we mean, for example, that if a police department has racist officers who decide to stop and frisk minorities more often than the general population, then we would naturally expect officers to find contraband on those minorities more often, not because they're more likely to be carry in contraband, but simply because they've been stopped and frisk more often. And certainly that is a form of bias in data. And if you kind of start a machine learning process with data like that, you shouldn't expect the machine learning to detect and undo that bias. But even without that kind of over bias or discrimination in data sets, there are many other ways in which machine learning can introduce demonstrable discrimination and unfairness in its modeling process. And as we discussed at length in the book, the high level reason for this is that machine learning as it is largely practiced today is a very sensible, principled optimization process. And usually that optimization process mentions only one criteria, which is predictive accuracy. And if all you mention is predictive accuracy in your optimization, you shouldn't expect to get anything else for free. So for example, if the most accurate model overall on your data set achieves that overall optimal predictive accuracy by being much less accurate on black people and more accurate on white people, well, you didn't ask to have the error rates equal on those two populations. You just have for the overall error rate to be equal to the optimal. And so you won't get that for free by kind of vanilla standard machine learning processes. You would need to modify that objective. And that's one of the proposals of the research that we described in the book. Sounds like a tough problem. How do you get around that? Scientifically, you know, from sort of a foundational perspective, it's really a very minor tweak on the sort of standard objective of optimized for overall accuracy. We're just going to add some fairness constraints to the process in our data. Where things get tricky is that, of course, when I add constraints like that, I should expect the overall accuracy to go down. And so we have to get used to the fact that we were going to face trade-offs between our overall predictive accuracy and how fair are the models we learn on. I guess the, you hinted a more subtle form of bias or fairness in the book, too. When you talk about enshrining and fairness, I guess that's it. example would be it may be objectively true and in that sense fair that poor people are a high risk in relation to loans, but if you hard code that if you like, if you make a hard decision rule or something, then essentially you've preyed on fairness, but if you've locked it in, applied it monolithically, can you tell us a little bit about that phenomenon? Yeah, this is where things can get quite tricky both conceptually and technically, which is, you know, if you take the many situations in the real world where algorithmic decision making is used or just normal human decision is making is used, but those decisions in turn kind of influence kind of the future date of the you gather or the state of the world as an economist might say, then things can get quite hard to think about and so one example would be let's say in consumer lending, if a poor neighborhood can be demonstrably shown to have a worse lending risk than a lender might legitimately say, well, look, you know, really, this is a higher risk for me. I should be allowed to give loans at a lower rate in this community. The problem, of course, is kind of causal feedback, right? Which in this case might be, well, the reason this community is a higher lending risk is because historically they've been under invested in. They have fewer resources to educate themselves or to build small businesses and if only lenders for some period of time went ahead and took on this higher risk, that might actually raise the fortunes of this community to the extent that they were a lower lending risk now and then everybody wins, right? Now the lender has a whole new lucrative market they can lend to and now the community is a lower lending risk. To account for that sort of thing, you have to think about not just the decisions you're making in the state of the world as it is now, but how your decisions and actions might influence the future state. And these kinds of problems are difficult, even really without fairness constraints, but certainly fairness constraints don't make them easier. There are frameworks for thinking about these problems in machine learning and particular reinforcement learning is one popular one for reasoning about models that have feedback loops with the future state of the world, but I would basically say thinking about fairness in such settings is most definitely in a very nascent state at this point. MSA book force me to think quite deeply about this example because I guess there's a problem of definition. Do you really want equal error rates? Is that what exactly what you mean? It forces you to say, well, in this particular case, I want more than fairness. I want positive bias, let's say. It made me think about feedback loops outside of the algorithm. Your book is called ethical algorithms, but in some sense, there may be a larger algorithm, the social feedback loop, which is also an algorithm in the sense, which we might need to address. I guess it brings up the idea of freshness in the sense that the the starting state is not necessarily the end state. The algorithm can influence the state. So you want to think about not just what you want to do, but where you want to get to. Although some of the nuances that are appropriate to your argument. We talk about these kinds of issues a little bit in our book. Again, the book is meant to describe to a general audience the state of the science on these topics, and so we have less to say where there's less science so far for better or worse. But we do try to point out that the machine learning pipeline is only a very small piece of an overall pipeline that in some circles people would call socio-technical systems, where upstream from the machine learning there are data gathering processes. So, for instance, in a criminal justice application, it could be that you start upstream with police making decisions about who to investigate or who to stop and frisk or who to charge or who to arrest, and then downstream from that there's a whole court process and decisions are made there. Finally, downstream some machine learning algorithm may get some data from these upstream processes and then build a little model for just making a very localized decision, like should this person receive bail while they're awaiting their next court date or not. And then, of course, those decisions have all kinds of downstream consequences. I mean, some people are trying to work two jobs, and if you don't give them bail then they'll lose their jobs, which has further negative consequences beyond the scope of the little decision that the model made. It's very important, even though there's not a lot to say or do scientifically about these kind of larger contexts right now, to remember that they're there and to, you know, as we say towards the end of the book, not be too guilty of rearranging the deck shares on the Titanic, so to speak, by just myopically focusing on the pure machine learning modeling part of these problems. I guess the mental trap there is to think that the computer algorithm is the whole algorithm. We've over-aligned one part of a process and so I like your distinction between computers and computation. You need to look all of the computations, including the social computations. Will that be the basic idea? Yeah, that's exactly right. And the algorithm part that involves computers rather than some more general notion of computation is really the part over which, you know, people like Aaron and I have the most control right now. It's a scientific process. It's well understood from an engineering perspective and a practical perspective. We know a lot now that we didn't know 10 years ago about making algorithms more fair, more private, more explainable, and the like, but if we fool ourselves into thinking that of just because we do that to our algorithms and there's still a long way to go implementing even what we know and fool ourselves that we've kind of solved the whole broader social problem, we will be sorely mistaken. So let's come on to the state of the art in business. Are these issues that you talk about in the book essentially things that we're thinking about in academia but not in real life for our most major digital players on top of these issues is the right sort of regulation about to appear. How would you characterize the state of the game in terms of how well we're addressing these issues in the real world? I guess my summary answer there would be we're just at the beginning of that but I think it is starting to happen. So in particular differential privacy for example is now starting to make its way into large scale deployment. So most notably the 2020 United States Census, the Census Bureau has promised that every summary statistic it releases to the world from the underlying raw you know block level census data will be released under the constraint of differential privacy. So they are going to add noise to those statistical reports and entities in a way that promises differential privacy to individuals who participate in the census. So that's in some sense a bit of a big moonshot for differential privacy. There have been a number of other deployments of it by tech companies for collecting behavioral data from browsers and mobile devices in the life. This is a big application of differential privacy. On algorithmic fairness you're now starting to see more and more open source packages for machine learning that implement different notions of fairness in the model training process. I think it's still a bit early you know we've not sort of seen widespread deployment of such methodologies in machine learning but I think the next five years are going to see a rapid increase in that especially in tandem with all of the increasingly forceful demands for more specific and stronger technology regulation. On the topic of regulation since you mentioned it this is of course a much much longer conversation. I think my basic view and my my co-author Aaron Roths as well is that current technology regulation in the US and in the EU is sort of woefully inadequate at this point and woefully under specific and I think one of the painful transitions that is going to have to be gone through in the next decade or so is a whole scale rewriting and revisitation of technology laws and regulation. On that topic of regulation I think about this in business more generally and I guess there are different regulatory philosophers different ways of ensuring the trustworthiness of a complicated black box in the case in algorithm. One would be to design in as a technical feature and your book sort of deals with where we can do that and where we can't do that. I guess another one would be tove fair process if you have a transparent process where there's consultation and involvement that sort of guarantees some degree of trustworthiness. I guess another one would be an expert endorsement you know somebody says look I'm an independent party and I assert that this algorithm meets certain qualities. I guess regulation itself do this don't do that standards self-regulation the financial markets you know we know the quality of bonds not because they comply with government regulation but because there is a rating agency that is a private sector business that assesses the business of other businesses and then more recently we have sort of market-based mechanisms like rate of rating systems on e-commerce sites as you think about a way to tackle this problem that is you know technically accurate enough adaptable to many situations and fast-moving technologies that doesn't throw out the baby the utility with the bathwater as it were what would be your philosophical approach to the problem regulation. So I think that there is a role for all of the mechanisms that you describe things like seals of approvals there even are already consultancies that offer to help corporations make sure that their model training processes and algorithms are fair in various ways and I don't have sort of strong opinions on kind of the effectiveness of those consultancies yet but you know another. possibility, which is at least conceptually even simpler than some of these more organizational solutions, is just odd, it's of various kinds. So in particular, many of the headlines that your listeners might have read in recent years about this or that model being released by some company, recent one, for example, was the claim. I still don't know how carefully it's been scientifically documented, but there were anecdotal reports that when Apple came out with a credit card last year underwritten by Goldman Sachs, that a number of people immediately anecdotally noticed that there seems to be gender discrimination in the credit limits that were given. So you had people claiming, "Oh, my wife and I file our taxes jointly. She has the higher income and credit rating than I do. We both got the Apple credit card, but I got ten times the credit limit that she did. Why?" And to the extent that discriminatory behavior like this appears in predictive models, if this is a real problem, this is something that if only somebody had thought to check in the model before releasing it to the world, they could have easily done it. You kind of just take your training data said, "There are details here, right?" But at a high level, you look at what your model is doing on men and you look at what your model is doing on women, and if you notice sort of gross disparities, then you sort of say, "Hmm, maybe we should revisit the training process with some constraints to approximately equalize those credit limits." And so a lot of violations of fairness are things that can be easily caught and measured if somebody bothers just to think about it and to check. And, you know, as a self-regulatory matter, I think companies should already be contemplating doing this themselves, but it's also something that regulators could demand in a way that I think wouldn't throw the baby out with the bathwater. It wouldn't have to be too heavyweight, and it could be done only periodically, and maybe only in the case where there was some evidence that there was a problem. So our audience is mainly C-suite of large companies, and I'd like to ask you, Mike, whether you think this is a strategic issue that we're dealing with here, whether this is an issue of great import that CO should be thinking about, or whether this is best delegated to the IT department. If you think about the consequences for an individual company of having senior focus on this versus not having it, what would you say about the significance of the issue? I would even go so far as to say that the topics we're discussing here are maybe the first time in which machine learning AI issues should be and are being elevated to the C-suite, right? And that's because the risks here are no longer limited to kind of operational risk or simply to, oh, if our machine learning isn't being done in a state of the art way, then we're losing some profit or revenue because our models are a little less accurate. The risks here are many fold. So first of all, I think the coming regulatory risks around these topics is growing rapidly. Already, the headline risk is significant, right? I mean, if your listeners who regularly read, just say, even the Wall Street Journal, you know, I mean, of course, it's amazing to me as somebody who's been in this field my entire career that you can now read the Wall Street Journal or the New York Times and it's clear they expect you to know what machine learning is, right? And this is simply wasn't true 10 years ago and now they have columns and journalists devoted to these topics. And so if you're a large corporation that's consumer facing and somebody gets hold of one of your products or services, maybe because it is consumer facing. And so it's possible to do third-party external experimentation on your products and services. And they discover that your models are demonstrably biased by race, by gender, by something else. It's going to be in the papers and it's going to be in the papers the next day. So I really think that this is something that should be elevated to the C suite and the companies I'm familiar with already have done so and have people who think hard about regulatory issues specifically around AI and machine learning and specifically what they should be doing technically within their technology divisions about the problems that we're discussing here. Let me ask you about accountability. So supposing I am a CEO and I want to do something about this. In the case of human misbehavior, accountability is usually not such a complex issue, right? The person that committed the offense is responsible and if he's an employer of a company, then in some circumstances the employee can be responsible. But with the algorithms you're talking about here, some of them might be open source, so it might be a problem with the original program. It might be the programmer who adapted the program. It might be the process of gathering the data, or it might be the documentation. It might be the unintended use of the program in the marketing department. It may be the algorithm, the socio-technical algorithm, as you called it, outside of the technical algorithm. How accountability seems to me to be a complex matter? How do you think about that? That is a great common question and I entirely agree with you. And this is one of the reasons I remarked earlier that I think whole-scale revisitation of our laws and regulations is required. Because exactly as you point out, compared to human organizational decision-making historically, what I might call diffusion of responsibility in algorithmic decision-making and machine learning and AI is so great that our current laws and regulations basically don't apply at all to a very nice paper by a couple of legal scholars a few years ago, studied exactly this kind of diffusion of responsibility in the context of Title VII law in the United States, which basically is the body of laws and cases governing anti-discrimination in hiring and employment in the United States. And this very nice paper basically in a painstaking way goes through and says, like, if you look at all of the landmark cases historically for Title VII and you revisited them changing only one thing, which is instead of people or the company having maybe these decisions and algorithm made them, could a good defense attorney kind of wiggle out of the decision because of this diffusion of responsibility. And their answer was an overwhelming yes in every case. In other words, our laws just weren't written with this kind of diffusion of responsibility in mind. And so I think that's one of the things that's going to need fixing. And as I say that, I know that this will not be easy and it will take a lot of time. But I think we are on the path to that. It sounds like we're at the beginning of the journey and there are still many things that we don't have. We don't have the right laws. We don't have technical solutions to all these problems. But on the other hand, there's a good case that CEO should take action already. If they wanted to further this agenda, but what are the two or three things that they could do to deepen their own process? So I definitely think starting with designating a person or a group whose sole job is to think through these issues in the context of whatever the particular company or business is, is the right first step. And I think it's important that that group be a diverse group. Both diverse in the normal senses of that word by gender or by race, by other demographic attributes, but also by background. So I think one of the biggest challenges moving forward is going to be that even though there's a huge amount of science, we don't know and understand yet. We're on the paths to developing that science. At the other end of the spectrum, there's all of the inadequacy of the current laws and regulations and higher level societal ways of thinking about these topics. And for real progress to be made on these topics, it's going to require a marriage of those two things. So scientists and engineers are going to have to get used to talking to regulators, legal scholars, and the like and vice versa. And a common language is going to have to be developed between those two communities that simply doesn't exist right now. In some ways, the science is the easier part of this, right? Because the science is developed by scientists working with and writing for other scientists. But the hard part is kind of navigating that middle ground between the kind of fuzzier side of things that's necessarily fuzzier, like laws and regulations, because you can't have laws that are overly specific because they'll just get too complicated. And that's why we have courts to mediate that ambiguity. But on the other hand, machine learning AI and algorithms are incredibly precise artifacts engineered and designed by humans. And some kind of language has to develop between these two ways of thinking that lets us kind of make progress not just technically, but sort of a broader society and in a business context as well. Michael, thanks so much for spending time with us today describing this very important emerging issue and perhaps one that's relatively speaking neglected in the boardroom. We've been talking about Michael Kahn's latest book, The Ethical Algorithm. The science of socially aware algorithm design which is published by Oxford University Press. Thanks again, Michael. Martin, thanks for having me. Very much enjoyed it.