Game Theory for Model Interpretability: Shapley Values

Hey Katie. Hey Ben. Today we're talking about shapley values and I don't know what they are. I can't pun off of it. And so I'm hoping maybe you can tell me what they are in this episode. And then at the end, if any punsters are listening who have puns that I could have used, well, they can send, they can send an email to Bennett linear digressions like them. Sounds great. Okay, cool. Wonderful. I'm off the hook today. You were listening to linear digressions. So what is a shapley value? And like, why are we talking about it? Good question. So shapley value, I'm actually going to postpone that for a minute, because I think it makes a little bit more sense once you give the machine learning context. And since this is a machine learning podcast, that's where we'll place a start. And one of the things that can be really hard about machine learning is that models, especially models that tend to have the highest predictive accuracy tend to be pretty black boxy and hard to understand. So in other words, they'll give you an answer, but they won't tell you why it got that answer. And especially in order to establish trust in a model in order to understand the underlying processes. Maybe that's what you're trying to do. Maybe you want to have insights into how to improve the model. There's all these reasons why you might want a model that's interpretable. And so the fact that the really good models are hard to interpret is a problem. Yeah, we have that this seems like it's kind of a running theme. We've talked about this a couple of times or a couple of different methods of interpreting why different algorithms give the results that they give. I'd never thought about it in terms of trust. That makes a lot of sense because you can have an algorithm that's really good on the things that you're training it on. But maybe when generalized, there are some corner cases that might give you really, really bad values or bad results. And so if you have some idea of what the model is actually doing, you might be able to preempt those bad results or maybe put in certain protections or find some things to train it on that that would help keep the negative pieces down, I suppose. I think it's even a little more basic than that, although those are really good points. And this is flavored a little bit by the work that I do kind of my day job, which is I work at a company that does a lot of data science working with other companies because there are clients we do like consulting work and technology work and things like that. And so what that means is that very often when we go and work with these companies, there might be some folks in the client organizations that understand data science or they have some interest in doing it. But most people don't, right? And so very often in order to get a data science initiative to succeed, you need to get people who don't really understand these models to be using them to make decisions. And so for someone like me who has a lot of technical understanding of how the models work, I'm very comfortable with that because you know, it doesn't feel it doesn't feel scary to me. But if you're someone who's never worked with these methods before and now all of a sudden there's a computer that's telling you what to do, having something that can help you understand that a little bit better is you know, that level of trust is also pretty important in order for stuff to actually succeed when you try to roll it out. So not just not just the person doing the model building and whatnot, but the people also who might be relying on the results of the model. Yep, yep. And anybody else, you know, having trust in a model is a that's a good thing. And like you said, a lot of times if we have a simpler model, then the model itself can be kind of self-explanatory. So a good example of this is linear models where you can just read off from the coefficients of the model itself, kind of how each feature impacts the final prediction that you make. But for more complex models, it can be a lot more complicated. So if you're thinking of something like a neural network or a random forest where you have a whole bunch of decision trees that are all making decisions kind of in parallel and then aggregating up those those results that's just a lot to hold in your head at once. And so what you want to do in those cases is usually come up with some kind of simplified model or simplified inputs that approximate the real model in the region that you're interested in, which is like the region near a case that you want to predict. And then that simplified model is easier to understand and can kind of give you an approximation. And so that's the general idea behind a lot of the model interpretation tools that are out on the market right now. So we've done one episode at least about this before with an algorithm called Lime. And I'll dig up a link to the old episode if any of you have started listening more recently than in the last year or two, which whenever we covered that. But there's a whole bunch of these. There's Lime, there's Deep Lift, Layerwise, Relevance Propagation, Shapley values. We're going to talk about Shapley values today. But the general idea is that there's a whole bunch of different options that you have about ways that you can build out some of these simpler models that allow you to do the interpretation a little bit better. That sounds awesome. That makes a lot of sense. Awesome. That context helps a lot with kind of understanding the the problem and why we're doing this in the first place. You just mentioned a bunch of algorithms. How do you know which one to use? I guess like are they all just different means to the same end and you just try all of them and see which one does does the best or is there some other way to decide? Well that can lead you into trouble because sometimes they give different answers. And so then you're like oh well so what makes something like a good explanatory algorithm. And I'm following right now a paper called a unified approach to interpreting model predictions. We will have a link to this on the website per usual. But they have a they cite in this paper a few different attributes that a good feature feature importance or feature attribution method should have. So the first is what they call local accuracy which means if you have a simplified model that's approximating a more complex model in the neighborhood of a particular point that you want to make a prediction about that if you put the same inputs into the simplified model as you do into the original model you should end up with the same answer. So the idea is that it gives you if you have a model that's easy to interpret but it gives you the wrong answer like it's not a good approximation then. Right. That's a violation of local accuracy. So it should give you the same approximate answer as the model that you're trying to interpret. So that's one. The second is a property called missingness and this one's kind of interesting because sometimes in different cases you end up in a situation where there's features that are missing for different cases that you want to interpret. So a good example of this is maybe if you're doing text classification. So in text classification usually what you're doing is putting in a bunch of words into some kind of complex algorithm. Now what if there's a word that's missing in a particular case. Well depending on how you structure the inputs that could be a problem but the general idea is that if there's a feature that's missing then it shouldn't have any impact on the answer that you get or it shouldn't have it shouldn't be showing up in the feature attribution that you have. So I'm not doing a very good job of explaining this. Let me let me try with an example. Let's suppose you have a decision tree and the decision tree is trying to figure out if somebody has pneumonia or not. So you have a list of symptoms and then you build your little binary tree and you go down the leaves and tells you whether you think you have pneumonia or not. Now let's say that the features that you have in general in your data set are whether you have a cough whether you have a fever and whether your toe hurts. Now you might build a model that depends on cough and fever but doesn't have anything about whether your toe hurts so whether your toe hurts is a missing feature. Well in that case if you're trying to make an explanation of whether somebody has pneumonia or not some prediction that prediction should not have anything about whether your toe hurts or not because that's irrelevant in the original model so it shouldn't be showing up in the explanation. So again not super complicated stuff but just things that are well we'll get to why this is tricky in just a second but let me come to the third one which is consistency which is where if a feature is increasing in its contribution to the outcome then that feature is increasing in importance. So again this is fairly simple but it's the idea that if there's some relationship where adding more of a feature to a case so let's say let's say again with the with the pneumonia case let's say there's some kind of relationship where the higher your fever is the more likely it is that you have pneumonia. Well in that case then increasing the value of if you're trying to predict for any individual person does this person have pneumonia or not and you're going to take that case and dial up the number that you put in for someone's temperature so increasing the fever. Then that should be an increasing share of the explanation that you get relative to if they didn't have a fever at all. I see. I see. Alright so yeah there's a proportionality to the algorithm that you want explaining the rationalization behind the the outcome. You want that to have that proportionality to have that understanding of the relationship of the inputs to the output. Yeah. Yeah. So it seems kind of simple. Right. Yeah. So these are all kind of ways of encoding or intuition I think in a sense but it's a useful exercise because what ends up happening is that for a lot of these methods you can come up with pathological examples or sometimes they're not even that pathological but just areas of phase space where you can't get all three of those at the same time which is kind of strange. Except in the case of Shapley values. So Shapley values we're going to explain those in just a second but there these quantities that come to us from Game Theory which is sort of a subfield of economics originally but the idea is that a Shapley value gives you a unique solution that satisfies all three of those criteria at the same time. So it's the only it's the only solution that does that and some of those other cases are kind of subvalues or like particular subcases of Shapley values. So the argument of this paper like I said the title of the paper is a unified approach to interpreting model predictions. The argument here is that Shapley values maybe are a way of uniting a bunch of other different feature attribution methods or in cases where they aren't uniting them kind of making the implicit argument that Shapley values have these particularly nice properties of local accuracy, missingness and consistency at the same time and kind of like the Shapley values are good ones. You know the best option that you can take. I will let the listener decide whether you think you agree with that assessment or not. Not here to pass judgment but it's a it's a pretty bold and interesting claim. So it's worth talking about. Okay so I guess I guess my next question is what so what are they like we're talking about them but yeah. Fair question. Yeah so I hope so. I hope so and an episode dedicated to Shapley values and so. Yeah so let me explain them kind of in the game theory context first and then okay the analogy to model interpretation I think is actually pretty straightforward. Yeah this is good so the context of the problem in machine learning. We'll talk about Shapley values with respect to game theory and then we'll bring that back to the original problem that we set up. Yeah yeah so the original problem that they were trying to solve in game theory is that sometimes in game theory you're have a situation where there's different people who are cooperating with each other to achieve an outcome. So a good example of this might be the composition of a soccer team. So a soccer team has 11 people on it and there's usually a pretty well formulated set of different positions that each player plays and so then if you have a soccer team that's really good one of the questions that you might have is how important is each one of those individual 11 players to the overall performance of the team and it's a little bit complicated because there's just one outcome that you see but there's 11 people who are contributing to it. Right so if you have a game where the other team almost scores a winning goal and the goalie the goalkeeper I don't know. Yeah you can you can tell I'm a big sports fan and the goal defender person blocks it then you would say oh that the goalie was really really important in that particular game. That's true we're not quite deconstructing it on that level so let me let me pose it to you this way it's a little bit of a strange thing but let's imagine that you had you were running an experiment where you had a whole stadium full of soccer players you have thousands of soccer players and you're pulling out different subgroups of soccer players and composing teams of them and so sometimes you have a team that's just one person and then that's probably not a very good soccer team and so in general they're not going to and let's suppose that instead of it being in discrete goals so usually like soccer games they end up being scores of like two to one let's imagine it's something it's a little bit more continuous so let's say a typical score might be between zero and a hundred so if you only have one player on the team then they're probably not usually doing that well because you only have the contribution of one person so maybe they're really good maybe it's a they score from zero to a hundred they score a ten but you know if you have if you were to have two of those people then maybe you have one who's really good and one who's like also pretty good so one of them who's a ten by themselves and let's say a second one who's a nine by themselves but maybe when you put them together because they can pass it back and forth instead of it just being ten plus nine maybe they get like 21 or 22 because there's gains to having them both in there together so the idea is that as you're adding in people maybe there's they're not just adding up kind of linearly but there's certain combinations of them that are more or less yeah impact so and then maybe you had a third person who's a five by themselves but they play like really really well with the first two so you know they're not adding on five they're adding on you know another ten when they join with the combination of the first two they're already and then let's imagine that there's a fourth one and that fourth one gets in fights with the first two and so that for the fourth one might be a seven by themselves but when you add them in they're actually just a two because they kind of like make it worse in some cases and then let's imagine that you add in a fifth one and it's a goalie and so oh my gosh all of a sudden you have this position that's totally different from any type of position that you have before so this isn't a person who's passing or anything like that but they're playing a really unique position on the field well that's that might be very very valuable and so you add that in but now let's say you add in a sixth person and they're also a goalie well okay but you already have a goalie so that person if they were the first goalie you added in that would be a really big deal but because they're the second goalie it's like not as high impact so the idea is you know you're building up this team you're trying to understand the coalition that you're creating here but it depends on the different combinations of people and the different the order even in which they're entering the game so like I said the first goalie that you get is going to be really valuable but maybe the second goalie is like doesn't add anything and so in order to understand how the impact of you the composition of the overall team how that reward can be attributed to each of the individual players becomes a much more complex problem because it's like what depends on who else is already there it depends on these non-linear interactions between all of the all of the different ingredients and just the the credit attribution problem gets really really hard I see what you're saying yeah that's yeah that is complicated it it almost feels also like like workplace dynamics when you're working with like a project manager and you're working with a designer and you're working with a couple developers and and everyone is kind of playing these different roles and if you have two managers that second manager is not going to add all that much necessarily or might even detract well as a manager you can never have too many but now I'm just kidding I forgot you're a manager yeah I should be careful what I say so anyway and so the question is is there some way that we can kind of summarize but at the end of the day maybe you have certain players who just are really really good or they're really really high impact and they just have higher values than the other ones when you kind of imagine you were to add up every different combination and every different ordering of players that you could compose here there might be people who consistently are higher impact or lower impact and so there should be some kind of calculation that you can make looking at all of those different scenarios that summarizes the credit that that person deserves overall and that is what a Shackley value is so what it the way you calculate Shackley value is you look at all the different combinations and of coalitions that you could have so coalition is just like the teams that you can combine yeah and you have to try out all the different you know all the different orderings because like we said depending on who else is in the coalition already somebody might have a bigger or smaller impact you have to you know kind of do like this very large multiplication problem and so for each one of those different scenarios that you construct you pay attention to when I add in a given actor to this coalition what changes in the overall credit that the team gets and then kind of add all of those different contributions up for each player you divide by the number of scenarios that you have and then that's the Shackley value that is associated with that person so it's a fairly straightforward thing to calculate it's just gets really that's an interesting concept so like I hadn't thought of that before you know like when you think about the value that someone brings to to a team of any sort or to a group of any sort at least as a as a human being my brain automatically thinks okay what what does this individual or what does this player or actor bring to this situation in the best case or in the worst case like I I have this bias that I enter that calculation with but this is specifically just talking about I guess it's just averaging every single possible contribution that the actor or player or person could have in every single situation just averaging all of that out and then and then you can kind of imagine okay if you add this person to some arbitrary situation they're gonna have an average impact of whatever yeah yeah that's the exact idea and so it's it's a fairly straightforward process I'll attach a link there's a pretty good explanation that I found that's talking about how to do this for a marketing application but it kind of walks through a scenario that does one of these calculations in a simple way so if you're really interested in the details here or don't believe me you can kind of go through the exercise but the main point that I'm trying to make is that the main thing that makes this difficult is just for situations where you have potentially lots of different compositions of coalitions then you have to go through the exercise of knowing what each of those credit assignments that you get is and doing all the just sort of doing all the math anyway here's the here's the tieback though so that's what a shoply value is is it's that like average over all the scenarios of how much an actor adds so now instead of composing a team of soccer team of players instead all of our players that the outcome that we're interested in is not how many soccer points did you score the outcome that are interested in is what's a prediction that a machine learning model is making on an individual case and the players instead of being soccer players are all of the features that could be going into making that prediction and so as we turn features on and off how are the outcomes changing and what does that tell us about the importance of that feature to the model so tell me if I got this right I might not but you have a complex model you want to make a simpler model so you can more easily understand what things are important and so you figure out you calculate the shoply value or the average impact in all of the configurations for each of the features that are in your complex model and then based on that you decide which ones to turn off for your simple model I'm not sure if it's turn off exactly but I was with you until yeah until like that last little bit but it's basically like that shoply value tells you how important a given feature is for a given prediction and then that helps you with the interpretation oh I oh I see I see I see so this is about this is not about creating a simpler model necessarily but about understanding how the prediction was made by looking at the shoply values of the features to figure out which ones were important yeah so and it's that it's not what it's not doing is deconstructing the model itself so for example if you have a neural network model what a shoply value is doing is it's not telling you anything deep and profound about the structure of the neural net right but what it's doing is it's kind of like turning features on and off or like adding them in in different power set combinations looking at the outputs that you get and then waiting all of those different scenarios against each other to understand the impact of the feature we're now starting to get into some of the stuff that I wanted to cover in the next episode a little bit and we've been going for a while yeah so I think I think maybe here's where I call it but to answer your question from the top to answer the question from the top I think the scenario that we have here for your pun question is that usually what happens with the pun scenario is that I come with some idea of what we're going to talk about and you come in with some some pun skills and then the thing that happens is when the two so each of us alone is not enough to make a machine learning pun no sometimes when we're together there's that like interaction effect from the coalition that we form and then occasionally we have good puns that's as far as we both have though so we both have mediocre shoply values and it's kind of interesting because well I'm now I'm really getting into the next episode stuff but you can like you can do shoply values for the interaction terms between features which is kind of cool what oh man oh yeah we got to stop yeah we got to say next week yeah next episode okay cool all right well so bear with us for a week as we're signing off now but to be continued a little bit more about the the guts of how sometimes the stuff is used for machine learning linear digressions is a creative commons endeavor which means you can share or use it any way you like just tell them we said hi to find out more about this or any other episode of linear digressions go to linear digressions dot com and if you like this podcast go ahead and leave us a review on iTunes so other people get to listen to this content too you can always get in touch with either of us our emails are ben at linear digressions dot com and Katie at linear digressions dot com in case you have comments or suggestions for future shows you can tweet us at Lynn digressions thank you for joining us and we'll see you next time