38. Should we be using AI to predict patient preferences? With Nicholas Makins

Ethics Untangled is made by the Idea Centre at the University of Leeds. It's a podcast about the difficult ethical questions we face in everyday life. In each episode, I interview someone, usually a philosopher, who's spent time thinking through one of these questions, and we try between us to work out what's going on. This episode is part of what's becoming a bit of an informal series of ethics untangled episodes on ethical issues relating to artificial intelligence applications. The particular application we're looking at this time comes from a healthcare setting and is called a patient preference predictor. It's a proposed way of using an algorithmic system to predict what a patient's preferences would be concerning their healthcare, in situations where they're incapacitated and unable to tell us what their preferences are. These assists have raised concerns about these systems, and these concerns are worth taking seriously. But Dr Nick Makins, postdoctoral research fellow in philosophy at the University of Leeds, thinks they can be answered, and that the use of these systems can be justified, at least in some circumstances. Nick Makins, welcome to the podcast. Hi, Jim. Thanks very much for having me. So we're going to talk about a thing called a patient preference predictor, and the question we're going to ask, as opposed, is whether we should be using patient preference predictors in making clinical decisions about people's care, right? So just so that we can get a clear idea of what we're talking about before we get into the ethics of this, I wondered if you could just tell me what is a patient preference predictor. Okay. That's obviously a very natural and sensible first question, but of course it wouldn't be a philosophical topic if the simplest questions weren't also somewhat the hardest, almost controversial. So I guess we'll come on to exactly why there are some complications in answering that question as we go through. But there are at least some fairly uncontroversial things that we can say about what a patient preference predictor is or would be. So the basic idea is that this is some algorithm, some computer system, that would take known information about a particular patient, and on the basis of that information, make predictions about their healthcare preferences. Yes. Okay. So it's interesting that you said it is or would be there. So as I understand it, this isn't something which is currently being used, right? Or even that has been necessarily developed, we're talking about a hypothetical system that one might use in clinical settings, but not something that's currently being used. Yeah, exactly. That's right. This idea has been sort of bouncing around in the medical ethics literature for a good 10-15 years, but it's not gone beyond there so far as far as I can tell. It's very much still an idea that's on the table, but not a concrete piece of technology that's out there to be bought or implemented in hospital settings. Right. That's really interesting. So it's a kind of statistical model, which might be kind of algorithmic in nature, where it's taking kind of demographic information about the patients, and on the basis of kind of statistical truths, I suppose, about people from those demographics, it's making judgments about the likely preferences of that individual patient in terms of what care they would want to receive. That's right. So there are some reasonably well-known and fairly robust correlations between certain socio-demographic facts about people and the healthcare preferences that they express. And so those correlations are really the kind of heart of the idea here that because those correlations are fairly robust, they could perhaps be used to make predictions about the preferences of particular individuals. Yeah. And it's in settings where somebody's incapacitated and decisions have to be made about their care, right? So it's not like you can make no decision. So you're in the position where you're trying to work out what the person's preferences will be because they're not in a position to tell you directly, and whichever decision you make is going to have consequences for that person. Yeah. Exactly. Well, yeah. I mean, most of the literature has focused on these cases of patients who lack the capacity to make a particular decision for themselves. Some people actually have spoken about, perhaps even if you have a patient who has capacity to make a decision, maybe they want to know what prediction would be made about them based on information about them, and some people have suggested that might even help them kind of go through their deliberation, but yeah, the focus has certainly been on decisions that must be made for patients who lack capacity to make that decision themselves. Okay. Great. It currently happens, then, so these things aren't being used currently, but we do have these situations where people are incapacitated and decisions need to be made. So how are those decisions made at the moment? Yeah. This is quite a big question, and there's quite a lot to it. So these sorts of decisions in the UK, and I'll just speak about the UK context, things vary, of course, a little bit internationally. In the UK, these sorts of decisions are governed by the Mental Capacity Act, and that has various provisions for different ways that decisions might be made. So if somebody has designated a lasting power of attorney, then that's another individual who might be involved in making the decision on their behalf, or if they have other documents in place that make specific statements about what they would or would not want to happen to them in the red event circumstances, then those can be used. But ultimately, and usually what is called a best interest decision is made by the clinical team. The MCA, the Mental Capacity Act, includes various guidance about the sorts of considerations that should be born in mind when making a best interest decision. But that'll be made by healthcare professionals taking into account the views of the individual themselves, if they can express some things about their preferences, other people who know them in factors like that. And it's important to have some idea of what the patient's preferences are, or hypothetically would be, because otherwise you're in a position where the doctor's making a best interest decision independently of what the patient thinks about it, which is kind of paternalistic, right? And that's not what we want to see happen. Yeah, exactly. And there's a lot of philosophical debate about exactly what the relation is between an individual's preferences and their well-being. And it's their well-being that a best interest decision is supposed to promote. But on pretty much any plausible view, the preferences of the patient are really important. Even if there's not this kind of constitutive relation between the satisfaction of one's preferences and one's level of well-being, on any view they're really taken to be, to be very important. Okay. Great. And returning to the patient preference predicts it then. So you've told me about how the process occurs at the moment, but how might a patient preference predict or improve on what's done currently? Talking about it in terms of improvement is interesting, because one of the other ideas that's important in the background behind the proposal for a patient preference predictor is that there's a decent bit of evidence suggesting that people are really pretty bad at predicting the preferences of other people when it comes to healthcare. That applies to family members and next-of-kin sort of people, but also to healthcare professionals. And so it seems fairly natural to think that if a patient preference predictor could be developed that had a reasonable degree of accuracy, there's a fairly low bar to get over in terms of improving on the existing state of affairs with regards to how accurately we can predict the preferences of an individual. Yeah. Okay. Right. So it seems at least plausible that something like this would be a better predictor of what a patient's actual preferences would be than the kind of proxies that we use at the moment. Yeah. But of course we don't know that for sure, I suppose. So it's not like these things have been tested and that there are kind of studies telling us how accurate they are, but the hypothesis is that they at least could be quite accurate. Yeah. And it seems like a fairly strong hypothesis. So we don't have direct evidence because these things haven't been developed and tested. But we do have lots of other evidence that would sort of lend support to the hypothesis that they would be helpful and they would be an improvement. I mean, most of this evidence falls under sort of support for the view that the bar is, the existing bar is pretty low. Yeah. People are just not that accurate at this already. Some suggestions that surrogate decision makers are no better than predicting on the basis of just the base rate in the population of a particular preference ordering. And so that would mean we could even with a very, very simple statistical model, we could do better than surrogates. Yeah. Yeah. Okay. And so we're going to get onto the ethical questions around patient preference predictors in a moment. But so I was wondering, is the reason that these things haven't been developed and tested and aren't in use, is that because of the ethical objections that people have, or as far as you know, or are there other reasons why people aren't using them? It's really hard to say. And I guess this is a question that is going to bring in all sorts of other factors about who develops new medical technologies, what their incentives are. They're going to be sort of economic incentives in play. They might be because of these ethical debates that they've been rumbling on. There might be some reluctance to develop this kind of technology because it looks like it might be kind of mired in a regulatory mess at some point down the line and maybe that's the sort of thing that puts off people who might develop this kind of technology. It might also just be that this is a, in some ways a slightly unusual idea and it's not immediately obvious to me exactly who it would be that would develop this sort of technology. I mean often new medical technologies that involve AI are developed by companies that have a mixture of computer scientists and people with a particular, like, clinical specialism. And I don't know exactly who those people would be here because this is a very, a very general problem that crops up all across different areas of medicine. So yeah, it's hard to know exactly why this hasn't been developed, given that it's been discussed for such a long time. But there are a number of kind of ideas around about why it might not have taken off yet. Yeah. I think just in a kind of pre-theoretical way, it's something that I can imagine making people a bit uncomfortable, so I can see why people might kind of shy away from it without really being very clear on what the ethical issues that it raises are. Yeah. And that's actually one of the things I kind of wanted to do with this paper because when speaking to people, when you raise this idea, lots of people think of it in a very particular way and that the way they think of it really puts them off and makes them think that it's really, doesn't seem like a good idea. And that way of thinking about it is something that also seems to be behind some of the debates that emerge in the literature, but I don't think we have to think of things in the way that seems to put people off. And so the kind of aim of the paper is to try and reframe this idea and the discussion around it a little bit to see that actually this doesn't need to be, take the form that makes people uncomfortably on that and media reading. Great. Well, let's get into some of those ethical issues and objections then, so why might somebody object to using a patient preference predictor? Yeah. There are plenty of arguments out there actually. There's been quite a lot of literature on this, but in the paper I focus on three objections in particular that have been raised. The first is essentially an argument from analogy with a problem that's well known in a legal scholarship. So in the legal context, there's this idea that legal decisions should not be made on the basis of purely statistical evidence. Right. And that case is made with a very well known example. So suppose that an individual is in an accident with a bus and is injured. And they're owed some compensation by whichever bus company is responsible for operating the bus which caused the accident. Suppose this goes to court and everybody agrees about that, but it's unknown exactly which bus company was operating the bus which caused the accident. It could have been the blue bus company or the red bus company. And suppose that the only thing that's known about this question is that 80% of the buses on that route are operated by the blue bus company and 20% of the buses that travel that route are operated by the red bus company. Now most people have the intuition that that fact alone would not be able, would not be enough to settle the case against the blue bus company. So even though 80% has an 80% chance that this bus was run by the blue bus company, it seems that there's something wrong with making this legal decision on the basis of that purely statistical information. We need something else. What's been suggested in the medical context is that whatever is objectionable about relying on bare statistical evidence in legal contexts is also going to be objectionable when we rely on that kind of bare statistical evidence in a medical context. And it seems as though the patient preference predictor would lead us to make decisions in just that way. So the patient preference predictor makes its judgment on the basis of bare statistical information about people. Yeah. And so it seems as though if there's a problem with making decisions in that way in a legal context, exactly the same sort of problem is going to come up in the medical context. Yeah. It strikes me that one important difference between the bus case and the patient preference predictor case might be that in the bus case, well there's always an option of not prosecuting anybody, right? Because you say that the kind of standard of proof hasn't been reached and therefore you just aren't able to bring a prosecution. Whereas in the patient preference predictor case, a decision has to be made, right? So either way, you're going to have to make a decision. So you might say that the standard of proof could be lower there because at least, you know, you just want the best decision-making tool that you can get in order to make the best decision you can because you don't have that option of not making a decision on behalf of the patient. Yeah. So I suppose even, I mean, not making a decision in a legal context is a form of a decision in its own right. And it also seems as though in the medical context, yes, a decision has to be made, but there are different ways that we could approach that decision on the basis of different considerations. Right. So I think the people who see this as a problem might think that, yes, of course, the decision is unavoidable in the medical context, but we should just approach that decision in a different way, not using this particular type of technology. Okay. Great. All right. So that's the objection from the use of bare statistical evidence to make decisions. What other objections are there? So another one, I say one, is really a family of objections where people point out that there are some distinctive advantages to making decisions or to having particular surrogate decision makers make decisions on behalf of the patient and that those particular advantages are not secured if we use a patient preference predictor instead. So for example, people often think that there's some particular special relationship that exists between individuals and perhaps they're close family members or other people that they have these very kind of special and close relationships and that perhaps that means they can form some kind of shared agency and that thereby the agency of the individual can be preserved if decisions are made for them by those particular surrogates. Or a more sort of simple and theoretically modest view is that sometimes people have preferences about who make decisions for them. So if I would prefer to have my brother make a decision on my behalf and we're concerned with satisfying my preferences, then that's one of the preferences we should try to satisfy. Yeah. So we should have my brother make that decision on my behalf. Yeah. So there are these kinds of advantages to surrogate decision making. They're not the kind of advantages that can be captured by making a decision with a patient preference predictor instead. Right. So in that kind of case, the thought is even as a matter of fact, the family member who's acting as a surrogate is not accurately dividing what the patient's preferences would be. The very fact that the patient has nominated that person is kind of close to that person and has decided to entrust their well-being to that person. That counts for something in addition to the kind of accuracy of the predictions that they're going to make. Exactly. That's right. Yeah. Okay. Good. All right. So that's another objection. So that's something that wouldn't be possible in the case of using a patient preference predictor. And then you've got one more objection to what was the last one. Yeah. So this last objection is based on the idea that the motivations for the patient preference predictor rely heavily on respect for autonomy. It seems as though the conception of autonomy that is motivating the patient preference predictor is people, the importance of people having their preferences satisfied. Yeah. But that might not be the best way to think about autonomy. It's been suggested that instead in order to properly respect someone's autonomy, what we need to do is pay proper attention to their capacity for rational deliberation. And what that might mean, one way that we could do that is by making decisions on the basis of the kinds of reason that that person would endorse as a reason within their own deliberation. Okay. If they were to be making the decision for themselves, what kinds of consideration would they be thinking about? Right. So if we think about autonomy in that way, and we think that what matters is making decisions on the basis of reasons that would be endorsed by the patient themselves, it looks as though the patient preference predictor is kind of going about it the wrong way. That's because the sorts of considerations that feed into the patient preference predictor are usually described as socio-demographic information. So age, gender, nationality, perhaps, things like this. And it doesn't seem as though those are the kinds of considerations that people tend to make decisions on the basis of. So if I had some important healthcare decision to make, I'd be thinking about how is this treatment going to make me feel, how is the illness that it aims to treat, going to make me feel what effect will the side effects and symptoms have on my relationships, my career, not what would someone who is a 33 year old white male from London do, right? Those are the kind of socio-demographic factors that just don't seem to be the kind of reasons that I would endorse in my under liberation. Yeah, it does seem to capture something that's a bit kind of weird and queasy about the idea of using these kinds of models, right? It's just, it doesn't seem to be pointing at the right kind of thing. So yeah, I can see, I can see where that comes from. Hello, this is just a brief interruption to tell you about an event that idea the ethics centre of the University of Leeds is hosting as part of its 20th anniversary celebrations this year. We're delighted to be welcoming Gavin Esler, journalist, presenter and author, to present a public lecture, dead cats, strategic lying and truth decay, on the evening of Tuesday 17th of June, at Clothall Court in Leeds. Gavin is an award-winning journalist, broadcaster and author. He's best known for presenting the BBC's flagship Newsnight program. He also spent a number of years as the BBC's chief North American correspondent, as well as reporting in other countries. He's the author of five novels and four non-fiction books commenting on politics in the modern world. Drawing on his recent book, Britain is better than this. Gavin will analyse the state of current politics and truth-telling in the public realm. He will argue that people no longer trust businesses or politicians or journalists, as much as they used to, and that this is the result of what he calls the politics of distraction. He will discuss how a loss of trust and truth decay is eating away some of the core principles of democracy in the United Kingdom, the United States and elsewhere, and will suggest how trust and truth can be reinforced. You can put your place at the event by going to our website at AHC.LEADS.AC.UK/Ethics and clicking on the link to find out about our 20th anniversary events, but also put a link in the show notes for this podcast. Thanks. Now back to the interview. I wonder if you could just clarify that a bit more, though. How might a decision be made on the basis of reasons that a patient would endorse themselves? Can you give us some examples of how it would work? I suppose if we think about a case where a decision is being made by a close family member who has been given lasting power of attorney, suppose this is a decision, so let's say we have an elderly patient living in a care home and they become unwell and some decision needs to be made about whether they should be admitted to hospital for some potentially invasive, but quite effective treatment to treat that infection, or they should remain in their care home where the treatment they get is going to be far less invasive, but it's also just not going to be as effective in treating the infection. Now somebody who knows that patient well is going to be making that choice on the basis of things like how much does this person value extending their life versus preserving the quality of life that they have currently? Are there particular events on the horizon that they really want to live to see? Or are they really just at the point where they just want to become and not have invasive medical treatment or be in a hospital environment? Those sound like good considerations to be bearing in mind when making this kind of decision, they sound like the kinds of considerations that in fact would be taken into account by someone making this kind of choice, but they're not the sorts of considerations that would factor into the patient preference predictor as it's typically conceived. Yeah, okay, great. And again, I suppose the thought has to be something like even if the patient preference predictor would do a better job of working out what decision that person would make, just the very fact that it wouldn't be taking into account the kinds of reasons that that person would be taking into account means the decisions being made in the wrong kind of way. So it's not by the outcome of the decision it's about the actual decision-making process. Exactly. That's right, it just puts pressure on this idea that accuracy when predicting what people should want is the sole concern that matters. Yeah, good, okay. So we've got three objections, so we've got the objection based on bare statistical evidence with the bus example, we've got the objection based on the idea that somebody might appoint a surrogate because they're involved in a kind of shared agency with that person, so it's someone they're close to, and they've entrusted that person with their wellbeing, something like that. And then you've got the objection based on the fact that patient preference predictor would not use, would not appeal to reasons that would be endorsed by the patient themselves in making the decision. So we've got those three objections. So your view is that those objections essentially fail because they don't distinguish between the use of algorithms as predictive models and decision functions. Is that right? That's exactly right, yeah. Yes, okay, so I wondered if you could explain that distinction first between predictive models and decision functions. Yeah, so this distinction I think is something that's fairly natural and easy to understand in a very general sense. I mean, here are two different things we can do. We can make predictions about what's going to happen or about some way the world is, and we can make decisions where we select an option to be pursued. These are clearly different activities. These are also things which algorithms can be used to do. So we can employ an algorithm or any kind of computer program to make some prediction about what's going to happen, or we can use it to make a particular decision, where what that involves is selecting a particular outcome that is the outcome to be pursued. Yes, yes, and how might that work in the case of a patient preference predictor then? Yes, so if what we have is a basic statistical model, which takes the possible preferences that an individual might have and sort of assigns a probability to each. So it just says this person is 70% likely to prefer treatment A to treatment B and 30% likely to prefer treatment B to treatment A, then that's just making a prediction. It's a probabilistic prediction, but it's just making a prediction. Alternatively, we might have it do that plus something else, so that something else would be a decision function, a decision rule, and what that might be something like. Pursue which ever option is most likely to be preferred. So then the output of that system would be the identification of a particular option on the basis of this sort of probabilistic prediction and the decision rule that's built into it, and so it just bits out one option and just says do this. This kind of distinction, we can think of this as a distinction that exists within different types of computer algorithms, but it also just exists within lots of other sorts of tools and aids which are used in medicine already. So there are lots of heuristics and sort of flow charts or things that you can put data into that will spit out some prediction at the end about the risk that somebody for example has a heart attack in the next five years or something like that, but there are also clinical guidelines that doctors sometimes follow which have as their output particular recommendations for which treatments should be offered. And so this sort of distinction between making predictions and making decisions is a very general one. OK, so if you use the patient preference predict as a predictive model rather than as a distinct decision function, the decision about the ultimate care would still be made by perhaps by clinicians or perhaps by clinicians with family members and things like that, right? So they would be presented with a model whose output would essentially say, you know, there's this percentage likelihood that this person or a person with these demographic features would prefer X, Y and Z decisions to be made, but then those decisions would actually be made by the people. That's exactly right. And of course, there are then big questions to be asked about how exactly those decisions are made. There were also interesting questions I think to be asked about how that prediction is understood what we should take the probability to actually tell us. And those are all other debates that are going to have to be had, but the basic simple point is that we need to think of these systems as making decisions themselves. We can just think of them as providing us with some evidence which can feed into the decision much like all sorts of other kinds of evidence that we want to factor into these sorts of choices. Yeah, so it strikes me that one thing that might be important here is how transparent or explainable the algorithm is, right? So if it's just giving you those percentages and you've got no idea how it's reached those percentages, then that might be somewhat useful, but it's going to be much more useful if it can say why it's reached those decisions. Because then you can say, oh, I say, okay, it's because this person lives in the north of England or whatever. But actually, I happen to know that this person is kind of unusual and therefore I don't think they would follow the trends for people from that demographic group and so on. So actually having some kind of explanation behind the statistics might make it more useful. So I'm not actually committed to any particular view about what kind of transparency or explainability is needed in these sorts of systems. My view, rather, is like these are the sorts of questions we should be asking. So we shouldn't get distracted by objections which think of the patient preference predictor in one particular way, which is not the way that they need to be used. We should be focusing on these sorts of questions and they're good questions, they're important questions and ones that we need answers to and I think they're the right kind and really my aim here is to kind of shift our attention a little bit towards these sorts of questions which I think are the right ones. Yeah, okay, great. All right, so maybe we could go back through those objections then and just say why you think those objections lose their force if we start to think about the patient preference predictor as a predictive model rather than a decision function. The first one was an objection to making decisions based on bare statistical evidence. Yeah, so to see how we can avoid this objection, it's worth just returning to the legal context again and thinking a little more detail about exactly how the arguments work there. So early on in some of the debates about the role of bare statistical evidence, it did seem as though people were making a quite a strong claim that these kind of evidence should not be admissible at all. So where amissability is just a question of what kinds of evidence are we allowed to consider what kinds of evidence should be presented to a jury or a judge in which sorts of information should not. But actually the debate has moved on from that and there's now more focused on a question of which types of evidence are sufficient for settling a legal question, whether that is just to say if you only had this piece of information, would that be enough to make a decision one way or the other? That shift has largely happened because there are lots of forms of bare statistical evidence that it just would seem ridiculous to rule out as completely inadmissible. So for example, lots of types of scientific evidence are produced by statistical analysis and are, in some sense, bare statistical evidence. So DNA evidence, for example, right. And it just doesn't seem right to rule out all evidence of that form entirely as inadmissible. The better question seems to be should these kinds of evidence be considered sufficient for making a decision on their own or not? Yes. OK. So that's the legal context. If that's the right way of thinking about the question of bare statistical evidence in the law, then it seems as though, and the analogy holds between medical contexts and legal ones, it seems as though the most we should conclude is that this type of bare statistical evidence that we get from a patient preference predictor should be admissible, but just not sufficient on its own to settle medical choices. OK. So with what I've suggested that we think of the patient preference predictor is just providing one source of evidence, we can clearly see that we might factor in lots of other pieces of evidence that we have. That could be further evidence from next of care, or people who know the patient particularly well. And if we're not using the patient preference predictor on its own to make a decision with no other input, then this question of sufficiency is kind of avoided, because we're not in a situation where we do need to rely on this as the only information that's feeding into the joy. OK. Great. And then the second objection was that there is a value to a surrogate making a decision on the behalf of a patient, because those people are in a position where they have a kind of shared agency, or they've been entrusted to make those decisions. So I suppose it's reasonably clear, right? How think you about it as a predictive model helps defeat that objection, because the people would still be making the decisions, right? So there still would be satisfying that kind of shared agency. Exactly. This is the simplest one to respond to, because, of course, having a piece of evidence doesn't constrain who can make a choice one way or the other. And so if there's something particularly good about a particular person making that choice, then great. They can make that choice. They might well take this other piece of evidence into account, but that's fine. We can still secure all of the advantages of them making the choice, if those are the things that matter. Great. OK. And then the final one was to do with the reasons that a plate being taken into account and the requirement that those should be reasons that the patient would themselves endorse. So how does the, yeah, how does the predictive model defeat that objection? So there are a couple of thoughts here. The first is that I think it seems at least plausible that using a patient preference predictor might in fact help people to identify the reasons that an individual would endorse. So it can be very hard to adopt the perspective of another person and think about what preferences they would have and what their reasons for those preferences would be. That's probably part of why the evidence suggests that people are so bad at predicting and so inaccurate at predicting the preferences of others. And it might well be that once you have some prediction about what preferences this individual would have. You can use the further information you have about that person to make a kind of inference to the best explanation. So thinking about, OK, well, what reasons are there that I think this person would genuinely care about and that would lead them to have this particular preference ordering? Of course, there's no guarantee that once we have this additional piece of evidence, we're going to sort of infallibly make choices that are in line with their preferences and made on the basis of reasons that they would endorse. But once we think of the patient preference predictor as just providing one piece of evidence, it's hard to see why that should make it harder to identify the reasons that they would endorse in their decision making. Yeah. Yes, I can see that. I can see how it's hard to see how it would make it harder. It's just another piece of evidence and it can be weighed against the other evidence that the person's using to make that decision strikes me that like somebody might object anyway, there might just be a kind of discomfort with being told, well, here's a statistical model, that can tell you something about what your loved ones preference would be that you don't already know when, you know, it's just kind of taking into account these very general demographics. I can imagine like a surrogate being presented with that and thinking, well, that's not helpful to me and it's kind of offensive to suggest it. Yeah. Does that sound right? Yeah, I think there were a couple of things here that it's important to bear in mind. So one is the way that this kind of evidence is presented to somebody is going to be important in determining how they receive it and whether it is indeed offensive. I mean, if someone walks up to you and says, I know you're the surrogate and you've lived with them for 40 years, but don't you know that statistical models are far better than you and you don't really know what you're talking about? Okay, they're going to be offended. That sounds bad. Yeah. But there might well be ways of communicating this, that acknowledge the particular role that is played by a surrogate that perhaps even emphasises what's special about the role that they hold and still saying that he has some evidence which might help you. It's also worth noting here that one of the other central motivations for developing this sort of system is that surrogates regularly report a lot of distress at making decisions on behalf of their nearest and dearest and that a large portion of that distress comes from the uncertainty, just not knowing what this person would prefer. So again, it's plausible that if presented and communicated in the right way, in fact, this sort of system might help. I mean, there might be appetite for this. Yes. Yes. Okay, I can see that. The last thing to note here as well is that developing this kind of technology, looking at how it might be used, doesn't mean we need to ram it down anybody's throats and just as with other forms of medical evidence, people have a choice about what sorts of tests are carried out on them, what sorts of information they're provided with. And they can say, "Look, don't tell me about that, I don't want to know." And it seems as though those kinds of general considerations would be reasonable to play a role here as well. Okay. Good. And then the other thing I was wondering, I suppose, is a kind of converse worry from the one that we're just talked about, which would be sometimes one of the ethical issues that's raised by kind of technical solutions, generally is that there can be an over-reliance on the solution, right? So I was wondering, should we be worried maybe that if we introduce these things, then people would kind of defer to them and might just kind of automatically think, "Oh, well, if the model says that, then that must be right." And therefore, it becomes kind of de facto a decision-making model. Yeah. I think this is a good worry and a real worry. There's already some evidence that something like this is playing out in some legal contexts where when people are given predictions about reoffending rates, that just automatically sort of, they don't then take that into account and make this kind of richer decision, but that their decision-making just falls in line with whatever they're told. So I do think this is a real problem and one that should be born in mind. That said, exactly how this plays out and what effect it has on decision-makers is something that we can only really know about once we start looking at it. This is not a question we can settle. Just by doing abstract philosophy, we need to actually look at what kind of effect this sort of technology has on the decisions that people make. And if it starts to have negative effects on their decision-making, if it does just start to become this automatic process, then that might be a reason either to cateil its use or to think about ways of changing its implementation in order to try and bring back in this kind of more considered decision-making and not just let it become a defacto decision-maker. The last thought in response to this challenge is that the way I've described it, it's making a probabilistic prediction. So it's just giving us likelihoods for different patterns of preference. And so it's not even immediately clear what an automatic decision would be like. Right. There are some cases if there are two options and the algorithm says 90% chance that this patient prefers option A over option B. Okay, it's easy to see how that could become automatic and just dictate the decision. But if we have more options and the probabilities are smaller, because they're spread between more options, I mean, probabilities just don't determine choices in any context. The way that we balance up probabilities and the possible values of the outcomes that might be resulted is a complicated thing. And so it's just not the case that there is a defacto decision that is recommended once you have a probabilistic prediction. Yes. But I wonder if that kind of raises another possible worry though, which is that people just generally aren't very good at understanding probability, right? Being presented with potentially quite complex probabilistic information, people might just make decisions badly on the basis of that or they might not know how to interpret it and so on. I think this is a really interesting question and is kind of exactly the purpose of the paper, is to bring attention to these kinds of questions about, right, if we do have some probabilistic information about the different preference orderings that an individual might have, what should we do? How should we deal with them? Should we have a kind of idealized decision theoretic model or a decision theoretic approach, or does that assume a kind of probabilistic reasoning that people are just so bad at doing that we should stray away from that kind of approach? There are questions about how to communicate this kind of information and exactly what types of statistics to give people, because people are better and worse at interpreting and using different kinds of probabilistic information, and I don't have a full set of answers to all of these questions. But again, these are the questions we should be asking. Not just does this kind of thing get the wrong sense of autonomy to core, rather if we were to use it, how should we use it and exactly what are the constraints on that kind of decision making? Yeah. Okay. But kind of all tools, I mean, with all the caveats that you've expressed and the course of this conversation, do you think that we should be, well, either using patient preference predictors or at least kind of investigating them and seriously thinking about how we might use them? Yeah. Well, I think those two options are clearly very different, so I don't think this is something that should just be immediately rolled out across the NHS and everybody should be following them. But I do certainly think that the important questions about whether this kind of system is any good or not, our empirical ones, well, they're a mixture, but they require more information about how these things actually work and what effect they'll have on actual decision making. And so I think it is really important that we start investigating it and taking it seriously, and that extends a little beyond just the kind of pure philosophy that we can do. We actually need to start looking at how these kinds of systems work in practice. Okay. Great. Well, that's all we've got time for. Nick Makins. Thank you very much for being on the podcast. If you enjoyed this episode of Ethics Untangled, please like and subscribe and recommend us to your friends. The podcast is edited by Mark Smith at Leeds Media Services and our music is by Kate Wood. Ethics Untangled is brought to you by the Idea Centre, a specialist unit for teaching, research, training and consultancy in applied ethics at the University of Leeds. We offer master's programs in healthcare ethics and applied in professional ethics and PhDs, available full or part time and face-to-face or by online learning. We also offer consultancy and training services for business and the professions, including help with the difficult ethical issues faced by your organisation, commissioned research projects and online or face-to-face CPD. If you'd like to find out more about these or any of our other activities, please visit our website at ahc.leeds.ac.uk/ethics