Thomas Frost on Clinical RL with Natural Timings

Talk rl podcast is all reinforcement learning all the time featuring brilliant guests both research and applied join the conversation on Twitter at talk rl podcast. I'm your host Robin Chohan. Dr. Thomas Frost is an emergency physician based in the London UK. He's also in the final stages of completing a PhD at the University College London where he has been looking at offline reinforcement learning applied to healthcare settings. Welcome Dr. Thomas Frost. Hello Robin, thanks for having me. So I've had the pleasure of getting to know you a little bit at the past two rl conferences and I've been able to see your PhD develop and follow part of your works. I'm super excited to talk today. Also it was an area I've found fascinating for years clinical rl. So I can't wait to get into it. Can you can you give us kind of a high level view of of your thesis? Yeah, absolutely. I mean first thanks for having me on. I've really enjoyed our conversations about kind of like you say healthcare rl where my work fits within that. I think if I had to summarize my research, I would say that I'm interested in reinforcement learning applied to sort of clinical decision making. So are there aspects of clinical care that could be done better through an rl framework rather than sort of current clinical practice. And within that I'm sort of interested in offline rl. So learning from exclusively historical data sets and the sort of topic of my thesis is how much processing we should or shouldn't do to that data before we try and train rl agents on it. So I'm really interested in sort of where timings and decisions fit within that. And that would sort of be sort of the overall theme throughout the thesis, I think. So you are a medical doctor and you worked in emergency rooms for I understand a long time and you left that to do this work. Can you can you talk about that decision? What led you to to that path? Yeah, so I've been a doctor for about eight going on to nine years now, initially qualified practice full time, worked through kind of a COVID and coming through the other end of COVID, I think I was a little bit burnt out as a lot of doctors were. I'd already been quite interested in sort of machine learning and in particular rl by that point. I the thing that kind of got me interested was the really the alpha zero work. I was I remember being so excited by this idea of something that could learn to make better decisions in a way that was free of human bias. So if this kind of like self play, you only know the rules of of chess, figuring out a good way of of playing games through that without any of our sort of human based historical biases about how one should or should not play chess. And I remember thinking that would be such an exciting thing to apply to health care because so much of clinical practice is unknown and so much is kind of hereditary knowledge that sort of passed down from one generation to the to the next, but obviously a lot of biases come along with that. Anyway, so then post COVID, I was sort of looking to do a side step and have a change of pace to something a bit different. And I came across this PhD program at UCL in London, which is around base. And in the UK, we have these things called CDTs centers for doctoral training. And what's nice about CDTs is essentially you have guaranteed funding for sort of three to four years over a very broad research team. So for example, the theme of my CDT is AI in health care. And I mean, you're very free to lead the research in the direction that you think it should go as you are doing the research. So you're not really bound to a particular funding call or anything like that. And that worked out really well for me because I sort of went into it, played around with a couple of other themes like temporal clustering of sort of COVID ventilated, ventilated patients of COVID. And then about a year and I sort of remembered that the thing that had got me interested in the first place was our rel. And this belief that there are, and saying this very much as a clinician, there are aspects of clinical decision making that are certainly not optimal. We're sort of the best available option, but we're not really practicing. It's not really a solved problem at how we practice. And so yeah, I've just been really interested in exploring which aspects of that decision making maybe we could improve through sort of autonomous AI. So for the listeners who might not join us for the whole episode, can you give us a preview of some of your main findings in this work, both on the RL side and the clinical side? Yeah, absolutely. So from the clinical side, I mean, I think I've got some results that I'm quite excited about that. Essentially, when you're, if you look at sort of an intensive care setting, so really the sickest of the sick, one in six intensive care patients won't survive their admissions. So these are really high mortality patients. A lot of those patients will get drug infusions. And a lot of those infusions, there's no one fixed dose that we give to all patients because everyone is a little bit different physiologically speaking. So instead, what happens is we often need to continuously titrate or adjust those drug doses depending on each person's response. So a really simple example of that would be insulin and blood sugar. And critically sick patients often have quite high blood sugar. And we know that if we do nothing about that, it can lead to worse outcomes for them in the long term. So we often give those patients insulin infusions that then get adjusted based on what their blood sugar is. But sort of the catch is that in terms of clinical practice, what we're doing is we're adjusting for short-term biomarkers. So we're really trying to keep blood sugar in some pre-specified optimal range, but with some air quotes around the word optimal. And the reason there are air quotes around the word optimal is that we don't really know what is optimal for different cohorts of patients. And patients themselves don't care about blood sugar as an outcome. It's not a patient-relevant outcome. Patients aren't thankful that you kept their blood sugar at eight. Well, they care about as not dying, getting discharged sooner rather than later, getting fewer infections, etc. So I think what's exciting about my work is this idea of being able to train a reinforcement learning algorithm to optimize drug doses, specifically insulin doses, for those long-term outcomes directly. So being able to say, you know, here's Robin in our intensive care unit. And right now, giving him this dose of insulin will actually optimize his long-term survival, rather than simply trying to optimize a short-term number or measurement. And so my early results suggest that we have successfully done that, that we've identified cohorts of patients where certain changes to practice would improve survival. So from the clinical side, I think that's really exciting. And then from the RL side, I think I really spend a lot of work in my thesis talking about the state of the sort of current healthcare literature. There's a lot of challenges when you're doing healthcare-based RL because the data is really, it's very chaotic, it's sparse, it's regularly sampled. And one of the sort of quick and dirty ways of dealing with that is to pre-process that data very aggressively into sort of a tidier, neater, simpler version. And then do offline RL on that. And I think probably one of the biggest findings for my thesis is showing that when you train RL models on data that's been processed like that, you're essentially training them on a fictional version of reality, which means that then when you deploy them back into the real world, they are liable to perform worse than you expected. Because the environment you've put them into is actually different from the environment they thought they were trying to optimize. So a really simple example of that would be maybe I'm treating patients at random intervals, 30 minutes, 90 minutes. But when I pre-process my data, the model just sees fixed regular decisions every one hour. Well, from the models point of view, that's the environment it expects to be deployed into is having decisions once an hour every hour. And so when you start exposing it to an environment that isn't like that, it may well do poorly. But I think the most interesting bit about that is that if you are trying to predict how your model is going to do and you do that analysis on your pre-process data, then your predictions are also wrong because you're predicting how it will do within this fictional pre-processed data set, not how it will do in your real world messy regular decision making. So I think those are probably the most exciting bits of the thesis. So your off-policy evaluation is kind of using a different MVP than where the data came from. Is that the one we're seeing? Yeah, I think that's exactly how I'd summarise it. You've just essentially completely changed the nature of the decision making that was generating the original data. Because obviously the data was generated using unpredictable irregular decision frequencies, but you're pretending to the model that it was generated using a regular decision frequency. And we call this a semi-MDP use, right? When the time, time, step varies? Yeah, I think semi-MDP is the right way to frame it. It's in a way to continuous time setting, but unlike certain other continuous time settings, like robotics, where maybe the scale, the precision in which you want the frequency of decision making is very small. For healthcare, it doesn't really matter if you make a decision at one minute resolution versus three minute resolutions, but patients, the physiology doesn't really change that quickly. So you can kind of discretize it into like really, so well-defined discrete time steps, like every five minutes. But if the decision making itself is happening over it, irregular numbers of steps, then that is best framed under a semi-MDP framework, for sure. So I kind of came across this line of work a number of years ago, looking at the kind of a famous paper AI clinician, the Kamarowski paper that was on the cover of nature, and it claimed like an alpha-zero type, you know, we're going to optimize sepsis treatment. And there was a very specific, quite a complex methodology used there, and I gather you're doing something quite different now. Do you want to talk about the kind of this line of work and where it came from and how to end it up like this? Yeah. Yeah, so absolutely very, very happy to talk about that. So let's say alpha-zero comes out in around about sort of 2017 or so. There's a lot of increasing hype around deep reinforcement learning, very much led by sort of deep mind research. And then in sort of 2018, you get this paper published in nature that is really building on that hype by saying, well, we've taken reinforcement learning, we've applied it to healthcare data, and we think that we can treat sepsis, which is kind of a condition of a disordered response to infection. They can have quite a high mortality rate. Basically, a paper saying that we have found a better way of managing sepsis through reinforcement learning for these sick patients, published in nature. And I think to the best of my knowledge is the most highly cited paper in healthcare based reinforcement learning. And it's probably led to a lot of interest post that paper being published. And I think much in the same way as deep mind doing our research in Atari is not reflective of real world challenges, but was still a very valid, like, important sort of initial step. I think I feel similarly about the AI condition paper in that I think for an early bit of work, it's, it has a lot of sort of positive aspects. But I think that the field of healthcare RL, specifically offline RL, has sort of fallen into a little bit of a stagnant, no, that's the wrong word, a little bit of like a trap. So essentially in the AI condition paper, one of the things they do is that they they bend this data, so they take this irregular sample data, and they bend it into four hour fixed regular windows. And that's immediately problematic because sepsis is not managed like that. Sepsis is a rapidly evolving condition. You often need to do a lot of higher frequency decisions early on in the management. Once they've stabilized, then it might be sort of a longer amount of time between successive decisions. But early on, it's not the case that you take a single decision and then you wait four hours. And what has happened post that paper is that we're seeing a lot of other reinforcement learning papers in healthcare also use sort of bending or temporal resampling of their data. And I think the reason it's done is because it is a very convenient fix for the problem of irregularly sampled and sparse healthcare data. And you have this paper, this sort of seminal paper that's so highly cited. So I think it's very reasonable for someone to do a literature review, see this paper, see the works that follow and decide to continue in that vein. But as I've said, it's problematic because at some point, we're going to want to run clinical trials with these models. I mean, I think that should be always the long-term goal. There's not to publish papers, it's to actually get something to the patient bedside that's going to change our management. And if that's our end goal, you know, clinical trials are very difficult to set up, they're costly, they're expensive, they're risks to patients. We want to make sure that the models that we get to clinical trial have the best chance of success. So how do we do that? Well, we need to make sure that when we train them, we train them in a way that is reflective of the environment they've been deployed into. And so I suppose where my work differs is my work is essentially saying, look, you know, we have, I'll be more than 90 percent of health care offline oral papers are are pre-processing their data in this way into fixed regular intervals. And I think that that is problematic because we train models on a fictional version of reality. And I also think that we're not going to accurately identify that this is problematic. Because when we do our evaluations, most of those evaluations are retrospective, they're down on the same bin data. So they're not accurately reflective of how the model's going to perform. And to date, I think there's maybe being by my last count, there were, I could find three prospective trials of a reinforcement learning models trained on historical data and then tested out on real patients. One of which was a behavioral intervention. So very low stakes. One of which was sort of managing deep brain stimulators in a very controlled setting. They didn't need to do any sort of temporal binning because of the nature of their, the stimulators. And then there's, I could only find one inpatient prospective trial of an offline oral algorithm which was done in China. So the vast majority of the literature is relying on retrospective evaluations. And so what you're seeing is you're seeing papers like the air clinician publishing very positive sounding evaluations. But you have to bear in mind, those evaluations were done on heavily processed simplified data, the vast majority of the time. So the sort of the first chapter of my PhD looks at exactly that. I essentially get a, there's an FDA approved clinical simulator for, for diabetes management. And so what I do is I take that simulator. I modify it so that we have irregular intervals. I collect a bunch of data from that simulator. And then I do varying amounts of pre-processing to that data. So now we have some re-sample training data sets. We have some non-re-sample training data sets. I do some offline oral on those various different data sets. And what I'm then in a unique position to do is I can prospectively evaluate my models in the original, unpredictable simulated environment. And so you end up seeing exactly what I sort of theorize would be there, which is that one, when you train on pre-processed data that has a fictitious temporal sequence, your model does poorly if you then deploy it back into a non-predictable environment. Whereas if you train it on data that hasn't been pre-processed, the model can actually improve and do a lot better. But more concerningly, when you do retrospective evaluation of all those models on their original data set, retrospective evaluations don't predict that difference at all. So retrospective evaluations think that the model is going to do very well because in fact, it has solved the optimal actions for your fictional data set. So from retrospective evaluation point of view, your model was doing exactly what you wanted it to do, which was to find optimal actions for that data set. But it just so happens that that's not the same as optimal actions in an unpredictable environment. So like if you do off-policy evaluation on the the Kalmerowski data set, which I've done, it'll tell you, well, this policy would be better, it'll find a better policy for that MDP. You're seeing that MDP isn't, doesn't represent that MDP we really want to solve. That's exactly what I'm saying. Yeah, and there's a few, I want to zoom in a little bit. There's a few different aspects of this, right? There's the time-bending forces, the observations to be been in time where they're generally not. And also the actions to also be lined up with the observations, which maybe not the case in clinical work. And then also the this notion of was there a no-op? Was there actually, did any doctor actually by at that moment and decide, no, I'm not going to take an action versus there was just no decision made. Is that correct that we don't see the difference between those two things on the data? Yeah, I think that's, I think what will come to the know-up aspect in a moment, but I totally agree with you that there are several overlapping things going on when we observe that models trained on bin data, then perform poorly in unpredictable environments. The first is exactly as you say, we're if you imagine a bunch of irregularly timed input data or irregular timed decisions, if you're binning over a fixed interval, there are going to be some intervals where you have multiple measurements, so you have to aggregate those together into a single measurement, so that could be averaging or something or taking the last measurement, whatever you like. And there'll be some windows where you don't have any measurements at all. So you end up imputing a value using whatever method you want to use for imputation. So that's already somewhat problematic because you're saying to the model, so let's say the patient received a drug dose of one and then three minutes later they received a drug dose of four. When you bin that data, you're saying to the model the patient received a drug dose of six, because we're just adding those together, but that's not really what happened. What happened was they received a drug dose of one and then some time passed and then they received a drug dose of five. It's physiologically quite different. So you have this sort of alteration of what the numbers themselves mean, so that's sort of one issue. The second issue is that you have this, what I like to call a failure of temporal expectation, so what I mean by that is let's say that your model was trained on data that had been binned to a very high frequency resolution, so maybe binned to every ten minutes. So you're saying to your model an independent decision was made every ten minutes. Then your model starts to learn this expectation that it will always be able to adjust its actions after ten minutes. And if you then deploy into an environment where it takes an action under that assumption and then an hour goes past before it can make a snake decision, then it's at risk of making decisions that are actually unsafe that it probably wouldn't have made if it had known that it wouldn't get an opportunity to intervene again for another hour. So a really simple example would be maybe their blood sugar is a little bit high, the model might learn to give it a very, very big dose of insulin or a very high rate of insulin of fusion, knowing that in ten minutes time it can just check the blood sugar again and adjust it once it starts to normalize. But obviously if an hour goes past and for every minute of that hour the patient's just getting a high dose again and again and again, then you end up over treating. So this kind of failure of temporal expectation I think is important. It's important for the model to know that there might be a longer than ideal delay between its decisions. So I think those are sort of the two main ones. The third one is a little bit trickier and it has to do with bias. So there are a couple of different ways that you can aggregate your time window data for observations and for actions. Now you can aggregate them over the same time window. So you get all your observations from T0 to 60 and all your actions from T0 to 16 you aggregate them and then you say this was the observation and this was the action. Now the problem with that is if multiple actions are happening and multiple observations are happening then you actually have look ahead bias where the observations that you're giving to your model have already been confounded by the very action it's trying to model. So you measure blood sugar, you give some insulin, you measure it again, you give some insulin, they'll be changes to the observations based on those two insulin doses you gave. But if you aggregate them together, so now you just have a single average blood sugar measure and a single average insulin dose. When you train that in offline RL what you're saying is to the model given this observation, this was the action given and that's not really true because the observation has already been changed by the action itself because of aggregating it. Yeah, the MVP or the original SMDP has a certain causal structure and then when you apply this meaning it completely changes the causal structure of the resulting. Yeah, that's exactly right. So that's from a piece of work by Shenpu Tang at Emory University. I'd encourage anyone to have a look at that paper. So then you could say okay, well that's a problem, how do we deal with that? Okay, well what you could do and what in that paper is recommended is you average your actions from T to 60, sorry T equals zero to T equals 60 and then you average your observations over the previous window. So now all of your observations that you've aggregated are all before the action that you've aggregated. So you've removed, look ahead bias, which is great. However, because of how these aggregation windows align, you can end up with very weird sort of causal inversion artifacts where you have these strange things where maybe in the original data, the patient had a big glucose spike and the clinician responded by turning the insulin up. But if you then average with these kind of separate windows, so the observations are all averaged after the actions, it can look like the order of that has changed that the insulin was maybe turned up and then the blood sugar went high, which is obviously also not ideal because that's not actually what happened in the environment. So essentially there's no good way that you can do these aggregations when you're having multiple observations in each window, which I think is sort of the third important problem with with binning as a practice. I mean it's it's like we've tried to turn the patient into an entire learning environment game and we're plugging in like the classic DQN stuff is all very simplistic when it comes to regular time steps. So I mean it makes sense. We had all this RL machinery that was assuming that. But you're really you're really zooming into how that really doesn't figure. But I want to point out it's not just I mean you're solving this for clinical RL. But what you're saying really applies a lot more broadly to I would guess all sorts of areas in real world RL, industrial RL where where that has these properties, right? That has the properties that you're you're finding here where the irregularly spaced actions, observations, decisions are not made on a on a on a clock tick. Ways in which the real world is very different than the assumptions would under DQN. Yeah, I'd say that's that's completely accurate. I'd say the conclusions from the chapter we've been speaking about, they're really specific to a setting where you have irregularly sampled decisions in particular, the important one, but irregular sampled data and you're trying to do reinforcement learning on that data by first binging it and turning it into this neat, clean, MDP when the underlying environment itself at least in terms of how the data was collected was not an MDP. And so there's obviously not limited to health goes limited to any setting where you have data sets like that where the intention is to sort of deploy it back into the environment from which you source the data. And I think I think that should be intuitively obvious. I mean if even let's say I had a big data set of just chess games and I'm trying to teach you Robin had to play chess, it's so intuitively obvious that if I just show you every fourth position from the games, then you might think you've learned how to play chess, but you obviously haven't learned how to play chess because I've lied to you about how chess is actually played. It's not the case that you make this move and then you see this new position that actually happened four moves later. And I think that the real world is no different. So we really want to be training these models in a way that is as accurately reflective of the source environment as possible. So Thomas, do you think I guess you think that a RL in healthcare and clinical RL is a worthy pursuit and it'll lead to real benefits? Yeah, so I think if for anyone who's kind of listening who does research in RL or is interested in doing research in RL, I'd really like to really encourage people to consider healthcare as one of those domains to be working. I think it's very challenging, but also the potential benefits are huge. So healthcare is very important. Anything that anybody cares about is only possible because of good health and it's very easy to underappreciate health until you no longer have good health. then you realize the sort of value of it. I think it's a really interesting reinforcement learning challenge because if we think about most reinforcement learning, most recent reinforcement learning is online. You have this opportunity to interact with the environment, make mistakes and get feedback from that. And part of that is this idea that RL is always changing, sorry, RL is always affecting the state seeing, the data it's seeing. So in supervised learning, if I deploy a model, the predictions of the model don't really necessarily affect the data the model is being exposed to. But in RL, the decisions it makes affects future states that it gets exposed to. So that sort of distribution of data is always changing as the model evolves. And that's all well and good, unless you are encountered with a domain where you can't actually train our own models in that way. So health goes often a canonical example for that sort of problem because you can't let algorithms experiment on people for kind of obvious ethical reasons. So instead, we end up using kind of offline RL where we're looking at historical data. And there are pros and cons to historical data. Advantages are that we can really cut out a lot of unnecessary learning where we're sort of rediscovering the wheel. We've got decades of clinical trial knowledge, clinical expertise, all of which is distilled into the behavior of everyday clinicians. So all of that learning has already been done and is present in the data set. But then the limits are that you can't really learn past the behavior you're seeing in the data set. So even though clinicians are noisy decision makers, you can only really perform at the towards the best level of known clinicians. You can't really extend much beyond that just because you can only learn from the data you're seeing. So I think that's a really interesting challenge because we have these huge data sets. We have so much data. But in much the same way that online RL, you have algorithms affecting the data you see when you train in an offline setting, when you deploy your algorithm, it's going to change the data it's seeing based on the decisions it makes. So then that becomes sort of an interesting challenge in and of itself because you want your agent to be a bit better than how clinicians are behaving, but you don't want it to be too much better. So you're sort of aiming for like the top 20% of clinician behavior. So why don't you want it to be too much better? Can you help me understand? Yeah, of course. Yeah. So if your if your model's behavior is too far away from the behavior of clinicians, your model ends up learning out of distribution behaviors. Oh, you just don't have coverage there. You just don't have the coverage. So it could be that it's better, but you don't know. And if we're thinking about the sort of safety, critical nature of health care, you want your model to learn behaviors that you have good reason to think are are better actions rather than just out of distribution actions. So that in and of itself is interesting. And I think is part of the reason why models trained on bin data then perform poorly is that they're being exposed to unfamiliar data. They make decisions that are out of distribution because the data is unfamiliar. They then get exposed to even more unfamiliar data and you get the sort of compounding distribution or shift where the model sort of just collapses. So yeah, trying to have your model learn behaviors that are relatively within the distribution of the data is important, which earlier in the earlier in our discussion, you talked about sort of know up actions. And I think now is kind of a good time to talk about that because when you bin data, you establish every single decision point very clearly. You know, you have time steps one, two, three, four at regular intervals and at every time step, there must by definition be a decision. But if you are only interested in a regular decision intervals, then you can fall into a trap where you may not know all of the decision points in your data. So to give you a basic example, if we look at outpatient clinics, we know when every decision occurred because even if outpatient clinics are directly spaced apart, we know that the only opportunity the clinician had to change their management of the patient was at the outpatient clinic. And between clinics, there's no opportunity to interact with the patient. Whereas when you're in an inpatient setting, it's much more of a continuous time setting where in theory, the connection could go into the ward or the bed bedroom of the patient at any point in time. And you know all the times when the clinician has changed something, because that might be recorded or should be recorded in the data set. But what you don't know is all the times when your clinician went into the room, thought about doing something and decided, nope, I think the best move is to do nothing. And then they go away. And that's, you know, that's a missing label. And it's a label where imputing that becomes very important. So you could naively say, well, why don't we just treat every single instance where they don't make a decision as being a labeled no-op action. And the problem with that is you then fall into this trap that we talked about earlier that the model then just gets this false temporal expectation because you're giving it labels every single minute of do nothing. And so it thinks it can intervene frequently. Is that what you mean? The older papers have handled this? I think it implicitly yes, by bending you are effectively doing that because you're saying if there are no clinical decisions made in my, in a particular time window, then we'll just impute the label of, of do nothing. But that's obviously not really what really happened. And what's really important in terms of minimizing this risk of sort of distributional shift, and maximizing the chance the models learned and accurately reflective visualisation of the environment. What you want to do is you want to make sure that when you impute those no-op labels because they are important, it is important sometimes to do nothing. That you're imputing them at a frequency that matches up with how often you think the model is going to get queried when you deploy it. So the model is then learning, okay, here are two no-op labels that are about an hour apart. So I'm going to learn to expect that there might be about an hour until my next decision. And in some settings it can be easy to impute that. So if you take my clinical setting in my thesis, so insulin infusions, we have a very strong prior that we can rely on, where we can say it in current clinical practice clinicians only adjust insulin straight after a blood sugar check. That is a mostly true statement. No clinician is going to go fiddling around with insulin when they don't know what the recent blood sugar was. So that's great because then we can just look at how often bedside nurses, for example, are checking blood sugar. And then we can just anchor our no-op labels onto those blood sugar checks. So every time a blood sugar gets checked, we look for any nearby actions. And if there are no adjustments made, then we can impute a do nothing label. And it is, it's a proxy, but any sort of imputation is going to be trying to approximate when we think these labels should. And what's nice is I can then you can then deploy a model and you can say to the hospital, the ward you'll deploy in the model, listen, this model expects to be queried every time you check blood sugar. So check blood sugar as often or as infrequently as you would normally do so, but just know that when you check it, the model is expecting to be queried. Which is nice because it is naturally irregular. It doesn't change how the bedside, like clinical practitioners are currently acting. Basically, say, do what you would normally do, but when you check a blood sugar query the model. So that's a sort of an easier example. But there will be other situations where you don't have those clinical priors. And there's some sort of research that looks into that problem, not by me, but by some other researchers. A random idea of can we use sort of kernel-based method or similar methods to identify certain states, certain moments in time where there's a very obvious spread of actions being taken by people. In other words, this is a point of contention. Some clinicians go for action A, some clinicians go for action B, whereas maybe there are other points in time where all the clinicians are very consistent with the action they take. Does that make sense? Yeah, so I mean, that's really interesting. Why would clinicians disagree? And I wondered, are there different philosophies of care, different schools of thought, or possibly could there be an issue with state construction? Like, something wasn't recorded, and so we don't notice the difference between these two states. Yeah, I guess these [BLANK_AUDIO] questions come up when I look at this type of data. What causes clinicians to disagree or apparent disagreement in the data? - Yeah, I think all of the things you've suggested are very insightful and definitely are part of it. I think at the end of the day, we are clinicians are human, which means we're subject to all the same restrictions as other humans. In other words, differences in clinical experience, differences in local guidelines, differences in cognitive biases, for example, if you recently missed some catastrophically low blood sugar or high blood sugar, then maybe you're more likely to be extra acutely aware of whenever a blood sugar was checked, whereas certain other clinicians maybe might be more prone to missing results. And then you see no op actions where you wouldn't expect to see no op actions. So there are loads of different reasons to do with both individual differences and also kind of regional differences between hospitals and areas and countries. And I suppose the additional factor is in some settings, we just don't know what the right thing to do is. So even, maybe there isn't a best standard, maybe there's some ambiguity, and that's where personal preference starts to become more obvious that some clinicians think that certain ways of behaving are better than other ways. All of which is not good for patients, but is good for offline reinforcement learning, because it means that we get to see a spread of different actions for the same set of kind of physiological patients, which means that we can use these big data sets almost like natural experiments, where we can say, okay, are there groups of patients that receive a spread of actions and the groups of patients who received action A do slightly better than the groups of patients who received action B, so maybe we should only take action A. So it's only possible. - Yeah, so we're not getting full epsilon greenie exploration or we're getting a little bit of diversity. - Yeah, that's absolutely right. And if we don't have that diversity, then you can't do offline RL, because you don't have the coverage. If every state has a deterministic action, it doesn't work. - I mean, so you've been on the other side, so how does the, you know, I've seen doctors and for certain things, they have flow charts or decision trees to work with. How does a doctor coming into doing this? What level of guidance do they have and what level of freedom do they have in executing their own policy? - Yeah, so there's normally a four insulin infusions, that many hospitals will use a sort of a pre-specified protocol, they're very simple. They essentially say, what is the blood sugar and depending on what band the blood sugar is in, so it might be like four to seven, seven to 10, et cetera. You should adjust the insulin by this amount. And maybe you might have different columns for whether they're already unknown diabetic versus not a patient with diabetes, but their blood sugar is high anyway because they're very sick. So these kind of protocols do exist. There's not a single right protocol, so different hospitals will have different protocols. And different clinicians might be more prone to overriding those protocols. So they might say, actually, I think that we need to take a different action for whatever reason. Maybe we think that the action is too aggressive from the protocol or maybe we think that the action recommended by the protocol undermanages these patients. And then obviously there are the cases of missed blood tests which happen semi-frequently where you don't miss them forever, but it might be the case that a blood glucose was checked and someone only really registered that it was higher than normal or lower than it should be an hour later or 90 minutes later. Just because it's especially in intensive care, there can often be a lot going on and it might just be the certain one of multiple tests was done and that particular blood test just happened to get overlooked. So yeah, there can be generally standard protocols for these things, but I don't think they're optimized protocols. I mean, the argument I make in my final chapter is that when we are optimizing something like insulin infusions using off-finaleral, what we should be doing is using the power of RL to optimize for long term outcomes directly. So for example, for me, I'm looking at survival, but it could be length of stay or it could be some other longer term outcomes. And we sort of disregard this idea of an optimal blood sugar range kind of completely. I have one of my chapters essentially talks about a pre-processing pipeline for putting together a data set that is sort of naturally timed, realistically timed, both from an input data point of view and from a decision label point of view. And the pipeline is sort of agnostic to the domain you're working with in, but along with the pipeline is sort of a published data set that I call insulin for RL, which is encouraging researchers interested in health care RL to try their hand at off-finaleral using healthcare data that is naturally timed within the context of insulin infusions in critical care, yeah. - Okay, so let's talk about insulin for RL. Now the mimic data set that you drew this from covers many different types of conditions. And we've seen historically a lot of work on sepsis. And you went a very different way here. Can you talk about how you decided to focus on insulin and why that's different and how you built this data set? - Yeah, so I'm really happy to ask that question because if you look at the healthcare RL literature, offline RL literature in particular, what you will see is that most papers look at sepsis. And I think part of that is because of the AI clinician paper, the Matthew Comerowski paper. And part of that is because sepsis is just, it's such a good headline grabber in terms of the domain you're working in. You know, the mortality of sepsis is high. So there's a lot of potential gains to be had for helping patients. But I actually think that sepsis is probably the last problem that we should be working on because it is so fiendishly difficult. There's so many things going on. It's such a heterogeneous condition and there are so many different aspects of management. And so why I think is that we should really be going about it the sort of deep mind Atari route. IE, we should be starting with a much simpler version of that kind of problem and solve that and then iterate our way up to more complicated settings from there. So the reason I choose insulin is that insulin is the simplest example of what we call titratable infusions, infusions where you have to adjust the dose at sort of maybe not regular intervals, but certainly frequently. You just have this very simple feedback loop where the patient is getting insulin and you're checking their blood sugar. If the insulin is high, the blood sugar will go down. If the blood sugar is high, you probably need even more insulin. So it's a very clean, simple feedback loop. And the argument I make is that we should start by just really being happy that we've solved that problem because then when we move on to something slightly more challenging like say blood pressure control, we have far fewer problems that we need to solve for that. So that was the rationale behind choosing insulin. In terms of the insulin for our data set, a lot of patients as I say often put on insulin infusions when they're critically well, regardless of whether they have a diagnosis of diabetes or not. So the insulin for our audit set within mimic, I think identified approximately 12,000 patients who were a minute ICU and had a period of time where they were receiving an insulin infusion. And most of those patients just have a single episode of receiving the infusion, but there are maybe sort of 5% or so who had two or more episodes. So about 12,000 patients. And if you include the sort of no op imputed decisions with every blood glucose check that didn't have an action, then in total the data set has about 375,000 labelled decisions. So just over I think half a million cumulative hours of sort of patient exposure. And I think that should be more than enough samples for us to kind of make good headway on that kind of a problem. And as I say, optimizing insulin is not a solved problem. Yes, we may not get the same glamorous mortality improvements as something like sepsis, but we're much more likely to be able to defend to be say, we've probably solved this problem based on the data that is available to us. So yeah, that's probably the sort of the headline things that I would say about the data set. Maybe one other thing I would add is that as far as I'm aware, it's the only reinforcement learning data set for healthcare that uses naturally timed data that is specifically intended for reinforcement learning. And hopefully if people access the data set, to find that there's set up four reinforcement learning researchers. So there's a data frame that you can use, but there's also tensors that are just states and actions and rewards, kind of the more familiar format. I mean, I think part of the reason so many reinforcement learning researchers rely on binning is that there are already well-established binned healthcare data sets. And if you're not super familiar with healthcare data, it could be very intimidating to try and clean that from the scratch. So people are probably prone to wanting to use pre-established clean healthcare data sets, which I think is entirely reasonable. So the goal of the data set is basically to say, look, we've cleaned the data for you so you can just focus on doing the offline RL with natural time data. - Can you spend a minute on the state construction here? Like Kamarowski had this clustering thing with discrete states. And I always wondered about how that was over the logic behind the clusters. Yeah, how does the state, or maybe this doesn't assume too much 'cause you're just providing the data set and then, but how do you do state construction when you build your policies? - Yeah, of course. So the Kamarowski paper, for example, uses, as you say, it uses clusters. So you end up with a discretized number of states. I think off the top of my head it's around about 750 states. Now you could ask the question of whether or not all possible ICU phenotypes can be grouped together in 750 states. I think that's sort of a separate question. My approach is that I think we should give deep learning models the poorest possible data we can and let them form their own latent impression of what they think, so the patient state is. So the way that I define states and implement states in St. Ferrari is that we have our decision point. So blood glucose has just been checked and there will be some label that we could learn from. And what we do is we look at the last about seven days of historical events. And by events, I mean lab test results, changes to infusion rates, drug boluses, those are sort of the main ones. And what I sort of recommend in the data set, as I recommend using something called the medical event data standard. So you can take this history of medical events. And for each medical event, you represented as a tuple of the particular feature. So it could be their sodium level was checked. The feature, the value, let's say 140, the time, so let's say 60 minutes ago. And between those three aspects, you sort of captured the most important things related to that particular medical event. And you can order these as a sequence. So you have a sequence of these tuples. And all of a sudden, you can just run this through any sequential network that you like. So it could be an LSTM, it could be a transformer, it could be state space model, whatever you like. Where you start with an embedding network to embed the tuples into kind of a single vector. So that that medical event is represented as just sort of one vector. But you have this sequence of irregularly sampled events that you then just provide to the model. And from that, the model draws its own sort of latent impression of what the state might be. And I think that's good because it means that you have done really as little as possible to influence it. There's no hyperparameters around state clustering. There's no bending of the input data. There's no imputation. There's no missingness because the model should develop this implicit understanding that the only observed data is the data being given to it. And so anything that has not been explicitly observed by definition is unobserved. So you don't need to explicitly state, sodium wasn't measured at 61 minutes, 62 minutes, 63 minutes, it's sort of not necessary. It doesn't really add new information. So I think it's a more powerful way of representing the state. And it was inspired by a paper that came out I think five years ago called-- Oh, it's called Straits, so it's a self-supervised transformers for sparse and irregularly sampled time series data, something like that. But they're really sort of the origin point that and the medical event data standard between those two was what inspired me to set up the input state in that way. So in chapters five of your thesis, you focused on using temporal difference learning for mortality prediction. So you're using TD learning for something in a very different way here. Can you tell us more about that and what you found? Yeah, so the inspiration for that chapter sort of trying to use temporal difference learning for this purely mortality prediction of task, the inspiration came from the chapter that followed it, which was, I know that I want to optimize instant infusion doses for survival, which is sort of a novel thing that people haven't really tried to do before. And I know that I want to use this kind of new naturally irregular health care data. But I suppose there's this question of how effective is temporal difference learning at learning to predict mortality when you have such an irregular structure? And also we know that health care data is sparse, the states are partially observed. So maybe this is actually a really challenging thing to do. So I wanted to sort of empirically evaluate that a little bit more. So what I did was I took all the patients in mimic, so not just those on instant infusion, it's just every single patient in mimic and I took every single medical event in their history within my sort of feature set of lab test and drugs. And I set up a TD learning, temporal difference learning, sort of framework or approach where, well, I'll take one step back. In temporal difference learning, the idea is that your bootstrapping from your predictions from the next state. But here in our sort of messy, irregular sample environment, it's not really clear what the next state should be. So you could take the super naive approach, you could just say, okay, we'll add a particular point in time where we've just measured something, we've just had a medical event and we've got some history that we can give to the model. The next state is just gonna be whenever the next observation comes in. Now that could work, but in somewhere like Intensive Care, where you're having observations happen very frequently, that interval might be really small. So it could be like every two minutes or three minutes or five minutes. And we already know the temporal difference learning can really fail to converge when the intervals between states shrinks to really small intervals and you're getting very small differences or imperceptible differences between your states from one moment to the next. So there is an advantage in you implementing some form of delay in time between your two states. And what I suggested was that we should basically implement like a mandatory delay, let's say 24 hours. So we've got our current state, which is a bunch of recent history, sort of medical events up to now. And then our next state is gonna happen, we'll wait 24 hours and then whenever the next medical event comes in, we'll just pick that point and we'll just say, okay, whatever the history is leading up to that point, that will be our next state. So I trained a temporal difference learning model in that way. And the task was to predict end admission survival. So this is very sparse reward setting. You've only got one binary reward coming in at the very end of the trajectory and everything else is bootstrapping. So there's obviously a lot of risk that you can have like significant bootstrapped related bias, your model might not be very well calibrated. It may really struggle to converge. And pleasantly what you see is that none of those problems arise, that not only does it converge, but it converges very well. So it ends up with very well calibrated predictive probabilities for predictive mortality. But what I think is interesting about that chapter is that you kind of want to have some sort of baseline to compare to. So what I did was I also trained a set of supervised learning models to predict mortality at various horizons. So predict one day mortality or seven day mortality. And I would compare the performance of those models to the performance of the TD trained model. Again, at each of these different sort of labeled time points. So one day from whenever your state was constructed or three days from that state. And what you see is as the prediction horizon gets pushed further and further into the future, TD starts to lead to much better performance than supervised learning models that have been trained on similarly distant mortality labels. And the theory that I put forward is that this relates to the variance of the label. So if you imagine predicting 28 day mortality, a lot can happen in 28 days. They can influence whether the person then died at the 28th day. And a lot of those things might not necessarily be predictable from the information you have today. But the problem with deep learning models is that they're encouraged to overfit the data. So if you have this very high variance label, what will happen is the model will start to overfit to spurious correlations. Either correlations that don't mean anything at all, or correlations that are really super, super specific to the mimic data set. But they don't really generalize in a physiological way. And what I found is that if you evaluate all of those models on an external data set from a different continent, you see exactly what I predicted, which is that the supervised learning models trained on distant mortality labels, they have big drops in their performance when externally evaluated. Because they overfit to this kind of noisy distant label. Whereas temporal difference learning models, because they're trained against this bootstrapped target that is much closer in time horizon to sort of now. So in my experience of 24 hours from now, the variance of that is much smaller. And we know that this is an advantage of temporal difference on something that you have this bias variance trade off. But yeah, what you see is models trained using TD learning, they have way less overfitting when you externally evaluate them. And you also see that their mortality predictions, even if they're not as well calibrated, they're less well calibrated in a way that is uniform. So what I mean by that is that it systematically overpredics mortality, which means that you can do post-doc correction to try and adjust that. Whereas all the supervised learning models don't have that. They have quite a messy disruption to their calibration. So to sort of summarize all of that, in essence, I sort of showed that you can use TD learning to predict long-term horizon outcomes in a setting where it resolves this issue of overfitting. The models generalize really well to unfamiliar data from different settings, much, much, much better than you would get otherwise from supervised learning. And I think that's kind of cool. I think that's interesting because, as you say, it's applying TD learning outside of an oral approach to just a purely prognostication prediction setting. I had to really stare at that and cross my eyes a bunch of times. I was surprised by your finding. And you mentioned that in this setting, you're facing the deadly triad, which is the function approximation, bootstrapping, and off-policy learning, you have all those factors in play. And yet, yeah, I was surprised. I was actually really surprised. Were you surprised or are you really so expected this? Or I had to look at your diagram. I had a good diagram explaining it as well. I had to look at that for quite a while. Yeah, I think I was pleasantly surprised at just how well calibrated it was. I think I expected it-- I expected its ability to predict survival to be better than supervised learning for the reasons that we talked about. But I also expected the predictions to not be very well calibrated because of bootstrapped bias. So I was very pleased to see that actually the calibrations were even better than supervised learning. I think the reason I talk about the deadly triads in the early bits of that chapter are because, as you've sort of already hinted at, there are lots of good reasons for this to not work, which means that before we even get to the task of trying to optimize insulin for survival, we need to know the TD learning can actually converge to accurate predictions in such a setting, where you have this partial observable data, function approximation, bootstrapped bias, et cetera, et cetera. So for me, it was very important that we actually drilled into whether TD learning works in that specific setting. Obviously, TD learning works when everything has been optimized and it's a wonderful technique. But there are scenarios where it fails, like the deadly triads. So yeah, I thought it was important to drill down into that. And my expectation was that it was going to outperform supervised learning. But I didn't expect that it was going to be so consistently well calibrated. Very cool finding. So should TD learning be used across the board for clinical predictions of this type? Yeah, I mean, I would love to see more interest in TD learning from a predictive point of view. I suppose there's a caveat to that. There was a really interesting comment piece. I think that came out of, I want to say, King's College London. And it was sort of making the point that, in a clinical setting, a lot of our prediction algorithms, like an AI, are really decision models in disguise. So why do we want to predict that a patient is at a higher risk of deterioration? Because we want to intervene in those cases. Or why do we want to predict that a patient is at higher risk of developing sepsis? Because then that means we probably need to give them antibiotics. And there are some problems with that that reinforcement learning-- well, we're very familiar to reinforcement learning practitioners that if you're going to be training a decision model, then you need to be wary of these things that we talked about earlier around having behaviors that are still in the distribution of your data set, conservative learning, et cetera. So having a pure prediction model that is a decision model in disguise has its pitfalls. Nonetheless, I would like to see kind of a wider uptake of temple difference learning as an alternative approach to supervised learning for these particularly longer term horizon predictions. And that's not specific to healthcare. I mean, in any setting where you are trying to predict long-term outcomes, and you have this sort of time series data that is being raised-- maybe not regularly measured, but you have a lot of time series data-- exploring whether temple difference learning might be an alternative way for you to accurately predict these long-term outcomes, I think, is worth a look really by anyone. So you found some pretty interesting things regarding insulin policies. Do you want to get into that now, Thomas? Yeah. So let's talk a little bit about that. So for your listeners, in terms of the kind of standard best practice, we know that really high blood sugar is bad, and we know that really low blood sugar is bad. And then there's a middle gray area where we don't know whether patients being slightly high on their blood sugars is bad for them, or whether it's a protective mechanism that their body is intentionally using. And trials that have tried to figure the answers to that question, trials that have explored that have really struggled to get a definitive answer. So I trained some models on InstaFrarau to optimize survival, bearing in mind there doesn't know anything about optimal blood sugar ranges. It just knows that you have a patient state, and you can change their insulin infusion, and it's trying to reduce their expected mortality for certain actions. And what you see is you see that, first of all, from an off-policy evaluation point of view, there's a predicted reduction in mortality of about 0.5 percentage points. And what's interesting about that is that that sounds realistic. Insulin correctly managed will have some influence on survival, but not a massive-- it won't halve your mortality. But certainly it's sort of a realistic looking number. I was very keen throughout my thesis to be using as robust statistical methods as I could come up with. So I did a lot of this sort of deep reinforcement learning at the edge of the statistical precipice. I was very into training lots and lots of models, boost up confidence intervals, et cetera. And so I'm quite confident that within the limits of what we can know from the data we have, that that probably is a realistic figure. But then you can kind of get into the qualitative side, which is what are these models doing differently, that is leading off-policy evaluation to think that survival will be better. And what you see is that most of the time, it doesn't do anything differently from clinicians. So, you know, 90, 3% of the time, it is happy with what clinicians are doing, as in, it doesn't think that there's a survival benefit that is being missed out on. But if you have a look at the sort of three small minority of states where it differs, then two broad patterns emerge. The first is that there are a subgroup of states where patients are, they have very high blood sugar, and the model thinks that we are under-treating them. So we need to be more aggressive with our incident. And if we look at the sort of physiological aspects of those patients, we can see that they're sick patients. You know, they're more likely to have biochemical markers of being very sick. But they're also patients who are on high insulin rates, which suggest that these are probably patients, where we're following these protocols that say off their budget is high, 10 in sum up by one unit. But because they're already on quite a high instant rate, in terms of relative adjustment, we're probably under-managing them. You know, if you can imagine if someone's on 15 units of insulin and then you turn it up by half a unit, that's a very small relative change compared to if they're on one unit power and you turn it up by half a unit. And then the other set of patients are patients whose blood sugar is actually normal or at the lower end of the normal range. But if you have a look at the biochemical characteristics of those patients, these are, again, patients on a high instant rate, so they're getting a lot of insulin, and their blood sugar is trending downwards. And we know that low blood sugar can be much worse for your survival. So these are patients who are on a high instant rate, their blood sugar is falling, but currently it's normal, which means clinicians are probably missing out on those patients until they become, you know, hypoglycemic with low blood sugar, and then the intervene at that point. And that's the sort of interesting thing is that the patients whose blood sugars are actually low, the model is happy with how we manage those patients. So once we know that blood sugar is low, we're managing them correctly. The problem is we're not always identifying the higher risk patients who are trending towards that state. And I think for me, that's really satisfying to see because it means that the model has converged to two quite realistic sounding policies of behavior in terms of where it wants to overall clinicians, rather than something that would look very weird and curious, and it doesn't really make sense. And you know, maybe it's figured out some sort of high-dimensional benefit, but we're not really sure why it thinks that's a better action. No, you have two sets of patients where it's pretty obvious why the model wants to overall what clinicians are currently doing. And if you look at the existing literature, it fits with what we know about which patients are most likely to get survival benefits from certain approaches to insulin and blood sugar control. So I think that's quite a cool finding to come out of that for me anyway. Absolutely. So you've opened the black box and some of your findings, I mean, maybe there's a chance you could even influence your practice with this work, is that right? Yeah, I mean, I think if I said either of those things to a clinician, I don't think I don't think a clinician would find that controversial if I said, like if the patient's body sugar is normal, but trending down, maybe you should turn in and down. If the patient is sick and a blood sugar is really high, maybe you should push the insulin a little bit more aggressively. I think those are not too controversial, so definitely could influence practice. I think the sort of future steps would be, you know, as I said, there's almost no perspective trials of RL. And I think if we think about the long-term future of reinforcement learning and healthcare, I strongly, strongly believe that reinforcement learning is going to become a widespread presence in healthcare because there's so many examples of decision making that we as clinicians are just we're not that good at. And I think it will come, but I think it will come late, if that makes sense. So, you know, healthcare famous is one of the last to get any technological innovation. I mean, there are still hospitals all throughout the world that use paper and paper notes, paper records, and we're in, you know, 2026. So, what I suspect will happen is that we'll see autonomous AI in other bits of daily life. So, for example, in London, we're having rollouts of both Waymo and Wave, so autonomous taxi service type vehicles. I think when we have many more self-driving cars on the road, when we are seeing a lot more autonomous robot behavior in daily life, at that point, people naturally wonder, well, shouldn't we be doing this in healthcare as well? But probably not before. And I suppose what I'd like to see is I'd like to see these successive rollouts of these types of policies, like the one that I trained in my final chapter, where you sort of do, you know, maybe like a silent trial, just get the model to make some recommendations and get some clinicians to say whether they think it's safe or not safe. But you never affect a patient's care. And then if that's looking encouraging, maybe we do a human in the loop trial where clinicians get the opportunity to decide whether to act out a recommended action or not. But yeah, longer term, I think the trend needs to be not to so much to change clinical practice being done by clinicians, but rather to actually devolve or delegate certain aspects of practice completely to autonomous AI. That's why I think long term, the field is ultimately going to go. Okay, so if but but if you had a choice right now of deploying the policy that you had built for insulin or rather distilling insights that you found and updating what the clinicians are doing, like would you would you pick the human interpretable route or how do you think about that? Yeah, I mean, I think that's really interesting because what you can say about clinicians is that we, although that isn't anywhere near as much evidence in medicine as people might like to think, clinicians are very responsive to evidence. They really they care about trial evidence, they care about seeing the evidence for things. I suspect my research on its own would be analogous to sort of case study level research as in probably not something was going to change most people's practice. The thing that would probably change people's practices is a formal trial. And if you're going to do a trial anyway, then I suppose you're asking the question, do you want to do a trial where you've just adjusted these pre-specified protocols using the distilled knowledge from the model or do you want to do a trial using the model itself? But I suspect at the moment, I mean, for me, I would have no problems with adjusting my practice from what the model is recommended because I think what the model has recommended is very very reasonable and not it's not something that would be controversial, but clinicians can be quite wary creatures in terms of formalizing whether something is best practice. And I suspect that the the expectation would be that the evidence that would change practice would come from a trial. So one way or another a trial would have to be done. I guess it's just how you want to go about conducting that, either with an adjusted protocol or with sort of an all-round model itself. So your thesis represents years of hard work, lots of insight for the community. What are you going to do now now that you're just about done, Thomas? What are your plans going forward? Yeah, I think that is the big question. What are we going to have a little bit of a break from just have a holiday? And then after that, I don't have fixed plans. I have ideas. One of the ideas is that, you know, I think maybe I would like to try and be the person who takes this sort of research forward into like a prospective deployment type setting in the UK. But there are a lot of barriers to that. It's going to, if I did do that, it would be a long and hard road. So I would just need to spend a bit of time thinking about whether that's what I want to do. And I know that some people when they get to the end of their PhD, very much welcome the breath of fresh air that is a completely different, you know, problem setting. And I certainly sometimes think about that. Other than that, you know, I think I'm open to opportunities. I think I've loved clinical practice. I've absolutely loved it. But I'm also, you know, I'm also in love with the RRL side of things. And I suspect I'm probably happy to hang up the stuff that's open and pivot to just the AI stuff full time. And I probably will be AI in health because that's where my skill set is kind of best utilized. But the question is, you know, is that going to be an academic role, industry role, startup role? I haven't fully figured that one out yet. And sometimes it can be frustrating working in health care, seeing that things don't move as quickly as you'd like them to. Just because of the nature of health data. But when you can get things to work, you know, you can have such a big impact on people, society, et cetera. And as I said before, it is just such a challenging area to work in. You know, it's got so many technically interesting, complex, difficult things to work at that I probably wouldn't mind. Continuing, doing more health based RRL. But yeah, I think I don't have one fixed answer to that question other than that I'll probably continue focusing on on the RRL and health stuff. But it's just what the right setting for that's going to be. Now, we did get a chance to speak to Harvard professor finale Doshi Belez about RRL and health care. And she was she was commenting on how different countries have different approaches to this in terms of their Imaginati of the data and their ability to move on this stuff. So I guess but how do you have any comments about that like how that? No, you're in the UK, right? So how the UK system Versus the American system or she'd mentioned Singapore You'd mentioned trials in China So just yeah Yeah, no, I was gonna say are you I unsurprisingly finale is obviously completely correct in saying that I sort of sometimes joke to people that The UK is like a great place to do health AI research and then the US is a good place to try and implement the health AI research But then they're both lacking what the other one has so in the UK obviously we have this national health care service and we have growing interest from the public and the government to try and figure out how we can get Research done on Large amounts of data to improve the care that we're giving to patients on the flip side Because the UK is responsible for so much of the health data. It's probably prone to being overly conservative with it by which I mean, you know Humans are hardwired To feel that if someone does nothing and something bad happens then no one person is at fault But if one person signs off on People getting access to data and then something bad happens then, you know that person is at fault Which means that people are generally incentivized to Say no to a lot of sort of research applications Access to data et cetera, et cetera So it's a very kind of conservative mindset from from that side of things Whereas, you know the perspective I get is other countries are a bit more willing to to move quickly Which I think is good actually I think that if you're moving slowly or missing out on opportunities to really improve people's health and sticking with the status quo for 10 years Potentially causes a lot of I'm a lot of avoidable harm if you're sacrificing 10 years of of progress But it does depend as you say on on which countries you're working in I suspect that the UK It's probably a better place to be working in from that side of things just because We're a bit better getting access to that data But the question is whether the right people can be convinced to start implementing these types of models And we'll see you know as I say I think Society in general will start to change its attitude when there's lots of RL everywhere else So I love to ask yes About lines of work that we may know and mentioned yet that you find interesting That the that you want the audience to be aware of and anything like that Yeah, I think the the two things that come to mind Neither of which really have anything to do with language models But the first is I really loved the the continual learning Paper that came out of Rich Sutton's lab by one of his students I just thought that was so cool this idea of Having models where you'll you know that you're ending up with So dead neurons in the network Having the sort of pasticity where you can continually identify Neurons that are really consistently not contributing at all to any of your outputs and being able to reset them and reset the pasticity I thought that was just such a cool bit of work And I'd love to see more stuff like that Other things that I've been interested in recently I Have had like a an interest from a distance in More neuromorphic architecture type stuff spiking neural networks I think there is this big question around energy usage of these massive neural networks and Where the future of that is going to go and I suspect the neuromorphic architecture is probably still haven't had their heyday yet but it probably will come in due course and then Just in general, I'm really excited about the advent of Start-ups that are going to focus really on like RL as a way of Trying to achieve a GI, so things things like obviously David Silver et cetera I think that to me is that as a doctor as a biologist that's so Intuitively appealing, you know, the only examples we have of general intelligence are what we see in nature and in that setting it evolved as a way of basically Doing movement better making actions more effectively in order to reproduce you know pass on genes. So actions around Finding food making sure you don't become food reproducing et cetera and so to me the idea of trying to achieve AGI through like an action-based perspective it just It makes sense because that's the only way we've really achieved not we but the anyway nature has achieved AGI whereas stuff that's like world modeling But without sort of an action RL-based element to me feels like it's putting the cart before the horse So you know, it's not specifically like any one research paper But I'm I'm excited to see more more of these startups that are at a RL based Dr. Thomas Frost. This has been fantastic. It's been a long time coming. I've been looking forward to this episode for honestly a couple of years at least and It's it's been fascinating here about your work Congratulations in advance on your PhD and What so when people are citing you are they citing your papers or are they citing the thesis? Yeah, well, I guess First of all, thank you. This has been there's been a lot of fun I love talking about my work and and I enjoy all of our conversations. I think if people are citing me The mortality papers been published so they can take to look at that the other two papers are under submission But they're a pre-prints available, so they're very welcome to cite those the thesis will probably not come out for another Yeah, I'm probably like two to three yesh months or so But yeah, they can definitely the main bits of work minus the insulin policy stuff The main bits of work are are out there. I have a Google scholar profile that people will be able to find fairly easily So yeah, I just encourage them to to take a look at that and yeah, thank you again. This has been so much fun It's been great Thomas. Thanks again, and we'll have links links to the key papers on the and your profile on the on the episode notes. Thank you Thomas Frost