"Improving Systems of Care" with Michael Barkham, Ph.D. and Jaime Delgadillo, Ph.D. - s1, e3

[Music] Welcome to Making Therapy Better, the podcast that brings together some of the top minds in psychotherapy as well as everyday clinicians to talk about where the field is headed and how we can achieve better mental health care for everyone. Making therapy better is hosted by Professor Bruce Wampold, who has dedicated his career to understanding how therapy works and advocating evidence-based methods for improving outcomes. His guests today are Michael Barkham Ph.D. and Jamie Delgadillo, PhD. Michael is a professor of clinical psychology at the University of Sheffield, UK. He has spent the past 35 years promoting the measurement of psychological therapies and routine practice, and in 2019 he received the Senior Distinguished Research Career Award from the Society for Psychotherapy Research. Jamie is also a professor of clinical psychology at the University of Sheffield, and director of the Psychological Therapy Research at R-NHS Foundation Trust. He has 20 years of clinical experience and has published over 90 scientific papers and book chapters in the field of mental health. Making therapy better is brought to you by Care Paths. Care Paths has been helping in-person and virtual therapy practices thrive for over 20 years with a complete web-based EHR and Practice Management platform, as mental health care evolves, Care Paths is leading the way in making measurement-based care easy and cost-effective for therapists. Visit Care Paths.com to sign up for a free trial today. And now, without further ado, episode three of Making Therapy Better, improving systems of care with Michael Barkham and Jamie Delgadillo. Jamie and Michael, this is a great opportunity for me to talk to you guys about the research you've been doing and the implications for mental health care. So I'm excited to have this conversation. I know in England there's a program to really provide mental health services to those people that are suffering through the program called Increased Access to Psychological Treatments, iAAPT. So you've had access to data generated by this program and we want to talk about that today. But first, let's share with our audience about what iAAPT is, how it began, how it works, and more generally what it does for people. Well, probably Jamie can talk best to what it actually does and how it works because he actually worked in an iAAPT service. But maybe I can kick it off because I was involved in an initial evaluation when it was started up. So maybe if I start with that and then Jamie can run with how it actually in reality works. And in some ways it's actually quite interesting, particularly for Americans if lovers of history and post and tea party, etc. But actually the source of it in some ways goes back literally to a tea event, tea party at the British Academy. 2003 is when it's supposed, the idea is supposed to have germinated. And the British Academy is basically a learner society. And two folk were in the queue for tea. One of them was Richard Layard, Lord Richard Layard, who was an economist and an advisor to the Chancellor of the Exchequer. Okay, so has direct access to funding. And he has an interest in happiness and a concern about the mental health of the nation, etc. And behind him, supposedly, is this person, David Clark. And he tells David Clark and says, do you know anything about mental health? And David Clark says, yes. And it's interesting because that's actually the germination of it, which is basically Richard Layard has this vision of how to deal and address with the mental health of the nation in some ways, unhappiness, particularly depression anxiety, which he knows as an economist is costing the country, he quotes 46 billion pounds or whatever. David Clark has the evidence for the fact that there are effective treatments for depression. And so the point is that Richard Layard between them and David Clark realized, okay, we've got a problem. We've got a we've got treatments. It's not as if we need new treatments. What we need is a vehicle for the delivery of those treatments to the population at large. So basically because Richard Layard is so well connected, right at the very top, George Brown, he later became Prime Minister. He put forward a strategic document in the beginning of 2005 to make a bid for a new program of delivering psychological therapies, which actually became IAVT. And what they did was they took two, what they called demonstration sites, one in New Uman, London, and one in Donkester. And they tested out various forms, which I won't go into detail, but basically out of that configuration became IAVT. But what's particularly interesting is that if any of us psychologists were about to instigate a program, we would run pilots, yeah, we would run a pilot program. They weren't pilot programs. They were demonstrations. In other words, they were absolutely committed to this idea from the get go. And the purpose of these two demonstration sites, New Uman and Donkester was to show that actually they could deliver psychotherapy, psychological therapies, effectively on mass in a way. So the whole point was about improving access. I don't think, and Jamie might sort of have a different view on this, I don't think it was ever really the intention to improve, to devise a new model of therapy. It was actually to find a vehicle for the delivery of therapy in an effective way, which actually is a model of stepped care. And I mean, I think Jamie can then sort of talk about how a stepped care works in that kind of way. But I just think it's actually interesting that background history. Because really, if you think about it, it's really the sort of a mixture of serendipity of these two characters meeting up in a TQ, okay, access to government right at the very top and then money to actually deliver it. So IAVT is not something that just happens. Needed that configuration of all those factors to make that kind of happen. So yeah, that's my understanding of how it actually, and we were involved with Glenis Perry, who led the program to sort of evaluate that those two demonstration sites, which is why kind of, it's buried in my brain rarely, the operations of it. Well, before we get to Jamie, I just want to emphasize the point that you made, we have treatments that work. We know psychotherapy is affected. The issue is access. And here's a program, as you say, we're not going to develop new treatments. We're not going to do things in new innovative ways. We know what works. We need to get it into the hands of those people who are suffering. So Jamie, your experience with IAVT because you kind of, in a sense, grew up there. Well, picking up on the expression of a demonstration site that Michael pointed to, indeed, the intention of the program is to scale up and improve access to empirically supported interventions. So this was not about demonstrating whether the interventions work or not. It was to demonstrate whether it can be scaled up to a routine care population. So how does it work in practice empirically supported interventions that have been demonstrated to be effective in efficacy trials and prior studies. We're organizing a step-care model. We're a by at least in theory, the majority of patients with common mental health problems would initially access a brief and low intensity intervention that may be effective to alleviate their symptoms and improve their functioning. Based on the expectation that these interventions could be cost effective in the sense of offering the least expensive and least restrictive intervention that could lead to some functional improvements. Those patients for whom these brief interventions are not effective then have the opportunity to be what is referred to as stepped up. to more traditional empirically supported psychotherapies. And there is a list of these treatments that have been established to work for the treatment of depression, anxiety, and common mental health problems. In the literature of this step care system has been described as a self-correcting model. Where by eventually people receive the intervention that meets their needs, albeit with some to agree on certainty about which treatment might be best for which individual. And it has been argued that it is efficient, cost-effective way to organize scarce resources for large populations. Indeed, this view is backed up by empirical evidence of step care clinical trials where this model has been compared to control conditions such as usual care or other alternatives that were available at the time and the studies that investigated this. So there's a pretty good empirical basis to establish this model. And indeed the demonstration site that Michael was referring to show that in the implementation of this model in routine care, you observe the kinds of pre-post treatment to fact sizes that you tend to observe in efficacy trials as well, where that data is available, where pre-post treatment effect sizes are used as benchmarks. So far, so good. But those of us that have, as you put it, Bruce, grown up in the system. So I did my clinical training CBT some years ago when sponsored, in fact, by the IAP service that I was working in at the time. So I've been working in the system for about 12, 12 years as a psychotherapist and clinical supervisor. Now, those of us who see it from the inside also noticed that there are some problems. One of the problems that has fascinated me, and I'm sure that Michael is also very-- - Jamie, let me interrupt just for a second. 'Cause I want to just establish one fact before we get to the problems, which I think are really important. Because this isn't just a rollout to a few clinics. This is a national program. And you're talking about the model being cost-effective and that the effects of the services being offered are effective. And I just want to emphasize that point if you agree. This is working. People who use the service, the step-care service are at the population level benefiting. And they're benefiting to a degree comparable to what we see in clinical trials under the best conditions. Is that right? - Yes, that was the claim at the very start of a program that were the conclusions of the initial demonstration sites. And 10 years later, very many studies have been conducted within this environment, within the IIA program. You might think of these as practice-based evidence that arises through the system. Also confirms broadly speaking the same, let's say, headline. I've got some figures for you to discuss this in more detail. For example, one of our doctoral students, Sarah Wakefield, conducted a systematic review and metanalysts of practice-based studies arising from the IIAP program in the last 10 years. This was published in the British Journal of Clinical Psychology in 2021. The high-level findings are that when you aggregate pre-to-post treatment effect sizes for depression and anxiety symptom measures, the effect sizes that come out of the metanalysts are for depression in the range of.87 on co-endee scale. And for anxiety,.88 tone co-endee scale. Now, how does this compare to the wider literature? More recently, one of our doctoral students, Chris Gasko published a large-scale metanalysts of practice-based evidence worldwide. This was published this year in the Journal of Administration on Policy and Mental Health and Mental Health Services Research. This included metanalysts of over 250 studies. When you pool data for those that report depression outcomes, pre-to-post treatment effect sizes, the pool effect size for practice-based evidence on depression is.96. And for anxiety, it's.8. Now, when you look at the confidence intervals around these estimates, they're very much overlap with the data that's coming out of the IIAP practice by this study. So what can we conclude? It appears that the effectiveness observed in the IIAP system is largely comparable to international benchmarks. Of course, we all know that these are aggregated group-level means that mask huge variability. And that word, that expression of variability in treatment outcomes has been very much a topic of interest for both Michael, myself and other colleagues in our department. Yeah, I think we're going to get to that because your efforts have been aimed at how can we improve the quality of service because we do have this variability. But before we get to that, I interrupted Jamie because you were saying, well, there might be a few problems as well. So let's talk about that part. OK, so headline figure, this stuff works. It seems to work as well as we might expect, at least by comparison, to benchmark some selected clinical trials, which was the method in the demonstrations studies. More recently compared to the wider literature practice based evidence internationally, we find very similar comparable effect sizes. So far so good, this stuff seems to work. However, it doesn't work for everyone. And it works much better for some patients compared to others. So when you zoom into this topic of variability, we start to notice that there are considerable problems. Just a name of you. Many people drop out somewhere between 25% to 30% of people drop out early, typically during the first three or four sessions before they have an opportunity to maximize benefit from available interventions. Now, the dropout rates in the IAP system are somewhat comparable to the dropout rates of systematic reviews of psychotherapy in general. So the problem isn't more acute in this system than an elsewhere, but it's certainly a problem that's particular of psychological services. Other problems are that some patients derive considerable benefit from these interventions in a very short space of time, and they stay well, impressive. But of course, other patients don't benefit at all, despite having several interventions in the stepped care pathway. And some people who apparently benefit from these interventions have relapse within a year and come back into the system. Oftentimes more impaired than they were in the first treatment episode. So these are just a few examples of some of the problems that we've noticed, and we've been especially interested in understanding the sources of the variability in these types of outcomes. I hope you have other questions. If I can, I mean, just to capture particularly the sort of that kind of more kind of not quite ethical, but just a subprincipal point out the step care model, is that I mean, it works well, particularly where people can get access very quickly to the step two, which is really the kind of innovation in many ways, compared with step three, which is the more traditional. But for those people for whom that doesn't work, they have to in effect fail experience failure to that in order to get stepped up to step three, which is the more traditional, what's called high intensity treatment. So you question a model, notwithstanding everything that it's sort of done in a sense well, or maybe there's no model that can actually sort of satisfy everybody, but you question a model that actually is principled on the fact that enough people have to fail in order to kind of get. And they've got to kind of deal with that failure in order to then find themselves with a treatment that they think they then got to wait for again for step three treatment to work. So the vehicle, so to speak, has been spoke about it, has been this is in some ways a bit clunky. And for some people, as Jane is saying, is probably not fit for purpose for particular people in that kind of way. But you can't take it away from the model, given that this was thought of virtually 20 years ago. Everything changes. And maybe one thing that I apt has to do is to actually sort of begin to question itself and be able to look at how can it modify itself. And of course, the other kind of major feature, which is worth mentioning, is that one whole component of it was that it introduced sessional measures. So that-- all the patients attending, whether it was low intensity or high intensity, completed PHQ, 9th, GADSET and the work and social adjustment scale. And in a sense, I think that that was really using the expression "a step change" in terms of access to data, which actually provided so much sort of golden riches really for the likes of Jamie and myself, and other people. It was an absolute sort of gold mine for researchers to investigate psychological therapy. So I've always kind of talked about eye-act as being the biggest social experiment in psychological therapies. And I think that's absolutely the case. But it's also the biggest opportunity in terms of data provision and big data to look at how psychotherapy, how psychological therapies work over time across the course of treatment. In that sense, it's probably one of the world's most impressive exemplars of the concept of a learning health system, where you have constant accumulation of data in an intensive way that enables you to interrogate how the system's working and enables you to do natural experiments within the system, or pragmatic trials, to see whether innovations or changes may make a measurable difference to patient's outcomes. Well, I think that's one of the commendable features that people around the world really recognize, that it wasn't just increased access, but it was assessment of how we're doing, which to be clear is led to both scientific publications that are important principles of how people change. But also, then, gives you that information, how can we tweak the system to make it better? Yeah, and just to say, I mean, in terms of that, I think David Clark has written using the word transparency. And of course, it's not just that this data is collected and completed by people, but it's actually made public. It's actually been made public every quarter of each year and published on the website so people can actually access it. So this is actually publicly available data in principle, although there's whole issues about what level you get access to. But in that way, I think, you know, from a from a research point of view, it is a real revolution in terms of being able to kind of look at the process and outcomes of psychological therapists. So just one editorial comment from me, because Jamie, you're talking about dropout, you're talking about unsuccessful cases, you're talking about some patients needing additional episodes of care. We should really put a context to this because even in medicine, we have these interventions that we all assume are uniformly effective. But if you look at the data in cardiology or in respiratory diseases, the efficacy of those treatments, many of which are very, very expensive, are not universally effective. So that doesn't mitigate against the this idea that we need to improve. But let's be clear that the services that we offer in IAC, but all around the world, are remarkably effective. And I think we always have to keep that in mind. Good point. Could they be better? Sure. That's where that's where the that's where the focus of the practice-based research is. Yeah. So this big data, let's explore that a little further because you have, as you say, every patient is completing measures session by session. So you had this remarkable data set. And now, you know, around the world, there are other people doing this, but clearly this is probably the most comprehensive and one of the earliest efforts to actually measure the effects of psychotherapy and psychological treatments. So using this big data to improve the quality of service, talk about some of the things you've learned and some of the innovations that you've discovered implementing and that we could benefit from that knowledge. Jamie, do you want to. Well, where did it start? There are several sources of variability in treatment outcomes, many of which we've been very interested in examining. For example, variability that may be attributable to therapists, variability that might have to do with the patient's features, variability that may have to do with process in the therapy, variability that may have to do with contextual features, where people live, really interesting. So where do we start? Maybe as you touched on the session by session, data, I could say something about the feedback related studies and we can build from there. So as you explained, Bruce, we have access to session by session outcome measures on three domains that Michael referred to earlier, depression and anxiety and functional impairment. This enables us to aggregate large data sets across very many patients to model typical trajectories of symptomatic change over time. And we have followed some of the literature referred to as progress feedback or outcome feedback, which is very well established as an effective quality assurance system that enables you to track whether your patients are responding in a typical way, as you might expect, based on clinical norms from other patients with similar symptoms. Or whether a patient may be showing signs of stagnating, so not improving, or even deterioration. The latter case is referred to in the literature as cases that are not on track. So one example of the work we've done is to design outcome prediction methods using session by session data to generate clinical norms of the expected trajectory of symptomatic change over time. And we've developed computerized systems that in real time alert therapists the cases that are not on track. We've done studies in the same. Yeah, well, you certainly have. I mean, the research on whether it's called measurement-based care or routine outcome monitoring, very convincing studies that you've published that providing this information to therapists improves the quality of care. So is this a mandated for all therapists in the IAPT system, or is it still in a pilot or demonstration stage? So there are two sites to this. What is mandated is routine outcome monitoring. So the patient reported collection of measures of depression, anxiety, and functional impairment. It is a standard feature of the system. You can overlay the technology to alert therapists to the treatment response that they can expect from each individual. This isn't mandatory, but in the work that we've done, we've taken a typical incremental, let's say a developmental program of studies. First, to design a prediction system and to evaluate whether we can trust the predictions accuracy or not. We then moved on to doing qualitative work, including patients and therapists to assess the extent to which this may or may not be acceptable and to learn what we need to improve to make it more acceptable. We then did a field test and followed by a large randomized control trial. The punchline of all of this is that the method is broadly speaking acceptable and it can improve treatment outcomes. However implementation is difficult. You typically get very enthusiastic and committed practitioners that embrace the method, use it well, and use it effectively. But you get other practitioners that are less enthusiastic about it and don't use the system. So we're now at a stage of implementation related research, where the system that I've been involved in has been already implemented in several IEP services. And we are now investigating how we can enhance the implementation efforts across other services. Yeah, I think it's important in terms of people listening to this, particularly outside the UK, trying to understand that as Jamie is saying, the feedback system is not mandated. IEP is a structure that's very difficult to add things to or to change. So the work that Jamie's kind of being leading on feedback is about via negotiation with services and getting their agreement. And in some ways that defines a feature of our particular service, those services that really want to engage and take part and take a step further, whereas there will be many other services nationally who are just hard pushed and managed the system as it is because the AIAT system is very demanding on therapists, particularly step two, the psychological well-being practitioners are really so run off their feet in that way. So probably for many services, there's a kind of hesitancy or reluctance to take on any further kind of research demands because everything is a demand, a button beyond what they already have to do. So it's not as if the AIAT system or services are all sitting there waiting to be candidates for further research. And I think that's where Jane has done such a brilliant job identifying those and building collaborations with those people. And a lot of this has been done on no additional research council funding, for example. It's got IT funding and industry back up, but not your classical research council funding in a way. Yeah, I mean, this really speaks to institutional difficulty because we know Mike Lambrut and Scott Miller and Barry Duncan and others now, I guess decades ago, shown that under the right conditions, these feedback systems do improve outcomes, particularly for those cases that are not on track. But to implement that throughout a system of care because the implementation is important is not that easy. So these comments kind of motivate me or instigate me to pursue one area of your results that I think are really interesting. As I recall, and correct me if I'm wrong, you see variability, and Jamie, you talked about the variability is where we have to look. Variability in outcomes among various clinics or services within AIAT. So some clinics are doing better in terms of their outcomes than others. So talk about what you think of the sources of that variability. And I think you mentioned some briefly, but I think this is fascinating because we tend to think about which treatment is given or even, and we'll talk some more about therapists, but about which therapist, but the clinic seems to make a difference. Michael, do you want to take this one on variability at clinics and maybe that dovetails with therapists? Well, I think probably in terms of the variability or what makes effective delivery, I think one of the better sources of that is actually David Clark's kind of paper where he generally looked at masses of data, was it half a billion, you know, at a kind of organizational level and identified those factors that actually make for an effective delivery. In other words, he was arguing that we should move beyond new treatments, etc. and it's all about how treatments are delivered within a service. So he was going for the kind of organizational level. And in a sense, I think my reading or memory of it was really that there was nothing kind of sort of that you wouldn't expect in the findings, which is basically, for example, to know what the presenting problems are within the population that's been presented. Otherwise, you don't know what treatments to kind of deliver. You know, you minimize your wait time period because that's a clear, that was one of the driving dissatisfactions that led to the instigation of improving access. You minimize sort of cancel to appointments, etc. And you give sufficient dose of therapy. You know, so the conclusions that he came to, but what that led to sort of more effective delivery are not sort of revolutionary in that kind of way. They're the ones that you actually would expect to get. And when we did the evaluation of the demonstration sites, for example, we found that there was one of the biggest differences in terms of outcomes was, for example, people completing their course of therapy. So, you know, that I think there's a lot in the system that we've known for a long time that actually perhaps hasn't been nailed down and adhered to in terms of, you know, sufficient dose, you know, in terms of completing treatments, these kind of factors that actually gives you a sort of a sound basis for the delivery of whatever model of therapy it is that you're delivering. Above and beyond then, therapist effects, which, you know, I think is an important contributor together with, yeah, you're going to get that. We're going to get to the therapist. But there's also, but there's also, we know that there are kind of, there's a strong sort of socio-economic deprivation argument as well. And in fact, when in a paper, we did big colleagues looking at data from the eye out, one of the biggest differences in terms of outcomes. And I think it was something like 17% difference in outcome was in terms of social deprivation. So in other words, the social context in which people live is one of the primary indicators. So we spend a lot of time looking at sort of the minutiae of within therapy sessions and all this. And yet there's one of the biggest contributors to determining outcomes is sitting out there in the real world. Right, right. The social context and the deprivation. I agree, Michael, we tend to think about what happened in the session. And we realized, I mean, there's so much work in sociology about neighborhoods and the effective neighborhoods on people's functioning and well-being. And yet we tend to think of patient problems at an individual level and look at kind of what's going on inter-psychically rather than socially. Yeah, well, we have a colleague Nick Firth who's done some wonderful work looking at exactly, you know, the therapist clinic and neighborhood effects. So that is all on its way out for dissemination. So the one part of this which I'm interested in and I didn't hear you talk about, is the actual work environment for the therapist. What is the climate within a clinic or service? What is the rate of burnout and pressure on therapists? How satisfied are they with the work they're doing? And those effects on patients. Okay, so yes, this is another interesting contextual factor maybe. So you can think of context as in neighborhoods and the socioeconomic aspects and how they impact on patients. But you could also think about organizational climate and context. So one of our shared interests is therapist effects. So Michael, myself and another colleague Dave Sachs and collaborated some time ago in the study where we collected data on the therapists, job satisfaction and occupational burnout levels, self-reported by therapists. We then followed up all of the patients they treated over the course of one year. And we found significant associations between therapist burnout and job satisfaction with their patients' outcomes. Yeah, yeah. Such that therapists with higher indices of burnout, particularly a facet of burnout referred to as disengagement, had poorer treatment outcomes. And those that were less satisfied with the job's poor treatment outcomes. So that made us curious about the organizational aspect and the pressures of the job and this whole area of occupational health and well-being. Since then, I've collected data in another study, but we failed to replicate this finding. So in a follow-up study, which is currently under peer review, so it's not yet published, there were no significant associations between therapist-reported burnout and job satisfaction with clinical outcomes. So it's still mixed evidence. Nevertheless, we think that therapists' occupational health and well-being in itself is certainly an important outcome of interest, both from our research perspective. What can we do to make the job more? But less difficult for patients who, for therapists who experience burnout. And whether improvements in occupational burnout at the therapist level trickle down or have an effect on patients outcomes is yet to be demonstrated. But it's certainly a very interesting work while area of inquiry. So let's get to this issue with therapists because it keeps coming up in our conversations. We know and and some of the best and most convincing research has come from these data that. The right from I act in the national health service. There are there is without a variability in outcomes due to therapist. Some therapists consistently get better outcomes than other therapists. And as I say this, I see the graph. Michael, that you guys published with the green and the red therapist always in my mind. So therapists are not interchangeable. What are the implications for a large system of care for this variability among therapists? Michael, do you want to start and I'll follow. Yeah. Well, backtracking is just slightly. I think one of the frustrations of all the work on sort of therapist effects and the variability is that so much of it is. And in a sense anonymous in some ways we don't know actually who these people are. You know, I can go back sort of 20 maybe even 25 years. When we're building up collecting initial development of the clinical core outcome. So we're writing data from patients is fine. But actually therapists have been continually resistant about giving information about themselves. And I think that is one real sort of block that we have in terms of. So in fact, you know, we end up with these. And we're talking about the kind of data showing, you know, groups of therapists being twice as effective as other groups of therapists consistently. As the main point, but actually who those people are and what they're actually doing is the real stumbling block and the glass ceiling. I think that's that's in research at the moment. So I mean, I think that that's that's really what I think is needed. I mean, I think there's work coming, you know, your chapter is brilliant chapter in the handbook, you know, looking at the characteristics. But I still think that we actually need to go a real step. Change in terms of being able to access and look at what people are doing that makes them. Some people so much more consistently effective than other people. And I think that's that would be my one dream research project. I think that will be left on me if we could find out that. So because I think gone. Yeah, essentially we have this large variability and Jamie, you know, that's that's where we're looking. What is the variability in outcomes and how can we use understanding of that variability to improve quality of care. And here we have therapists, as you say, Michael, that's a large source of variability. But because of the way the system set up because of the way research is conducted, we don't really know very much about the therapist. And even if we do, it's difficult in a system of care to intervene in a way. So maybe I would rephrase the question, say in an idea where we knew who the therapists were and what they were doing. It's a thorny question to think about how we get improved service given that knowledge. So as a thought experiment. What could we do? I have two replies to this. It's on it's on controversial now that substantial proportion of variability outcomes is due to differences between therapists. This is very well established. So the question is, what can we learn from the most effective therapists that we could then try and instill or encourage. And we can implement somehow in the rest of the workforce or indeed in our trainees and the new generation of therapists. So that might be a perspective on understanding either the states or traits of effective therapists are these more stable characteristics that define the high these effective therapists. So that's fluctuating states, for example, when a therapist is burnt out, they may be less effective when they are less burnt out, they may be more effective. Now that's one angle you could take. There's an alternative perspective on the therapist effects, which is an interesting position to have. Even if we don't know what characterizes the high effective therapists, we can actually reduce the variability between therapists using feedback. This year published a meta analysis of clinical trials conducted in the United States and led by Michael Lambert in his group, where we found that when you implement progress feedback in university counseling settings, mostly in these studies. And then we can reduce the therapist effect. In other words, the variability between therapists becomes constrained becomes less of an influence on the on the outcomes of that particular cohort relative to usual care without feedback. And in other way, all you have to do actually is introduce feedback supported and ensure that it's well implemented in a system. So that the who your therapist is. And determine the outcomes of patients. So you can reduce variability. You can reduce it, but it's the very early still there, is it not? You can't eliminate it. But reducing it is probably a good step forward in the sense that the who the therapist is becomes less of a lottery. And I just can I just say here as well. In terms of that, we've actually kind of replicated that empirically within an IAP service with a PhD students just completed where going through a whole series of programs and the final one was actually devising a bespoke sort of deliberate practice light really in terms of the skills. And so we have this evidence of exactly this effect of reduce reducing the variability. And actually the outcomes increasing slightly so not at the cost of reducing the outcomes. The outcomes take the same improve slightly actually, but the variability across the therapist and these are the same therapists we try to over five years for different interventions. And that bespoke deliberate practice light reflective practice intervention actually reduce the variability. Right. So there's some intervention. At a service level that worked. Well, here's my concern. And that is in the research that Simon Goldberg did with the therapist who receiving routine outcome monitoring feedback didn't improve over the course of their career. So might reduce the variability, but the critical question is how do we help maybe all the therapists gradually improve. Because it doesn't look like just receiving feedback. Really results in therapist improvement, which is not surprising to me because we're giving feedback at a pretty global level. If you look at the idea of deliberate practice, you need feedback about particular lease particular skills that are necessary for performance. And maybe what you then have to get into is the realm of sort of matching better matching. So that actually, because I think this is all about trying to, you know, we have effective therapies we have broadly effective therapists, etc. And what we're looking to change is a whole lot in this system we're not going to change. So I think a lot of what we're looking for are the finer tunings that actually to try and how can we tweak this system to make things a better fit so that actually, you know, person X gets the treatment, which is most appropriate for them with the therapist and it's most appropriate them. And we can start doing that kind of configuration in a more informed intelligent way sort of smarter assessment smarter assignment. then I think there's a way there where we can actually move therapists forward. without doing anything really revolutionary in any way in that kind of way. But just doing better. And I think that's partly the way of the strat care kind of design of actually trying to decide what who is best for the low intensity and who can go straight to the high intensity and not having to go through that experience of failure to get to the treatment, which is most appropriate for them. And I think if we can sort of start configuring those parts of the jigsaw together, then I think we will potentially improve the outcomes of therapy. But there may be an absolute limit on what we can reach anyway. So I think we need to have reasonable expectations of our outcomes. Yeah. So what you're really describing is precision mental health services. Can we learn from the big data how best to match patients to the therapist, to the type of treatment, to the intensity of treatment that's going to result in optimal outcomes? And I know you guys are working on this. So you have the data to do it. And I think that's pretty exciting. Your last comment, Michael, really comes back to. Let's remember where we started. And that is mental health services are remarkably effective. And so what we're doing and talking about here is tweaking the system through feedback, through matching in a way to marginally improve the outcomes. Now, I use the word marginal, not in a derogatory sense, because you guys have written this as a small change in the benefit for particular patients aggregated over the entire service is a remarkable benefit in the mental health. And there's an additional angle to this, which is the cumulative benefit of small incremental changes. So in the feedback trial that we did in eye-up services, the effect size favoring the feedback intervention is 0.2, small effect size and conventional terms, but scaled at a large population level, as it's clinically important. Now, if you do that, you bring in feedback into services. Then you bring in stratified care, where you use data to determine which patients may benefit more from a high intensity dose instead of a low intensity dose. And you match patients to the dose of treatment. The effect size that you will get for that is around 0.15. Some people might argue that that's negligible, but if you add those in a cumulative fashion with the effects of feedback, you've got 0.35 additional enhancements of therapy by bringing in smart decision tools. And so on, you then match patients to the right therapist. And you get another incremental benefit. Overall, in the long run, through tweaking around the edges, you could potentially, you could potentially add the effects of therapy. Never eliminating error, never making therapists completely effective for every patient. That's probably unrealistic, but probably better than they are today. With how to doubt. Let's see, I was thinking something where you're talking about the incremental care. I mean, that's really the challenge is how do we keep learning from the data we're collecting to improve quality of care? So it seems like the biggest challenge is what I wanted to talk about is implementation and scaling up. So how do you do that? Because you talked about the difficulty of the different services acceptance of feedback systems, for instance. So what have you learned about the scaling up and the implementation part of this? Besides it's really difficult. Yeah. I mean, I can only-- I'm not in an AI app service in that kind of way. I mean, with Jamie and Kim and Wolfgang, we've just reviewed all the kind of implementation stuff in relation to feedback and stuff. And it just comes up as the consistent thorn in the kind of neck of the whole implementation. It's just the whole issue, really. It just gets to the organization, the problems, resistance, the money, the funding, all the difficulties. And I don't know. I wouldn't claim to have an answer to that one at all. I just think it's an ongoing problem, really. Well, when you think about it, Michael, psychotherapy is a very personal, one-to-one relationship. But so is the implementation. It's a relationship between the system of care or the clinic with these ideas. I know the biggest feedback effect ever found was Inganambla's OQ study in Norway. And she always talked about when you ask her, why was this effect larger than others? She said, I was the OQ mom. She was there working with the therapist, encouraging them really working with the climate of the various clinics. But of course, that becomes the weakness of it. Because it's an observation I've made about some of the studies. Some of the feedback studies still employ-- we had an international consultant. So they've had the great and the good involved in the study. And I just say to myself, this isn't generalizable. You can't keep running a system of, for example, feedback implementation relying on the great and the good, having their word and making that. It's got to be sort of-- you've got to take them out of the equation, really, for it to be a reality test that this is something you can actually use in your to be scaled up. It's got to-- you've got to take away the charismatic people involved in that. And it's got to drive itself, really, for its own good. Because the charismatic people won't be around. Yeah. I'll push you a little bit, Michael, because we're all mental health professionals. We know the importance of relationship and the personal relationship between the actors. So yes, you can't have a charismatic figure leading your feedback and form system. But it is important to think about the climate and the clinic, who's implementing this, how it's done, so that the personal relationships of the therapist and the staff and so forth with the system is built in a positive way. So I think the implementation can be systematically designed to improve that in a way that it is scalable. But I think the relationship is important. But actually, in the end, it has to be about ownership. It will only work if the people who are actually using it have a sense of ownership of that. So the charismatic person fine and whatever. But in the end, if the person doesn't own that that procedure is going to be a useful one, in the same way that you've talked about relationship between sort of therapist and patient. If the patient isn't owning the work that they're doing, et cetera, doesn't matter how good the therapist is, if the patient isn't accepting it and owning it, it's not going to work. And I think it's the same principle within implementation for feedback, et cetera, is about ownership. So that's what I want to see is actually the priority on ownership, that this is actually-- that they buy the idea. In some ways, regardless of who is actually selling the product, it's actually the product itself, not the seller. Although some sellers can be worth convincing than others. Yeah, the ownership, I think, is really important. I mean, mandating any of these kinds of practices is not going to be optimally effective. It has to be that the people using it find value in it and have that ownership, as you say. I'm noticing the time where I've been doing this about an hour. I want to say thank you in two ways to both of you. First, thank you for the amazing efforts to make psychological care accessible to people. And the work you've done and the research that you've produced, which has been absolutely astounding. funding in its kind of rigor, but also its applicability. And then to thank you for today having this conversation. What a great opportunity to talk to both of you. It's been a pleasure. Wonderful. Thanks so much, Bruce. Thanks for listening. Care Paths offers a complete behavioral health EHR and practice management software solution, including claims, billing, clinical notes and documents, scheduling, and teletherapy, all for one simple and affordable monthly price. Care Paths EHR is hip-a-compliant and ONC certified and can also support electronic prescribing for an additional fee. Their latest release, Care Paths Connect, includes automated measurement-based care and the ability to create a digital front door for your practice, as well as a free mobile app designed to increase patient engagement. If you're just starting your practice or dissatisfied with your current EHR, go to Care Paths.com to start your free trial today. To find out more about Bruce Wompole and his work as Care Paths Chief Clinical Officer, visit makingtherapybetter.com.