Go back

National Commission into the Regulation of AI in Healthcare: Recommendations. With Prof Alastair Denniston, Chair of the commission

from Digital Health Podcast- Royal Society of Medicine

47m 24s

National Commission into the Regulation of AI in Healthcare: Recommendations. With Prof Alastair Denniston, Chair of the commission

The UK has established a comprehensive national AI commission to address the urgent need for a modern, adaptive regulatory framework in healthcare. As generative AI evolves rapidly and outpaces existing regulations, the commission’s 44 recommendations aim to strike a balance between innovation and safety. It proposes a shift toward proportionate, lifecycle-based regulation—allowing lighter-touch oversight for low-risk tools and stronger, continuous monitoring for high-risk applications. Central to the framework is a shared responsibility model across manufacturers, clinicians, and health systems, ensuring that safety is embedded from design to deployment. Patient transparency and choice are prioritized, with clear rights to know when AI is used and to opt out where possible. The commission also highlights the importance of system readiness—where clinical needs, technological capability, and systemic preparedness intersect—as the true source of value in AI healthcare. While direct-to-consumer devices face unique challenges, they are regulated as medical devices, offering potential for real-time feedback. Critical to implementation are investments in digital infrastructure, workforce skills, and post-market monitoring systems. These recommendations signal a strategic pivot from rigid, pre-market rules to a dynamic, responsive model. However, turning theory into practice will require coordinated efforts to resolve ownership, funding, accountability, and human oversight challenges—ensuring that AI integration in healthcare remains safe, equitable, and truly patient-focused.

Transcription

7859 Words, 44306 Characters

English
your listening to the Royal Society of Medicine Digital Health Council podcast, where we explore health-tech innovations that are transforming healthcare, with me your host Dr. Annabelle Painter. We are at a unique moment in AI and healthcare. Generative AI has arrived in health and it's evolving at extraordinary speed, but it's being regulated by frameworks that were largely designed before we had this kind of technology that's non-deterministic, there's general purpose and that's constantly being updated. And for the UK there's another layer to this, AI arrived just as we were leaving the EU, putting us in an interesting position. As a relatively small but globally important regulatory player, what do we do? Should we follow the EU, follow the US or try to forge our own regulatory path? How do we make the UK an attractive market for AI and a home for innovators? And how do we make the most of the NHS and its scale, its data, while still protecting patients? These questions led to the launch of an unprecedented national commission on the regulation of AI and healthcare convened by the MHRA. For the course of a year, the commission brought together industry, clinicians, regulators and members of the public to advise government on what a future regulatory framework in AI should look like. I was fortunate to be part of that conversation, sitting on the technology working group, and it's been fascinating to see how the different perspectives have come together. On the 10th of September, the commission published its recommendations to government. 44 recommendations covering how we should approach regulation of AI and healthcare. The commission was chaired by Professor Alistair Deniston, an ophthalmologist, academic and professor of regulatory science, and I'm delighted to have him with me on the podcast today. We're going to unpack what the commission found, what its recommendations mean, and perhaps most importantly, what they could mean for the future of AI regulation and innovation in the UK. Alistair, welcome to the podcast. Hi Annabelle, thanks, it's great to be here. It's such an honour to have you here Alistair, I'm really excited to talk to you about everything to do with the commission and the recommendations about the regulation of AI and healthcare. But before we jump into that, it would be great to hear a little bit more about you and to understand how you came to be sitting in the position as chair of the National AI Commission. So tell us about your background and what's brought you to this point. Thanks Ava, so I'm an NHS doctor, I'm an ophthalmologist at Happens based at University of Hospitals Birmingham, and I'm a professor leading a research group in the University of Birmingham, focused on the evaluation regulation and implementation of AI health technologies. Path to getting there was kind of always just interested in science and so on through school, wants to be an engineer, sort of slightly accidentally explored medicine and really enjoyed clinical medicine, but always continuing to sort of explore the research elements and their kind of innovation and I guess always kind of interested in the technology and how we use technology to improve the delivery of healthcare. And then specialised in ophthalmology, did a PhD in ocular immunology, so super lap-based kind of looking at cells and immune staining and so on. But it was really during that that I discovered that actually what I really cared about was not so much that kind of very discovery and lap-based stuff, but it was actually how do we get innovation through to patients, how do we improve what we're doing today so that tomorrow be better. So as a clinician thinking okay, these diagnostic tests we're using are better than we had ten years ago, but couldn't we do so much better than this and similarly with treatments, they're kind of okay but they still have lots of side effects and they don't work for everyone and I'm sitting here in clinic and maybe I can rapidly diagnose 80% of the people coming to us and maybe we've got good treatments for 70%, but what about the other 20% and what about the other 30%. So there was just a sense of okay what we're doing is good but surely we can do better than this. So I guess that's fed my desire to always push us to do better through research and innovation. I think if you look at a theme through my research both when I was working in the sort of vertical depth of ophthalmology and now as it worked more in the horizontal of AI, I think a common theme has been around just how can we do this better? So it's almost like you know at the meta level, how can we do just this whole thing better that kind of moving from discovery of in this case a new technology and all the amazing things that are happening in the kind of discovery of AI and the opportunity there but how can we flow that through to actually being able to be have something that's useful and safe, inclusive, equitable in our day-to-day clinical practice and that's that's not just a technology question that's not just a research question you know that's a kind of people question as well isn't it so no it's a policy question and a practice question. So now as Chair of the AI Commission how did that come about? How did you get into that role? I guess what most people won't know is that's actually the second report I've done to government on how to regulate AI. I sit on something called the Regulatory Horizons Council which is not specifically a medical thing it's set up by the UK Government to advise them on where there is an opportunity for regulatory reform to better support innovation and it's really interesting role because partly because it's not purely medical so I sit with other experts in this space and we look at things like nuclear fusion or regulation of space or looking at hydrogen power and so on but we've also within that looked at regulation of medical devices and specifically AI health technologies and one of the values of it is that you can learn so much from from other sectors and actually many of the principles are very similar and of course something like AI cuts across multiple sectors so as part of the Regulatory Horizons Council I led a report to governments published in November 22 and actually if you look at that report now you still see that many of the recommendations are highly relevant I was you know a very pleased I thought we had met the challenge very well but something else happened within a week or two of us publishing which was Chatchy PT went public so that was that for me just highlighted one of the challenges here is that you can you can create a framework that is good for now but it's almost immediately good for yesterday and this space is moving so fast and so I think if you did read it now you would go well yes this feels like it belongs from a previous era because it really hardly touches on the challenges of generative AI at all so I have been working in this space for a while the team that I have the privilege to lead based out of Birmingham have been doing some really good work and working with regulators to try and help both evaluate the existing regulatory system but also think about how we can optimize the regulatory system so very much sitting in an academic space rather than in a policy space but I guess those were the reasons probably that the MHRA came to me and said we would we are looking on behalf of the Department of Health and Social Care to establish this National Commission would you be prepared to lead it and about you know me well enough to know there was definitely a moment of going oh my goodness am I the right person for this and really took time to reflect whether actually leading something that was so important you know you want that the right person to be there you don't want to just jump at it because it sounds interesting but I did say yes and I think the value of having an academic potentially leading this is that you're bringing a particular set of skills of impartiality kind of independent scrutiny of the opportunity here the risks here the evidence the ability to convene people together recognition that no one person or no one sector has all the wisdom we need we're all like jigsaw pieces coming together and I guess that's what I saw my role within the commission is to just ring together this wealth of expertise and knowledge the diverse viewpoints around how this should be done both in expert groups in the wider public and practitioners on the front line and health leaders in kind of the regulatory space across government international UK-based foreign edge based just bring that all together and work systematically through to say okay where are we now where do we need to get to and how do we get that and I think from your answer it's It's very clear that you're happy to find out. And certainly from my experience, I think everyone would agree that you have done a brilliant job in leaving the commission. Why now? Why was the use of mission convened at this particular moment? Because it is quite an unprecedented commission. So I would really like you to sort of give us that bigger picture view on what is it about this moment, technologically, politically, strategically that has made this feel like it was needed right now in the UK. Yeah, so I think it's really interesting. And to highlight the uniqueness of this is that we have had a lot of interest from around the world, saying this is really interesting what you're doing. Can you tell us more about it? How have you set up this commission? What's the scope that also tell us about how you're bringing different voices in? And what they said was look, every country and every kind of organisation within the country is bringing out AI policies and bringing out really thoughtful papers, et cetera, about this. But they're all based just on expert groups and they're all kind of a single part of the system. What we think is really unusual about what you're doing is that you, the UK, are actually taking a moment to reflect on what you want your regulation to look like in order to build and shape that future healthcare system that we want to live in. That is understands that it's going to be AI enabled. And I think what this reflects is a real clarity that we had right from the beginning that before we got into the weeds of the regulations and, you know, what turned out to be 44 recommendations in a report. There was a need to set direction and affect agree our compass points. And I think that's what spoke so powerfully both within the UK and across the world. So those compass points were effectively we wanted to ensure that any new regulatory framework was achieved was safe. It was fast recognising it needed to be agile for a rapidly moving technology space, but it also needs to be fast in the sense of being able to get technologies through and either weed them out or through to benefit patients as, you know, faster than we do now. It needed to be trusted. It needed to both earn and maintain trust when that's always true of regulation, but even more in this space. And fourthly, and in fact, most importantly, we need to be person centred. So what we're looking at is a future healthcare system that will be increasingly technology enabled. But part of our role is to make sure it was always person centred. And that of course is about being patient centred first, but it is actually also around enabling professionals. I think we took a view though, to be honest, if the sort of introduction of these technologies didn't improve the working lives of health professionals, then it wasn't really very good. You know, surely with all the technology capabilities, we can achieve both. We can really improve patient experience, improve patient outcomes whilst also empowering health professionals and enabling them to actually have better working lives, tackling burnout at the same time as we're introducing technology that improves the lives of patients. So I think that those those compass points of safe, fast, trusted, and person centred have defined how we then set up the commission to achieve those aims, but that was where we wanted to get to. That's a great vision of what we want regulation to be going forward. I would like to dig into some specifics about why that vision was needed and why it was felt that there's something needed to change. And I'm specifically thinking about what the advances in A.I. that are pushing the boundaries of current regulation and where current regulation is struggling and where that change was felt it was needed. So that from a technological perspective, but also from like a geopolitical perspective about the UK's position compared to other regulators. Thanks, Annabelle. And yes, you're quite right. I didn't answer your previous question. I think there were a few things coming together at this time. And in a sense, I don't think these discussions about how we regulate A.I. going to go away because, you know, I think you and I could be sitting here in 20th time and the exact detail may have changed, but this is an ongoing, both opportunity and challenge as these technologies continue to evolve. But I think the reason it happened, particularly now, was as mentioned earlier, a need to respond to the opportunities and challenges of generative A.I. I think it's also that we have moved from, say, four years ago, where the introduction of A.I. was much talked about, but actually there were probably far fewer examples. It was in a sense much easier to contain and constrain to actually being very widespread, including through the use of A.V.T.s and so on. So it's just like the challenges of A.I. and ensuring safety is now widespread across the whole health system, rather than just being in a few higher resource settings. And we just dig into that one thing, because we mentioned that generative A.I. is the problem, but can we just go into why generative A.I. is a problem with current regulation? Because to my mind, it's things like the fact that it's non-deterministic, that it might be more rapidly updated. It might be more generalist, rather than having, for example, a static deterministic model that can be evaluated pre-market. Like, what is it about the generative A.I. that particularly makes that a challenge? That it's broad purpose and the kind of fact that you can potentially have infinite inputs going in and potentially infinite outputs in non-constrained generative A.I.P.T.s model, the fact that at least kind of the base models have often not been designed for use and health care. There's a whole host of problems, but I guess what you're trying not to do is you're not trying to say, "Oh, well, I'm going to put this in the two difficult box, and therefore we're going to just ban it outright." It's clear that there is an opportunity here, so we need to move regulation to the opportunity of the innovation, not hold back innovation to where regulation happens to be from 20 years ago. I think one of the things I haven't said yet was that the commission was actually partly set up in response to the NHS 10-year plan. It's a describing analog to digital shift, and they talk about A.I. as one of their big bets, and they mention A.I. more than a hundred times through that report. But it's just a recognition that if we are going to use A.I. as an enabler for what we want our future health care system to be like, then we need to have a regulatory system that keeps the safe, is trusted, but also is efficient in the ability to get new technologies that could be genuinely helpful out the front line through into routine use. And as I say, getting beyond the, oh, it's another pilot, and it's another proof of concept, and it's generated this academic paper, but to a kind of, you know, business as usual, where actually don't really talk about the fact that it's A.I. not. It's just part of, it's just another tool that we use, and that's a bitch, you know, I'd love to get to the point where we're just, we're just talking about health care, and it happens to be A.I. enabled just as it happens to use a bunch of other tools, do you know what I mean? The focus is on the quality of health care, not on the tool we use. Let's get into the kind of strategic political reasons as well, positioning the UK, because obviously we've got, you know, big global regulatory jurisdictions in the US and Europe. What is it about this moment that made us think what do we need to do as a country? Well, I think there is an opportunity here. One of the things is that I, as Chair of the Commission, I, it's been a privilege to take this big overview picture, including all the political aspects and the kind of opportunities. But I'm, I always come back to the kind of, what does it mean for patients, what does it mean for health practitioners in the frontline, which probably reflects, you know, where I sit in the system. I'm unapologetically pro innovation, and I'm unapologetically pro patient safety. So, and I don't see these as operating in opposition. I see these as part of just getting the system right. I see this as making regulations smarter so that we get better innovation through, through faster, whilst still weeding out stuff that it just isn't ready or simply isn't useful or safe at all. And the reason I come to this first is that I think actually every health system in the world, every government in the world is struggling with that balance. And I think the UK is really well set up to get that right or achieve that in a way that most countries will really struggle. And partly that is that we do have a national health system. We know that it operates and quite fragmented, disconnected way at times, but still overall there is an ability. And it's exemplified actually by the work of the commission to bring you know which bits of the system do what and therefore who you need round the table to try and work this out in a connected way. I think that there is an opportunity for the UK to work out how it positions itself in terms of enabling good technologies that are developed elsewhere to come through faster. So in regulatory terms we talk about recognition and reliance routes. So to what extent can we recognise a regulatory approval from outside the UK? And so yep this is this meets the standards we we require and yes we can safely bring that in to be used in the UK. There is also we have as a country we actually have a really strong medical device sector who've also been engaged through the commission it's been really great to have have their input but we can build an environment where they can flourish and that's good for them as companies it's good for the economy it's good for our employment but it's also good for patients and health practitioners because again that's a kind of engine room of great technologies coming through faster. I guess when I look at regulation I'm looking at two things how do we make it easier for people to do the right thing and how do we when you do have people deliberately or accidentally do the wrong thing how do we detect that early and actually bring down corrective action and stop happening again for ideally preventing it before before it happens and in terms of being able to do the right thing again I think countries around the world struggle to provide the clarity that developers need but one of the things you'll see throughout the report is a real emphasis on both improving predictability for innovators so that they know what the pathway looks like they know what they need to do when and clarity so that both innovators themselves but also all of us are much clearer about for example when is a medical device a medical device as opposed to a non medical device health technology if it is medical device what is the risk class it sits at what are the exact requirements in terms of the kind of evidence that needs to be provided in order to show that it is it is safe and and who needs to do what once it's out in the system who has you know what related to the roles and responsibilities to control the risks and so on. Yeah and it was great to see that that recommendation really running center there early on about clarity in terms of risk clarification because that's you know something that obviously gets talked about a lot in the industry and let's go go into what exactly the commission was what did it comprise and what was the process. So yeah it was a big initiative formally announced in September 2025 and ran to earlier this month so September 2026 and the final report landed just a few days ago the commission had a core group of commissioners chaired by myself and Henrietta Hughes 14 people representing kind of senior leadership and senior voices from across the wider sort of health and the government ecosystem and then critically sort of feeding into that or commission group were four working groups of more than 100 experts and they represented there was a technology working group which represented for example small companies large tech also clinical entrepreneurs in the NHS there was a health systems working group which was chaired by Henrietta Hughes which was frontline staff senior health leaders people working across the health system then there was a cross-white hold across government group which included things like professional regulators different government departments because again recognising if you want to make changes the regulation of AI and healthcare you may need to get sign off from other parts of the system and then really importantly we had four nations working group where you had senior digital leaders from across Scotland Wales Northern Ireland and England meeting together and essentially and you know because you were part of one of these working groups what we did was we brought really quite detailed papers to look at hot topics there might be around post-market surveillance it might be around professional responsibility and agentic systems and all sorts of interesting challenging areas which the working groups would work through they could feedback on and this would then continue to be worked on and iterated throughout the course of the year so that was if you like the kind of main route for expert inputs in a kind of formal really deep dive way then there was a public call for evidence which ran December to January which we had 761 detailed responses big chunk of those were from members of interest in members of the public but also some sort of formal responses from many of the sort of professional bodies and other institutions etc fantastic material came through that great description of the problems that we are in the challenge we see but also very thoughtful suggestions of what needs to change and we then had big surveys so we surveyed more than 10,000 people all together 8,000 members of the public more than 2000 health professionals that was run by health foundation and has been published and looking at people's views around AI etc and then we had like international contributions and so on but perhaps the thing I'd want to most highlight because maybe people go yeah yeah we'd expect people to you know the commission to do things like that I think one of the things that I was most struck by was their sort of public detailed public dialogue we did so this was a bit like citizens juries which people would be very familiar with the concept and so we had one set of these public dialogues led by health foundation and one by national voices and we took members of the public away and spent time over several days just looking in detail about the opportunity of AI the challenge and we took three case examples of specific technologies we took an ABT we took a sort of digital mental health chat box we took a skin cancer triage-diagnostic tool and we explored with each of those those example technologies the potential how it could be used taking really drawing from our experience of real life but also some of the challenges the decision points for example what happens with that skin diagnostic where the evidence is strong for using pale skin but is much less we don't have such good evidence in darker skin etc exploring that's meant for public we then also talked through different ways of regulating how the regulatory system works now and different suggestions in terms of how it could be regulating the future including self-regulation which is obviously quite in the news at the moment in terms of wider kind of AI frontier models we looked at things like trusted developer status where effectively people could earn the right to self-regulate we looked at the current system where it's very heavily pre-market it's all on a pre-market stage of manufacturer to MHRA or to an approved body conversation versus something that is more ongoing assurance across the life cycle and so on and and we got their feedback and what was just really interesting for me is that it was possible to have this very nuanced conversations and detailed conversation about the regulatory system just removing some of the technical more technical jargon but still absolutely not dumbing it down in terms of what was actually happening and it's very similar to the kind of conversations that you and about we've been part of in the working groups the kind of themes and the recommendations came through were very similar so for example in terms of what the members of the public in those dialogues wanted from regulatory system they wanted they were really clear they wanted a risk proportion of that system they were happy for very light touch for vulnerable risk products but they definitely wanted much stronger controls on their kind of high risk products and they definitely wanted continuous monitoring they were very uncomfortable with an approach that put all the way right up front but they were also comfortable then with the fact that we might have less evidence at the point of it coming to market and it would be okay to come on to into healthcare what people commentated called the learner plates on so that there was an appropriate risk controls in that initial phase with a view to it then getting it full market assurance, market approval slightly later on so it was a big process it ran for a full year involved more than 12,000 people gave me a lot sleepness nights and but I'm really really grateful to everybody who committed their time through that process the level of engagement these are all very busy people I think we've landed somewhere very significant where it is a sort of foundation that we can now build from it sets direction it is as you know it these are recommendations independent recommendations to government this is not government policy we have to wait now hopefully it's in the order of a few weeks in terms of government response but that's outside of my control and we will see what changes are actually going to happen in response. So as you can hear a huge piece of work with very widespread involvement and what this has resulted in ultimately is the publication that's gone out last week which contains 44 recommendations and you know each of those is there for a reason but I want to hear from you about which of those recommendations do you feel will really make people sit up and take notice, which are the recommendations that are most likely to make headlines or raise eyebrows? There's a few different questions in that. Well, I've been asked before as, you know, what's your favourite recommendation? So well, I don't have any favourite children. They're all important. But I think you're right. There are a few themes, I think, I would first want to call that. So if you look, they're ordered, the fruitful recommendations have a coherence to them because they fall under three key themes. And those themes are a shift to more proportionate life cycle regulations. So that real recognition that it may be, you know, that we can be light to touch in some areas, essentially, that the regulatory system we've inherited is not optimized for these technologies. So it may be over heavy in some areas and overlight in others. So something around risk proportionality, but also this life cycle approach. Now, the life cycle bit is the bit that I think is going to, it's going to be most challenging. People, you know, those universal acceptance, I would say, that that's where we need to go. But there is also very legitimate, yeah, we need to, we know, we know we need to go in this direction about how. And I guess one of the things that I would caution against, it's kind of magical thinking in the space. So you could be in a position where they're, ah, we're going to manage everything through post market surveillance. But if you don't put the infrastructure in place to, to ensure we have robust post market surveillance systems, post market monitoring systems, then you can say we're doing something, but actually, you haven't built the, the safe system that enables that to happen. I think the second major theme is around system wide responsibility and safe management. And I think, I think that might also challenge some people in the sense that what it does is it says, look, in order to ensure these technologies are used safely, we all have a role in responsibility. And that includes very much, including to manufacturer. So the manufacturer has responsibilities in terms of the design of the product and the kind of development and ensuring that safety is sort of baked in the whole way through and identifying what are the risk controls that need to be put in place at the point that is kind of leaves the factory in quotes. And so the responsibility doesn't stop at the end when it's packaged out. But the health provider or a hospital or a GP practice or wherever also has responsibilities to ensure that their system in which their technology is being deployed enables that to be used safely. And then you or I and about as doctors and wider health professional community also have responsibilities in terms of how we use this tool in the same way that we do for using any other tool that is part of our practice. I think most people again have really received this and said, yes, we recognize that we all have responsibilities. We just are asking for greater clarity in what those responsibilities are. And that touches on the liability conversation. And you had a brilliant discussion on your last podcast with Sarah from NPS on the exactly subject. And they asked for clarity. But I think most people accept that we do have responsibility. But I think some people will go, oh, no, the manufacturer should do everything. This should be perfectly safe out of the box. But for a complex tool, I don't think that is reasonable. I think there is conditions of use effectively. And then the third principle which cuts right through this is around trust transparency and productivity. And in fact, when we surveyed, it's part of the call for evidence, you know, 83% on average across when you put everything together, ask for more transparency around these tools, kind of when they're being used, what the safety profiles are. So there are again, there's some quite challenging recommendations into the MHRA and the wider system about how we can improve the transparency of these tools. There is a kind of right to know for patients that they should know absolutely when technology is being used, where possible they should have the opportunity to opt out. I personally don't think it is always possible, you know, if we take the analogy of electronic health systems, we can't opt out and say, well, actually I'd like a paper system, please. And so once there are many examples where you probably can, it may be very reasonable to be able to explore whether you can have an opt out option. I think that won't always be possible. But I can see of no circumstances where it would not be appropriate to be informed. I think that's something we would all expect. So yeah, there's a few in there. I've not picked out one for you, but I've tried to hide it. Some of the things that are important, but may challenge the state's group. Thank you. And I think I would agree with you. I think the ones that stand out for me are certainly this question or this concept of the life cycle and the so-called learner plates or L plates for technology. And I think that feels like, you know, a very sensible step, but also one that raises quite a lot of questions as well about, you know, how we actually make that happen. I think some other ones that stand out for me as well is the concept that the powers of the MHRA might increase in terms of fines and things as a consequence for organisations who aren't compliant. And then yeah, this question about accountability, system-wide and what that means for organisations, what that means for clinicians, and also the point about redress and kind of how, whether the manufacturers of organisations might be held to account. I want to just get on to this question about the ongoing monitoring, because I think whilst most people would agree that we definitely need ongoing monitoring, I don't think there's agreement on is how we really shouldn't make that happen. You know, you've mentioned infrastructure, and there's a big question there about who owns that infrastructure, who's responsible for it, who pays for it ultimately. And as part of that discussion, one thing that you haven't spoken about yet, but I think is an important thing to inject here is quite a lot of medical technology, medical devices are actually D to C, so direct to consumer and aren't implemented within a healthcare system. So whilst it may be responsible for organisations and clinicians, etc, to take responsibility for ongoing monitoring when it is deployed in that kind of setting, a lot of these are increasingly so more and more commercial products are sold D to C. So who funds this? How do we make it happen? What are your thoughts on how we make that ongoing monitoring a feasible reality? So I think one of the things is, some of this goes beyond the scope of the commission, so I'm just trying to be a bit clear that much of this is me taking my commission chair hat and saying where, you know, just as a commentator like you in terms of somebody who has the privilege of seeing, you know, into different parts of the system. From a commission point of view, there is clarity about where we need to get to, sometimes there are multiple ways for getting the same route and the, you know, mapping that route and what I think we should look for the government response to see some of the details to kind of how some that infrastructure is going to be put in place to respond to that challenge from the commission. I think there's a lot of good work already happening. If you lurk across groups across the UK, organizations always kind of bottom up. There's been a lot of innovation in terms of, I don't know, hospitals or GP working out how to kind of put the monitoring in place. There are tools being developed to help institutions set up systems, so a concept that we've talked about a lot actually, this is me wearing kind of my researcher hat on is AI redness. There's a team that have been working on AI redness framework and a checklist that goes with that as part of the centre of excellence that I lead, so I'm called Cersei AI. So if you look up there and look up AI redness, you can see some tools that we've developed that kind of help organizations that want to deploy AI technologies, essentially look at how they ensure that they've got safe systems in place, including thinking about monitoring. And that's just to say they're kind of, it's worth looking broader. Those are just the ones I've been involved with. There's increasing interest in preparing, deploying organizations in terms of, in terms of these technologies. I think I'm particularly interested in where we can reduce the burden on humans for doing the monitoring and they can use technological solutions. And I think if I was an innovator and manufacturer, I would be very much looking at what can I embed within my design and my product to actually make it easier to collect both for us as manufacturers but also for the system to collect that ongoing performance and safety data. I think there is, there is an opportunity, so it's a for the NHS to bring its digital infrastructure up a level. The Health Foundation has done a report estimating the cost of that, which is significant, but they also highlighted that without doing that kind of digital renovation, if you like, we simply won't be able to unlock all the opportunities of things like AI. I think that the point around direct to consumer, it's interesting. I think you can look at that two way. Well, first of all, from a commission point of view, everything we said about medical devices applies, whether it's going to be deployed in a hospital or a GP practice or it's direct to consumer. If it qualifies to medical device, it's a medical device, whether it's direct to consumer or not. So all those things still apply But again, from a scientific point of view, I think when the interesting things about Direct to Consumer is you can look at that and say, "Oh, well, it's introducing new risks because there's not a health professional in the mix." But you could also make the argument, it has less variables. If you're going Direct to Consumer, you actually have fewer variables to think about. You as a manufacturer potentially have a direct line of sight into the device in which the user is using. You've got much more opportunity for direct feedback in terms of safety monitoring, improvement, etc. So I think the Direct to Consumer space is really interesting. I think there's an opportunity to improve our monitoring systems and a real-time feedback in a way that actually, it may be that the innovation that happens in Direct to Consumer space is something that we then learn from back into the more complex healthcare system. So I would actually see that as a really important area, but that actually is an opportunity to learn how to do this one. I was to this so many more questions I want to ask you at a really conscious time, and I know that we're having to the end of our time together. So I want to close by asking you a final question, which again involves stepping back and looking at the bigger picture here, and saying out with the commission if we're thinking about regulation and success of AI and healthcare in general. What are the things that you feel like we need to change most to make that a reality? Is that practical? Is that infrastructure? Is that policy? Is that how we think about AI? Yeah, so that's probably another whole podcast, but maybe just to sort of so a few sort of concluding thoughts around that. I have the privilege of being asked quite regularly, you know, where do we, from an NHS perspective, where do we find the value of AI, where do we unlock the value of AI in healthcare? And I think there are lots of complicated questions that we've been talking through just now, but I think in my head this one's quite simple, and that is that the opportunity or the value of AI in healthcare lies in intersect, and that intersect is defined by effectively three circles. It's a defined priority need met by a technological capability in the context of a system that is ready to use that technology or system readiness. So those three circles have an intersect of need, technology capability and system readiness. And I would argue that that intersect is relatively modest now. It's probably a lot smaller than many people outside your listenership think because I think if you listen just to the headlines, etc., you imagine that's like a massive intersect that kind of like the AI could be used absolutely everywhere in healthcare, which I really don't think is true. But it does have a value, and that value will grow over the time, and it will grow for two reasons. The needs will stay the same, but the two things that are going to change are the technology capability is going to increase. Now, but step change from more narrow AI to generative AI and general purpose AI is an example where that technology capability has suddenly expanded to cover more of the things that we would find useful where it really defines needs in the health system. But the other thing where we can really make a difference, and I think many of the people who listen to your broadcast sit in this space and could really make an impact is that system readiness. And what I mean by that is if you imagine building a bridge, we've been so focused on building the one side of the bridge, which is like that technology capability, you know, that half the bridge is looking really shiny, it's so much investment's gone into that space. But the other side of the bridge is potentially looking a bit rupee. It's that ability of the system to use that technology. And to get that side of the bridge in a state of readiness to be able to meet in the middle and start actually creating something that's useful and that addresses our needs. We need to invest and that's kind of financially invested. It's also thoughtfully to share that it's about people, skilling up people, it's about building digital infrastructure, it's about getting the systems right, which includes a regulatory system. And I would see the commission as being one of those contributions to that level of system readiness to meet in the middle. So I think what I would hope to see over this next period is that effectively those of us are working the health system and who are interested in this space, working coherently and together to just to ensure that we are making sure that we are equipped to, you know, we have systems in place, we have the people that are both competent, but also confident to use these technologies. And we can actually start to use those safely and moving from it being kind of pilots and research studies to something where it's just business as usual. And we're, we're starting to see a very, a step change in the kind of quality of care we can deliver because it benefits people. You know, there's a really interesting line in the tenure plan that talks about making the NHS the most AI in the healthcare system in the world. I have a different vision, which is about making the NHS the best AI in a healthcare system in the world. What a great note to end on, Alistair. Thank you so much. It's been such a pleasure having you with me on the podcast. And look, we will share the recommendations in the show notes for everyone to have a look at. And I do encourage listeners to do that. And it will certainly be keeping an eye out for all of the ramifications of these recommendations as they're put into action over the next months and years. So thank you, Alistair, for all you've done. Thanks very much, Annabelle. So what should we take away from all this? The recommendations highlight how the UK is answering a key question in AI healthcare regulation. How do we construct a regulatory framework that isn't just fit for now, but will be adaptable and robust enough to regulate technology that's accelerating capability at an unprecedented rate. These recommendations are an interesting signal into what might be ahead, but they really are only the beginning. Whilst they answer some important questions about approach and strategy, they raise many others that will require a coordinated system level perspective. Questions like where responsibility sits, what meaningful human oversight actually looks like, who owns and funds the infrastructure and workforce capabilities needed to make this a reality, and how different organisations will be held to account if regulation is not followed or harm occurs. Answering these questions responsibly and aligning levers and incentives strategically will be crucial to determining the impact of these recommendations. As with technology itself, the most challenging part comes when we turn theory into practice, when recommendations meet the reality of clinical deployment, navigating human factors, commercial incentives, and increasingly capable systems.

Podcast Summary

Key Points:

  1. The UK launched a national AI commission to develop a future-ready, adaptive regulatory framework for AI in healthcare, responding to rapid technological advances and geopolitical positioning.
  2. The commission emphasized four core principles
  3. Key recommendations include shifting to risk-proportionate, lifecycle-based regulation with "learner plates" for early deployment and ongoing monitoring.
  4. System-wide responsibility is assigned to manufacturers, health providers, and clinicians, with clear expectations for design, deployment, and safety oversight.
  5. The framework promotes transparency, including patient rights to know when AI is used and to opt out where feasible, while recognizing limitations in direct opt-out scenarios.
  6. Direct-to-consumer medical devices are treated as medical devices under regulation, offering new opportunities for real-time safety feedback and innovation.
  7. Ongoing post-market surveillance requires infrastructure investment, shared responsibility, and embedding monitoring into AI product design.
  8. True value in AI healthcare lies at the intersection of unmet clinical needs, available technology, and system readiness—highlighting that investment in systems and skills is as critical as technology development.

Summary:

The UK has established a comprehensive national AI commission to address the urgent need for a modern, adaptive regulatory framework in healthcare. As generative AI evolves rapidly and outpaces existing regulations, the commission’s 44 recommendations aim to strike a balance between innovation and safety. It proposes a shift toward proportionate, lifecycle-based regulation—allowing lighter-touch oversight for low-risk tools and stronger, continuous monitoring for high-risk applications.

Central to the framework is a shared responsibility model across manufacturers, clinicians, and health systems, ensuring that safety is embedded from design to deployment. Patient transparency and choice are prioritized, with clear rights to know when AI is used and to opt out where possible. The commission also highlights the importance of system readiness—where clinical needs, technological capability, and systemic preparedness intersect—as the true source of value in AI healthcare.

While direct-to-consumer devices face unique challenges, they are regulated as medical devices, offering potential for real-time feedback. Critical to implementation are investments in digital infrastructure, workforce skills, and post-market monitoring systems. These recommendations signal a strategic pivot from rigid, pre-market rules to a dynamic, responsive model.

However, turning theory into practice will require coordinated efforts to resolve ownership, funding, accountability, and human oversight challenges—ensuring that AI integration in healthcare remains safe, equitable, and truly patient-focused.

FAQs

The National AI Commission for Healthcare was established to develop a future-proof regulatory framework for AI in healthcare. It was created in response to rapid advances in generative AI, the need for a person-centered approach, and the UK’s unique position between EU and US regulatory systems.

The core principles are safety, speed (agility in regulation), trust, and being person-centered—ensuring patient and health professional needs are at the heart of AI deployment and regulation.

The commission advocates for a life-cycle regulatory approach that is risk-proportionate, moving away from strict pre-market evaluations. It recognizes that generative AI is non-deterministic, fast-evolving, and broad-purpose, requiring ongoing monitoring rather than one-time approval.

Health professionals, patients, and the public were actively involved through expert working groups, public dialogues (including citizens’ juries), and surveys involving over 12,000 participants to ensure diverse perspectives shaped the recommendations.

L-plates refer to a phased regulatory approach where high-risk AI tools are introduced with lighter, more flexible oversight initially, allowing for learning and adaptation before full market approval and stronger controls are applied.

The commission emphasizes system-wide responsibility and ongoing monitoring throughout a product’s lifecycle, requiring manufacturers, health providers, and professionals to maintain safety and performance, with infrastructure and funding being key challenges.

Chat with AI

Loading...

Pro features

Go deeper with this episode

Unlock creator-grade tools that turn any transcript into show notes and subtitle files.