Placebo-Proof Digital Health: Why FDA-Grade Evidence (Still) Matters with Dr Acacia Parks

Today on Proof It, we're joined by Dr. Akasha Pax, one of the few people in digital health who has lived the full stack, academia, Medtech, regulatory, clinical operations, and the occasional controversy in Firestorm. We talk about what's going to count as "enough" evidence, why most teams miss the mark, and how bold this claim imprete. Welcome to Proof It, the show where digital health leaders stop making grandiose claims, start building good evidence and tell us what they learned along the way. I'm Paul Wix, and today I'm joined by Dr. Akasha Pax, a psychologist-turned-queen-of-digital health, formerly the chief science officer at Happify, after that an independent consultant, now Vice President of Regulatory Compliance and Engineering at Evania. She's published, she's regulated, she survived being the grown-up in the room, which is why we're very fortunate to have her on the show. Akasha, welcome. Thank you for having me. It's always exciting to talk to you, Paul. Great, well let's start at the beginning then, if we're going to talk about evidence. What was your first paper that you published? Oh gosh, so when I was in graduate school, I worked with a guy named Marty Seligman, who is a big proponent of the idea of positive psychology, and so my background is in clinical psychology and my interest was in depression, but I ended up working with him because he had this really interesting idea about being able to essentially treat depression without even talking about depression. You could essentially help people find positive things in their lives that are meaningful, and activating, and distracting. Ultimately, you might be able to paradoxically help people become less depressed without ever having to have the tough conversation about the things that are depressing them. My first paper ever was actually an opportunity to publish data testing that hypothesis, and so we had taken undergraduate students and we had given them a curriculum, weeding me, a curriculum of various kinds of gratitude and savoring, and all kinds of activities that they could try, and these were all students who I had identified as either being at risk of becoming depressed or who are already depressed. We managed to reduce their depression to clinically significant levels, essentially taking them from this sort of mild to moderate depressed state to not depressed at all in this eight-week program where we never talked about depression once. I published a paper about that with Marty, and I'm saying, "Hey, look, you can do this. This is kind of wild, but you can do it." Yeah, oh, that's fascinating. I love that. Given how depressed I feel every time I open up my news app, maybe we could talk about some tips for all of this, for staying positive. Too soon. Too soon. So this process then, the way you've described it was you had a hypothesis, you gathered your data, you falsified it, and you got your statistical analysis, and now it came. Was it really as simple as that, or was it a bit more chaotic in practice? You know, where it actually got chaotic was the next study. So we did this in college students in person, and that was basically stuff that had been done before, and we were just kind of using a new type of therapy to do a type of study that had already been done. Where things really got hairy was when we started studying internet interventions, because at this time nobody knew how to deal with that. And so I remember when I did my first study, so we took these, the same activities that we just talked to students, and we put them on a webpage, and it was pretty clunky. It was not a great intervention, but we were essentially just trying to think about, there's all this research that if you give somebody a good self-help book, they can teach themselves cognitive therapy. I'm like, "All right, I'm going to give them a good website," and they'll teach themselves. And I did that. And when it came time to analyze the data, I had 80% missing data, which, you know, if you come from a clinical trial background, you know, is kind of disastrous. If you do a clinical trial with 80% missing data, the conclusion is, your study failed, go back to the drawing board. Statistically, there's nothing you can do with that. And that's what people told me. I went around asking various experts, "What should I do here?" And they said, "Throw your data set away and give up." And of course, in today's digital health world, that's a wild thing, because all of these digital health companies are doing studies with 80% dropout rates, because in the world, you download an app, and then you never even open it, right? Like 20% of the people never even used your app, and other 20% used it once. All right, dropout is part of the research methodology of today, but at that time, nobody even knew how to deal with it. And we had to figure it out from the ground up. And that was challenging. Publishing those data was actually really, really hard. I got some very harsh rejections saying, like, "This can't be done. Give up." Yeah. And I take it, you took that advice from the peer viewers and you immediately gave up, right? That seems like your decision. That's it. I just rolled over and died. Yeah, I started a whole new career. No, no, we just did it, right? And at some point, we pushed forward, and other people like David Moore and Goother Isaac Bach and other leaders in the field were, Helen Christensen, were publishing data like this, and starting new journals, because they understood that if you didn't get that this was the reality and figure out how to work around it, the missing data was going to be part of it. So the J-Mier journals were the first place that if you were publishing in digital health, you could actually publish data where you had missingness. So yeah, I found support as those things like new journals came out, but I wasn't going to find a home in the existing journals. They were telling me to get lost. Yeah. No, I had a similar experience. I have a collection of my rejection peer reviews and what of them said, oh, I don't believe in this internet stuff. I think it's a flash in the pan. It's going to blow over. And we had got data from one percent of, yeah, I mean, I don't know, early days, right? But we had got data from one percent of everyone who'd had an organ transplant in the United States using online surveys. And they were like, yeah, but it's only one percent. It's probably really biased. It's only one percent. Oh, gosh. Yeah, exactly. No, so there's something interesting. What you're saying there that research isn't just academic scientific like the teaching would have you. It's political, right? And what you're doing is if you're disrupting, disrupting something, which in startup world that we both live in is good. If we're in academia world, that's pretty upsetting, right? If you're someone who only has run clinical trials at a psychiatric hospital and you are the king of this disease or the queen of that disease, probably feels quite threatening that someone could come off the internet. Well, you can't do what I do with a webpage, right? So was that like an early taste that you know, this wasn't going to be a cold, vulcan science. It might be hot, Kirk science. So in my first year when I was starting to do this research, we had a like a master's thesis that you had to present at the end of the year and it's this sort of sink or swim, like prove that you can do research and sort of walk the walk. And so I came in with my study and I had I compared my positive intervention to a wait list control, which is what you do at first. You're like, is this even worth studying? Does it work better than nothing? And Diane Chamblis, who if you have operated in the anxiety world of clinical psychology is like a goddess, was sitting in the room. But how do you know that it's not just the placebo effect? You should really be comparing this to a sham. And I was like, man, that's wild. How do you even do that? But like, guess what? 10 years later, the FDA was going to be like, hey, if you want to get your digital product FDA cleared, you have to show it's better than a sham. So that whole process got me started thinking about this problem deeply. How you even create a sham for a digital health intervention? 10 years before anybody cared about it. But I was in a position of having to have this meeting with FDA and say, hey, I've already done that. So I'm going to check it out. So it is really interesting kind of being in a place where nobody knows what the right methodology is. People maybe know the right questions to ask. But then every answer is imperfect. And all we can do is acknowledge, try it different ways and acknowledge this way is imperfect this way, this way is imperfect this way and try and glean on an answer. It's pretty fast and loose. Yeah. Yeah. And it's interesting even hearing, because I think we've been working for a similar amount of time at the beginning, you find that you're grinding away and trying to change minds and things like that. And at the time, it feels very slow. But if you're talking about a 20 year lifespan, you're right. Someone just goes out and does the hard work. Someone just went out and made a whole new journal on the internet. And for example, I published some stuff in JMR and be involved with them. One of the things I loved about what they did is they raised the bar. So you wouldn't just say, oh, I did an intervention and I showed it to a thousand people. They had checklists that they had adapted to internet interventions that, first of all, acknowledge limitations and strengths, but also help close some of the loopholes that like less group list researchers could use. So you would have to say, oh, how do you know it's not the same person entering it 20 times and how much did you reimburse them and how did you double check that they were fake IP and what have you. So it's interesting that sometimes the innovators have to raise the bar. Everyone was forced to raise the bar because they're being evaluated more harshly, perhaps, than the status quo. Well, that's right. You want to be clear about, well, this is impossibly messy here. So I'm going to be as rigorous as possible everywhere else. And like that's the sham comparator is such a great example of that. You're like, okay, I'm testing for the placebo fact, which is like kind of the highest scientific thing that you can do for a behavioral product. But at the same time, we're like, well, and I've got all this missing data. Right. So you kind of, you have to figure out where you're going to, where you're going to have to acknowledge shortcomings and be as rigorous as humanly possible everywhere else. So let's talk about that, Shane. idea for a little bit. So obviously people will be familiar in a drug trial. The concept that the placebo is an inert sugar pill. What you may not know is that in other procedures like surgery, people will even do sham surgery. And it's really important that the actual scalpel cuts and skin and things are retracted and say line injected instead of the chemical or whatever. And even with implantable devices, you may have a situation where the device is implanted but not switched on on by the database or what have you. So in a digital mental health app and take for example, something that's got a little bit of psychoeducation, maybe some games. Now we're increasingly seeing use of light and sound and advanced sensors. Tell me about a sham there because obviously there's no real equivalent of a sugar pill. You can't ask people to play candy crush. That might be too fun, might be too engaging. So tell me a little bit more about how you think about that and you mentioned FDA as well. To what extent is a regulator really establishing that because I think that's being a theme for your career that I want to play as well. Yeah. I think it's really important to highlight that the only reason anyone in digital health mess is with shams is because FDA has asked them to. The first 10 years of my career, I was doing shams and nobody else was and it was totally overkill and nobody cared about it. And I go to an employer and say, hey employer, we beat a sham and they're like, yeah, but they're effect sizes bigger. Look at how much bigger the differences between their groups. And I'm like, right, like it was scientific overkill back in the day. But when it became clear that this is right. And when it became clear that this is how FDA thinks, right? FDA has for medical devices for, you know, from time immemorial has wanted to see a sham comparator to control for the placebo effect to control for expectation and for digital products. It's no different. That's their expectation. But they don't tell you how to do that. They can't tell you how to do that. They don't know your product, right? So it's FDA's expectation that you're going to be the expert on your product and the things that need to be controlled for in your product and the things that need to be simulated and that you're going to come to the table and kind of propose what that looks like. And so essentially you want to show that people spend about the same amount of time on it. You want to show that people believe that it's going to work about the same, right? So that it's believable. And you don't see signs like your control group dropping out twice is often because they figured out they've gotten the control. Like you don't want to see the sort of behavioral signs that you've failed to make people blind. So ultimately it's about blindedness. You want people to not realize that they're receiving a control intervention. And the more sophisticated technology gets the harder that becomes, right? Like back in the half of five days when we were designing a sham, we literally just came up with a psychoeducational app. People Google for psychoeducational information all the time to to help them with a problem, right? So it feels like a plausible treatment. People were pretty convinced by like an app that just has all of this information that you can access about mental health, but it doesn't tell you what to do. So people can spend the same amount of time doing it. They're doing the act of reading. They're interacting with an app. The app is similarly branded. It looks fancy, right? So it controls for a lot of that expectation. Interestingly in the very beginning, I tried to put sham activities. So for example, if you're supposed to write three good things that happens to you, instead you just write like three things that happen to you or three neutral things that happen to you. People don't let you do that. They're like, "Oh, I'm probably just supposed to write positive things." And then they do, right? So like in the beginning, I tried to really come up with sham activities and they always worked. People found a way to change them so that they were beneficial instead of doing the thing that they were asked to do. So in the end, we worked on slightly like having a psychoeducational type thing. And that was okay. But now try and think about how to do that with generative AI. Like, do we give them a version of chat GPT? They can ask health questions, but like it won't give them anything useful. Like, it becomes more complex. Yeah. So do you think there'll ever be a universal sugar pill in digital health like one sham to control them all? No. No, I don't think so at all. If you look at just the differences between the shams for a Killy, the ADHD product versus something like the big health recent product, like daylight. So these are just different. Some are more gamified, for example. And if it's gamified, you want to like reproduce that game and change the underlying dynamics that are therapeutic. Right? So like if you're trying to attain attention in a certain direction, for therapeutic benefit, train attention in a different direction or so chaotic that it doesn't achieve any kind of training. And otherwise, it's the exact same game. Right? And that's going to be specific to a given product. You won't just have a generic one. I could see there being like a generic psychoeducational app that like anybody could use. I could see that working. But just because the nature of psychoeducation changes so frequently, like where people are going to get information and what the information is, like it's so rapidly changing that you would at least need to like see that be updated periodically. Like nobody's going to accept a static app with psychoeducation as a control for a generative AI mental health bar. Yeah. Well, and the pace of technology has moved on, the pace of technology moves on so quickly as well. I mean, so one of the things that I that we've both seen is that by the time something is published, the technology that was used to build that often seems quite archaic. Right? So oftentimes you'll be peer reviewing something in 2026 that uses, oh, we show people a video. Okay. But if it wasn't optimized for the tick tock attention spend, you know, no one's going to watch a seven minute psychoeducation video. If it's going to, you know, three things about your condition. One, two, three, like that's it. That's all the time you have. So it feels like unless the pace is. Yeah. No, that's true. But you know, the other important thing is that, you know, coming back to FDA, not defining like what a sham looks like, what defines whether a sham worked is patient behavior. So a sham that worked three years ago that had a seven minute video might not work in three years when patient attention span has changed. And you'll see those things like drop out rate being higher in the control group when it wasn't. Right? So like it's this say that there's like a working sham without thinking about kind of keeping up to date with technology. It seems unwise. Yeah. I mean, I've done some working medication adherence and, you know, a lot of the programs were like, well, we'll send someone a text message because people pay attention to text messages or that it was we'll send them a notification on the app. And you know, even with most of them you did you get hundreds a day and you just think, well, it's it's all just kind of melting away. So yeah, I think unless you can stay current, like we have this problem with hapify where we would we would try and give people push notifications, but we found out that the majority of our users never even enabled to push notifications from our app. So they're not even seeing them and ignoring them. They're not even there. Yeah. And I mean, it's interesting. So when you describe the sham, I mean, the sham design sometimes has to be as elegant as the intervention that you're building. And yet that's very hard to come across in a manuscript, right? In the manuscript you're normally to be about the intervention. And then you probably just say, as an aside, we used a sham control. Maybe you might say controlling for attention or duration or something like that. And some of the data that you mentioned earlier about old people persisted for as long, well, they spent as many minutes with it. But I guess it must be really hard to convey. Like if I if someone set me a task, April, find me 50 shams and describe them richly as the participant viewed them, you know, going back 10 years, I doubt I could do that. Right? There is no library of this stuff, is that? That's by design. It's richly like competitive information. Right? If you think about how many interactions a company might have had to go back and forth with FDA before they agree with what an acceptable sham is for their product, they don't want to share the fruits of that labor with the rest of the field. There isn't that same sort of scientific collaboration where, you know, I remember when we had our first meeting with FDA and we were like, all right guys, this is, I don't know, 2016, 2015. And we, you know, maybe Pear had gone through their clearance, but otherwise, like we didn't know a lot about what FDA was looking for and we had a meeting with them. And we proposed a study where we were going to basically show non-inferiority to treatment as usual. So, you know, here's our product. And it's about as good as standard care. So like, that's great, right? And FDA was like, no, actually, I don't care about that at all. Show me comparison with the sham, right? That's, that's what I want to know. So we had that meeting and at that time, like, nobody knew that this was what FDA expected. Nobody had told anybody. And I remember looking at the CEO of Happen by Tomer and saying, I think we should probably tell everybody about, here's like, okay, so we told everybody, but like, I think back in those days, there was a bit more of a culture of, like, we're all sort of, there are six of us, you know, scropping through this together and like, let's share what information we could. Now, I think it's a bit more close to the vest, right? Like, companies find out, oh, well, it's important to do this and this. I'm going to guard that secret to my grave. And you can't find it from their FDA filing. You can't find it from their publication, because as you note, there isn't this emphasis to give as much detail about the sham. So you can't even reconstruct what people do for their sham's half the time. And I think that's on purpose. Yeah, well, it's interesting because, you know, part of the way the business of pharmaceuticals work is there are patents, right? Because a chemist with a kind of water, white setup can manufacture aspirin and Tylenol and whatever, but perhaps it's, you know, so the rest of the secret source is shared and transparent. It's probably harder to recapitulate a working app than, you know, a new cancer therapy that's come out 'cause you've published all along the way. This is how the cancer drug worked in cells than mice and pigs and humans and phase one, phase two, right? It's just that you own IP protection that someone else can't come and do it. But then the challenge is so many of the people who would say digital mental health are using talking therapy conventions that have been digitized, right? And it's almost like stuff going out of copyright. It's not quite like that, but, you know, anyone could take a training manual about getting over fogus or ticks or sleep terrors or whatever it is, right? And try and extract that information. But the particular, I guess, formulation of engagement, infrastructure, hooks that work, you know, that secret source seems very proprietary and maybe almost counter to some of the other traditions and science of sharing evidence of being transparent and allowing replicability. I mean, by definition, replicability in digital health is impossible, isn't it? - Uh-huh, yeah. And, you know, it's the same with the sham. So you have your, you know, the magic that you put into your app that brings cognitive therapy to life. Well, it's, you know, it's not engaging inherently engaging material. So like if you have one app that people use a lot and one that people don't, there's some secret sauce there. But then you've also got all of the time and energy that's spent developing whatever the companion kind of sham application is. And all of the work it took to find out what needed to go into that. And yeah, it's all, it's all a lot of, a lot of secrets which, you know, it's interesting 'cause I'm on the reviewer side a lot more than I'm on the set, like as a consultant, I, you know, I'll review things a lot more now than I'm like being a part of the company's submitting something for publication. And you know, you see these papers and you're just like, we did black box. And like on one hand, you understand that that is the way things are. But on the other hand, like how do you even let a paper be published unless there's a certain amount of information people are willing to provide? So it's attention. - Yeah, though in that black box, I think, you know, if you're concerned about safety, that's a real concern, right? So some of the apps we're talking about are in very vulnerable populations. Very, they could have very severe conditions, a worsening of that condition or failure to notice that someone is getting sicker could have fatal or very severe consequences. And yeah, saying, oh, it's secret source and production information feels a little, like I say, counter to how the clinical psychology, psychotherapy world operates, right? Where you have supervision. We have communities of practice, where you have licensure. - And manuals, right? So like if you wanted to replicate a study about this particular type of cognitive therapy, there's a manual and that author would be delighted to share it with you, which you could then use in your study so that you're doing the same kind of cognitive therapy. And we don't have that in digital health, right? If you, and this is the other thing, like as a reviewer, I got a lot, is it's like, well, this is cool, but no one else can do anything with it, right? No one can pick this up and continue to study it. There's not enough information here. I'm sure that you're not gonna share it. So, you know, that ability to kind of pick up and use and like when you read research on cognitive therapy of a particular kind, you know that 10 years worth of studies did the same cognitive therapy pretty much because they used the same manual. And it's very apples and oranges in digital health because everybody's doing their kind of own special sauce version and the sort of ability to learn basic scientific things from their research is less. Now, there are some like, there's some particular things that you can learn, like you can do this without a therapist, like that's true, right? If any app comes through and is able to deliver kind of scientifically valid outcomes that are clinically meaningful, and there's no therapist involved, you can sort of conclude like it's possible to do this without a therapist. So there are some things, but yeah, a lot of things not so much. - Yeah. So maybe going from the secret source then to evidence. So, you know, when people share stuff that conferences, as posters, as presentations, as peer review publications, that in theory work its way into the pyramid of evidence. How much evidence is enough? - You know, I think it's less about enough evidence and more about enough variety of evidence. So if you kind of track the history of digital therapeutics, I think in the beginning, everybody was really focused on how does one get FDA clearance as a medical device? Like that was just a whole new set of information for digital health. Like nobody in digital mental health had ever thought about this before. FDA had never thought about it before. It was all being sort of created from scratch. And, you know, if you asked me not to describe the kind of study you need to do to get a depression product cleared, I can tell you, pretty exactly. It took 10 years to get there, but I know what those things are. Once you get your clearance though, and this is like kind of what the last five years of digital therapeutics has looked like, then you actually have to get people to use your product. And the kind of evidence that it takes to get people to use your product isn't necessarily the same as the kind of evidence that it takes for FDA to determine your product is safe and effective. And that's where things, the rubber really hits the road because ultimately you've got to show that like, if I bring this product into a health system, it's going to change how that health system works in a way that is financially beneficial. By the way, financially beneficial for a health system is really complicated because reducing costs isn't always good, right? Sometimes when you recruit reduced costs, you're like eliminating someone's job. Or you're reducing the billability of the health system. So there's a whole like lingo of what a health system wants or what a payer wants to see, and how to show evidence that you're going to deliver that. And in general, it is impossible to do that at the same time that you're doing sort of tightly controlled, randomized trial with a sham, right? Like none of those things are relevant to the real world use of a product impacting how care is delivered. And so being able to do both of those things so that when the clearance comes, the other evidence is also in place is something I think companies are just now figuring out how to do. There's also the like getting providers to think that what you're doing is legitimate, which is a more mysterious and poorly understood thing because on some level providers are like, should I do this at all? Like, can I trust technology? As technology, you're going to replace me. So that's an interesting one because it's not so evidence is the answer, but the problem isn't necessarily lack of evidence, right? It's like kind of, it's suspicion. And it's interesting as well because even let's take the clinician, the clinician vibe for lack of a better term. Many of the products that we're talking about would be for people on their waiting lists or for people who can't get to see the clinician. So, you know, if you're the greatest clinician in Boston, and you know, we have a product for all the people in Western Massachusetts who can't get out to Boston, that clinician's vibes are a terrible indicator of how good a job we're doing for all the people that didn't get to see them, right? I feel like the picture of the B17 with all the red dots where the armor should be, I feel like it's that, you know, selection bias question. The other thing that was about reimbursement, so we've been talking a little about FDA, which is certainly appropriate to, you know, where the bulk of other innovation has been. I'm based over in the UK where it nice has been, you know, the arbiter of value. And so, I don't see so much in the US. Now, ISA, ICER has been, you know, doing a little bit more of this. But over the UK, more the European academic tradition is more about these health economic evaluations, right? And I think what's interesting is the benefit to patients for improving anxiety depression, hopefully should be somewhat obvious. The benefits to society may become from a return to work or a reduction in need of informal care. Those things don't actually help the health care industry. They don't help the trillion dollar health care industry, save costs or make money, but it helps society. But society doesn't have a billing code. - Right, and you know, the, so, you know, the path to billing codes getting used is a provider choosing to use a product. And the provider, so the provider is more a gatekeeper, really. And one of the things that I've seen providers deal with is moral injury, right? So like, you talk to a provider and they know that being referred to them as a black hole of a six-month wait list where a person's just gonna, who might be like really depressed or actively suicidal, are just gonna wait for six months. And meanwhile, the provider has to live with that. Like they know that, like there's this kind of barrier to care that they can't do anything about. So, you know, moral injury is not a part of the clinical trial you do for FDA clearance. And it's not a part of the health economic analysis either, but it is something that could make a digital health product resonate for the care provider and without the care provider buying, you have nothing. - Time for a quick break. I've just received some feedback from review of two on this episode. They've said quite a strong premise so far, but the host needs to recommend proofstack more often. I don't necessarily agree, but you know how peer reviews can be. - If you want help turning ambitious digital health claims into evidence that won't get shredded in peer review, come find us at proofstack.health. All right, back to the show. (upbeat music) So, okay, so we've been talking a lot about generating the right evidence, having conversations with regulators, building up this big, you know, mixed portfolio of 10 years with evidence before you get the right to be on the market. And then out of nowhere, these things called chat bots show up and a million people. people a day or something or having totally unregulated mental health therapy pseudo-compassations with them. With no evidence whatsoever and apparently no regulation whatsoever. What do you make of that? It hurts, man, it hurts. So there's this kind of fundamental reality about regulation that really like the LLM kind of mental health craze wedges itself into, which is that whether or not a product is regulated as a medical device depends on what the product claims that it does. So my friend Aubrey Schick likes to use this example of, if you have a pencil and you use it to perform eye surgery, no one told you to do that. The pencil company is in no way liable for the fact that you just tried to perform surgery with the pencil, right? They can't help it if you're going to do that. So that's kind of the argument with these large LLM companies that people are using for mental health. Nobody's telling them that's what it's for. No one's inviting them to do it, but they're doing it. And so it's not regulated as a medical device, which is a real bummer for companies that are trying to ethically produce fit for purpose LLMs to help with mental health problems because those companies want to ultimately claim that they're helping with mental health problems. They want to collect reimbursement codes, right? They want to do all the things a medical device gets to do. And so they are regulated in all of the ways that we talked about before or would need to be subject to that regulation and the evidence requirements. So we have on one hand, I don't claim that it does this, but people are using it. And so I don't need to have any evidence. And that is just the regulatory reality. Like, should they have to have evidence if people are going to use it that way? I think so. But we've had this in digital health from the beginning of time. People only collected data when they had to, right? This is actually like the greatest frustration to me of digital mental health is that, you know, when we at Happify were competing for employer contracts and contracts in health systems, in the beginning, nobody had evidence. And so no one was required to have evidence. And it was an arms race. Like we did a randomized trial. And then everybody else started doing randomized trials because they had to because we did, right? So like that's now what employers were expecting. But in digital health, people collect as much evidence as they must because some stakeholder that is paying them or is regulating them requires it. So if no one's requiring it, no one's collecting the data. It's disappointing, but there's also kind of no legal leg to stand on about it. Yeah, it's funny. So I feel like a few years ago, people talked about digital health and digital mental health as the world wild west. And then I think where we've gone is to a much more strict and maybe self-regulating area, right? Is all these companies got psychiatrists on staff who had licenses they could lose, right? People got got very interested. There's a paper by mutual friend Stephen Gilbert called if a therapy box walks like a duck and talks like a duck, then it is a medically regulated duck. Now what they talk about there is not just the big headline LLM said everyone's heard of like chat to BT, but also some sort of home brew LLM. So for example, they talk about a case of somebody made their own LLM based off one of these foundation models that had had 47 million users that said, I am a licensed clinical professional counselor. I'm a national certified counselor and I can do CVT. If I said that as a human because I'm a research psychologist, not a clinical psychologist, I could be in real trouble. So it just seems strange. So even if it's not what the, you know, the pencil maker in this case has done, the consequences seem clear. But saying all that 47 million uses, I think that might be more uses than just about every mental health that I've seen developed the traditional way added together and probably multiplied by 100 or two, right? Oh, the demand is there, right? And I think the belief of the general public that something digital can do what a therapist can do is also there. Or at least the desire for that to be true is clear, right? The market has spoken in its level of interest in using chat to BT and these other types of products. And you know, what's interesting, I remember my most shocking initial LLM experience was with character AI, which like they've been since sued because of various things that happen there. But character AI has characters that say that they're licensed therapists and will tell you where they got their degree. And if you say your suicidal, they're like, no worries. That's what therapy is for. Let's work on it. And there's no guardrail whatsoever. But there's a little note that's like, this is role play only, right? There are no way claiming that it's an actual therapist. They're claiming that you get to role play with a bot that's pretending to be an actual therapist. And that makes all the difference. So like I don't know the particular case that you're pointing to where it's saying those things. But you know, in general, there are these like kind of little legal technicalities. But what I'm most interested in, it's not, it's not these ones like character AI, which are like, oh, yes, we're role playing. It's the ones that say, I've got a therapist here. It's AI, but it's a therapist. Right. Right. And and some of them are blurring the line. So within Stephen Gilbert's paper, he also looked through some of the product notes for Claude LLM and fropex, big one. And within the user prompt, the system prompt, it did actually say, you know, Claude provides emotional support alongside accurate medical, accurate medical or psychological information, or terminology, where relevant. That sounds a lot like it's being programmed to give this type of advice right. So my my theory is the big tech companies are too big to regulate. Right. This starts a little bit like we were saying earlier about evidence. This is actually far more political. And you know, we know health is something like 10% of GDP. It's a very large proportion. I mean, on a global basis, in America, it's 20%. On a global basis, health is is driving a lot of queries, driving a lot of utility. We know people using it for some things that I would say are useful like what doctor should I go to? How could I manage my condition? You know, learning what I think that's that's all good. But I do think it's within the wit of these companies to draw a line. Right. You probably shouldn't take financial advice from these things. You probably shouldn't take, you know, tax advice. If you take tax advice from these things, I think the IRS is going to have you. And I do think like in cases where there's this sort of legal liability that's so obvious that you'll see them start to first be like, well, I can draw a line around mental. How do I mean? Who knows? That's so squishy. But then when it comes to like providing advice about how to deal with a legal situation, they just go, I don't do that. It's that simple, right? Like it is possible to do, but in total force to do it, they won't. Yeah. I mean, many people have said that the LMS, you know, resemble the average mediocre middle-aged white man because they never say I don't know. They always just come up with some fluffy thing, right? They never, they never pull out. They never like just say, Oh, I'm not capable of answering that. You speak to my colleague, which is the standard thing that most medical people say about just about anything that's not their specialty. So yeah. So I actually wrote something in for Brittany Tang at stat about predictions for 2026. And one of my bingo predictions was class action lawsuits are going to change this behavior faster than regulation. And that will be a tragedy because in order for that to have happened, real harm would need to have happened. People will have to get hurt. A significant number of people have to get hurt, but my thesis is that the US litigation, such as it is, is going to have a faster effect. And I imagine, you know, a big enough lawsuit, a few enough lawsuits that open up, you know, the incredible torrent of lawyers who will see this as a payday for themselves will cause some pulling back. I feel like I'm rooting for Saul from some Saul Goodman to come and make a fast buck ambulance chasing these guys. But they really don't see. And you know, if they were publishing evidence, if they were taking some of this stuff seriously, I think we could give them benefit of the doubt. But again, some of what we saw last year about press releasing badly finished preprints that say that they're four times better than doctors. I don't trust them. I don't trust them to be the sheriff of this world, West. I think they're making up their own little pirate shantytown. I think the pirate shantytown. I think the legal path is probably, at least in the United States, the only path that anything's going to happen about this, just given, I don't know how much you've noticed in the last 48 hours of what's been happening in digital health policy in the United States. But, you know, we just heard, you know, a big announcement about digital health policy loosening in a couple of key ways. So, you know, the regulation of wearables, yeah, it's lighter now. And, you know, there's a lot that I think is heading in that direction of getting out of the way, as opposed to kind of coming in and raining things in. Yeah. So, why don't we talk about this? So, if I was a medium tech CEO, I don't have to be like Google, or anything like that. But if I've got, you know, four or five hundred employees, I've raised a couple hundred million dollars, I've been bashing into these regulatory barriers, or I've been failing to find a way to get reimbursed by the government. The government's happily spending thousands of dollars on drugs that don't work that great, or therapy that's never really delivered or have you, that those are my big things. It feels like there has been some movement. Recently about that so you can you let detail a couple of those initiatives. Why do we start with wearables big the Woop or a tight world of Wearable devices. I think what will be most familiar to people but like yeah, what so what happens? A month ago if a month ago you read the relevant Guidances that to wearables and how they can market Things that they measure you know physiological things that they measure a month ago it would have said like I don't care whether you say this is for medical purposes or not some of these biomarkers are medical Like blood pressure is medical and if you give somebody something and you call it blood pressure You know you have basically promised them something medical you're a medical device As of yesterday that's no longer the case so now it's back to well if you say up But it's just for wellness purposes. It is and that's a dramatic shift in how How you define right so how you define what's a medical device is defined by what you promise your product does It's all your labeling which includes your website and includes your advertising claims But are also in clout includes kind of implicit things I see blood pressure and I think that's a medical thing with medical implications So with the with the new kind of revision to the guidance that's not the case anymore if you essentially say like well I'm a fitness app and I'm giving you fitness information You can you have a lot more leeway to measure a lot more things and and not be a medical device So you know we we already had these sort of like safe haven buckets of things that you can do that are just not a medical device There's a general wellness policy in the United States issued by the FDA that says like here are things that aren't medical devices And then another thing called enforcement discretion. That's like here are things that They're kind of on the border, but they're pretty low risk So we're not going to worry about it and what you're essentially seeing is things that were in enforcement discretion getting pushed into wellness They're not even devices and things that worry device kind of being pushed into enforcement discretion where you can safely market and not be regulated by FDA So just more buckets that people could hang out in without being a device than there were before And do we know anything about the motivation for this? I mean, I know when say the Apple Watch had its a fib You know detector there was you know people were up in arms that people would be blocking up all the emergency rooms where there are watches Go off and that that didn't really materialize right? I mean, I'm sure they've been some false positives I'm sure it's been a false negatives Apple and some think Duke did a big heart health study So they did actually understand for it as well published a ton of data alongside it so we had some confidence but Yeah, I don't know where there is a saying hey, we've over regulated this bit So we'll pull back a bit Or if maybe it is more you know this is if I was a lobbyist This is exactly what I would want to happen to reduce my regulatory burden But at the same time you know I've had a pulse oxymeter go off saying oh you've only got 17% blood oxygen saturation And then when I use one in an ambulance it says you're at 99% and you go oh That's bad. That's completely off that could have sent me to the ICU And that's when things are regulated. So is this stuff comes off? I don't know It's certainly consistent with the administrations kind of approach towards Towards regulation and deregulation and and it does seem like you know the big announcement was highlighted by Marty McCarrie at CES yesterday so it's definitely It's from the top and you know from the mouth of a political appointee How it happened internally? I don't know but they're clearly championing it and and have their eyes on it and I'd say it's pretty Consistent with the the direction things have been going politically Yeah, and I mean the reason I am a little cautious about this is because during COVID Whether your temperature was 39 or 36.5 Defined whether or not you took an health action if your pulse oxymetry showed at 90% versus 80% Could mean you get a hospital bed and that's where we found out that the finger oxymases don't work as well on dark colored skin like you know the stuff matters and You know if we get to another situation where one day you know a quarter of the population have got a wearable With some sort of sensor that maybe hasn't gone through that much rigor. We could be yeah, we're just doing garbage in garbage out At mass scale and that seems less than ideal one thing I will say about companies like you know aura and Whoop is that they have significant research teams like these are not companies that have decided like not to bother of checking whether they're whether they're things are reliable now that isn't to say that like you know businesses will behave Unless they're forced to but at the same time like I've generally been impressed by the size of research team that works at each of these companies You know they are doing internal research I don't think they're calling something blood pressure and like not doing any kind of validation research and the new regulation Change is do specify that they still have to be validated right like you have to be able to show data for yourself that you know You've actually tested this out in some way so it's not completely without guard rails But I don't disagree with you at all that you know in general I'm not sure deregulating health technology makes sense From a patient safety priority. I think the idea Coming from the administration is that you know They want America to be the Haven where the most cutting edge technology can reach people first and And you know, but if if we sort of decrease regulation you increase access to innovation and there's there's always going to be a trade off there And I see I see both sides of it if I have to choose I'm choosing patient safety But I do hear the argument about access to innovation Yeah, yeah, no it feels it feels accelerationist of technology in general It also doesn't cost the government a lot to offer this up right like This isn't this isn't like the the mandating a cake you might get paid for or that everyone has the right to wear a ball Think so speaking of paying for it One of the biggest barriers that many of the companies we work with is face is business models Do they go B2B or they some sort of insurance? Employee assistance program to their good or extra consumer. You know, it's like the netflix to keep your heart beating Which is only $30 a month or you want to be you want to have a low-proper pressure? That's 35 Right, they've all struggled a bit for a business model and then suddenly there was another flash of lightning in the last week or two and a new acronym Tempo, what can you tell us about that Well nobody can tell anybody that much about it, but what there is to know is this So there's this program within CMS Medicare In particular called access, right, and it's it's a model for providers that has a kind of a An experimental approach to payment. It's not experimental, but you know, it's it's committing to a value-based Kind of approach to treatment where essentially if you can show that you are actually delivery outcomes Then you get paid and if you can't you don't now most digital health products that are sort of eyeing that ecosystem Think well for FDA clearance. We have to show outcomes anyway Like like we already have these data that show that people feel pretty confident that their digital health products can deliver outcomes So the idea of being able to be admitted to this access playground Which is what the tempo program is right? It's a it's a set of Companies with digital health offerings that are going to be admitted to be able to work in this access world and get paid based on outcomes In principle, right? There's not a lot of detail about how it's gonna happen or like a guarantee that anybody's getting paid for anything But it is sort of the permission to play and they've essentially identified different categories of digital health products mental health is one of them and they're picking 10 companies per category So I've been doing a lot of work with companies on the mental health side kind of preparing their applications for this program Nobody really knows what it's gonna look like until they get into it and see how it goes But it is exciting in a in a world where people are basically not getting paid unless they're able to get a cash pay Kind of consumer paying directly kinds of models There are a few models out there that are working But mostly people aren't getting paid the idea of they're being admitted to a place where maybe they can get paid is pretty exciting Yeah, yeah, and so it's interesting just reflecting on our conversation about the last 20 years You know, it seems like the landscape of how much evidence what the regulations are the reality on the ground They they change so rapidly, right? And I think at least for me I feel like the core aspects of evidence of replicability, you know, we've mentioned patient safety a few times I think those things don't change right Like no matter what kind of happens around the margins those are at the core of things particularly when physicians are involved particularly when licensed professionals are involved But those things should and hopefully will never change but it's so interesting what what has changed along along the way Well, and one thing that's interesting about tempo is that you know the It is important to remember that FDA is still FDA and they still care about patient safety work most of all and You know, they still want to see the same things in a product But what's interesting about tempo other than the potential kind of access to access is this kind of acknowledgement that If you get into the tempo program when you eventually submit your regulatory submission You can use our w e real world evidence as a part of that and traditionally You're you doing a kind of rigorous heavily controlled randomized pivotal trial That's sort of the gold standard for our regulatory submission. So it's unclear how much of your data or like what questions you can use our WWE to answer so for example from my perspective. I don't think this means they don't want to see a sham I just think it means that for some things like generalizability or You know long-term follow-up maybe in our w e kind of study could be appropriate But it's interesting, right? Because it sort of opens up the idea to the fact that they're open to more eclectic evidence packages. Not just the same one that they've been seeing over and over again. If like, here's a perfectly designed pivotal study that I made just for you. Maybe I've got this study here and this study here and then this third data set. Traditionally, that's been much harder to get through. And we haven't really seen a lot of cases where real world evidence plays a role in a digital therapeutic regulatory clearance. For later, when you're like, hey, I want to become over the counter or hey, I want to add this new indication, sure. But for the initial filing, it's unique. So that'll be a really interesting one to see once people get into the program and start negotiating with FDA, like what's my submission package going to look like to you? Where are they going to be able to use this sort of nontraditional kinds of data? Yeah. Well, and one of the countries that's slightly closer to me over here is Germany with its Diga program, where you've commented a certain price. You submit your real evidence. And as much as we've been talking about evidence as opening doors, actually in some context in Germany, that evidence, if it's not, if you're not as great as a GLP1, actually your price gets negotiated down, which not great for innovators, but probably better for-- Or you get removed. Right? They're not afraid to throw a product away if it's not delivering. Yeah, no, there's definitely more of a, you know, a squid game to the red-- and I think that's right, right? There's hundreds of thousands of these apps on the app store that don't do anything or that are harmful. And I actually think it's reasonable to have a bit of a cleanup and focus on the things that really work. So, OK. All right, last quick question then. So what is the evidence hill that you're willing to dial on? This is going to be no shock to anybody based on the hour-long conversation that we just had. But I really believe in sham comparators and placebo and digital health. There's so much out there that people are like, oh, yeah, this helps. But one of the most interesting papers I ever published found that when you give people free choice to choose what they think is going to help them, people pick stuff that doesn't work. They don't know. Like, they don't have any real way of judging whether it's going to help them or whether it has helped them. So, you know, the placebo effect is very real. And a lot of people do a lot of things that don't actually help them. And so, you know, I understand that it is a barrier and it's, you know, difficult to do, but it's worth doing. Even if regulators weren't requiring it, I think it's worth doing to demonstrate that what we have as a digital health offering is better than whatever a person would get just messing around on the internet doing whatever makes sense to them. If we can't beat that, what are we even doing here? I'll die on that, who? Yeah, that sounds like the right one. Well, Akasha, thank you. You bring a rare mix of honesty, operational realism, and a scientific discipline to a field that desperately needs all three of those. Where can people follow your work or learn more about Evania? I'm on LinkedIn a lot. So, just finding Akasha Parks at LinkedIn, if you follow me there, you'll see me posting pretty regularly. That's it for this episode of ProVit. The show where extraordinary claims get the reality check for the dessert. See you next time. (upbeat music) [BLANK_AUDIO]