Atopic Dermatitis, Large Language Model Accuracy in Pediatric and Adult Medicine – Ep. 251

from the American Academy of Pediatrics. Nearly 10 million children in the United States have atopic dermatitis or eczema. A new clinical report from the AAP provides updated guidance about this common condition. We have a lot of options. The toolbox is much bigger. You can see that outlined in the clinical report. And I think it's going to continue to grow. I know there's a few exciting things on the horizon. Plus, we'll discuss the accuracy of large language models in pediatric and adult medicine. For learners and medical education, I think first of all, using tools like LLMs to practice for exams while on pediatric rotations, different subspecialties given the range of questions would be a great way to study. But keeping in mind that those have limitations, they're not always accurate, but who is? The story is more on the Pediatrics Uncalled podcast on children's health from BAAAP. I'm Dr. David Helm. And I'm Dr. Joanna Pargoblinke. This podcast is for everyone in the medical community keeping our children healthy and safe. That's a show by pediatricians, four pediatricians. But anyone who cares about kids will get a lot out of it as well. Let's start today's episode with a discussion on treating atopic dermatitis. David, what was the hardest part of your med-peed residency? Oh, I remember it so well. I was coming off on my first three months on internal medicine and I started in the pediatric outpatient clinic the first day. I had this patient with the most mysterious looking rash and I just couldn't make any sense of it. So I got the senior resident and she glanced at it and rolled her eyes and said, "That's eczema. Her tone made it clear that I was already far behind my pediatric classmates." But eczema can be complicated. It can present with bumps or patches, scale or in patego and it could show up in so many places. I'm still not sure that even I completely understand it. There was a while there when I thought I did understand it, but the science is moving so fast that my understanding may be out of date. Maybe we can get some help today from Dr. Jennifer Schuch, a professory pediatric dermatology and physician scientist at the University of Florida and one of the authors of a new course of the clinical report from the AAP, guidance for the clinician in rendering pediatric care, atopic dermatitis, update on skin directed management. Jennifer, welcome. Hi, thank you. Thank you for having me. Jennifer, I think you heard David's story from his internship. How do we diagnose eczema and is it one of those, know when you see it kind of things? Well, it is for me, but I get that it's not that easy. I think that there are the clinical patterns that vary depending on age. So in infants we see more eczema on the cheeks, for example. And then older kids we see more eczema in the fluxures, like in the elbow creases and the hindonies. But it can be hard because eczema can look like anything from bright red, dry, cracked, and scaly skin. To, in our darker skin patients, can be a little more subtle. It can just be a lot of hyperpigmentation and thicken skin. So I think that I usually can figure out what eczema is. And pediatricians do a great job as well, but we're all stumped sometimes. Well, Jennifer, I'm going to go back in the past again. And I'd love you to settle an old argument between my pediatrician father and me. I was trained to think of eczema as sort of an allergic disease, like asthma, allergic rhinitis, you know, a topic. He said it was really an issue of skin moisture and dryness. We're either of us even close. I would say you're both right. I can't decide this more right. Sorry. I can't settle the debate. So both things are a factor, right? So the skin barrier is defective. It's not holding onto moisture well. And then the immune system is activated. And sometimes we don't know if it's inside out or outside in is the immune system activated. And it is causing defects in the skin barrier or is the skin barrier defective? And then it activates the immune system. The latter seems to be a little bit more true, but honestly, we're still not completely sure. And then there's other things that play a role like the microbiome on the skin. If there's any infections present, things like that. So I'm sorry, I cannot settle the debate. We keep talking about skin. And of course, the clinical report title includes skin directed management. But does eczema affect only the skin? Oh gosh, no, no. I mean, that's what we see in dermatology, but it doesn't. It affects sleep. And that's a big issue. And then sleep is important to everyone's mental health, not just the patients, not just the child's, but then family as well. And that's definitely true for infants and toddlers and their families. But also for the older kids, where sometimes it contributes then to ADHD to having more problems with depression or anxiety. So it's really a whole person disease that maybe starts in the skin, but there's a lot more to it clearly. And can you speak a little bit to the infectious complications that you can get with eczema? Sure. Eczema exists outside of infection and then sometimes infection joins the party and makes things worse. I think that the best example of that is is meloscum, meloscum contagious. And that tends to go with eczema, like they like to make each other angry. And we also see something called eczema herpeticum, where the HSV virus really likes that defective ui-gui barrier of the eczema skin. And there's a couple other things a little bit less common, like something called eczema-cock stachium, which is a virus that can also, you know, particularly happen in eczema areas, even if a kid otherwise has their eczema under pretty good control. So definitely complicates things. I do think there's an overemphasis on infection, like bacterial infection eczema. I think a lot of that's more in the microbiome shifts than we're really starting to understand that over the last few years. And can you speak to concomitant fungal infection as well? Sure. That's an issue, particularly with separate dermatitis and infants. That's when we see it the most. So the infant has a little bit more malacisia on their skin, or I guess it's probably the same amount, but we tend to see a pattern of an overlap of subrate dermatitis and atopic dermatitis, where the atopic dermatitis is kind of happening in the separate areas. And my guess is somebody who studies the microbiome is that it's the immune system, which is already going to be primed towards eczema over reacting to that malacisia shift on the skin. And we see that in later kids too. It's actually something that we sometimes see on kids, wouldn't kids who are in biologic therapy, where we see malacisia, which is a fungus, as a trigger for eczema. So that's the tightest connection we see. We don't see a lot of like athletes foot with eczema or anything like that. For a while there it seemed like there was a lot of focus on preventing eczema, either by controlling the child's early diet or by applying moisturizers daily, or either of those preventive measures effective and if not, are there others that are? Great question. So this is really shifted over the last few years. There was a great study that came out that everyone got excited about, that we thought, okay, if we can put a mullion on children earlier, maybe protect that barrier, then we can prevent eczema. But unfortunately subsequent studies showed that's probably not true. So I was really hopeful I was hoping that would be something we could do to prevent eczema, but it didn't really pan out like we were hoping. I do think that there's a little bit of something to treating the skin gently if you are an eczema prone family so that you're not aggravating the immune system, getting it more active, particularly in early childhood. So I still recommend gentle skincare, gentle moisturizers for our families that have a topic to be tightest or atopic disease in their family. The other thing about diet gosh, you know, we get asked this all the time and I just really don't think diet makes a difference. There's a lot of data. There's a lot of different opinions on this, but at the end of the day, there's really not a strong trend in the data that I'm aware of that shows that there's anything we can do diet wise to prevent eczema. There's a little bit coming out in terms of oral probiotics and that will be interesting, but I don't think that eating particular things or avoiding particular things will change the outcome for eczema. In fact, if you're familiar with the leap trial, so we actually kind of shifted our thinking on that. And that trial we found out that gosh, actually we're doing the wrong thing by avoiding peanuts. We're causing more peanut allergy by avoiding peanuts. So I try really hard to encourage our families with eczema to not avoid foods, but instead expose their kids to as many foods as possible. And what about treating eczema when you have it? It seems like the days when we just had antihistamines and topical steroids are kind of far behind us now. Oh, yeah, we have a lot of options. But you know, topical steroids are still great. Antihistamines evidence is a little bit more iffy. You'll see that in clinical report too. We do rely on topical steroids. We try to emphasize in the clinical report that they are still considered first line and that it's okay to use them safely. There's an overemphasis. And this is generally felt amongst dermatologists and overemphasis on like a fear of topical steroids. But unfortunately we end up under treating a lot of kids and kind of forgetting that eczema is a chronic disease. So we really encourage families whether it's topical steroids or a topical steroids sparing agent to treat the disease until the disease is under control. And then if we don't get any control of topicals, we just have to go to the next steps. So though if topical steroids don't work, if you're really encouraging them in a patient and having them. use them liberally, what else can you recommend? - Well, again, yeah, our toolbox is much bigger now, thankfully. So recently, topical jack inhibitors were approved and we have those available for kids with eczema. And then after that, there's next steps where there's systemic treatment and because the reports focused on topical therapy, we just touched on that briefly. But there are oral jack inhibitors in addition to the topical jack inhibitors and those are sometimes used for eczema. First line for systemic treatment though right now is biologic therapies and they're really great. The advocacy date is really great. The tolerance is really great. So we have a very low threshold for the kids that need it to move from the topical therapies into those targeted biologic treatments. And I think at threshold, they're approved all the way down to six months of age, actually. And so that threshold has really gotten a lot lower, even just in the last couple years because we've got good 10 years of experience now and they're seeing, they really do seem to be safe. And if you're using steroids as a first line treatment, but then you're worried about a super infection with a bacteria or a virus, would you have to pause the steroids or can you keep going for that treatment for eczema? - Great question. Usually no. In fact, there's studies for eczema herpeticum that show that we should continue to treat with topical steroids. And for most bacteria, infections, quote unquote infections is actually probably more of a shift in the skin microbiome because there's more in the skin staff in that Iwi-Gui eczema. So I think that the whole, like, is it infection or is it just the eczema thing in general and continue the topical steroids? The only minor exception to that is moloscum because moloscum, the erythema you're seeing around that is a response to the virus and is fighting off the virus. And it's a longer term fight, right? So what I tell people is if the eczema's so bad that they're itching at it and it's miserable, then you have to treat with the topical steroid, but if it's just the redness there fighting off the moloscum, then leave it alone. - And in addition to pharmacologic therapies, I know in the past, we've used non-pharmacologic interventions, including dilute bleach baths or wearing wet pajamas, or any of those measures still useful? - Yeah, I relied on those heavily in residency and training. They're still useful. They're just not used as commonly. And so I think the part of this for me in terms of my practice was kind of becoming a parent and realizing how cumbersome those things are. So I tell parents all the time, like here's some additional tools if we need them, but again, with the availability of the biologic there being the bigger toolbox, we don't rely on them as much. So the data on bleach baths is a little bit mixed. It probably helps to make a healthier microbiome and helps a little bit with atopic dermatitis, but certainly not a cure. And then wet wraps are really effective for focal, atopic dermatitis or a flare in a patient to otherwise as well control with topicals. - So I remember back when Kelsey Nuren inhibitors first came out, we were afraid they might increase children's risk of cancer. Where are we on that now? - Thank you for asking that. So I don't think we need to worry about that. I get calls from pharmacists about that. Every once in a while somebody sees the black box warning and freaks out, but those studies that we were initially concerned about were animal studies and higher doses. And there's subsequently been published studies with thousands of patients that show they are safe and effective. We use them in infants younger than the age that they're approved for. So I guess that just speaks to how comfortable pediatric dermatologists feel with them. And it's usually my first line after topical steroids. So definitely a huge tool in our toolbox. - And you already mentioned that Eczema is something that really affects the whole person. And I've often heard dermatologists call Eczema the itch that rashes. So knowing that itching is a key symptom that drives so many patients crazy, what can we recommend when it is really bothersome? - So I was told people the best way to treat the itching is to treat the eczema. So it really doesn't actually work that well to treat the itching if you're not effectively treating the eczema. So what that tells us is that we need to step up our eczema therapy, whether it's going to biologics, even though maybe we're not seeing as much eczema as we thought, or working something else in from our toolbox. The other thing is that, you know, like I just mentioned in the clinical reports, antihistamines aren't really as helpful as we thought. They sometimes can be helpful because a lot of kids have dermatographism where they scratch and then they hive up. And that just makes it worse than they have eczema and they scratch and they hive up in those areas. But it's not really the cure. I guess I will say since we talked a little bit about systemics, there is a new biologic coming out that focuses more on itch. And it'll be really interesting to see for our severe patients if that is going to be maybe used in the toolbox, maybe even with some of the other biologics for the patients that are really itchy. So you talked about using antihistamines. Do you recommend Benadryl? Oh, thank you for bringing that up. No, we try to avoid the older antihistamines. We try to stick with a newer generation antihistamines so that they're not causing as much drowsiness and that they last longer. So Benadryl doesn't last very long. There can be side effects. And there's actually some studies that show that maybe using those medicines long term, there might be some issues with learning problems, which maybe are just confounded by the disease, by the eczema. But I think there's enough there that I almost never recommend Benadryl except for emergencies. You know, you mentioned mental health as one of the issues with eczema. How do we counsel teenagers who are just really embarrassed if they've got visible lesions on their skin? Yeah, that's a great point. This happens all the time in our office where maybe on that day, the eczema doesn't look that bad but the teenager is describing, well, I can't participate in sports I want to because it makes my eczema worse or I'm really embarrassed by how this looks. And again, thank goodness we have more tools in the toolbox and we just go to the next step if needed. And really, that's where we use the biologic therapies if we need to step away from the topical therapies. And we take that into account in terms of making the treatment plan. We don't undervalue that. We make that part of the decision as well, even if that's not what we're seeing that day. Well, Dr. Schuch, I may feel up to date on eczema for the first time in my career. So thank you. And I'm going to make sure my dad listens to this podcast. I imagine our listeners want to read this report and its entirety. But before then, can you leave them with one takeaway? I think my takeaway would be that please do not be afraid to treat eczema. Please be our first line, treat eczema as a chronic disease. Follow up with maintenance therapy. You'll see that in the report where we're treating the eczema in between the eczema flares. And then make sure to refer to your subspecialist when you need us. We have a lot of other tools including those biologics. And we are very happy to partner with you to help our patients get better. Dr. Jennifer Schuch, thank you so much for talking with us today. Yeah, it's a pleasure. Thank you, guys. To read the clinical report, guidance for the clinician and rendering pediatric care a topic dermatitis update on skin directed management is at aap.org/podcast. Coming up, we'll look at large language model accuracy in pediatric and adult medicine. Looking for practical resources to support your work in child development and disability, the American Academy of Pediatrics offers free pedaling courses for pediatricians and other child health professionals. Explore topics including ADHD, Tourette syndrome, autism, developmental surveillance, and more. These self-paced courses can help strengthen your knowledge and support your care of children and families. Visit aap.org and search keywords, child development, and disability courses to review the course list and MOC credit information. The other day I was in the Neonatology Fellow's workroom and we had a question on milk fortifiers for preterm infants. So my fellow asked ChatGPT to help us out. Okay, Joanna, I'm really curious now because ChatGPT caused me to miss a question on a practice test a couple of weeks ago. How did you when you were asking about something as nuanced as milk fortifiers? Honestly, it was mixed. It pulled information we could only assume was from various companies that make milk fortifiers on the internet. So we did feel the recommendations that gave her a little bias. And I'm nervous about asking large language models, pediatric questions because I don't know how much they incorporate information about human development. Well, Joanna, you are not alone. There's a new paper in Pediatrics Open Science, PEDMQA Comparing Large Language Model Accuracy in Pediatric and Adult Medicine. And the senior author, Dr. Esley Osmanliu and lead author Nick J. Swall, are here to discuss. Dr. Osmanliu is an assistant professor of pediatrics and an emergency medicine physician at Montreal Children's Hospital. He does research focusing on how innovative technologies can be applied to patient care. Nick is a graduate student and first year medical student in the Department of Family Medicine at McGill. Esley, Nick, welcome to the show. Thank you. It's a pleasure to be here. Thanks for having us. When I was reading this article, I didn't know what MedQA was. So I had to use a large language model to find out. Nick, can you describe for our listeners what MedQA is? Of course. And that QA is a- a large benchmark dataset of United States medical licensing exam style multiple choice questions or USMLE questions. Why do you suggest how well large language models can answer medical questions? And it contains roughly 13,000 questions and answers. Well, that's a lot. And so what's PID med QA? So PID med QA is the pediatric subset of med QA, which is not organized currently by patient age. And actually using med QA and PID med QA, what did you learn about pediatric multiple choice questions for tests like the USMLE? Well, it was great to see that there's actually hundreds of questions per age group. And we went between newborns all the way through adolescents. Each age category had hundreds of questions and those are multiple choice questions. And we also were pleased by the range of sub-specialty based questions and general pediatric questions within that subset. So Nick, how well did these large language models or LLMs answer pediatric multiple choice questions? Were they accurate? Well, ChachyPT4 Turbo, which was the latest model at the time we conducted our evaluations, scored roughly 75.5% on adult med QA, and roughly 78% on PID med QA. And actually, what does this mean for how we use these types of large language models in medical education? Well, I think, first of all, we're wondering whether we can rely on these tools in pediatrics. There have been some reports in the last few months, last couple of years, showing that performance in pediatric questions from these large language models was not as impressive as an adult medicine. So it was actually, first of all, reassuring. And this was, again, an assessment of multiple choice questions. Those were not complex reasoning questions as we've seen in prior publications. But it was reassuring to see that it was on par, if not actually a bit better, about 2% of points better than an adult medicine. So for learners in medical education, I think, first of all, using tools like LLMs to practice for exams while on pediatric rotations, different sub-specialties given the range of questions would be a great, great way to study. But keeping in mind that those have limitations, right? They're not always accurate, but who is? And for instructors, I think it's a good way to see how learners are doing to personalize feedback, perhaps seeing how they're interacting with the material. If there's some sections within pedMed QA that learns are having a harder time with, maybe that's a way to personalize medical education. But, Essley, can you comment a little bit on what you were thinking about, which is using large language models for more nuanced things like diagnosis and maybe even treatment of patients? Do you have thoughts on where we are with that? Joanna, I think in your intro, that was a great example. I imagine the trainer you were working with didn't ask, "Tad G.P.T." a multiple choice question about the milk, right? That's right. So, the evaluation we conducted was a bit more constraint, right? It was easier for us to assess the performance given this being an established data set. So we're looking for a way to see how to do some pediatrics. But, this is one component of performance, right? This is accuracy evaluation. It's an important component of safety to make sure that the information that's being released and being proposed by the LLM is actually accurate, but there's so many other components. Of course, we're unable to assess for bias, which is a big concern, right? And interacting with LLM's in health because there was no significant information on social demographic characteristics or structural determinants of health for these cases. So we couldn't really see if by changing some of these characteristics, the responses would be different, the performance would be different. But that's definitely something looking forward to evaluating the future. So, that's one. Also, in terms of uncertainty management, right? It was accurate on average about 70% of the time. But it'd be interesting to see if it's able to conduct some form of reasoning about uncertainty. Does it know when it doesn't know? It can give us a sense of, hey, you know what? This answer, I'm maybe 50% sure. This one, I'm 99% sure. I think for clinicians, having that type of transparency in terms of the management of uncertainty would be really helpful. I say, I'm absolutely loving that thing you just said. Because as clinicians, that's a lot of what we do is say, hey, I think this may be the case, but you know, I may be wrong. That's fascinating. Well, in addition to that, I would love to hear a takeaway from each of you that will make pediatricians want to go read this paper. Esley, can you start? Absolutely. My main takeaway is that for multiple choice questions in pediatrics, different age groups, different sub-specialties, chat GPT seems to perform on par between pediatric and adult questions. And Nick. So, one thing that I'd like people to take away is the ability to use Piedemet QA as an automated and open source way to evaluate LLMs over time in clinical context. While I'm hoping that pediatricians will look to this paper to start thinking of doing those things. Dr. Esley, Osmond Liu and Nick J. Swal, thank you so much for joining us today. It was a pleasure. Thank you for having us. To read the paper, Piedemet QA, comparing large language model accuracy in pediatric and adult medicine, visit our website at ap.org/podcast. At the end of every episode, we'd like to leave you with something positive or enlightening. We call this segment, say, "David, what's your say this week?" Well, China, most people do not look forward to long car trips, but I got to drop our youngest Julian off at his summer internship in DC. And that meant close to seven hours trapped in the vehicle with him and boyd it he opened up. Sometimes, in ways that were a little frightening, not gonna lie, but I treasure that time, some of the best time to just hear where he's at in life and what he's thinking and share thoughts and I wouldn't give it back even for a short flight. So you know this already, David. I don't think I would listen to you, but my daughter Zelda, my 17 month old, broke her arm with a fush, which was an acronym I learned. Fush fall on outstretched hand, very common. So she has a little super-congratly fracture. And that's not the positive thing. The positive thing was she had this incredible orthopedic surgeon, Dr. Williams, who saw us a week after the injury after she got casted in the ED and took off the cast and you could see that it was like a little bit of a question if it was a fracture, but it truly was. But it was just this beautiful Dr. Patient interaction where we had this great kind of discussion about what we wanted to do with her care, whether we were gonna recast her or whether we were gonna splinter. You know, and I was nervous about splinting her, David, because I haven't done that since my days in the ED as a resident. Right. But the splinting is going well, and I just really loved to be with a surgeon who did such great shared decision making. So, my say-as for Dr. Williams, who was just a really supportive doctor in all of this, and it just really made me appreciate that subspecialty. Oh, wow. You know, we talk about effective communication so much. I love hearing that you saw such a wonderful real-world example for it when it really counted. That's it for today's episode. If you like Pediatrics On Call, please subscribe. On your favorite podcast app, and help us spread the word. Follow us on YouTube, Facebook, and Instagram at Pediatrics On Call. Pediatrics On Call is a production of the American Academy of Pediatrics. Our producers and editors are Anne Johnson and Claire Keating. Our audio engineer is Doug Nagle. Joe Puskars is our associate producer and Susan Martin is our executive producer. Our theme music was composed by Matthew Simonson. Join us next week when we'll discuss achieving equity in clinical guidance. We'll talk to the authors of a new article, Dr. Joseph Wright and Elise Portillo. I'm Dr. David Helm. And I'm Dr. Joanna Parker-Villinki. Thanks for listening.