Bridging the gap to medical knowledge - OpenEvidence

Hey everyone and welcome back to the Medical Innovation Podcast. My name is Brunov, I'm here with my co-host, Urban. And today we're speaking with one of the co-founders of arguably one of the most important companies in healthcare today. We're speaking with Zachary Ziegler, the CTO and co-founder of Open Evidence. Zachary, it's so great to have you. Thank you so much for joining us. Would you mind starting off by telling us a little bit about yourself? Yeah, absolutely. So I'm the CTO and co-founder of Open Evidence. I come from the technical side of things, so my background is academic machine learning. I come from a PhD program at Harvard Computer Science, working with Sasha Rush. And my research there focused on the mathematical underpinnings of the types of models that we now call large language models. We used to call them probabilistic gender models and all sorts of crazy stuff. And they've now evolved into this new world we live in with large language models. And so, yeah, that's kind of my background where I come from. My connection to healthcare originally is kind of as a consumer. We live in this world where there's an enormous amount of noise about healthcare and frankly a lot of other things as well, where from any different source on Google, there's just tons and tons of garbage and very little actual truth. And what we started kind of working through very early on with Open Evidence is kind of taking really what should be kind of simple questions or easily answerable questions like does a key to genetic diet really work? Is there any evidence that some skin cream reduces wrinkles or whatever it is? And it turns out it's like pretty hard to answer those questions normally. And what's crazy is there's actually a huge amount that we know as a species about medical science and biomedicine. There's about 40 million publications that have ever been published on biomedical sciences. We see updates that are about 10 to 20,000 new or updated publications every single day. So there's this huge amount that we know as a species, but it's really hard to access that information. And all these questions are going to answer actually do have really good answers. And what we set out to do with Open Evidence is make a system that can essentially address that question. Yeah, and I think it's been incredible to see the uptake of it in the clinic and seeing not only myself as a student but even attendings that have been practicing for the years. And we'll have to use it. We'll have to hear a little bit more about what kind of inspired you to start Open Evidence in terms of what was the initial product, was it kind of what it looked like today in the sense of being able to pull from medical literature to support claims or was there completely different first MVP, I guess, of what the product was? Yeah, that's a great question. So we've had as our North Star this idea that really resonated a few years ago and still resonates today that there's just this absolutely enormous haystack and this enormous fire hose of information. And the real question is how do you make that information accessible? That's the problem that Open Evidence solves. Early on we explored a few different modalities. We started pursuing this before Chattachypt and before the whole world woke up one day and realized that AI was the next greatest thing in the entire world. And so we've explored a handful of different modalities. But I think especially the beginning, COVID was a big wake up point for all of us where it became clear that some of these questions like I'm talking about about diets or different kinds of things that are relevant for consumers, that these types of questions are actually not just consumer questions, but really at least as important or challenging for positions for all sorts of, you know, both in like med students and residents and trainees, but also attendings and senior physicians as well. In terms of, in terms of being able to, you know, when at the very beginning of COVID, there is something like, I don't remember the number, some, you know, in the order of like 10,000 papers that are published in like the first week or the first month or something, which is crazy. And it's like, how do you, how do you practice in a world where there's both and so much information that's out there, but also, you know, uncertainty about that information or nuances that are really important around the details of what you can learn from different pieces of evidence and what's applicable in different situations and stuff like that. And so really honing in on finding exactly the right references and helping positions meet this need is really what we started. That makes a lot of sense. I mean, the amount of medical data in this world is growing at an unimaginable pace and it's next to impossible to keep up with everything. So you're definitely solving a real problem in trying to make the latest information, the latest research available to providers and clinicians, people that are actually caring for patients and need to have the latest information to be able to provide the highest quality of care. So it's a phenomenal idea and a phenomenal product. You mentioned a little bit about how the, or the idea came to be, but I tell us a little bit more about the beginning. How did the team form? How did you and Daniel come together? How did you get started? Yeah, so Daniel, I met with both high connections through Harvard. And then we grew at our team initially, largely from Harvard MIT. And some of our biggest objectives with building a team for this were to kind of merge extremely strong engineering talent and really passionate engineers with kind of equivalently passionate and exceptionally talented scientists and physicians. And so that was kind of our ethos in building out a team for this. We've maintained an ethos of keeping an extremely small team. So even though now at this point we're helping hundreds of thousands of that millions of physicians around the world, three patients were still a team of about 20 people, I'll tell maybe 25, I'll think added in. And I think there's a lot of really good reasons for that. We've always tried to just more than anything else make a really, really excellent product, make something that just feels really good to use. I think people kind of have a sense of what that is, kind of the design ideology of like an iPhone, for example, where you just pick it up and it just, everything just works. Everything's really just really thought out and really, really well designed. And that only really happens, especially when you're just starting out, if you have a small group of people that is just like perfectly in sync and perfectly aligned, not to mention being really great engineers and really thoughtful future thinking physicians as well. And so we put together a team that was really focused on both building really great things, but also thinking very carefully about what we should build and what the product is and how to make something that just feels really excellent when you use it. Yeah, and it's really, I mean, from using open evidence, like right now I think it's very clear how powerful it is. And I think originally, sometimes I'd get I'd put in queries and it would, you know, maybe not process it as I would like, but then like, you know, as, I mean, that was probably like quite a while ago, but now it's really strong and how it's responding and being able to pull evidence and high quality guidelines and things like that. We'd love to hear about what are some kind of challenges that you've had initially with building open evidence. I could imagine it's difficult, especially like you said, when there's so many sources and there's a lot of research out there, but at the same time, there's a lot of research that's maybe not as high quality as, you know, we'd want it to be. So I guess sorting through those things, making sure that the system's pulling, you know, source evidence from the high quality sources as well as any other, any other challenges you might have faced, especially working with clinicians and the medical field. Yeah. I think actually in a lot of ways, some of the hardest challenges are kind of product challenges. So one of the one of the central tenants that would be have an open evidence is that AI is about empowering humans. You know, I don't think I don't, I don't, I don't really see a future where positions are replaced by computers and frankly, you know, where I think a lot of positions and jobs in the world are replaced by humans. I think AI makes us substantially more efficient and more effective at our jobs. But you know, I've had, for example, medical experiences in the past and it's just, it's so clear to me that there's so much more to the active, let's say, healthcare that is inherently really human and is really about interpersonal connections and engagements at a very human level. So I think actually one of the biggest challenges in building open evidence and I think something that contribute, the success of which contributes to its adoption is how can we build tools that are extremely helpful and make positions more efficient without, without, you know, falling into the trap of trying to automate positions or trying to replace positions and walking that line between being able to be useful but being able to be a tool in a position's pocket without kind of being a replacement. I think that's actually one of, in a lot of ways, kind of the core, the core challenge that we addressed in open evidence. And for us, the solution is about being able to make tools and systems that don't aren't overly prescriptive. So we don't try to tell you, you know, here's the, you know, small set of things you can do with open evidence. What we try to tell you is we want to allow you to do everything under the sun that you're going to need to do as a physician and we want to help support you largely with, you know, wherever we can with evidence. We want to be able to support all your needs and help address different tasks, you know, whether that's helping with diagnosis as an aid for a patient or even, you know, writing insurance, documentation or just looking for answers about different treatments and the comparative efficacy of different treatments like, right, there's a wide range of different things. But, you know, what really we really believe in is that this, we should be, we should be treating, we should be conducting health care in an evidence-based way and all of these aspects in different ways revolve around, you know, at a, at a core sense evidence. And so being able to be able to give that to users and provide a helpful service and a helpful tool without being overly prescriptive, I think is the balance that we've really been trying to achieve. Right. So you want to be the go to tool that provider's or physicians can use each step along the patient care journey, whether it's trying to diagnose the patient or figure out treatment options or follow-up care, open evidence should be the go to tool that can help them get the latest information and make the best decisions for their patients. I think that makes a lot of sense. Would you mind going to a little bit more detail about how the, who will actually work? How does open evidence process an input? If I say I'm seeing a 48-year-old male with hypertension on drugs x, y and z, what, but is hypertension is not under control what's the next step? How does open evidence try to understand the query? How does it process it and how does it develop the output that it will provide to me to best answer the question? Yeah. For sure. So there's a bunch of components, as you can probably imagine. I won't go super in depth into detail about too much. But really at a high level, the system uses a pretty general strategy that aims to compose a number of different components that are all kind of designed to meet some pretty specific, real position needs. So, you know, evidence and references are obviously a really core aspect of that. Even more than many other fields, the ability to have traceable tracks through to where sources are coming from, where answers are coming from, what's the information that that's based on is obviously super critical. And so that's one of the components. The system that I think is pretty central is the ability to, for any arbitrary query, kind of find some of the most relevant publications or papers or guidelines or government resources or whatever that is. That's a pretty challenging problem because there's a number of axes that are somewhat interdependent but also separate at the same time. So for example, what do you do when there's a New England Journal of Medicine publication from 2020, let's say 2017, that has some new phase three results of a drug. But now there's in 2024 something that's published in maybe a second-tier journal, but has some added information or additional context or something like that. How do you take those into account? How do you weigh the relative value of those different contributions? That sort of thing ends up being very central to everything kind of evidence related that we provide. And so that's kind of a core piece of the system. And then beyond that, basically every query is handled on a case-by-case basis. And we're able to basically analyze queries and route them to different subsystems that are pretty specialized and pretty different depending on the queries asked. So we can provide analysis that are based on guidelines. We can provide analysis that are based on, let's say, drug information in the FDA. We can provide analysis that's based on clinical trials. And talking through these, all of these have the core property that there's enormously valuable, essential information in code of them. But it's just impossible to manually go through and find them. Nobody is going through the FDA labels to very carefully find information about drug interactions. And same thing with guidelines. These incredibly long documents. And the thing you're looking for might be one sentence on page 413. So what we aim to do is basically be able to identify those needs and then meet those needs. Yeah. And I mean, I think you nailed it in terms of the pulling the guidelines. I think it's hard to know exactly where the information that's relevant to the particular query that a person's having is in those guidelines. And it takes forever to actually go through those things. And so I guess I also wanted to follow up in terms of like evidence, like levels of evidence in the way that at least how you see it at open evidence say a person puts in a query. You know, there, I think there are some queries that it's very easy to pull a guideline. You know, if you're asking like, what's the first line hypertensive or something like that? There's a very easy place to go or like you're asking questions about vaccination schedules. You can pull from the US preventative task force. I guess with questions that maybe you're a little bit more nuanced, where it's not a clear guideline. What is the, I guess, ranking of evidence in the sense of how does open evidence decide, like, okay, this is the source that that we can trust. And is there some kind of built in like, hey, there's only case reports or like systematic you know, analyses that support this data. Is there some kind of I guess information there that is provided to the user to know like what the quality of the guideline that's or the guidance that's given by open to evidence is. Yeah, it's a great question. So the truth is it really depends on the type of question, right? Because there are plenty of questions for which, like you say, there's really great evidence and clear things we can point to, guidelines we can point you to that will really just give you exactly what you're looking for. And then there are plenty of questions where like maybe all you have is a case study, right? And that's kind of the best you have to go on. What we try to do overall is provide a system that is as much as possible, mimics how the best physicians in the world triage data and triage evidence. And so often that means trying to identify given the full slew of types of things that we can take into account, you know, are there, you know, is there going to be that guideline that says exactly exactly the right answer that you're looking for? If so, we're going to want to we're going to want to highlight that, right? And then you kind of just go down the list. So, you know, for when they're relevant, phase three clinical trials, especially publishing great journals are fantastic sources of evidence, especially when we can be quantitative about efficacy, quantitative about safety. Those are some really valuable resources, but also they're pretty rare. There just aren't a lot of phase three trials about many of the drugs in practice that are being used or just, you know, questions that really aren't about drugs and are about much kind of broader topics in general. So it really depends a bunch of the query. What we do is we build systems that just take all of that into account holistically. That's kind of the best way to do it is if you focus too much on any one thing, right? You're going to spend forever trying to, you know, chase all sorts of different users. You'll satisfy the people who just want the answer and then you'll upset the people who want really, you know, deep primary source references. If you focus too much on primary source references, you'll have to the people who don't have time to read those references. And that's why they're on open evidence in the first place. So it's really as much as possible. We really try to kind of treat it like a human would, honestly, like an excellent doctor and just make those decisions on a case-by-case basis. Yeah, and that's, it's really exciting because I think, you know, everyone aims to practice evidence-based medicine, but in reality, I think it's very difficult to do so. And, you know, there's been Dr. Howard. He's an OB-GYN that has a bunch of published, he published a lot about his thoughts in medicine and he has an article about, you know, the different types of physicians there are based on like, how they interpret literature and what kind of sources they pull from. And he basically says, you know, there's the spectrum from zero to three, but like type one is, you know, the folks that do a lot of deep research and like, we'll really look at the literature. Type two is where people will look at guidelines, maybe not dive way too much in the primary literature, but kind of take what the general findings are from guidelines and practice medicine that's based on those things, which is exciting. Maybe that's the evidence-based side that we want at least. And then type three is the folks that kind of practice based on precedent, like people that, you know, have been doing something for a long time. I recently had a patient where their orthopedist had prescribed them antibiotics following some knee surgery that they had and said that, okay, anytime you get a dental, even a dentist clean it, you have to take prophylactic antibiotics. And I remember going into open evidence and be like, is that something that people still do? It was like since 2015, like people have not done that, but you know, people have people kind of have their precedent and will kind of stick with it. But what's really exciting is I think you're basically bringing down the threshold to practice evidence-based medicine because I think there's so much effort that is required for folks that especially are seeing a lot of patients. And so I'm almost thinking there's like a 1.5, you know, type 1.5 physician who's now can use open evidence and, you know, have questions that pop into their head and maybe normally they wouldn't look that up or not have a med student to ask to look it up, that they now can put it in a place to like plug it in and actually get up to date evidence with the sources to actually go through. So it like personally, I'm just really excited to see this really promote evidence-based medicine amongst clinicians. And I'm curious to hear from you too, like have you seen that happen, you know, working with physicians? Have you seen people that maybe were a little bit hesitant or maybe were maybe not as inclined to follow primary literature that are now using this tool a bunch and asking those questions that maybe they didn't do before? Yeah, it's really good question. The place that we see this the most actually is, or what I found I guess is that it's not even so much about individuals and clinicians and often about kind of like the settings folks are in, right? So if you're in a community practice, if you're in a community oncologist in Burl, Georgia, or you know, some rural part of the United States, it doesn't matter if you'd love to dig through the literature and you know, really thoroughly like find, you state, read every single paper that comes out and try to stay on the top of your game as much as you can, because you're just, you know, you're spending all your time trying to actually treat patients and that is what those folks should be doing, right? So it's almost not even about like, if from what I see about like, you know, people's, you know, are they more or less a academic or whatever, and it's just like kind of the reality of jobs is that it's kind of unrealistic in a lot of cases to expect that you really can spend the time that almost any patient deserves to really go and up to about a lot of these things. And so one of the things that we're the most excited about that we see and that we hear from people who just reach out to us is people telling us, you know, from these rural, rural areas and telling us, you know, this has been an absolute game changer, right? One person reached out and said, I'm in community practice. Open awareness has been an incredible lifeline for daily practitioners. This is from Cancer Center in County in Georgia, which is 75% African American and has a median household income of $43,000 a year. And there's tons of these places across the United States where, you know, I think this is one of the biggest impacts we can have in a lot of ways. I think we also have different and, you know, additive uses for many types of positions, but especially for these folks that are really just trying to give the best care that they can and had no ability to do this kind of, this kind of research before can now kind of get a lot of, if not the entirety of the benefits of spending, you know, all of that time and energy much faster and much more efficiently. Okay, you bring up a really good point in that any provider can benefit from using open evidence, whether it's a veteran attending physician in academic center who has access to all the publications and has, you know, residents and medical students working with them, or someone who's in a rural part of the country in a small clinic that's dealing with bread and butter patients, not the, you know, complex, unique one-off situation that an academic physician might see. Everyone can benefit from having access to the latest academic or latest scientific literature that helps them make the best decisions in terms of treating their patients. But what I wonder is when you first started building open evidence, when you first sort of camp with an idea, who was the intended audience? Were you building it more for, you know, medical students and residents, people and training? Was it more for people who didn't have access to scientific literature like small-town clinic providers? Who do you see as the intended user and how has that changed over time? Did you start with a narrow scope and work your way out or was it always intended to be sort of a one-stop shop for all physicians and providers regardless of where they're located or who they're treating? Yeah. The answer is kind of like all, right? Like, you know, my north story that I keep coming back to is there exists an enormous amount of information that we want to have access to and it's hard to have access to the information. And it's not just one user that needs access to that, but I think their use cases look very different. We talked about the doctor and rural parts of the country that are swamped and really just looking for quick answers to questions. That's a use case that we see a ton of. But just as much, you know, we also talk to plenty of academic physicians, folks in really well-plunded research institutions that are using it, you know, much more for research and to write papers and define references. One of our, it was an early anecdote we heard from a neurologist at an academic institution who was, I think, you know, one of the leaders of his specific field within neurology. And he just, he was like, I just tried out open evidence and, you know, even though I'm like the world's expert in this thing, I actually just searched around and I was able to find like four references that I had never even heard of before, but were exactly what I needed for my review paper or stuff like that. So that's a need. We see medical students using it for test prep and for education. We see residents using it to help kind of earlier in their careers as trainees. There's really a wide range of stuff. And I think what, for me, what underpins it all is like, it's not really about addressing any one of those specific needs because they are kind of fundamentally all asking the same thing. They have, they have, they come from, because they have different specific needs, but at the end of the day, it's about finding access to the best information and having that information be grounded in the best possible references. Yeah. And I think there's, that's, that's really unifying message and looks different for different groups. But at the end of the day, I think is everyone would benefit from the same product at its core. You know, you said you did a talk at, I think it was at Berkeley that really, you showed a graph showing kind of where people are in terms of the, the users for open evidence. And there's a lot of people that were in rural areas that were utilizing it as we talked about a little bit earlier. You know, at least where I am, where I am right now, I'm in Asheville, North Carolina, which isn't really an academic center, but like most of my preceptors and a lot of my residents are using open evidence pretty regularly. And so, you know, I will go back to Chapel Hill next year and I could imagine it's going to be a lot greater. But zooming out a little bit more, I would love to hear from you about, I guess, if there are any numbers that you have in terms of what does the user base of open evidence look like right now, whether that's nationally, globally, just how much impact is, is open evidence having in clinical medicine right now? Yeah. So right now we're used, we've been used one way or another by about 40% of US positions. We're in every country, essentially. And it's, yeah, I mean, the growth and the usage of the uptake has been really incredible. And I think it really speaks to the fact that open evidence in a lot of ways hits on a real nerve and a real thing that people need as a cross, you know, as we talk about across all ranges of the different types of physicians and medical and healthcare practitioners that you can have. We've seen about like 300x growth in the last, I don't know, 10 or 12 months or something like that. And that continues to this day. So it's something that we're really thrilled about. We're honestly super thrilled that even as a small team, we can make something that is legitimately just very useful to a lot of different people. And we're really excited about continuing to make that as good as it can be and help me positions even more efficient and more effective. We've reached that open evidence has and the number of providers it has spread to is just really phenomenal. And it's great to see that one product can impact the lives of so many patients and arguably improve patient care so much. But it's interesting that you bring up that it's being used all around the world, but it's not just in the US. So I'm curious, how do you address the differences in practice? For example, the guidelines vary not only from country to country, but even if you look at North Carolina, but California, the recommended treatment guidelines can vary significantly. Additionally, if you go from country to country, there's medication availability. There's common terminology made different. Just practice the how medicine or practice can vary significantly if you go to different parts of the world. So how does open evidence take this into consideration? Is able to understand different languages? Is it able to sort of tailor the responses to where the provider is from? How do you consider all these factors or is it always a sort of standard response at this point? Yeah. So we end up being like pretty US specific. There's a handful of reasons for that, at least of which are kind of regulatory reasons. So that ends up being kind of our large focus. But the benefit of modern AI and the types of models that we work with is that they're pretty language agnostic. So either inside the United States, outside the United States, users can ask questions in any language. They'll get answers back in that language that they asked. And it's pretty, everything's pretty isomorphic. It's kind of at the end of the day, this fluency layer at the end that can be tweaked to be like just speak in Spanish or whatever it is. So that's a really cool piece of the technology. But other than that, the guidelines do change from region to region. But at the end of the day, humans are the same. Human biology is the same. And so I think this really goes back to our mission of supporting physicians. For a tool that is about helping and supporting and giving information, it's ultimately pretty similar, no matter where you are in the world. What you do with that information might end up being different. And I think healthcare is practiced pretty differently in a lot of places. But for the point that open evidence sits in terms of not automatically writing prescriptions for example or something like that, I just as a tool to help physicians, it's pretty global. Yeah. Yeah, and I think we keep touching on this. And I think this is what really makes open evidence really interesting is that it's literally like a tool that's as ubiquitous as something like Google specifically to medicine. Because you think about a lot of these tools that are being built with AI right now. It's like, okay, this can generate these insurance prior off forms or you can do notes really well or you can do clinical decision support really well. It's kind of one or the other, but it being able to do all these things in one, I think is really exciting. We touched on this a little bit with the research and with some of the rural locations where open evidence is used. But in your experience working with clinicians and seeing how open evidence is being used by these physicians, what are some common workflows that you see physicians or different people that are using open evidence commonly used? I can see it being from the generated question and then putting that in, but also is there a certain setting where it's used more in the hospital where it's like internal medicine or do you see it using the ED a bunch? Just what kind of flows does that happen in which settings the most? Yeah. So we've done a little bit of analysis of this. It's used pretty ubiquitously across different specialties. Let's say the least being radiology, maybe probably it's reasons. But we do see some trends in kind of overall how folks are using open evidence. So some of the biggest categories of types of questions are around treatment decisions and treatment questions essentially. That can include cases with instances where people will put in information about patients and cases and ask for treatment options. I think these are characterized by pretty challenging cases or pretty challenging instances where it's not obvious what to do. And that's one reason to consult a literature. But also kind of, for as much as there's challenging cases, there's also plenty of easy cases. In terms of I know that I've learned it. I've done it 20 times. I forget right off the top of my head, let me just ask real quick and get a fast and easy answer to it. We see both ends of that spectrum. Another large chunk of usage is around side effects and especially drug interactions and that sort of thing. And that's a place that I hope it is going to be really powerful. One of the sources I would drop from is the FDA, which is an enormously rich resource. That's just impossible to get into and find specific information from. And they have actually incredibly detailed reporting of side effects, for example. So those are two, I think of the biggest categories overall. But the thing that I've been kind of surprised by and has become a bit of a mantra for us in some sense is that everything is the long tail. That just especially with medicine and this is probably even true beyond medicine, that every specific, every person has their own questions. And almost no two questions are exactly the same. And it just truly is remarkable to see just the extreme long tail of instances that people will ask about and that sort of thing. And it is, the vast majority are kind of just really specific, unique, uncurrentizable questions. Yeah, that's a really good point in that the chances of two different people asking the exact same question are very slim. And that's the thing with natural language processing. People will work at how they're used to speaking or they'll work it in a way that pertains to their specific situation or their specific patient. There may be unnecessary contacts. There may be specifics to that scenario that don't apply or are not really relevant. But open evidence, your platform will have to sort of figure out what's relevant, what's important and what question should I be using to do the research and pull the information I need to actually respond in an effective manner and answer that provider's question. So that's a really interesting problem or challenge to have and overcome. But I'm sure the models are only getting better and better at doing that. I guess shifting gears a little bit, I'm curious about your go-to-market strategy. Once you started, you know, commercializing and moving past just initial pilots or testing, how would you think about commercialization? How would you approach the B2B versus B2C? Because I think I heard you speak at a UCSF, or I saw a video of you speaking at a UCSF lecture or a presentation. And you mentioned that there's really pros and cons to both. In B2B, you have system integration. You can be integrated into the EHR. You are very easy to access. Your platforms are very easy to access. But it takes more time as hard to get by-infraintire system, health system compared to an individual provider or user. And the benefit of B2C is that you can get very quick user onboarding. You can get quicker feedback cycles and you can get quick feedback. And you can upgrade very quickly based on the feedback you receive. So how do you think about that? What made you choose a route of trying to not think about EHR integration and going that B2C route at least initially? Yeah, I think fundamentally one of the things that we're the most excited about is really just making something useful. And so early on, it was very natural to build something for positions and therefore to talk to positions. And as we started to see more and more usage, that kind of became a meaningful strategy for us in terms of wanting our users to be very central, thinking, you know, waking up every day and thinking, how can I best make something that's really great that will improve the experience of positions? And so because our kind of process for burning this into the world, so to speak, was very much focused on users. It was very natural to just keep doing that and not want to move that focus away and start dealing with administration and hospitals and go that sort of route, which always has one step of kind of barrier, infriccion, and separation from people on the ground to actually just need something to use. And so that was, and truly, you know, the reason why we kind of very much went this route to really just make something that can be used. And, you know, like you say, there's other aspects of this that play in and stuff like that. But it's, yeah, it's, I think it's just really special to be able to have a really close connection with our users. And that's why we kind of always just want to maintain. Yeah, I think what's really cool with that is when we think about clinical decision support, it's actually incredible how many tools there are right now that are doing, being able to process queries and, you know, they all work, I think a little bit differently in the back end, but the same essential, like we're, you know, in putting in a query and we need some help with some clinical decision making, like tools like Glass, tools like AVO and other folks as well. And so it's interesting, you know, how different people are going to be to be route versus be to see. And I think looking at open evidence, whenever anyone thinks about a tool that's using AI, that to put in queries, open evidence is at the top. Like everyone, everyone knows open evidence that has the, has the, the largest usage. And when you have that kind of grassroots movement of like the people that are actually using the product, love it so much, there is that additional pressure on the, you know, institutions that they're working for to incorporate it in some way over the other tools. So I think that's a, that's a really great way to approach it. And, you know, as, you know, maybe you're thinking about EHR integration or like, you know, into the future, it could be, it could be one of those things that you have so many people that are championing the product and really believe in it. But also like to talk to you about, you know, you're generating, you know, so many queries, I can't imagine maybe you have a number, like a daily query, a number that open evidence generates, but it's probably incredibly high and, and requires a enormous amount of service, service side support and, and, and cost there too. Just curious in terms of the revenue model, I know you mentioned ads in the past for people that are using it on the B2C side. Have you thought more about how we're, you know, you're thinking about charging for it or like making it sustainable, I guess, for the long run? Yeah, I think there's been probably at least a few thousand since you started speaking just just, that sentence. That's incredible. Yeah. So we're, we're in, that's pretty business. And that'll, that'll, that'll be true for the rest of the company's existence. You know, as, as I said before, we're, we're really just thrilled to make something available and, and making it free is kind of a big part of that for us, that we get to just put this into the world. And, you know, when, when, when you go with the B2B route and when you, when you have the subscriptions and all this stuff, then all of a sudden it becomes a tool that is for, you know, let's say all the schools around the Boston area, for example, or the MGH in the world, and that sort of thing, which are great institutions and there's a ton of really great care that happens there, but there's also a ton of really important care that happens for the rest of the country. And allowing open evidence to be free also, you know, allows anybody regardless of resource, you know, amount of resources to be able to get value from what we're building. And so our adds some more to our added-spotted model allows that and enables that. Yeah, is a piece of what we're really excited about. I really like that perspective in that this tool should be available to everyone. I mean, different healthcare resources have taken different approaches. If you look at up to date, for example, I think health systems have to pay a subscription or a fee for their providers to be able to use it. So it's not easily accessible to, say, independent providers or people who are in smaller towns maybe who are not affiliated with the big system. I think it's really important for all providers, regardless of where they're located, who they're treating to be able to have access to latest information and be able to stay up to date with medical literature. So I really like your business model of being added to poor, so that the users aren't the ones that are paying for its usage, but it's more of the people who are paying for the adds that are supporting providers in their ability to care for their patients. That's a really great way of making it truly accessible to all. But shifting gears a little bit and thinking about the bigger picture, stepping outside of just open evidence. I'm sure you're pretty well versed in what's going on in the health tech industry in other ways that AI is being used for scribing, for note taking, for clinical decision making. There's lots of really interesting things for surgical applications. Lots of really interesting applications that are being developed or even in use by providers. What is something that you're really excited about or something that you really hope to see in the future as AI and technology continue to play a larger and larger role in healthcare and medicine? Yeah, it's a really question. I feel like healthcare and health tech specifically kind of historically has been in a challenging place because on the one hand, it's one of the most impactful places that you can do work in as an engineer or as someone building because rather than making someone point zero, as there are one percent more money, you're actually ultimately improving the care that patients give. It's compelling for that reason, but also there's traditionally an enormous amount of red tape and challenges with building technology into systems in a way that can actually find users and can find real impact at a large scale across the healthcare system. That's traditionally been challenging. There's a moment where there's an opportunity to rethink many of those technology integrations and rethink the ways that we should be interacting very broadly with technology with the HR systems, with hospitals, with the whole medical world. I think that these tools, as is immediately apparent to everybody, are too valuable and too useful to be ignored or to be left behind because either for regulatory reasons or for fear honestly of change. There's too much obvious good that these tools can do. That's one thing that I'm really excited about. As these tools continue to evolve like you were saying earlier, every day or every week, there's some new development that's coming out with AI models or tools or whatever it is. We're still in the early innings of what this looks like overall. I can't imagine that the outcome is anything less than true tool-based modernization, ultimately, of what can sometimes and historically has been a bit of lagging industry. I think you nailed that. Calling healthcare a lagging industry, especially in terms of digital health, I think it's so true. Dealing with EHRs on a daily basis just reminds me of that every day. I think this is going to really accelerate a lot of it. I think that's what personally, too, really excites me about this time and why I'm thinking about taking some time off in medical school because it just feels like a golden opportunity to get involved with the technology being built and be, I don't know, have some role in changing how things are done. Also would love to hear about what the future of open evidence looks like right now. What are some things you're looking forward to in the next few months and the next year? One thing that I thought was interesting was this whole space of being able to be a great tool for doctors to put inquiries into. Also, maybe think about the side of when we have patients that come in that have done their doctor googling before they came in. Is there any thought of potentially making a patient facing side that isn't some sort of evidence-based way to provide information that's helpful but not overkill, that's not too specific while also maybe knowing how to direct them into medical care when they need it? I guess that was just something I was thinking about maybe if the technology behind open evidence could play into that but also would just have to hear what you're thinking of in terms of the next few months and next year with open evidence. Yeah, the patient's thing is something that we're looking at and then we've looked at in the past. It's honestly pretty tricky because one thing that's extremely clear is position should be the center of care. If we believe that humans are an integral part of this process, there's no world for example that I think leads to good outcomes for humanity, frankly, where patients are primarily going to anything digital, let alone AI to have their prescription's filled or their health resolved or stuff like that. I don't think that's it. I don't have some future that I'm especially excited about. Part of that with open evidence is we know that already, like you say, it's a really substantial challenge for doctors and frankly just annoying when patients come in with more materials and you have to spend half or even three quarters of a patient visit talking them down because they're missing context about something. Now, time that could be spent to actually helping is really spent just fixing the misinformation or problems that they're getting from finding that information on people. On the one hand, open evidence should be able to help that in general because we can give patients access to better information. That should be a strictly positive thing. Even so, ultimately, myself as a non-medical person, patients in general are just not going to have the same kind of context that physicians are going to have. On the one hand, open evidence should be able to give patients better access to the access to information. I think right now we're so focused on making something that is really useful for physicians and really hyper-focused on that use case because I think that does end up being pretty different than the patient type thing. There's still a lot of widget chop there and that's really where we're focused and I think probably for the foreseeable future where we're going to be focused. It's definitely something that we're looking at in the K-Bow. You also ask more broadly. I think there are probably two main thrusts. One is always probably always going to be answer quality and reference quality. Really just making sure that we're always making it whatever is better, that we're always pushing on stuff. There's a bunch of stuff that we're working on right now in this vein and I'm really excited about all of the advancements that we're going to hopefully be talking about the next few months as we roll some of this stuff out. Really giving access to the best information I think always is going to be first and foremost what we're focused on. The second thrust is kind of just more broadly pushing the boundaries of what the mixing of AI and healthcare means because there's a lot more to that even if we're talking about aspects that are grounded in evidence or references, then kind of a place you can go to answer questions. That's awesome and I'm glad we're doing that. I think we've really just only begun to scratch the surface of when you think about what these models can do in a very practical sense and the very specific needs that healthcare has, what that interaction looks like in its fullest form. That's something that very broadly speaking, I'm excited about in terms of where all this is going. Yeah, I think one of the most exciting things about artificial intelligence in general or even more specifically in healthcare is that it's moving so fast. Like you said, there's developments being made every day, every week, every month. There are literally game changing. So it's really exciting to see what the future holds and I'm really here to see how open evidence evolves and as new features because like you said, the quality response they're getting better and better, pretty much every single day. So you never know how AI could be used by clinicians or how it's integrated into the workflow in just six months or a year from now. So it's really exciting stuff and that's sort of why we're so excited to have you onboard to hear your perspective because clearly you have a really solid understanding and of also really in the trenches building and seeing the new updates day in, day out. As we wrap up, I would love to ask you about any advice you may want to share. Our listeners, there's lots of people who are interested in developing in health tech and trying to solve challenging problems and just really make a positive impact in patient care. What advice would you have for someone who's maybe considering launching a company or developing their own solution to try to improve provider efficiency effectiveness or to improve the patient experience? What would you share in terms of advice or tips as they look to get started or grow their business adventures? Yeah, I guess one of the, one major lesson that I've learned with open evidence is that if you're passionate about something to just like put it out there, the way I think about it is there's a lot of people in the world. There's many, many more people in the world than you expect. And if you have an idea and you can build something that you think has value, just don't hold back from just doing it, from just putting it out into the world. There's a million reasons why things shouldn't exist or why you shouldn't do something. And what I think has been a very cool lesson about open evidence is we would start from just like a really core belief that the ability to find evidence and information should be easy. And I still believe that. I believe that and I believe it now. And on the basis of that, we kind of just put open evidence out into the world. And I think if there's stuff that speaking to anyone, anyone's excited about building, passionate about, believe should be true, just put it out into the world. And if you're right, people will just start using it because I don't know, we're always looking for better ways to do everything. I think it's incredible advice. I think there's a lot of friction. And oftentimes we're, you know, you ask ourselves too many questions before we build the logistics and how things go. And that can cause a lot of paralysis when it comes to wanting to create things, especially in healthcare where you feel a lot of that more. But that's incredible advice. And Zachary, we're really thankful that you took the time to be on the podcast. It's been an incredible, it's been an incredible conversation. And we really enjoyed learning a lot about you, all about open evidence, really the magnitude of the impact and the really exciting future of healthcare and evidence-based medicine that open evidence is really paving forward. So again, thank you so much for being on the podcast. It was truly an honor. Thank you so much. And keep your ears open. Hopefully we'll have some pretty exciting announcements coming soon. So thank you very much.