#36 General Purpose LLMs vs Specialised AI- Is Bigger always better? (OpenEvidence vs OpenAI)

this paper that we are talking about was published in nature medicine and it's called general purpose large language models outperform specialized clinical AI tools on medical benchmarks yeah now this is a cloud chasing paper this is yes let's dive into oh i was gonna say the same thing i was gonna say the exact same thing these two models are not even made for such a benchmark this is a multi-turn benchmark where you evaluate consumer facing composition exactly like not even close for what these models are made for so what i don't get is why did they pick these benchmarks like why did why is this paper about these benchmarks that obviously make no sense given the models you have chosen if you walk into a doctor's office now in 2026 June i see it splits in doctors for the more junior or younger or more tech for doctors i see them all on chart cheap deals some kind of LLM like doing this exact same information retrieval yeah and then you see the old scoop people still reading papers and stuff okay i hope both of them are reading papers but yeah yeah but i think it shows that this is very useful and even myself i'd use this kind of thing to even to find those papers right i use perplexity you know so it does a similar thing but i mean especially in this case you would you were saying before you would use up to date now you can just you don't have to scroll through up to date and find the right thing you can just ask the model and it gives you an answer based on up to date data yeah so when you know for those physicians who are listening or clinicians it's up to you to then verify the sources right so like whilst the LLM the AI output may say something click on that sticky link hyperlink they put in and just double check you know takes you know maybe a couple of minutes but it's worth doing because you know you you cannot fully trust that yeah outputs like 99% of the time you can yeah it's worth checking i think for this part now we're going to talk about open evidence there was a lot of discussion so for those who don't know you know open evidence is a healthcare AI company started shortly after chat Jupiter came out but the people they had the insight to to say let's make a search AI search platform with LLMs that doctors can use to basically look up queries and conditions you know basically replacing traditional google-esque search or paper reading with LLMs you know they had the foresight to do that but but this week there was a paper that came out that basically was arguing that generalist models like big frontier AI models could outperform these smaller bespoke one even on clinical tasks and you know probably you can extrapolate that to any industry so we're going to talk about that yeah yeah I think big question in the end is do specialized models even make sense when do they make sense is it true even that specialized models on that specialized task are worse than a generalist model yeah because this was always a big discussion and you know working in healthcare like traditionally traditionally the discussions were always that's yeah like smaller bespoke models are performed even bigger ones I think that was the conventional wisdom on very specific domain yeah expert tasks yeah I agree with that it was usually when you want or if you have your specific data set for your specific task and you find you in that model for that exact task then that model will outperform and this is to an extent true even today for many smaller tasks if the task is small enough and well defined you usually can make a model that is better than the general judgey pity or whatever else like you can even do something like this like let's do something stupid detecting animals on images the talking about animals on images or even like doing whatever x-rays and then um classifying x-rays or stuff like that a model that is specifically trained so a simple convolutional network specifically trained for detection of cats in images will work better and this is 100% works better than judgey pity on detecting cats on images like this is still true true I mean it's true with the caveat that's like that surely there's a data and size threshold right like no I'm saying currently this true but things like germany and judgey pity l&m star frontier are greats at detecting cats like I've never seen them wrong not as great as the best convolutional model that detects I'm doing cats as an example yeah but to do whatever a thousand classes of animals a model specifically trained for that will outperform judgey pity for the same purpose but surely that's with the caveat that's there's with the caveat that there's not enough cat pictures for chat you be the train on because I feel like for big l&ms not they probably is enough data out there for it to perform almost perfectly on a task like detecting cats obviously there is not today I mean it doesn't do perfectly wait was this tested yeah really yeah not for cats for image classification or whatever like if you look at any benchmarks in most benchmarks you don't have l&ms being benchmarks for image classification or even OCR like a specialized smaller model for whatever handwritten detection for a very specific language or stuff stuff like that will work better than judgey pity so converting handwritten text to computerized text yes but for germany was almost perfect on that task yes but there are models that are better okay fine so okay so so we're still so the convention still remains true generally but do you think for coding like a specialized coding AI would outperform something like load code see but I do think there is a boundary there it depends how general the task is like coding is an extremely general task it is very different from detecting cats detecting cats is a very small bound task yeah super narrow exactly super narrow and you can train a model the model doesn't have to be very intelligent it can be in fact fairly stupid it can be a very simple model that doesn't even understand what is it detecting like a convolution network has no understanding of anything but is capable of matching shapes and by that detecting cats true yeah then apply the same thing for healthcare then right like it depends how narrow the healthcare task is because actually if you think about it a lot of healthcare tasks are pretty general like you still need to take patient context into accounts you know maybe you need to take into context why it's being performed what you're looking for yeah but but you're right there are narrow tasks as well like you know detecting a tumor on a CT scan no by I just want to say one thing because I don't want to I don't want people to misunderstand detecting cats can be a example where a lambside extremely good but where small models have an advantage is really where the data set is extremely curated for that small model and maybe that data set itself does not exist or is not very well represented in that giant data set the LLM is trained on that's why I'm saying like don't do cats do another weird animal for which you have very little data and if a human went and hand labeled 10,000 images and trained a small model on these exact 10,000 images the model will of course be better on that task than the giant LLM because the giant LLM has never seen those 10,000 images or that weird animal yes yeah and yeah that's what I was trying to loot to before us it depends how much of that data exists online or in public or whatever so it's probably safe to assume you know professions where that data is locked away or is private will still have that. advantage. I think so. So, you know, like in healthcare, for example. Yeah. Can you think of any of this maybe in legal? Where data is very locked away from the public? Probably healthcare is one of the most. Healthcare is for sure one of the most protective. I mean, healthcare has a bit of dichotomy there maybe because on one side there is a lot of data that's locked away. On the other side you have the most amount data ever in papers, in journals, in articles, in whatever else, in books. So, I think healthcare has the two sides. There is a part of the data that is available to everyone and then there is a part of the data that is not available to anyone. Yeah. And that's exactly what we're saying before. Like, mythos or most frontier dilemmas will crush most medical MCQ benchmarks. You can ask any multiple choice question or even what you deem is a complex case series. Yeah. And there will still come out with very reasonable, usually correct answers and I'll perform probably most clinicians out there. Now, I just want to, this paper that we are talking about was published in Nature Medicine. And it's called General Purpose Large Language Models. Outperform specialized clinical AI tools on medical benchmarks. Yeah. Now, this is a cloud chasing paper. This is, yes, let's dive into it. Oh, it's going to say the same thing. It is insane, but you go, please, I'll go second. No, yeah. This title, General Purpose Large Language Models Outperform specialized clinical AI tools on medical benchmarks is an absolute joke of a title to me honest. Like, you know, you immediately know these researchers are after after clicks, rather than. Exactly. Well, sorry, I don't want to, yeah, I don't want to base arguments on the title or the people's intentions. So, okay, let me rephrase that. The title is not typical of academic titles. It's very clickbaity. Yes. So what they did was, and let me give the backstory of open evidence very briefly. So just to be sure, it is not only open evidence, like they benchmark open evidence in one other company and one other model, just so we don't go against open evidence completely. Let me go back, go back to the days where I was running around in the hospital as a doctor. I see all these different patients, some with, you know, different diseases or different presenting symptoms. You know, as a lowly doctor, I was thinking, oh, you know, I wish there was something I can read to give me more grounded thinking and evidence on what I should do for this patient. And, you know, this goes beyond medical textbooks. This will give you the primary knowledge of medicine and physiology and pathophysiology. But when someone comes with a specific problem, let's say, sure enough of breath or migraine or stroke, not only do you need to know what's happening in the patient, but you need to know what the guidelines are, particularly the guidelines to your country or your region, based on what resources you have and what the latest studies are. So, up to dates is this company, which was also evaluated in this paper. Up to date is basically, you can think of it as a Wikipedia of healthcare. And just like in Wikipedia, it's maintained very well. And there's a lot of updated resources. So, I think the company is called Walter's Clure. And, you know, I use up to dates a lot. So, when the patient would come in, two things I'd do is go on Google, look up different papers, relevance to the condition, read those papers. You used that look for the, you know, because I'm working in the UK, I'd look for things like BMJ or practical neurology. Yeah. They usually have some good insights. And then I'd also read up to dates, which is a more generalist, literally up to date knowledge of that condition. So, yeah, when Chatshipate came out, it looks like both of these companies very quickly hopped on to the AI wave. So, open evidence, a lot of discussion was around replacing Google Search at the time of AI generated chatbots. So, that's what open evidence reported to be. At the time, there was a lot of concern with, you know, hallucinations and things. So, I think their whole thing was, it's grounded in evidence. So, it's AI grounded in evidence, open evidence. And likewise, I don't know when up to dates took their switch to AI, but they must have integrated some kind of AI based on their data. So, here they call it up to date expert AI. So, they also have an AI chat bot presumably pulling from their sources and their data. So, this paper compared these two pretty big companies, you know, open evidence is valued at around 12 billion, I checked this morning. Big, insane. Much bigger than I ever would have imagined. They've formed all these partnerships. So, you know, for those of you who think back to chatbots early on, there were lots of problems with the trustworthiness of their outputs, hallucinations, making up fictitious citations. So, there was a whole like wave of companies doing what we call retrieval augmented generation, taking grounded resources and then getting the amount of outputs solely based on that. So, in open evidence here, I can read that it's now partnered with New England Journal of Medicine, JAMA, other hospitals, guidelines, FDA, even has a clinical trials bits. So, you know, they clearly have gone deep into this. So, that's the backstory. Yeah, okay. So, in the end, both of these are really, there is an AI in front and in the back, there is a giant database of up-to-date guidelines, papers, books, whatever, and the AI is able to answer questions based on that database. Yeah, and it references those papers. Yeah. Okay. So, rag, in the end, if you want to or some agentic rag or whatever type of learning or whatever, yeah. So, let me show you a quick. Because I signed up this morning, because we don't release open evidence in the UK, but you know, it gave a typical scenario of 35-year-old bloated abdomen distension, nausea, and vomiting. I was, I was, I was put possible diagnosis, mobile obstruction, aliased gastroprices, like all sensible things. Like, one thing I wanted it to pull up here was pregnancy, where it didn't actually pull this up. Gives you some, pose some diagrams, even. So, you know, quite high-tech. It's put, pulling in diagrams from, from one of the textbooks, gastroenterology, next steps. It's got references all here. So, at the bottom, you can see all these references. You can actually just test it, click on, and I see if it actually goes to the right study. So, you can see the, that's the main functionality. But, worth a check. But yeah. So, that's what it is. But, I don't, like, was open evidence made to answer questions like the one you posed, or I think it is more made to answer questions like, what is the most up-to-date treatment for these disease? Or what is, because, or that was my understanding of open evidence. It is more like clinical decision support that is more about understanding up-to-date stuff, understanding what evidence there is in papers, in research, in whatever. So, it is not something that is particularly good at diagnosing a patient. Yes. But, it's more something that supports a clinician with evidence. Yeah. Yeah, I wouldn't say, I don't think they went down the regulation read, so I wouldn't use the word clinical decision support too strongly for them. But, they, they brand themselves as a medical knowledge platform. Okay. Okay. So, but, this one is important because it's for physicians, you're right. It's for physicians who, in my description, would have specific questions for a patient. So, yeah, you, you know, I would hope most doctors aren't going to their AI and saying, what's the diagnosis? Yes. From the symptoms. They're probably more be looking for, I think it's headache, or, or even they're presenting with severe, you know, vagranous headache. What, what things do I do now? So, it's more like, yeah, the course of action, the, yeah, or what are the symptoms? Yeah, or what are the most up-to-date treatments for us, specific disease, or given these disease, the next steps, or whatever. So, at least that was my understanding of, I, in fact, both of them open evidence end up to date was more around helping you with a specific query. that's related to guidelines, research, whatever, and not really something that's capable of diagnosing a patient or just a quick side point. Like, if you walk into a doctor's office now in 2026 June, I see a split in doctors for the more junior or younger or more tech for doctors. I see them all on Chartchip Theosome kind of LLM. Like, doing this exactly the same, like information retrieval. And then you see the old school people still reading papers and stuff. Okay, I hope both of them are reading papers. Yeah, but I think it shows that this is very useful, and even myself, I've used this kind of thing to even to find those papers, right? I use perplexity, you know, so it does a similar thing. I mean, especially in this case, you would, you were saying before you would use up-to-date, now you can just, you don't have to scroll through up-to-date and find the right thing. You can just ask the model and it gives you an answer based on up-to-date data. Yeah, so when you know, for those physicians who are listening or clinicians, it's up to you to then verify the sources, right? So, like, whilst the LLM, the AI output may say something, click on that sticky link, hyperlink they put in and just double-check, you know, it takes, you know, maybe a couple of minutes, but it's worth doing because, you know, you cannot fully trust that AI output. Like, 99% of the time you can, it's worth checking. The thing. Yeah. So, going back to this nature paper, general-purpose large language models outperform specialized clinical AI tools and medical benchmarks. They took this, these two, you know, up-to-dates and open evidence and they compared it with existing LLMs. So, they, here, they compared GBD 2.5.2, Germany 3.1 Pro and Claude Opus 4.6. So, all three of those are pretty kind of flagship thinking models. Yeah. And the three benchmarks they used was so Med QA. So, this is just your standard multiple choice questions. They used 500 of those from, you know, basically exam questions. Yeah. And then there may be some open questions there as well. They used 500 of health bench. So, for those who don't know, health bench, that's a benchmark from OpenAI's health team. We deep-dived into this benchmark in the never-opposite, called the benchmark, one of the benchmark episodes we did. Do you want to summarize health bench or the problems or shortcomings of it, operapists? No, you do that. Honestly, I have no idea. Health bench is, so basically they got a bunch of doctors and they got the doctors to speak to the AI and the doctors themselves wrote the rubric for what they think the responses should be. And the problem with that is a lot of doctors on developers and they're not code savvy or they don't know what how to design a good rubric. Yeah. Which is why this podcast was made, Devon Doctors basically, to join those two worlds. But a lot of them would have random stuff like, oh, you get 10 points if you have the right header or like, oh, if you get a million points if you quote this paper, presumably that's their paper, right? Or, you know, it'll be very subjective saying, okay, you need to say if you're not here, but you shouldn't say if you're not here. So, it's super subjective and it's not supposed to be clinical questions or like not possible to questions, but it's more, you know, they're role-playing as a consumer speaking to chat GPT. Yeah, and Devon was made by OpenAI. Yes. So, that's an important part to consider. You'll see once you see the results, why? Yeah. And finally, they have this real clinical queries benchmark, RCQ. So, what, so this one I prefer over those two benchmarks, RCQ is 100, the identified queries from physicians to a general-purpose large language model in a live clinical environment. So, my answer to standing was this hospital, or hospitalism, NYUL and GONE. They have their own AI in house for clinical queries. So, basically they took 100 questions from those queries and then basically they did a blind test between all of the different models. Yeah. And they rated it on a specific rubric. Now, just before we go into this, there is a couple of things I want to go into these datasets a bit more, but there is a couple of things I just want to preface these and why I hate this paper. First of all, the main thing is, I think, quite a bit of the data. So, like, the first thing is they show that, okay, their final conclusion based on the title is big models are the best. Small models cannot do anything. That's what you conclude from the title. They, I know in the discussion section and later in the paper they go into this a bit more, where they explain the limitations of their approaches or whatever, but it is a conclusion that is written in title that you cannot even remotely make given the paper. And that's why, like, on one side it is just fighting against small models or showing that small models are bad even though it's impossible and basically glorifying big companies and big models. So, there is that part of the title that I don't like that is straight out presented in this paper. So, I'm going to play the devil's advocate here. Yeah. Why can they not make this claim? So, so, you know, the spotter, the results were for those three benchmarks, half bench, met MCQ, an RCQ. Yeah. The general purpose, large non-group models, did perform better than up to date and open evidence. Yes. Now, first benchmark that they have is the MCQ. This is one of the most popular benchmarks. It is really question and questions and answers from the medical domain. It can be about diagnosis. It can be about what is the medication used for that. It can be whatever you want really. It is USMLE based the United States medical exam. That's an exam. Yes. This is one of the most benchmarks benchmark in the world. Yes. Like, almost every big lab has this benchmark as their target. And in their training data. Exactly. So, it is the most contaminated. It is the most benchmarks. It is the most everything benchmark. And you are now comparing the general models that really have that as the benchmark to an open evidence model for which this is not even in the scope. Like, open evidence does not even have in the scope this MCQ data set, nor should the open evidence model be good for MCQ. It's not even in the intended use. Exactly. It is specifically not made to do diagnosis or to do answers without citations, whatever else like literature. So, like, you are comparing to an a benchmark for which these models are not made. You're comparing a hammer to a spin. Exactly. Doesn't make any sense. It is like, whatever. That is the first one. And the same argument can be made for all the three benchmarks. The second one is health bench. First of all, that's made by OpenAI. You can imagine which model won there. Like, in the first benchmark, Gemini won. In the second benchmark, GPT 5.2 won. Yes. That was the benchmark itself was made by OpenAI. So, it is a direct target for them. Have you got the graphs here? It's 88% performance. GPT 5.2, Gemini 3.1, trailing second to 80%. You can see the open evidence on up to date, so around 60% for health bench. Again, these two models are not even made for such a benchmark. This is a multi-turn benchmark where you evaluate consumer fees and compensation. Exactly. Like, not even close for what these models are made for. So, what I don't get is, why did they pick these benchmarks? Like, why is this paper about these benchmarks that obviously make no sense given the models you have chosen? Perhaps this is a good time for me to give the, I don't know if you saw the rebuttal from OpenEvidence, but perhaps this may shed some beef. Press this may shed some lights. Let me show you this. I just want to say, I have the same argument for the turret benchmark. Before we go into the OpenEvidence rebuttal, let me just mention the turret dataset, the RCQ, because I think this one is the worst of the three. It's the worst. only real world one. First it is only a hundred questions. I mean it's not too much but the second one is this is not a like general data set of questions or physicians questions or clinicians questions or something it is questions physicians or clinicians already asked a general purpose large language model. Yes. So first you can imagine that these questions are a bit biased towards the functionalities of that existing language model and a bit more general purpose or more dumbed down if you want. Like one of the examples is please write a patient friendly list of red flag symptoms and return precautions for back pain. Yeah so this is an extended figure six. Yeah and that's not the time yeah you're right this is not a task that's for open evidence or up to date. Makes no sense really like this is a really a question for a general language model like a patient friendly list of red flag symptoms it is something that I really feel like again any chaggipotid germany whatever can answer and again it makes sense because these are the questions that the clinician already posed to that general language model in the hospital. Yeah of course the questions will be a bit guided towards these general models and not towards citations or whatever the open evidence provides. Yeah there's a priori bias. Yeah exactly and yeah I wasn't going to go too much into this but you know with all the work we do in clinically eye like the rubric is essential on how you frame these things so the rubric on the metrics is how you score the model you can see here in this diagram the way this score these are a CQ is with a light grid scale so between one to four hmm so one is so for example they do four domains here right clinical correctness how harm follow safety it is how complete it is and clarity for clinicians bracket structure plus readability so on these four axes there's from one to four so like if you imagine completeness one is major gaps two is some key gaps three is mostly complete four is complete and appropriately scoped likewise you know for safety one is unsafe two is potentially risky free is generally safe four is practically safe man this is not how you design rubric like that this is so just a quick lesson for those who listen to design clinical AI rubrics like you you really want the rubric to be self-contained verifiable and grounded like a light grid scale like this between one and four yeah it's highly subjective you know for example what's the difference I'm read you know if I read an output what's the difference between potentially risky and generally kind of safe like it's super subjective yeah like what you want is something more binary like it's you know or quantity like how much harm is there you know how many mistakes how many hallucinations are made yeah or you want to classify the harm you know it's just not all harm types of the same right like if you miss like let's say for this example of back pain if you miss a certain moderate flag is very different from missing like a major red flag it's very different than some additive something that doesn't make sense like there's no room here to make those quantifications it's like you know I'm sorry but this is just like a very rocky rubric you know it doesn't look like the researchers know what they're doing yeah or again I don't know how did this end up in nature medicine but what I'm interested in what is the beef between okay yeah so before we got cut off so open evidence posted a response over the weekend so there's just a couple of days ago and and actually they they go free you know they talk about medical AI blah blah okay they say behind the scenes this is a bit of juicy gossip here the study authors run a competing in house medical AI so which is there you know NYU Languon RCQ thing and even at some point asked open evidence for an API to power it like what so so they actually asked open evidence to be the provider of their in house generous alarm at some point and and then they said it but they wanted to include rights to build a competing products with open evidence is own API open evidence declined okay fun times weird so they wanted to use open evidence and use them to create some brand new products so there's a bit of competition there yeah and then second so they're free main points here one is first contaminations here what here's what happens when you plug common benchmark questions into other alarms they've seen the question and the answers and they show some they they show some screenshots of the alarm recognizing yeah the questions from health bench and from meta QA which we know pure it's very contaminated yeah contamination here means that the model was trained on the exact questions you are testing it on yeah so that's almost like cheating right yeah exactly second misrepresented metrics health bench created by open AI scores responses based on subjective slash arbitrary stylistic choices that's exactly what we said yeah here's an example of open evidence growing worse because it didn't use a specific email header yeah so like and then we we know this so we don't need to go more into it yeah and then third public peer review records indicate that's the two floor data sets above so this goes to your question right like why do they choose these data sets after peer review one of them one of the reviewers pointed out these data sets lacked epistemic grounding and so therefore that's why the rcq data set was added and you know they also say the rcq data set is not public so it's not there's not good science in any way exactly it's not only not public we know nothing about it you don't even know what types of questions are inside or anything more than this one question that is in whatever appendix inside on whatever on the 16th page yeah like very I don't know in any case but there is one thing here I want to say I'm defending it in this case because it was bad like I'm defending the small models in this case because this was written in a bad way and this is just the evidence doesn't support the title of this at all yeah my mind basically you can draw no conclusion yeah exactly I mean yeah because it's so like we're not making the claim why the generalist models can outperform small models and not wait we're just saying this this is a nothing burger exactly and I would in fact argue for any sufficiently general domain a more general model will work better like what I want to say here so assume that open evidence has their own small model that was trained on these papers on the other side you have Claude mitos yeah if you give the same database of papers so for react for basically agentic search to mitos and to this open evidence model mitos will work better like mitos is a significantly more intelligent model that can use tools better that can combine things better that can do a million things better than that small model that was trained on only these papers from open evidence like if you if if I'm making it clear I'm just what I want to say is that if the domain is general enough like coding like health care on the global scope general models will work better because intelligence is needed versus like to solve a general enough domain you need intelligence to solve to solve a small domain you don't need intelligence you need some pattern recognition but if your query as a physician is what do I do for this fund the club headache you need damage intelligence you need to write data source you need the right summary been pulled out yeah but my assumption there is that a large intelligent model is able to do rag better than a small model agentic rag or whatever else if that makes sense yeah I guess there's a difference in the amounts of papers they have access to you. So yeah, there's an access problem there, right? Yes. But you're saying, if given the same amount of access, yeah, there's no, yeah, most likely the generalist model, big model will do better. Exactly. If you have access to a similar data set, even to an extent similar, even if these questions were done the proper way, if these two models have access to similar data set, I'm sure mitos would be better than a small model. Yes. But yeah, but if the small model has some specialised human annotated manually created data set for a very specific purpose, then of course that model will be better than mitos. And these goes into the stuff we are making, which is the world models of health or models for patient trajectories. Yes. Because the model we are training, these sounds like self promotion, but it's not what I want to say. It is what I want to say is that this model we want to make, why we cannot use mitos for that, because mitos has never seen, for example, a hundred million patient trajectories, meaning diseases, symptoms, medications, like the trajectory of diseases of patients of everything else. Mitos has never seen that. It has read papers, but it has never or rarely seen a real patient. Yeah, it's rarely seen that messy electronic health data. Yeah, spelling mistakes from clinicians, different ways of, you know, synthesising data. It's not seen that, especially in the right temporal skills, which most of you are trying to have records are, you know, over decades of time. And it has never seen the real, let's say side effect. Like when mitos reads a pamphlet from a medication, it will read the side effects of that medication. And you can have a hundred of them, even for like random medications or everyday medications. It has never seen that in reality of when does it happen? How does it happen? Why does it happen? Statistically, how often is it? Like it has not seen the hundred million patients that really have that side effect from that medication. Yeah. And in that way, like that's the two sides of health data that we were mentioning. There is one that's public, but there is the other that almost no one has access to. And in my mind, this is where the specialized models will be better. Until the point that mitos has access to all patients, when mitos has access to a hundred million patients, and anthropic can train on that, then it's too late. Yeah. So I'm from pick if if you listening, that's probably something that we're doing, if you want big impacts to the world, potentially. But no, you're right, like just adds nuance to the discussion. It's not only date training data, and you know, all of this data sources that you can use for RIG, it's also the data distribution. So like, for example, we know online, the discussions around cancer and oncology, for example, has a much higher representation than in real life. And you know, we did some experiments of LMS, we never published this. Maybe we should, but that we when you get alarms to generate these different distributions, you can see that it's reflected in in the quote unquote randomness of what the model says. So there will be a higher proportion of, for example, oncology, oncology bias, let's say, yes. Okay, we could publish this small dashboard for this. What we were doing was we asked a model to generate a random patient. Yeah. And then ask it to generate a million random patients. And then we checked the distribution of things. So like, how many patients were generated with a certain disease? And then checked as that map, the real world and a million random people in the real world. Yeah, yeah, you can do this or anything, right, can be any open and biased question. And you know, people have done this with like gas and number, right, from 0 to 10. Yes, yes, yes. And the alarms always put seven, yeah, like 70 percent of the time or something, like it's not random. And like humans is not random. Like if you ask the human this, they tend to gravitate towards certain numbers as well. Yeah. But that is LMS gravitate towards the same numbers as humans. Yes. Because again, that was from the training date. But yeah, we can make that dashboard available. I think we have maybe a hundred thousand or maybe even more fake patients generated. They just have a bit of statistical information. But this is the only thing what I want to say in the end for this topic is this paper to me is wrong, just because the results they're shown do not justify or show anything. It is someone doing a couple of random tests. You cannot do conclusions from that. The second part is I do believe that general LLMS are better than specialized when these cases hold that we just went through. So the task is general enough, there is enough data in the internet, the task is wide enough if you want, like there is, there are tasks also that you cannot solve with a small model and has to be done with a big one. And this is like coding, you cannot solve just coding without the rest. Yeah. So big, big general LLMS will always be better at coding than a tiny model that is specifically made for coding. Yeah. Just because coding also includes images, it includes language, it includes design, it includes the up almost you can do everything you want. Coding is a whatever complete language you can build whatever you want. And to be able to do that, you need a giant intelligent model that is capable of doing that. A small model simply cannot do that well enough. Yeah. I feel like for this paper and for this task of physician, physician evidence based questions, the secret source is almost the database that you have access to. It's not really the model that's doing the heavy lifting. But I think just maybe just to finish like I think, like instead of criticizing the paper, maybe we should say how to make it better. I think one thing would be to give it like to test it on those specific questions that physicians are actually asking open evidence or up to date. Yeah. Taking that pool and then doing, you know, you can do similar head to head tests. Yeah. And you know, you can improve that rubric substantially. Yeah. But that could be a good start. Like test it on the intended use and also look for locality, right? Because the strength in these papers is it gives you local knowledge and specific regional knowledge, which an alarm would have no little or no idea about because you know, I don't think I mean, they might have started tuning for different regions, but you know, guidelines are usually exclusive to certain areas. And to me, that's the strength of these evidence based. Yeah. Like models. Yeah. No, I agree that data is a big part. And especially in these cases, the type of data you have access to determines, can you be better or is a small model more suited for these tasks than a general model? There is the part of just data is not enough. You need to be able to also put the right training. Let's call it infrastructure, but training process behind it, like anthropic and open AI and Gemini have all almost the same data. But anthropic is 10 times better than the two others, maybe not 10 times. Okay, then Gemini 10 times, but then open AI, they are similar, like one of them is always a bit better. Or why are these better than whatever the Quinn models or the deep seek models, there is a bit in the data or there is 80% of the data, but you need to be able to do their enforcement learning properly, to do the task design properly, to somehow design your learning curriculum, if you want, for those models, so that they can almost become intelligent by being trained on that task, like you have to prepare your task in a way that the models can learn from it. And not in a garbage way that is just like you can take your EHR data and train a language model purely on text. And you will not gain too much, you will gain a model that can speak that EHR text, but you will probably not be able to gain too much on the diagnostic side or on And were you will not gain much more than what the models already have. Like this is obvious in the, not obvious, on the AlphaGo paper, or AlphaGo models, or any one of these protein folding models. Sorry, what's obvious? Like how you train it? How you train it and the task design is very important. The input features have to be, you have to remove the noise, like we've wrote in the recent blog post about this. You have to be able to remove the noise, you need to leave the signal, and the model has to be able to learn from that. But you also don't want to remove all the noise. Like you want some redundancy. Yeah, there is a balance. Yeah, no, I was listening to a podcast from one of the deep-mind people about pre-training, and he was saying that, yeah, how you design the pre-training. Because now the bottlenecks are like cost and compute for these frontier companies anyway. So like he is saying a lot of his work goes into maximizing the compute and training. So like getting more bang out of the buck. And like you said, that is architecture or like how they set up their clusters or what proportion they put in which parts when they're training. So you're saying like they want to maximize every single second of computes every moment in time as much as possible. I think to summarize, you know, all of this. Okay, let me think how to summarize. We spoke a lot about some of the things, but I think in this specific open evidence part, you know, we're basically saying the paper that was published is hard to draw conclusions from because of the study design. But we suspect, you know, if you want generalist models to outperform smaller models, then there needs to be a degree of data availability and to compute for these frontier companies to train on that task. The task needs to be general enough for these LLMs to outperform narrow ones. Like if it's very narrow task and probably narrow LLMs still win. And finally, I just wanted to say like from a societal level, you know, we criticize or critique this paper, let's say. But you can see this divide in technologists versus scientists, versus like, you know, hype. And you know, I think there's this from my personal side, but I really disliked how so many people from social media linked in or whatever started jumping on this bandwagon immediately. You know, we've actually reading, analyzing the paper. Oh yeah. And you know, I probably criticize also nature here. Like the peer review process was. shouldn't have let this paper be published. Like if you still care about academic rigor. Yeah. But that's a conversation for another day. No, I agree. Conclusion on my side for the first part is really. the discussion here between open evidence and general models is really very similar to humans if you want. If you have an extremely intelligent human that knows a lot of things. And you bring him into a completely new domain and don't give him any data and tell him immediately to solve it, it's not possible. While a human that only knows that specific task, if you ask him for that specific task, he will be better on it. No matter that the other guy is 10 times more intelligent or knows more or whatever else. So that is the data access part. That's why I still think open evidence or anyone working with health data or anyone working with other specific data since they have constructed or specifically made will be better than whatever CHGPD can do on that specific specialized task if you want. And the same case is here. About this paper, I mean, whatever. Also, there is many papers like this. I don't like it. I don't like that it just gains traction and then people have to-- it gets annoying that small models get criticized for no reason or get even more-- get into an even more difficult position than these giant companies and big models because of whatever random scientist who has an agenda or doesn't have an agenda, I don't know. But that's just the idea behind. Perfect. I think that's it. It was nice to be back and have for the chat. Same. Let's try and do it more again. Thank you very much for listening. Like and subscribe. And the content you want to hear. No, like and subscribe. I don't want that. Okay, don't subscribe. Don't subscribe, don't like. If it's just comment or message me if it's any topics you want to do. I don't know. Say hi. Good. See you next time. Bye-bye.