Episode 7: Lively discussion about tests with Elena Plante: why we use them, what makes one valid, i

(piano music) - Welcome to Seaheer's Speak Podcast, episode seven. In this episode, I talk with Alina Plant, professor of speech language and hearing sciences at the University of Arizona, she's also the current American Speech Language Hearing Association, Vice President for Science and Research. We have a lively discussion about the use of tests. As educators and clinicians, we use tests all the time, but how often do we stop and consider why we use certain tests? How tests are created? What makes a test valid? Is it better to administer more than one test to determine a child's ability and how to test results play into our decisions for treatment? We talk about the importance of theory-driven assessment, key concepts for determining risk for dyslexia, to be in compliance with dyslexia laws that are sweeping the nation state by state and how current tests differ from past tests. Per usual, we end our conversation with Alina describing her current, most exciting project and favorite book. I want to thank you for listening. And don't forget to check out www.seaheerSpeakPodcast.com to sign up for email alerts for new episodes and content. Read a transcript of this podcast, access articles and resources that we discussed, and find more information about our guests. Also, don't forget to subscribe to the podcast in Apple Podcast or wherever you are listening. (upbeat music) Welcome to SeaheerSpeak Podcast. Alina Plant, I'll have you start by introducing yourself. - I'm Alina Plant and I am a certified and licensed speech language pathologist. I am a professor of speech language and hearing sciences at the University of Arizona. - And you have a new position with Asha recently, correct? - Yes, I'm currently, if you're a national member, I am your vice president for science and research. - Oh, fantastic. So you've spent a big portion of your career thinking about psychometrics and the appropriate use of tests and SLP practice. And you said you're currently teaching evaluation right now, so you got your head in it even more so right now in the semester. So what are the main purposes for using tests and how are those purposes used to inform test, or how are those purposes informing tests are chosen by SLPs? - You know, this is a really, really important concept that I think that many speech language pathologists aren't really getting in depth in their graduate programs. And that is that a test isn't valid or invalid. It's only valid relative to a purpose. And so in speech language pathology, we actually have a lot of different purposes that we give tests for. So we may give one to screen people who are at risk for having a disorder. We may give one to identify a disorder. We may give a test because we already know that the child or adult has a disorder like aphasia. We don't need to give a test and to determine whether someone has aphasia. We pretty much know when they walk in the door. But we might want to know how severe it is or what the profile is or the strengths and weaknesses. These are all completely different purposes and require completely different sources of evidence to prove validity and to provide that evidence base for using the test in that way. - So if I hear a clinician say, "Oh, my favorite test is XYZ." And I use it all the time for everything. What's the likelihood that test is appropriate for every purpose? - Slim and none. - Yeah. - And here's the reason. When you're asking different questions, sometimes the evidence for one to support one kind of diagnostic purpose, diagnostic purpose, is diametrically opposed to the evidence you need for another diagnostic purpose. So let's take the issue of ranking severity against identifying a disorder. So when we want to identify the disorder, it's essentially we're answering a yes, no question. Is this person impaired or is this person normal? So really, in an optimal world, you would have something like, if you do this, you're impaired, 100% of the time. If you do that, you're normal, 100% of the time. So the distribution is actually not a normal curve. It's two lines. (laughing) You can think of this in terms of a medical test that if you have this pathology in yourselves, you have this disease. And if you don't, you know. So that's really kind of a binning exercise. You're one thing or you're the other thing. Severity ranking, what you're trying to do is determine where a score falls in a very wide distribution of scores. So for one case, you need this wide distribution and for the other case, you need a distribution of one point, optimally. - Yes. - Now, of course, in our field, it doesn't quite work out quite that cleanly, but you can see from the example that really a test that is really good for ranking people within the normal range, when you develop that test, you're gonna develop it in a way that it really does that well. But conversely, it might make this same test not so good at answering the yes, no question. - Yes, that makes a lot of sense. And then I think there's also the purpose of wanting to better understand what treatment targets to use. So then that would be even a different purpose. - Oh, absolutely. And I did a study with a master student 20 years ago, has to be by now, where we looked at tests that had the same content. So we were looking at Morpho syntax. And so, you know, test A would test, you know, third person agreement and is verbing and plural S and test B would test the same things. But we looked at how often if you failed plural S on the first test, would you also fail it on the second test? And the answer was not that frequently. And the reason why is the context in which those tests embed those items will make it more or less difficult. Here's a great example. This was an actual test, it's not on the market anymore. So I won't name it, but one of the test items was me. That the pronoun me and me is acquired at about, oh, H2, okay. But the way they tested me was an item that said, the what you want a cookie, but the woman gave me a, and she's holding like a piece of pie. And then there's this thought bubble over her head that has cookies in it. And you're supposed to say, but the woman gave me a piece of pie. And I give this to grad students all the time. And you know, two thirds of the class failed the item. And they should have acquired me at two. So just by embedding it into a context, it makes it difficult. You can pass or fail that item on any given test. Now that's an extreme example, but the reality is there are lots of things that will affect performance that are much more subtle. So for example, do you have a four picture array or three picture array that turns out to make a difference? If you're using colors, the palette of the color and the relative intensity of the colors across the pictures can draw a response even when the test taker knows the correct answer. So there are lots and lots of subtle ways in which this plays out that we as clinicians can't know whether or not the item is actually acquired or whether it's just that it's in a context that is masking true performance in to some degree or another. That's why you don't take therapy targets straight off of tests. You really need to use the test. You can say, okay, we're looking at this. These are the kinds of items that are being passed and failed. Let me follow up on that. Maybe I should get a little language sample analysis and see if they're able to use them spontaneously. Maybe I should get in there with some clinical probes and see if I sets things up so that there's no obligatory context for the child to use the item is he able to, those sorts of things. But just going straight from a test right to therapy targets. Usually that's going to be really problematic. - Right, and I can even think in the case, the extreme case again, as the example he gave me that if that test was given, the clinician could think erroneously, oh, they don't understand that concept of me and then maybe even work on it. I mean, it could go even that extreme. It really could. - Right, right. But in the master's thesis we did with Andy Merrill, what we found was that even for tests that didn't have extreme examples like that, Again, they could pass it on one test and fill it on the other. Wow, that's. Yeah. And then, yeah, then you could see how that would be used erroneously in treatment. But I also think clinicians are told you need to use standardized, nor-in-reference assessment. And they're told, you know, these homemade assessments aren't good or even functional. But there is that balance between choosing targets that are functional and using appropriate standardized assessments. Well, again, this gets back to purpose. So you need to be really clear about why you're using that particular test. And it's probably not to select therapy targets. So under the IDEA, which all school-based clinicians are constrained by in the practice, the IDEA does not mandate giving any standardized tests. You can qualify a child under the IDEA without giving a single standardized test. So I think what clinicians think they're supposed to do actually always isn't the case. So it certainly isn't the case under federal law. And what the IDEA does say is that if you use an assessment method, it must be valid for the purpose for which it is used. And that's the kicker for the purpose for which it is used. So again, you might use a standardized test to make the identification decision, but use other methods to decide what treatment is needed. So one of my favorite things to do for school-aged kids is just go get their worksheets. Go get them. And then look at the content of the worksheets and just say, OK, to get a good grade on this worksheet, what skills do I need to have in language? And if you go over a whole bunch of worksheets, those skills that are showing up in the schoolwork that the kids can't do will come to the surface. You will see them over and over again that when a poor grade is happening, what the kid needs to do that is phonological decoding or metal linguistics or semantics, or relational semantics. You can even get that specific just by looking at the content. So clinicians can use what they already know about the nature of language and language disorders to look at other materials to determine where the need is. And that's completely appropriate under the IDEA. And in fact, the IDEA mandates that the children have to have an educationally handicapping condition. And by looking at their work, you're already linking it directly to their educational needs. And I like that to someone who thinks often, as you do about the connection between oral and written language, is if you want to make it something that's functional in the school setting, you can't deny that connection between oral and written language. It is critical for academic success. Absolutely. And we see this all the time that the things that we've cleaned up as clinicians. Therapy works. That's a good thing. We can clean up early morphemomissions and things like that and wacky morphic syntax. But it'll show back up in the writing. And then it'll show back up as a weak phonological system in decoding. So this is not rocket science. You just have to give up the idea that somehow language in its written form is somehow unique and special. Absolutely. And also embrace the knowledge we have about language and how it just directly applies to written language. I like to what you mentioned about cleaning up certain aspects and feeling like we've made improvement. That's true. But I love Hollis Scarborough has a paper which talks about illusionary recovery. And we know that language disorders is a lifetime disability that needs to be addressed and considered over time and the different manifestations. Yeah. One of the things that's come out of the Department of Ed, there's a really nice chart that shows that there is a systematic decrease in the use of speech language impaired as the handicapping condition label over time in schools and simultaneously an increase in the use of learning disabilities as the handicapping label. And that's basically a reflection of the crossover point is exactly where you'd expect it, where children start having to use literacy as an educational tool. And their language disorder is now impacting their academic progress very obviously. And you start seeing kids who had been in maybe speech therapy for unintelligible speech. We'll cleaned all that up. And all of a sudden, they're previously undetected or even unlooked for language deficits are now impacting reading and writing. And then they come back into the specialized system under that LD label. Yeah, that's really interesting. And I think it also relates to, as you said, the purpose of the assessment, but also having a theory and scientific basis behind your assessment protocol. That's something I've been really hitting at with reading is that you can't detect a certain aspect that's difficult in reading if you don't measure it or you don't look for it. You have to have a theory that says, these are the key components. I'm going to systematically assess those key components to uncover the deficits and then address it appropriately, whether it's through the functional and even tying in if it's appropriate, an assessment that's comprehensive. Yeah, let me just comment about the word theory there because I think the lay public really thinks about theory as a guess or a conjecture. In science, a theory is the very best thing you can have because it is an explanation that ties together a plethora of known facts and then allows you to predict future facts. So that's much, much different than a guess or a conjecture. And in fact, it is a tool that allows you to say, when you've got a child in front of you, what thing should I be looking for here? If I see this, should I also be looking for that? If I see deficits in oral language, should I be keeping my eye on literacy? Why, yes, yes, you should. So these are very important things. And the current crop of tests that are coming out these days are different than tests that used to come out when I was coming up through the professions as a young speech language pathologist. So back then, tests were largely built on models of normal language. And they didn't perform that well for identifying language disorders. And the reason why is that language disorder kids are not bad across the board at all aspects of language. They're, in fact, differentially bad at certain things. And so as time has gone on, you've seen tests that are more oriented to loading the tests with items that are more sensitive to disorders rather than items that capture the breadth of language in typical kids. And generally speaking, those kinds of tests are going to be better identifiers. Now, the side effect of that is that the models you learned when you came out of school may not reflect the best practice today and the best that we know about language today. So for one thing, one example, when I was raised as a speech language pathologist, we were really taught expressive versus receptive. And we were told that the vast majority of kids with language disorders have expressive deficits only. And then there's a subgroup that have receptive plus expressive. Well, that turns out to be not true. We've got about five or so really large scale studies that have looked at how language skills cluster. And you know what? They don't cluster that way. They don't. That receptive and expressive is not the divide. The divide is things that happened at the sound or the word level and things that happen at the sentence level. And if you test it, things that happen at the discourse level. And some studies tested discourse, and some didn't. But that's very, very different than what was in the field 30 years ago. And so if you're an old dog like me, you really have to stay up with what the current conceptualization is, what the current data says is going on in order to do your best in terms of what your theory is guiding your assessment. Right. So instead of-- I was raised in the same vein. So it's receptive, expressive. What type of a parent they have? And of course, the DSM, I think, still has the receptive, expressive tied into it. But ultimately, these new assessments, and the way I do my research studies is, you can collapse across receptive, expressive tasks if you're looking at a certain domain. Like you said, at the word level, the sound level, the sentence level, it's really just what is the appropriate assessment for that child's age and the task that you're completing. It doesn't have to do with the receptive, expressive. It's tapping the same construct that just in slightly different ways. And someone said something that is stuck with me. It's just a logical argument about this. But unless you have a mental model of what you're trying to produce expressively, you're not going to be able to produce it expressively. So the fact that you have an expressive deficit implies to a very large degree that you also have some problem at the mental model level, which would be receptive. Absolutely. And is that paper Larry Leonard has that ASLP paper? I believe the talks about the receptive expressive component tries to tie this together and say there really is not a distinction. The research is not playing out in that way. So that's a good resource. Yeah. It probably was him that said it. Yeah. Yeah. Maybe I'm not going to be in that paper because he goes systematically through the evidence showing that there should not be this distinction, that the research just does not support this distinction in these large studies. So it may be and that's a good resource. I was asked recently, speaking of thinking about diagnoses and purposes. I was asked recently by a colleague who's an editor of a reading journal and she's wanting, you know, she's getting more work on a DLD language impairment and she said to me, okay, so what is the standard criteria to diagnose childhood language disorders? And of course based on your, you know what I did, I said, we should talk. I'm not going to respond to that. And what I did is share the paper that you wrote with Tammy Spalding in the LSHSS 2006 paper, eligibility criteria for language impairment is the low end of normal, always appropriate because I told her there's really no way to obtain a standard criteria it has to do with the distribution and the test itself. So can you tell us about that paper and what is the main take home for clinicians? Yeah, if you saw, if you're a national member and you were reading the recent issues of the Ashe leader, you also saw quite a few letters to the editor hitting back on a paper that also said the same thing that there's a standard. Well, no, okay. So what we found was that when we looked across all child language tests that have been published by that time, I think there was like 42, 43 of them. But the average score difference between a normal group and an impaired group was 1.2 standard deviations, which means half of them were above that. I mean, that's what the average means. So half of them, for half of the tests, the scores lie between, you know, 1.2 standard deviations in the mean. So that's just fundamentally not true at the population level of tests. And one of the things that probably feeds into this gets back to something I brought up earlier and that is how sensitive the items are to the disorder. So are they testing language broadly? Are they testing language in a way that really hones in on those aspects that reflect the disorder? Or the test hones in on the aspects reflecting the disorder, the more sensitive and specific it'll get. So the more accurate it will be in saying you are impaired, you are not because normal kits will pass those items and impaired people will fail them to a much greater degree. So it's kind of, this is kind of one of those things that was, people thought back in the, you know, 70s and 80s that if your language impaired, you're just going to be at the low end of normal and that's how we're going to identify you on a test. But as I said, the data doesn't actually prove that. So you actually need a different piece of evidence and that is sensitivity and specificity. So sensitivity is the rate at which a test calls people with impairment impaired and specificity is the opposite. That's the rate at which people without a disorder are identified by the test as not having a disorder. So you need to know both of those things so that you don't under-identify disorder and you don't over-identify normal as disorders. Since I've been in the field, when I came in the field, none of the tests had this, none of them did. Now newer tests as they're coming out, I'm pretty routinely seeing this, not always, but we're often the not-I'm seeing it. So right now, if you take all the tests that might be on a shelf in a university clinic, probably about a third of them have it. And two-thirds of them don't. That's the problem. That two-thirds of the time, if you just randomly pick a test, it will not have the evidence you need to determine whether you can correctly diagnose somebody. Now, getting back to the, is there a standard? No, there is not. There is a unique score for every single test that will maximally differentiate normal from disorder, and it's not the same score across test. It can be anywhere. It could be negative one standard deviation for one test and negative three standard deviations for another test. So I'll give you an example. So one of our master's students is at in a school placement, and she said to me, she was asking another question, but she said to me in the course of this conversation that her supervisor at the school said, "How to test in her hand?" And said, "This test always over-identifies." And it turns out that their school system was using a criteria, a standard criteria across all tests of negative 1.5 standard deviations, and everybody was impaired. And my response to that was, that's because the cut score is somewhere else for that test. And so if you're routinely over-identifying, it means that the cut score of 1.5 standard deviations is probably too high for that test. So the test manual should provide you this information. If the test manual doesn't provide it and you've just ordered the test, send it back. If you've had the test for a while, you could use it to prop up tables that rock or something like that. But otherwise, it's, I mean, you don't have any evidence to justify using it for identification. Maybe it's good for another purpose, but it's certainly not for identification. So that sort of gets to a related question, which I'm sure the listeners are already asking in their mind. But if my school district makes me use the cut score. Right. Well, first of all, they're probably not making you. Where that cut score originally came from was probably you or your colleagues 30 years ago, with 30 years less information about diagnostic practices. So it's up to you to borrow a phrase from the late Tom Hickson to keep your corner of the cave clean. Yes. Okay. This is your practice. You need to be abiding by the ethical principles of your profession. And it's up to you to keep things up to date. Now this may seem really daunting, but it's not. It truly is not very often in school districts, the school or the district's director of specialized education or the state director of special education are not speech language pathologists. So they are waiting for us to come and say to them, you know, this is really badly out of date. We've got to update these. So in our state, we did this. We went to the state director of special education and said, you know, your best practice guidelines are really out of date. And response was not, well, that's the way we do it. The response was, oh, really? Perhaps we should put together working meeting on this. And you know, it took a while for us to work through all the issues and get all everybody on board and headed in the same direction, but we got it done. So now our state guidelines specifically say that using an arbitrary cut off score across tests is not appropriate. So if the school district is still doing that, they can change it tomorrow and still be in line with the state best practices. There is no barrier to doing that. And I think too, what I do for my class when we talk about this article in particular is I have little pieces of paper and I give, I have 10 people stand up in front of the class and I give them scores. Each person has a score. And I say, okay, you're now in this district and their cut score is 1.5. So then I go through and I say, now we're going to cut here. So all of you below this, you're impaired and all of you aren't by this district standard. Okay, now you move to a different district and that score is 1.2. So then you give the assessment and I say, okay, now if you gave that same assessment, then you say, okay, now you aren't impaired anymore and you are because it also creates this kind of confusion if students are moving or parents are checking in with peers. that you can have one district that has a cup point. Another one has a different one. And the child themselves can go to one district and be impaired and the other one not. The child's not changing. It's the criteria that it doesn't make any sense. - Well, absolutely, absolutely. And there's kind of two ramifications for this. Let's see if I can remember both. (laughing) Okay. One is that the procedure for both of those school districts is not only not evidence-based. It's not like there's just no evidence per hour con. There's actually evidence against doing it. So if you went to do process or God forbid, landed in court, you would lose big time. Big time. And I find that school officials are really responsive to the legal ramifications of bad practice. So. - Absolutely. - The other ramification of this is that what we're doing is we're basically setting cut points that assure that the most severe kids are the ones who get treatment. Well, the other half of that is that the kids who we couldn't move from underperforming at risk for not graduating low SES consequences for their life, those people who we really could shift into a middle-class life. They get nothing. - Right. - And the tax basis that those people could generate versus the taxes they're going to use because we couldn't move them into a functional range because they weren't eligible is enormous. So the societal consequences of these are quite strong. - Absolutely. And the shame associated with having difficulties and yet being told, well, you're not trying hard enough, you're lazy, those kinds of things because you haven't been identified as requiring services. Those ramifications for everything and all the decisions you make are strong. And you know, listeners might be interested to know that in longitudinal studies, in larger studies, even in smaller studies, we find that only about 30% of children with language impairment are even being identified in the current practice. So there's a real problem with identifying these children and getting them services. They're just not being seen. They're missed. And I see this all the time in the studies I do that I talk to parents and they'll say, I knew something was going on. Are there things they're told about their child when they bring forth, not if they have the capacity to even bring forth some concerns and try to fight for it? The things they're told are really demeaning. Your child's just not academically inclined. That was when I heard. And it's ridiculous. And it has to do with these practices. So it's frustrating. I tell my students in that example, that if they're in a district that has a certain cut point, then minimally, they have to find a test that has the same cut point with good sensitivity and specificity. Yeah. And that's a hard job to do. I mean, occasionally you can do it. But it's so hard to be much more functional and you'd have a lot more options in front of it. If you and your colleagues would just take care of the problem, just take care of it. I mean, I'd love to hear your story that that worked in Arizona. I think that's really empowering and fantastic that you've been able to move the dial there and get more evidence based when it comes to the diagnosis. We're really dealing with this in my work in dyslexia. So there's been this nationwide grassroots movement to say dyslexia in schools. So teachers are being educated on dyslexia. And schools are being mandated to identify dyslexia early. So we have a new law saying in Massachusetts, for instance, that in the fall, this coming fall, 2019, schools need to determine risk for dyslexia and all kindergartners. But what I find is that teachers, schools, administrators don't have the education and the appropriate use of assessment and just data interpretation. And so concepts such as false positives, false negatives, sensitivity specificity, as you mentioned, positive predictive value, which I know you said you discussed in your classes as well. And it's important to know how all these tie together. And that's something I've been trying to get the word out about is what does this mean and how does that inform your decisions? How can you tell us about the relationship between these variables? What do they mean and how would that inform a mandate such as you have to determine risk for dyslexia and kindergarten? Yeah, when you're talking about determining risk, you're really talking about a screening. So you're not talking about does this person have the impairment or not? It's really, you're trying to identify a cohort of people who deserve a closer look. So when you're dealing with screenings, this is one thing that students struggle with. And clinician struggle with it as well is that they're not really clear about what constitutes a screening. So a screening is not a test that is short. Short has nothing to do with it. If you think about mammography, that is a screening. It's not short. We're certainly not short enough for those of you who've had it. It not only takes your time, but then there's this whole back end where a radiologist is reading these things. So that's not short. Yet it's a screening. Why? Because it's very good at telling people, telling physicians who needs a closer look as a biopsy or an ultrasound or some other diagnostic procedure to determine what's going on. So if it's not short, what is it? It's how accurately does it determine who's likely to fail a diagnostic test? So when you're screening, the issues of sensitivity and specificity still come into play, because you want to identify everybody who's likely to fail that later diagnostic. Because those are the people who need help. And to get that, you're willing to tolerate some over-referral. So for a diagnostic test, you want the best balance between sensitivity and specificity, and you want both of them to be high. For screening, you can give a little bit on the specificity, but the sensitivity's got to be high, because you don't want to miss people and not have them get the services they need. There's a third metric, positive predictive validity, which is based on sensitivity and specificity. But it also calculates how frequent the disorder is in the population to help you sign a gauge, how good your screening is. So basically, the concept is the more frequent the disorder is, the easier it is to find it, the more rare the disorder is, the more you're searching for that needle in the haystack. That is going to affect your overall accuracy of correct referrals and over-referrals. But again, it all boils down to. It's still based. The foundation of this is still sensitivity and specificity. So if those things aren't high, if particularly sensitivity, you're not going to have a functioning screener. So the idea that you can just make up some items, and it'll serve as a screener, is frankly ludicrous. As a researcher, I can tell you when I'm trying to build a new experiment, I can be wrong seven, eight times before the thing works. So the idea that I could just come up with something, and it'll work the first time with the 3,000 kids that I'm going to screen, is just ridiculous. So you really do need a tool that is built as carefully as a full diagnostic test. The bad news is that right now, today, we don't have a lot of great options for screeners. But one of the things that we do know works pretty darn well is parent and teacher report. There is a screener. It's not one of mine, the student language scale, that has north of 80% sensitivity and specificity. It takes about three minutes for a teacher to fill it out, about that long for parent to fill it out. And it's really accurate compared to every other screener that's on the market right now. So in the next 10 years, we're going to see better and better screeners coming out. But you have to know how to determine which is a good screener and which is a poor screener. And so sensitivity, specificity, and positive predictive validity are the three main elements. If I don't see those, I don't even bother looking at reliability because if it's not accurate, why do I care if it's reliable? So many of us when we went through school, we're taught to look at a checklist of psychometric criteria. I don't do that. I go for the main evidence. And if that's not there, I return the test to the publisher if it's a new test. Because all of them will allow you to do that, all of them. Or as I said, I use it to fix rocking tables or you'd doorstop or something like that. Because there's no evidence to support my use of it, I'm not in compliance with the federal law at that point. That's right. And then we think about-- so the sensitivity being the testability to detect impairment, You have to. have an outside measure. So I think what's not always understood and where I appreciate even more and more as I've done it myself is that you have to have to get that sensitivity as specificity data. You have to have a sure diagnosis. And so that sure diagnosis has to be given separate of the screener and it has to be taking in lots of information and you feel very confident about that diagnosis. And then you give the screener and then you look at how well the screener is able to match up if the screener says risk and the child has the impairment, that's the sensitivity versus if the child, you know, the full battery says note the child's typically developing and the screener says the risk is low, then that's specificity. But like you said, whenever you have these decisions you're making about sensitivity and specificity, you're willing and screaming to have a few more false positives because that there's a kind of gain in yang and there's some great online tools. I'm sure you probably use them too where you can change your sensitivity and specificity and move it around and see how the curve moves. So if you change sensitivity, usually take a hit and specificity and so forth, you want to have a good sensitivity, that means your specificity is going to go down, you're going to have more false positives, that means you're going to have some children that you set are at risk but they really aren't. But you'd rather over-identify and then that's just a red flag and then you're going to further follow up with them. I often use the example of cholesterol. So if you go in and you have this cut point in its arbitrary as well, 200 for cholesterol and it's above, then it's not automatic that you're going to have a heart attack. It means that you are at risk and you need to further evaluate your food intake, your exercise, your genetics, huge factor, what is your family history. And then from there you do some response intervention typically. So if you think, "Oh, I've had a lot of ice cream before I did this test." I've been really digging that and I took the test. I think that's what it is. Then you need some time where you get your diet and check and then look at your cholesterol. If it goes down, well, then you have a little bit more of a causation going on. You tweak something, cholesterol change. But what if you do all, you're someone who exercises, you eat well and you have a family history of heart attack, your doctor's probably going to skip over some of these other tweaks and say, "We need to get you right on some medicine," or something. So I think I try to think of these kind of more medical examples because we all live them. We're going to get you in checkups, right? Or like the mammograms, a great example. We live these things every day and because of those experiences we go, "Oh, yeah, of course. If I have over 200 cholesterol, it doesn't mean I'm automatically going to have a heart attack. There's something I can do about it, usually and figure it out further." Right. That's a really good example of what risk is. And again, when you're screening, you're really talking about risk. You're not talking about hazard, has not the diagnosis. Absolutely. So one piece of advice thinking again on this line of dyslexia risk is that I hear often that it's better to determine risk if you use multiple measures. So there was a recent piece in the International Dislex Association by Richard Wagner and all quote what he says, "Individuals with dyslexia are commonly misdiagnosed or even misentirely. Part of the problem is unreliability and diagnostic that occurs for definitions that feature a single indicator. A promising solution to this problem is use of hybrid models that combine multiple indicators or criteria, improving reliability of diagnosis. So is it always the case that if you add more measures, you're going to increase your sensitivity and specificity?" No. Absolutely not. There are actually two issues at play here, but let's deal with the sensitivity and specificity issue first. So the problem with sensitivity and specificity metrics is that if you have one test and it is bank and good, it's 100% sensitive and 100% specific. I'd love to know about that test. I don't know about that test, but let's say that test exists. And you give that test and you get an answer. Is your answer going to be any better by adding a second test? Well, let's take a test is more realistic. It has 85% sensitivity and just so I can do math in my head, 85% specificity. So now if you say you have to fail test one that's 100% and 100% and test two that's 85% and 85% what happens is the error, the over and under identification rates add. So it's straight for those of you who had any programming back in the olden days I actually had a basic course, but the good thing is that it teaches logic. So in Boolean logic when you say this and this, you're combining those two things and the error combines. So your total error now is 15%. When it was 0% before. And so if I had two tests that were 85 and 85 and 85 the total possible error now is 30%. Now your sensitivity rate is is poor is 70%. That's below what I'm comfortable using clinically. That's just too much error. So if you are arbitrarily combining tests, this is a real problem that you're better off using just the best test and going with that, then trying to say, oh, I've got to fail this one and I've got to fail this one. And I got to say I still see this a lot in district eligibility criteria where they say you've got to fail two tests at some arbitrary cutoff point. So those school districts are even worse than the school districts that are just saying, well, you know, he has to fail this at this arbitrary cutoff point. There's another point here though that's a little bit more subtle about in the Wagner quote. And that is the idea of trying to stick to the phenotype. So it is the case that with, you know, dyslexia, again, they're not bad at all aspects of reading. They're bad. It's some aspects of reading. And they also have some other deficits that are known to exist but are not quite so prominent like working memory. So if you develop a test that combines those aspects of the phenotype, one test that combines those aspects of the phenotype, you will do better than just testing reading very broadly in terms of what normal people should do when they read. The way that these different aspects are handled, that's where the devil is in the details. Because if you make a separate test for each of those and say you've got to fail this and you've got to fail this, you're into the error adding problem. But if you say, okay, we gave this battery of tests and we determine what scores in combination predict being dyslexic, that's fine. What that ends up having to do though is it weights the different test scores. So you get test score A, which maybe is a decoding test and you wait it by a factor of say, I'm going to pull this out of my head, five. And then you get a working memory test and they are, may or may not be bad on that, but it isn't as big a part of the phenotype so it gets a weighting of two. And you get a vocabulary comprehension test and some dyslexic have problems with that, many do not, so that gets a weighting of one. And so when you combine all those scores multiplied by their weighting, what that means is that if you are dyslexic and you're really bad at decoding, you're going to get identified, okay? But if you're a little bit bad at decoding, but you're also a little bit bad at working memory and you're a little bit bad at vocabulary, you're also going to get identified. So in those cases, paying attention to the phenotype really does help in how the test is developed, but it takes a very different kind of scoring to get the accurate sensitivity and specificity. That makes sense to me in terms of thinking about structural equation modeling, right? So you have a construct and you have multiple indicators, but that model is pulling only the shared variance and leaving the error out. So kind of speaks to this a bit, right? Like you're pulling in what's maybe shared in that phenotype and you're giving some leeway to that weighting and that will give you a better diagnostic accuracy, but that can't be just cart blank, give multiple tests. It's going to be a better sensitivity and specificity. I think, yeah, no. No. You know, and the thing is there's there's also kind of a groundswell of using latent variable approaches to identifying dyslexia. And I'm sure we'll see that bleed over into other fields of behavioral disorders. But it's not good enough to just say you're high on this variable, this latent variable that represents dyslexia. You also need to know how accurate that is. So it still comes down to sensitivity and specificity. You can have a latent variable and a cut score for how much of that latent variable you have to have, but you in the end need to be able to say how accurate that method gives. Absolutely, and I think what I'm seeing is some states are saying, "Okay, to determine risk, you have to test these three constructs that we know are critical." But there's not a focus on the sensitivity and specificity. Even though theoretically, yes, those are the phenotypic aspects that are key. It still has to be, like you said, the sensitivity and specificity. That's what's going to determine whether that specific measure is sensitive, because you can have a lot of different measures of biological processing, for instance. But that doesn't mean that they're going to be good for screening and have the diagnostic accuracy you're interested in. So I think that's a tricky part of it. It's tying it all together, but really having the central part be that sensitivity and specificity. And just to reinforce something that we already covered, is that one test of phonological decoding is not equivalent to another. That you can't just go to the publisher and say, "Well, there's a decoding test. That's the one I'm going to take." You need to know something about how well it performs in detecting known decoding problems, which is sensitivity. Well, I'm an eternal optimist, so I'm actually quite excited about these laws, because I think it's going to force these issues more. And I think that it's rubber meets the road when you have, I'm setting with districts now, and they have all this data. Then you do have to say, "Where's the arbitrary cut point? Where is the cut point? It makes you think, what should I do? Or when you're thinking about how many children are at risk, you have to start thinking about sensitivity and specificity." So I think this could be a really nice byproduct that we start to see more of a focus on this evidence base. That's my positive view. I also have to say that I've been involved, and this is driven, some of this positive view, in something called the tools chart. And that is funded by the Department of Education, and it's been going on now for a decade. So we first got together, it was like 10 of us, and we were, you'll probably relate to this. 10 of us had a set in a room in Washington, D.C. and think of a user-friendly definition for screening. It took us a full day to write one, like two sentences. You can imagine, right? It's now evolved, and it's been revised many times. But for 10 years, what we've done is we created a website that has more consumer reports. And it has like, you know, full bubbles, all the evidence, good job, half bubble, empty bubbles, not good. And you can kind of look at it across the chart. But what we did is solicit publishers to send us information about all of these aspects, sensitivity specificity, reliability, all of it, but it was all driven by purpose. And so there's a tools chart for screening. So we, you know, you can imagine what we're looking at as criteria for screening is very different. We look at, like, how did you determine the child had the impairment in the first place to tie to your sensitivity specificity and where your cut points and all, what's your base rate? And what we found is initially, many of the tests did not have good ratings, as you can imagine. There was a lot of empty bubbles. So then what we would do is we'd send feedback to the publishers to say, here's why you didn't score well. And in the 10 years, the publishers have improved their assessments so much, because they want to be on the tools chart, because the tools chart is used by administrators and Department of Education at the state level to make these decisions about what tests to purchase. So I've seen this evolve over the decade that we've seen this improvement as kind of a pushback from, you know, that back and forth science publisher approach, and it's made me feel more optimistic. And I'm hoping we'll see that same kind of evolution occur with these laws in place. I don't know. Yeah, well, I'm also an optimist, so from your lips to God's ears. No. And I love the idea of these tools charts, because it, you know, instead of having to go search publishers websites and a lot of times the publishers don't actually provide the critical information you need on their website. You end up calling them, or at least I end up calling them. Yeah, which does save some time in terms of having them ship something that I'm just going to have to ship back. Right. Yeah, so that's a good thing. That's a great thing. Yeah, I'd like to see some of that done for speech language pathology, just because it would just be such a handy tool to have. And publishers are very responsive to consumers. And so, you know, the chart is a form of consumer pressure, but the other form of consumer pressure is when you return a test, stick a note on it, say, and why. Yes. One of the questions my students always ask me when they discover how bad many tests are, is, well, how can they sell these? And my point to them is publishers are not in the business of ethical practice. That's your business. Okay. They're in the business of selling things. And as long as people like you buy them and keep them and don't return them, then they will keep selling them in exactly the form that you are buying them in. Making a test better is costly. So before I develop my first test, I always wondered why in the world tests cost so much. I don't wonder anymore. Oh, yeah. A decade. So, yeah. So, I spent 13 years on the TILs and other people were on that team longer than I was. And by the time we were ready to publish, we were millions into the development. If you count our time, our salaries, data collection costs, all the costs of the tri-outs that we did, yeah, easily. But the point is that publishers will publish whatever will sell. That's ethical practice is not their business. That's the practitioner's business. That's your responsibility to make sure that you are doing evidence-based practice. And if the test is not providing that evidence, don't use it. Absolutely. I just saw something on NPR that was about the headline was something like, do we need an FDA for educational practices? And I've heard you use the term educational mount practice. And I think this is kind of like, you know, we kind of think sometimes, well, there must be someone watching, like your students saying, why are they selling this? There must be someone watching over this. No, that's our job. I mean, we are the ones that need to do this. We don't have an FDA for our tests and our, you know, practices that we use. It's our ethical code and our knowledge base. So I did like the idea, actually, of an FDA for educational practices. I thought, that's pretty good. That would be nice. And actually, you know, Lynette, speaking about the tools chart, I'm thinking, hmm, I think I'm talking to someone who might be able to do something like that for an action. Should I be a business name? What about? Yeah. Actually, we do have a committee that's relatively new that is specifically designed to be dealing with putting out information on evidence-based practice toward professions. And one of the things that they're doing is developing practice guides. So I think that this is possibly one of the areas that we should add to that. And you know, we're saying here that there's a lot of bad test and child language. Guess what? There are a lot of bad test and motor speech. There are a lot of bad test and aphasia. But in each of these areas, I'm starting to see good ones. So there are choices. So if people are throwing up their hands and saying, well, all tests are bad. No, I know that in child, there is at least one choice at every age. Every age. So you just have to get the right test and get rid of the test that are not doing it. Or sometimes, if you have a hard time tracking down the information you need in the test manual, it's either because it's not there or because it's being disguised in a way that they don't want you to know the full story. So these are ways that red flags should go up for you that are not that time-intensive to check out. I would say that for clinicians that I speak to, 90% never open the technical manual. For me, it's the only thing I open when I order a test. Right. Don't get past that. I don't even look at how to give the test or what the content of the test is. That makes a lot of sense. And I think that having these resources out there, like your article, like what's being driven by these practice guidelines, and it's there. And also hearing about your empowering story, changing the procedures in Arizona. I think it's really the future's bright in that way, and we can be our own advocates. Well, I know we're running out of time, and I want to ask two questions. I always ask my guess. And the first one is, what are you working on now that you're most excited about? Oh, gosh. We're going a lot of things. I have to say, though, I can give you some ideas because I know some of things. What about your physical learning kind of stuff? You know, probably that's what I'm most excited about right now. So we have a line of research that's designed to make treatments better and more effective and faster by using principles of learning that are grounded in how children and adults learn implicitly, which is a very rapid form of learning. And we're just wrapping up an article, our study now for resubmission, our resubmitted yesterday, that shows that we can cut the treatment time in half from 30 minutes to 15 minutes, still get kids giving us the same levels of performance. And again, this is a rapid form of learning and it doesn't require you to ask five year olds to think about, well, this is a boy, so we say him. And there are no metal linguistic skills that aren't developed for young kids involved. You don't have to ask them to think about a rule and apply that rule. It's very rapid, very input driven and fun to do. We like it. The students love it. That after they get the training through the research project, they just love it. So I'm very excited about that. We really do have a couple of new tests that are in the pathway. Oh, really? Yeah. So I have two that I'm working on now. One's a little bit further along than the other, but yeah, they're certainly getting closer. Probably we're about a year off for one of them and maybe a couple years off for the other. That's fantastic. I know how long that takes to get through the pipeline as we talked about. So that's great. And I will circle back to the statistical learning just to say that I very much appreciated the language to be sharing services. It's clinical forum that Mary Alt led on statistical learning. And I was a gradish. And you had an article in there, correct? I did. So I didn't want to. I team with a developmental psychologist to talk about what are these basic fundamental principles of learning that can be incorporated into treatment. And it's not like one kind of treatment. It's any kind of treatment. So if you are doing conversational based research or treatment with a four year old, it'll work. And if you are doing worksheet based treatment with a school aged child, it still will work. So yeah, we were really excited to put all that together in one place and give it some structure so that people could get their heads around it. Yeah, I'm excited. I read maybe three. I'm working through it. I just absolutely love those principles. And having been an Arizona with you several years ago, it was really mind-blowing to think about the input that you're providing and how children can pick up on these grammatical structures, for instance, or vocabulary learning based on the types of stimuli you're choosing. And it didn't require met and awareness, which is quite problematic for children with difficulties anyway. So I'm excited to get that, you know, I'll attach that to the podcast resources. I think that'll be a great-- Oh, thank you. --to shine a light on. The other question I asked, the last question is, what is your favorite book from childhood or now? I'm going to say pride and prejudice. Oh, great. And, you know, I started reading it when I was a teenager and I probably rereaded every other year or so, often around Christmas. I don't know why. Oh, I love that because I-- most people choose something that's more-- they're young childhood. I like that you're choosing more of the adolescent book, because I like-- Yeah, yeah, yeah. And of course, I, you know, I've read all of St. James' works. Oh, that's fantastic. Yes, I do. That's fantastic. But I don't know that really is my favorite by a little bit. There are really good ones in this series, but yeah. Okay, fantastic. Well, I appreciate you spending the time chatting. It was really nice. I look forward to getting it out to the podcast listeners. Okay, it was my pleasure. Thanks. Thank you. Check out www.cherespeakpodcast.com for helpful resources associated with this podcast, including, for example, the podcast transcript, research articles, and speaker bios. You can also sign up for email alerts on the website or subscribe to the podcast on Apple podcast or the other listening platform, so you can be the first to hear about new episodes. Thank you for listening and good luck to you, making the world a better place by helping one child at a time. (gentle music)