Ep4: Exploring Item Types

Are you teaching a critical thinking skills class or verbal comprehension class? No, so that should not be the thing that is affecting how they answer that question. And that's the thing that we run into all the time. Hello and welcome to the It Depends Podcast. Your alternative guides the often mystifying world of psychometrics. In each episode, your host, myself Timbernet and the brilliant Dr Amanda Danis will serve you a cocktail of psychometric wisdom garnished with a twist of sarcasm and humour. Alamission to help you actually enjoy the process of building better tests. So grab your favourite beverage and get rid of your dive into the awesome world of psychometrics. Amanda, welcome. Welcome to It Depends. Welcome listeners. Here for another rollercoaster of an episode. This time we're going to be talking about item types. How you doing Amanda? How are you? I'm doing well. How are you? I'm good. I'm good. I like that you already said roller coaster because I have already thought a little bit about this topic this morning and I'm just going to go back and forth and probably say some things that people are going to want to email me about. So we'll see how it goes. Are you going to be contentious? I might be. I might be. That's good. As contentious as a psychometrician can get possibly. Excellent. Okay. So we're talking about item types and I think I always might kind of go to one and think about different types of items that are out there. Is the multiple choice one? So my first question to you, Amanda, today is is multiple choice the best question type for a test? Well, it depends and I'm easy. So when you might see certain it depends. There we go. Really early on in the episode. And again, like a lot of the other topics that we've talked about, it depends on what the content is, what your resources are for building the test, how you're going to score the test, what type of protestor appeals you might expect to do or to get, how many smis do you have to create. There's also things that go into what type of item types are best. And I think most psychometricians would agree that at the end of the day, if you could do a performance-based test, a practical exam for every job, that would be ideal. That would be, that would remove some measurement error. You would actually see the person doing the job, but that's just really, really impossible in a lot of arenas for different reasons. And practical exams, performance-based test are my favorite tests to work on. I'm sure we'll talk about them later. But multiple choice items, I think, get a bad rap these days. So I am still a fan of a good multiple choice test question. And I think that they are more difficult to write than anybody thinks to make a good one. And I think that they can be done very, very well, but they can also be done very, very badly. So there's a lot that goes into them, both in discussing them and developing them. So yeah, it depends. There's nothing wrong with doing a multiple choice test. But if you've got the resources to do something fancier and more accurately measuring the skills and knowledge, then I vote to do it. Do you think that makes sense? I didn't really answer the question. I mean, it's just kind of. I suppose no, no, it's fine. So it's kind of, you know, there's a slightly saying, there's a lot of factors you need to consider as to whether or not you're going to use a multiple choice question type or any other question type, really. When you're starting out, do you. When you're starting looking at a test, do you always immediately have in your head this is going to be multiple choice test or at what point do you ask yourself, what kind of question types you're going to use in this test? I think that's a good question. I think it is sort of the default, right? It is the most common by far, it is probably the most commonly known for SMEs to write a test question like that. If you start talking about hotspots and drag and drop and these other kind of more technology enhanced items, you know, there's also a learning curve to get your SMEs on board or your item readers on board to write that. So I think first off the bat, normally when I ask somebody, what is your test? I'm surprised if they don't say it's multiple choice. And lots of times it does raise actual red flags for me if I get a new client that's not doing a multiple choice test. Because if they're doing an essay or a rubric based test or portfolio based test, I'm immediately on guard as to like, okay, here comes 5,000 questions as to how did you develop it? How do you score it? How do you validate it? There's 5,000 things that have to go into that type of item type, whereas a multiple choice can be fairly more straightforward. So I do think it is sort of the default, I think, in new programs starting out, especially if they don't have a ton of money, you're just kind of going to kind of start with a multiple choice. And there's nothing wrong with that. I have, we have a few clients who are doing performance based tests, other, either in simulators or in like mock up systems for drywall or whatever it is. And it's really, really cool. And the budget is extensive for those. So yes, I think multiple choices is the default. I think it's way better than doing true false or fill in the blank, which I hate. Why? Or if you go against fill in the blank. Go on. I'm true. Fill in the blank. I think is a lazy way of writing a bad multiple choice question. It is harder to grade. I can turn pretty much any fill in the blank question to a multiple choice question. You just have to come up with better distractors usually or actually write out your response options properly. And we don't really allow fill in the blank for our clients unless they are, I mean, I don't think we have anybody that uses it. Nor true false. Yeah, I think that it's true false. Now that's fine. True false is one that I would kind of imagine, yeah, okay, fine. That's quiz territory. And it's kind of maybe okay for a quite quick formative solution or something like that, just to quickly go to the classroom. And I suppose in a way, it kind of stems from people in the classroom putting their hands up when they're kind of who would agree with this voting, isn't it really in a way? But filling the blank, it's interesting you said though. But what are the question types are there out there that you think are kind of quite common these days and yeah, well, let's kind of expand that a little bit. So before I say I hate what I need now I need to caveat my I hate fill in the blank. If you have a small sample size and you're going to hand grade your tests fill in the blank line. If you have somebody to pay to do that, but if somebody is like our test, it's a fill in the blank and I say that silly to have somebody to pay somebody time to go in and grade it. And then what if it's sort of the right answer but it's not really the right answer. I mean when you do an open construct answer, you're opening a can of worms with you know what if they spell it incorrectly? Are you still going to accept it? If it's computer based test and you're doing fill in the blank, if they type it incorrectly, if they have spelling issues, are you measuring your spelling issues or not? Write a better multiple choice question and you kind of remove. So for me, the fill in the blank is crappy from a psychometric like scoring logistical standpoint. Right. You mentioned it. You mentioned open construct. Can you just just say anyone who doesn't know what that means? Open constructors means you are just or constructed response open response, constructed response, open constructors. You're just writing it's either an essay or your your create the test taker is creating the answer. They're not selecting an answer. Yeah. That makes sense. Okay. That's fine. Yeah. That's fine. That's fine. That's what's it. And I can yeah. Yeah. Okay. So we've upset the fill in the blank squad there. We're going to get a breakthrough our windows through that. But you're right. It's going to be certain other times. What are the question types? Is a bit of a kind of a more of a go-to for you then? So if not multiple choice, what would be your next kind of option in your question item type arsenal? Some sort of practical tests item, even if it's still so people will call this is another another fighting phrase, but people will call non multiple choice items. They'll say it's not a multiple choice. It's a drag and drop or it's a matching or it's but they're all sort of those are all sort of essentially still multiple choice. Yeah. And the same thing if you use audio or video or diagrams or schematics and images, a lot of it is still could be broken down into multiple choice, even if it's a series of multiple choice items that lead up to the final answer. So I you know we do a lot of incorporating a video audio images, especially in a lot of our technical exams for certification exams, you know diagrams, blueprints, all that kind of stuff. They're still essentially multiple choice because at the end of the day if you're testing thousands of people, that's the best way to be able to do it. And an efficient and a good way as long as the as long as the items are written well. They're sort of hard to right. But I do think that there are, you know, there's newer ones like there's drag and drop and there's you know hot spots and there's you know data entry ones and so like data entry into a spreadsheet that's still sort of right wrong, it's still sort of binary but you are still sort of doing a practical exam because you're measuring how well they're doing that skill of doing data entry. So there's lots of cool stuff and CVT that you can do that's borderline practice performance exam and not really multiple choice. So I just like to get creative. We have one group that's doing a portfolio assessment and they're new and they came in and they were very proud of their portfolio assessment and I felt really bad because we just kind of ripped it apart and said you would be better off doing a multiple choice exam at this point because what you're doing is inconsistent across candidates and that's to me is more important it's really important is fairness. So we do a lot of different types of items but I would say at the end of the day most of the items are multiple choice or some format of multiple choice. Does that make an error? Yeah. Yeah, it just so I'll tell you a quick funny story in a second but yeah one of the many a lot of the platforms out there that provide you know assessment delivery platform with they kind of this we've got 50 question types. Yeah and I want them to all get together and I want them to make me a beautiful pie chart of the proportion of items in their banking system that are multiple choice versus everything else and I've had one vendor partner do that and it was came out exactly as you would expect which is fine and there's nothing wrong with that and can you use cool items? Yes but you also sometimes I also have to talk to client out of using so I'm like why do you want to do that type of item? Well we have the technology so we should do it. I'm like but why you're measuring something that has nothing to do with that type of item like why would you that your people don't see that on a daily basis in the job so why are you trying to use that just to like be cool? Yeah it's it's and it introduces I would say some measurement error in there because if I get to an item type that I've never seen before and I'm like what is this? I mean I'm old so I I was like I don't know what these buttons are so you've got that learning curve there so you just have to you can make a good argument to use some of those cool items but lots of times I'm just going to shoot it down so no no again we make a very good point about it fighting words fighting words from that user building point that you you excellent because I've seen a million different one not a million but I've seen a lot of variations of people's implementation of things like graph types and stuff like that as to how you kind of manipulate it and sometimes people get a bit kind of concerned about the scaffolding that's then put in place to try and help people use those items and if that's then helping them do that you know with things like spreadsheet item types you know they're great when they replicate word I remember working on one particular project where we we were told not to try and replicate word even though word was vastly you know kind of used platform it was because it was too word specific and we were being a bit unfair to the other platforms and systems are out there so there shouldn't be yeah and I know I should have had you on my team at the time and stuff but when you said about the hotspots and stuff there's a great little memory I have from it's not my meeting but when someone came out from a meeting and said straight away they were on a call to so this is from a British organization demonstrating their platform to an American organization for the first time and they they showed this I don't know what it was like a hotspot a graphing question type and the response from the the American list client that was on the thing was wow that's the last Vegas of item types that you know it was just we loved it as as Brits were kind of that's so funny because I think but one thing I've always I've never been in salt or as a compliment no no they were like wow yeah I can't do that I don't like Vegas so that's not my cup of tea but but I think it was just that kind of wow so yeah such a fancy funky item type maybe the one maybe they were taking them in me I don't know I think that if I say that if I say that to you that's not a compliment you yeah okay right yeah I think it wasn't just it's not just and distracting is what I mean about it there's too many colors and there's it's sensory overload and I can't do it so yeah yeah yeah yeah yeah yeah did they end up but did they end up using that item type I think they did yeah I think they they they well they like with most I think it was kind of like a sprinkling of the item and I think that's just one of the questions because I you know when you you know about blending item types and stuff in there the one stat I remember from a client that I worked with in past was of their millions of item bank that you know they had millions of items we were migrating from one system to another system and I think we did account there was some like 98% of the item types were more will choice within that that migration and the rest were they had some functional skills which is the UK based qualification so which is which is quite hard and tough have you ever looked at the functional functional skills tests have you ever heard of some tools think so like general I can you type stuff or it's like it's for level two and three students so kind of non GCSE level school living students who are taking a slightly more vocational route into education so if you if you leave school to go to college and university in the UK so at 16 you know 16 you take you know national tests when you take the GCSEs if you then go if you fail your GCSEs or you don't do very well at GCSEs in English and maths then you would take the functional skills one it would be quite a good episode for us to do actually just to look at that that test is a whole and that would be interesting because we don't have that we don't have that in the States I mean we could get into a whole standardized testing thing but we don't have we have the SATs that we don't have like a functional skills or like a something that lets you know what you should do if you're not going to college I mean there's career development tests and there's you know career inventory things and I've taken a several of them and they always tell me I should be a forest ranger so I'm not really sure what that means but it's not a diagnostic style test it's yeah okay I would be good to know about it yeah it's it's good to want to look at but one thing I've always noticed over time is that you know in Europe you know there are a lot more use of different item types I'm not saying that you know people still you know new programs that come aboard are probably worth choosing multiple choice as the starting point but I've always kind of observed that there is a lot more use of different item types in in Europe than I have seen when I've worked on testing America so is there a difference? and are there a lot of different to that? are there what are there a lot less psychometricians in Europe than there are in America? there are a lot less psychometricians if there's anyone out there there's a psychometric who wants to move to Europe you've just thought it to me because I know quite a few people are looking for psychometricians is that a is that a correlation does not mean causation issue or is that like that people are just like using all these fancy items without really much thinking about the measurement impact of them or because they don't have a psychometrician to say why are you using a hotspot nobody like what does that lend to the measurement of your tests? does it differ in the conditions? sorry I think this is the Nana this is one of this has always been one of the kind of differences in when we talked about this at the EATB conference in London and things like that and we were you know the different emphasis that people put on psychometricians in different spaces you have in in Europe a psychometrician you know will be present but wouldn't necessarily be leading the process you know that's quite a sweet statement whereas you know we've always talked about with yourself you know a psychometrician in the US or North America is kind of always leading that process a little bit more don't they? but that's that's a whole topic in itself isn't it? that is a topic because it depends on the psychometrician as well and yeah I'm just off-tee so one thing I wanted to ask you what do you see has been the difference between a performance test items and technology enhanced items what's your kind of view that do you think? do you mean like labeling? so performance based tasks is to me you know and I sometimes I wish we were like a radio show and we could accept collars live because I would be like you know I'd call out my colleagues and be like dial in I know you have an opinion on this I think to me a performance based task is basically one where somebody is demonstrating the skill that you are trying to measure flat that's just it there's there's multiple ways to do that whether that's in a simulator whether that's on a computer whether that's in on the job testing that's the performance based task you are performing or demonstrating the skill technology enhanced item to me can mean can help with performance based items it can also just mean you're providing writing audio, you're giving a video, you're making an adaptive kind of testlet of items where if I answer it this way, I go down this path of items. If I answer it this way, I go down this path of items. So there's lots of things that technology does that actually doesn't have anything to do with actually performing the skill. And so there's lots of ways that I think that technology enhanced items, custom process more efficient, but as I've been saying all along, I think it can also hinder kind of, it can add unnecessary bias or unnecessary flue for lack of a better or technical term, unnecessary flue to the test that people will do just because it looks pretty or it sounds cool. And maybe that's worth it, right? If you're launching a new certification exam and you have a competitor and that competitor is doing all these cool items and people just think that it's really cool, you've got to keep up with the Joneses and you might have to do that as well. But from a measurement perspective, I'm always going to be that like the devil's advocate who says, I need you to just explain to me why this item type helps me measure this person better. And it might be that it's more efficient. It might be that it's cheaper to create. That's absolutely true. But I just want to make sure that it's not hindering how I measure that person. And the performance based test is ideal. I mean, if we could make that, it makes me so happy. I love working with clients who have joined performance based tests. Super cool. Commit itself. Yeah. So I think I can talk in general terms without being arrested for non disclosure a violation. But one of our clients is the board of pilotage commissioners in Seattle, Washington. So the Marinor pilots are the men and women, there's one woman in the entire state of Washington who basically they will go out on a pilot boat and they'll meet the huge tankers, the ships that are bringing in cars and cargos. I mean, the ginormous, you know, and they will pilot them and drive them. Be the captain. They'll take over the controls and they will drive them into the pilot into the Puget Sound and park them basically. And the reason is that they do that is because these captains of the cargo ships are coming from all over the world. They don't know a Puget Sound. They don't know the ins and outs. They don't know where the silt is that day. They don't know the currents that you know all this stuff. And so it's for the environment. It's protecting the environment. It's protecting the structure of the Puget Sound Harbor, all this stuff. And they're highly trained. It is like the pinnacle of the Marinor occupation career. And it's a huge testing process. It's for every four years. They take a written exam. If they pass the written exam, then they get an assimilator and they do a simulator exercise. So what I've been able to do for the last six months is work with a bunch of the pilots in Washington state to design the exercise that's going to occur in the simulator exam. And it's really, really cool, really cool. Like I can't, I can walk into the simulator and I get motion sickness within 10 seconds. Even though I'm standing. I'm in a room. I know that I'm standing in a room, but it is, it is everywhere. And you're on the bridge of a ship. And I will vomit if I don't leave shortly. So I don't go in the actual simulator very often when they're running it. I sit in the control room. Um, but it is, it is amazing because how else are you going to, how great, how else can you perfectly measure ship handling besides putting them on the ship and risking a bajillion dollars worth of damage? So this is, this is really cool. We also have the glazers. They've got some mock up of glass, you know, in-saw systems that that ship around the country at different training centers and they test guys and they run through and say eight hour training session. And they get graded on this extensive rubric. The examiners are trained and calibrated every year. I mean, it's, it's a lot, but it is the psychometrician stream is to be able to design that because then you're talking about scoring, multi-dimensional items, weighted items. It's just really fun. So I could talk about that forever. Sorry. No, no, it's good. No, it's interesting. And it's interesting point you mentioned there, but we talk about the kind of comparison between performance testing technology and hand-stitum. So, you know, performance test is, is the, is the process and that could include technology. It couldn't, you know, it could just be purely an observation, observation based exercise. Whereas when we talk about technology and hand-stitums, they are, you know, items themselves that are leveraging technology to go beyond what you might have been ever been able to do if you're doing on paper or something like that. It's, it's kind of, it's taken a whole spin and I suppose the simulation, you talk about simulations, have flight sims and those kind of things. Yeah, that's still a performance based exercise that's been enhanced through, you know, through technology because it could be, like you say, it could be them piloting them about to do that. Yeah, I suppose, you know, when you're looking at, you know, things like multiple choices and stuff like that and you mentioned there about doing an essay at the start before you did that simulation. Do you think sometimes that things like essay response question to pipes, so those kind of things, do they measure deeper learning as part of the process or can multiple choice do as good a job? I need more coffee for that question. It depends, it depends on what content are you asking them about, right? Are you asking them to interpret a reading passage and express their own feelings using imagery? I mean, what's your promise, right? Are you asking just a simple question and they can just write an essay to answer it? Or are you asking something really complex and for them to really dig down deep and synthesize information, turn it back into metaphors and similes and all those things and imagery? I mean, is it an English literature essay or are you asking them to write about their experience on the job and how that will help them in the future? That can't be answered with a multiple choice test question because it's specific to the person. So one thing about essay questions is how unique is the answer supposed to be? Like, like, the population or the universe that the answer is coming from is that universe unique to that person or is it general knowledge? So that's one thing. The other thing that comes into essay questions is if I had to pick an answer, I would say, yes, essay questions can get at deeper knowledge because they might require all of that synthesis, that a multiple choice question might not, but a really, really good multiple choice question can get out a lot of things that people don't usually assume that it can. So if I write a complex reading reading passage and then I write some questions that make the person think and make connections from the reading passage to get to these solutions that at first you look at the response options, you're like, no, those really fit until you go back and you think about it and you think about imagery and then you put, you know, it's really difficult to write. And it's much easier to write an essay question that says using imagery, you know, describe blah, blah, blah, blah. But it's also a lot harder to grade because I'm always going to say, what is the rubric? Did you train your examiners? If you're using an automatic essay scoring system, like some of our big vendor colleague friends have, that they've spent a bajillion dollars, very technical term bajillion to develop, you know, for those large, some of the standardized testing to do essay scoring. And they've had some great success with it, but lots of people can't either afford that, you know, to run it through that or to develop the algorithms to grade them properly. So I think that there's, there's benefits to both. And again, it's that content. What are you really getting at? What are you asking about? And is it unique to that person or is it general knowledge? If it's general knowledge, I'm guessing that we could probably write some really cool multiple choice items. But if you have a small sample size and you have people trained and ready to grade them and you're ready to defend your process in a court of law, if you're dealing with employment or certification, then by all means go for it. Makes my job easier on the, on the front end, right? What's a lot easier? Exactly. Cause the way I kind of see it is that the, to do it, like you said a few times to write a really good quality multiple choice question. There's a lot of effort that he's going at the start. It takes more effort than it would do to write a kind of quite easy essay response question type. And then just the problems then come in the mark and aren't there. And the subjective nature of that and the making sure everything's level. And that's going to be the same with a performance test items and things like that. That aren't using technology to, to try and, you know, make sure there's a degree of fairness in there. If you just down on, because you can imagine a performance testing situation, someone walks in and for whatever reason, they just put the backup of the, the, the greater straightaway. Yeah. There's a potential there isn't that, you know, they're not going to do well because I don't know. Maybe they're just reversed into the car in the car park. Yeah. Well, and depending on, depending on the industry, they might know them. And so that becomes really difficult as well as how grade blindly and to have some checks and balances. So you can never have just one grader. There's all sorts of things too with the performance based test and part of my job coming in as a psychometrician is to be that kind of like that devil's advocate and say, okay, but you, used to work with him years ago and now he's applying to be, you know, a new blazer or lean building. I mean, we're doing a bunch of them, but, you know, how are you going to remove your bias? And I don't mean it with any disrespect to anyone, but we're all human, and it's really hard to do. So that my job is to come in and to make it as bias free, as possible, which isn't always, it's never going to be 100% on anything. So, as we, as you kind of come towards the closer this today, we've unpicked a few areas, and I think we've unpicked and discovered a few areas that I'd like to go into a bit more depth, like the multiple choice to you. What is, what does a good multiple choice question look like? Because I've got students trying to take some exams at the moment, and it seems to me that the way in which they've made the questions difficult is to really mess with you mind a little bit with the way in question. No. So that should not be the thing that is affecting how they answer that question. And that's the thing that we run into all the time, because we get people who double negatives or, you know, which of the following is not, and you're just like, come on. Like are you just trying to trick them? Like that doesn't make it necessarily, it makes it difficult, but not in the right way. You're measuring something that is not the goal of the test. I'm going to send you this specimen to paper, because these students are really struggling with, because they're vocational students, they're apprenticeships. And, you know, they're not academic students who are going to, you know, kind of read through the lines that they just look at this and it's like a jumble of words to them, it's kind of, you know, like, we only kind of lose teasers, you know, track it or something. Yeah. Yeah. It's really unfortunate. And I think all of us can think of those tests that we took in school that were just terribly written. And you feel, you go into a test, and you know that you know the material. You feel good about it. You know the stuff you've studied. You feel really comfortable and you get it and you bomb the test because the test is just total. Can we cast on this podcast? It's total, it's total crap. And that's really unfortunate, you know, for kids who are in school, you know, I coach high school sport and I've got teenagers coming all the time and they talk about a test that they thought they would do really good over there upset because they failed. And so imagine people doing that in certification licensure. So they've studied all year, their job depends on it or their promotion depends on it or getting a job depends on it. And they go in there and the test is crap. I mean, that just is terrible. That's just it really makes me angry. So you can tell. Yeah. No, it does for me. I've got trying to help these guys, these students who are test dates just and why I want to do it. But there's even one part of the syllabus which is actually wrong. So I'm having to teach them the wrong stuff so they can pass this test, which is a kind of step within it. Yeah, it's it's her. Anyway, teaching, we have in the state of Virginia, we have S.O.Ls and I won't say what teachers say that the S.O.L stands for, but it's the same kind of thing where teachers feel like they are teaching to the test, just to get the kids to pass, but it's actually not the greatest information. So I'm not against standardized testing. I think it's necessary for a lot of things, but it's not always done well. So and I don't know enough about this to know, but yeah. So that stinks. So yeah, and I would love to talk about it or read about it with you. Well, I'll pick that one. It's going to be taken off the syllabus soon. So we can we can pull it apart after it's removed. It's perfect. So what do I want to leave the listeners with if we can, is kind of three things to consider when they're looking at the item types. What would you say the kind of three top points that they should think about when thinking about the item types they're going to use within that test? Could you give it some read that? I can give you three things, but they're not necessarily in a ranked order. I think they're all fairly. That's fine. That's fine. One is the content of the job. You know, is it a desk shot? What are those skills and what are they best? How are they best measured? And to kind of look at that dream like ideal, what is that ideal and then work your way backwards from there? And because resources, not only resources to build it on the front done, but to maintain it. So let's say you've got a bunch of budgetary money right now. You know, everybody's excited to build this new test. That's fantastic. What about in one or two years when you have to refresh the item bank and you need more items and you need to do extra things? What's it going to look like that? Because you don't want to be changing up your format of your test over and over again. So content resources and probably scoring, I would say scoring, you know, how are you going to score those items? That has to be maintained as well. If you have a good group right now that's ready to volunteer or you're ready to pay them to be your scores and your examiners on your performance based test or portfolio or whatever, that's great. What are you going to do next year? What are you going to do the year after that? It just has to be- Does it scale as well? Right. Is it scalable and sustainable? So I guess that builds into the fourth, the third slash fourth one because scoring kind of fits in there. Because people just kind of forget about that. They're like, they get excited about the measurement part, which is cool, right? I'm glad that they've read some stuff or they've listened to me and they're constructs validity. They're all about that, but it has to be able to be maintained. And so I think that's why a lot of people end up with a multiple choice item, but again, they can be very, very, very good. Yeah. Okay. Don't give up on them. Yeah, yeah. Everyone, yeah, don't keep the mini heart really. They have a special place. Yeah. Use audio clips or video clips or images and your response options are in your stem. I mean, that makes the world a difference. That just text base. And that's fairly easy to do these days. So fancy, fancy multiple choice items. Scenario based technology enhanced, sure. Las Vegas style multiple choice. That's right. I'm not too much Las Vegas. It's Las Vegas. I'm shutting my eyes and I'm just going to get my ears. That's brilliant. And it was just a lot for me. I quite enjoyed it. Anyway, anyway, we're talking about our holiday trips and other time. But well, thank you. That was excellent. I think we kind of really kind of, yeah, we covered quite a lot there. I think we covered giving some people some great tips to take away on there. As ever, you know, listeners, please hit the subscribe button if you want to hear more from myself and Amanda as we put on pick more aspects of psychometrics. Then please do that. There's also a link within the show notes to the page where you can ask questions and you can post questions to us and we'd yeah, I love to hear your questions and we'll name check you as part of that. Or complaints or complaints about my absolutely. I would love to hear it. Absolutely. Yeah, please send that through and thank you, Amanda. Once again, it's been excellent. And no, catch you all another day. Yeah, thanks. I can't goodbye. Have a good one. Bye. See you later. Bye.