AI Can Fix Your Broken Research Repository

[Music] Welcome to the BoAG World Show, the longest running web design podcast, where we look at you for experience design, conversion, and optimization of working in the web. This month's show we're going to be discussing AI Powered User Research for Positories, and then in the very next breath, I'm warning about the dangers of AI. So, make of that what you will. My name is Paul BoAG, and joining me as always is Marcus Lillington. Hello Marcus. Hello Paul. You're right mate. How's it going? Yeah, not too bad, not too bad. We've been, it was really interesting. We're having a moment in the Agency Academy Slack channel, I don't know whether you noticed that. I didn't know. Not too busy Paul. Oh, we've actually been doing work. We're all feeling a bit burnt out, and we're blaming AI for it. So, yes, that's where we're currently at. It's because. That's the thing that is already a recognised thing. Yeah, and it really is as well. It used to be, you sat down and you did one thing, right? But now I've got like three or four agents on the go, all doing different things, and you're having to hold it all in your head of, you know, and you have to check and review and all the rest of it. And so, yeah, I'm a shit ton more productive, but I'm knackered. And I'm not the only one. Several people were moning about it. So, there's a thing, is it? If you read about it, some of them. Yeah, I'm even though I have for the first time in a very long time come up with some reads for this week, which are a little bit, well, they are quite AI related. Yeah, I'll see if I can dig one out for the next show because it's basically, I think it's called AI Burnout, which is to do with basically. I'm going to repeat pretty much what you just said, but having to manage too many things. Yeah. You basically become a manager, rather than a doer. Yeah. Well, that's a bit too strong. You know what I'm saying? Yeah, yeah. I'd probably be quite good at this because I've always just had a million things on. Yeah. And they just skip between them. But I'm not a big user of AI. Use it a bit. When it's specific, tell us what it's going to help. I don't help it run my day. Whereas I probably could cope with it. But I can completely understand that we're not. I think I saw a quote that, you know, we're only meant to sort of deal with, you know, whether we're hungry or. Yes. Or whether it's wet and we need to get under some shelter and having to deal with very complex issues, multiple ones and make sure that you're on top of it is going to make you go bleh. Yeah. Yeah. So, time. I reckon I can only be productive at that level for about four hours a day. And after that, I begin to kind of start dropping things and flicking up. Flitch. Yeah. We glitch. What about you? How are you? Things going alright with you? They are okay. Headscape. We want to clients. A new client the other day. Yeah. And we're bearing in mind how long I've been saying we haven't done that. I mean, I think we should have a little party. Not particularly big, big thing. But for another American university, so that's nice. Oh good. And we're also courting another. Courting in the old sense of the word. Yeah. Another American university, which we would love to work with. So, yeah, I guess our marketing work that we've been doing may be starting to pay off or it could be complete chance, who knows. You don't know. That's the annoying thing about it. You ask, how did you find us? And then, inevitably, the answer is, don't look quite no. I can't remember. Every time. Oh, I think I just looked it up on Google. Really? No, you didn't. At least I know where the. Where the. One of these. I know where both of these came from. One of them was. We do a lot of work for the University of Michigan. And this guy runs a journal for another university, which I'm not going to mention yet, so I don't want to jinx it. And he was looking to sort of, how can we improve our website, found the one we did, thought, that's what I want. Right. Got in contact with the people at Michigan that run it. And they said, oh, headscope did it. Go talk to them. So that. Perfect. Yeah, so that's got nothing to do with our marketing. No, I'd realize. No. The other one has, because she came. I basically decided that because we've done work for law schools, that I should be talking to other law schools. Right. And, you know, cold email, basically. But with a good story. Yeah, yeah. And a couple of people came back and she was one of them. Well, that's good. Well, anyway, also, Paul, I was down on your neck of the woods the other day. Yes, you were. Showing off your breakfasts to me. Well, yes, exactly. Paul, you put all I'm going to say is you must go there for breakfast. It was, you know, you were in our, in our sort of aging state, mustn't eat that sort of thing very often. But every now and then, you're in it. I was not younger than you. I want to remind you. Don't, don't tell me with your, with my, with my great age. Yeah. Exactly. Yeah. I don't. You've had, you've had, then, as I think probably discussed, maybe not on this podcast, but on other podcasts, you've had your issues with what, with, with, with your eating. So I was just looking off to you, Paul. That's very kind of you. Thank you very much. Anytime. So anyway, should we, should we talk about topic for today? So I was quite surprised that I haven't already talked about this topic. And I have to triple check to make sure I haven't, because it's something I'm really enthusiastic about. I think I vaguely mentioned the idea of AI-generated personas before. We have talked about that, I think, more than once, Paul. Yeah. But it's kind of a little bit broader than that, with that I want to talk about today. I want to talk about user research and user research repositories, right? Because there are, and yeah, an increasing number of the clients that I work with do use a research. So I've got their own in-house UX teams. And they, they might even have specialist UX researchers, although in a lot of cases, they've just got kind of generalist UX people, if that makes sense. And so the typical pattern from what I see in most organisations is that these people go away and do research normally for a particular project. They then give a PowerPoint presentation to stakeholders who either, not along and then completely ignore everything that was just said, or not along and agree. But in either case, that's then that research done and dusted, right? And yeah, that's great, maybe in the short term, if they, they not along and agree and you go and do something about it. But it does mean that that knowledge just evaporates and you don't get any long-term value out of that. So the solution to that has traditionally been to create some kind of research repository. But the problem is, is well, who actually is using these things, right? We create these repositories of information about all the different testing that we've done. But oftentimes, that feels quite cumbersome to do because if you're not on top of it, say, for example, you decided to create a user research repository today, you might have done years worth of user research that's in PowerPoint presentations and, you know, kind of user interview transcripts and surveys and all of this stuff is all over the place. And bringing it together and putting it into a repository feels like quite a big job. And even if you do that, you're then left with what, a load of documents that have become a dumping ground that are quite hard to kind of navigate and find anything of value if you're working on a specific project. And yeah, there are tools where you can search on them and that kind of stuff, but that's pretty, pretty hard as well. And so most of the time, if anybody is using these repositories at all, it's just other UX professionals, right? The very people who kind of understand the research anyway. And so what's the benefit kind of thing? And everybody else is just ignoring them. And so this can all feel like a massive wasted opportunity. So what I'm kind of interested in is how AI has changed the game around all of that. I know it's I feel like it's all week. I said this last time, all we talk about is AI, but I'm going to talk about something different on this subject, by the way. Oh, are you? Yes. Now old fashioned. Okay. What were you going to talk about? I want to know. Well, there is another way of doing this. Or ensuring that things stay front of mind. Yeah. Outside of people in UX and full writers designers, whatever, is print them out big and stick them on the walls, which is a really, really good thing to do. It is a really good thing to do. Absolutely. I agree with that. I wrote an article basically saying that a couple of months back. The conclusion of which is, because I can remember we looked at some good examples of personas that have got the high fidelity, well-designed personas that for, I think it was male chimps, maybe anywhere. They're the ones that I can do. They were really good. We've done that from many clients over the years. We've ended up getting the ones that you stick on the wall are less and less and less and less detailed. To the point of, actually, all you need is the face. Maybe that's a little bit too stumbling. It's a reminder that they exist. Unless you don't interact with these things very often, you know what each of these characters are talking about personas and you probably know about the user journey maps and what kind of thing as well. All you need is the face. I then went on to this last month to say, but what about having, should we be having faces on personas? I'll save that for another day. What I like about what I think you're going to say, Paul, is it'll happen automatically with what you're going to say? Well, it's not quite. I think there's a slight difference between what you're talking about and what I'm talking about because you're talking about personas, right? And I agree that there is a big advantage of having personas on the wall to remind people totally agree with that. However, a user research repository is a repository of all the user research you've ever done in its raw form, right? Not just personas. Not just personas, yeah. That is, you might argue, well, not everybody needs to know or have access to that information anyway. But the problem comes in that you often then end up repeating research in slightly different forms or research doesn't get done at all. When actually if you did a, created a repository of research, then people could just go to that to find out, you know, what has been done before. But it's kind of overwhelming. And that's where AI begins to change the game, right? First of all, AI is really good for creating a user repository, user research repository, because you can just throw everything at it, you know, every PDF, every old persona, every survey, every bit of data that you've got, anything you can lay your hands on, you can just pump it into AI and it will structure and organize that information into a repository for you, right? Which is incredibly powerful. So where it used to be this huge painful task to create a user research repository, now it's pretty easy to do in comparison. But the best bit is you can then make that content accessible to absolutely everyone, not just a UX specialist, because traditional search on these kinds of repositories require you to know specifically what you're looking for while AI lets you have a conversation with that repository. So once you've got it in place, you can then put a chat interface on it and say, I'm working on this feature in this product, you know, and I'm doing this kind of thing or I'm trying to appeal to this audience or whatever it be, what research of we already done on that that might be relevant or might be useful. So you can be much vager than you can be with a traditional search. So a product manager could ask something like, you know, what do users think about our checkout process and get this kind of synthesized answer that might come across, you know, come from like five different research studies that they never even you existed. And that it means a lot of new research requests can also get answered. Let me explain what I mean by that. So let's imagine somebody goes along and types in, you know, well, what I just said, what do you use to think about our checkout process? If you set up the AI in the repository correctly, it will be intelligent to know and go, do you know what? We've got no research on checkouts at the moment. But here's some best practice for the product manager. But at the same time, I'm going to make a note of the fact that we've got no research on that and send that to the user research and UX team so that they can now go away and do some user research in that area to fill in the gaps in the repository. So it's a great way of creating the initial repository, be enabling people to access it better, but see identified gaps that need to then be filled, you know, with more, you know, more research, is can AI, can you rely on AI to reliably do that though? Yes, you can. Yeah, at least in my question, along with that, using your example, I want some, my experience of AI, if you ask it, something doesn't know, it makes it up. Yeah, that's because that's because it hasn't been trained properly. Right? Okay. So you did say if it's been set up properly. Yeah. So what you would do, you do two levels to it, right? Well, you could even argue three. First of all, you've got your basic instructions that say very explicitly do not make shit up, right? You know, or whatever other requirements that you need from it, of how you want it to respond. Then as part of those instructions, you say before you return the answer to the user, go through this quality gate check, right? Okay. So, you know, have, you know, does it meet these requirements? And then finally, you could get it to even go, now I want you to check back against the original sources that you mentioned to make sure they're all existing and accurate. So if you add in these kind of layers of checks and balances in to make sure that what you get back is reliable. Okay. But even if, let's be super pessimistic and say one in ten things that it answers is incorrect, right? That's still better than having nothing at all. And even that incorrect, if it gives you an incorrect answer, it's still going to give you generic best practice. It's not going to suddenly say, everybody thinks your checkout process is wonderful, right? It'll go, I can at worst case scenarios, it'll go, I can't find any specific research on this. So what I'm going to do is make up some general best practice when it comes to checkout, right? So in this scenario, I agree entirely. Oh, yeah, it depends on what you're using it for. For example, medical advice, I would have a much stricter criteria on its answers. So it depends on what you say money away. Yeah, that kind of thing. Yeah, you said bloody common sense. And you can also caveat these things. And if you use Claude, for example, every time I start Claude, it's got a little line under it that says something like Claude makes shit up sometimes, double check everything. And you know, people yes, are inherently lazy and don't do that. But it's still in this case that it's the consequences of it making stuff up is not the end of it. It's a wall garden. Yeah, yeah, fine. So my, the reason I like this is that it gets, you know, it creates an easy interface for people to start asking questions about the user a bit like those virtual personas. It has huge cost saving benefits before, because before you're commissioning any new research, you could check what you already know. So those two things are really good. Now, I don't think that A, this replaces your big posters on the wall. But I would maybe my eyes see then what you could do is you compare this repository with those digital persona virtual personas I talked about before. So in case you didn't listen to that, that basically what you do is you take the content of your repository, you get AI to analyze it and then create a set of personas based on the data that's in your user research repository. So now you've got a set of personas that and you can create a chat interface on that set of personas so that people can then talk to those personas and get answers for them. Now, does that replace your pictures on the wall? Absolutely not. So what you do, because that's still, it means people have to think, oh, you know, oh yes, I need to use that amazing AI tool, right? Yeah, yeah. So what you do is on those posters, you put a QR code. So you snap the QR code and now you're talking to that user that's on the poster or a virtual facsimile of them, right? So I just think, I think that's bloody amazing as a way of getting the user's voice into an organization, organising the research that's been done, identifying gaps in it. And I just don't understand why more organisations, but I do understand why more organisations aren't doing it is because they UX teams are too small and too busy to do this shit. So hire me to do it. There you go. Ah, see? Oh, that was good, wasn't it? That was subtle. Yeah. So, yeah, that was basically all I wanted to say on that particular subject. Apple the week or Apple the month or whatever it is. So I thought I'd pick. This is Apple of your life, Paul. The Apple of my life, yes it is. But I did pick this one kind of on purpose because it relates to the conversation that we've just had. So the app of the month that I must have done this as a recommendation before because I'm obsessed with it is a tool called Notion. You've probably heard a notion because they're quite big tool and yeah, Marcus is right. I'm utterly obsessed with Notion. I run my entire business on it. If Notion goes down, I go down. I cease to exist. Right? So, but I wanted to talk about it specifically today because I am pretty convinced it is the best platform out there for doing these user research repositories. There are supposed specialist platforms that do it. But I actually, I don't think there is good as using something like Notion. What Notion gives you, it gives you two things. One is it gives you utter flexibility in the sense that you can structure your user research repository. However, you need to specifically for your organisation. And it can also take in pretty much any form of user research that you might have done. So whether it be a survey, whether it be data and analytics, whether it be a diary, study or a user interview or whatever it is or user testing, whatever it is, you can bring it into Notion pretty simply and pretty easily in order to create this repository. And then second, Notion has got a really powerful AI agent built into it that can run through this repository and access everything in it and search on it and that kind of stuff. So I think it is a really good tool. But that said, I am talking to the R&I, R&Li. Yeah, that is right. Is it B? Lifeboats. Lifeboats people. What is it? R&Li. Yeah, for some reason I wanted to put B on the National Lifeboats Institute. Yeah. They are not a client, but I have just got chatting with them. We have been talking about user research repositories. And in their organization they use co-pilot. And they can do the same thing in something like SharePoint for example, where they have all of their repository information in SharePoint and then co-pilot just looks at SharePoint and pulls out what it needs to do. So you can do it in a lot of different systems, but notion I think is the one perhaps on by, well, no, I am biased because it's the one I really know. But it's a good one. It's definitely worth checking out for organising any kind of information like this. So I thought I'd give it a mention. I can't help, but once I'm asking the question so I'm going to ask it, Paul. What if notion did go down? Have you got back up in different places? What if you lose internet access? Well, yeah, I suppose. Yeah. Yeah. I'll just write letters to people. Yeah, exactly. I mean, there are fair point. Yeah, fair point. Yeah. You know, there are things in our businesses that are single points of failure. It's a conversation that we've had. And now I would feel differently if notion was, it's like every time Amazon web services go down half the internet stops working suddenly, doesn't it? Cloudflare has been one of the reasons that we've been doing that twice. Yeah, yeah, yeah. So we do have those kinds of dependencies. Fair enough. Yeah. So that's my kind of attitude. Notion is big enough, right, that I'm confident in them. They're profitable, right? Which I consider another big factor in their favour. I mean, we could end up in a world where we don't want to be using American apps, for example. Yeah, yeah. Some people I know, some people you know, have dumped all of them American affiliations or subscriptions, whatever you want to call them. Well, we're also finding an increasing problem with organisations. I've got several larger organisations I've worked with that can't use American software because of GDPR and that kind of side of things because they're just not compliant. Yeah. If you've got to hold any kind of private data in them, which I have to say on a user research repository, you absolutely shouldn't be doing, right? It should all be anonymised and you don't want to hold people's personal information on that. So it's not relevant in that particular case. But in notion I've got contact information and confidential information held in that and it's like, should I be? Possibly not, although we're having an ongoing nightmare due to consent management at the moment. A lot of our clients were using cookie pro, which is run by, bought by one trust, I think. And the cookie pro product is being discontinued. All that's being changed to be a ten times more expensive. As you can tell, I'm not really part of this particular conversation. So we've been looking at alternatives and you think you find, yeah, that's the one. Then you go, no, it's not accessible. And all of this keeps going down, down this route and then you kind of come back to the conclusion that actually maybe the expensive one is the one you want. Yeah. It's like everything else these days that everything's just getting more and more expensive. Yeah, the other, it is interesting because the cookie notification thing drives me nuts. I mean, I've talked about this before. But it's a waste of absolute waste of time, worst piece of EU legislation ever. Really good intentions, you know, I understand. But all of the ad people that they were trying to target have moved to, the browser sniffing anyway. So it's not like helping at all with the problem that they were trying to solve and just making it an almost pain in the ass for everybody else's sites. And then the other one of that, which I feel a little bit differently about and it's more nuanced is the new legislation around unsuitable images. You know, so if you're going to use poor hub, you now have to prove that you're an adult, etc. Obviously, I, you know, I kind of fully accept that. I think that's very sensible. But the big problem that it's created is that a lot of companies that host imagery, right? In America, I've gone screw it. We're not going to support that anymore. You know, we're not, we're not going to comply with that. And so things like huge numbers of images now just you cannot see this image in your region because it exists on websites absolutely everywhere across the internet, but are being pulled from America who don't want to be liable for this. And so it's just broken big chunks of the web because they'd not thought through the consequences. It's really hard to get this kind of legislation right. I was speaking to somebody from off-com about AI, boyfriend, girlfriend things, you know, these fake boyfriend, girlfriend, and they're thinking about how to legislate around that and they wanted to pick my brain over it. And it's like, it's really hard to get right without having unintended consequences. The one with the imagery, it's kind of like, well, the idea behind the internet was that it shouldn't be, it shouldn't be kind of, I can't find the right word, but you know, close down in any way. It's going to be just this free platform. But equally, of course, as soon as you do that, then you end up, well, you end up with what we've got in the world now. So, which ain't great. So doing something, I think in this particular example, is better than nothing. And if it's broken, a lot of other things. I do. I do. In that case, I agree with you. Yeah. do. Yeah. Could be because at the end of the day all it means is that you know websites that want to operate in the EU are going to have to find an EU-based image provider, right? Yeah. And yes that is a big inconvenience but when you're talking about you know the safety of children I think that's a very you know a price worth paying. Yes. I don't feel like that with the cookie legislation because instantly it became meaningless because you know the companies that were the offenders, the ones that the EU wanted to target just switched technology. That's all they did you know. It's anyway. Let's talk about your interesting reads because you had some good ones. I the first one in particular I had no clue about me neither. This Dan at Headscate found this one which which I sort of was like really anyway so it's I thought to start off I've got a couple of interesting reads, a couple of articles this week. Well I thought I would take a different look at AI because obviously Paul's such a massive fanboy. And maybe we should be a bit careful or maybe that's not really what it is but it's just looking at it from a slightly different angle. And I'm picking on Google as well particularly don't know why but it just happens to be the case. So the first one is a tech juice article entitled Google is quietly rewriting headlines with AI in search results which is like is it and then I started reading the article apparently it's been doings for years anyway. Oh really? Yes. I'm not quite sure how it's been doing it maybe even manually. But I don't know it has been it has been rewriting the titles of certain content in its articles but with the advent of AI with with Gemini I assume in this case it's gone into overdrive and it said I think it said that for let's say four months ago we're just we're we're trialing this and then a month after that it's in use and it's happening every day. Anyway to use an example of the kind of thing that it's changing the one that's come from this article it's it it had an article that was originally titled I used the cheat on everything AI tool and it didn't help me cheat on anything that was shortened to cheat on everything AI tool which obviously completely changes its meaning you go from this is rubbish to here's get your cheat on it on everything AI tool. That's the fundamental problem with AI. Well it's not the problem with AI it's the problem with how people are using AI is you have to have a human in the loop if you take out a human out of the loop if you have no human judgment in it you're going to get shit like that you know what I mean yeah well absolutely but I mean I still I'm still left I'm still a little bit not knowing where this is all coming from because unless unless you've got your your conspiracy theory hat on and I'm not a conspiracist at all but I'm thinking well with that particular example you could say well AI is being pro AI but then I keep coming back to why what what's the point even like say before AI why was Google changing the time it's more well no because they don't really care about traffic that much they don't care whether you click through in fact they prefer you not to all I can assume is that it's meant to sort of help help better summarize so we're we're helping you understand what the content is behind this title because it's but all you're getting is the is the title of the link isn't it yeah yeah there are other articles there are other examples in this article where the the the title has been changed to use words and and subject matter that don't exist in the article at all so again yeah it is really weird but you know what I keep coming to is what's wrong with just keeping the original headline it wouldn't be something it would be something about driving traffic when it is got to be of of driving clickthroughs or try to you're proving engagement or some there's some logic there but I mean yeah I agree it's a it's a solution looking for a problem there's a lot of that with AI at the moment of people using it to solve something that's not really fundamentally a problem no that was so that was a real eye opener for me you can't trust Google to recreate your headlines that you spent I'm it's over I don't care anyway I don't use Google so don't give a shit will I do you do yeah I don't use Google search I should add it's not like I'm against using anything Google but yes you're right the vast majority of people still do so Google bashing part two yes let this one I've got more opinions on this one I do care a little bit more more about you are right right such a click baity title you you noted that I noticed yes this is the title testing suggests Google's AI overviews tell millions of lies per hour yeah which I did a lot that made me smile a bit but this is an ARS technical article and there appears to be something in it and I wrote the words in my notes we all use the AI summary overviews that Google presents us with then I thought well I do so I guess everyone does a lot of people use the AI summaries and I guess what I mean by that is and this is purely an ecdotal on my behalf quite a lot of the time I'll be looking up something I don't know like the golf club that I went to over the weekend down near you I'd look that up and if it told me what the address was I'd believe it I'll be going through to the website and check it but anyway so this article talks about some analysis that was done by the New York Times that suggests that AI overviews are incorrect 10% of the time which is interesting you used that figure earlier earlier to in this podcast Paul they used them an example in the article where when asked for the date on which Bob Marley's former home became a museum AI overview cited three pages two of which didn't discuss the date at all the final one Wikipedia listed two contradictory years and AI overviews confidently and that's a really important word confidently chose the wrong one and there are many many other examples of this Google of course said well you know your research is flawed bladdi bladdi bladdi bladdi bladdi bladdi which it is but it then went on to sort of some right saying well yes of course we can't get it right all the time or AI can't get it right all the time so that goes back to what I was just saying my beef with all of this is if I'm like if I'm the average user people aren't checking they aren't going through and checking responses they're not checking that the address of the golf club club is they don't go through the website anymore so being one being wrong one percent of the time if we're talking about the entirety of the internet is potentially a problem yes and no it's my response to that um first of all I think this isn't a Google specific problem I think this is a problem with large language models that they are predictive and they don't always predict accurately and so yeah they will get things wrong they're also reliant on the data that was you know that is provided to them of which you know you talked about the Wikipedia one had two contradictory dates and so I don't think it's fair to pick on Google other than the fact that I guess they've got a bigger responsibility to be accurate just simply because of the scale of usage but here's where I'm less sympathetic is that we do have a responsibility and we fully accept that we tend to trust humans we tend to trust what's written on the internet we shouldn't do that either right why is AI any different you're oh shocking something on the internet it's not true you know it's like and and so the truth is is you know why should I trust this article I bet there's things in this article that's incorrect I bet there's things that you tell me that are incorrect no that we live in a world where things are inaccurate so it comes down to an individual responsibility of it's a risk-benefit analysis basically that we have to make every single day with every piece of information that we're fed which is comparing the effort with validating whether that thing is true or not against the risk of getting it wrong right So for example, let's take your address, you know, you wouldn't go through and check that that address is real. I would personally check because me getting lost is a bigger pain in the ass than clicking through and checking. I'm not saying you were wrong doing it. I think we're all different, right? You know, it is a bigger pain in the ass than just clicking through on that website and double checking that piece of data. So I don't think it's, everybody makes out it's Google's problem or AI's problem. I just think it's a, we accept it in every other aspect of life that things will be inaccurate. Why do we suddenly hold AI to a higher standard? Other than, and you nailed it earlier, the word confidently. Right? The, I think AI says bang, bang, bang. Yeah. Yeah. It should, all it needs to add is something like it looks like the address of your place is, Dada, Dada, Dada, Dada. Rather than the, the, the, your, the address is, Dada, Dada. And this is where you get to a user experience thing and a conversation interfaces. Yeah. Yeah. Have a prominent link rather than, you know, yeah. Right. Right. But I think we're going to have to maybe agree to disagree a little bit on this one, because even though I, I think you're right in the only way that this is going to get fixed is if people change their behavior. But what is annoying is that you've got an extra step here added because of this. Potentially, extra step, we're in the past. You used to type in East Dorset Golf Club or whatever it was. It's changed its name. It's posh and it's the country club. And the top link would be East Dorset, which I click on. There would be no reading the AI summary. I'd just click on it and go there and find what I was looking for. But now I read the AI summary that confidently tells me something that I go, yeah, that's right. And walk off. But what you're, but you're right, I need to now go, okay, I need to go check that on the website. So I've added an unnecessary step has been added, which is just annoying. Yeah. I didn't know I get that. No, I've fully accept that. I think we do agree over that. I've literally just got, I mean, perhaps I've turned the AI off on something. I never use Google, but I don't get an AI summary. I didn't ask a specific question. It is. It is. Sometimes it appears, sometimes it doesn't. Maybe it doesn't. Maybe a dress. Yeah, a dress. When I added a dress to the end, it did. And then I, but immediately below that is a link to click through. Yeah. But do people click on those links? No, they don't. It's my. Yes. And I would agree with that. So what you're saying is that it's an added level of cognitive effort. Yes. Of, of having to ask yourself, do I trust this summary based on what you're saying about most of what's on the internet is a load of old tribe anyway, then nearly every time it should be more like better had. So what was the. No, I almost, I almost take the opposite approach. No, no, see, I see it the other way. It is. I see it as well. It's just as likely the bloody website hasn't been updated and is wrong. Right. Yeah, maybe. I mean, you can't complete. Obviously, completely trust that. But the organization's website is got to be the place to find their address. It's got to be the definitive secondary place. Yes. Absolutely. No, I accept that. So what's the answer then is, is it to get rid of those entirely? Yes. But they're not going to. People like them because they confidently tell them stuff quickly. But if 10% of it is wrong, then that's a problem. I don't know what the answer is. Paul, I'm not here to give you answers. That's that. I'm just, I'm trying not on duck duck go now. See, no, duck duck go is giving me a search answer as well. So it's giving me an AI answer. See, I like it. I just, yeah, I just said that. I think they're great. And I believe them. And it's like, this is handy. Don't have to fathom out going to websites anymore. It just tells me this stuff. And then this article tells me that anyone in ten or all right, no, no, nine. 19 of 10. Well, one, one out, sorry, you don't know when, one out of 10's wrong. But you see, even that I'm not convinced of that number. It depends on what you're asking. We're just all thinking, Paul. You know, that's based on on the benchmarks associated with the large language model. You see in that particular example of Bob Marley, it wasn't the AI's fault. It correctly quoted Wikipedia. Wikipedia was wrong. Oh, it had, or it had two or three days, and it picked it confidently picked the wrong one. But what does that mean? I mean, to say it confidently picked it. I mean, what is the wording? Let's have a look at what the wording actually is. That's the key is the wording. Yeah, so I mean, it is confident. The door, see it, a gold front country club is located in high near the postcode is. And it's probably right. Well, yeah, I suspect it is. Yeah, I just don't know if you can. It's a lovely part of the world. It just, oh, everybody just whos and winges about any new technology that comes along. And so, you know, there, I am probably. Good for you. It's an education thing. People just need to realize that AI gets it wrong sometimes. It's like going, why doesn't notion make my breakfast for me? Well, don't, don't think that's quite what that is. That is. I know. I know. It wouldn't be good if AI did do that. Why doesn't AI paint my living room for me? Yeah, wouldn't that be bloody fantastic? To be serious, my, the serious point there is that AI is claiming to make my breakfast and paint your living room, but is not doing either. That's the fundamental issue, isn't it? It's the claim, it's the claiming with confidence. I've they got any kind of disclaimer. All AI responses may include mistakes written in daily text. So I guess it, you know, they're covering, although that doesn't appear interestingly, that disclaimer doesn't appear until you hit the show more button. Yeah, quite a lot of the time you get what you want or you get what you think you want right at the top. Yeah, which is where the message needs to be. So, you know, it's not. It does have the links are in the top. Yeah, but it could at least say, see this, it's a basically what we're saying is this is a user interface problem. Well, how kind of I think it should just be dumped and just go back to clicking through to the what's more likely to be the definitive source. What's wrong with that? Nothing. Yeah, but they're not doing that because. No, they're not going to, but. No, but all right, within the constraints of the business model that they're operating under, which is that they want you to stay on the website so they could show ads, right? We can wish all they want to. Oh, let's strip out. I wish that they'd get rid of all of the ads. I wish they'd go back to the clean interface they used to have when they were making no fucking money. You know, that's a it's like a ridiculous argument. But so many things been ruined by this. I use the monzo banking app, which is still pretty good. Okay, why don't you pay for search then? But I'm paying for search. Paul, don't be ridiculous. There you go. That's the problem. So therefore you are the product. It should be subsidised. It should be like the NHS. Right. It's tax, which is taxed on it. Well, to be honest, I mean, that's not a bad idea. Everyone uses it all the time. So what should we just be taxed? And then it's not the American way clean. Yeah, no, it's certainly not the American way. No, God, no. You're quite an interesting idea. When a farmer comes in, do you want, do you want free healthcare? Oh my God, no. We're all like, what? Why wouldn't you want that? We need BBC search. That's what we need. Yeah, funded by a licensed pay of fee. BBC make it happen. There we go. Markus, we went off at a complete tangent then, didn't we? Which is a good job because the show is running short time wise. So now I got to try. Yes, I've got to talk. I have a joke. Paul Edmunds shared this one. And it made me giggle. So I'm entering. Sorry, I'm entering the annual give helium to a sheep contest again. And I'm a bit nervous. Last year, the bar was very high. I knew it'd be something to do with bar. And I was trying to think what would a higher pitched bar be. But yeah, no, that was better. That was good. Alright, I approve they're getting better recently markers. I think you're you're Diling in on the things that amused me. That's what it is. (laughs) All right, that's great. Thank you very much everybody for joining us. I'll be interested if any of you do listen to this. It hits up on X or LinkedIn or whatever to see what you think about some of these subjects because I really am be interested in what other people think. It's an interesting one. All right, thank you very much Marcus. Thank you for listening everybody and we will talk to you again next month. Goodbye. Bye. (upbeat music) (upbeat music)