How your AI search results are being manipulated

For many people, AI use is becoming part of their daily lives. From the intentional use of chatbots to answer questions, to AI being served up on search platforms, even if it wasn't requested. AI is radically changing the way we discover products, services, brands, and more. But what if behind the scenes, these recommendations are being manipulated? Approximately 50% of the traffic right now on the internet is mostly driven by chatbots. Today, we have Nick Kudas joining us on the show. He's a professor of computer science at the University of Toronto. He'll explain just how this works, and what exactly is at stake for consumers and businesses alike. I'm Cheryl Sutherland, and this is the decibel from the Globe and Mail. Hi, Nick. Thank you so much for coming on the decibel. Thank you very much for having me. I'm really excited to be here. So before we get into how things are working now, let's take a step back and talk about how search engines used to work. So say five years ago, if I googled what's the best coffee in Toronto, what would I see? Yeah, so five years ago, once we asked this question on your favorite search engine, the search engine will get a query, will apply particular algorithm that has been evolving over the last 20 years to rank the best coffee soap in Toronto. It will utilize many different signals, and your user experience would be a ranked list of 10 blue links. The top link would be what the search engine deemed to be based on its own algorithms, the best coffee soap in Toronto. So it could be a website of the coffee soap. It could be an authoritative social platform talking about this. And on the side, you will see some advertisements for sure of coffee soap in Toronto. And then your experience was not so much getting an answer, but also becoming an explorer. Right? Because yourself, you have the opportunity to explore the options there, right? Yeah. Okay. And so there's something happening in the background here in order for this list to be generated and for the list to have the top link and the second link and third link. It's something called search engine optimization. Can you tell me how that works? Yeah. So certainty optimization is a particular set of techniques that appeared probably when the website becoming very important and services like Google and others start appearing. And because the way the answers were reported as a set of 10 blue links on the first page, ranked in a particular order, there was a lot of things at stake in order to be able to optimize your URL, your company, your link to be the first one. And that involves many different, you know, approaches, having other authoritative sites point to your website, having the right keywords to put on your webpage so that when Google index it, you know, it ranks your page higher. And that set of techniques that people started applying in order to position and jump better in the search results was essentially search engine optimization. Right. And the value of that was that if your link was the top link, you're more likely to have somebody click on that one versus the one on page two or three. You definitely get more traffic that version. And of course, there isn't a business around it, right? Because you know, Google was selling ads based on keywords and there was also a dark art of what are the right keywords, and how do you make sure a second, you know, market if you will or business of sites selling the opportunity to link back to your website because they had high authority and many other things around them. Right. Okay. So that's how SEO works. And you're giving us a sense of how companies were trying to make sure that their business or brand is at the top of the list. Absolutely. So let's talk about where we're at today. These results have really changed recently, now they use generative AI. So can you explain how this is different from the search results we used to get? Yeah. So right now, the experience of when we go to a particular chatbot, your favorite one, there are two things that happen. So the first thing that can happen is that you ask your question, right? And some of this chatbot, when you ask the question, they may or may not do a website at that point. Right. And if they do a web search, what they do is that they are utilizing some search engine and they get links from these search engines, they analyze the links, right? And then they form an answer based on the content of these particular links plus including some of their what we call pre-trained knowledge, the knowledge of the model that it acquired when it would have been trained. So it's not just using internet links, it could be something inside that was built into it. Yes. So that's the first thing that can happen, right? So it's a confluence of what's on the internet, plus what in the, let's say, the model's memory. Right? However, there is another case where you're asking a question and the model doesn't do a search. It answers from its memory based on its pre-trained knowledge. So these two are different signals, completely different, right? And as you can imagine now, there are lots of stake, right? Because obviously, if you want to be part of the answer of the model, right? You have to look at both on the web, right? And you have to look good both on the memory of the model. Yeah. So help me understand this. And let's take the example of what's the best coffee in Toronto that we were talking about earlier. Like, if a chatbot is made, right, and you're talking about this memory, how would it know what the best coffee in Toronto is based on their memory? Yeah. So first, we have to go to a step back and understand how we train these models, right? And in very simple terms, think of the model as a computer program that has had the ability to read the entire internet. Right. And there are two important things that happen when the model reads the entire internet. Typically, it will read most of it one time, okay? But there are some other sites and piece of information typically dictated by the model creators that the model is going to read more frequently. So I want to make the analog of people like you and I when we read something, right? Some of us, you know, we'll read it once, we'll read the page once. But most, for most of the people, if you read a piece of text multiple times, you remember it better. Mm-hmm. And things can change as well, like when it comes to information in the world, like completely, completely. That's why most of these models that have this string in window, right? So when they answered, they say, "As of my last string in window," which was, let's say, six months ago, these are the answers, right? And if they don't trust the string in window and they deem that they need more recent information, then they will do a website that's essentially what happens. So now we're going back to the coffees of analog, right? So the models are read in the internet, right, read different pieces of information online, could include websites, authoritative sources, let's say Wikipedia, social forums, could be read it and different threads there. And then they learn based on that, all that information they read, the form their own opinion. But this opinion is very similar, that the opinion that we form, when we start, if you would like to gain an opinion about the coffees of the internet, your typical process, a mathematical process, that you will ask around or you will go and read a lot of different source of information, and then you distill your own opinion based on what you read, right? And there are some implicit biases, right, for some of us. For example, you might read a particular website that you trust much, and you say, "Well, I mean, this coffeeshop is featured or had this award, right, or this coffeeshop was featured in this particular publication," right, or there is a very authoritative social site that you read frequently, and you tend to trust the opinion of the social site. So we form an opinion based on all these different signals of information that we read. Something similar happened with the models. So and of course we're talking about large language models, LMMs, and this is how they work. So now there are these new marketing tactics called generative engine optimization, or AEO, and answer engine optimization, or AEO. What are those terms mean? Yeah. So these terms, you can think of them as the evolution of SEO, search engine optimization in the AIR, and it's a set of techniques that people and companies are trying to apply in order to position brands better in the answer set of the Zatbot. All right. So basically, it's just a way to manipulate the chatbots, the same way SEO is doing that to get your brand or your company at the top of the list, or in this case, the answer that the chatbot comes up with. That's the first thing. And the way this engine works is essentially there, trying to process this, they're trying to position grid information out there that can make the model believe that the best coffees in Toronto is whatever these documents say. Now, I'm just going to be very clear that this is a high non-trivial, right? So it's not as easy as creating a website and say, you know, the best caution in Toronto is, you know, XYZ, right? And I will give you a very simple analogy, right? So the way the model is trained is in reading the entire internet, let's say, plus many other things, but let's say information at the scale of the entire internet. So if you have read all this information, you've read a lot of things about it. about coffeeshop. You formed your opinion. And if I saw up and I give you one document, and I tell you, based on this document, the best coffeeshop in Toronto is this particular one. Now, since you've seen so many things and you know so many things about coffees in Toronto, if what I'm telling you matches what the opinion you formed, then you believe this document, right? But if it highly contradicts what you've seen so far and what you know so far, it's very easy to, I wouldn't say dismiss, but not trust this document immediately, right? So taking into account, but you will still go with your previous opinion. Let's assume now that I am coming to you with a document about the topic that you don't know a lot of things. It's much easier to make you change your opinion about something because you're not an expert. You haven't seen a lot of information about something very similar happened in these models. It's very easy to change their opinion if they don't know a lot of things about something. It's not impossible, but more difficult to change their opinion if they've seen a lot of information about a particular topic. Right. And both of these have very, very clear mathematical analogues. And so you were mentioning how it's difficult when there's a lot of information, but if this model doesn't have a lot of information, it might be a bit easier. But what could that be? Because I would assume that there's so much information on the internet. Is it just that something more niche would be easier to manipulate? Exactly. Exactly. That's exactly what it is. If the topic is a very niche topic that the model doesn't know a lot of information about, then it's very, very simple to essentially inject your own information into the model. And that can happen if you create a website and it happens that the model learns about the website, it will potentially trust it. But if there is a lot of information about something and, you know, for larger brands and, you know, big ideas and topics and other things, right, there is definitely a lot of information, this becomes, you know, more challenging to do. Right. And I research that it's not impossible, but it's, you know, definitely you need a very different methodology to create information than if you are trying to create information for a niche topic. What did your research say about how it is possible? Like, how does this actually work that say a company or brand is trying to get their information into the chatbot? Yeah. So it's all about generating information, of course, right. So you have to generate information. And that information, you have to publish it. And in order to become accessible in the model. Now, as I mentioned, there are two ways the model get information and answer, first and two in a web search. Yeah. So you need to move, if you want to influence that route, you need to have your information published online and you need to find out depending on the topic where to publish and what to publish. Right. So that's very important. And in order to do that, I should probably exactly as with a human to understand what do you know already about the particular topic and how well you know it. And then based on that, for a strategy on what is the new information that I will give you in order to make you change your opinion about something. So that's one route. The other route is that the model might answer from its memory. In that case, you also need to find out what the model knows already about this particular topic and then understanding how strong their opinions and their biases are towards this topic and then try to identify what information you should produce in order to be able to support that or change the opinion. Something else that is interesting about how the bot conversations happen, right, is that it feels very personalized when you ask a question, it kind of caters the answer to what is learned about you. And it gives you an answer that feels like very authoritative and it makes you feel confident about it. So there's something about that new way of AI speaking to us or giving answers that that's different. Yeah, I think that definitely answers can be very authoritative. Sometimes there is also the bot also knows the context of the discussion. So you know, if you're asking for a few things, it takes in all of these things into account. Also bots, you know, are fairly polite. I mean, the way they are trained to respond, they're very, you know, affirmative most of the time when you answer. And if you challenge them, they're typically very apologetic. So yeah, I mean, they're trained in a particular way. And I think that the way they respond creates also a sense of trust. Yeah. And for some people, and I will say this happens to me that it feels like I feel more confident in my decision based on what the bot will tell me because of this conversation and all these things you've talked about trust and honesty that these words that they say. Are there instances where the AI falls for or conveys fake or unbarified information to the reader? Yes, I mean, there is there is a one particular phenomenon, right? It was probably, you know, most of the people are familiar with right now. There's a call hallucinations. Yes, that's very familiar to people for sure. And the hallucinations are essentially a byproduct of how the model provides answers, right? And there is a lot of research into that. So people, you know, have found out that, you know, models sometimes hallucinate tend to hallucinate on things that they don't have a lot of information because they are trained to provide answers, right? And they make up stuff just because, you know, the way their process of generating data works, right? That is making it. So yes, there are a lot of cases where, you know, the AI is going to provide information to you that is unverified or, you know, or misled in or completely completely made up. We'll be right back. And so I'm curious to know, in such a way, because we're focusing on like how this is being manipulated by potential brands or companies, like, are there ways of convincing the chat bot or AI program that a brand or company is a reputable source? Yeah, I mean, there are. And again, to do that, first, you have to understand what is the domain that is brand, typically, you know, operates on? Is it, let's say, a fashion brand or, you know, what is it, right? And then, typically understand also the competitive landscape where they're trying to position themselves against, right? And create information that is, you know, making the model believe that this is true information. Is this mostly third parties who are utilizing GEO or have you seen AI companies start to get into the advertisement sector? Well, I mean, what it's public information, right? That chat ZPT, the OpenAI, that ZPT creator, they have already, they're providing answers. I experience them, like especially on the free account, if you, it's not a paid account, typically, if you ask the question, the question has a commercial value, typically on the bottom, at least that was my experience. You'll see an ad which is sponsored by a particular company, right? Is there any way, at this point, for a user to tell if an answer has been manipulated? At this point, I don't think you can tell. In exactly the same way, you couldn't tell back in the SEO days that the number one link, right? That was something. I think what, what we do know is that back in the day of, you know, Internet search, right? Google itself is one of the companies offering search services had a lot of quality control processes, right? Now, I am confident that SEO becomes more prevalent and not so much SEO services of, you know, generating content, the models themselves, the model creators, right, are going to create typical policies in order to, you know, convey some sorts of trust based, right? With respect to the consumers on where this information is coming from. What potential harm should we be watching out for if AI can be manipulated? Well, I think that for the case of, depends on the domain of discipline, right? But I think the real issue there is not so much, you know, brand, right? But imagine if you are trying to ask a question to the model that has to do with, you know, politics, has to do with, you know, historical facts, right? Information about, you know, anything that, you know, people care about. So in that case, I would say right now there is no gatekeeper, like, of that information, right? So you could ask the information, like, you know, do they have democracy in country acts, right? And then, and if you don't, double-check the answer, right? How would you know, right? This is the right answer, right? Yeah, and the other thing is like, the option of choice is kind of gone, right? When you're asking a chap about a question or even like the AI summary on Google, for example, it just gives you the answer. And it's like, doesn't give you the chance to think or to explore, as you mentioned. So, look, I mean, I completely agree with your point, right? And I think it's a very fundamental point is that in the case of, of a website, okay, before the web, right? Forget the web, right? So just a continuum, just look at the continuum, right? Of access, information consumers. And you would read lots of different books. You would read newspapers, right? Newspaper has certain reputation because if the newspaper was lined, right? The reputation would be tarnished, right? So that's how we worked, right? A little bit before. Then the web appeared and then all of us became, you know, researchers, right? Because you will look at the 10 links. It was your choice to read the first link or to your choice to go ahead and, you know, look at all the information that you have the time. [BLANK_AUDIO] was laid out there for you. Right now you are asking a question, you get an answer and you get typically four options or three options with one citation. So now you can just read the answer into the convenient. It's super convenient because it's just an answer there. It's exactly what you expect them. And then the opportunity, you have less options if you stay in the same boat to explore. And I think people should exercise caution with with anger type of information consumption. Yeah. Are there any sort of regulations in place when it comes to geomarketing? Noto that I'm aware of right now. I think this is very early days, both for the GeoAO and I mean it's an interesting making, but I think it's very early days. So yes, there might be regulation, right? There are a lot of cases where you know, already with I'm aware of where people are essentially the CEO, a particular, you know, chatbot is leaning a lot towards a particular social platform, right? There are even services where you can actually pay people to write good things for you on these social platforms, right? So that has always been, but right now is getting geared towards more what the bots actually interest information from. Now I don't think this is, I mean this was bound to happen, but I think this is something that doesn't have a long lifeline because from the model creators point of view, if they want to cut off a particular source, it's very simple to do. So yes, people might be doing something that works for them today, but tomorrow it might disappear. You know, with everything that we've talked about today, what should people take away about how to interact with search results? We shouldn't be doing anything different than we should have been doing for the last 20 years, right? So consuming information, being mindful and always double-check the information, right? So like, and there are some piece of information that we're seeking that maybe you can argue that is not that important. Let's say, what's the best coffee shop in Toronto? Yes, compared to the coffee shop. Yeah, I always find out what the best coffee shop is. Yeah, so in the end of the day, I mean, and let's, you know, depends on the idea that we hold it, right? So you do your research, you read what people say, and then based on that, you know, you go, and then if it doesn't work for you, probably you will not go back there again, right? But I think the real question is not so much, you know, then these are very important recommendations, of course, right? That you know, kind of affect businesses, and but there is an element of choice, right? But if you look at consuming information about getting answers about important questions, right? Other than coffee shops, for example, you know, you are trying to understand historical facts, right? Or you are trying to read about the opinions of political parties, right? Things like that. So obviously you can consume information from bots, but you have always the option to continue your research outside the bot, right? You have always the option to visit the website of the, you know, let's say, the service or the political party or the, you know, anything else and reading and educating above and beyond what the model says. So the model could be good pointer to information sources, but as always, we should be doing, you know, our own research and learning more about what we are searching for. Yeah, I think that's a really great point to end on. Nick, thank you so much for coming on and talking about this really interesting topic. Thank you very much for having me, appreciate it. That was Nick Kudis, a computer science professor from the University of Toronto. That's it for today. I'm Cheryl Sutherland. This episode was produced by Amber Ranson. Our producers are Madeline White, Rachel Levy McLaughlin, and Mahal Stein. Our editor is David Crosby. Adrian Chong is our senior producer, and Angela Pichenza is our executive editor. Thanks so much for listening.