AI Glossary Series: Retrieval Augmented Generation (RAG) [AI Today Podcast]

The AI Today podcast, produced by Cogmelidica, cuts through the hype and noise to identify what is really happening now in the world of artificial intelligence. Learn about emerging AI trends, technologies, and use cases from Cogmelidica analysts and guest experts. Hello and welcome to the AI Today podcast. I'm your host Kathleen Mulch. I'm your host Ron Schmelzer and one of the great things about our AI Today podcast is that many of you have reached out to us and say you're on your journey of implementing AI. Some of you are very early and sterile learning concepts and trying to understand many of the basic terms and terminology of AI, which is perfectly fine because I think a lot of people there. Some of you are sort of farther along in your journey. You're actually building AI systems. You're using one or more of those seven patterns of AI and some of you are even further along where you're building solutions for yourself, for your customers, maybe solution providers. And many of you are CPMI certified. You know that just building AI, that's the easy part. Well, that's the part that can get you in the most trouble. Honestly, it's doing it right. That's the hard part and you realize that the issues around data, data cleansing, data preparation, even understanding what business problem to solve issues of model evaluation and operationalization, there's a lot more than just making AI work to make your AI project successful. And so on that theme, we kind of go through each of these ideas. And in particular, we like to focus every once in a while back on the main idea of education because so many of you are on this path to understanding AI before you can apply it to your problems or those of your customers. Exactly. But we wanted to spend some time today talking about retrieval, augmented generation, also referred to as RAAG. We've had podcasts on large language models and generative AI, but we really wanted to spend some time today talking about RAAG, letting you know what it is and why people are suddenly talking about this, explain a little bit about how it works, and then we also go over some of the benefits and some of the challenges with RAAG. And this is part of our CPMAI training. So it's going to be an excerpt what you're going to hear next. And so if you're interested in really digging deep into this, I encourage you to listen to this podcast a few times and also sign up for CPMAI. For our listeners, we have our free intro to CPMAI course. You can go to AIToday.live/CPMAI, and I'll link to that in the show notes as well. And you can also sign up for CPMAI. But now you're going to hear an excerpt from our training about RAAG. You might have heard the term, or have seen the term, retrieval augmented generation, in the context of large language models. And the reason why this topic comes up is because there are a number of challenges with these LLMs and these GPT systems that are just so powerful. I mean, they are very powerful because general-purpose large language models, LLMs, are trained on petabytes, exabytes of internet data. And therefore, they're really good at general-purpose natural language processing tasks. For example, generating text or code or things that represent what you're interested in getting from the prompt. They're also really good at classifying text, doing an analysis of the input you provide. Some input it can provide some good analysis, and question answering in a bunch of general natural language processing tasks. However, as we know, large language models suffer from a few problems. One of these problems is that they can only perform these natural language tasks based on its training data. It's got a lot of it, but it's general training data. And so it can answer general questions. It can do some general classification analysis and question answering based on all of that training data. And that training data might be old. It might be irrelevant. It might involve content that is not specific to what you are interested in or even know about your particular domain or industry or specific application. And that means that it doesn't really have that knowledge of that domain specific or proprietary. You may have information that's not available on the internet. That's not generally available or private information. And you want to use the power of large language models to do all of that querying, that analysis, that generation, that's really powerful. But you want to limit and restrict it to your specific data. So really, the big question is, how can we use this power of large language models with its ability to understand and generate content and all that text information. But we want to use that power for our own data sets to focus the question answering on our own information, focus the content on things that are relevant to our data, and of course, constrain the responses that it doesn't generate stuff that is irrelevant, hallucinating, incorrect, or just out of the bounds of what we are looking to get. So retrieval augment to generation is actually a fairly simple idea in that we can use the power of LLMs, but constrain it to only look at the information that we're telling it. And I basically, if you will, injecting into the prompt that we ask the LLM system, the specific context, the information we are asking the LLM system to provide. So retrieval augment to generation, if you will, kind of grounds the LLM system with very specific data sources and context that we provide the LLM in the context of our prompt, the thing that we're asking. There's really three steps in retrieval augment to generation. The first one is getting data and original documents from a source, and we basically put that in some sort of source database. And then what the system will do is when you ask a prompt, it will basically augment the prompt that the user asks with the context from those data sources that have been stored. And then it presents that to the LLM. And then basically the third step is we get that output back from the LLM using that augmented prompt with the added context from our data sources. So we did talk a little bit about this idea of retrieval augment to generation in the context of things like Leng chain, we talked about taking LLM and combining it with web base data, other data, and maybe modifying the way that these LLM systems behave. This idea of retrieval augment to generation originated at Facebook AI research meta, but it's become widely popular because it's such a simple idea and you can implement it in a wide variety of ways. So how does retrieval augment to generation work? Well the first is that let's say we're starting with some documents and data that we have. You have to start with some context, something that you have. So it could be PDF documents, it could be websites with FAQs, it could be text, it could be a variety of sources. What we're going to do is we're going to take that information, those documents, we're going to chop them up, we're going to chunk them up into pieces. And we're going to load them into a searchable database. And the searchable database is important, it's not just an unstructured data store that just has documents in there. Now what we're going to do is we're going to vectorize, we're going to take this text information, we're going to create these called embeddings. If you remember from our earlier podcasts and earlier discussions and our training on that whole idea of vectors and vector databases and embeddings is that and tokens is that machines don't understand words, they only understand these numerical concepts. And think about vectors as we can relate one idea or one concept of information to another based on kind of where it is in this multi-dimensional space. We take these numbers and we embed them in space as we load them into our data store. And so this vectorized information, we usually store it in something like a vector database. And so when a user enters a query, so they type in some search, so like, you know, give me some information about some repair manual, right? I want to ask a question about the repair manual. Of course, the internet doesn't know about repair manual. But I first loaded my repair manual into the database, I chunked it up, I vectorized it, I got the embedding. So now all the concepts of the repair manual are now in the vector database. When the user enters a query, it doesn't go right to the LLAM. It first goes to our retrieval augmented generation system. What it does is it vectorizes your query too. And it looks up the relevant and tries to find the most relevant information searching this vector database. And what it does is then the next step is it takes the query that you asked and it combines it with the source data, the best match from the original data sources. And it creates a new prompt. And that prompt, which is the combination of the relevant context and the query, then gets sent to the LLAM system, could be any GPT or whatever system you use. And then the responses are provided back by basically constraining, making sure that we tell when we write the prompt that we restricted to only answer the question based on the context that was provided in the search. So basically the user enters a query, the rack system then queries the searchable database, based on what the user wants, which has also been vectorized, the content is then retrieved from the data source. So we find the content and then the content and the query are combined together as context that we use with a predefined prompt that we've created in our system. So it's not a freeform prompt. We created a prompt. We send this to the large language model and then the response that we get back is presented to the user. And hopefully what we've gotten is an LLM system, answering questions in our data, because we're telling the LLM to only respond to the context that we've provided by basically searching against our own data source, which we don't have to provide all that data to the LLM. So we can maintain that privately in our own database. And all the LLM is seeing is the context that we provide combined with the user's question. So I don't know if that seems simpler or complicated, but it's actually really very simple. And you can actually implement a rag in just five lines of Python code, where you basically take your data, you load it into the database, then you basically query the database with your question, then you combine that with the prompt, and you send that to the LLM and get it back. And so there's lots of benefits to these retrieval augmented generation systems. One of the benefits is that because rag systems use the latest, your most relevant data, rag systems can provide contextually accurate, relevant and detailed responses that answer that user's question directly, using that information source, which is hopefully as up to date as possible. As a result, these retrieval augmented generation systems are more adaptable. As we change the data, the responses can change. They're also more capable, performing well when we have very domain specific applications, things that require a lot of knowledge of a very particular system. You can make it work that way. And also rag systems, these retrieval augmented generation systems really can enhance the relevance of what these LLM systems are providing and improve the quality of output in general from our language models. The other thing, as I was mentioning, it's a fairly simple concept, a fairly simple idea, and therefore they're easier to implement than trying to take an LLM and say fine-tune it, because the alternative to a rag system when we have proprietary data is to retrain or fine-tune a big LLM by basically giving it new training data and then going through the training process. And yes, that would make the LLM system much more responsive to the data and perhaps it might even perform even better than this case, because prompts are always never perfect, right? But it just takes time, it takes effort. The constant retraining process is computationally and intensive, it requires data. We have to, it's a little more technically sophisticated as well. So retrieval augmented generation is easier to implement and faster to produce than fine-tuning LLMs. And it's also more reliable than simply just doing prompt engineering. If I'm trying to create some sophisticated prompt, I have to figure out the data in the context myself. And the thing is if we have a database that can look up among all of our different sources, what the right piece of content is that we can use in our context window, then it'll do better than just trying to be very, very creative with prompt engineering. So there are challenges. Rags is not a perfect solution here, retrieval augmented generation. Systems they can still hallucinate. I mean, we're still using an LLM system. After all, these LLMs have been trained on all of the internet text. And if a question goes slightly out of bounds with what our retrieval augmented generation system can provide, then the system will generate a response. And that response might be completely made up. However, we can limit hallucinations with our retrieval augmented generation system by making sure that the prompts that we use are very carefully engineered to reject responses that aren't supported by what's provided in that context. And of course, that context comes from the documents and data that we have in our data store. So the other thing to know is that this context window-- because we're basically-- that's all the LLM is seeing is what the user wants from the query, combined with our sophisticated prompt that we're going to ask the LLM, combined with the context that we're going to tell the LLM to only answer that question within the confines of that context. And if we have a lot of data that we're giving the LLM, we might be limited by what's called the context window. There's just a certain amount of text and information tokens that we can provide to an LLM that it will be able to respond to. And if we go beyond that, it'll just kind of ignore everything else. So if we provide a very long document, lots of data sources in our context window by embedding this text or maybe URL with lots of information, it's only going to work until we hit the limit of that context window. And then it'll stop reading it. And then we can have hallucinations. Some large language models have larger context windows than others. So you have to be very picky when you're building a Rags system, kind of which LLM you're basing it on and how sensitive it is to that context window. There's a little other advice we have on reducing hallucinations, even in a retrieval augmented generation system. We can engineer the prompt in such a way to say, only pay attention to a specific date or a specific ID or a specific number or a particular URL and say, don't, if your information comes from some other place, say, ignore it, don't respond or say, you can't respond if you can't find it. And even in these cases, we can't completely and fully eliminate hallucinations. And by hallucinations, as you know, we mean answers to a question that are completely made up, that there have no authority in any sort of source. And they come across as confident. So people might think that the system is saying, oh, yes, yes, you could file this claim at this place or you can open the stores open on these hours when the store is not open at those hours. And that claim cannot be processed. Clearly, if people are depending on this information, and you can't tolerate a high rate of hallucinations, you have to be very careful with the use of these systems. And as mentioned, there are alternatives. We don't have to use retrieval augmented generation systems. We can build a fine tuned LLM that's based entirely on our training data, or mostly on our training data, and get a little more control over it. That's especially if we need a high level of adaptation to the way that the model needs to behave. And we don't necessarily need to use a lot of external knowledge. But we could also just be really, really, really careful in how we do prime engineering. That's another approach, although that's clearly sort of going to be less sophisticated than the retrieval augmented generation approach. Finally, we can sort of combine the two. We can do some retrieval augmented generation and use some fine tuning LLMs. If we combine these approaches, we can have a very sophisticated model. We might say fine-tuned something, so it can handle a wide range of queries, perhaps, on a large set. And then we can use a retrieval augmented generation to even further enhance the fine-tuned system with information that might be changing in a more frequent basis. Because as we know, with fine-tuning, it just takes a while. And we may now want to be fine-tuning every week or every month, and they're both the resource cost, and the data cost, and the sophistication. But if we have some sort of fine-tuned system, we update periodically, but then we have a retrieval augmented generation system, we update much more frequently. The combination might be really powerful. And so people have been looking at this as a great approach and to solve many of these issues of LLMs and make them highly relevant in today's world. We hope that you enjoyed that excerpt from our training on RIG, and now you know all about retrieval augmented generation. If you enjoy listening to this podcast, I encourage you to subscribe to AI today if you haven't done so already. So you can get notified of all of our upcoming episodes. We have some great interviews lined up. We also have many more podcasts. In 2024, we really want to focus on education and really talking about relevant topics, continuing to showcase our thought leadership. And so that's what we are planning on doing. And we have many podcasts in the works that will do just that. So subscribe to AI today if you haven't done so already to get notified of all of those episodes. We also encourage you to reach out to us. You can rate us on iTunes, Google, Spotify, or your favorite podcast platform. We love to hear from our listeners. You can email us as well, [email protected] so that we can learn here about the episodes that you've enjoyed, additional topics that you'd like us to dig deeper into. We really do like to hear from our listeners. And also, as I had mentioned earlier, we will link to CPM AI in the show notes so that you can sign up for the certification or the training and certification. Or you can also sign up for our free intro to CPM AI. Many of our AI today listeners have done so. And it's a great introduction into CPM AI. So I definitely encourage you to sign up. You can go to AIToday.live/CPM AI. Like this episode, I'm going to hear more with hundreds of episodes and over 3 million downloads. Check out more AIToday podcasts at AIToday.live. Make sure to subscribe to AI Today if you haven't already on Apple podcasts, Spotify, Stitcher, Google, Amazon, or your favorite podcast platform. Want to dive deeper and get resources to drive your AI efforts further? We've put together a carefully curated collection of resources and tools. Handcrafted for you are listeners. To expand your knowledge, dive deeper into the world of AI, and provide you with the essential resources you need. Check it out at AIToday.live/list. This sound recording and its contents are copyrighted by Cognolidica, all rights reserved, music by Matsu Grabas. As always, thanks for listening to AI Today, and we'll catch you at the next podcast.