Building a Production RAG Pipeline — From Documents to Answers

Why Your 'Chat with Documents' AI Hallucinates Garbage Picture this. You are sitting at your desk. You know you have just been granted access to your company's brand new, highly touted millions of dollars in the making. Chat with your Documents AI system. Speaker 2 Oh yeah, the magic bullet. Speaker 1 Right, the magic bullet, the promise, is just massive. The internal marketing emails have been relentless about it. All the knowledge of your entire organization. So like every wiki, every financial report, every Slack thread, all perfectly synthesized and available right at your fingertips. Speaker 2 Sounds amazing on paper. Speaker 1 It does. So you type in a simple question, something you you actually need to know for a meeting in 10 minutes. You hit enter. You watch that little loading indicator spin. You feel this brief moment of like technological optimism. And then, and then the AI confidently, elegantly and beautifully delivers you utter unusable garbage. Speaker 2 Yeah, it is. It's the universal corporate experience at this point, the beautifully articulated, grammatically slawless hallucination. And the immediate psychological reaction is really fascinating because we inherently trust the interface. I mean, we look at that blinking cursor, we read the confident prose, and we assume the brain behind the text has simply failed. Speaker 1 That is exactly the universal reaction. Everyone immediately points the finger at the large language model. They blame the AI. You hear it in the hallways like oh the model is stupid or we just need to pay for a newer scarter model and then the engineering teams. And hey, if you are listening to this, you might have been in this exact agonizing position They waste weeks trying to prompt engineer their way out of the problem. Speaker 2 Prompt engineering desperation. Speaker 1 It gets so desperate they Add all these almost pathetic clauses to the prompt. Like, uh, you are a highly intelligent financial assistant. Think step by step. Do not hallucinate under penalty of death. Speaker 2 I will tip you $200 if you just answer this correctly. Speaker 1 Yes, exactly. And none of it moves the needle. Speaker 2 Well, none of it moves the needle because the LLM was never the problem to begin with. We have to completely reframe how we view this whole architecture. In the scenario you just described, the LLM is merely the brain. It is a reasoning engine. Speaker 1 OK, just the brain. Speaker 2 Right, but that pristine highly capable brain is currently being fed absolute nonsense by a completely broken mis wired nervous system. I mean, it can only reason over the information you physically hand to it in the context window. Speaker 1 A broken nervous system. So if you hand it paragraphs that are cut off mid sentence or or tables where the columns have been scrambled into just a word soup, it doesn't matter if it's the meat. Advanced neural network on the planet. Speaker 2 Precisely. If we look at the raw data from massive production systems, the statistics are just staggering. 80% of the quality of an AI's final answer comes down entirely to the retrieval mechanism. Speaker 1 Right, 80%. Speaker 2 80% Yeah, you can deploy a state-of-the-art model, but if you retrieve the wrong pieces of text from your database, you get wrong answers. Garbage in, garbage out. The LMM is just faithfully answering based on the garbage data it was handed. Speaker 1 Wow, so we are constantly blaming the AI for what is at its core, a deeply unsexy search problem. Speaker 2 Exactly. It is a data pipeline problem disguised as an intelligence problem. Speaker 1 And that is exactly what we are deconstructing today. The Hidden Dangers of Bad Document Parsing in RAG We are going on a massive structural journey in this deep dive. We are going to tear down the entire production R edge pipeline that's retrieval augment generation from the raw messy document all the way to that final answer on your screen. Speaker 2 It's a long journey, but a necessary one. Speaker 1 Definitely. We are going to explore why the most critical decisions, you know, the architectural choices that actually make or break the system, happen long before the AI ever computes a single token. We are going to show you why the pipeline itself is the product. Speaker 2 And to do that well, we have to start at the absolute bottom of that pipeline. We have to go to the basement where the compounding air is really born. Because before an AI can process any information, the system has to like physically read the raw materials. We have to talk about parsing. Speaker 1 Nobody wants to talk about parsing. It is structurally ugly. I mean, every developer wants to talk about neural network weights, multi head attention mechanisms, latent space. Nobody wants to talk about what happens when your beautiful, pristine AI pipeline encounters a corporate 200 page PD PDF created in 2014. Speaker 2 Because the reality of that document is terrifying. Think about what a standard enterprise PDS actually looks like. It is not a clean string of text. It has headers on every single page. It has footers. It is page numbers right in the middle of the text flow. Speaker 1 Oh, and those faint watermarks spanning diagonally across the page? Speaker 2 Yes, the watermarks. It has complex multi column financial tables. It has an image of a workflow chart embedded directly over a paragraph. Your very first absolute non negotiable job is to extract clean, logically ordered text from that visual nightmare. Speaker 1 Right. But I feel like the assumption is usually, well, we have OCR, we have text extraction tools, it'll pull the text and the AI is smart enough to figure out the context, right? Speaker 2 That is the assumption, but the failure modes here are catastrophic if you don't understand how these parsers actually operate. Let me paint a very real picture from production environment that perfectly illustrates this. Speaker 1 OK, lay it on me. Speaker 2 So a company built a massive financial RV system. They ingested all their quarterly reports, 10 KS all of it. The CEO sits down at the terminal evaluating the tool and asks a very simple direct question. What was our revenue growth in Q3? Speaker 1 A complete standard CEO question. Speaker 2 Right, so the AI thinks for a few seconds and confidently spits out a number that is wildly objectively wrong. Speaker 1 Oh man, the panic in the engineering Slack channel must have been instantaneous. I mean a hallucination right in front of the CEO. Speaker 2 Absolute chaos. Everyone scrambles. They are debugging the LLM prompt. They're checking the retrieval mathematical scores. They are looking at the advanced RE ranking algorithms. They spend days auditing the shiny AI components. Speaker 1 And what did they find? Speaker 2 Well, you know what they finally discover buried at the very bottom of the log files? The PDF parser choked. Speaker 1 The parser. Just the parser. Speaker 2 The PDF parser, literally the simplest tool in the entire chain, destroyed the data. The Q3 financial report had a complex table. Now a human looks at a table and naturally reads the columns left to right to connect a category with a number right. Yeah. Speaker 1 Obviously. Speaker 2 But the person library they used didn't understand semantic structure. It read the page visually based on spatial coordinates. It read top to bottom. Speaker 1 Wait, it literally read down the columns instead of across the rows? Oh, that is a disaster. Speaker 2 It was. The revenue numbers from 2023 were located physically above the headcount numbers for 2024. The parsers simply dragged A bounding box down the page, so the text that was extracted and permanently saved into the vector database was literally a mutant concatenated combination of dollars and employee counts. Speaker 1 And when the AI finally received that text in the prompt, it wasn't hallucinating at all, was it? Not at all. It was looking at a sentence that effectively said revenue was 5000 employees. Speaker 2 Exactly. The LLM was performing perfectly. It was being incredibly faithful to the text it was given. That is the fundamental, terrifying truth of these systems. Errors compound fatally at every single stage. Speaker 1 A bad parse leads to a bad chunk. Speaker 2 A bad chunk leads to a bad mathematical embedding. A bad embedding leads to bad retrieval, and bad retrieval guarantees a hallucinated answer. By the time you the user, see the output on your screen, you are 4 layers deep into a massive systemic failure that started just because a simple Python script choked on the table border. Speaker 1 It's like, imagine you hire a windowed renowned genius historian to summarize a highly complex diary for you. Speaker 2 OK, I like this. Speaker 1 But before you have them the diary, you run the book through a cross cut paper shredder. Then you blindfold yourself, take the shredded strips and take them back together completely at random. You've got a date from 1812 taped next to a casualty report from 1945, spliced with like a recipe for bread A. Speaker 2 Total mess. Speaker 1 Exactly. Then you hand the glue together nightmare to the genius historian and demand tell me what happened. When they give you a deranged story, you don't blame the historian, you blame the person who shredded the book. Speaker 2 That's a perfect analogy. The parser is the person taping the shredded diary back together. If they do not respect the original structure of the document, the intelligence of the historian is entirely irrelevant. Mastering Recursive Chunking for Semantic Context Preservation So, assuming we survive the parsing nightmare, let's say we use a multimodal model to read the document visually, or we use an advanced structural parser that actually understands what a table is. We extract clean text. It reads left to right. The tables are intact. Speaker 2 The headers are stripped out so they don't pollute the text flow. Speaker 1 Right, so now we have an incredibly long continuous string of text, but we obviously can't feed a 200 page book to the AI all at once for every single search query, we have to slice it up. Speaker 2 Right, and here we enter the art of chunking, and it truly is an art as much as it is a science. I would argue that determining your chunking strategy is the single most underrated architectural decision in the entire rag pipeline. Speaker 1 Let's explore the physics of that. Why is slicing the text so dangerous? Speaker 2 Think about physically cutting up a daily newspaper. If you take a pair of scissors and you cut the newspaper carefully article by article, following the structural lines, each piece of paper you have in your hand retains semantic independence. Speaker 1 Like it makes sense on its own. Speaker 2 Exactly. It makes sense on its own. It has a headline setting the context. It has a beginning, a middle, and a conclusion. Speaker 1 Right, it's a complete thought. Speaker 2 Now imagine you program an automated robot with a pair of scissors, and you give it a brutal, inflexible rule. Cut this newspaper strictly every 500 characters. I don't care where you are on the page, just cut it. Speaker 1 Oh, I see where this is going. Speaker 2 What happens? The robot cuts right through the middle of a vital sentence. You end up with one piece of paper that says the company reported an astonishing Q3 revenue of Beige, and then the paper just physically ends. Speaker 1 And the very next piece of paper starts with $4.2 billion, up 15% year over year, surprising investors. Speaker 2 Exactly. You've completely destroyed the relationship between the entity and the value. Speaker 1 So if a user asks the AI what was the Q3 revenue, the retrieval system is completely paralyzed. Speaker 2 Completely, the first chunk has the semantic keyword revenue but no actual number to answer the question. The the second chunk has the specific number 4.2 billion but lacks the word revenue, so the mathematical search might not even find it at all. You have effectively deleted information simply by how you wielded the scissors. Speaker 1 Wow, this sounds like a massive Goldilocks problem. How do we determine the perfect size for these pieces? Speaker 2 It is the ultimate Goldilocks problem. Let's play out the extremes to understand the mathematical trade-offs here. Let's say you decide to make your chunks incredibly small to ensure hyper relevance, say 100 tokens. Speaker 1 OK. And a token is roughly A syllable or a fraction of a word, right? So 100 tokens is maybe 75 words. A very short paragraph, yeah. Speaker 2 A very short paragraph, highly targeted but deeply vulnerable to context loss. You have 100 token chunk and the isolated text says tokens expire after one hour. They must be refreshed via the authorization endpoint. Speaker 1 Well, that makes total sense on its own. Speaker 2 Does it? What kind of tokens are we talking about? Speaker 1 Right. Speaker 2 The system doesn't know the chunk is so small that it lacks the broader architectural context. Are we talking about Jason web tokens like JW TS? Are we talking about OV access tokens? Are we talking about session cookies? Speaker 1 It's completely ambiguous. Speaker 2 Right. If a user asks how long do oath tokens last in the new architecture, the system might never pull up this specific chunk because the word oath you was located 2 paragraphs earlier, which got sliced away into a completely different disconnected chunk. You went too small and you lost the semantic anchor. Speaker 1 I see the answer is just orphaned from its subject. So the immediate engineering reflex would be to go bigger, right? Make the chunks massive. Let's do 2000 tokens, several pages of text per chunk. Now we have all the context. Speaker 2 You do have the context. You had captured the entire chapter, but now you have triggered the exact opposite failure mode. You've diluted the relevant score. Speaker 1 Diluted it. Speaker 2 Yeah, the specific needle you are looking for is now buried in a massive haystack of semantic noise. Speaker 1 Because the search algorithm is scoring the entire chunk, not just the one sentence we need. Speaker 2 Exactly, if that 2000 token chunk contains information about your entire authentication system, you know it has paragraphs on passwords, biometric server architecture, rate limiting, and finally token expiration, it is going to have a significantly lower mathematical relevance score. For the highly specific question, what is the token expiration time compared to a perfectly sized targeted? Speaker 1 Chunk so it just gets washed out it. Speaker 2 Gets washed out. The specific answer gets mathematically buried under all the general noise of the surrounding text. When the retrieval system ranks the results, that 2000 token chunk might rank #50 instead of #1. Speaker 1 It's just too muddy. So if you are a developer sitting at your desk right now architecting this pipeline, what is the exact blueprint? We need a validated default logic to build this correctly without spending months running trial and error. Speaker 2 The industry has tested this extensively against massive data sets. If you want the most robust general purpose strategy, you need to implement an algorithmic logic called recursive chunking. Speaker 1 Recursive chunking. Let's breakdown the logic of that step by step. If I am coding this right now, what do I do? Speaker 2 Step one is structural awareness. Your algorithm cannot just blindly count characters and slice. It must first attempt to respect natural human boundaries. You write logic that looks for double line breaks, which usually indicate the end of a paragraph. It looks for markdown headers. That makes sense. And crucially, and this is where most pipelines fail, if you are dealing with structured documents, you must write explicit exceptions that keep tables and code blocks completely intact. You never, ever slice a Jason payload or a financial table in half. You isolate it and you keep it whole. Speaker 1 OK, that is huge. We treat the structure as synchroid, but we still need a size limit, right? We can't just pass a 10,000 word table. What is the mathematical ceiling we are aiming for? Speaker 2 That's Step 2, enforcing the limit. You set your recursive character splitting targets strictly at 512 tokens. Speaker 1 512, that's a very specific number. Why not 500 or 1000? Speaker 2 It comes down to extensive benchmark testing across general purpose retrieval tasks across massive diverse data sets. Recursive chunking set at 512 tokens consistently scored around 69% accuracy in general benchmarks. Yeah, that is the highest baseline score for general strategies. It is. The mathematical sweet spot is long enough to hold meaningful self-contained context, but short enough to maintain a highly concentrated, relevant score when searched. Speaker 1 OK, 512 tokens is the golden rule, but that still doesn't solve the newspaper problem we talked about earlier. What if that 512 token limit happens to hit exactly right in the middle of that crucial sentence about Q3 revenue? Speaker 2 And that brings us to Step 3, which is really the magic ingredient of the entire strategy, the overlap mechanism. When you slice the document, you do not start the next chunk exactly where the last 1 ended. You instruct the algorithm to include a rolling overlap of 50 to 100 tokens. Speaker 1 A rolling overlap. Can you visualize that for us? Speaker 2 Think of the shingles on a roof. Have you ever analyzed how they are physically laid out? Speaker 1 Yeah, they layer over each other at the bottom of 1 covers the top of the next. Speaker 2 Exactly. Each shingle overlaps the next one. Why? Because if you put the perfectly edge to edge, the moment it rains, the water hits the seam, breaches the boundary and leaks into the house. Speaker 1 I. Speaker 2 See, the overlap protects the structural integrity of the boundary. It is the exact same logic here. Each text chunk overlaps the previous one by about 50 words, so context is never lost at a harsh arbitrary boundary. If a key sentence like that revenue figure happened to land right at the 512 token split point, the overlap ensures that the entire unbroken sentence physically exists in both chunk A and chunk B. Speaker 1 That is brilliant, the shingles on a roof. Even if the mathematical cut is awkward, the semantic meaning survives because it's duplicated just enough across the gap to preserve the context. Speaker 2 Precisely that is the blueprint for a robust chunking strategy, Structural awareness to protect tables, A 512 token limit for dense relevance, and A50 token overlap to seal the seams. If you implement that algorithmic logic, you have proactively solved a massive portion of your retrieval hallucinations before the AI ever gets involved. The True Economic Cost of Vector Embeddings in Production OK, so we have survived the parsing. We didn't scramble the tables. We have carefully, surgically sliced our text into beautiful 512 toe chunks with overlapping shingles. Now we have to make this text searchable by an AI, which means translating human language into high dimensional mathematics. Yes. Speaker 2 We're crossing the threshold into the realm of vectors and embeddings. Speaker 1 And this is where things get computationally heavy. And frankly, this is where I see a massive disconnect between engineering teams and finance departments. There's a huge track regarding costs here. Speaker 2 The API cost illusion. It is a very seductive trap. Speaker 1 Let's walk through the math of this illusion. Let's say you're an engineering manager planning the budget for this new RAG system. You look at the pricing pages for the major embedding models. Let's use Open AIS Text Embedding 3 Small as the benchmark. You do some quick back of the napkin math. You have a massive corporate archive, 10 million internal documents. Speaker 2 A very standard enterprise data set. Speaker 1 Right. You average it out to five chunks per document. So you have 50 million chunks of text that you need to translate into mathematical vectors. You plug 50 million Trump's into the pricing calculator for that API and the total cost comes out to what, about $100 A? Speaker 2 $100 to process the entire intellectual history of the organization. Speaker 1 It feels like stealing. It is absurdly cheap to start. You can embed the entire company's history for the price of a nice dinner. So you sign off on the budget. You build the pipeline directly around this external API, and everyone is thrilled. Where's the trap? What is the hitting cost? Speaker 2 You fall for the illusion because you are treating the vector database like a static hard drive. You are only calculating the cost of the initial ingestion. The true cost of vectors is not the single API call, it is the cascading operational overhead over time. Speaker 1 Because the data doesn't just sit there. Speaker 2 Exactly. Let's play out the reality of a production system. Six months from now, a new, vastly superior embedding model is released. It captures nuance better. It processes technical jargon Florida State, and it expands the dimensional space to capture deeper semantic relationships. You absolutely needed to improve your system. What do you have to do? Speaker 1 Well, I just update my code to point to the new API endpoint. Speaker 2 Right, you switched the endpoint for your user queries, yes, but the vectors sitting in your database were created by model A. They exist in a specific mathematical space, let's say 15136 dimensions maps according to model A specific neural weights. Model B mass concepts entirely differently. They speak different mathematical languages. You cannot compare a model A vector to a model B vector. Speaker 1 So I can't just mix and match them. Speaker 2 You cannot. To use the new model you have to re embed everything. Speaker 1 All 10 million documents, all 50 million chunks. Speaker 2 Every single one. So there is another $100, which admittedly is still financially cheap. But it is also a week of pipeline engineering, orchestrating massive database migrations, managing downtime and monitoring the ingestion pipeline. Speaker 1 That's a huge operational headache. Speaker 2 And that is just the upgrade cycle. It gets worse. Your company is a living, breathing entity. People are editing Confluence pages, updating Jura tickets, rewriting HR policies, adding new code to repositories. Speaker 1 The corpus is constantly mutating. Speaker 2 Exactly. Let's conservatively say just 10% of your corpus updates every single month. That means every month you are reparsing, re chunking, and re embedding 5 million chunks. It's an ongoing, perpetual cost. Speaker 1 OK. But honestly, $10 a month month in API cost is still basically free for a mid sized company. I'm still not seeing the catastrophic trap here. Why wouldn't I just comfortably use the API forever? Speaker 2 Because eventually, you hit the brutal wall of enterprise constraints. Let's say your company pivots to handling healthcare data, or you land a contract with the Department of Defense. The Compliance and Infosec departments knock on your door and deliver an ultimatum. We cannot transmit our raw, unencrypted, highly classified data over the open Internet to a third party API. Speaker 1 Period. Speaker 2 The security veto. Speaker 1 You must sever the connection. You must host the embedding model yourself on our own internal servers behind our firewall. Or, alternatively, your user base scales so rapidly that the network latency of calling an external API for every single search query is introducing A500 millisecond lag, and your product team demands you run the model locally to achieve 10 millisecond response times. Speaker 2 The constraints force the architecture, so now I can't just hand open AIA 100 bucks. I have to build the physical infrastructure myself. And that is where the economics brutally flipped from software costs to hardware costs. To rent a high quality embedding model yourself locally at production speeds capable of handling concurrent users, you need Sirius silicon. You need GPUs. If you go to AWS or Google Cloud and rent a self hosted A-100 GPU, which is the absolute workhorse required for this kind of tensor calculation, it is going to cost you roughly $1800 a month. Speaker 1 $1800 a month just to keep the instance turned on. Speaker 2 Just for the raw compute hardware, that does not include the storage costs, the egress fees or the $150,000 salary of the specialized DevOps engineer you now need to hire to maintain the CUDA drivers, manage the container orchestration and ensure the model doesn't crash. Speaker 1 Yeah, so if I am forced to self host due to privacy, or if I foolishly choose to self host thinking it will be cheaper at scale, where is the actual crossover point? At what massive volume of data does that $1800 server actually become more economical than just paying the tiny fractions of a cent for the API? Speaker 2 Data scientists have modeled this out extensively. The mathematical crossover point where self hosting actually becomes cheaper than paying the API is staggeringly high. You have to be processing roughly 50 to 100 million tokens per month, every single month in ongoing data turn to justify the baseline cost of the self hosted GPU. Speaker 1 50 to 100 million tokens every month. That is an absurd amount of data ingestion. Speaker 2 It is if you were below that volume, and frankly 95% of internal corporate RA tools are well below that. The API wins economically, hands down. The lesson here is that the economics of vectors are not about the microscopic price per token. They are about the massive architectural choices you are forced into by your organizational constraints. Speaker 1 That is a massive paradigm shift for how we budget these projects. Beyond 'Vibes': Combining Vector and Keyword Search with RRF But let's assume we have our infrastructure secured. Whether we chose the API or we are paying for the A-100 GPU, we have successfully translated our data. We now have a giant beautiful database filled with 50 million high dimensional mathematical vectors. Each vector represents one of our perfectly overlapped 512 token chunks. Now a user sits down, types a query into the chat box and hits enter. How do we actually pull the exact right chunks out of that massive database? Speaker 2 We entered the search phase and this is where the most common fundamental and damaging mistake in our edge systems is made. Speaker 1 The trap of simple vector retrieval. Speaker 2 Yes, if you look at almost every tutorial, every quick start guide, every developer boot camp, they teach you to build a pure vector search. The user asks a question, you turn that question into a vector using the exact same model, and you ask the database to calculate the cosine similarity. You find the vectors that are mathematically closest to the question in high dimensional space. Speaker 1 Which makes sense on paper. You are measuring semantic similarity. You are searching by concept, not just keywords. Speaker 2 You are measuring meaning, but here is the critical vulnerability. Pure Vector Search is completely obsessed with vibes. Speaker 1 Vibes. I love that description. What do you mean by that mathematically? Speaker 2 Vector models map concepts conceptually. They understand the general aura, the surrounding context of a word, but they fundamentally fail at exactness. They struggle with hyper specific terminology because those specific terms get mathematically blended into the broader concept during the embedding process. Speaker 1 Walk me through an example of how that fails in real time so. Speaker 2 Let's go back to our engineering team. A developer is using the company AI to debug a micro service. They type in a highly specific technical question. What is the RS256 JWT signing algorithm used in our legacy authentication service? Speaker 1 Highly specific. They need the exact code for RS256 JWT. Speaker 2 Exactly. So the system executes a pure vector search. The embedding model translates that query and effectively says. I understand the semantic 5 of this question. This is about authentication. It is about cybersecurity, digital tokens, and logging in. Let me find the chunks that share this conceptual neighborhood. So it dives into the database and brings back chunks about UTH 2 point O flows. It brings back chunks about session management. It brings back high level architectural overviews of the login portal. Speaker 1 Because mathematically, in that 1536 dimensional space, oath and session management are extremely close to the vibe of the question. Speaker 2 Yes, they are semantic neighbors. Oh, but it completely misses the specific technical string RS25 E 6. It missed the exact operational requirement because it was too busy looking for conceptual similarity. Speaker 1 So the developer gets back a generated answer summarizing general security practices, but completely lacking the specific algorithm code snippet they actually asked for the AI. Speaker 2 Fails. The AI fails because the retrieval failed. Pure vector search is simply not robust enough for enterprise production, where exactness matters just as much as meaning. You must implement a hybrid search solution. Speaker 1 Hybrid search. So we combine the semantic vector search with a different mechanism entirely, Yes. Speaker 2 You combine the vector search which gives you that beautiful overarching semantic meaning and conceptual matching with good old fashioned BM25 keyword search. Speaker 1 BM25 This is essentially traditional search engine technology, right? Speaker 2 It is the bedrock of classical information retrieval. BM25 is an algorithm based on term frequency, inverse document frequency, or TFIDF. BM25 does not care about vibes. It does not understand semantic meaning. It does not know that a dog is similar to a canine. Speaker 1 It just matches words. Speaker 2 Exactly what it cares about is exact term matches adjusted for how rare that term is across the entire database. If you search for RS256BM25 will brutally mechanistically filter the database for chunks that contain that exact specific string of characters. Speaker 1 Which solves the problem perfectly. The vector search casanet for the overarching concepts and the keyword search acts as a sniper rifle to nail the specific jargon, acronyms and part numbers. Speaker 2 Exactly. But solving that problem immediately introduces a complex mathematical nightmare. You now have two completely different search engines giving you 2 completely different lists of results. Vector search gives you a top ten list scored by cosine similarity, a number usually between zero and one. BM 25 gives you a top ten list scored by keyword frequency, a completely unbounded score that could be 5 or 50 or 500 depending on term saturation. How do you combine them? The scores are mathematically incompatible. Speaker 1 It's like trying to compare a temperature in Fahrenheit to a distance in kilometers. You can't just average the numbers together. Speaker 2 You can't normalize them easily because the distributions are entirely different. If you try to force them together 1 one search method will completely overpower the other. To merge them elegantly in production, you use a specific mechanism called reciprocal rank fusion or RRF. Speaker 1 Reciprocal rank fusion. Let's unpack the actual math behind how that works. Why does it solve the Fahrenheit kilometer problem? Speaker 2 It solves it by completely ignoring the underlying scores. RRF doesn't care about the vector distance of .89, it doesn't care about the BM25 score of 45. It only looks at the rank, the physical position of the chunk on the respective lists. Speaker 1 It strips away the complex scores and just looks at who came in for a second third. Speaker 2 Exactly. It uses a very simple, elegant mathematical formula. It's 1 / K plus rank, where K is a smoothing constant, usually set around 60. So it looks at chunk A, says OK chunk A was ranked number one by the vector search, and it was ranked #4 by the BM25 keyword search. It runs that reciprocal formula for both ranks, adds the two fractions together, and generates a completely new unified score. Speaker 1 That is incredibly smart because the math uses reciprocals. Fractions being ring number one gives you a massive mathematical boost, but the value DROs off a Cliff as you go down the list, so chunks that perform well on both lists naturally float to the very top of the combined master list. Speaker 2 Precisely. It creates a perfectly blended, unified list of the absolute best chunks without needing any messy score normalization. Speaker 1 So what is the actual tangible impact of doing this? If I take the engineering time to build a dual engine hybrid search with RRF instead of just relying on the simple vector search out-of-the-box, what do I actually gain in performance? Speaker 2 The impact is massive in rigorous production benchmarks. Implementing this hybrid approach, boost recall at the top 10 results by 10 to 15% across the board. Speaker 1 10 to 15%. I mean, to a layman that might not sound like a massive leap. Speaker 2 In the context of RHA, that 10% is everything. That 10% is the literal physical barrier between a correct answer and a catastrophic AI hallucination. That 10% is the difference between finding the exact RS256 code snippet and the model confidently making up a non existent signing algorithm because it couldn't find the real one in its context window. It is the difference between trust and system failure. Speaker 1 OK, so we've run our hybrid search. We've cast a wide net using vectors and keywords, and we have pulled back a master list of, say, 50 highly relevant chunks of text. Do we just take those 50 chunks, shove them into the prompt, hand them to the LLM and say figure it out? Speaker 2 No, absolutely not. That is another massive architectural trap which leads directly to the context degradation problem. We will cover in a minute, but first we cannot hand 50 chunks to the model. We have to refine that list. We need to introduce the ultimate game changer in modern retrieval cross encoder re ranking. Speaker 1 Cross encoder re ranking OK if the hybrid search was casting a wide net, how does this fit in wait? I think I see the analogy here. If the database has 50 million chunks, the hybrid search is like a resume screen. You're hiring manager with 10,000 applicants. You need to narrow it down fast. Speaker 2 That is exactly the dynamic. The hybrid search is the resume screen. Speaker 1 Use keyword filters. You look for semantic matches and you independently evaluate each resume to cast a wide net. You get it down to 50 candidates, but you don't just hire all 50 people. And you definitely don't hire someone just based on the superficial resume screen. You bring them in for a rigorous in person job interview. Speaker 2 A rigorous, exhaustive, highly specific interview. That is exactly what a cross encoder does. The 'Job Interview' for Your Retrieved RAG Chunks Re ranking is the interview. Speaker 1 So how does a cross encoder work mechanically? What is the neural architecture doing that makes it so much better than the vector search we just performed? Speaker 2 Think about how the first vector retrieval model worked, what we call a buying coder. It scored the user's query in the document. In completely separate universes, it turned the query into a vector. In isolation, it turned the document into a vector, and then it simply measured the geometric distance between them. It never actually looked at how the words in the query interacted with the words in the document. Speaker 1 You're right, it's just measuring proximity in space. Speaker 2 A cross encoder changes the paradigm entirely. It does not use vector distances. It takes the user's raw text query and the specific document chunk, and it feeds them together simultaneously through the deep transformer layers of a complex neural network. It uses full self attention mechanisms. It forces the model to analyze the exact granular interaction between every single word in the question and every single word in the text. Speaker 1 So it isn't asking are these two things conceptually similar? Speaker 2 No. It is asking a much harder, much more profound analytical question. Does this specific chunk of text logically entail and actually contain the exact answer to the specific query? Speaker 1 Wow, so it is actually reading them together, cross referencing the concepts to see if one fulfills the other. That is a profoundly deeper level of understanding. But that immediately raises the question, if it is so vastly superior at finding the truth, why didn't we just do that from the very beginning? Why bother with vectors and BM25 at all? Why not run the cross encoder on all 50 million chunks? Speaker 2 Because of the brutal constraints of computational physics, cross encoders are punishingly slow. They require massive computational power because they're running a full complex neural network inference, calculating attention across all tokens for every single pair. The computational complexity is O n ^2. You physically cannot run a cross encoder on 50 million chunks. If you tried, a user would ask a question, the server would catch fire and they would have to wait a week for the answer. Speaker 1 Speed trade off. So the architectural force is a 2 step process. Speaker 2 Exactly. You run the fast lightweight hybrid retrieval the resume MMA screen to violently filter 50 million chunks down to a manageable 50 candidates. Then you use the slow, computationally heavy cross encoder the interview to deeply evaluate just those 50. Running a cross encoder on 50 items only takes about 100 milliseconds. Speaker 1 That's completely imperceptible to the user. Speaker 2 It's light and fast at that scale, so the cross encoder evaluates those top 50 candidates, drastically rescoring and reordering them based on true logical entailment, until you are left with the pristine, mathematically verified absolute best top five chunks. Speaker 1 The pristine top five. Overcoming 'Lost in the Middle' with Strategic Prompt Formatting Let's pause and look at what we've built. We have survived the parser protecting our tables. We have chumped perfectly using a 512 token limit with overlapping shingles. We navigated the brutal economics of vectors. We cast our net with a hybrid BM25 and vector search, merged, urged them with reciprocal rank fusion, and we ruthlessly interviewed the results with a cross encoder. We now possess the five most perfect relevant pieces of information in the entire organization. Speaker 2 Yes, the nervous system has successfully delivered the signal. Speaker 1 Now, finally, we get to hand them to the brain. We pack those chunks into a prompt, hand them to the LLM and ask it to generate the answer. Speaker 2 We do, but we have to be incredibly careful at this exact moment because the AI, the brain itself, has a fundamental architectural flaw in how it processes large blocks of text. Speaker 1 A flaw in how it reads. We think of these models as supercomputers. We assume they digest data uniformly. Speaker 2 They can ingest it instantly, but they do not comprehend it uniformly. There is a deeply researched, well documented phenomenon in AI architecture called Lost in the Middle. Speaker 1 Lost in the middle. I have to say, when I first learned about the mechanics of this, I was stunned. It sounds so profoundly human for a mathematical system. Speaker 2 It mimics human cognitive fatigue perfectly. Think about when you have to read a massive, dense 40 page legal contract. You pay straight strict hyper focus attention to the very beginning of the definitions, the parties involved, the initial premise, and as you get to the end, you wait back up. You pay strict attention to the very end, the signature blocks, the final termination clauses, but the middle. Pages 15 to 35. Speaker 1 You absolutely stem it. Your eyes glaze over. You're just looking for paragraph breaks. Speaker 2 You skim it, you lose semantic focus. LLM's exhibit the exact same behavior due to how their softmax attention mechanisms distribute weights across long sequences of tokens. They are terrible readers of long, dense contexts. The attention scores heavily favored the very beginning of the prompt and the very end of the prompt. They literally skim, degrade their attention, or completely forget the critical information placed in the middle. The performance curve looks like a giant U High at the start, plummeting in the middle, high at the end. Speaker 1 Hang on, I have to push back on this because it completely contradicts what we're being sold right now. Open AI, Google Anthropic. They are aggressively marketing models with massive context windows. They are bragging about 128 K, even 1,000,000 token context windows. They're telling developers just shove the entire book into the prompt, the AI can handle it. Are you telling me that stuffing these models full of data actually makes the AI Dumber? Speaker 2 Yes, unequivocally. It is one of the most dangerous misconceptions in the industry right now. A massive context window is a brilliant marketing bullet point, but in engineering practice more context does not equal better answers. Retrieval accuracy aggressively degrades as you stuff the context window. If you fill A128K window, the models mathematical ability to extract a specific isolated fact from the middle of that text drops off a Cliff. Speaker 1 Think about the consequence of this For our pipeline. We just did all that brutal engineering work, the hybrid search, the cross encoder to find the perfect chunks. Let's say our cross encoder determines that chunk #4 is the absolute best, most crucial piece of data. If we just blindly handle list of 10 chunks to the LLM and chump #4 happens to sit physically right in the middle of the text block, all right. I mean right in the middle of the text block. Speaker 2 The AI might completely ignore it, the attention weight simply won't focus on it, the model might hallucinate an answer or simply state I don't know, despite the fact that the literal answer was physically located in the prompt you provided. Speaker 1 That is infuriating. So how do we mitigate this? How do we hack the attention mechanism to ensure the LLM actually reads the data we fought so hard to retrieve? Speaker 2 You mitigate it using strict formatting hacks. First, you must enforce ruthless brevity. Do not give the model 20 chunks just because you technically have room in the 128 K prompt window. Limit the context strictly to the top five to seven highly curated chunks. Give the model less text to read so the attention weights don't get diluted. Speaker 1 OK, enforce brevity. Cut the noise. What is the second heck? Speaker 2 Strategic ordering. You know for mathematical fact that the models attention curve is U-shaped. It focuses heavily on the beginning and the end. So you game the system. You take your five retrieve chunks instead of just listing them sequentially from 1:00 to 5:00, you orchestrate their physical placement. You force the highest scoring, most critical chunks into the very first position and the very last position in the text prompt. Speaker 1 You literally hide the lower scoring, less important chunks in the middle where you know that AI's attention is going to degrade anyway. Speaker 2 Exactly. You bookend the prompt with the most critical information. You put your best foot forward and your best foot backward. It seems like an incredibly simple trick, but reordering the context drastically alters the attention weight, significantly reduces hallucinations, and ensures the LLM actually utilizes the gold standard data you retrieved. Speaker 1 This is amazing. We have built an incredible, highly tuned pipeline. We parse visually and structurally. We chunk with mathematical overlaps. We navigate the cost of embeddings. We search using vectors and keywords. We fuse the ranks. We ruthless interview the chunks with a cross encoder. We strategically format the prompt to hack the LLMS attention span. Building Custom Pipelines for Real-time Data Updates We deploy the system. It answers questions. Florida State. The engineering team celebrates. The CEO is thrilled. Speaker 2 And then tomorrow morning at 9:00 AM, someone of the marketing team logs into the company wiki and updates a single paragraph on a single page. Speaker 1 Oh no, and the entire House of Cards comes crashing down. The unsexy infrastructure problem keeping the data fresh. Speaker 2 This is where 90% of RJ prototypes completely collapse when they try to move to production. We have built a perfect, pristine system for a static snapshot of data. But enterprise data is a living Organism. It mutates constantly. Let's walk through the cascading architectural nightmare of a single edit. Speaker 1 OK, so the marketing person updates a paragraph on a product page. Speaker 2 Right, the physical document changes because the text changed. The structural boundary shifted, which means the 512 token chunks change. Because the chunks changed, the high dimensional mathematical representation must be translated again, which means the vector embeddings change, which means instantaneously. The massive vector database we are relying on for our hybrid search is completely out of date. It contains ghost data vectors representing text that no longer exists, and it is completely blind to the new truth. Speaker 1 So the AI will immediately start giving confident outdated answers based on yesterday's wiki page. Speaker 2 Exactly. Now, in a traditional software architecture like a standard SQL database running a banking application, this is a solved problem. We have incredibly mature tools for this, things like de Bezium or transaction logs, which automatically detect changes in one database and elegantly sync them across downstream systems. Speaker 1 Right Change data capture, CDC. You just turn it on and it works. Speaker 2 Yes, but here is the terrifying reality of the AI engineering space right now in the vector database world, there is no standardized plug and play change data capture. It simply does not exist yet. It is the Wild West of infrastructure. Vector databases are largely stateless in this regard. Speaker 1 So how do we actually fix this? I assume the answer is not to reprocess all 10 million documents every single night. We already established that would cost a fortune in API fees and compute time. Speaker 2 No, you can't do that. You have to roll up your sleeves and build a custom, highly resilient change detection pipeline entirely from scratch. You have to weave a nervous system of custom webhook handlers across your entire organization's tool set. Speaker 1 What does that architecture look like in practice? Speaker 2 It means you write micro services dedicated entirely to listening. You deploy a service listening for Confluence webhooks like hey, a page was just saved, you have a service pulling your AWS cloud storage for S3 event notifications, hey a new PDF was just uploaded to the bucket. You configure webhooks listening to your code repositories. Forget commits. Hey, a markdown file in the documentation folder was just merged. Speaker 1 You are building a massive tripwire system. Every time a piece of data moves anywhere in the company, an alarm goes off in the pipeline. Speaker 2 Exactly. And when that webhook ping comes in, you execute a highly surgical isolated update. You do not reprocess the whole database. You fetch reparse and rechunk only that specific document that trigger the alarm. Speaker 1 OK, but even that isolated document might be a 50 page policy manual and the HR person only changed one sentence on page 40. Do we re embed all 50 pages? Speaker 2 No, because that wastes API credits in compute cycles. This is where you implement A cryptographic division process. When you re chunk the updated document, you mathematically hash each of the newly generated chunks. You use an algorithm like S HA25T6 to create a unique fixed length digital fingerprint for that specific string of text. You then query your database and compare those newly generated hashes against the old hashes of the chunks you already have stored. Speaker 1 Oh I see you are comparing the fingerprints. Speaker 2 Precisely. If the hashes match perfectly, it means the text in that chunk is mathematically identical. Nothing changed in that 512 token block, You skip it entirely. But if a hash is different, or if it's a completely new hash that doesn't exist in the database, that tells you exactly which specific isolated chunks physically changed. Speaker 1 That is surgically precise. You only spend the money and the compute time to run those specific altered chunks through the embedding model, and you only update those specific vectors in the database. Speaker 2 Yes, and building that infrastructure, the high availability web host, the state management, the cryptographic hashing, the diffing logic that is incredibly complex, deeply unsexy back end engineering. It has absolutely nothing to do with neural networks or prompt engineering. But without it, you're highly advanced AI is functionally useless within a week because it's memory is permanently frozen in the past. Speaker 1 OK, we are in the final stretch. Ensuring AI Truthfulness with the RAGAS Framework The system is live, the infrastructure is humming, the data is updating constantly via our custom webhook tripwires. The LLM is successfully navigating the middle attention problem thanks to our formatting hacks. Users are asking questions and getting answers. But this brings up the final, most terrifying question of all For anyone managing this system, how do you actually know if the AI is telling the truth? Speaker 2 The measurement dilemma. The ultimate blind spot of generative AI. Speaker 1 Think about this logically. In a traditional search engine like Google or an internal keyword search, you measure success with a very simple, highly effective behavioral metric, the click through rate. If a user searches for PTO policy and they click on the first link provided, you assume the search worked. If they scroll past the first page, click the 10th link, or abandoned the search entirely, you know it failed. Speaker 2 It is a clear binary trackable behavioral signal. Speaker 1 Right, but in our the user doesn't get a list of links to click, they get a conversational generated paragraph of text. You cannot track a click on a generated sentence. The user reads it and they either believe it or they don't. So how on earth do you evaluate if the AI is doing a good job? Are you supposed to hire a team of humans to sit in a room reading every single AI response, cross referencing it with the corporate wiki and grading it? You can. Speaker 2 It doesn't scale. You cannot evaluate 10s of thousands of queries manually. It would cost a fortune. You need automated, robust programmatic evaluation. And this is where we enter the final piece of the architectural puzzle, the Rgas framework. Speaker 1 Raji Ace Rags Let's do a deep dive into how this actually functions because it seems to be the gold standard in the industry right now. Speaker 2 The brilliance of Raji AES lies in a concept called Reference Free evaluation. Speaker 1 Reference free. Explain the mechanics of that. What are we freeing ourselves from? Speaker 2 Historically, to evaluate a machine learning model, you needed a massive data set of ground truth labels. You needed subject matter experts to write 1000 potential user questions and then meticulously write the perfect, exact correct answer for each question. Then you test the AI against those human answers to see how close it gets. Speaker 1 Which takes months to build and is immediately obsolete when the data changes. Speaker 2 Exactly. Raji S throws that entire paradigm out the window. Reference free means it does not require humans to sit around writing ground truth labels. Instead, it uses a powerful LLM to act as an impartial judge, evaluating the RG system live on real production traffic as it is happening. Speaker 1 It uses AI to grade the AI. OK I'm intrigued, but how does it actually judge the quality objectively? What specific metrics is it calculating? Speaker 2 Array J ace breaks quality down into 3 core pillars, 3 distinct calculated metrics. Pillar number one is context recall. Speaker 1 Context recall. Speaker 2 This metric evaluates the nervous system, the search pipeline we spent all that time building. Did the retrieval system actually find the right chunks to begin with? The Judge LLM looks at the user's raw question, and it looks at the five chunks we retrieved, and it evaluates that the text physically contains the necessary factual information to answer the question. If this context recall score is low, it tells you immediately that you're parsing your chunk limits or your hybrid search weights are broken. Speaker 1 OK, that diagnosis, the retrieval layer, what's the second pillar? Speaker 2 Pillar number 2 is answer relevance. This evaluates the final output generated by the brain. Did the generated text actually directly answer the specific question the user asked? Speaker 1 So if the user asks what is the Q3 revenue and the AI writes a beautiful, historically accurate Florida State written essay about the history of the company's founding in 1985, it would fail answer relevance. Speaker 2 Exactly. The information might be factually true, but it's completely irrelevant to the prompt. The AI got distracted. But the third pillar is where the true power lies. It is the most critical metric trick in the entire RAG universe. Speaker 1 The third pillar, Faithfulness. Faithfulness is the model loyal to the text it was given. Speaker 2 Exactly. Is the final generated answer strictly logically grounded in the retrieve context or did the LLM hallucinate and make things up based on its pre training data? Speaker 1 How does RAGS mathematically calculate faithfulness without a human checking the facts? Speaker 2 It uses a brilliant logical breakdown. It takes the AI's generated answer and breaks it down into individual atomic claims. For example, if the answer is revenue was 4.2 billion, up 15%, the judge extracts 2 claims. Claim one revenue was 4.2 billion, claim 2 revenue was up 15%. It then takes every single extracted claim and checks it against the retrieved chunks. Can this specific claim be logically deduced strictly from the provided text, Yes or no? If the AI stated the revenue was 4.2 billion, but the text chunk only mentions the headcount, the faithfulness score for that query drops to 0. Speaker 1 That is incredibly rigorous. It's essentially performing automated fact checking on every sentence. Speaker 2 It has to be that rigorous setting. The quality bar here is not an optional feature. Hallucination is the most dangerous systemic failure mode in enterprise AI. Speaker 1 Because it completely destroys trust in the system. Speaker 2 It destroys trust and it causes catastrophic real world mistakes. A confidently stated lie is infinitely worse than the AI simply saying I don't know. If the AI says I don't know, the user is slightly annoyed and they ask a colleague. If the AI confidently tells the engineering team to use the wrong cryptographic security protocol, you have a massive data breach. Speaker 1 So how do you operationalize this framework in production? Speaker 2 You establish A roofless automated rule. Every single production answer generated by the system gets a RAGAS faithfulness score calculated in the background asynchronously, and you set a hard mathematical threshold. Anything that scores below a .8 out of one point O on the faithfulness scale is immediately flagged. It is sent to a dashboard for the engineering team to review, and in critical systems you intercept the answer before the user even sees it and replace it with a graceful fall back message. Speaker 1 You are building an automated AI driven quality control inspector that looks at every single product before it leaves the factory floor, yes. Speaker 2 That is what a mature production grade AI pipeline requires to survive in the real world. Why RAG Architecture Outweighs the LLM Itself Wow. Let's just take a breath and look back at the sheer scale of the architectural journey we just went on. Speaker 2 It is a massive undertaking. Speaker 1 We started with the absolute messy reality of parsing corporate documents, stripping the watermarks, ensuring the bounding boxes read tables left to right to avoid the revenue headcount mutation. We moved into the meticulous art of chunking, building those strict 512 token pieces with 50 token shingle like overlaps to mathematically protect the semantic context across boundaries. Speaker 2 We navigated the hidden cascading economics of translating that text into high dimensional mathematical vectors, realizing that the real cost isn't the API ingestion, but the ongoing hardware constraints and infrastructure of updates. We. Speaker 1 Fixed the vibes only semantic problem of pure vector search by implementing a hybrid approach, merging overarching meaning with exact BM25 keyword matching using the reciprocal rank fusion formula and then violently occasionally filtering the results with cross encoder reranking to ensure true logical entailment. Speaker 2 We hacked the LMS U-shaped attention span, enforcing brevity and strategically ordering the prompt to mitigate the lost in the middle phenomenon. We built a vast network of custom webhook pipelines executing cryptographic hashing to surgically handle the unsexy problem of keeping our vector database fresh without going bankrupt. Speaker 1 And finally, we policed the absolute truth of the entire system using the Reference Free Rigs framework, automating the extraction of claims and setting a hard threshold for faithfulness to ensure our AI isn't confidently lying to our users. Speaker 2 When you lay the entire architecture out step by step like that, the core thesis becomes incredible obvious, doesn't it? Speaker 1 The LLM is the easiest part of the equation. You can swap out the reasoning engine tomorrow with an API call. The true engineering, the actual defensible value of an AI application lives entirely in the connective tissue. If you build the nervous system right, any decent brain will shine. But if you skip the architecture, if you ignore the structural parsing, the chunking boundaries, and the hybrid retrieval, the most advanced trillion parameter AI in the world will confidently, eloquently serve you. Absolute garbage. Speaker 2 The pipeline is the product. Speaker 1 The pipeline is the product I think that covers the architecture completely. I want to turn it over to you for a final thought based on all of this. Speaker 2 I want to leave you with a final lingering question to ponder as you look at the systems you were building. We just spent this entire deep dive detailing how this production RAD pipeline is a highly complex, fragile machine. It is meticulously built, tuned, and constantly monitored by human engineers. It is humans deciding the five and 12 token chunk limits. It is humans tweaking the hybrid retrieval weights. It is humans setting up the web hook trip wires. But with generative AI advancing as rapidly as it is, well, how long until we start using AI to automatically diagnose and dynamically adjust its own pipeline? How long until an orchestrator AI looks at a messy PDF, automatically determines the perfect dynamic chunking boundaries based on the semantic density, runs Monte Carlo simulations that test its own retrieval weights, and rewrites its own cryptographic hashing rules on the fly? How long until the system entirely removes the human engineer from the Riga pipeline altogether?