S07E86 - Context engineering: de onzichtbare motor achter betrouwbare AI

Hi, nice that you're listening to this short episode of EOTD Live. My name is Joke Schneider, CTO at ETC. And picture this, Barcelona 2025. You see yourself traveling through this beautiful city. You have an overview of the facilities, restaurants, activities and the most beautiful places to look over the city. And the most amazing thing is that you have these routes and let go through an AI assistant. But there is one problem. The AI doesn't know that you're roles are bound, that you're vegetarian or that you've been to Barcelona last week for your work. The result is completely irrelevant advice. And this is exactly why context engineering has become the most important part of AI development. And yet many people are still surprised with prompt engineering. That's why I'm telling you today about context engineering. Because without good context, the best AI agent in the world can mean nothing to you. Well, let's start with what context engineering is actually. Well, this context engineering is a lot of stories about this, but that's fine. That's what we're going to talk about today, much further than a smart question mark on Chatchapiti. It is the design and construction of, you could say, complete information systems that give AI models the right information at the right time. It makes sure that AI systems not only know what they have to do, but also discuss all the information they need to do it right. So with context engineering, you build dynamic systems that bring information from different sources. That can be users' input, conversations, external databases, data from tools that you set up, APIs, programming interfaces to business products. And they are all organized and presented at the moment that the AI needs them. It is a holistic approach that constantly adapts to changing circumstances, a changing situation, I have to say. But when it comes to this form of engineering, we also have to talk about the context window limits. Because here we focus on a limitation. AI models can't work at the same time with a lot of information. They have a context window, that is a limit to a number of words, that they can actually understand in one go. And that means that context engineering not only goes about collecting information, but above all about selecting it smartly. And you can't stop everything that is potentially relevant in the context. So you have to choose. And what information is the most important thing for this specific task? What can be summarized? What can be left out? And this is where context engineering really becomes a challenge. It is about making smart choices within the limitations. But what is the difference with context engineering? And that goes on, it is not a semantic difference or so. Many people think that context engineering is just a phenomenon for context engineering. But I think that the differences are good to understand when it comes to calling out a bit. The differences are, for example, in the focus. For example, in context engineering, you focus on writing a very specific instruction. Context engineering builds complete information systems that dynamically react to what I just said. You have different scope, there is a prompt, there is a single, let's say, sentence, drawing. But there is context engineering around all relevant data, tools, documents and the ongoing status of the system. There is a difference in the adaptability, prompt is largely static, you write that one time and use it more often. While context engineering is much more dynamic, it is per task, per use, per session, you can adjust what is eventually sent together with the prompt to the language model. And you have a difference in impact on the quality of the outcome, so a good prompt has, let's say, a medium impact, because it is limited to the context that is given. Context engineering has a very high impact on the quality of the outcome, because it is directly responsible for the memory, more steps possibilities of the AI, all the data you give. So this explains why organizations that are successful with AI often spend more time on context engineering than on prompt engineering. Context engineering ultimately determines what an AI system can and cannot. I want to tell you seven ways how you can add context, and that is not up to date, but I think these are very important. Well, it all starts with the prompt itself, you start everything, so these are the instructions and the instructions must be clear. And the AI can also tell what the role is, that is actually a little bit of context. But also the system instructions, they also determine how the AI behaves, what style it handles. And so you can give a clear goal, describe a role, or even use a placeholder, where, for example, you can give the current date with it. And then the prompt and the context run a little bit together. Another is, and maybe you have done that very often, is just a web URL, and a link to a web page. And with that you give the AI access to current information. And this ensures that you do not stick to the training data, but you can get real-time information on it. Simple, but very powerful. You can upload documents, maybe you have done that too. Business documents, manuals, specific knowledge sources, you can make them available directly. And you literally give your knowledge to the AI with it, also in a way of context engineering. Then we have the adapted AI assistants, so think of the custom GPT in a chat GPT, where you can upload a complete knowledge bank. Well limited, but you can really give a lot of context with it. So the AI always has access to that specific information and tools that have been provided with it. The next step is the RAC systems. A RAC is for Retrieval Augmented Generation. And these are systems that search for smart information from pretty large databases. And only give those relevant pieces to the AI. And this is actually the professional way to manage knowledge within a context window. And that is limited because you are only looking for relevant pieces in that very large range of data that is there. What I have already mentioned is the API access. So you give the AI a programmatic access to external systems. And that can be a CRM, your agenda or other company systems. And with this, the AI will be able to get the data in real digital colleagues and really know your context. And in the world of the developers, I also come out myself, you can see very much that you can configure it with Clot, Projects or GitHub Copilot. For example, with your own coding standard, your project structure. But they can also look in the code and get the context there. The AI then understand the way you work and then adjust your style to it. And then I have now talked about what you add in the context engineering. But in the context engineering, you also hear that you make choices in what is not in the context at home. There is a side of the story, because we also have to look at what is within those restrictions. And here it also becomes extra interesting for agents. Because just as important as what is in it, you also have to look at what is not in it. And I get that because you get to make it pretty quickly with so-called context pollution, context fraud. And that is really a problem in the word, especially with the limited context windows. So if you give too much information, an AI agent can do that, can manage it or confuse it. For older information, that can lead to wrong decisions and sensitive information that you do not hear at home. You can just deliver safety risks. So it really means that you have to filter smartly, sometimes have to compromise. And ask yourself what information is relevant for these specific tasks. What business history does not have to be kept? What systems can the AI agent do and not approach? And what can be done together without losing important details? Well, the quality of this context determines directly how confident your AI system is. In previous episodes about the future of AI agents, look at the show notes, we talked about how these systems are going to take more and more tasks from us. But that can only be done if we have the right context to perform those tasks. Optimized for the technical reality and the limits of the context of the word and tokens. So what concerns me is context engineering, which is perhaps the most important ability to develop if you really want to benefit from AI. But as always, AI is not the solution to any problem, but it is possible where it fits. If you want to listen again, do you find this episode interesting? Share it with friends, colleagues, family. We really like to find out what happens next time.