The AI Cheat Sheet for 2026 — 10 Concepts Every Engineer, AI Engineer, and Data Scientist Must Know

Speaker 1 So I was sitting there yesterday, right, just watching my terminal and I had this, this moment. Speaker 2 Oh yeah, like a good moment or a bad 1A. Speaker 1 Weird one, I was watching an AI coding assistant. One of those tools like Cursor or Devin and it was editing my files. I literally just sat back, took my hands completely off the keyboard and just watched it read my prompt. Speaker 2 Right, doing its thing. Speaker 1 Yeah, but then it paused. It decided entirely on its own to call a read file function like I physically saw the shell command execute in the logs. It viewed the contents of my React component, thought for another second, called edit file and generated a DIF. Speaker 2 Standard agentic loop. Speaker 1 But then it gets crazier. It actually ran the linter, saw the stack trace of the air that it just created pop up in the terminal buffer, and then it went back and fixed its own work. And I'm just like, OK, let's unpack this because it looks like magic. Speaker 2 It definitely feels like it's sometimes. Speaker 1 Or at least, you know, it looks exactly like human reasoning. But my brain, like the software engineer part of my brain, knows that underneath it all, it is just code. Speaker 2 It is exactly just code. There's there's absolutely no magic here. And honestly, to the software engineers, the ML engineers, the infrastructure folks, and the data scientists listening to this deep dive right now, you do not need a PhD to understand what just happened in your terminal. Speaker 1 Thank God because my math is rusty. Speaker 2 Right. You don't need to be able to manually calculate like partial derivatives on a whiteboard. You don't even need linear algebra. You already have all the mental models required to understand exactly how that system works top to bottom. Speaker 1 Well, on that note, welcome to System Design Deep Dive Session 101. This is the official kick off to our new series AI Internals Decoded, and I'm going to be completely honest with you. Speaker 2 Late on me. Speaker 1 I have shipped distributed systems for years. I know my way around the database message queue, you know a load balancer. I can debug a nasty race condition. But when I read AI research papers. Speaker 2 Eyes glaze over. Speaker 1 Completely. My eyes completely glaze over at the math, the Greek letters come out, the summation symbols start stacking up, and I just lose the thread entirely. So today I am going to be the proxy for you, the listener. Every single concept we cover today, I want it explained in systems language. Speaker 2 I love that constraint, yeah. Speaker 1 Things we already know. AP is caches, compilers, event loops, pipelines. If we start throwing around terms like gradient or attention without explaining the actual physical computing mechanism, I'm going to push back hard. Speaker 2 And you should. That is exactly the goal of this entire series. We are going to map the seemingly mystical world of artificial intelligence directly onto standard software engineering architecture. Speaker 1 So what's the road map look like? Speaker 2 O to set the road map today is the field guide. We are basically building the mental scaffolding, but over the next few sessions we are going to dive incredibly deep into the underlying hardware and infrastructure. Speaker 1 OK, where do we start? Speaker 2 In entry one O 2, we are going to look at how GPU's physically shape the transformer architecture. We'll look at the actual silicon layout. Speaker 1 Which is fascinating, by the way, because we usually think software dictates hardware, right? But here it's completely the inverse. Speaker 2 Precisely the hardware constraints force the software invention. Then in entry one O 3 we will breakdown the KV cache, continuous batching and page detention. Speaker 1 Page detention sounds like memory management. Speaker 2 It is. We will use the operating system virtual memory analogy, you know, pages, page faults, physical memory mapping to explain exactly why your second prompt to an AI is just so much faster than your first. Speaker 1 Oh I am really looking forward to that one. Memory fragmentation is something every back end engineer has had absolute nightmares. Speaker 2 About for sure, and it's the exact same nightmare in AI, just with a different coat of paint. Speaker 1 Then moving on, in entry 104, we will tackle mixture of experts or Moe and speculative decoding. Speaker 2 What's the system equivalent there? Speaker 1 We will explain how Frontier models rate requests internally using consistent hashing. Oh. Speaker 2 Wow, consistent hashing. Yep, something literally any engineer who has built a distributed cache already knows. And finally, in entry one O 5 we'll get into agents in memory. We will talk about building L2L3 and disk storage abstractions right on top of AI context windows. OK. Speaker 1 I am so in, but before we get into the actual architecture of these systems today, you told me right before we started recording that you have one singular meta analogy. I do 1 anchor point that ties this entire series together. What is it? Speaker 2 Weights are the compiled binary. Training is compilation, inference is execution. Speaker 1 Wait, what? A binary file? So when I hit an open AIAPI, or like when I send a prompt over to a massive Frontier model running on my local machine, they're basically just executinga.sofileora.exe. Speaker 2 Yes, literally. Think about traditional programming, what Ongridge Karpathy called software 1.0. You write source code in C++ or Rust. You write the discrete logic. If this then that. Speaker 1 Right deterministic rules. Speaker 2 Right, exactly. Then you run a compiler like GCC or LLVM. The compiler translates your logic into a binary. That binary is just a sequence of numbers, machine code sitting on a disk. You don't read the binary, you don't open it in vim and start editing the ones and zeros. You just execute it. Speaker 1 Right, because if there's a bug, I go back to the source code, fix it, and recompile the whole thing. Speaker 2 Exactly. Software two point O works exactly the same way, but the inputs and the compiler change. Your source code is no longer human written logic. Your source code is your training data. Billions of examples of text code and images. Speaker 1 And the compiler. Speaker 2 The training algorithm, specifically back propagation, is the compiler. It churns through that source code for months, and the output, the output is hundreds of gigabytes of floating point numbers. That is your compiled binary. You don't read it, you just execute it. Speaker 1 OK, let's test this analogy. Let's look at the actual artifact itself. Which brings us to concept one. We always hear about the quote UN quote model. If I download an open source model from a repository, what am I actually downloading? Because in my head a model is some massive, highly complex Python class with a million custom methods. Speaker 2 It's actually much simpler and honestly much Dumber than that. A model is just two glued pieces. You have the architecture and you have the weights. Speaker 1 OK, break those down. Speaker 2 The architecture is the wiring. It is the physical data flow graph. And what's fascinating here is that the architecture is almost completely standardized. Now whether you are looking at open source models you can run on your laptop or massive proprietary ones behind AP is the wiring is mostly the same causal decoder transformer across the board. Speaker 1 Wait wait, so the code executing the proprietary models and the open source models is basically the same? Speaker 2 Fundamentally, yes. The actual code that defines the network, the matrix multiplications, the activation functions, it's maybe a few 100 lines of π torch, that's the execution engine. But the weights, the learned numbers. That is what differentiates 1 Frontier model from another. The weights are the payload. Speaker 1 And we're talking about a massive scale here. Speaker 2 Tremendous scale. We are looking at models ranging from 7 billion parameters up to 2 trillion parameters. And these parameters are literally just numbers. They're typically stored as 16 bit or 8 bit floats, FP16 or BF16. Speaker 1 So how big is the actual file? Speaker 2 To put that in perspective, a 70 billion parameter model stored in 16 bit precision takes up about 140 gigabytes of space. That entire 140 GB file. The model is the compiled binary. The weights are simply the bytes inside that binary file. Speaker 1 So if I like somehow open that 140 GB file in a hex editor. Speaker 2 You would just see endless floating point numbers. Billions of them. No strings, no text, no discrete logic. Just numbers that dictate how much signal should pass from one node in the data flow graph to the next. Speaker 1 OK, I'm tracking if the weights are the binary file, the compiled artifact, how exactly do we get them? That's concept too, because you said the compiler is the training algorithm, but let's look at the system dynamics of that build process. Speaker 2 Sure. Let's look at the pipeline. Speaker 1 Right, because in traditional software I run make or NPM run build, it spins up my CPU for maybe 30 seconds and I have a binary. I assume building 140 GB binary of pure math is a a bit more intense. Speaker 2 Just a bit. If we keep our systems frame training and inference are just two entirely different operational phases of the software life cycle with vastly different hardware requirements. Speaker 1 Let's start with training. Speaker 2 Training is the compilation. It is a one time offline massive batch job. You are not running this on your MacBook, you're running this on a billion dollar GPU cluster, 50,000 GPU's wired together with massive InfiniBand networking, and it takes months of wall clock time. Speaker 1 Months. What is the system actually doing for months? Speaker 2 It is balancing the compilation process according to some very strict mathematical rules, the most famous of which are the chinchilla scaling laws. Speaker 1 OK, I've seen that term thrown around a lot on Twitter and blogs. What does Chinchilla actually dictate for the build process? I assume it's some sort of optimization formula. Speaker 2 Exactly. It dictates the optimal allocation of your compute budget. When you are burning hundreds of millions of dollars on electricity and GP rent, you need to know you are getting the smartest possible model. The formula is roughly C = 6 ND. Speaker 1 Let's write that out mentally. C = 6 * n * d. Speaker 2 Right, where C is your total compute budget and floating point operations, N is the number of parameters in your model, and D is the data size and token. Speaker 1 OK, 6 ND let's break that down. Why 6? Where does that magic number come from? Speaker 2 The six comes from the actual hardware math of the forward and backward passes during training. For every single parameter for every single token of data it sees, the GPU has to do roughly 2 floating point operations on the forward pass to make a prediction. OK, that's two, and then about four floating point operations on the backward pass to calculate the error and actually update the weight. So 2 + 4 is 6 operations per parameter per token. Speaker 1 Got it. So C = 6 and D What does that equation actually tell a systems architect when they are designing this batch? Speaker 2 It tells you that if you want to increase your model size, say you want to double the number of parameters to make it smarter, you cannot just use the same amount of training data. You have to proportionally increase your training data size to actually get optimal performance. If you double N, you must double D They have to scale together. Speaker 1 What happens if I don't? Like? What if I build a massive 100 billion parameter model, but I only train it on a really small data set just to save time? Speaker 2 You get an under trained model. It's essentially A bloated binary. You have all these weights, but they haven't seen enough data to actually settle into useful configurations. You wasted memory and compute. Speaker 1 And the reverse. Speaker 2 Conversely, if you have a tiny model and you train it on way too much data, it stops learning. It runs out of physical capacity to store the underlying patterns. The Chinchilla law basically says you are balancing the size of your final binary against the size of your source code repository during the compilation phase to get the absolute maximum intelligence per dollar spent. Speaker 1 OK, so that's the batch job, the massive $100 million compilation step. What about inference? That's what happens when I actually use the thing, right? Like when I hit enter on my prompt. Speaker 2 Exactly, Inference is execution. It's the online per request execution. And from a systems engineering perspective, inference is fascinating because it actually has two distinct stages that behave very differently in the hardware. Speaker 1 Lay them out for me. Speaker 2 First you have the prefill stage. This is when the model reads your prompt. Say you paste in a massive 10,000 word server log file and say find the bug. The prefill stage processes that entire log file at once. Speaker 1 All at the same time. Speaker 2 All at once. This stage is heavily compute bound. The GPU's are doing massive matrix multiplications in parallel. The silicon is red hot, running at maximum floating point capacity. Speaker 1 Because it can look at all the words in my prompt simultaneously, there are no sequential dependencies. Speaker 2 Yes, there are no dependencies in the prompt. It knows the whole input. But then you transition to the decoding stage. This is where it starts generating the answer one word at a time. And decoding is a completely memory bound. Speaker 1 Why? Generating text seems like it should be computationally expensive. It's creating new information. Speaker 2 Think about the data movement. To generate a single word, the GPU has to read the entire model, all 140 gigabytes of weights from its high bandwidth memory, pull it into the actual compute cores, do the math, spit out one single word, and then discard the weights. Oh wow. Then to generate the next word has to pull all 140 gigabytes across the memory bus again. Speaker 1 Wait, really? It loads the entire binary into the processor just to output one word. Speaker 2 Yes, that is the fundamental bottleneck of autoregressive generation. It hits what hardware engineers call the memory wall. The compute cores are actually sitting idle most of the time, just waiting for the memory bus to finish shuttling the weights back and forth. Speaker 1 So if decoding is bottlenecked by literally just moving 140 gigabytes of numbers around memory chips, how do we ever run these things on a smartphone or a laptop? Because my phone definitely does not have the memory bandwidth of a massive data center GPU. Speaker 2 This raises a critical systems challenge, and the answer introduces model compression. If we want to fit this massive binary on a smartphone for edge AI, we have to shrink it. We do this primarily through quantization. Speaker 1 Quantization. Is that like when you compress an audio file and drop the bytrate? Speaker 2 That is the perfect analogy. Instead of storing our parameters as high precision 16 bit floating point numbers, we round them down. We compress them into 8 bit integers or int 8. We are literally chopping off the decimal precision. Speaker 1 So we just dropped the trailing decimals. Speaker 2 Yep, and recently we've even pushed it to 4 bit integers or INT 4. Speaker 1 But doesn't that destroy the model? Like, if you take a beautifully precise math equation and just round all the numbers to the nearest integer, shouldn't the output just turn to garbage? Speaker 2 You would honestly think so, but neural networks are surprisingly resilient to noise. It turns out the relative magnitude of the weights matters more than their absolute precision. Interesting. It's like a heavily compressed JPEG. If you compress it too much, it gets artifacting, but you can still tell it's a picture of a dog. By moving from 16 bit floats to 4 bit integers, we reduce the memory footprint by 4X. That 140 GB model suddenly fits in 35 gigabytes of RAM. Speaker 1 And I guess we also speed up the decoding because the memory bus only has to move 35 gigabytes for every word instead of 140. Speaker 2 Exactly, you just solved the memory wall problem for edge devices. The other technique we use is pruning. This involves analyzing the neural network and finding connections that have a weight of 0 or very close to 0. We just sever those connections entirely. Speaker 1 So pruning is like stripping the debug symbols and dead code paths from a binary to shrink its footprint. Speaker 2 Spot on. You strip the debug symbols, you compress the assets via quantization, and suddenly that massive cloud binary can execute entirely on a local embedded device. Speaker 1 OK, I want to back up to concept 3. Let's look at the compiler itself, the actual mechanism that builds this binary. We mentioned back propagation and training, but I really want to dejargonize the math here. When I read about loss and gradient descent, it sounds like some multidimensional calculus nightmare that only a mathematician could love. Explain this to me in systems terms. Speaker 2 Let's strip all the Greek letters away. At its core, any compiler needs a feedback mechanism. When you compile traditional code, you get syntax errors, right? Right. Speaker 1 It fails and tells me line 42 is broken. Speaker 2 In AI, the compiler needs to know how wrong was the model's guess. That is all loss is. It is just an error metric. Speaker 1 So it's just a percentage like you were 80%, right? Speaker 2 It's a bit more nuanced. We use something called cross entropy loss. You feed the model and input, say, the first half of a sentence, and it predicts A probability distribution for what the next word should be. Speaker 1 So it ranks the possibilities. Speaker 2 Yes, it says. I am 10% sure the next word is Apple, 50% sure it is run, and .1% sure it is the. Then you look at the actual truth in your training data. Speaker 1 And if the true next word was the, the model was very wrong. Speaker 2 Exactly. The gap between its prediction distribution and the actual ground truth is the loss. It's a quantifiable number representing how shocked the model was by the real answer. And in systems engineering we actually divide this into two categories, reducible loss and irreducible loss. Speaker 1 Reducible, meaning errors we can actually fix. Speaker 2 Yes, reducible loss is the gap we can close by training longer or using a bigger model, but irreducible loss is just the inherent noise or ambiguity in the data itself. Speaker 1 Give me an example of irreducible. Speaker 2 If a sentence genuinely could end with five different words that all make grammatical sense, the model will never be 100% perfectly predictive. That's irreducible. You can't optimize away human unpredictability. Speaker 1 OK, so loss gives us the error rate, a numeric value of wrongness. But how does that actually fix the model? If my CICD pipeline fails, I have to go in and manually rewrite the logic. How did the system automatically rewrite its own binary? Speaker 2 That is gradient descent. If loss asks how wrong are we? Gradient descent asks which direction do we tweak the parameters to reduce the wrongness. Speaker 1 Walk me through the mechanism. How does it know which of the 70 billion parameters to tweak? Speaker 2 Through the chain rule of calculus, which in systems terms is basically just a massive reverse trace. Let's trace it. The model spits out a wrong answer. We calculate the loss. We then take that error value and propagate it backwards through the data flow graph. At every single node, at every single matrix, we calculate the gradient. Speaker 1 And a gradient is just. Speaker 2 Slope. It's a multi dimensional vector that tells you the steepest path downhill. For any specific parameter, let's say weight #5 billion. The gradient tells us if I increase this weight by .001, will the final air go up or will it go down? Speaker 1 So if increasing the weight makes the air go down, you bump it up. If increasing it makes the air go up, you dial it down. Speaker 2 Exactly. You do this for all 70 billion weights simultaneously. You take a tiny step downhill, and then you do it again. You fetch the next piece of training data, make a prediction, calculate the loss, trace the error backwards, calculate the gradients, and tweak the 70 billion weights again. Speaker 1 So it's an iterative loop. It's just brute force. Speaker 2 Yes, Gradient Descent is a compiler with a feedback loop. Think of it like your continuous integration pipeline. It runs your test suite. The test fail. But imagine a magical script that automatically tweaks your source code based on the stack trace and runs the CI again. And it does this over and over billions of times, continuously nudging the code until your tests finally pass. That is what training is. Speaker 1 That is wild to think about. It's algorithmic trial and error on a massive scale. Which brings us to concept 4, the actual stages of this compilation, pre training, fine tuning and RLHF. Because the reality is as a software engineer, I am not compiling the entire operating system kernel from scratch every time. I just want to write a simple Python script. Speaker 2 Exactly. Building this binary happens in very distinct phases with completely different economic costs. Pre training is the $100 million step. This is the massive batch job we were just discussing. This is where the model learns the foundational structure of language, logic, reasoning, and code. Speaker 1 What data goes into pre training? Like physically where are we getting the source file? Speaker 2 Everything raw web crawls, every public book, all of Wikipedia, every open source repository and GitHub. It is reading trillions of words and a huge systems challenge here. Arguably the biggest challenge for the data engineers is data deduplication. Speaker 1 Why is deduplication so critical? I mean, if I have duplicate rows in my database, it's annoying. It wastes a little storage, but it doesn't break the system. Speaker 2 In neural networks, duplicate data completely warps the compilation. If your pre training data has 1000 copies of the exact same boilerplate MIT license text, the model will encounter that text 1000 times during gradient descent. Speaker 1 Oh I see. It thinks it's super important. Speaker 2 Yes, it will over index on those specific patterns. It will dedicate too many parameters to memorizing that license and it's generalized performance on. Other tasks will degrade. You are essentially building the base image of your docker container. It needs to be perfectly balanced. Speaker 1 OK, here's where it gets really interesting to me. O if retraining builds the massive heavy base image, what is fine tuning? Because I keep hearing developers talk about fine tuning models for their specific apps. Is fine tuning like forking the binary? Speaker 2 It is exactly like forking the binary. Fine tuning is the cheap step. You take that massive 140 GB pre trained base model. It already knows English, already knows Python, but now you want it to do a very specific task like formatting customer service transcripts into strict Jason objects. Speaker 1 But I definitely don't want to spend $100 million running back propagation across the entire Internet again just to teach it, Jason. Speaker 2 Right, so you just show it a few 1000 high quality examples of your specific Jason format and you run the compiler feedback loop again, but only for a few hours on a single GPU. But here's the trick. We don't even want to recompute all 70 billion weights during this fine tuning step. It's still too memory intensive for a consumer GPU. So we use a technique called Laura, which stands for Low Rank Adaptation. Speaker 1 OK, I see Laura mentioned on every AI forum. How does it work physically? Don't gloss over the math, give it to me in systems terms. Speaker 2 Think of Laura as a hot fix patch. In linear algebra, the weights of a neural network layer are represented as a massive matrix, say a 10,000 by 10,000 matrix. That's 100 million parameters for just one layer. If we update all of them, we have to store a brand new 10,000 by 10,000 matrix. Speaker 1 Which is huge if I have 50 layers. I'm basically duplicating the model size for every fine tune. Speaker 2 Exactly. Laura freezes the original binary. You don't touch the 140 GB file, it is read only. Instead, Laura says what if the changes we need to make to this matrix can be represented by a much smaller amount of information? It decomposes the update into two tiny matrices. Instead of learning a 10,000 by 10,000 update, it learns a 10,000 by 8 matrix and an 8 by 10,000 matrix. Speaker 1 Wait, let me do the math 10,000 * 8 is 80,000, so two of those is 160,000 parameters. Speaker 2 Yes, you just went from needing to update 100 million parameters down to 160,000 parameters. You train this tiny separate set of weights, maybe just a few 100 megabytes in total. It acts as an adapter. Speaker 1 And how do they combine? Like when actually run the prompt. Speaker 2 At runtime, during execution, the engine takes the output of the massive frozen matrix and simply adds the output of your tiny adapter matrix to it. You literally just add the two vectors together. You applied your tiny patch on top of the frozen base binary, altering its behavior entirely. You patch the artifact without recompiring from source. Speaker 1 That is brilliant and honestly that explains why open weight models are such a massive disruption to the industry. The cost asymmetry is insane. A massive tech lab spends $100 million paying for the compute cluster to build the base image. They open source the base binary and then I just some random developer can spend 50 bucks on a rented cloud GPU to fork it, freeze it, and patch it with Laura for my specific enterprise use case. Speaker 2 Precisely, it democratizes the application layer completely. But there is one more critical step to make the base model actually usable as a chatbot or an assistant, and that is RLH reinforcement learning from human feedback. This is the Polish. Speaker 1 Why do we need it though if it already knows everything from the pre training? Shouldn't it just answer my questions? Speaker 2 No, because a raw pre trained model just wants to predict the next word on the Internet and the Internet is a very weird place. If you prompt a raw base model with the question How do I reverse a linked list? It might reply with How do I reverse a binary tree? Speaker 1 Why would it do that? That's totally useless. Speaker 2 Because on sites like Stack Overflow or Reddit, a list of questions often follows a question like FAQ page. The model isn't trying to be helpful, it's just continuing the pattern of a forum post. RLHF forces the model to shift from document completer to helpful assistant. Speaker 1 OK, but how do you program the concept of helpful into a binary? Helpful isn't a mathematical objective function, it's entirely subjective. Speaker 2 That's the genius of it. We use frameworks like Iterated Distillation and Amplification, or ID. First, you get thousands of humans to interact with the model. The model gives two different answers to a prompt, and the human simply clicks thumbs up or thumbs down on the better one. Speaker 1 But you can't have humans sit there and grade billions of interactions. It doesn't scale. Speaker 2 Right, so you use those human grades to train a second, completely different, smaller AI model. This is called the reward model. The reward models only job is to look at a prompt and a response and output a score predicting how much a human would like it. Speaker 1 Oh wow, so you build a simulation of a human grader? Speaker 2 Exactly. Then you let your main model generate thousands of responses, and you have the reward model automatically grade them instantly. If the main model gets a high score, you use reinforcement learning algorithms to tweak the main model's weights. To do more of that, you are distilling human preference into a robust automated testing suite. Speaker 1 That is fascinating. OK, we've spent a lot of time on the artifact itself. We know exactly how the binary is built, how it's patched with the lore, and how it's aligned with RLHF. But let's look at the inputs. This is concept 5. What data type does this execution engine actually process? Because I know mechanically you cannot multiply a floating point matrix by the letter A. It's not raw text. Speaker 2 Yes, that is a fundamental barrier. Neural networks only do math, so we have to convert text into numbers, and the fundamental unit of computation we use is the token. When you pass text into the model, it gets chopped up using an algorithm called Byte Pair Encoding or BPE. Speaker 1 I have always wondered about this. If we need numbers, why not just map every word in the dictionary to an ID? Like apple is 1, banana is 2. Or alternatively, why not just feed it raw ASCII characters? A is 65, B is 66. Why this weird middle ground of tokens? Speaker 2 If you connect this to the bigger picture of system memory management, it makes perfect sense. Let's look at the word approach first. If you map every word to an ID, your vocabulary blows up to an unmanageable size. Think about all the words in English. Now add French. Now add every variable name on GitHub. Now add every typo ever made on Reddit. Speaker 1 Right, it's infinite. Speaker 2 Millions and millions of unique IDs, the embedding tables, the memory structures required to store all those unique words would be too massive to fit in GPU RAM. Speaker 1 OK, so a word level vocabulary is out due to memory constraints. What about character level? That are only 256 risky characters? That's a tiny vocabulary. Speaker 2 A character level vocabulary is highly memory efficient, but it destroys your compute efficiency. Characters are too small to carry any semantic weight. The letter P has no meaning on its own. The model would have to waste massive amounts of its computational depth layer after a layer of matrix multiplication. Just figuring out that APP forms a fruit. Speaker 1 Right, it's doing extra work just to read. Speaker 2 Exactly. Furthermore, your sequence length would be huge. A-100 word paragraph might be 600 characters, and as we'll see later, processing long sequences is computationally lethal. Speaker 1 So tokens are the Goldilocks solution. Speaker 2 Exactly. Tokens are the compromise. Byte pair encoding starts with characters and iteratively merges the most frequent pairs into chunks. So common words like the OR apple become a single token, but a weird rare word like supercalifragilistic might get chopped into 3 or 4 tokens. In English, a token averages out to roughly 4 characters. The total vocabulary size is usually capped between 50,000 and 200,000 unique tokens. Tokens are the bytecode of language. Humans write words, which is the source code. The model executes tokens, which is the bytecode. Speaker 1 That clarifies so much, and from a developer cost perspective, looking at API dashboards, I always notice I get billed way more for the tokens the model generates than the tokens I send it with a discrepancy. A token is a token, right? Speaker 2 It comes right back to the execution mechanics we talked about earlier, prefill versus decoding. Reading your input tokens happens in the compute bound prefill stage. The GPU processes your entire prompt in parallel. It just chunks the whole array of tokens through the matrix math at once. It is fast and efficient. Speaker 1 Generating output tokens. Speaker 2 Generating output tokens uses autoregressive decoding. It has to generate token 1, append it to the sequence, run the entire sequence through the model. To generate token 2, append it. Run the entire sequence. To generate token 3, it hits the memory wall. Output tokens cost three to five times more computationally because the process is inherently sequential, unparallelizable, and memory bandwidth constrained. Speaker 1 That makes total sense. All right, we have our tokens. We have chopped our text into byte code IDs. But we still have a problem. This is concept 6. To the computer, a token is just an integer ID. Token ID four O 5 might be the word dog. Token ID 8090 might be puppy. How does the model know what a token actually means? Because four O 5 and 8090 have no mathematical relationship, how does it relate them? Speaker 2 It transforms those integer IDs into high dimensional vectors. This is called an embedding. Typically an embedding is an array of 768 or in larger models 15136 floating point numbers. These numbers represent spatial coordinates in a massive multi dimensional semantic space. Speaker 1 OK, so it takes the integer 405, looks it up in a massive table, and pulls out an array of 1536 floats. It's literally plotting the word on a massive graph. Speaker 2 Yes, and the unbreakable rule of this graph is that semantic similarity equals spatial proximity. Let's take dog and puppy. They have completely different integer token IDs. They share no letters, but in the 1536 dimensional embedding space, their coordinate vectors are almost identical. They're plotted right next to each other. Speaker 1 How did they get plotted there? Is there an engineer manually assigning these coordinates? Like saying puppy goes here, dog goes here? Speaker 2 No, it's entirely emergent from the pre training compiler run. During gradient descent. The model noticed that dog and puppy were used in the exact same surrounding contexts. It's. I walk the dog on a luge and I walk the puppy on a leash. It saw they were both associated with Bark Park and Vet because they share the same context hole in sentences. The back propagation algorithm naturally nudge their coordinates closer and closer together in order to minimize the loss I. Speaker 1 Love this. Wait, let me just make sure I had this conceptually right because I've seen this specific math trick online. If I take the coordinate vector for king, subtract the vector for man and add the vector for woman. Speaker 2 You land exactly on the coordinate vector for queen. The geometry of the space perfectly encodes human semantic relationships. Embeddings are the symbol table for meaning. In traditional compilers, a symbol table maps your variable names to physical memory addresses. In AI, embeddings map discrete integer tokens to continuous coordinates in semantic space, allowing the system to perform fluid math on concepts. Speaker 1 OK, so we've mapped our tokens to coordinate vectors, but language isn't just isolated words, it's sentences. Context changes everything, which is concept seven. If I say the Bank of the river versus the bank on Wall Street, the token bank has the exact same starting coordinates, right? Because it's the same token ID, correct? So how does the system alter the meaning based on the surrounding words? More complexly, if I say the cat sat on the mat because it was warm, how does the system know that the word it refers to the mat and not the cat? Speaker 2 You just hit on the exact problem that stumped AI researchers for decades, and the solution to that problem is the core innovation of the transformer architecture. It's the attention mechanism. Speaker 1 Breakdown attention for me. How does it physically work? Speaker 2 Attention evaluates how every single token relates to every other token in the sequence simultaneously. It does this by creating 3 new vectors for every token, a query, a key and a value. Speaker 1 Query key value. This sounds exactly like a database look. Speaker 2 Up. It is functionally a soft probabilistic database. Look up. Let's take your sentence. The cat sat on the mat because it was warm. Let's look at the token for it. The model generates a query vector for it. Think of the query as asking I am a pronoun. What noun do I refer to? Speaker 1 OK, so it broadcasts a query. Speaker 2 Exactly. Meanwhile, every other token in the sentence is broadcasting a key vector. The key vector is a description of what that token contains. The word Matt broadcast the key saying I am a noun, I am an inanimate object. I can have a temperature. Speaker 1 And how did the query and the key interact? Speaker 2 Dot products. The hardware calculates the dot product, which is a mathematical measure of similarity between the query vector of it and the key vectors of every other word. It compares it to cat. It gets a low score because cat is animate and warm usually applies to the resting place. In this context, it compares it to Matt. The dot product is massive. It's a match. Speaker 1 So what happens when it finds the match? Speaker 2 It uses that match score to multiply against the third vector, the value vector. The value vector contains the actual semantic meaning of Matt. The system takes a heavy percentage of the Matt value vector and physically adds it to the representation of the word IT. By the time the word IT exits the attention layer, its embedding has been completely modified. It is no longer just a generic pronoun, its coordinates have shifted. It now mathematically carries the context of matte and warm inside of it. Speaker 1 So in database terms. Speaker 2 Attention is scatter gather. It is a dynamic sequel. Select late where clause executed for every single token at every single layer of the network to pull relevant meaning forward and combine it. Wait. Speaker 1 If it evaluates every token against every other token, that sounds incredibly computationally expensive. Speaker 2 It is the primary bottleneck of the entire architecture. Attention scales quadratically O of n ^2. If you double the length of your text prompt, the attention computation quadruples. If you pass in 1000 tokens, it's a million dot products. If you pass in 100,000 tokens, it's 10 billion dot products per layer per attention head. Speaker 1 Ten billion operations just to route the context of a 100K token prompt? How is that even physically possible without melting the server? Speaker 2 It requires massive hardware level tricks to make it viable. This is where systems engineering saves the math. The most famous optimization is Flash attention. Speaker 1 I see flash attention on the change log of every new open source model. What is it actually doing? Speaker 2 Flash attention is a pure memory hierarchy optimization. Normally, calculating those ten billion attention scores requires writing intermediate matrices back and forth to the GPU's main memory, the High Bandwidth Memory, or HBM. But each BM, while fast compared to standard RAM, is incredibly slow compared to the GPU's extremely fast on chips RAM cache. Speaker 1 So it's an IO bottleneck, like reading from disk versus reading from memory. Speaker 2 Exactly. Flash attention uses a technique called operator fusion and tiling. It breaks the massive end by end matrix into blocks. It loads a block into the Super fast RAM cache, computes the query key dot products, immediately, applies the softmax function, multiplies the value, and writes only the final result back to HPM. It fuses the operations together so it never has to materialize that massive 10 billion float intermediate matrix in the slow memory. It avoids those expensive GPU memory round trips entirely. Speaker 1 That is insane. It's literally just rewriting the order of operations to stay within the L1 hardware cache limits. But even with flash tension, there has to be a hard limit. You can only scatter gather across so much data before you literally just run out of physical memory. Which brings us perfectly to concept 8, the context window. Speaker 2 The context window is the maximum number of tokens the model can hold it once. Depending on the model, this ranges from a meager 4000 tokens on older models to 128,000 tokens up to over 1,000,000 tokens on frontier models today. Speaker 1 So what is the system analogy for the context window? Speaker 2 Context window is L1 cache. It is the working memory of the model. It is bounded. It is highly expensive to compute across, as we just established with the quadratic scaling, and it must be managed carefully by the application developer. Speaker 1 Wait, I want to push back on this analogy. L1 caches on C GPUs are measured in kilobytes, maybe a few megabytes. But you just said frontier models have a context window of 1,000,000 tokens, 1,000,000 tokens mapped to 5000 and 36 dimensional float vectors is gigabytes of data. How can you call that an L1 cache? Speaker 2 It's an L1 cache conceptually for the intelligence engine. Obviously it sits in HBM or system RAM physically, but functionally to the model it behaves exactly like an L1 cache. It is the only data the execution engine has immediate synchronous access to. If it's not in the context window, it does not exist to the execution loop. Speaker 1 OK, so when I'm using a chat bot or an AI coding assistant and an hour into the conversation that completely forgets an instruction I gave it at the very beginning of the session. Speaker 2 It's a cache eviction you slid out of the context window. Explain that when you hit the limit, say 128,000 tokens, the system has to make room for your new prompt. Most chat applications just use a sliding window protocol. They drop the oldest messages from the payload. Your initial system instructions were literally pushed out of the buffer. People anthropomorphize AI constantly. They think it has biological long term memory. They get frustrated and say why can't you remember what I told you? It does not have memory, it just has a sliding buffer that gets overwritten, which. Speaker 1 Perfectly leads to the biggest issues software engineers have with these systems. Concept 9, The ultimate bug hallucination. If it has no long term memory and it only knows what is currently injected into its L1 cache, how does it answer factual questions at all? And why does it so confidently make things up? Like I will literally ask an AI for an AWSAPI endpoint that doesn't exist and it will give me the endpoint URL, the required parameters, and beautifully formatted example code. It lies with total conviction. Speaker 2 Because of the fundamental nature of what we built, the model is a probability machine, not a fact database. Speaker 1 Say more about that because as a I expect deterministic systems. Speaker 2 It doesn't look things up in a SQL table. There is no select from osatilochis where service equals S3 when you ask it a question. It is just executing autoregressive decoding. It is calculating the highest probability next token based on the massive generalized patterns encoded in its weights during pre training. Speaker 1 So when it invents a fake API. Speaker 2 If you ask for an API endpoint, it generates tokens that structurally look exactly like an API endpoint. It knows that URLs usually start with Https://followed by a sub domain followed by Amazon, oz.com. It knows parameters are usually camel case. It is optimizing for fluent, structurally correct text, not for empirical truth. Speaker 1 So it's not a bug. Speaker 2 In ML engineering terms, we look at this through the lens of distribution shift and Gulmus generalization. The compiler back propagation trained the model to minimize the loss on predicting the next word. It was rewarded for sounding plausible. It was not trained to say I don't know when its internal confidence is low. It has no default failure state. It will just keep generating the most probable sequence of characters. Speaker 1 So it's the system working exactly as designed. It is generating highly probable bytecode based on the context. If I want hard facts, I shouldn't be querying the weights. Speaker 2 Exactly. The weights encode reasoning and structure, not specific trivia. If you want facts, you cannot rely on the binary. You have to inject those facts into the context window at runtime. Speaker 1 Which is the perfect transition to the final piece of the architecture Concept 10? If we can't trust the internal facts of the binary, how do we make these things? Do real, reliable software engineering work? How did that agent in my terminal actually edit my file? Speaker 2 We connect the model to the outside world. This is where we build infrastructure around the model. The most common pattern is RAID retrieval augmented generation. Speaker 1 How does Rd. read map to systems? Speaker 2 Instead of asking the model to remember a fact from its weights, we use a traditional database. We intercept your prompt, we run a vector search against a database of document embeddings, we retrieve the top five most relevant documentation pages, and we inject that raw text directly into the L1 cache, the context window right before the. Model generates its answer, we populate the cache. Speaker 1 OK, that handles reading facts, but what about the loop I saw with cursor? The read file, the linting, the edit file? The model was actually taking actions. Speaker 2 That is tool manipulation. The model itself can't execute code, it's just a matrix multiplication engine. But during fine tuning we teach the model to output a specific strict format, usually a Jason object, when it wants to take an action. Speaker 1 OK, so it spits out Jason, then what? Speaker 2 We parse that Jason in our traditional software one point O code or Python or node JS wrapper. If the model output say ACTION SEARCH QUERY error code 500, our wrapper intercepts that. The wrapper executes an external Google search API, grabs the search results, formats them into a string, appends them to the conversation history, and feeds it all back into the model's context window. Speaker 1 So it's literally just an event. Speaker 2 Loop. It is exactly an event loop. The model yields control, the host application executes the side effect, and the result is passed back to the model. For complex reasoning, systems now use advanced loops like Monte Carlo Tree Search or MCTS. Speaker 1 What does that loop look like? Speaker 2 The wrapper prompts the model to propose 3 different ways to edit a file. It spawns 3 parallel executions. It simulates or evaluates each path. The wrapper scores them based on whether they pass the linter. It discards the bad paths, takes the successful 1 and loops again until the task is complete. Speaker 1 So what is the grand systems analogy for an agent? Speaker 2 Model plus RPC plus loop. The model is just the brain doing the text evaluation and routing. The tools are just standard remote procedure calls and the loop is your traditional event driven architecture, just like a standard node server. Speaker 1 OK, my mind is slightly blown, but I feel incredibly grounded. Let's unpack everything we just covered. Speaker 2 I'm going to rapid fire the mental models one last time so they stick. Model and weights. That's a compiled binary training and inference. Compile and execute gradient descent. A compiler feedback loop tracking the stock trace. Pre train and fine tune. Building the base container image versus adding the app layer tokens, the bytecode embeddings, the symbol table for meaning, attention, scatter, gather, query routing, context window, the L1 cache hallucination. It's a probability machine, not a fact. Database agents model plus RPC plus loop. Speaker 1 That is the whole stack from the billions of floats on the disk to the agent editing files in my IDE. So to you, the listener, here's your homework. Tomorrow morning, the next time you open an AI tool, whether it's Copilot, cursor or just a chat window, I want you to actively identify which of these architectural concets you are touching. Label the context window in your head. Spot the RPC loops when it searches the web. Realize that when it confidently makes up a fake library, it's just predicting the next most probable bytecode. Once you do this, the magic disappears, and magic is bad for engineering. Speaker 2 Next time in session 102 we look at the physical silicon, we're going to explore exactly how the architectural bottlenecks of GPU's force the invention of the transformer in the first place.