Does Claude Have Private Thoughts? (Everyone Settle Down) | AI Reality Check

Last week, Anthropic released another one of their infamous research reports. This one was titled "A Global Workspace in Language Models." And it was accompanied like all great scientific research by a lavishly produced animated movie. Now, this report, not surprisingly, soon led to some breathless excitement on X. Here's one such tweet I'll read the beginning using my best sort of scary X voice. Anthropic just admitted, they have discovered what I and many others have been claiming exist for a very long time, explicitly, clod my friends by all counts is a conscious entity. Clod my dear friends is a moral patient. All right. The traditional tech media also quickly began writing about this report using intensely anthropomorphized language in Axios headline red. Anthropic says clod has carved out its own space to ponder. The MIT tech review exclaimed, "Anthropic found a hidden space where clod puzzles over concepts." All right, so what do we do make of this report? Has Anthropic revealed evidence that their LLMs are more human-like and alive than we realized? Or like so many such reports in recent months, is this yet another overwrought cynical push to generate a fresh wave of relevance reinforcing digital-ic? Well, it's Thursday, which means it's time for a reality check episode of this podcast, which makes it the perfect opportunity to go searching for some measured answers, which is exactly what we'll do. As always, I'm Cal Newport, and this is Deep Questions, the show for people seeking depth in a distracted world. All right, so let's start by looking a little bit closer on how Anthropic describes their findings and the introduction of their paper. I'll load it up here, and I'll go down to the introduction. All right, so here's what they say. We find that clod has developed a small collection of internal neural patterns that, compared to all its other internal processing play a special role. We call the collection of these patterns, the J-space, named after the technique we used to find them, involving a mathematical concept known as the Jacobian. Each J-space pattern is linked to a particular word, but when one of these patterns lights up, it doesn't mean the model is saying that word, just that the word is on its mind. If you've heard of language models having a scratch pad or chain of thought, text they write to themselves while reasoning, the J-space is something different. It operates silently in the model's internal neural activations, allowing the model to think about a concept without writing it down. Notably, the J-space wasn't designed or programmed by us, but instead emerged on its own during clod's training process. All right, so an isolation that intro summary sounds pretty impressive, kind of like clod, this was their italics, on its own, made some sort of leap and is now behaving sufficiently human that we can't help but feel at least a little bit of digital-ick, but what's really going on here? Well, to answer this question, I'm going to start with a high-level tutorial on how large language models actually work, and I'll add a little bit more detail to it. Stick with me here, because once you understand these basics, you're then going to understand that description I just read from Anthropic in a completely different light. All right, so this is an exercise that's worth doing. All right, I'm going to start at a very high level here. At its core, a large language model like those of the phabial clod, opus, or GBT families can be described underneath the hood as a sequence of what are called transformer blocks that are arranged in layers, one follows the other, so sequential collection of layers. GBT-3, which was the last major LLM they actually published stuff about, details about, had 96 of these transformer block layers. Two LLMs probably have more, but we don't know how many more. All right, let's start at the very high level here. I submit as input a prompt that I have typed to an LLM. You can imagine that this input is going to pass through each of these layers one after another. It'll go into the first layer and come out the other end. It'll go into the second layer, come out the other end, and so on. Now, here's what's important, conceptually speaking. Those layers can add what we'll call for our own purposes annotations to that prompt. So as the input goes into the first layer, it'll come out the other side with some annotations added to it. Some information added to it that the first layer came up with in its own analysis. Now that's prompt with the first layer annotations goes into the second layer, and it comes out with even more annotations added onto it. And so on. So as it moves to these layers, the original prompt is there, but you're getting all of these annotations added layer by layer. And critically, later layers can use the annotations from earlier layers to help do their analysis, right? So it's, it's, you can imagine it's almost like you have like a bunch of long tables of scholars arranged in rows in some vast like Hogwarts, Great Hall style dining room. And you're passing this prompt from table to table and it arrives at each table. The scholars in the table pour over it, they look at the annotations from the scholars that came before them to figure out how to analyze it. Maybe the annotations from the scholars before them will let them know which of the scholars of the next table should look at it and then they add their own annotations and it passes on. The very final layer in a large language model is special. It takes this heavily annotated version of the prompt and then it maps it to what word or part of a word to output next. Never going to be superpindantic, it really outputs a probability distribution over all possible tokens and then the control program selects one probabilistically. But just think that the last layer takes all the work that all the transformer blocks did and says, this is what we are going to output next. All right, so that's what's happening at a very, at a very high level. I'm going to add a little bit more layer level of detail now, right? So now let's add a little bit more level of detail. What are these annotations and how are they written down and how are they passed from layer to layer? Okay, let's look a little bit closer at what happens to this prompt that you have of hand it on to your large language model. The first thing that happens is the prompt, which presumably you typed in with words on your keyboard, gets broken up into what are known as token, so in the pieces. And a token might correspond to a single word or a longer word might get broken up into multiple tokens that represents the different parts of the word. But we kind of just, let's break this up into this fundamental unit we call tokens. Next, those tokens get embedded into a high dimensional space and all that really means is, okay, each token is going to be described by a long list of numbers, a long list of numbers is otherwise known as a vector, so it's a bunch of numbers that are in some order. So we've got a long list of numbers to represent each token and there's something called a token embedder that takes each token, which is now like letters, right, a word or part of words and transforms it to a long list of numbers. The long list of numbers, the so-called token embeddings that come out of this embedding, the numbers that are in there at first are basically capturing the meaning or details of that token. You train in a sort of semi-supervised fashion how do these embedding? So at first, these long list of numbers are the numbers that are in this long list to sort of capturing what this word means or what this part of a word means, right? We want these things in numbers because that's what we can, that's what language models are going to, they understand because in the end, we're going to be manipulating numbers here, not letters. But here's the key thing, there's a lot of room in those long list of numbers. So they don't just encode, here's what this word means. As the, this long list of numbers, which is how we represent the prompt now, as it moves through each layer, the layers mess around with the long list of numbers for each token. And then messing around, which means messing around with the values in there. In part, that's where the annotations live. So this is what's being passed from layer to layer, is I just imagine like I have my, my prompt, it's been broken up into tokens and hanging off of each tokens is a long list of numbers. And this big, so it's a big table of numbers, we call this a matrix, as this passes from layer to layer, those numbers are being updated to include among other things, the analysis that's being done by each of the transformer blocks. There's all, there's other things that, lots of things are stored in here. Like for example, if you open up a transformer block, there's really two parts in it. There's a self-attention mechanism layer and then a, a small multi perceptron style feed forward neural network, which does the analysis. But like the self-attention layer is kind of cool. It adds information in each of these numbers, each of these lists to try to help each, each token understand how relevant each other token is. So it's like the, this is like the, the working space. Here's the way I think about it. They often think about these tokens and these long list of numbers, these, these vectors to come with each of them. It's, it's almost like if you had, it reminds me of the Talmud, right? So if you, if you look at the Jewish Talmud, what you see in there on every page, if you actually look at a, a printed one, is you have in the very center of each page, not taking up all that much space is the actual text from the, the, the Mishna, right, the, the sort of the oral law written down by Judith the Prince and the early common error. It's a small part of the page. And then surrounding it all on the page, if you look at a page of the Talmud, it's commentary. So you got the, the, the, the, the, the, the commentary around it, okay? That's how I kind of imagine, conceptually, The prompt being passed through a large language model, it's like you have a book and on each page you put in the very middle of the page, one token, right? So this book has the whole prompt, but tons of white space around each token. And as it gets kind of passed from layer to layer, scholars are writing more commentary around the token. And then the next layer gets said, and they look at that commentary and they look at that, they build on it and write more commentary. So I sort of imagine it like we're filling out the gamara around the mission in the Jewish town. So that's roughly speaking what's happening. So we got this matrix moving from layer to layer, which is really just a list of numbers for each token. And those numbers are capturing everything we've learned so far as it moves through, as it moves through these various layers, right? So that's, that's what's happening. We know like probably, so what type of analysis is happening? Well, you know, we don't know exactly because these are neural networks doing it that are trained in a supervised manner, like any neural networks from an image recognizer to a handwriting, whatever. Any sort of neural network, the whole point is you start it with random weights and you train it with lots of data till it gets good and you don't know exactly how it does it. It just does it until you really kind of open up the box and try to understand what's going on. But we know, I've written about this before in the New Yorker, I've talked to a bunch of experts about it. There's kind of like two things that are happening together with a generative AI model, like a large language model. Actually, it's learned during training. What, when outputting the next word, what words are syntactically appropriate? This is actually pretty easy to train. You can do this even with like a simple in-gram model. This is an easy thing to learn to sort of statistical nature of language. Given this sentence that stops here, what are the words that could follow that, from a syntax, like a grammar standpoint makes sense, right? So like that's kind of the easy part and that's baked in. In fact, that's probably captured like primarily in the final layer, we don't know, but probably in the final layer of a large language model. The problem is of all the words that could follow that make sense, which one makes semantic sense? So which one actually makes the most sense, given the meaning of what the prompt is asking? So not just what word would grammatically make sense here, but which word not only grammatically makes sense, but actually matches the meaning of the prompt. And that's really in my understanding where all the annotations that follow along these token vectors really are helpful. So you kind of have these two things mingled together. They're not separate systems, because this is all kind of trained and muddled together. But you kind of have these two things happening at the end when you're in a generative model, like an LLM model, is like, you know, the quick brown fox and grammatically there's lots of words that would make sense there. Like you want to put the word the next, but there's lots of things, jumped, dance, died, talked. And a lot of words make sense. But then the annotations where you're like, oh, this is a saying, this is a saying that already exists. It's a common saying and it always says, jumped. And then so of those words that would privilege jumped us next one to go, again, I'm separating out here things that are muddled together and the neural networks themselves. We don't, they're not going to work as cleanly as a human mind. But that's the way to think about it is that the annotations really help you figure out semantically the right next token output. And then the easier part, it's like baked into probably these final layers is the grammatically or syntactically what words or next tokens would be valid. So those two comes together and we get a good output. All right. That's roughly speaking what happens in a large language model. Now that we know this, let's return to the anthropic paper. Okay. I'm going to see if I can find it again here. Let's ask what, so what again, did they actually find? Okay. So they used a mathematical tool based on a linear algebra notion called a Jacobian to essentially try to understand what is in those vectors of values attached to the tokens that are moving through the layers of the large language model to their study. So we're going to look at the annotations. Now I know they're just like this a bunch of numbers, but we're going to figure out a way of trying to make more sense of what those numbers actually meant. And so using Jacobian, this is a linear algebra way that involves taking the partial derivatives of a lot of things. Like basically what they can figure out is like which combination of numbers from this giant matrix, which pattern of these numbers are seemed to be connected and imported. In other words, like having a high influence on what the ultimate output is. So they're trying to decode the annotations that are captured in numbers in these vectors. And they're able to figure out certain patterns of these numbers seem to be really influential for the ultimate output. And you can do this experimentally, if we change it, we change exactly this pattern. We're much more likely to get a change in the output than if we change other patterns. That's simple, but it's something like that. And then, and this is what's cool, through a lot of experimentation, they also try to, as much as possible, associate different patterns through a lot of trial and error experimentation with human interpretable concepts, like words or numbers or something like that. So they call this the J-Lens because it allows them to say looking at a, you know, they run a prompt through and they can kind of watch it go all the way through. And as it gets towards the end, they look at these vectors of numbers that have been evolving and updating as it moves through and they can say, we can actually decode in the English or in human interpretable way, sometimes it's not like English words, some of the annotations. We can kind of understand what some of the annotations are that these layers are using to help come up with the answer, which is pretty cool. So I'm going to go to the paper here to show an example from the paper. This table is called the J-Lens revealed some models and journal thoughts. I'll put one example up here on the screen. So this shows a prompt that says the color of the fourth, the planet forth from the Sun is. So that's the prompt. The language model needs to expand as with another word or part of a word. When they looked at the annotations that this accumulated as it moved through the large language model, they discovered that there is an annotation that corresponded to Mars, which is the fourth planet from the Sun and an annotation that corresponded to color. So what's happening here is as that prompt was moving through it, the conceptual tables of scholars that are studying this sort of Talmudic commentary and adding their own commentary to it, somewhere along the way, one of the layers said, oh, I recognize the fourth planet from the Sun that sequence of tokens, and this part of some neural network and some layer is all about planetary stuff. It's like, that's the Mars. I'm going to write Mars down on my annotation here. We're talking about Mars. That's an important annotation. And somewhere else, some other layer was like, this is asking what the color of something is. The thing that comes next is a color. This is asking about a color. Let me write that down. This is, you know, we're asking about a color. So then when you kind of get to the end of this large language model and it's like looking at all the possible words that just statistically make sense to follow this sentence, to color the planet forth from the Sun is, and all sorts of nouns there would make sense and adjectives would make sense. It looks at the annotations like, oh, we're looking for, of the words that would make sense syntactically, we need something that's a color and a color describes Mars. Hey, what is that? Don't we have that somewhere that's red? Okay. And then an output's red. And see, that's cool. It's like, oh, maybe it's not as mysterious as we thought. Like we know it annotates numerically, but when we, we're able to interpret the annotations and annotate it with Mars and color, which is like exactly like what you should do if, you know, it makes sense. So I think that was, that's pretty cool, right? So they could do that type of thing. Then what they did, which again, I think it's fun research, is like, let's mess with this. So let's, you know, freeze this matrix right before we get to that final layer, we're going to output the next word. And let's change the, if we know what these annotations are, what if we change them? It's like one of the things they did with that example I just talked about, I'll bring it back up here, right? Is they replace the numbers that corresponded with Mars in that matrix of the, the, the annotation vectors that hang off each token. And they replaced it with the sequence of values that they had discovered corresponds with Earth. So now, even though the, the prompt, you know, is the color of the fourth planet from the Sun is, when it gets to that, when it got to the final layer to output a word, it was looking for words that make sense grammatically. And then it's annotation said, Earth and color, so it output blue. So they were showing them, yeah, hey, yeah, these, and these conceptual annotations really do influence the word or part of word that are output. Another thing they did is they called it, I, I really don't like the anthropomorphize in here, they, they said they ablated some of the information that was in the, these sort of key annotations. That just means they zeroed out the values. And what they found there, ablated by the way is like a procedure you use in like an actual human or animal nervous system, or you use heat or electricity to sort of fry a, a nervous connection, which is anthropomorphizing, let's put that aside. But they would, you know, in this example, right before we got to the final layer, take those numbers together, correspond to Mars, and let's like zero them out. And what they got in those instances is you would still get grammatically correct outputs, but they weren't semantically connected anymore. So they would just be arbitrary colors, which would be equally likely you get a bunch of colors, but nothing that corresponded to Mars in particular. So it shows, right? I mean, you kind of get this very rough sense about how these things are working, it's like. some combination of like of the grammatically syntactically correct next thing we can do, which one should we choose? And we have all these annotations that help us narrow that down to be semantically correct. That's oversimplifying it, but something like that behavior emerged. So that's, you know, I think that's interesting research. All right. So what does this mean, though? Is this scary? Is it not? Is it interesting? Is it breakthrough? Like, how do we think about this? Two thoughts, I think we have to keep true in our mind at the same time. One, it is interesting research. Is it interesting because they're the first people to be able to like look at these embedding vectors and use the Jacobian to figure out patterns of values that are important for the influential, for the final output to sort of Jalen's approach. Turns out they're not the first ones to do that. So it's not like they had a breakthrough idea about doing this. I want to, I'll bring up a tweet here. This is from a you Illinois professor, Yuzhou Zang, who said, "Jay Space is really something we have been exploring since 2022. Glad to see it continues to work well at scale. Some of the related work along this direction. She lists three papers and then she lists three more papers. The point being she was kind of responding to the anthropic paper like, we've been doing this for a while with neural networks. This isn't new, but no one's ever done it on a massive language model like Cloud before because the only people who have access to the innards of a massive data, language model like Cloud are the actual companies themselves." So that's why she said, "It's great to see this being done at scale. The idea wasn't new." But so it's, anyways, I think they executed it probably well. Again, these are not, I always put quotation marks around reports because these are not computer scientists. These are, they're more press-releasy. They describe things relatively high level with lots of pretty graphs and animation, but we can't really get into the gut. So what's really happening here? But anyways, it looks like, hey, we took this idea and we applied it to these massive models people are using and that was interesting. So that's idea number one that keep in your head. But idea number two, the way that anthropic described these results and the way that they were echoed after anthropic described them, I think is incredibly disingenuous because once we understand how a large language model roughly works, like I just explained, we see the thing they were describing with their J-Lins is exactly how we've always understood large language models to work. I mean, I went back. I wrote a long article explaining large language models for the New Yorker in 2023. I used exactly that analogy of adding these annotations of the high level concepts and annotations built in other annotations, then you used that to help figure out the word 2023. 2024, I wrote another long article about the architecture for the New Yorker of the architectural language models. Again, tables of scholars adding the annotations, which then allow you to narrow down conceptually the next token to output in a way that has some semantic meaning. This is just how we understand large language models to work. It's also how we understand essentially any deep learning neural network to work. The original breakthrough work in deep learning was image recognition, like Jan LaCoon, figuring out that you could recognize handwriting better with a multi-layer neural network, then you could with other types of approaches. Again, the way we always understood those to work is that different layers were picking up different features or annotations, and they were then combining those different features later layers to try to figure out and make a good guess of what the letter was in a very robust way. This idea of we're picking up higher level features and descriptions of what we're analyzing, this is just how deep networks work, and it's how I always understood large language models. I see this as like, yeah, good. We saw with the J-Lens exactly what we expected to find. This is how we assumed large language models, what was happening in those embedding factors. They're accumulating annotations that build on each other that capture higher level conceptual meanings that help influence the token you put out, and if you change the annotations, you get different tokens. How else would this work? But if you read the press release, it's all sorts of anthropomorphizing and implication and egg generation. It talks about the global workspace model of human consciousness. It somehow implies that this seems kind of similar to that. It uses these loaded terms about pondering and puzzling, and it's thinking through these things, and it silently thoughts, and it's not things that's saying out loud, and it makes it seem like this is all icky and human-like and new, and it is what any LLM researcher you would talk to you for the last five years would say, yeah, that's how LLMs work. I mean, it's oversimplifying it to talk about it like annotations, but like, that's how they work. There's nothing new and surprising here. I mean, of course, the annotations are picking up high level features of the input to help figure out what the output next. That's how deep learning networks work. That's why you have multiple layers to get all sorts of different levels of abstraction of understanding different parts of your input. And so it's cool research, but research that most people shouldn't care about, and it certainly doesn't imply what the implicit, and again, they're careful in the paper, they're like, well, it doesn't really mean it's conscious. We don't say that, but it's interesting. I mean, look, I'm going to load this up. Let's go back to this, let's go back to the paper for a second. I don't want to go on too far here, but again, let's look at these things. The JSpace, it operates silently in the models internal network activations, allowing to model to think about a concept without writing it down. What does that mean? It's thinking about a concept without writing it down. The way deep networks work is that you figure out different features, which help you understand other features, which you combine in ways that you, you know, through neural network circuits that you learn during training that then help you, like, put out the right recognition or generate the right thing. Not thinking. Was it mean for it to write it down? It is writing it down. It's putting, that's how it writes things down. It has this matrix that it updates the numbers every time. That's its workspace. That's its workpad. That's how these type of neural networks work. I'm going to jump forward here to the conclusion. All right. So here we go. It's like, what about consciousness? And it's like, well, we borrowed a lot of ideas from the study of consciousness. Many of our experiments were designed to test for connections between the JSpace and global workspace theory, a framework for explaining how conscious access works in humans and animals, given those connections that's natural to ask whether these experiments provide evidence that A and models like cloud might be conscious, well, our experiments don't show that cloud can have experiences or feel things, you know, it's like, but dot, dot, dot. That's all very suggestive that there's something new going on here. Probably the most, I think, disingenuous thing at all is the way that they punched in the intro, cloud did this on its own. We didn't program it. It's a machine-learned neural network. Nothing is programmed. That's what machine learning is. You could say that about any machine learning system. This image recognizer is figuring out how to recognize it. We didn't tell it how to recognize the images. It just figured it out. Yeah, because it was machine learning. You did semi-supervised training, the trained the neural networks weights until it was good at minimizing loss on the image recognition task. You don't program it. It learns. So I just think that I just think the language around this is so disingenuous. They took an existing tool that people had been doing for the last four or five years, and it's a cool tool, and they showed it, could work on scale and very big networks. It's cool. And I confirmed, like, yeah, the way we thought these things work is that you have useful, like high-level properties are identified as you move through these layers, and then those are used to help narrow down the choice of possible tokens to output. Yeah, good. It works like we thought it did. Nothing is new, and nothing has to do with human consciousness. Global workspace theory, first of all, has been largely at least as controversial. But the thing they're not saying and that all the people talking about elements and consciousness are not saying is, what do you think about global workspace theory and human consciousness? It's a center that's integrating ongoing inputs that are coming in and out. It has all sorts of different things coming in, and it's selecting different things to pay attention to and not, and it's just evolving stateful system. This is all feed four. Remember, this is feed four. One layer after another in order, and then it's done, and the token is out, nothing is saved. There's no states that are changing, that neural network. Nothing has changed at all, and then you can feed it another prompt and you'll get out another token. And so they're just looking at the annotations that build up, and to, like, at some point towards the end, we see some of these annotations describe high-level properties of the type of thing that it's trying to output a token for. That's how LLMs, at least I understand, are supposed to work. So, like, I want to, I will say this. I think there's good researchers and inthropic, and this is good research. I think the PR people who talk about this technology do so in a way that is, I think, disingenuous. I think it is with a particular agenda for trying to make people feel in a certain way, which is mainly just a general sense of, like, wow, this stuff's too powerful. Why? Because if this stuff seems like weird and alive and emergent, it's such an important technology, and we so worry about that, that we'll forget the ask, hey, anthropic, how are you going to make a profit? But, anthropic, your token cost is this high. You have no competitive mode on the small number of applications where people are willing to actually pay for token API access. Smaller, more specialized models with, you know, hand-coded, complicated harnesses are going to do just as well. What's your plan? How do you justify a trillion-dollar valuation? Like, these sort of real questions are the ones you forget to ask when you instead are trying to wonder, huh, does this mean clear? Claude is a moral agent or conscious. So I guess my final thing would be to tell the researchers at Anthropic, I like your work and this is good work. I wish they would let you write real computer science papers and not these glorified press releases, but I know they don't let you. But maybe next time when the PR department calls, wait a second before you pick up the phone because I do not like the way Anthropic talks about their research. Even if there's cool stuff going on in there. So back to the original question, did we just reveal something unnerving, cool, or blockbuster about LLMs with this research and the answer is no. We confirmed that yes, language models worked a way that at least I always thought they did and you can check the print because I've written about this for years. So Anthropic, when you stop with the press release research reports, write computer science papers, sell products, convince us to buy the products. This weird in between kind of bastardization of actual research is something that's stressing a lot of people out and I think it's disingenuous, but that's just my opinion. All right, that's all the time we have for this week. I'll be back on Monday with an advice episode, probably no AI reality. Check the next week because I'm on vacation, but I'll do my best. We'll be back soon enough and until then as always, remember care about AI, but not everything you read about it. Hey, if you've made it this far, you must be ready to join my fight for depth in a distracted world. Now, the best way to do this is to join over 125,000 people who receive my email newsletter each Monday, you can sign up at calnewport.com/ideas and when you do, I will send you a free guide to my seven best ideas about cultivating a deep life. Sign up today at calnewport.com/ideas.