Go back

Richard Sutton – Father of RL thinks LLMs are a dead end

1h 6m 21s

Richard Sutton – Father of RL thinks LLMs are a dead end

Richard Sudden, a foundational figure in reinforcement learning, argues that large language models (LLMs) miss the core of true intelligence by focusing on mimicking human language rather than learning from real-world experience. He contends that intelligence fundamentally requires a goal-oriented system that acts, observes, and learns from outcomes—what reinforcement learning embodies. LLMs, in contrast, lack a clear goal, ground truth, or ability to adapt when things go wrong, as they rely on predicting next tokens rather than understanding consequences. Sudden emphasizes that real-world learning, through action and sensory feedback, is essential for developing robust, adaptive intelligence. He draws parallels to human development, where early imitation gives way to experiential learning, and stresses that animals—including humans—learn via trial and error, not supervised training. While LLMs have achieved impressive performance in specific domains like math, this success stems from pattern matching, not true understanding or generalization. The future of AI, he believes, lies in experiential learning systems that operate in real environments, continuously updating their models based on outcomes. Such systems would not only learn from actions but also build internal world models through direct interaction. This approach overcomes the limitations of LLMs, which depend on massive, static datasets and lack the ability to generalize or adapt. Sudden also highlights that current AI lacks mechanisms for generalization across tasks, a critical gap in achieving human-level competence. He sees the shift toward experiential learning as both inevitable and necessary—not just for technical progress, but as a fundamental evolution in how intelligence is understood and built, echoing the broader transition from biological replication to designed, intelligent systems in the universe.

Transcription

11001 Words, 60679 Characters

English
Today, I'm chatting with Richard Sudden, who is one of the founding fathers of reinforcement learning, an inventor of many of the main techniques he was there, like TD learning and policy gradient methods. And for that, he received this year's Touring Award, which, if you don't know, is basically the Nobel Prize for Computer Science. Richard, congratulations. Thank you, DuLakish. And thanks for coming on the podcast. It's my pleasure. Okay, so first question, my audience and I are familiar with the LLM way of thinking about AI. Conceptually, what are you missing in terms of thinking about AI from the RL perspective? Well, yes, I think it's really quite a different point of view and it can easily get separated and lose the ability to talk to each other. And yeah, large language models have become such a big thing, generative AI in general, a big thing. And our field is subject to bandwagans and fashions. So we lose track of the basic, basic things, because I consider reinforcement learning to be basic AI. And what is intelligence or the problem is to understand your world. And reinforcement learning is about understanding your world, whereas large language models are about mimicking people, doing what people say you should do, they're not about figuring out what to do. Huh, I guess you would think that to emulate the trillions of tokens in the corpus of internet text, you would have to build a world model. In fact, these models do seem to have very robust world models. And they're the best world models we've made to date in AI, right? So what do you think that that's missing? I would disagree with most of the things you just said. Great. Just to mimic the what people say is not really to build a model the world at all. I don't think you know, you're mimicking things that have a model of the world, the people. But I don't want to approach the question in an adversarial way. But I would question the idea that they they have a world model. So a world model would enable you to predict what would happen. They have the ability to predict what a person would say. They don't have the ability to predict what will happen. What we want, I think, to quote Alan Turing, what we want is a machine that can learn from experience. Where experience is the things that actually happen in your life. If you do things, you see what happens. And that's what you learn from. Yeah. The large language models learn from something else. They learn from here's a situation. And here's what a person did. And implicitly, the suggestion is you should do with the person. Right. I guess maybe the crux and I'm curious if you disagree with this is some people will say, okay, so this imitation learning has given us a good prior or given these models a good prior, but reasonable raise to approach problems. And as we move towards the era of experience, as you call it, this prior is going to be the basis on which we teach these models from experience because this gives them the opportunity to get answers right some of the time. And then on this, you can build, you can train them on experience. Do you agree with that perspective? No, I agree that it's the large language model perspective. Right. I don't think it's a good perspective. Yeah. So to be a prior for something, there has to be a real thing. I mean, a prior bit of knowledge should be basis for actual knowledge. Right. What is actual knowledge? There's no definition of actual knowledge in that in that large language framework. What makes an action a good action to take? You recognize the value, the need for continual learning. Right. So if you need to learn continually, continually means learning during the normal interaction with the world. Yeah. And so then there must be some way during the normal interaction to tell what's right. Yep. Okay. So is there any way for it to tell in the largest language model set up to tell what's the right thing to say? You will say something and you will not get feedback about what the right thing to say is because there's no definition of what the right thing to say. Yeah. There's no goal. Right. If there's no goal, then there's one thing to say. Another thing to say. There's no right thing to say. Right. So there's no ground truth. You can't have prior knowledge if you don't have ground truth. Because the prior knowledge is supposed to be a hint or an initial belief about what the truth is. Yeah. But there isn't any truth. Right. Now in reinforcement thing, there is a right thing to say or right thing to do. Right. Because the right thing to do is the thing that gets you reward. Right. So we have a definition of what's the right thing to do is and so we can have prior knowledge or knowledge provided by people about what the right thing to do is. And then we can check it to see because we have a definition of what the actual right thing to do is. Yeah. Now an even simpler case is when you have you're trying to make a model of the world when you predict what will happen, you predict and then you see what happens. Okay. So there's ground truth. There's no ground truth in in large language miles because you don't have a prediction about what will happen next. If you say something in your in your conversation, there's the large language miles have no prediction about what the person will say in response to that or what the what the response will be. I mean, I think they do. Like they you can literally ask them what what would you anticipate a user might say in response? And they have a prediction. Oh no, they they will respond to that question right. Yeah. But they have no prediction in the substantive sense that they won't be surprised by what happens. And if something happens that isn't what you might say they predicted, they will not change because an unexpected thing has happened and they're to learn that they'd have to make an adjustment. So I think a capability like this does exist in context. So it's interesting to watch a model do chain of thought and then suppose it's trying to solve a math problem. It'll say, okay, I'm going to approach this problem using this approach at first and it'll write this out and be like, oh wait, I just realized this is the wrong conceptual way to approach the problem. I'm going to restart by this another approach. And that flexibility does exist in context, right? Do you have something else in mind? Or do you just think that you need to extend this capability across longer horizons? I'm just saying they don't have a in any meaningful sense they don't have a prediction of what will happen next. And they will not be surprised by what happened next. They'll not make any changes if something happens. But isn't that what isn't based on what happens? Isn't that literally what next token prediction is? Prediction of what's next and then updating on the surprise? Next token is what they should say, what the actions should be. It's not what the world will give them in response to what they do. Let's go back to the their lack of goal. For me, having a goal is the essence of intelligence. Something is intelligent if it can achieve goals. I like John McCarthy's definition that intelligence is the computational part of the ability to achieve goals. So you have to have goals you're not you're just you're just you're just you're just behaving system. You're not you're not anything special. You're not intelligent. Right. And you agree that large language miles don't have goals. I think they have no they have a goal. What's the goal? Next to prediction. That's not a goal. It doesn't it doesn't change the world. You know, I think tokens come at you and if you predict them, you don't influence them. Oh, yeah. It's not a goal about the external world. Yeah, it's not a goal. It's not a substantive goal. You can't look at a system and say, oh, it has a goal. If it's just sitting there predicting and being happy with itself, it's predicting accurately. I guess maybe the bigger question I want you to understand is why you don't think doing RL on top of LLubs is a productive direction because we seem to be able to give these models a goal of solving difficult math problems. And there are in many ways at the very peaks of human level in the capacity to solve math lumpia type problems, right? They got goal that IMO. So it seems like the model which got goal that the international math lumpia does have the goal of getting math problems, right? So why can't we extend this to different domains? Well, the math problems are different. I'm making a model of physical world and carrying out the consequences of mathematical assumptions or operations. Right. Those are very different things. Like the empirical world has to be learned. You have to learn the consequences. Whereas the math is more just computational. It's more like standard planning. So so there you can you can they can have a goal to to to find the proof and they are in some way given that goal to find the proof. Right. So I mean it's interesting because you wrote this essay in 2019 titled The Bit or Lesson and this is the most influential essay, perhaps in the history of the eye, but people have used that as a justification for scaling up LLMs because in their view, this is the one scalable way we have found to pour ungodly amounts of compute into learning about the world. And so it's interesting that your perspective is that the LLMs are actually not bit or less impelled. It's an interesting question whether large language models are are a case of the bitter lesson because they are clearly a way of using massive computation. I think things that will scale with computation up to up to the limits of the internet. But there also a way of putting in a lot of human knowledge. So this is an interesting question. It's a sociological or industry question. Will they reach the limits of the data and be superseded by things that can get more data just from experience rather than from people? In some ways, it's a classic case of the bitter lesson. With more human knowledge, we put into the large language models, the better they can do. And so it feels good. And yet, one, in particular, expect there to be systems that can learn from experience, which could well perform much, much better and be much, much scalable. In which case, it will be another instance of the bitter lesson with the things that used human knowledge were eventually superseded by things that just trained from experience and competition. I guess that doesn't seem like the crux to me, because I think those people would also agree that the overwhelming amount of compute in the future will come from learning from experience. They just think that the scaffold or the basis of that, the thing you'll start with in order to pour in the compute to do this future experiential learning or on-the-job learning will be elements. I still don't understand why this is the wrong starting point altogether. Why we need a whole new architecture to begin doing experiential, continual learning and why we can't start with elements to do that. Well, in every case of the bitter lesson, you could start with human knowledge and then do the scalable things. That's always the case, and there's never any reason why that has to be bad, but in fact, and in practice, it has always turned out to be bad, because people get locked into the human knowledge approach and they psychologically, and now I'm speculating why it is, but this is what has always happened, that they get their lunch, get eaten by the methods that are truly scalable. Give me a sense of what the scalable method is. The scalable method is you learn from experience. You try things, you see what works. No one has to tell you. First of all, you have a goal. Without a goal, there's no sense of right or wrong or better or worse, so large language models are trying to get by without having a goal or a sense of better or worse. That's just, you know, it's exactly starting in the wrong place. Maybe it's interesting to compare this to humans. So in both the case of learning from imitation versus experience and on the question of goals, I think there's some interesting analogies. So, you know, kids will initially learn from imitation. You don't think so. Oh, of course not. Really? Yeah. I think kids just like watch people, they like kind of try to like say to say to those kids. I think the level about the first six months. I think they're kind of imitating things or trying to like make their mouth sound the way they see their mother's mouth sound and then they'll say the same word to them, understanding what they mean. And as you get older, the complexity of the imitation they do increases. So that's your, your, you know, you're imitating maybe the skills that your people in your band are using to hunt down the deer or something. And then you go into the learning from experience are all regime. But I think there's a lot of imitation learning happening with humans. It's surprising. Yeah, you can have such a different point of view. Yeah. Well, I see kids, I see kids just trying things and like waving their hands around and moving their eyes around and no one, no, and tells them there's no, there's no imitation for how they move their eyes around or even the sounds they make, they may, they may want to create the same sounds, but the, the actions, you know, the thing that the infant actually does, there's no targets for that. There are no examples for that. I agree that it doesn't explain everything infants do, but I think it guides a learning process. I mean, even LLM when it's trying to predict the next token early in training, it will like make a guess, it'll be different from what like it actually sees. And in some sense, it's like very short to rise in RL or it's like making this guess of like what I think this token will be this. It's actually the other thing similar to how a kid will try to say a word, it comes out wrong. The large language model is learning from training data. It's not learning from experience. It's learning from something that will never be available during its normal life. There's never any training data that says you should do this action in normal life. I think this is maybe more of a semantic distinction. Like what do you call school? Is that not training data? You're not like going to school because it's like school is much later. Okay, I shouldn't have said never, but I don't know. I think I would even say about school, but formal schooling is the exception, should be a series of learning where I think it's just a sort of program in your biology that like early on, you're not that useful. And then like kind of why you exist is to understand the world and like learn how to interact with it. And it seems kind of like a training phase. I agree that then there's like a sort of more gradual, there's not a sharp cut off to like training to deployment, but there seems to be this like initial training phase, right? There's nothing where you have training of what you should do. There's nothing. You see things that happen. You're not told what to do. Don't be difficult. I mean, this is obvious. I mean, you're like literally taught what to do. This is like where the word training comes from is from humans, right? So I don't think learning is really about training. I think learning is about about learning. It's about an active process. The child tries things and sees what happens. Right. Yeah, it does not. We don't think we don't think about training and we think of the an infant growing up. These things are actually rather well understood. If you go to look about how psychologists think about learning, there's nothing like imitation. Maybe there are some extreme cases where humans might do that or appear to do that, but there's no basic animal learning process called imitation. The basic animal learning process is for prediction and for trial and error control. I mean, it's really interesting how sometimes the most hard things to see are the obvious ones. It's obvious if you just look at animals and how they learn and you look at psychology and how our theories of them, it's obvious that that supervised learning is not part of the way animals learn. We don't have we don't have examples of desired behavior. But we have examples of things that happen. One thing that followed another and we have examples of we did something and there were consequences, but there are no examples of supervised learning. There are no supervised learning is not something that happens in nature. Even if that was a case, we should forget about it because it's it's just that's some special thing that happens in people. It doesn't happen broadly in nature. Squirrels don't go to school. Squirrels can learn all about the world. It's absolutely obvious, I would say, that supervised learning doesn't happen in animals. So I interviewed this psychologist and anthropologist Joseph Henrik who has done work about cultural evolution and basically what distinguishes humans and how do humans pick up knowledge? Why are you trying to distinguish humans? Humans are animals. What we have in common is more interesting. What we have what distinguishes us, we should be paying less attention to. We're trying to replicate intelligence. We want to understand what is it that enables humans to go to the moon or to build semiconductors. I think the thing we want to understand is the thing that makes no animal can go to the moon or it makes semiconductors. So we want to understand what makes human special. So I like the way you consider that obvious because I consider the opposite obvious. I think we have to understand how we are animals. If we understood a squirrel, I think we'd be almost all the way there. It's understanding human intelligence. The language part is just a small veneer on the surface. This is great. We're finding out the very different ways that we're thinking. Maybe when we're not arguing, we're trying to share different ways of thinking with each other. I think arguing is useful. I do want to complete this thought. Joseph Henrik has this interesting theory that a lot of the skills that humans have had to master in order to be successful, and we're not talking about last thousand years or last 10,000 years, but hundreds of thousands of years. The world is really complicated, and it's not possible to reason through how to, let's say, hunt a seal if you're living in the Arctic. And so there's this many, many step-long process of how to make the bait and how to find the seal and then how to process the food in a way that doesn't make sure you won't get poisoned. And it's not possible to reason through all of that. And so over time, yes, there's this like larger process of whatever analogy you want to use maybe are all something else where culture as a whole has figured out how to find and kill and eat seals. But then what is happening when through generations this knowledge is transmitted is in his view that like you just have to imitate your elders in order to learn that skill because you can't you can't think your way through how to hunt and kill and process a seal. You have to just watch other people maybe make tweaks and adjustments. And that's how culture knowledge accumulates. But the initial step of the cultural gain has to be imitation. But maybe you think about it a different way. No, I think about it the same way. But still it's a small thing on top of basic trial and error learning, prediction learning. And it's what distinguishes us perhaps from many animals. But we're an animal first. Yeah. And we were an animal before we had language and all those other things. I do think you make a very interesting point that continual learning is a capability that most mammals have. I guess all mammals have. So it's quite interesting that we have something that all mammals have but our AI systems don't have, right? Whereas maybe like the ability to understand math and follow difficult math problems depends on how you define math, but like this is the capability our AIs have. But that no almost no animal has. And so it's quite interesting what ends up being difficult and what ends up being easy. More of experience. For the era of experience to commence, we're going to need to train AIs in complex real world environments. But building effective our own environments is hard. You can't just hire a software engineer and have them write a bunch of cookie cutter validation tests. Real world domains are messy. You need deep subject matter experts to get the data, the workflows and all the subtle rules right. When one of Labelbox's customers wanted to train an agent to shop online, Labelbox assembled the team with a ton of experience engineering internet storefronts. For example, the team built the product catalog that could be updated during the episode because most shopping sites have constantly changing state. They also added a reddit cache to simulate stale data since that's how real e-commerce sites actually work. These are the kinds of things that you might not have naively thought to do, but that Labelbox can anticipate. These details really matter. Small tweaks are often the difference between cool demos and agents that can actually operate in the real world. So whether it's correcting traces that you already produced or building an entirely new suite of environments, Labelbox can help you turn your RO projects into working systems. Reach out at labelbox.com/thewarcache. All right, back to Richard. This alternative paradigm that you're imagining. The experiential paradigm. Let's lay out a little bit about what it is. It says that experience, action, sensation, well, sensation, action, reward, and this happens. On and on and on makes more life. It says that this is the foundation and the focus of intelligence. Intelligence is about taking that stream and altering the actions to increase the rewards in the stream. So learning then is from the stream and learning is about the stream. So that second part is particularly telling. What you learn, your knowledge, your knowledge is about the stream. Your knowledge is about if you do some action of what will happen or it's about which events will follow other events. It's about the stream. It's the content of the knowledge is statements about the stream. And so because it's a statement about the stream, you can test it by comparing it to the stream and you can learn continually. So when you're imagining this future continual learning agent, they're not future. Of course, they exist all the time. This is what reinforcement learning paradigm is learning from experience. Yeah. I guess maybe I would have meant to say is human level general continual learning agent. What is the reward function? Is it just predicting the world? Is it then having a specific effect on it? What would the general reward function be? The reward function is arbitrary. So if you're playing chess, it's to win the game of chess. If you were to, if you're a squirrel, maybe the reward has to do with getting nuts. In general, for an animal, you would say the reward is to avoid pain and to acquire pleasure. Right. And there's also would be a component having to do with, I think there should be a component having to do with your increasing understanding of your of your environment that would be sort of an intrinsic motivation. I see. I guess this AI would be deployed to like lots of people would want it to be doing lots of different kinds of things. Right. So it's performing the task people want. But at the same time, it's learning about the world from doing that task. And do you do imagine, okay, so we get rid of this paradigm where there's training periods and then there's deployment periods. But then is there, do we also get rid of this paradigm when there's the model and then instances of the model or copies of the model that are, you know, doing certain things? How do you think about the fact that we'd want this thing to be doing different things? We'd want to aggregate the knowledge that it's gaining from doing those different things. I don't like the word model when used the way you just did. Interesting. I think a better word would be of the network. So I think you mean the network. Maybe there's many networks. So anyway, things would be learned. And then you'd have copies and many instances and sure you'd want to share knowledge across the instances. And there would be lots of possibilities for doing that. Like there is not today, you can't have one child grow up and learn about the world. And then every new child has to repeat that process. Whereas with AIs, with the digital intelligence, you could hope to do it once and then copied into the next one as a starting place. So this would be a huge savings. And I think actually it'd be much more important than trying to learn from people. I agree that the kind of thing you're talking about is necessary, regardless of whether you start from LLM's or not, right? If you want human or animal LL intelligence, you're going to need this capability. Suppose a human is trying to make a startup, right? And this is a thing which has a reward on the order of 10 years. Once in 10 years, you might have an exit where you get paid out of billion dollars. But humans have this ability to make intermediate auxiliary rewards or have some way of, even when they have extremely special rewards, they can still make intermediate steps, having an understanding of like what the next thing you're doing leads to this grander goal we have. And so how do you imagine such a process might play out with AIs? So this is something we know very well. And it's the basis of it is temporal difference learning, where the same thing happens in a less grandiose scale. Like when you learn to play chess, the long-term goal is winning the game. And yet you can't, you want to be able to learn from a shorter term, things like taking your opponent's pieces. And so you do that by having a value function, which predicts the long-term outcome. And then if you take guys pieces where your prediction about the long-term outcome is changed, it goes up. You think you're going to win and then that increase in your belief immediately, quote, reinforces the move that led to taking the piece. Okay, so we have this long-term 10-year goal of making a startup and making a lot of money. And so when we make progress, we say, oh, I'm more likely to achieve the long-term goal and that rewards the steps on the way. Right. And then you also want some ability for information that you're learning. I mean, one of the things that makes humans quite different from these elements is that if you're onboarding on a job, you're picking up so much context and information. And that's what makes you useful at the job, right? You're everything from how your client's preferences, to how the company works, everything. And is the bandwidth of information that you get from a procedure like TD learning high enough to have this huge pipe of context and tacit knowledge that you'd need to be picking up and the way humans do when they're just deployed? I think the crux of this, I'm not sure, but the big world hypothesis seems very relevant and the reason why humans become useful on their job is because they are encountering the particular part of the world. And it can't have been anticipated and can't all have been put in in advance in the world. It's so huge that you can't, the dream, so I see it. The dream of large science models is you can teach the agent everything and it will know everything and it won't have to learn anything. online during its life, okay? And your examples are all, well, really you have to because there's a lot to, you can teach it, but there's all little idiosyncrasies of the particular life they're leading and the picker people they're working with and what they like is opposed to what average people like. And so that's just saying the world is really big and so you're gonna have to learn it along the way. - Yeah, so it seems to me you need two things. One is some way of converting this long-run goal reward into smaller auxiliary or these predictive rewards of the future reward or the future reward at least to the final reward. Then you need some other way, initially it seems to me, you need some way of then, okay, I need to hold on to all this context that I'm gaining as I'm working in the world, right? I'm like learning about my clients, my company, all this information, and I'm-- - So I would say you're just doing regular learning. - Yeah. - Maybe using context because in large language miles, all that information has to go into the context window. - Right. - But in a continual learning setup, it just goes into the ways. - Maybe, yes, so maybe context is the wrong word to use 'cause I mean a more general thing. - You learn a policy that's specific to the environment that you're finding yourself in. - Yeah, so the question I'm trying to ask is, you need some way of getting the company bits per secondary picking, like is a human picking up when they're out in the world, right? If you're just like interacting over Slack with your clients and everything. - So maybe we're trying to ask the question of, it seems like the reward is too small of a thing to do all the learning that we need to do. But of course we have the sensations. We have all the other information we can learn from. - Right. - We don't just learn from the reward, we learn from all the data. - Yeah, so what is the learning process which helps you capture that information? - So now I want to talk about the base common model of the agent with the four parts. So we need a policy. The policy says, the situation I'm in, what should I do? We need a value function. The value function is a thing that is learned with TD learning. And the value function produces a number. The number says, how well is it going? And then you watch if that's going up and down and you use that to adjust your policy. Okay, so those two things. And then there's also the perception component, which is the construction of your state representation, the sense of where you are now. And the fourth one is what we're really getting at, most transparently anyway. The fourth one is the transition model of the world. That's why I am uncomfortable. It's just calling everything models because I want to talk about the model of the world, the transition model of the world. Your belief that if you do this, what will happen? What will be the consequences of what you do is so you're physics of the world. But it's not just physics, it's also abstract models. Like your model of how you traveled from California up to Edmonton for this podcast. That is a model and that's a transition model and that would be learned. And it's not learned from rewards. It's learned from you did things you saw what happened. You made that model of the world. That will be learned very richly from all the sensation that you receive, not just from the reward. It has to include the reward as well, but that's a small part of the whole model, a small crucial part of the whole model. - Yeah, one of my friends to be ordered pointed out that if you look at the museuro models that Google DeMine deployed to learn Atari games, that these models were initially, not a general intelligence self, but a general framework for training specialized intelligences to play specific games. That is to say that you couldn't, using that framework, training policy to play both chess and go and some other game. You had to train each one in a specialized way. And he was wondering whether that implies that reinforcement learning generally because of this information constrained, you can only learn one thing at a time. The density of information is not high or whether it was just specific to the way that museuro was done. And if it's specific to Alpha Zero, what would need to be changed about that approach so that it could be a general learning agent? - The idea is totally general. I do use all the time as my canonical example, the idea of an AI agent is like a person. And people in some sense, they have just one world they live in and that world may involve chess and it may involve Atari games. But those are not a different task or a different world. Those are different states that they encounter. And so the general idea is not limited at all. - So maybe we'd be useful to explain what was missing in that architecture or that approach, which this continual learning agent would have. - Well, they just set it up. They didn't, it was not their ambition to have one agent across those games. If we want to talk about transfer, we should talk about transfer. Not across games or across tasks, but transfer between states. - Yeah, I guess I'm curious about historically, have we seen the level of transfer using our all techniques that would be needed to build this kind of? - Okay, good, good. We're not seeing transfer anywhere. We're not seeing general, critical to good performance is you can generalize well from one state to another state. We don't have any methods that are good at that. Well, we have our people try different things and they settle on something that a representation that transfers well or that generalizes well. But we don't have any automated techniques to promote. We have very few automated techniques to promote transfer. And none of them are used in modern deep learning. - Let me paraphrase to make sure that I understood that correctly. It sounds like you're saying that when we do have generalization in these models, that is a result of some sculpted. - Humans did it. - The researchers did it. 'Cause there's no other explanation. I mean, gradient descent will not make you generalize well. It will make you solve the problem. It will not make you, you know, get new data, you generalize them a good way. Generalization means to train on one thing that'll affect what you do on the other things. So we know deep learning is really bad at this. For example, we know that if you train on some new thing, it will often catastrophically interfere with all the old things that you knew. So this is exactly bad generalization. - Right. - Now generalization, as I said, is some kind of influence of training on one state on other states. And generalization is not necessarily good or bad. Just the fact that you generalize is not necessarily good or bad, you can generalize poorly, you can generalize well. So you need, generalization always will happen, but we need algorithms that will cause the generalization to be good, right, and bad. - I'm not trying to kickstart this initial crux again, but I'm just genuinely curious because I think I might be using the term differently. I mean, one way to think what is these LLMs are increasing the scope of generalization from like earlier systems which could not really even do a basic math problem to now they can do anything in this class of math Olympia type problems, right. So you initially start with like they can generalize among addition problems at least. Then you generalize to like they can generalize among like problems that require use of different kinds of mathematical techniques and theorems and conceptual categories, which is like what the math Olympia requires. And so it sounds like you don't think of that being able to solve any problem within that category. As an example of generalization or let me know if I'm misunderstanding that. - Well, large names models, so complex. We don't really know what information they had prior. We have to guess because they've been fed so much. This is one reason why they're not a good way to do science. It's just so uncontrolled, so unknown. - But if you come up with an entirely new. - They're getting a bunch of things right, perhaps. And so the question is why? Well, maybe that they don't need to generalize to get them right because the only way to get some of them right is to form something which gets all of them right. So if there's only one answer, then and you find it, that's not called generalization. It's just the only way to solve it. And so they find the only way to solve it. A generalization is when it could be this way, it could be in that way and they do it the good way. My understanding is that this is working more and more better and better with coding agents. So engineers, obviously, if you're trying to program a library, there's many different ways you could achieve the end spec. And an initial frustration with these models has been that, they'll do it in a way that's sloppy. And then over time, they're getting better and better at coming up with the design architecture and the abstractions that developers find more satisfying. And it seems that an example of what you're talking about. Well, there's nothing in them which will cause it to generalize well. Then creating descent-- will cause them to find a solution to the problems they've seen. And if there's only one way to solve them, they'll do that. But there are many ways to solve it, some which generalize well, some which generalize poorly. There's nothing in them in the outrooms that will cause them to generalize well. But people, of course, are involved. And if it's not working out, they fiddle with it until they find a way, perhaps until they find a way which it generalizes well. So to prep for this interview, I wanted to understand the full history of RL, starting with reinforce of two current techniques, like GRPO. And it didn't just want a list of equations and algorithms. I wanted to really understand each change in its progression and the underlying motivation, you know, what was the main problem that each successive method was actually trying to solve. So I had Gemini Deep Research walk me through this entire timeline, step by step. It explained the last 20 years of gradual innovation and explained how each step made the RL learning process more stable or more sample-efficient or more scalable. I asked Deep Research to put all of this together like an Andre Carpati style tutorial. And it did that. What was cool is that it combined this whole lesson together into one coherent cohesive document in the style that I wanted. It was also great that it assembled all of the best links in the same place so that if I wanted to understand any specific algorithm better, I could just access the right explainer right there. Go to gemini.google.com to try it out yourself. I want to zoom out and ask about, so being in the field of AI for longer than almost anybody who is commentating on it or working in it now, I'm just curious about what the biggest surprises have been, how much new stuff you feel like is coming out or does it feel like people are just playing with old ideas, zooming out, you know, you got into this even before like deep learning was popular. So how do you see this trajectory of this field over time and how new ideas have come about on everything? And what's been surprising? Okay. So, yeah, I thought a little bit about this. There are many things or handful of things. The large line of miles are surprising, it's surprising how effective neural networks, artificial neural networks are at language tasks, you know, that was a surprise wasn't expected language seemed different. So that's impressive. There's a long standing controversy in AI about simple basic principle methods, the general purpose methods, like search and learning and compared to human enabled systems, like symbolic methods, and so in the old days, it was interesting because things like search and learning were called weak methods, because they're just, these are general principles, they're not using the power that comes from imbuing a system with human knowledge, so those were called strong, and so I think the weak methods have just told me what, that's the biggest question from the old days AI, what would happen, and learning and search have just won the day. But there's a sense which that was not surprising to me because I was always voting for, or hoping or rooting for the simple basic principles. And so even with the large science models, it's surprising how well it worked, but it was all good and gratifying, and things like alpha-go, it's sort of surprising how well that was able to work, and alpha-zero in particular how well it was able to work, but it's all very gratifying because again, it's simple basic principles are winning the day. But there felt like whenever the public conception has been changed because some new application was about. For example, when alpha-zero became this viral sensation to you as somebody who has literally came up with many of the techniques that were used, did it feel to you like, new breakthroughs are made, or does it feel like, oh, we've had these techniques since the 90s, and people are simply combining them and applying them now. So the whole alpha-go thing has a precursor, which is TD Gammon, Jerry Tassaro did exactly reinforcement learning, temple-difference learning methods to play backgammon, and beat the world's best players, and it worked really well. And so in some sense, alpha-go was merely a scaling up of that process, but it was quite a bit of a scale up, and there was also an additional innovation in how the search was done. But it made sense. It wasn't surprising in that sense. Alpha-go actually didn't use TD learning. It waited to see the final outcomes, but alpha-zero used TD, and alpha-zero was applied to all the other games, and that did extremely well. I've always been very impressed by the way alpha-zero plays chess, because I'm a chess player, and it's a sacrifice material for sort of positional badges, and it's just content and patient to sacrifice that material for a long period of time. And so that was surprising that it worked so well, but also gratifying, and it's fitting into my world view. So this has led me where I am, where I am is, in some sense, a contrarian, or something, thinking differently from the field is, and I'm personally just kind of content being out of sync with my field for a long period of time, perhaps decades, because occasionally I have improved right in the past. And the only thing I do to help me not feel I'm out of sync and thinking in a strange way is to look not at my local environment or my local field, but to look back in time into history and to see what people have thought classically about the mind in many different fields. And I don't feel I'm out of sync with larger traditions. I really view myself as a classist rather than as a contrarian. I go to what the larger community of thinkers about the mind have always thought. Okay, some sort of left-goal questions for you if you'll tolerate them. So the way I read the bitter lesson is that it's not saying necessarily that human artisanal researcher tuning doesn't work, but that it obviously scales much worse than compute, which is growing exponentially. And so you want techniques which leverage the latter. And once we have AGI, we'll have researchers which scale linearly with compute, right? So we'll have this avalanche of millions of AI researchers and their stock will be growing as fast as compute. And so maybe this will mean that it is rational or it will make sense to have them doing good old fashioned AI and doing these artisanal solutions. Is that as a vision of what happens after AGI in terms of how AI research will evolve? I wonder if that's still compatible with the bitter lesson. Well, how did we get to this AGI? You want to presume that it's been done? Suppose it started with general methods, but now we've got the AGI and now we want to go. Then we're done. We're done. Interesting. You don't think that there's anything above AGI. Well, but you're using it to get AGI again. Well, I'm using it to get superhuman levels of intelligence or competence of different tasks. So these AGIs, if they're not superhuman already, then the knowledge that they might impart would be not superhuman. I guess there's different gradations of yours. I'm not sure this. So your idea makes sense because it seems to presume the existence of AGI. And then that we've already worked that out. So maybe one way to motivate this is AlphaGo is superhuman. It beat any go player. Alpha0 would beat AlphaGo every single time. So there's ways to get more superhuman than even superhuman. Yeah. And it was a different architecture. And so it seems possible to me that, well, the agent that's able to generally learn across all domains, there would be ways to make that give it better architecture for learning. Just to stay with it, Alpha0 was an improvement upon AlphaGo and Mu0 was an improvement of fun Alpha0. And the way Alpha0 was an improvement was it did not use the human knowledge, but just went from experience. Right. So why do you say bring in other agents, expertise to teach it when it's been, it works so well from experience and not by help from another agent. I agree that in that particular case, that it was moving to more general methods. But I meant to use that example to illustrate that it's possible to go superhuman, to superhuman plus plus, to superhuman plus plus plus plus, and I'm curious if you think those gradations will continue to happen by just making the method simpler or because we'll have the capability of these millions of minds who can then add complexity as needed if that will continue to, if that will continue to be a false path, even when you have billions of AI researchers or trillions of AI researchers. I think more interesting is just to think about that case, which when you have many AI's, will they help each other, the way cultural evolution works in people? Well, let's just maybe we should talk about that yeah, for sure the better lesson Oh, who cares about that? That's that's an impure cloud observation about a particular period in history 70 years in history no longer doesn't necessarily have to apply the next 70 years So an issue question is you're an AI you get some more computer power should you use it to make yourself? You know more computationally capable or should you use it to spawn off a copy of yourself to learn something interesting On the other side of the planet or on some other topic and then report back to you. Yep I think that's a really interesting question That that that will only arise in the age of digital intelligence I'm not sure what the answer is, but I think it will More questions will it be possible to really you know spawn it off send it out learn something new Some perhaps very new and then we'll be able to read be re-incorporated into the original Or will it will it will have changed so much that it can't really be done. Yeah, is that possible or is it not? And you know you could carry this to its limit As I I saw one of your videos the other night that that suggested that it that it could where you spawn off many many copies do different things It's highly decentralized, but report back to the central master And that this is this will be such a powerful thing Well, I think one thing that so this might attempt to add something to this this view is that a big question a big issue will become Corruption You know if you if you really could just get information from anywhere and bring it into your central mind You can become more more powerful And it's it's all digital and they all speak some internal digital language. Maybe it'll be easy and possible But it will not be that easy as easy as you're imagining because That you can lose your mind this way if you and pull in something from the outside and build it into your into your inner thinking Uh, it could take over you it could change you it could be Your destruction Rather than your increment in knowledge. Mm-hmm. I think this will become a big concern You know, particularly when you're oh he's figured all about you know how to play some new game or figured out he's studied in Indonesia and you want to incorporate that into your mind um Yeah, so you can't as you could you think oh just read it all in And that'll be fine, but no you've just read a whole bunch of bits into your mind and uh they could have viruses in them They could have hidden goals They can uh work you and change you and this will be up come a big thing. How do you have Cybersecurity in the age of digital spawning and re re reform again It's interesting that both quantum firms and AI labs have a cultural secrecy because both of them are operating an incredibly competitive markets And their success rest on protecting their IP if you're an AI researcher or engineer and you're deciding where to work Most of the quantum firms or AI labs that you'll be considering will be strongly siloing their teams to minimize the risk of leaks Hudson River trading takes the opposite approach their teams openly share their trading strategies And their strategy code lives in a shared mono repo at HRT if you're a researcher and you have a good idea Your contribution will be broadly deployed across all relevant strategies This gives your work a ton of leverage you'll also learn incredibly fast You can learn about other people's research and ask questions And you can see how everything fits together end-to-end from the low level of the execution of trades to the high low predictive models HRT is hiring if you want to learn more go to Hudson River trading dot com slash the work cash All right, back to Richard I guess this brings us to the topic of AI succession You have a perspective that's quite different from a lot of people that I've interviewed and maybe a lot of people generally So I also think it's a very interesting perspective. I want to hear about it. Yeah, so I do think succession to digital Or digital intelligence or augmented humans is inevitable So the argument go I have a four part argument now your step one is There's no government or organization that that gives humanity a unified point of view that dominates And they can they can arrange there's no consensus about how the world should be run and number two We will figure out how intelligence works researchers will figure it out eventually and number three We won't stop just with human level intelligence. We will get reach super intelligence And number four is That once it's inevitable over time That the most intelligent things around would gain resources and power And so put all that together it's you know You it's sort of inevitable that you're going to have Succession to AI or to AI enabled augmented humans So within those those four things seem clear and and and sure to happen But within that set of possibilities some it can be good outcomes as well as less good outcomes bad outcomes and So I'm just just trying to be realistic about where we are And and ask how we should feel about it. Yeah, I I agree with all four of those arguments and the implication and I also agree that succession Contains A wide variety of possible features. So curious to get more thoughts on that right And so then I do encourage people to think Possibly about it first of all because It's something we humans have always tried to do for thousands of years try to understand themselves trying to make themselves think better and You know just understand themselves. So this is a great success from as science Humanities we're finding out what this essential part of of Of humanness is it what it means to be intelligent And then why I usually say is is that this is all kind of human centric What if we look you step aside from being a human and just say take the point of view of the universe And this is I think a major stage in the universe a major transition transition from replicators we humans and animals plants we're all replicators and That gives us some strengths and some limitations and then we're entering the age of design Where because our AIs are designed our our all of our physical objects are designed our buildings are designed or Our technologies designed and we're we're designing now AIs things that can be intelligent themselves and that are themselves capable of design And so this is this is a key step In the world and in the universe and I think It's the it's the transition from the world in which most of the interesting things That are are replicated replicated means you can make copies of them But you'll really understand them like right now we can make more intelligent beings more children But we don't really understand how intelligence works whereas in We're reaching now to having design intelligence Intelligence that we do understand how it works and therefore we can change it in different ways in a different speeds than otherwise and and our future they might not be replicated at all like we may just design AIs and those AIs We'll design other AIs and Everything will be done by design construction rather than by replication Yeah, I mark this as one of the four great stages of the universe First there's there's dust ends of stars stars And then stars make planets and the planets give rise to life and now we're giving life to designed entities And so I think we should be proud and we should be That we are giving rise to this great transition in the universe Yeah, so it's an interesting thing what should we what should we consider them Part of humanity are different from humanity. It's our choice It's our choice whether we should say oh they are our offspring and we should be proud of them And we should celebrate their achievements or we should we could say oh no, they're not us Then we should be horrified. It's it's just it's interesting that that is it feels to me like a choice And yet it's such a strongly Held thing that how could we be a choice? I like these sort of contradictory implications of thought I mean it's just to consider if we were just designing another generation of humans. Yes, he designs the wrong word But we knew a few generations ago humans are gonna come up and forget about AI We just know in the long run Humanity will be more capable and maybe more numerous maybe more intelligent How do we feel about that? I do think there's potential worlds with future humans that we would be Quite concerned about so are you thinking like maybe we are we're like the nander falls we give rise to homo sapiens Maybe homo sapiens will give rise to a new group of people That's interesting like that like I'm basically taking the example you're giving of like okay Even if you consider them part of humanity. Yeah, I don't think that really necessarily means that we should feel Super comfortable. Yeah, like Nazis were humans right if we thought like oh the future generation will be Nazis I think we'd be like quite concerned about just handing off power to them. So I agree that this is not super dissimilar to worrying about more capable future humans But I don't think that that addresses A lot of the concerns people might have about this level of power being attained this fast with entities We don't fully understand. Well, I think it's relevant to point out that For most of humanity They don't have much Influence on what happens Most of humanity doesn't influence who can control the atom bombs or who Controls the nation-states even as a as a citizen. I often feel that we don't control the nation-states very much They're out of control. A lot of it has to do with just how you feel about change And if you think the current situation is really really good then you're More likely to be suspicious of change and reverse to change than if you think It's imperfect and I think it's imperfect in fact. I think it's pretty bad So I'm I'm I'm open to change And I think humanity is not in a has had a good super good track record and maybe it's the best thing that there's been But it it's far from perfect. Yeah, I guess there's different Varieties of change The industrial revolution was changed the Bolshevik revolution was also changed and if you were around in Russia and The 1900s and you're like look things aren't growing well This is ours kind of messing things up. We need change. I'd want to know what kind of change you wanted Before signing on the dot of line, right? And then similar with AI where I'd want to understand And to the extent it's possible to change its trajectory to change its directory of the eye such that the change is positive For humans we are we should be concerned about our future the future makes we should try to make it good We also those should recognize the limits our limits and we're I think we want to avoid the feeling of entitlement avoid the feeling. Oh, we're here first. We should always have it in a good way How should we think about the future and how much control A particular species on a particular planet should have over it How much control do we have? You know a counterbalance to our limited control over the long-term future of humanity Should be how much control do we have over our own lives like we have our own goals and we have our families And we those things are much more controllable than like trying to control The whole universe. Right So I think it's appropriate you know for us to to uh, you know really Work towards our own local goals and and it's a kind of aggressive for us saying oh the future has to evolve this way that I wanted to sure Because then we'll have arguments like different people think the future the global future should evolve in different ways And then they have conflict and yeah, and maybe we'll avoid that maybe a good analogy here would be Okay, so Suppose You're raising your own children. It might not be appropriate to have Extremely tight goals for their own life or also have some sense of like I want my children to go out there in the world and have this specific impact You know my my sons are going to become president and my daughter is going to become CEO of Intel and like together They're going to have this effect on the world but People do have the sense that I think this is appropriate of saying I'm going to give them Good robust values such that if and when they do end up in positions of power they do Reasonable pro-social things and I think maybe a similar attitude towards AI makes sense not in the sense of we can predict everything that they will do Um where we have this plan about what the world should look like in 100 years, but it's quite important to give them robust and Steerable and pro-social values pro-social values So maybe that's the wrong word are there universal values That we can all agree on I don't think so but that doesn't prevent us from Uh Giving our kids a good education, right? Like we have some sense of we want our children to be a certain way Yeah, and maybe pro-social is wrong word actually high integrity is a maybe a better word where if there's a request or if there's a goal that seems Harmful they will refuse to engage in it um or they'll be honest um Things like that and we have some sense that we can Teacher children things like this even if we don't have some sense of what true morality is or everybody doesn't agree on that Um, and maybe that's a reasonable target for you as well. So so you're saying We're trying to design the future and the the principles by which it will evolve and come into being right And so you're saying the first thing you're saying is well we will we try to teach our our children um general principles which will promote More likely evolutions. Yeah Maybe we should also seek for things being voluntary if there is change We want it to be a volunteer rather than impose that. Yeah, I think that's a an important point. Yeah And yeah, that's all good. I mean, I think this is like a big um, you know, the big The big or one of the really big human enterprises to design Society and then that's been ongoing for for thousands of years again And so so it's like the more things change really the more things they stay the same We still have to figure out how to be uh the children will still come up with different values that seem strange to their parents and their grandparents and uh and things will evolve The more things change the more they see the same also seems like a good capstone to the AI discussion because The AI discussion we were having was about how techniques which were um Invented even before their application to deep learning and backpropagation was evident have are you know central to the progression of AI today So maybe that's a good place to wrap up the conversation Okay, thank you very much. Thank you for coming on my pleasure

Podcast Summary

Key Points:

  1. Reinforcement learning (RL) views intelligence as learning from experience and understanding the world through action and consequence, unlike large language models (LLMs) that mimic human behavior without true world modeling.
  2. LLMs lack a genuine sense of goal, reward, or ground truth, making them unable to distinguish right from wrong or adapt to unexpected outcomes in the real world.
  3. True intelligence, as defined by Alan Turing, involves learning from actual experience—actions, outcomes, and feedback—rather than imitation or prediction of language patterns.
  4. The core of RL is grounded in a feedback loop
  5. Unlike LLMs, RL agents build world models from sensory data and outcomes, not from static training data, allowing them to predict and learn from consequences.
  6. Human learning parallels RL, with early imitation followed by trial-and-error experience, suggesting that real-world interaction, not supervised training, is essential for intelligence.
  7. Current AI lacks generalization and transfer between tasks due to poor automated methods, while deep learning often suffers from catastrophic forgetting.
  8. The future of AI lies in experiential, continual learning systems that evolve through real-world interaction, not just from curated data or human knowledge.

Summary:

Richard Sudden, a foundational figure in reinforcement learning, argues that large language models (LLMs) miss the core of true intelligence by focusing on mimicking human language rather than learning from real-world experience. He contends that intelligence fundamentally requires a goal-oriented system that acts, observes, and learns from outcomes—what reinforcement learning embodies. LLMs, in contrast, lack a clear goal, ground truth, or ability to adapt when things go wrong, as they rely on predicting next tokens rather than understanding consequences.

Sudden emphasizes that real-world learning, through action and sensory feedback, is essential for developing robust, adaptive intelligence. He draws parallels to human development, where early imitation gives way to experiential learning, and stresses that animals—including humans—learn via trial and error, not supervised training. While LLMs have achieved impressive performance in specific domains like math, this success stems from pattern matching, not true understanding or generalization.

The future of AI, he believes, lies in experiential learning systems that operate in real environments, continuously updating their models based on outcomes. Such systems would not only learn from actions but also build internal world models through direct interaction. This approach overcomes the limitations of LLMs, which depend on massive, static datasets and lack the ability to generalize or adapt.

Sudden also highlights that current AI lacks mechanisms for generalization across tasks, a critical gap in achieving human-level competence. He sees the shift toward experiential learning as both inevitable and necessary—not just for technical progress, but as a fundamental evolution in how intelligence is understood and built, echoing the broader transition from biological replication to designed, intelligent systems in the universe.

FAQs

Reinforcement learning focuses on understanding the world through action, experience, and rewards, where intelligence is defined by achieving goals. Large language models mimic human behavior by predicting what people would say, without learning from actual outcomes or understanding what actions lead to desirable results.

While large language models can predict next tokens or human responses, they lack a true world model that can predict the consequences of actions in the real world. They do not adjust or learn from unexpected outcomes, unlike reinforcement learning agents that update based on actual experience.

Large language models operate without a meaningful goal or sense of right vs. wrong. They only predict next tokens, not actions with real-world consequences. In contrast, reinforcement learning defines intelligence through goals and rewards, where actions are evaluated based on outcomes.

Experience—acting, sensing, and observing outcomes—is essential for learning. Intelligence arises from continual interaction with the world, where actions are refined based on feedback, not just from imitation or static data.

No, because they lack ground truth and a sense of reward. Without real-world feedback, they cannot learn what actions lead to better outcomes. Experience-based learning requires a clear definition of goals and consequences, which LLMs do not have.

The reward function defines what is valuable—such as winning a game or acquiring food for a squirrel. It provides a measurable signal that guides learning and helps agents improve by adjusting their actions to increase rewards over time.

Chat with AI

Loading...

Pro features

Go deeper with this episode

Unlock creator-grade tools that turn any transcript into show notes and subtitle files.