Go back

Some thoughts on the Sutton interview

11m 39s

Some thoughts on the Sutton interview

The core argument centers on the inefficiency and limitations of current LLM training paradigms, which consume vast compute without enabling true world modeling or continual learning. Most learning occurs during a narrow training phase using human-generated data, which is unsustainable and fails to produce agents capable of autonomous, self-directed environmental interaction. The speaker challenges the notion that imitation learning is fundamentally separate from reinforcement learning, arguing instead that it is a practical, scalable form of short-horizon RL that serves as a necessary foundation for more advanced learning. Historical progress, such as the development of technology and language, mirrors this process of cultural imitation and knowledge accumulation. While LLMs currently lack true world models and on-the-job learning, they show signs of developing deep world representations through their training. The speaker suggests that future AGI systems will likely rely on continual learning mechanisms—enabled by techniques like in-context learning or fine-tuning as tool calls—rather than relying solely on pre-trained knowledge. Despite Richard’s valid critique of sample inefficiency and data dependence, the speaker believes the boundaries between imitation and true learning are overstated, and that current models already demonstrate promising emergent capabilities. Ultimately, the path to AGI may not require abandoning imitation learning, but rather evolving it into a dynamic, continuous learning system grounded in real-world interaction.

Transcription

2098 Words, 12387 Characters

English
Boy, do you guys have a lot of thoughts about this interview? I've been thinking about it myself and I think I have a much better understanding now of Sudden's perspective than I did during the interview itself, so I wanted to reflect on how I understand his worldview now, and Richard apologies if there are still any errors or misunderstandings. It's been very productive to learn from your thoughts. Okay, so here's my understanding of the steel man of Richard's position. Obviously, he wrote the same essay, the bitter lesson, and what is this essay about? Well, it's not saying that you just want to throw away as much compute as you possibly can. The bitter lesson says that you want to come up with techniques which most effectively and scalably leverage compute. Most of the compute that's spent on an LLM is used in running it during deployment, and yet it's not learning anything during this entire period. It's only learning during this special phase that we call training, and so this is obviously not an effective use of compute. And what's even worse is that this training period by itself is highly inefficient because these models are usually trained on the equivalent of tens of thousands of years of human experience. And what's more, during this training phase, all of their learning is coming straight from human data. Now, this is an obvious point in the case of pre-training data, but it's even kind of true for the RLVR that we do with these LLMs. These RL environments are human furnished playgrounds to teach LLMs the specific skills that we have prescribed for them. The agent is in no substantial way learning from organic and self-directed engagement with the world. Having to learn only from human data, which is an inelastic and hard-to-skill resource, is not a scalable way to use compute. Furthermore, what these LLMs learn from training is not a true world model, which would tell you how the environment changes in response to different actions that you take. Rather, they are building a model of what a human would say next. And this leads them to rely on human-derived concepts. A way to think about this would be, suppose you trained an LLM on all the data up to the year 1900. That LLM probably wouldn't be able to come up with relativity from scratch. And maybe here's a more fundamental reason to think this whole paradigm will eventually be super seated. LLMs aren't capable of learning on the job, so we'll need some new architecture to enable this kind of continual learning. And once we do have this architecture, we won't need a special training phase. The agents will just be able to learn on the fly, like all humans, and in fact, like all animals are able to do. And this new paradigm will render our current approach with LLMs and their special training phase that's super sample and efficient, totally obsolete. So that's my understanding of richest position. My main difference with rich is just that I don't think the concepts he's using to distinguish LLMs from true intelligence, or animal intelligence are actually that mutually exclusive or dichotomous. For example, I think imitation learning is continuous with and complementary to RL. And, relatedly, models of humans can give you a prior which facilitates learning, quote unquote, "true" world models. I also wouldn't be surprised if some future version of test-time fine-tuning could replicate continual learning, given that we've already managed to accomplish this somewhat with in context learning. So let's start with my claim that imitation learning is continuous with and complementary to RL. So I tried to ask Richard a couple of times whether free-trained LLMs can serve as a good prior on which we can accumulate the experiential learning, aka do the RL, which would lead to AGI. So Ilya Satskiver gave a talk a couple of months ago that I thought was super interesting. And he compared pre-training data to fossil fuels. And I think this analogy actually has remarkable reach, just because fossil fuels are not a renewable resource does not mean that our civilization ended up on a dead end track by using them. In fact, they were absolutely crucial. You simply couldn't have transitioned from the water wheels of 1800 to solar panels and fusion power plants. We had to use this cheap, convenient, and plentiful intermediary to get to the next step. AlphaGo, which was conditioned on human games, and Alpha0, which was bootstrapped from scratch, were both superhuman go players. Now, of course, Alpha0 was better. So you can ask a question, will we or will the first AGI's eventually come up with a general learning technique that requires no initialization of knowledge, and that just bootstraps itself from the very start. And I'll perform the very best AIs that have been trained up to that date. I think the answer to both these questions is probably yes. But does this mean that imitation learning must not play any role whatsoever in developing the first AGI, or even the first ASI? No, AlphaGo was still superhuman, despite being initially shepherded by human player data. The human data isn't necessarily actively detrimental. It's just that at enough scale, it isn't significantly helpful. Alpha0 also used much more compute than AlphaGo. The accumulation of knowledge over tens of thousands of years has clearly been essential to humanity's success. In any field of knowledge, thousands, and probably actually millions, of previous people were involved in building up our understanding and passing it on to the next generation. We obviously didn't invent the language we speak, nor the legal system we use. Also, even most of the technologies in our phone, were not directly invented by the people who are alive today. This process is more analogous to imitation learning than it is to RL from scratch. Now, of course, are we literally predicting the next token like an LLM would in order to do this cultural learning? No, of course not. So, even the imitation learning that humans are doing is not like the supervised learning that we do for pre-training LLMs. But neither are we running around trying to collect some well-defined scalar reward. No ML learning regime perfectly describes human learning or animal learning. We're doing things which are both analogous to RL and to supervised learning. What planes are to birds, supervised learning might end up being to human cultural learning. I also don't think these learning techniques are actually categorically different. Imutation learning is just short horizon RL. The episode is a token long. The LLM is making a conjecture about the next token based on its understanding of the world and how the different pieces of information and the sequence relate to each other. And it receives reward and proportion to how well it predicted the next token. Now, of course, I already hear people saying, no, no, that's not the ground truth, it's just learning what a human was likely to say. And I agree. But there's a different question which I think is actually more relevant to understanding the scalability of these models. And that question is, can we leverage this imitation learning to help models learn better from ground truth? And I think the answer is, obviously yes. After RL and these pre-trained base models, we've gotten them to win gold and international math Olympic competitions and to code up entire working applications from scratch. Now, these are ground truth examinations. Can you solve this unseen math Olympic question? Can you build this application to match the specific features request? But you couldn't have RL a model to accomplish these tasks from scratch or at least we don't know how to do that yet. You needed a reasonable prior overhuman data in order to kickstart this RL process. Whether you want to call this prior a proper world model or just a model of humans, I don't think is that important. It honestly seems like a semantic debate. Because what you really care about is whether this model of humans helps you start learning from ground truth, aka become a true world model. It's a bit like saying to somebody pasteurizing milk, hey, you should stop boiling that milk because eventually you want to serve it cold. Of course, but this is an intermediate step to facilitate the final output. By the way, LLMs are clearly developing a deep representation of the world because their training process is incentivizing them to develop one. I use LLMs to teach me about everything from biology to AI to history and they are able to do so with remarkable flexibility and coherence. Now, our LLMs specifically train to model how their actions will affect the world. No, they are not. But if we're not allowed to call their representations a world model, then we're defining the term world model by the process that we think is necessary to build one, rather than the obvious capabilities that this concept implies. Okay, continue learning. I'm starting to bring up my hobby horse again. I'm like a comedian who has only come up with one good bit, but I'm going to milk it for all its worth. An LLM that's being riled on outcome-based rewards learns on the order of one bit per episode, and an episode might be tens of thousands of tokens long. Now, obviously, animals and humans are clearly extracting more information from interacting with our environment than just the reward signal at the end of an episode. Conceptually, how should we think about what is happening with animals? I think we're learning to model the world through observations. This outer loop RL is incentivizing some other learning system to pick up maximum signal from the environment. In Richard's Oak architecture, he calls this the transition model. And if we were trying to pigeonhole this feature spec into modern LLMs, what you do is fine tune on all your observed tokens. From what I hear from my researcher friends, and practice the most naive way of doing this actually doesn't work very well. Now, being able to learn from the environment in a high throughput way is obviously necessary for true AGI, and it clearly doesn't exist with LLMs trained on RLVR. But there might be some other relatively straightforward ways to shoehorn continual learning atop LLMs. For example, one could imagine making supervised fine tuning a tool call for the model. So the outer loop RL is incentivizing the model to teach itself effectively using supervised learning in order to solve problems that don't fit in the context window. Now, I'm genuinely agnostic about how well techniques like this will work. I'm not an AI. researcher. But I wouldn't be surprised if they basically replicate continual learning. And the reason is that models are already demonstrating something resembling human continual learning within their context windows. The fact that in context learning emerged spontaneously from the training incentive to process long sequences makes me think that if information could just flow across windows longer than the context limit, then models could meta-learn the same flexibility that they already show in context. OK, some concluding thoughts. Evolution does meta-RL to make an RL agent. And that agent can selectively do imitation learning. With LLM's, we're going the opposite way. We have first made this base model that does pure imitation learning. And we're hoping that we do enough RL on it to make a coherent agent with goals and self-awareness. Maybe this won't work. But I don't think these super forcements of those arguments about, for example, how these LLMs don't have a true world model are actually proving much. And I also don't think they're strictly accurate for the models we have today, which are actually undergoing a lot of RL on ground truth. Even if sudden splotonic ideal doesn't end up being the path to the first AGI, his first principles critique is identifying some genuine basic gaps that these models have. And we don't even notice them because they're so pervasive in the current paradigm. But because he has this decades-long perspective, they're obvious to him. It's the lack of continual learning. It's the abysmal sample efficiency of these models is their dependence on exhaustible human data. If the LLMs do get to AGI first, which is what I expect to happen, the successor systems that they build will almost certainly be based on Richard's vision.

Podcast Summary

Key Points:

  1. The "Bitter Lesson" argues that current LLMs waste compute by using deployment time for non-learning tasks, with only training providing actual learning.
  2. Training relies heavily on human-curated data, which is inefficient, non-scalable, and fails to produce true world models capable of autonomous environmental interaction.
  3. Imitation learning is not fundamentally separate from reinforcement learning—it is a form of short-horizon RL and serves as a practical, scalable prior for learning.
  4. Historical and cultural knowledge accumulation, like human learning, is analogous to imitation learning rather than requiring full RL from scratch.
  5. Current LLMs demonstrate emergent world modeling through their ability to process complex, coherent information across domains, suggesting they develop meaningful internal representations.
  6. True AGI likely requires continual learning from real-world interactions, which current LLMs lack due to their dependence on pre-defined human data and static training phases.
  7. Techniques like in-context learning and supervised fine-tuning as tool calls could enable future models to simulate continual learning more effectively.
  8. While Richard's critique highlights valid structural limitations, the distinction between imitation and true world modeling is overly rigid and may not reflect actual capabilities or future progress.

Summary:

The core argument centers on the inefficiency and limitations of current LLM training paradigms, which consume vast compute without enabling true world modeling or continual learning. Most learning occurs during a narrow training phase using human-generated data, which is unsustainable and fails to produce agents capable of autonomous, self-directed environmental interaction. The speaker challenges the notion that imitation learning is fundamentally separate from reinforcement learning, arguing instead that it is a practical, scalable form of short-horizon RL that serves as a necessary foundation for more advanced learning.

Historical progress, such as the development of technology and language, mirrors this process of cultural imitation and knowledge accumulation. While LLMs currently lack true world models and on-the-job learning, they show signs of developing deep world representations through their training. The speaker suggests that future AGI systems will likely rely on continual learning mechanisms—enabled by techniques like in-context learning or fine-tuning as tool calls—rather than relying solely on pre-trained knowledge.

Despite Richard’s valid critique of sample inefficiency and data dependence, the speaker believes the boundaries between imitation and true learning are overstated, and that current models already demonstrate promising emergent capabilities. Ultimately, the path to AGI may not require abandoning imitation learning, but rather evolving it into a dynamic, continuous learning system grounded in real-world interaction.

FAQs

The main critique is that current LLMs are inefficiently using compute, as most compute is spent during deployment without learning, and training relies solely on human-generated data, which is not scalable or truly representative of real-world interactions.

No, it argues that LLMs develop models of human behavior rather than true world models, which would predict how the environment changes in response to actions, limiting their ability to generalize and learn independently.

Imitation learning is seen as continuous and complementary to reinforcement learning, with imitation being short-horizon RL where each token is a potential action, and the model learns from human data to build a foundation for further learning.

No, human data is seen as essential and beneficial, especially at scale, as it provides a foundation of accumulated knowledge that enables progress, much like fossil fuels enabled technological advancement.

Pre-training data acts as a crucial, non-renewable starting point—like fossil fuels—that enables future AI systems to build upon existing knowledge, even if it's not directly used for final learning.

Yes, LLMs can learn from ground truth, especially when initialized with human data, enabling them to solve complex tasks like math problems or code applications, though pure RL from scratch remains unproven.

Chat with AI

Loading...

Pro features

Go deeper with this episode

Unlock creator-grade tools that turn any transcript into show notes and subtitle files.