David Silver and Lasse Espeholt of Ineffable Intelligence: A Superlearner for the Age of Experience

collectively you know we started with this kind of gray mush in our head um we somehow started out knowing nothing at all and somehow discovered like everything that humans know today was like this process of collective discovery like of of languages and technology and science and mathematics and all the rest of it you know everything we learned we discovered it for ourselves and that's an incredible thing that no one has cracked yet in ai and so to me that is the essence of what it will take to get to super intelligence like a system that can just keep learning and discovering knowledge forever i'm thrilled to have dave silver and lese as behold with us two of the co-founders of ineffable intelligence uh great name by the way fun fact dave you were the first person i called emailed to come on training data when we started the show a hundred ish episodes ago and it feels like a full circle moment to finally have you on the show so thank you for joining us um dave let's start with you your 2009 phd was on reinforcement learning applied to the devilishly difficult game of go you demonstrated for the world for the first time that superhuman in one domain machine intelligence was possible and i think you were really the figure that inspired the entire current generation of of reinforcement learning and and deep learning researchers and so thank you for your contributions uh to where the field is i'm curious um thinking back to that point in time what did you learn what did you have conviction on and what has remained the same and what has changed as ai has changed uh over the course of the last decade i think it's been really interesting to see what you've learned and what has remained the same and what has changed as ai has changed over the last five years or so yeah thank you and pleasure to be here and i'm glad i finally made it after these hundred episodes it's a pleasure to be here um so i suppose i always had this conviction starting my phd and going into um the game of go that the right approach to solve ai was was through learning from experience through systems that discovered things for themselves that that were learning through reinforcement learning to get better and better and better and that that would be the thing which would lead to the emergence of intelligence and ultimately super intelligence and so i suppose we were able to show that in in the game of go but it's a conviction that has held true throughout my career and um here we are now now applying that that conviction ineffable trying to show that indeed um super intelligence can emerge all the way through through systems that learn from experience and in terms of things which are different although we are not pursuing the the human data path that the llm world has pursued i do think there's lots of interesting lessons to learn from that um in particular it was it was definitely a surprise to me how a system that was trained on a massive amount of data could then um be very quickly um fine-tuned on a a very small amount of data to kind of completely shape um change its behavior that was something which i think was something which sort of changed my mindset in terms of you know just suggesting that you get things at the right scale that a lot of things which seem like incredibly hard or like they might be very slow that once you've kind of experienced enough data that after that the things that were slow can become very fast i think that's a a lesson that wasn't obvious to me in advance until it until it happened okay we're going to come back to that later um you could have run this program inside deep minds uh where you know could have probably had more access to compute than than just about anybody you decided the right thing to do was to start a new lab uh why'd you do that yeah um so look first of all i had a fantastic time at deep mind i i loved my years there and um i have a you know huge amount of respect for the people there and still get on with them very well but at the end of the day you know i have a really strong belief about how we can go beyond where we are today with ai and all the way to super intelligence and the path that i believe in um doesn't actually go through llm-based products as an intermediate point and when i really thought about that i thought about it really hard um i realized that i felt that so passionately that this deserved to be more than just the corner of a of an organization that's focused on on lm-based products and deserve to be um represented by uh an elite ai lab that really just gives this approach its all and and represents it 100 and you know once i realized that it felt very natural to to uh create this company i suppose another way to look at it is that you know the world already has tens of thousands if not hundreds of thousands of really talented researchers all pursuing exactly the same approach like really just kind of doubling down on on llms with pre-training post-training you know coding and and and that kind of llm path it am i going to be the hundred thousand and one person that's like pursuing that approach uh is that the most effective use of my time and my my energy and in the end i i realize that there's this other lamppost that right now no one is looking under and where if if i actually put my effort there has a much higher chance of really having a massive impact on the field because if this works it's going to change the whole of ai um so i think so i think that that kind of realization also meant that this was something which just means that you know it's the best way to have impact upon the field through through this approach plus you had a company name that couldn't go to waste so that's right yeah do you want to say where the mind's called ineffable yeah so i suppose um the the name ineffable it came from this idea um that you know there's really just knowledge could be so incredible like the things which which super intelligence will one day discover like this knowledge is going to be so unbelievable incredible and it's going to be something which is you know profoundly amazing and and perhaps so profound that we can't even express it in in human language and so really it's like you know in line with this other path that doesn't go through through you know human language-based approaches but instead build systems that discover things for themselves and has the the possibility to to discover knowledge that's unbelievably profound and and ultimately accessible and i think that's that's unbelievably profound and and ultimately ineffable i should say there was also a secret second reason that that we used um which was um that you know it's this idea it really it has to work like if you really just double down on learning from experience it's an approach that that just has to work um and so you could say that it can't possibly f up so it's it's ineffable so there you go there's the the secret reason as well amazing amazing but best lab name um let's say that you you were at google you were in a different group than dave uh in the in the deep learning group at deepmind and you spent years scaling up reinforcement learning on gpus what made you decide to leave deepmind and what drew you to uh to co-found ineffable right um yeah so i joined like deepmind in january 15 and actually i started working a bit with dave around like 16 where i was doing this work on like scaling up reinforcement learning um so as you said i was part of like the deep learning team and i was part of like the deep learning team and i was part of the reinforcement learning team and he was like leading a reinforcement learning team but we were working on an internal project together i actually left deepmind twice so i left deepmind in summer 18 and went to like google brain then eventually google brain and deepmind merged and then i was like back again but then day i left deepmind again like a year before dave left um and the first time i heard about ineffable was actually through one of our eventual investors it's like reaching out to me asking what i thought about dave well my first like reaction was damn i missed out of an opportunity i should have you know reached out to him but i was you know i recall like dave as more often than not being one of the smartest people in the room and you know shortly after that uh sync with that investor um they actually reached out to me so i was a great yeah so i really love to work with the dave um i knew why tech and chris uh from back in the day and i thought like the team was a great and i also thought it was like amazing to have like an opportunity like this in europe it should be said as well like um we we worked together um on some of the early approaches to to um combining reinforcement learning with deep learning and trying to scale things up and and it should be said that lasso who by the way is very modest but he actually worked on some of the original approaches that that um were the first attempts to actually scale up reinforcement learning and show that it could be done at large scale and like ways to paralyze reinforcement learning and ways to kind of build out architectures that were scalable and if you look today at the way the field's gone like actually all of the ideas that people use today kind of um sort of hail back to to those that original work that that lasso did back in the day that's where all the roots come from of of um scaled up rl so he was the the first to figure it out wonderful we should have you guys introduce each other instead of you're both too humble to brag about yourselves okay let's talk about ineffable and what you're building you have a beautiful mission statement as a company you're going to build a super learner and make first contact with super intelligence can you define both of those things for us what is a super learner and what does making first contact look like yeah so so to me a super learner um it's it's something which is it it's got incredible capacity both in its its depth of what it can learn and in its breadth so in the depth dimension it should be something that can start from not knowing anything at all and can go all the way through to discovering things that go way beyond what humans know so an analogy to this would be the way that human collectively you know we started with this kind of gray mush in our head we somehow started out knowing nothing at all and somehow discovered like everything that humans know today was like this process of collective discovery like of of languages and technology and science and mathematics and uh all the rest of it you know everything we learned we we discovered it for ourselves um it wasn't that someone came down and gave us like a blueprint of how this should work and it wasn't that someone came down and gave like a a trillion element data set of like oh this is what these things should look like we actually figured it out for ourselves and that's an incredible thing that no one has cracked yet in ai and so to me that is the essence of what it will take to get to super intelligence like a system that can just keep learning and discovering knowledge forever so the second element is breadth that we want something which can do this incredible discovery from like all the way um to incredible things and it should be possible to drop it into any stream of experience so whether you know it's it's like a simulated world or a digital world or a physical world you know whatever it's dropped into it should be something that can just learn and learn and learn so that's the dream i think if we crack this then to me it it gives rise to a super intelligence that will be like it'll be like encountering magic you know it'll be something which is so so profoundly amazing that it will it will be be able to do wonderful things what will it feel like when we first encounter it or is it one of these things where you you can't predict ahead of time i think it will be amazing i think we'll we will you know that we should have an experience of like wonderment and the only thing i can say from my own experience is what happened when we got to this point in an admittedly much narrower field but when we got there in in in board games you know when we got to the point of superhuman play in in the game of go like it brought wonderment to the to the field like the the the go players who who saw like alpha go playing you know they they were amazed and like it was it was like a joyful experience for them to kind of witness this kind of go playing same happened when we got to superhuman play in chess and you know the chess players they they were so excited and and just imagine like going far beyond that that like in you know things which are discovering incredible things new fields of science new types of of of technology that we've never even dreamed of i mean it will be it'll be incredible so the current paradigm these large language models uh with reinforcement learning seem to be you know at least knocking out millennium problems and making some sorts of scientific discoveries does that you count as a super learner or what's the difference between something that can make novel scientific discoveries versus a super learner in in your sense of the term yeah so look i think llms are doing a lot of great things and i have a lot of respect for what's being achieved what i mean by a super learner is something which first of all discovers completely autonomously um and i think what we've seen so far at least with these kind of approaches that have been taken is there's a lot of human in the loop um effort that's required to to make these breakthroughs uh it's not fully autonomous yet and it's also something which should be applicable to kind of um you know real world embodied domains where where where the agent can like learn from from its physical experience and we're a long long long way off that one it's not at all clear that llms and language-based approaches are even the right like vehicle to achieve that that kind of knowledge thinking about these things in terms of language might not even be the right paradigm and it might just be that all of this human data that's being used in llms is kind of weighing things down so a lot of respect for some of these uh the progress that's made but i don't think we're anywhere close to this like true super learner of something which can be like autonomously just discover everything for itself what does building this look like in practice for you and you know we backed janice who was on your original alpha go team and he he told us that you know one of the things that's distinct about you is you can have these incredibly ambitious visions but you also ground the team in very practical uh research milestones and and i'd love to understand big big big vision what does this look like in the day-to-day and what are you what are you implementing how do you spend your days right now yeah great question so um so day-to-day um i suppose there's a lot of hard technical work with um by people like like like lasser to kind of build the whole stack that means we've got you know a state-of-the-art reinforcement learning stack that allows us to run things at scale um inhale data um at the optimal way be able to kind of learn and and have systems that keep learning and learning and learning and then on top of that um so we've spoken about the super learner the other aspect which we're building is um we're building out environments which are are capable of super learners like discovering everything like environments in which simulated environments in which in which a super learner can discover everything that we mean by intelligence uh we call those crucibles um so they're environments in which it's possible for agents to to make incredible discoveries in which it's possible to express and learn everything that we think of as intelligence and so we have been putting a certain amount of effort into creating these environments um and so it's a combination of of building out the super learner doing the research to make super learners incredible and and beat the state of the art of what's possible with reinforcement learning um both in terms of the engineering and the research that happens and creating and placing the super learners into these crucibles to kind of see the emergence of of incredible behaviors and in terms of the grounding you asked about how to ground this in in the day-to-day so we ground it in in milestones and the milestones are kind of analogous to you know what happens with the human um journey um where we discovered incredible things and and so those become natural milestones for us to for our agents to rediscover the great moments that that humans went through in our discovery of knowledge what do you think the agents will rediscover first when that's going to be your first moment of oh this is working i imagine we're going to be surprised like i think one thing which is different about the way we take to doing reinforcement learning to a lot of the rest of the field is that we are not breadcrumb trailing the particular um objective to be learned about a lot of the way things people do rl with llms at the moment is to kind of say oh you know the the agent should be learning about this problem and then this problem and this problem and after kind of following that trail we're kind of pulling the agent by its nose towards a particular desired outcome we're not taking that approach we're giving the agent some simple objectives and asking it to you know discover things that will will surprise us and amaze amaze us so you know the the agent is in principle capable of discovering all kinds of things and we don't necessarily know in advance what those will what those discoveries will be and it's kind of just going to be exciting to to witness that journey so so i hope to be surprised i think is the answer to your question interesting so so what you're saying is instead of explicitly giving the agent the goal to go rediscover language or you know build a calculator or build a computer you're going to give it other more open-ended goals and the agent may discover that the most efficient way to actually or the best way to actually accomplish that goal maybe to build some of these tools that we as humans have already built is that is that one way of putting it that's one way of putting it i think i think another way to say it is people often think in rl oh my god how do i come up with a reward function that kind of perfectly specifies this really complicated thing that i want to discover like you know how can i how can i specify exactly what i mean by by science or how can i specify exactly what i mean by interesting maths but there's an alternative um and the alternative is that to give the agent something you know really straightforward optimize but in a really complex environment where there's a lot to learn about and so it turns out that you know if you look at the way the world has worked in in nature the pursuit and optimization of a very simply easy to describe quantities can often give rise to incredibly hard to describe complexity and emergence so so examples of that would be you know evolution which kind of just optimize a kind of survival of the fittest um um you know some kind of fitness signal and the pursuit of of of fitness gave rise to all of the complexity that we see in in life um in the human journey like the pursuit of relatively simple kind of goals that we as humans have gave rise to like all of the complex behaviors that humans ultimately um acquired and all of the knowledge that we eventually ended up with so the kind of emergent complexity is much greater than the kind of um starting point that was given so it wasn't like you know evolution that was given so it wasn't like you know evolution arithmetic was useful and then we arithmetic was useful and then we arithmetic was useful and then we started to discover that started to discover that started to discover that you know tools were useful and then we you know tools were useful and then we you know tools were useful and then we discovered that some discovered that some discovered that some basic science was required to help basic science was required to help basic science was required to help understand how the tools would work and understand how the tools would work and understand how the tools would work and that gave rise to more complex technology that gave rise to more complex technology that gave rise to more complex technology and understanding the technology required and understanding the technology required and understanding the technology required more complex maths and so forth and so more complex maths and so forth and so more complex maths and so forth and so forth so all in the pursuit of relatively simple goals, this incredible complexity emerged. So if we can get that process right, you know things are going to take off and and and lead to you know unbelievable things it's fascinating are the rewards for your agents kind of similar to like human biology and evolution then like you know try to make it to the next generation um what does a reward function look like for you all yeah i mean we've we've we've been experimenting with quite a number of different rewards so it's not like we have one answer to that question um but i think one thing we're not doing is is that kind of you know crafting a reward that exactly describes what we mean by science or exactly what we mean by technology or exactly what we mean by language we expect our agents to discover things and if we can't crack that you know like if we can't crack this problem as ai scientists we're never going to have agents that discover whole new fields of knowledge that we don't even know about there are vast fields that will exist that are out there that which we don't even know um have the potential to exist at this point in time you y'know it's like imagine imagine you're back in you know the days of newtonian physics and and no one even knew that like quantum physics was a thing or or that you know relativity was a thing like we should be in that process of open-minded discovery where our agents are free to kind of figure things out for themselves and i think that's going to lead to you know much more exciting outcomes what do these crucibles look like in practice and you know you're contrasting it to today when people say environments oftentimes it's a it's a set of tasks and you know you're contrasting it to today when people say environments oftentimes it's a it's a set of tasks and you know what the agent should have done on the task it sounds like your environments are your crucibles are very very different what do they look like in practice so we are building crucibles which are without giving too many details away have the capacity for a lot of emergent complexity like the key thing about them is it's like it's not like we've kind of handcrafted a whole load of puzzles for the agents to to to work with um there's just a huge amount of emergent complexity there some of that emergent complexity can come from different places it can come from from like physics engines that give rise to kind of complexity as you've got you know millions of interacting objects with with like a complex physics engine you can have millions of interacting agents and uh and and the world becomes much more complex because it suddenly now and count includes the complexity of all of those other agents within the the world that the agent's building so there's ways to build worlds that are vastly more complex than the kind of simulators like you know atari 57 or whatever is that that people have been working with in in the past and which is certainly more complex than the kind of standard benchmark kind of tasks that people are using um and if you get that right you have a world in which there's the possibility for discovery by the agent there's the possibility for the agents to discover everything from from technology to science to mathematics to languages to everything else you know just the same way our humans all the way through from from more basic things like you know motor control or um coordination you know the the whole spectrum is possible like if you look at like a game like go right which is like very simple to represent it's just like a bit map uh the simulation is extremely simple um but you still see this like emergent behavior of something complex right it's like super interesting to us so like of course like go is like way simpler than we can imagine doing uh but at the same time going all the way to the realistic physics will be like very computational expensive right so we want to like create this range of crucibles where we have like simple ones that can execute quickly and we can iterate on but where we can still see like very complex uh behavior interesting yeah it's almost like what you choose to represent is is uh it's tricky right because there's for example i can imagine a lot of economics arises from game theory um and most of that is completely unrelated to physics and so there a crucible that represents physics is probably not that helpful but if you have information theory represented who which agents know what then you probably do end up discovering rediscovering quite a bit of economics is that is that a fair way to put it like the the primitives you choose to capture in the environment kind of give rise to the type of learning that emerges yeah like the end goal is to have something that is grounded in in the real world right but stepping stones i want to come back to the go example because i really like this example you'll be surprised to hear that i found going to be a good example um but what what i find fascinating which i think is what lass is um talking to here is is that you know go is just like a 19 by 19 grid where you're trying to kind of maximize the territory and if this was a single agent game it would be like the most boring game you could imagine it would be you know you just place down stones on a grid until he fills up the whole grid and there'll be no strategy whatsoever the moment you put a second agent into the environment you get this emergent complexity which is suddenly um because the agent contains like one agent has to reason about all the possible behaviors of the other agent suddenly it kind of massively increases the complexity in the environment and and to the best of our knowledge you know um certainly all of the compute which we sank into it when we were working on it there was just like this endless learning curve that where with without end you could continue to discover new things in this game none of this is to say that go is like the end point of course it's still just a board game and quite narrow in a certain sense but it shows that very simple environment dynamics can lead to kind of infinite like an infinite onion of layers of complexity that kind of never stops infinite to the capacity of what our computers can do of course do you think that the agents will end up rediscovering language and will that language look like ours or will it look different i mean my personal belief is that um anyone who thinks that human language is the end point of of like or the best way to to to reason it's it's you know that that perspective is a little naive i think that it must be the case that there are far more sophisticated approaches to to reasoning you know maybe using um paralyzable rich complex um approaches that neural reasoning or whatever it is these approaches will will ultimately be far more expressive um and far more capable than approaches which are based on on human language i would also say that starting from human language as like the the the approach for reasoning is somewhat problematic um because it pulls in all of our biases about how we think agents should think so for example a nice way to think about this is to imagine um you know what if we were to step back you know 500 years or a thousand years or two thousand years and only like were able to use all of the language that that humans used at that time um so maybe our agents would believe that the world was flat or or that the diseases were caused by uh the four humors or um there'll be all kinds of you know strange beliefs that would be embedded in that will be very hard to kind of remove because they're so fundamental to the kind of language and the processes of thinking that that humans have used so i do think that that you know going beyond this um is is a really important goal i'd love to talk more about this concept of removing human priors uh and i think your experience with alpha go and then alpha zero alpha go zero uh is maybe the the best example that anyone can point to of what happens when you remove human priors maybe if you take us back to alpha alpha go and then alpha zero was it surprising to you that removing human priors would remove human priors results in even smarter intelligence i wasn't sure honestly so i remember i remember i i always like to do things in research where i think um you ask research questions which are like on a knife edge where it's like 50 50 um i think it's not very interesting to do research where you already know the answer and you're like 95 certain this thing's gonna work it's kind of like boring honestly to do to do those things so you know i think i think research good research is about asking questions where you're really not sure what the answer is in advance and in this case i was about 50 50 and i thought that maybe there was a good chance that it might um but i wasn't confident and and it was very exciting to see it um i should say that you asked about human priors so so we are making a bet right ineffable and the bet we're making is that we can go further by giving up like on on the priors that everyone's relying on today like the human data like the whole internet of knowledge that that everyone else is is building their systems upon that's a uh an important choice and so we should probably talk about like why we're doing that and i think the reason for me of course it's you know a short-term sacrifice like we we give up a certain amount of performance um in order to access something which is potentially far greater and it comes back to this like you know what does it mean to build a super learner like to me building a super learner again is about building something that that has this like potential for for full depth and breadth so in the depth direction like we need to get to something which is capable of discovering things which goes so far beyond human knowledge that that you know it goes just far beyond the prior and and the more we lean on human data the less able we are to actually discover the algorithms that are really going to take us all the way to something that's going to for example discover a whole new field of knowledge and then in the breadth direction you know we need to have algorithms that can can be placed down and discover things in in environments for which language and human data that we started with is just not a good fit i mean i mean you know there are just many many areas you know different kinds of sensors for example um where where human language data or or human data in general is not a good fit for some of the this this description and so so i think by letting go of this this um the human data we put ourselves in a position where we can actually discover something which can which can like get go far far beyond it so if we come back to the alpha go example what we found there was that the moment we let go of the human data which we'd spent by the way we'd spent about two years like building alpha go somewhat similar to the llm approach where we kind of you know we built every level of this you know we kind of hand put together the next level we kind of um put in a lot of human data and then we were kind of human in the loop like improving it where we would kind of take the previous generation and choose how we were going to improve it to give the next generation and and so we were right in the loop of this process it took us two years to get to alpha go and beat lisa doll when we got alpha zero right you know we gave up on the human data and finally focused on what would it be like to build an algorithm that could discover for itself um so once we got it right that algorithm it took just a few hours like it took a few hours to like outperform the two years of work which we'd done as with human in the loop um so in the in the game of chess it took four hours to beat the world champion the best ever ai that had ever been created in the game of go it took um you know like uh i think it was a couple of days or something but it was incredible progress like and and so you have to let go of the human data to allow yourself to discover these things that can help you build your own algorithm and you can't do that without you have to learn how to build your own algorithm and you have to learn how to build your own algorithm then shoot past the the competition and discover the things that just keep learning and learning and learning forever so so you know it has to be an active choice and if you don't make that active choice you know what i see in a lot of the kind of llm companies today is they're basically you know building uh semi-autonomous human the loop kind of learning algorithms with thousands of people in the loop of the learning progress which is you know it works and and a lot of respect for how far it gets but you know until we step back from that and really discover the process that's going to learn for itself i don't believe we'll get all the way to something which is truly a super learner and unleashes the potential that we've talked about so far that's very profound as far as i know it that there are humans at ineffable and um some of you are creating environments or crucibles and what does it look like to actually try and make sure that human priors aren't kind of leaking into what you're doing it was so when we design like environments um and like rewards function and so on like obviously there's a human bias in there but we don't want to as dave was also mentioning earlier we don't want to like bias it towards uh building a machine based on like the von neumann architecture or something like this right we want to like but if if you're stating like um some problem where like a machine would be useful it would be very interesting to see what type of machine it comes up with like what computer like machine it comes up with so i totally agree with this i would also say that we're not religious about this it's not like a religion that you can't have any human prior anywhere um it's more like a pragmatic position that you know until we give up on on like relying on the trillion like item data sets of human knowledge that we're not going to make the progress that we that the field deserves to see so it's more like a actually a practical position and so in practice of course there's going to be a certain amount of priors which like as lasser says there'll be priors which which apply to many different like worlds um as opposed to like you know trying to put in these very strong priors that exactly encode how all the knowledge we know about about this one particular problem or this one particular world so kind of weak environment agnostic priors seem perfectly reasonable but kind of these super strong priors with like trillion item data sets we're trying to avoid that because we think pragmatically it's going to lead to better outcomes yeah absolutely you mentioned that you like philosophically that you're drawn to research that's more on the knife's edge 50 50. i'm curious what your confidence levels are for you know ineffable broad at the broadest level like does it will a super learner emerge and then i guess at the more pragmatic level in terms of the things that you're trying to accomplish right now are you are you 50 50 on these as well so first of all i would say i strongly believe that the approach will be feasible at some point like we in fact it's the only thing for which we have an example of like you know i'm not going to go into the details but it's the only example in our brains is the only existence proof of something which has discovered like an incredible intelligence that's very general and works in you know embodied world and discovers how to do all kinds of different things so we have this incredible example of the of the human brain that um we can look at so then you know do i think we can achieve it well i think like many great like moments in science it's about timing so so you know it's all about timing like um you when you create a startup you want to be at the right time you you don't want to be too early or too late you know if you're too late everyone's done it you know it's not interesting like a thousand other people have already done it and if you're too early um you know you might be banging on the door um but not have the the right kind of you know maybe the field isn't advanced enough or there isn't enough compute to be able to succeed yet so to me it's all about timing for example with computer go um you know i made a prediction i think 10 years before um computer go was solved and i got it right to the year um i got it right for um starcraft which i also we had a bet and made a prediction um saying for alpha proof uh when we predicting when when the imo was going to fall so you're a super forecaster of sorts so so far so far i've done very well at this um and i do have a very strong sense that this is the right time for this to fall that super learners are possible like there's been enormous progress in the level of technology the the ability to accelerate research using kind of um llm coding methods or or auto research methods like like we've produced in the past uh the ability and understanding of scaling laws and and the sophistication of technology and the stack there's a lot of things that fall into place that i think make this the right time so that's my bet that this is the right timing but yeah of course like this is research yeah um research is never guaranteed and um and that's what makes it exciting if you get the timing right you can be the first you can be the ones to change the the whole field of ai and um you know um that that's that's what it means to to take risks and do something exciting for ineffable and also all the previous things that you correctly forecasted is the biggest driver of timing compute i think it's a very big driver i think it's an enormous driver i think um yeah i mean we're we're in a great position with the compute that um that we've we've managed to get hold of um and i think it gives an opportunity to do things which are you know orders of magnitudes um more sophisticated than than approaches that have tried to kind of do self-discovered rl in the past so i think yeah if we compare to i don't know something like hide and seek from open ai which which was like an early example where where agents discovered um kind of some basic tool use um i think you know we could probably run experiments that like uh use a hundred thousand times more and so it's kind of amazing that people haven't come back and really pushed on these types of approaches given the the availability of compute and of course hide and seek i i don't want to pretend that that's the end goal i think it's it's very basic but i think it just it's suggestive of of what could be done like if we really you know so i think compute is a huge driver but it's not the only one yeah i think we're in a very fortunate situation that we have been able to secure like a very large cluster right so at a large amount of beautiful cluster at that yeah so so um um so this is like really like an opportunity for us right like we really have like the cars on our hands like to test this out um very few other people are doing something similar so yeah we can we can really test this out in a way that we couldn't do at deep mind for instance absolutely it seems like one of the maybe determinants of how far this can go is how far you can push simulation and there's ultimately you know the principles are you know they're they're digital right and i think simulation has probably come quite far in the last handful of years uh but it's not it's not perfect right um uh we recently had your your friend rich sutton on the show and i asked him about simulation and he said he was very dismissive of simulation um i'm curious how you guys think about this and unless you've been on both sides of this i think you you obviously did a lot in the digital world built metnet uh which you know touches real world atmospheric uh data how do you all think about where is the ceiling on simulation and what when is the right time to bring an apple into the real world so i guess metnet was actually like an example so this is like a weather forecasting model we did in the google brain amsterdam office together with google research where like we were doing like a machine learning model to predict short term like weather forecasts and there's like way better than the physics-based simulations right like so the physics simulations were like accumulating errors over time so there's like a point to that that that physics simulation is not perfect i do think there's there's a lot of relatively simple environments that we can build where we can see this like emerging behavior of like let's say like technology and so on where we don't need like exact physics and i actually do think it's interesting to see how much we can build these days we got like a very small team in terms of these simulations and so on so there's definitely like a ceiling um is there a ceiling actually i'm curious about this yeah well i guess quantum physics and whatnot right but but so there's like a ceiling somewhere but but we can do like very interesting things without real physics i would say so i think the right question to ask is can a super intelligence be built in simulation so if it is possible for it to be built in simulation i believe it will happen 10 times or 100 times faster just because simulation can be infinitely paralyzed you get perfect reproducibility has a lot of really nice properties it's also the safest place for a super intelligence to be built so it has these really nice properties then we need to ask is it possible um and and i think for the reasons we discussed earlier there are very clear approaches to building a super intelligence and i think it's a very clear approach to building a super intelligence and i think it's a very clear approach to building a super intelligence that we can add into the simulation um that allow it to be sufficiently complex to represent arbitrarily interesting things so for example like let's say you wanted to build a computer in simulation well all you require is that your simulation is is turing complete if you have a turing complete simulation suddenly everything that we mean by computation like everything like provably everything we mean by computation can be expressed completely inside simulation and indeed humans have built incredibly complex computers or um related things in inside simulated worlds the same is true for various other um ideas like you know if you want to build an economy like you know money can be built out of coins or out of paper or out of bits um and as long as you have um one of those you know some some representation of money everything that we mean by economics can be can be expressed inside a simulation the same is true for language if you've got multiple agents and they've got some uh ability to express a channel of communication between the agents suddenly everything that we mean by language is expressible in simulation mathematics we know is expressible in any universe science in some sense is the study of our world so will a simulation be sufficient to to study what we mean by science well there are some aspects of science which are like expressible in many things like you know energy or entropy or evolution but also it's it's true that you know the agent will discover its own science like whatever the simulator is if it's got this emergent complexity to its environment the agent will discover its own science about whatever that world is it's very like provably possible to build environments in which it's possible to express and represent enormous complexity um of all of these things well i agree with lasso there might be some things which which aren't well expressed you know there will be things which can only be expressed in the real world like quantum computation is a good example the real point is that if it's possible to get all the way to a super intelligence in simulation it will happen faster and it will happen safer and so for those reasons it's something that we choose to to put a lot of energy into um i actually agree with rich saturn that um that you know i think his point was something along the lines that you shouldn't expect that the things that you learn in simulation to like transfer like sim to real into the real world because the real world is like just like vastly more more complex and and in very interesting ways that are hard to simulate but i think what we can do is we can do a lot of things that are hard to simulate and we can do a lot of things that are we can do all of the science to discover the super learner um and um and and once we have that super learner we we we will have the answers and we'll know how to build something which that that same algorithm should be applicable to the real world and we can also you know make our simulations more and more high fidelity towards the real world and check that they still work before before putting them into the world great answer thank you i'd love to talk about alignment and maybe less i know this is a politically charged topic right now and so i'm less interested i'm i'm zero interested in having the politics or policy discussion around alignment but i am very interested in the scientific discussion around it it seems like so much of the misalignment that's emerged has arisen from you know scaling reinforcement learning to the extent that that it's been scaled do you think that the super learner you build will naturally be aligned to human interests um and have human values is it important for that for it to have alignment with with human interests human values so so this is a a big topic so let me you know start by saying a few things so so where we are today with alignment we are essentially relying very heavily on alignment approaches that are based on the language that the agents use um and priors that are based on on human data that has been reasonably been very effective up to this point but we're starting to see that that approach is reaching its its limits and by itself i think it's going to be difficult to see a path where as agents become inevitably more and more powerful we can already see this happening as agents are trained more and more they they are kind of um going to start to overwhelm their priors they're going to start to learn to say one thing and act in a different way um and it's going to be increasingly difficult to to align to the things that we uh that we that humans uh want them to do so my belief is that we need something more um and that this by itself is not sufficient the most important single thing that we can do as a company um is to actually understand the the science of super intelligence so that this means to say that if we actually can in simulation really build a super intelligent agent and start to see these super learners like go all the way um we can start to actually understand like what it takes for a super intelligence to actually um interact with peacefully um so we can answer really important questions like these are fundamentally important questions like you know under what conditions like what are the what are the conditions what are the the the structures what are the kind of leverages that we need to put into place to make sure that that like a society of super intelligences actually interacts peacefully and in particular interacts peacefully with other lesser intelligences that are also going to be in in their world and i think if we get this right um we actually can get to some answers like to some a real science that will help to understand like what the right things to do are in the real world because we're hurtling towards that real world outcome where there are going to be a whole load of different interacting super intelligences um and and we don't quite know how that's going to play out and i think the faster that we can actually get to these answers in simulation and publicly share that um the better situation we'll be in um so that's my strong belief um i also believe that we can use this to kind of gauge the the actual decision making of of like what we do in in the real world so you know one of our um commitments is that we will gate anything we do in the real world on having sufficient um evidence from this kind of science of super intelligence that we are really confident that things are going to go well um and we'll only push forward um to the next level of what we do in the real world once we are um absolutely confident that that can lead to a good outcome um so i think this is these are two things that i believe we can do you know we're in this world where where um a lot of things are happening very quickly we're hurtling towards an outcome with lots of interacting super intelligences i personally believe that this will lead to a good outcome for humanity but it's happening very quickly and i think that if we um are able to actually understand this and and and run ahead in simulation we'll be in a much better place to actually do the do the things that ensure that this goes really well fascinating and it seems like one of the primary things that current safety researchers rely on is being able to monitor the agents chains of thought in english and so when they start speaking the release that scares people because we can no longer understand the monitor it as well in your case you have your agents kind of going straight to rediscovering language what do you think that means for for monitoring and is or is the point that you know if you get the reward right and if you get the environments right then you shouldn't need to monitor yeah i think i think monitoring is is still really helpful um i think that like it or not i believe that the approach of monitoring the human language that these systems use is going to be increasingly difficult to as agents get more powerful and we're already seeing that that the monitoring you know agents are you know maybe they're producing like a tome of you know a million words of of of reasoning um or maybe they're actually actively you know using sub fuge to kind of think in one way and then act in another um it's increasingly difficult to rely 100 on that um so i do think we need other elements in our armory anyway um so i think it's actually really important that some people are looking at how to get this right i think there are ways to to monitor agents that don't that don't use language um i think we can do things like um start to for example you know you can use shadow um actions where you start to have actions taken where you can look at what they would actually do if they were allowed to act in the world but you don't actually let them take that action yet you can start to have other um super intelligences which are acting as the as the monitor so for example you might um have the idea of like a an incredible scientist agent that sort of treads lightly upon the earth and then you have another super intelligent that's kind of monitoring the kind of magnitude of its actions and regularizing it to make sure it can't do things which are kind of affecting the the world too much i think there are lots of ideas i would say that there's a lot of research to be done here i'm not claiming to have all of the answers but i think it's something that that deserves deserves a lot of investigation. And if people aren't looking at this because they're only looking at the language-based approach, I think we're in trouble because I think that language-based approach is not enough in itself to actually get to where we want. I should say in terms of having said in trouble, I am optimistic about where the world is going. So by trouble, I mean that we're not going to have agents that do the things that we want them to do. I do think we're going to end up in a world with lots of complex, interacting, super intelligent beings. And I'm optimistic that that will lead to a kind of a balance of actors, some of which are supporting the flourishing of humanity, and some of which may not be. But the ones which do support the flourishing of humanity will end up kind of supporting us and protecting us against the ones which aren't acting on our best interests. So that's the world I think where we can work. Yeah. Well, given your super forecasting abilities, I'm happy to hear that. What do you think about all the conversation around pacing the frontier? I'm actually very much supportive of the idea that there should be pacing of what happens for the frontier of AI in the real world. At the same time, I think we should move as fast as possible in simulation. Now, the reason is that I think in the real world, there's a lot of things we can do. And in simulation, we can actually develop a lot of things like the science of super intelligence that will actually help us to find solutions that help things to go well in the real world. And I think that kind of work should happen as fast as possible to make sure that when things really happen in the real world, that things have the best possible opportunity to go well and lead to human flourishing. So I think that would be my thought. But yeah, in general, I'm supportive of the suggestions, like the kind of direction that people like Demis and Daria and so forth are suggesting. I think it's generally a good idea. Maybe part of that is that I think that things are moving really quickly. And I do believe that when we accelerate AI and get to super intelligence, it will lead to a really great future for humanity. But I also think that it's something that the speed at which things are moving is not something that we are capable of adapting to that fast. So I think just by pacing a little bit, I think it gives us the opportunity, A, for our governments and our organizations and our emotional responses to adapt to the changes that are coming. And also for this work to continue a pace at really figuring out, in simulation, how to, you know, the real science of doing this, doing this right. So accelerate in simulation so we can understand what's happening and, you know, do science around it and then be careful in the real world. Yeah, I think that would be my summary. Yeah. I'd love to transition to talking a little bit about compute. This is going to be an incredibly compute expensive journey. And you are pushing the limits of what compute can do. I mean, both Google and NVIDIA have been great partners for you. Can you talk a little bit about your cluster? Like you're not simply renting spot compute from Neoclouds like many other Neolabs. You're actually, you're doing some really serious work on your cluster and the different other forms of compute that you have. Can you just talk about your compute setup? Right. So, yeah, as I mentioned earlier, like we are very fortunate with the compute that we have. Like we are the first customers of the largest Ruby cluster on GCP by the end of the year. Well, actually started like one month now. So we are like extremely fortunate in that. And I actually feel quite bad about like the startups that are like getting compute now because it's becoming like harder and harder. But yeah, so we have like this like Ruby cluster, which is based on GPUs, right? Like NVIDIA GPUs, which is really fortunate for us because it means that we can easily do simulation and like training and inference, right? They're quite like general chips. So that's like amazing. I think like the Ruby specifically, they have this architecture where like one rack can basically consider like one machine. There's like a very high bandwidth between the nodes and one rack, which is very important for reinforcement learning where you have like really fast iteration cycles compared to LLMs that are relatively speaking quite slow. So that's quite fortunate for us. I think like some of the things we did like eight, 10 years ago at Google where we basically had like abundant CPUs at Google. That doesn't scale anymore. So we really need to make sure that everything we do is on the GPUs and we don't need like millions of CPUs beyond these GPUs. So our simulations can run on the GPU inference and training and so on. So we have some good architectures for that. I think like the LLM companies now have kind of like the problems that we had many years ago where they need like sandboxes running on CPUs, like simulating computers and so on. And we're trying to not put ourselves in that situation. All the compute is, is it roughly fungible between the different things that you're doing? I imagine you're, you're building crucibles and then you're kind of doing these big rollouts where the agents are, are actually learning. Is that, is that the rough buckets of what compute is being used for? Yeah, you could split it up into training, inference and simulation. Is the same compute equally good at training, inference and simulation? No. So I think it's pretty clear what has happened with GPUs over the last many years or like since deep learning took off is that there have been more and more tailors towards training, right? And, and maybe in the last few years also inference, but less about simulation, right? So, but we are like, similarly to the LLM companies, we also want to be in the case where like scaling laws matters. Like we know that roughly, like if you add this amount of compute and experiences, then we get a certain amount of quality out of that. So I guess if, if GPUs aren't great for, for scaling simulation, then, then what's the solution? Right. So, so there are many different solutions here, right? So there's like model-based reinforcement learning where like the model becomes much more expensive compared to, to the simulation. So there's many different tricks you can do. There's like various approaches for like replay-based learning, right? So there's, there's a lot of, there's a lot of, there's a lot of offers, so like reusing data many times. So yeah, there are ways around that. And all in that we do spend quite some effort on just making the simulations as fast as possible and make them use, make the best use of the GPUs. Are you seeing scaling laws yet, or is it too early to say? So for reinforcement learning, like scaling laws are like slightly different, right? So like scaling laws is the component of how much compute you put in, how much data you put in, and like the model size. Well, reinforcement learning is a little bit different because the problem is like non-stationary. So as you progress in the problem, the data changes, the data distribution changes. So they are a little bit different. It's like a very interesting research direction. I understand they're different, but are you seeing some, some sort of scaling law where more compute in gets you closer to achieving the goals or, or is it hard to say right now? Yeah, but it's, it's something where we are actively pushing to towards. Yeah. And especially as we like transition from like a smaller cluster to the bigger cluster, we want to be ready for that. Yeah. They say at the beginning when David introduced you, he mentioned that you had done some very foundational work to compute some algorithms for reinforcement learning upon which the field really relies on today. Maybe talk about some of that work and in particular your, your work around Impala and Seed. What, what are those and how do they relate to the work that you're doing at Ineffable? Right. So when I joined like the deep learning team at DeepMind, most people were running like single machine CPU based reinforcement learning. So the scale was like quite small. So what I did together with colleagues was to make an agent, reinforcement learning agent that really utilized GPUs that could scale like linear with compute. I think like partially because it was very efficient, it was very good at scaling. And like the third thing was it was, it was very easy to extend and like modify and do whatever you wanted with it. Contributed to it being like the most popular reinforcement agent at the company. And at a thing at one point, most RL team used it or like derivatives of it. So that was quite successful and quite a nice project to work on. Then like later on, like we got TVUs, right? We got like racks of these TVUs. So we need to scale beyond that. So some of my work at Google Brain after I left DeepMind to found the Google Brain Amsterdam office was to scale to like racks of TVUs. And that project was called like Seed RL. And so there was like really about like utilizing the TVUs both for like inference and training in the best possible way while keeping the environments as flexible as you wanted to. It could be like anything from Go to very expensive physics simulations. And how's that being used? Yeah, so well, we actually implemented a few of those algorithms, also like derivatives of these algorithms. So that is like what we are using today. We have like depending on like the environments we use like different algorithms right there like pros and cons between them so yeah also we talked about like we have some very simple um like fast to simulate environments and also some that are more expensive so for those who are working on large-scale rl architectures today and haven't read those original papers it's definitely worth a look because i think you'll discover that you know many of the ideas that are currently in use like kind of trace back to those uh yeah early early work that that lesser kind of did and and figured out which was uh yeah a little bit prescient i would say i think it's actually kind of fun to see um the reinforcement learning space now versus like 10 years ago or like let's say like eight years ago maybe um like the llm community is also using reinforcement learning in particular the last two years so and like a lot of the things that they're using are you know being rediscovered approaches that were used uh many years ago so it's kind of like interesting to see with a lot of novelty as well on top of it right but yeah can you talk a little bit about your team how big is the team right now we have intentionally kept the team relatively small um so we're not in a hurry to build a a giant organization i think it's um you know you can do an awful lot um nowadays with a small number of incredibly talented people so we have hired 20 people in total um and yeah the people we've hired it's it's about finding some of the exceptionally talented people that um but not just it's not just about talent for us it's also about bringing together people who are um kind collaborative human beings that's really important to the kind of culture we're trying to build and people who are you know really well aligned with the kind of mission that we're following um and people that sort of fit the the the needs that we have um so we try and build the team with the right right kind of people and we're trying to build a team with the right kind of people set of skills as well um so all of those four things are really important and um and yeah we kind of have very high standards and very high expectations of the people that come in so we'll be very lucky to get um some yeah amazing really amazing people and so just to tie it to the compute as well right like give like this relatively small amount of people so far means that we have a lot of compute per person right and like these days you can you can do a lot of experiments just like one person and then you can do a lot of experiments just like one person so it's really also to keep the compute to person ratio quite high so we're not like fighting for resources uh from day one i think three of the co-founders worked together on on alpha star um many of you many of you did work together are you are you assembling the avengers from especially given the recent changes from deep mind are you assembling the avengers of folks that you've worked with in the past have you hired new people and if so kind of what are the attributes that make somebody new to you um likely to be um you know i think it's it's a it's a it's a it's a it's a it's a it's a mixture so i mean alpha star was an exceptional team um it was sort of intentionally assembled within deep mind by choosing the kind of best of the best to work on that project so in some sense it's not a surprise that i think actually four of our co-founders come from from that team um but we are not exclusively um drawing from that team or indeed from from deep mind i mean the other three founders uh all came from um very successful startups um and we hire people who are brilliant um but also kind and nice people to have around um who are going to work together well who are aligned with the mission and who uh you know fit the things that we that we need so um but in terms of finding those people we put a lot of effort into getting the right people so as uncomfortable as it is i've had to say no to a lot of people who are brilliant um but also kind and nice people to have around um who are going to people i've worked with in the past just because our bar is so super high you know i've worked i've been fortunate to work with like some amazing people and so you know i've been choosing amongst like a a very exceptional prop and bringing in yeah what i consider to be a yeah superstar team um and so yeah it's it's um it's it's not easy to get into ineffable but um the people who we have are wonderful human beings and yeah very very proud of the people we brought in yeah i can see why people are drawn to working in ineffable especially the time when you know the the biggest clusters the biggest labs are kind of you know they're in they're locked in this prisoner's dilemma race and you know there's a lot of dollars at stake and so you know decisions get made differently i can imagine for for a set of people that especially really really deeply believes in the scientific mission of what you want to accomplish that's what i'm trying to say yeah i think that's a really good point yeah i think that's right and we've tried to establish a a research-led culture so it's it's not a company that's led by you know building an incremental next product v 17.1 on the thing that's come before it's a company that's built on uh researchers pursuing um their passion and trying to produce the breakthroughs that they believe are possible and we center the company around that from the outset to create like a haven for research you know where where we push back on on on product pressure because we believe that if we make these breakthroughs it will lead to you know far bigger products downstream far bigger successes than if we kind of start off from day one on like you know oh you know here's the product we're doing from the beginning so yeah so it's research-led we're creating um uh i think a really special environment and yeah people are loving it um i'm loving it yeah yeah wonderful wonderful um i'd love to close with some predictions maybe especially since you're such a super forecaster what's your guess for when when you'll have the first strong evidence that the approach is working and then what's your forecast for when we'll achieve the mission that you've set out to achieve this first contact with super intelligence yeah so these are fair questions um so there are good questions there are good questions there are good questions there are good to be some reasonable bars of uncertainty on this but you're being recorded so we're gonna hold you to it exactly yeah um so i think we will see some extremely strong evidence that we're on the right track um within a year and i think we will achieve something which is a true super intelligence like the according to the definition i gave earlier of something which you know discovers things which are far beyond what humans can achieve in across you know everything from science mathematics technology languages etc etc i originally thought within three to five years um my timelines have accelerated a little bit so i'm gonna say three years because of um yeah what can be done with um with like coding agents to accelerate things um i think um yeah i think things are gonna move fast buckle up it's gonna be a journey wonderful uh let's say david this is a wonderful wonderful conversation um i'm so inspired by what you're building and thank you for taking the time to sit down and with our audience what you're building today thank you sonia thank you you