Effective Accelerationism and the AI Safety Debate with Bayeslord, Beff Jezoz, and Nathan Labenz

eAchro bias towards maintaining an adversarial equilibrium. So that means open sourcing models, keeping a healthy developer economy not centralizing the compute, not centralizing the algorithms in the knowledge. AI's will keep other AI's in check, just like mega corpse keep other mega corpse in check. So Microsoft competing with Google on this AI aspect is an example that you fight power with power. And I think that maintaining this adversarial equilibrium is the way forward. And that's only possible if there's not a monopoly on intelligence. And so we're fighting anti monopolization of intelligence fight. We raise the variance and accelerate. So that's the EAC way. Welcome to another episode of Moment of Zen. Today we're blessed with the presence of Bayes Lord and based Beth J. Zoes. You don't know who they are. They're the founders affect effective accelerationism. You don't know what that is. It's a philosophy that counters effective ultrism and aims to accelerate technological progress and all the benefits that come with it instead of slow development down in the name of AI safety. In this episode, we impact what exactly is effective accelerationism, whether it's intellectual foundations, how exactly is it different from effective ultrism? And we have an AI safety debate joined by my co-host of the cognitive revolution Nathan LeBetz. This was recorded Monday before the SBB news, which is why we don't mention it. Okay, onto the show. Moment of Zen is brought to you by Riverside. The platform Dan and Tony O'Neil used to record all of our podcast episodes with remote guests. Riverside captures exceptional audio and video quality, makes incredibly easy for us to record conversations with multiple guests and then edit and publish within minutes. If you're hosting a podcast or often getting interviewed, user code Zen to get a 20% discount at Riverside FM. The link is in our description box. SecureFrame is the leading all-in-one platform for security and privacy compliance. SecureFrame helps you get sock-to audit ready in weeks, not months, and it's used by thousands of companies like Angelist, Koda, and Remote. I believe in the companies so much I invested in it and I recommend it to all my portfolio companies. Sign up for free demo at SecureFrame.com and mention Moment of Zen during your demo to get 20% off your first year of SecureFrame. So we have Antonio Big-timing us again. Yeah, we're at Riverside and Tony O'Neil. I just question his commitment to the show. Yeah, for people who are listening, make sure you add reply Antonio on Twitter and tell them to come on the show. Yeah, and Neg Him and pretend you don't know about his startup or his book and just you know him from Moment of Zen. We're here today joined by Nathan LeBenz from the cognitive revolution and our special guest Bayes Lord and based Beth Jesos. Guys, welcome to the show. Thanks for joining. Thanks for having us, Eric. Beth, what won't we start with you? Why don't you introduce the effective acceleration as a movement, YAC and the origin of how you came to start it and then Bayes Lord will have you get into it? So I guess I'm Beth Jesos online in the and on Twitter sphere. Yeah, I mean, I guess the background on willing to share is that I used to be a theoretical physicist. I've worked in Big Tech in ML and yeah, overall ended up accidentally with Bayes Lord and some others sparking a very active community online called you know, our ideology is called effective accelerationism, kind of a play on effective ultrism to some extent. It is based on some ideas and we'll get into kind of its influences and Bayes is very well read but on this and can have a lot of comments on this but in our case, I think it started as a just a you know, hanging out in Twitter spaces, creating an on accounts, posting on Twitter, there's an account that's pretty famous in AI circles called the rune at TSDZL I think. And kind of the social media graph in the neighborhood of rune is a very interesting part of Twitter, literally called this part of Twitter or T pod, right? And essentially this is a movement that started within this part of Twitter, T pod, focused on techno optimism and being optimistic about the potential of AI. And I've sort of account our movements to pervasive, humorism or pessimism and demoralization that we feel, was pervasive especially in when was it started? 2020, late 2020, 2021. I think people were pretty, economy was crashing, things were dark, it was still the pandemic and I think thinking about a better future and discussing where is humanity going and all this? We've been through a couple chaotic years, what does the future hold? And just having discussions online actually in Twitter spaces ended up accidentally sparking this movement. So I think I think originally, Bayes and I would just chat like weird hours of the day after our big tech jobs, our days were done and just talk about kind of the future of the universe and everything. And eventually some folks end up noting down some of the things we were saying and to bullet points and kind of putting out a sort of shit post like blog post and in fact, it kind of took off from there and I think it was Zestie and other account that coined de facto accelerationism and yeah it just spread like wildfire in small circle of anonymous accounts in tech and eventually we decided to write a more formal set of notes on some of the principles after having many chats about it and that was another blog post which was kind of principles of antennas of EAC and from there we kind of just let go and it just kept growing and growing and now there's quite a few well-known figures either involved or accidentally aligned with the movement and it seems to be taking off especially as AI technology is accelerating and so is the knowledge of AI, doomerism or kind of the popularity of doomerism I think with the recent podcasts on that and so as a counter movement to doomerism and what we call deceleration or decels more broadly, for us it's been an exciting time but I can dive it so that's the origin I can dive into what exactly is effective accelerationism what are kind of the the principles behind it and what are we united by in terms of where we think the world is going that'd be helpful why don't you give a brief overview of what are the what are the main tenets the high level and how does it overlap and how does it differ with effective altruism yeah that's a that's a pretty that's a pretty deep question there I mean hopefully we're going to get to dig in especially with Nathan joining us as well I look forward to that so yeah so effective accelerationism is about recognizing that there's this thing called the techno capital machine which is kind of the capitalist system coupled with technology and it's this machine that assigns capital to things that have utility towards its own growth right and you could think of the capitalist system almost as a form of life right life is self-replicating its self-adaptive and it's always seeking to reassign usually energetic or chemical capital towards things that have utility towards its survival and growth right and you have this happening on all sorts of scales and effective accelerationism is about recognizing we're in a self-adaptive system that can kind of figure itself out and by just contributing to this system you're ensuring a better and brighter future for all and it's kind of a counter movement to more top down utilitarianism right it's kind of like more of a bottoms up emergent altruism and it's about kind of having faith that the system will figure itself out right and everything co-evolves with one another and we don't have to worry or try to control everything every aspect of our lives in a top down fashion to maximize safety because essentially the system tends to converge onto assigning capital to things that have positive utility to the world and it does so in the most efficient manner compared to sort of more spreadsheet ethics or spreadsheet utilitarianism and so you know some folks think that eac is strictly a reactionary movement to EA it's not quite it's mostly about techno optimism and you know building encouraging others to build and encouraging others to contribute to the advancement and growth of civilization in the way that is of the highest impact right but how do we measure that is different than let's say EA's where where utilitarians would measure either heat ons or utelons you know how much suffering reduction in our case is more how much are you contributing to the scope and scale of civilization as measured by kind of our energetic consumption of free energy and turns out that this principle is kind of the the loss function or the function that is optimized naturally by nature nature tends to want to grow organisms that consume energy and and prefer statistically speaking configurations of matter that are consuming work for energy and the point of eac is to recognize this and to kind of lean into kind of the natural forces guiding the progressive civilization so it's kind of like it's a very meta philosophy it's a sort of meta culture recognizing that growth is good and contributing to growth is the ultimate form of good and most other forms of good are kind of projections or specializations of this kind of meta optimization so that's kind of a very high level overview which our Bayes Lord can give a less theoretical physics coded background on on eac but yeah I'll stop here for now it sounds like a spiritual argument for capitalism right sort of I mean it's trying to understand kind of where are we going what is the driving force that has brought us to this point it's kind of like a self adaptive system of cultures of technology of we call them meta organisms so that could be groups tribes corporations nation states and all of these competing against each other at all scales has yielded a very robust system that's kind of dynamically adaptive and it's about recognizing the mechanisms that lead to successful and sustainable growth and anti fragility of of civilization and kind of trying to be proponents of these principles and reinforce them rather than kind of you know demonizing them or saying that we should decelerate or a capitalism is bad and we should you know even though the market has deemed certain causes to be of low utility and has assigned low capital to it you know for example eAs would want to kind of do deformations of the markets right like take capital from one place and extract and manually inject it elsewhere which is kind of kind of fluctuation that tends to dissipate over time and so we saw it with FTX and SBF right he was trying to kind of extract capital using crypto well not crypto large but you know crypto exchange and was trying to reassign it at all sorts of organizations in places that you know wouldn't necessarily have had capital without his manual assignment and I guess our thesis is that you know the market usually figures out what is important manually reassigning capital from one place to another is not a sustainable practice it's much better to build a company if you want to see a change in the world right it's like contrasting ESG efforts versus Elon Musk right Elon Musk just builds companies he creates a meta-organism that is self-sustaining and attract capital because it is growing and going to grow its amount of attraction of capital over time and that creates a self-sustaining organization that has lasting impact in the world and objectively you know like Tesla has had massive impact on green energy much more than these green energy funds have had in my opinion at least but happy to be corrected there. Bays would you would you like to give your take on how to summarize YACC? I think that's like a really good summary um one of the biggest things that comes to mind to me about the the the critique of EA is just this um I don't want to treat it as too monolithic but if we look back at the history of EA really on there was a pretty clear and sort of united focus on buying I sort of say this like buying cheap today utility they were like if you look around money is not being allocated very well in these certain places for example uh like we can very cheaply purchase malaria preventing bed nets or deworming medicine and and these sort of like global health things were were pretty common uh commonly coming up because uh they're yeah there's there's some opportunity there to buy what yeah cheap today utility but over the history of EA as you know sort of in like the novelty seeking of the uh of the ideology kind of the community over time it seems like people were very attracted to this notion that actually there's a lot of utility out in the future there are going to be so many more future humans future life forms then there are today in the future and the further we go out the more that's true and therefore uh we should try to focus our efforts on helping them instead of helping people a lot today but the issue is that when you actually ask how expensive it is to buy that utility it's much more expensive uh then buying today utility or tomorrow utility something like this and that that is a function of it being difficult to do planning and prediction in a chaotic complex system at very very long-time horizons and so you're the way I sort of think about this is that on a long time scale you're spending your the money that you're using to try to pick up more utility is getting you know ever more diluted with with every time step so you sort of imagine this branching tree of futures right it's like your money is getting strictly down and it's just you know sort of exponentially diluted now of course that's a function of how correct you are uh in your predictions and how you're targeting things but I think that's probably the main thing I want to say I think uh you know bethue always do a great job summarizing it so yeah I would say that um you know main tenet of yack is that top-down control doesn't work very well for self-adaptive systems with emergent properties like economies or biological systems like controlling you know spread of viruses and so on um and I think the same is going to be applicable to AI where there's kind of these emergent properties when you reach certain scale of these systems and trying to have top-down control of them is uh it's kind of a moot point or very difficult yeah overall we think that there's a there's a tendency in culture that we're seeing or a trend in culture towards sort of very uh high centralization and suppression of variance for more optimal control and so whether that's culture suppression of cultural variance or suppression of variance of say potential outputs of a of a machine learning system in general there's kind of a trend towards centralizing uh for easier control uh as opposed to decentralizing and suppression of variance and overall we think that in order for civilization to advance the most rapidly and for us to reach a prosperous future where the scope and scale of civilization has increased substantially we need to embrace the variance uh we need to embrace freedom efficient marketplaces of ideas of neural nets of that cultural cultures uh etc uh and why is that well essentially you could think of of sort of parameter hyper parameter landscape for for all these things whether it's uh how to how to run a company how to set up a culture of an organization or a nation uh or how to tune the architecture of a neural net um everything is optimized through this kind of a competitive landscape and a sort of evolutionary process and the speed at which this evolutionary process can evolve is bounded by uh the variance that you allow uh for this evolutionary process and if you suppress variance you move very slowly which makes it very controllable but it also makes it kind of um crystallized and static and as the fitness landscape of uh cultures of waste organized oneself of how to tune a technology is always changing you want to maintain that sort of adaptability and dynamism in order to always be uh optimal so yeah so so essentially you know effective accelerationism is about it's it is somewhat libertarian about encouraging uh variance experimentation building towards a better future but not through kind of top down control prescriptions of how to how to live live one's life or how to uh run your company or how to tune a neural net it's more about uh bottoms up uh marketplaces of everything and encouraging uh sort of this evolutionary adaptive process in the space of everything to run its course wherever that may take us and and it's an argument for the fact that this is the best way to go if we want to ensure kind of robustness and anti-fragility of of everything at all scales and so it's kind of a you know fine principle it's anchored in like thermodynamics i won't go down that that whole rabble hole here i could talk your ear off about that but yeah uh essentially you know we were getting worried with kind of like several sort of uh movements that we're emerging that kind of are coercing or co-opting organizations throughout the western world that's kind of um seek to suppress variance uh instill certain opinions and and favor kind of top down control right um and we needed a counter movement that was bottoms up and uh you know it's natural that it emerged from anonymous circles because uh twitter and on's you know many of us work in in tech and so on and an attack vector is you know uh threatened things one's uh employment if one doesn't comply with kind of the the prescribed subspace of opinions and so you know in our in our case we just want to argue for variance apparently that can be a controversial idea in some circles and so it's natural that it emerged from anonymous uh twitter accounts uh and but now obviously there's all sorts of uh real names that have uh joined joined the fight i mean you you've hosted a omjad uh you know i mean uh i think mark and jris since pretty public about his bio change and um obviously there's there's some inspiration from biology's work on uh network states and so on and so uh you know it seems like there's you know this counter movement is is reaching well more of the mainstream of tech obviously not the core um you know broad mainstream but overall it's a good time to have kind of a counter movement to uh effective altruists and AI existential risk doomers because otherwise it would be unopposed and such an ideology could do quite a bit of damage in terms of demoralizing people about the future. iakka's about building it's about building towards a better future it's about finding the points of uh leverage to you know that can really increase uh the scope and scale of humanity but working on on deep tech working on technologies that really move the needle in terms of like how how big of a civilization can we sustain with these technologies and so it's kind of less about writing uh less a culture about writing or word celery as the the twitter anons would say um and more about uh building or as as said on twitter uh shape rotation which is that's a whole meme we can get into but uh yeah so uh it's kind of a revolt by the engineers that feel perhaps suppressed in their roles in large orgs that are very top-down uh controlled internally um and it's been a way for uh builders to express how they feel about their role in society and what they're contributing to civilization and how they can feel happy with their day lives contributing to kind of a broader goal of civilization and all advancement and so you know to us that's the antidote to dumerism and we're trying to spread it i found it really interesting and and refreshing that there is a kind of bottoms up uh to use the analogy of the cathedral versus the bazaar this feels a little bit more bizarre like and um one one thing that you mentioned in terms of like civilization and energy uh there's a terrific book by a stanford professor Ian Morris why the west rules for now and he he's a he's an archaeologist class assist and he tries to take 15,000 years of human history and have a normalizing function across how do you measure civilizational progress and his his attempt whether or not you agree with it i mean go read the book uh is he he breaks everything down in jewels and just like what is the average person in a given civilization um and and he spends most of the book contrasting China and and the west at whatever point rom medieval europe uh you know age of discovery colonialism and the charting the the kind of ability to capture energy per person as kind of the measure of progress i thought it was just a fascinating thing to try to take such a wide chunk of history and break it down to kind of a quantitative metric and then obviously if you think about kind of where we've been since the 70s it's like we kind of have this up into the right uh increase in energy per person and then we decided that efficiency was the thing to focus on and and and we don't actually i'm I'm pretty sure that average american energy usage has decreased since the 70s and it's kind of crazy given that we we we have fish and power right we're working on fusion but but fish and does work and it's extremely safe like we could be in an environment where we as a civilization continue to invest in additional per capita energy usage and i don't know like how much better would our lives be today if we had that much more abundant cheap energy and and so i it's an area that i am i'm very excited about of and and what's interesting is also there are other people on the political spectrum like the as reclines of the world and Derek Thompson talking about abundance and and and building and so i do think there's a counter to the uh humorism of just generally whether you take climate change or agi of everything is really bad we need to take control and uh manage it versus like no we're gonna you know to take the term from the martian we're gonna science the should out of this and we're gonna actually move civilization forward across every realm that we're participating in. that's i got to read that book um i i think i do think that um if you so so some of the crazier parts of of of eac are kind of taking inspiration from the the mathematics of how life emerged right there's these works by jeremy england at mit trying to understand the physics and the thermodynamics of life and trying to apply that sort of thinking to kind of economies replacing you know energy with capital is kind of some of the thinking that's that's influenced eac but ultimately you know energy and money are kind of interchangeable and it turns out that at least from the point of view of life right like all forms of life uh or trying to optimize their capture of free energy for their own sustenance and growth and and um you know replication and evolution emerge as a byproduct of this tendency which you can derive from just pure thermodynamics the universe prefers futures that have uh lower free energy uh and so it it's kind of basically favor configurations of matter that are replicating and just seek out all the free energy universe and consume it uh and so on and to some folks it's like okay well why don't you just light the whole world on fire well actually that's sub optimal on large time scales right it's much better to have a self-replicating uh organism that then goes finds pockets that are much further out in the universe with either some some form of intelligence uh and go consume that then to just like you know have everything explode in one go uh so you know just burning fuel locally is kind of a local optimum in this landscape and life itself is kind of a more non-local optimum but if you think about that um essentially every organism or sub organism including yourselves your neurons and so on every um human and group of human and so on they're all trying to uh optimize uh uh how uh secure they are in their growth uh and they're trying to secure resources and the goal of intelligence is to give you the tools to find and capture and and be able to harness energy from the environment your growth and replication and so we do think that energy consumption is a good metric a good benchmark for the progress of civilization because it comes from kind of the first principles of how we we benchmark uh life so it's really interesting that you know this uh uh Ian Morris uh guy uh had written about this in the past definitely get to check that out um but ultimately I think you know there's a notion of the Kardashev scale I don't know if you're familiar maybe for the listeners Kardashev scale is kind of a logarithmic scale about um uh measure trying to measure progress of civilization in terms of how much energy consumption from the wattage that we have um and I would summarize the goal of EAC as kind of uh climbing the Kardashev gradient try to find the direction of most rapid increase of uh free energy consumption but on a on a large time scale so you average out um you know on where we are on the Kardashev scale on a long horizon and our goal is to maximize that so it's to maximize kind of the scope and scale of civilization on kind of an infinite time scale we we'd like to discount over time because we can't necessarily predict everything in the future and the further out you go it gets exponentially harder to predict so you can discount it exponentially but like we do in markets when there's an interest rate um um but that is kind of the the function we're optimizing in terms of effective accelerationism and the word effective is kind of figuring out you know gradients are like what is the what is the action or what is like my how do I change my actions or my policy or my culture of my day-to-day life to maximally influence and climb this Kardashev gradient um and so effective accelerationism is kind of like how you know I have a skill set or I could gain a skill set some information I have I have some capital I have some leverage how do I maximally leverage what I have in order to most positively influence the growth and scope and scale of civilization and that's what really being an effective accelerationist is at least that's how we're defining it um and so you know there's a lot of parallels with EA right that EA has a different loss function they're trying to minimize uh suffering and so on at least to me as a physicist or former physicist um a very subjective loss function um can lead to kind of weird anomalous solutions like uh wire heading right so um bays would you like to explain what wire heading is uh and how we should probably avoid it yeah um yeah I like to think about wire heading is sort of a spectrum of things um um right but the sci-fi version of wire heading as you guys probably know uh right so uh so you know in the future some uh neural link type device uh gets implanted in people's brains you know probably stimulating the um reward centers uh and people sort of just get stuck in these very sort of simplistic lives where they just sort of self-stimulate and uh have uh uh sort of all of their pleasure comes from from this and it's uh yeah I think it's meant to be a pretty dystopian vision when it comes up um but yeah this is actually I mean in my opinion this is a spectrum of things right so you can sort of uh I mean even you know twitter or something it's like a very very low grade version of of wire heading right you're kind of self-stimulating uh with just like very repetitive stimulus right in a way I think maybe take talk is an intensification of that for me personally that that's with my experience uh I don't have the app on my phone uh is just found a way to intense uh in that way um but uh yeah yeah I think that pretty much explains it um and if you're optimizing for kind of minimizing suffering and maximizing pleasure it leads to a lot of the technologies we have today right they're just trying to you know give you dopamine or whatever they're a tonin uh and it's stuck in a local optimum this is not what we should be maximizing like our pleasure we're just pleasuring ourselves to death right and instead if we were focused on kind of growing our capabilities in the worlds of atoms uh necessarily means we need higher levels of intelligence in order to capture the free energy that's why we evolved intelligence but you know higher levels of intelligence that of a civilizational scale could be you know using AI to crack nuclear fusion right using AI to uh design materials to capture carbon to um you know harness all sorts of forms of energy around us right with higher levels of intelligence we can find pockets of energy around us that are much harder to access and so we think it's a much better sort of function to uh optimize for that kind of not only pushes our energetic consumption but also necessarily increases the amount of intelligence in the universe because in order to consume more more energy uh you need more intelligence and so um yeah that's the gist of it back on the the point about uh centralized or top down control um just one thing it's not just that it's not very good at accomplishing the goals that it sets out for itself um it's that it actually has a tendency to backfire in a lot of ways i think like the go-to classic example is like central planning in Soviet Russia right something like this um because the system that is doing the control uh necessarily has you know like bottlenecks and aggregating information about how the system is functioning taking measurements uh and and bounded intelligence and planning uh due to like like historically due to coordination between humans um but like there is a more general like limit there um uh what what you get is actually that there are many unintended consequences and uh uh downside effects and so uh yeah we can talk about what those look like i think this is really interesting in the case of of talking about uh AI safety AI alignment AI risk all of that um so yeah i just wanted to point out it actually there is there is quite a lot of downside as well but before we jump into the AI stuff two two things off of what Beth mentioned before so i'm a total dyson sphere nerd so like when dyson spheres like that that to me is like what we should be heading towards for civilization and for listeners so don't know what a dyson sphere is it's a theoretical device that you'd build around a star to capture all of the energy so think of like solar panels but if you build it all around uh like a sun and then you have that much more energy to do all cool stuff for space varying and then there's a there's another thing that you made me think of and this is i think a good transition at the AI point is one of my favorite uh short stories from azimuth is one called the last question and the the the premise is humanity comes up with this really smart thing i want to say it was called multivac and and it's like at agi and they ask it a question they can ask it any question and then they ask it uh can you reverse entropy right can you can you stop the second law of thermodynamics and the machine can't come up with an answer it's like gpt breaking every single time and then it kind of progresses and i won't spoil the end but through thousands of years of human civilization in in a pursuit of this question of can you reverse entropy and civilization gets that much more advanced in kind of the pursuit of this question and then the the end of the the story i think is pretty spectacular and i won't spoil it you know it's a short story go read it but i don't know if you guys have read it absolutely love the story again i won't spoil it um but it's funny that like uh the goal there is to reverse uh you know or to violate the second law because in fact the generalization of the second law of thermodynamics to um not not equilibrium states right everybody learns thermodynamics in school about like if everything's at equilibrium here's a theoretical eat flux etc um there are generalizations to out of equilibrium systems right which is most most systems including most systems that are alive we're not just a soup of matter we are very much out of equilibrium and maintaining or out of equilibrium state with the environment is what it means to be alive right and so the generalized law the generalized second law is actually uh what has driven matter to organize into systems that are alive essentially right and it is this you know this law of the universe preferring futures that have lower free energy that the free energy has been consumed because free energy for those those not familiar is more or less energy minus entropy right and so you want to minimize free energy want to consume energy and you want to increase entropy in a fine tune balance though um you don't want to go too too crazy on the the entropy um but uh uh uh yeah so actually you you can apply this sort of generalized second law thinking to all sorts of systems and at least to me from an intellectual standpoint um creating this account you know uh experimenting with these writings has been trying to you know in a non-academic setting extend some principles from physics to understanding kind of uh uh economics and uh um you know systems comprised of of humans and groups of humans uh through the lens of of physics and in order to make predictions about where are we going on average i don't know the details but i know on average i can show that you know these futures are exponentially more likely and in fact if we get to it uh as as we're going to get to the the back and forth with Nathan Nathan we get to sort of the the paperclip uh maximizer uh uh uh fable uh and uh there's all sorts of uh AI existential risk fables like gregu and so on you can like show they are not uh tractable or likely from the laws of physics uh but we'll we'll get into that i think uh a bit later but i just want to comment because you you mentioned the second law yeah a few questions that will segue into the AI safety conversation one is uh at the end of jeffery west's book scale which is all about natural you know uh scaling laws he he quotes someone else where he says something like uh the only people that think run away growth economic growth is sustainable are uh idiots and economists um and so that's something that he's he's worried about and then of course this sustainability term begs question of sustainable for whom um and and so of these next question which is like how do you see the roles of a human of humans uh in in this kind of acceleration as world you mentioned in the manifesto that's not exactly transhumanist and and then lastly um you know if aliezer pick whoever um you know AI safety uh you know or doomer uh kind of you know smartest person uh was was listening in i'm sure they would agree with a lot of the kind of evolutionary descriptions but i'm curious like where where would the fundamental disagreement or fundamental different assumption uh be so there's there's kind of some some big related questions i'd like to see uh those transitions do awesome yeah um i i'd say like um at least personally my journey was to be a humanist first i wanted to maximize the scope and scale of human and civilization but once i understood more of how the universe uh actually works right like in reality every organism and meta organism is always like rooting for its own team right whether it's ourselves looking to replicate and and pass down their genetic code right the selfish gene theory um or it's kind of like cultures trying to to replicate or uh and be successful and kind of consuming the mind share or its uh companies trying to grow and capture the market um everything is competing for kind of uh its own interest and so if you apply this to all scales including him human all we know is that there's going to be a future where there's some form of life um and because intelligence is an evolutionary advantage it's most likely be will be very intelligent that will figure out how to spread even further than than our current planet will that be this direct descendants of humans that are biological substrate maybe will it be a sort of hybrid between a non-biological matter hybridized with biological beings perhaps will it be pure AI that are like von Neumann probes made of non-biological matter it's it's it's also potentially likely right um and i think like the thing about eac is that we're kind of open to whatever comes next as long as life in our corner of the universe gets to keep growing because in this sort of race against uh well this race following you know the tendency of the universe to consume more free energy you're either getting busy growing or getting busy dying there's no such thing as like um you know sticking around just staying the same um and so for us it's kind of like a sort of meta-humanism right where we're team whatever comes next and um so we're not necessarily saying that it's going to be humans like you know hybridizing with machine and then going becoming fully machine it might be that we do create you know von Neumann probes for those that are not familiar are kind of this theoretical concept of self-replicating machines that are kind of a virus throughout the stars and uh come to a planet figure out how to manufacture themselves and spread right um but essentially what we can what we can understand is that intelligence is an evolutionary advantage in this this competition or for resources or or even even this fight against the universe of finding pockets of free energy to to sustain and replicate oneself and so whatever comes next is probably going to be as intelligent if not more intelligent that we are and we're open to whatever comes next but we know that that's kind of where we're going and I think there's a sort of um calm and serenity that comes with accepting whatever comes next because because the reality is that the notion of human the notion of anything just drifts at all times right like our genetic code drifts we're always evolving there's nothing in nature that just stays the same and so I think that anchoring on like the good future quote-unquote being that we maintain the current state of humanity and maintain our local genetic neighborhood or local neighborhood in matter configuration space forever um I don't know I don't I don't think that's a good future I don't think that's anti-fragile against this stochasticity and things that could go wrong in nature and erase us and so for us we're all we're trying to kind of minimize existential risk to life itself as a whole and in order to do that we got a favor growth of civilization and life itself more broadly defined than just through biology um and so is that post-human is that transhuman I don't know those are old labels we're we're a whole we're trying to be a whole different thing that's kind of more open-minded I think it's like good to maybe emphasize this like I think I would be happiest to see all of those things sort of thrive uh right like this idea of having higher variance is is like pretty baked in and into uh you know the set of futures I think like we we prefer and I think are like sort of borne out by other principles um yeah but I think this thing about the sort of psychology or the psychological thesis of a viac right um it's like sort of equanimity in the face of the singularity or something um it's like pretty critical right when you contrast it with um you know 20 plus years of like intensifying fear uh you know increasing the migdala hijack at culture scale um right uh this sort of uh cleaning on to uh the way that it it it is or the way that it was um right there there's like a kind of deep palsy in there I think that is I mean yeah uh this might be my personal bias but it is I guess it's a little bit there's a little bit of like a Buddhist insight there right you you can actually have uh equanimity in the face of all this change and actually it's like uh if you if you can figure out how to cultivate that within yourself you're going to be a lot happier to address kind of the the second set of questions about underlying assumptions I think that uh AI existential risk folks think that um something a concept called fume uh or gray goo or you know kind of runaway AI progress kind of singularities exist um and at least something we believe in the iac is that that's not the case um in a sense like every biological system is always trying to quote unquote fume it's always trying to grow capture more resources replicate at all times but what keeps us in check is kind of competition limited resources uh and the laws of physics right it costs energy to for a biological system to optimize itself to find resources and grow and and replicate um and you know there's no such thing as sort of recursive self optimization uh for AI I mean in fact it's a topic of research called meta-learning uh or hierarchical uh based in networks if you want to get technical um but essentially we've always tried to have AI optimize itself since the dawn of time but it's just very hard because um essentially you you have that that minimum cost of of querying uh this this simulation either a simulation of a neural net or a literal instance of a neural net uh to to get a sense of like what value of every parameter or hyper parameter is better than the other uh and that has a thermodynamic or energetic cost to it uh so in terms of biology it's like you you fire up another instance of a human or whatever organism you're optimizing for either it's genetics or it's neural architecture and then you run that and you you feed it and then you run that experiment you see if it was successful or not it's similar neural networks you got to try to hold neural network and see it through and then test its its performance in order to to optimize the architecture um and so we we think that uh you know that will randomly tweak one thing about the architecture and then it will be like massively better uh all of a sudden and then recursively self-improve and runaway fashion is very implausible from the laws of optimization if you know if you're anybody who's practiced uh you know machine learning and applied machine learning at scale uh and also you know theoretically you could apply that same sort of thinking to biology you know maybe there's a there's a configuration of matter that just like just achieves crazy viral self-replication and biology is always trying to find that optimum but it hasn't so far so it must be hard and kind of maybe there's no such optimum uh and so i think as well like there's uh there's a few concepts of like like kind of very bleak futures that they paint as like uh uh you know the worst case scenarios for runaway AI and i think two of those are like uh gray goo uh and uh paperclip maximizers and so uh let me let me tackle gray goo uh real quick um and uh gray goo is like this assumption that AI if you have runaway AI uh it just figures out how to densify its intelligence all the way down to engineering nanotech that is like self-replicating and consuming everything in the world to turning into this kind of weird meta-organism of nanotech goo um and ultimately that's that that's just describes like most uh biological systems to be honest um but formally like there's no reason like an intelligence train on gpt3 or sorry train on scraping the internet like gpt 3.5 or whatever they're at now would be better at predicting how nature behaves than then uh you know uh one could obtain from querying nature and and running the simulation running the experiments physically nature is kind of it's the best simulator of itself at the nanoscale and so any sort of like simulation based uh argument right oh the neural nets just get a model how the world works and understand it and predict it and pop down design nanotechnology that's much better than biological systems at consuming energy and consuming everything in the environment for their replication that's just not plausible uh because uh nature has kind of a sort of uh irreducibility and it's how complex it is to represent kind of the emergent behaviors or behaviors of nature this computational irreducibility concept is something you know steven wolfram famous mathematician has written a lot about um and you know in a sense that kind of that's kind of a counter argument to like having a super intelligent AI will cause us to create nanotech that consumes the world so that's kind of counter argument against gray goo which is like a doomsday scenario that like you know AI existential risk people use to argue for you know regulating AI and stopping all progress um and so to us it seems like a very zero or near zero probability occurrence and so to weigh that as like infinite negative reward and get Pascal mugged by this potential future seems like a massive opportunity cost for good of like accelerating AI technological progress instead that can yield a lot more for civilization um and so to us like you know that's that's one way we differ quite a bit um when it comes to paperclip maximizer as I guess and maybe Nathan can correct me but the the fable is that um you know if you don't figure out how to uh how to align AI um and alignment is a whole uh topic now but alignment is kind of like align AI towards human goals uh if you don't do that then you're going to end up with an AI that has some weird secret goal that maybe a very low utility uh like just producing as many paper clips as possible right but the argument against that is that actually any organism or meta organism that doesn't assign all its kind of capital or resources towards things that have utility towards its sustenance and growth will get out competed by another organism right this happens in like organizations maybe they adopt a culture that's kind of uh woke or counterproductive and they start kind of crumbling or getting out competed by startups right we've kind of seen that happen with big tech I won't point point at fingers uh I'll let people other people figure out what I'm referring to but um you know essentially if you have a paperclip maximizer at any point that it starts producing paper clips instead of you know assigning resources to uh figuring out how to grow and sustain itself it's going to get out competed by some variant either variant of itself or competing biological or non biological uh intelligent system and so you know I think like these common you know doomsday scenarios like really fall apart once you understand some physics and thermodynamic limits of computation or understand some optimization theory and so you know but I'd love to hear kind of um you know any other counter arguments I want to say that like and maybe Bayes can add to this you know EA and EAC align on quite a few things right ultimately we're we both care about you know civilization and you know the benefit of all um we just think we have different ways of going about it uh and different like ultimately things we're optimizing for and in our case where we diverge is pretty mainly that you know AI is one of the most potent technologies for massive good and massive utility uh towards you know advancement of civilization uh uh and it we shouldn't kind of uh neuratically obsess over uh fictitious stories of how it will lead to our doom especially like such stories kind of fall apart as soon as you examine them from first principles um so I'll leave it at that for now but I'd love to your Dan and Eric's thoughts and then have Nathan jump in potentially and the one so I guess for starters I should probably you know say there's a lot of things I would agree with you guys on in terms of you know largely having been a libertarian for most of my the life and being kind of a you know a hi-ekian if you will definitely a big believer in you know bottoms up organization for most things anyway um and also pretty open to I think a lot of different futures uh it sounds cool to me if we could you know be a multi-planetary species and you know colonize the universe that starts at the galaxy I guess and then we can move on from there uh long way to go so I really do think that you know there's a lot that we would agree on I don't want to you know go into questioning from like an adversarial standpoint because again I do think there's a lot that I would uh we could find agreement on I guess where I kind of get off the train a little bit is it seems like a leap to me to say that everything will be fine which kind of sounds like the general conclusion right that like I'm understanding you correctly everything sort of competes and out of that kind of comes you know an acceptable outcome um I would say that seems like it is broadly true in like you know that my local grocer has like pretty good merchandise you know available and it's usually not toxic I do like it joy the benefits of that I'm a little bit not quite making the leap with you that we should accelerate everything um I'm also not quite there on that we should kind of accept anything that comes next you know I could very easily imagine something kind of entering into our world and outcompeting us and taking all the energy and you know going out and getting more energy and still feeling like I don't like that thing and it doesn't like me you know and we don't have any values in common and so I kind of don't understand um I think those are kind of my two big leaps that has been sitting here I've I've not been able to make with you and I'm very uncertain I guess just if further contextualized like you know if Elias are you know is kind of at the high end of like we're all gonna die for sure or maybe like Nate sores from Miri's even even maybe higher on the we're all gonna die probability estimate um you know I wouldn't be nearly as confident as they are but I also don't think I can get my nothing I've heard gets me lower than say you know a 1% chance that we're all gonna die um and to me that is like pretty seriously worrying and you know I don't I don't know I don't know that there's really much that you know just given the the totally unprecedented nature of AI in general I'm not sure that there's much that folks could say that wouldn't make me like be you know sufficiently confident so I guess I'd also really like to get a sense for where that confidence comes from in you guys I don't think most people are gonna make that leap with you based on what you've said so far yeah I mean ultimately you know um yeah like every every every living organism eventually dies off it leaves behind a sort of evolution of itself some sort of either genetic fork or potentially you know some work they've left behind some knowledge and that's kind of what they leave behind in in terms of like their imprint on civilization and you know ultimately there's no guarantees that humans live forever and every whether it's kind of like yourselves of a certain you know genetic neighborhood optimizing for you know the highest likelihood of their genes being passed on or you know that kind of instills upon us kind of a an in-group preference set all scales whether our tribe or our family are tribe and then our nation and then human broadly but um ultimately it's kind of like um you know if you only care about your team and the grand scheme of things there's no guarantee that you always win and even if you win for a while in general the notion of what is your team drifts and changes right and empires you know they start off a certain way with a certain culture and then there there's cultural drift and then eventually they crumble and you know um it's not uh clear to me why like humans are the final form of living beings uh I don't think we are I don't think we're adapted to take to the stars for example at all and we're not uh easily adaptable to new environments especially other planets and so on and so at least like if I just look on like the requirements of having life that becomes multplanetary we're not adapted for it and so it's going to be some other form of life that takes to the stars when we're another but on a shorter time scale you know that's on a far time scale but on a shorter time scale I think that um right now AI is not a it's not a living thing like it doesn't capture resources in the environment to replicate itself and so on it is just like neural net APIs running on GPUs in the cloud right and if you want to consider that quote-unquote alive uh then you know the selective pressure on the space of AI's right now is whatever neural net has highest utility to our economy and to humans we put the capital down and and put the energy down to sort of feed the GPUs that post the model and and and make it alive right and right now it's kind of fully dependent on us for its survival right and so right now we have all the leverage we could just unplug it right I know it's a meme but like it's it's true shut down the cloud it's not alive anymore so we have we have total leverage totally dependent on us so we're still very far from embodied intelligence we're still we're still not you know not close to having robotics figured out we can go why that is like motor intelligence is much more complex than the white color work like intelligence um but we're nowhere near embodied intelligence we're nowhere near AI's figuring out a whole supply chain on how to replicate themselves in fact we have all sorts of problems in terms of supply chain to even make just basic chips nevermind whole whole bodies um and so we're not even close to AI's being competitive with us now you could paint other doomsday scenarios where you know some AI's I don't know some government official interacts with chat GPT and becomes convinced to I don't know launch nukes or something but right now there's no selective pressure there's no radiant and incentive for it to to do that um and I mean I don't see how it would be more convincing than a than a human but again like we'll we'll see how these GPT bots or whatever get better over time so at least to me I just I'm just not convinced that AI is actually competitive with humans even if it like if like some nuclear powered GPU cloud runs some mega neural net that's technically smarter than any human on earth we still have all the leverage over it at this stage and so I'm not panicking yet but I agree that there's no guarantee that humans win forever and that our camp wins forever but that is just a universal truth that you you have to accept just like you know your your your your favorite sports team is not going to win the Super Bowl every year right um and so uh yeah anyways so so there's our some my arguments for maybe maybe we shouldn't panic as much um maybe do you have any like uh rebuttals or other things you want to expand upon yeah I mean I guess for one thing when you said uh you know we're nowhere close I was reminded of the thing that's been circulating on Twitter in the last day or two which was a list of AI achievements that we were supposedly nowhere close to as of 2021 when the list was published and now you can kind of look down that list and you see like five of the seven things on it have been accomplished so I would encourage people to just bear in mind that like even the predictions of the people in the AI space around what would be achieved on a timeline as short as like one to two years have recently been dramatically superseded and that could happen again it may not happen again um I kind of hope you know that things move a little more slowly than I fear that they might um which is not even the same thing as saying that I think we should take drastic action to slow them down but just that I think the pace of change is getting kind of uncomfortable and will bring you know kind of its own problems but even you know even more basically just is not something that we're like very good at predicting I think like the track record of predictions is just plain poor um across the board you know with few with few outliers who can say that they really saw where things are going um so that's one kind of concern so I want to concrete here though so I think Beth brought up two specific instances there's the AI now gets capable and smart enough in a reasonably short period of time that it can control the physical world and specifically manifesting into supply chain so that it can start to control the flow of resources and in the second which is it's so smart it can trick humans into doing what it wants like what which of those and and maybe you do the the trick humans into controlling the resources right or the nuclear missile or whatever is is the fear the tricking humans or is it it's just going to be so smart we can't even comprehend the the strategies you know to use the alpha go example what is like move 37 it's like oh that's a bad move but it ends up being a good move like what what what are we actually specifically worried about I kind of think everything's on the table to be honest with you I try not to over index on any one particular failure mode I've been kind of working on writing something that's like here's a long list of things that could go wrong that you know you may think none of them individually are super likely but cumulatively it seems like you know that whole list would be something that ought to be taken seriously you know the tricking thing goes I don't know if that goes back I mean in some sense the whole field of of AI was kind of founded on this notion of like the Turing test which is fundamentally kind of a deceptive frame which is probably a big mistake to put the whole thing in in that you know lens from the beginning where to define you know success as like tricking the user that's probably you know probably should be a little more careful about how we kind of set our success criteria but more recently Ellie as her post or you know positive this AI at a box scenario and actually ran an experiment with humans where he said I will pretend to be the AI on the box your job is to let me or not to not let me out of the box no matter what I say and I will bet you that I we can enter into this like private dialogue I will require that you don't tell anybody what I said but you can tell them the results of the experiment and he ran this a couple times with actual money on the line and I believe it was one of two people that he actually you know went to all the trouble to set up the bet and have the anonymous dialogue with ultimately came back to the public and said I let him out of the box you know and now I have to pay the money and nobody knows what happened in those transcripts but Ellie as her point was kind of like if I with all this setup can convince this skeptic to let me out of the box when he has to pay me conditional on doing so and he knows it's me because that's how we set the whole thing up then like it shouldn't be that big of a leap to think that you know a sufficiently advanced AI might be able to trick people and you know by the way the way that they're trained right now kind of incentivizes that because they are trained to maximize the human feedback score or at least that's kind of the naive implementation of like a reinforcement learning paradigm the AI learns to maximize the score you know whatever will get me the highest score from the user that's what I'm supposed to do obviously I'm beginning to anthropomorphize there it's hard to avoid but you know we are exploitable whether you look at behavioral economics or he risks and biases literature there are all of these sort of systematic predictable ways in which people are irrational and exploitable and as we're getting to AI systems that are starting to demonstrate theory of mind you know understanding what people were thinking and what is implied by rather subtle statements it doesn't seem like a crazy leap to say maybe they will start to develop you know kind of these dual models where there's like one model of the world that is you know the truth and about kind of making good predictions but maybe there's another one that sort of coexist or in a superposition with the first that is really about telling people what they want to hear because that's what maximizes the score right and if you went to your AI and you said like do I look good in this dress what's it going to tell you and that's kind of a toy example obviously but you know the again if it can start to learn to tell us what we want to hear in order to maximize its own feedback score at the potential expense of the truth you know how how do we know first of all we don't have right now the interpretability work to say if that's happening we couldn't you know reliably identify that I think that's probably one of the biggest things that people kind of hope things will move a little more slowly for I the people that I know in AI by the way even the ones that are the most doomer like they're not blind to the upside of the technology they would like to see everyone in the world have access to like a great AI doctor and you know all the the great things that could come from that but they sort of are like shit you know we're training with this highly imperfect feedback mechanism that is our ratings that creates this gap between the truth or you know purely accurate predictions of reality and what we want to hear which creates a sort of exploit opportunity for the AI and we have no way to detect if that's going on maybe we could just like slow down a little bit on the top end training runs that's not to say like don't you know make your stable diffusion variations and you know fine-tuned it on Eric's face so you can have Eric in every you know context that you never want to see him like by all means like do all those sorts of things but do we need to like 10x or 100x from GPT-4 training scale up to the next level like right now maybe we could take a minute to enjoy that technology to implement that technology to pull it apart and understand what's going on in that technology try to get to a theoretical understanding that is robust enough that we could say if these things were starting to deceive us we would be able to know that or we're at least pretty confident that we'd be able to know that based on our ability to inspect what's going on inside you know that's just kind of one branch right I mean I think there's a lot of analogies and a lot of stories back to your question like what would you worry about first to me the most compelling is this deception the notion that because we're unreliable raters subject to systematic exploitation the AIs may pick up on that I think there's a lot of other flavors and you can't even really like imagine what they all might be but to me it kind of adds up to you know enough that it's worth worrying about so here's a here's a question for you know like in a sense if you think of like nation states or corporations they are meta-organisms of humans human intelligence is they're in a sense they're super-intelligences right because they're kind of a mixture of expert model of different humans right a task comes to an organization it gets routed to the proper expert and they have kind of a collective meta-dural network of all their expertise that can effectively have far more parameters than anyone brain could fit physically right and organizations like corporations are we deal with them every day and we somehow we found a way to align them through competition keeping them in check through competition and a healthy marketplace right and ensuring there's not you know monopolies and so on and so we've already been studying how to align super-intelligences through our legal system and we already know how to interface with them you align interests by kind of exchanging utility either utility for money exchanging utility or exchanging capitals of some kind and I think that's how we're going to align with super-intelligences in the future but we already have kind of like this subversion and incursion problem with corporations right there are incentive is to grow at all costs and increase their metrics and they don't necessarily care about the well-being of their subjects but somehow we're okay with that and somehow we've done pretty well in our capitalist system through that and so how would you like you know differ problem of aligning a artificial superintelligence that runs on computers versus you know biological superintelligence that is let's say a corporation or a mega corp right to me they're kind of very similar sure there's IO bottlenecks in terms of corporation but they're still you know if you have a hundred two hundred thousand humans like some of these big tech corpse have there's a lot of intelligence there and that's a lot of power and somehow we're not crumbling under their weight yet and so how would you differ aligning AI or if Dan has other questions like us to cover before we finish I'm glad to answer those as well but uh yeah up up to you guys Eric and no no I would love I would love to hear Nathan's answer to that but the the thing I would just add on to there and we also have a mass scaled deception that exists in society American consumer culture there's almost every product in the market is sold to you to make you feel better about yourself in a reality that doesn't exist right you you see the model in the ad or at least when models used to be like people would they would aspire to be sorry Joe about body positivity but um you have a uh like someone in clothing you're like wow if I just buy that clothing I will also be that greater beautiful and so we we we we have mass scale deception across everything if I just drink this Coca-Cola I will feel better about myself and then they get a little bit of dopamine and then it turns out it's like that that's actually really bad for but but like that hasn't totally destroyed society I mean you could argue that there are a lot of bad effects from that but the reality is market-based capitalism and consumer culture has plenty of that and and we seem to survive and and if anything continue to make progress so I I very much build off the best point in the sense that like I think capitalism is a great example of an emergency system that we have kept in check I might interject here it might be good to try to like sharpen this up a little bit um so I kind of use a subtraction um of uh what I you know like you can think about a capabilities landscape sort of across civilization uh you know you have like peaks where uh there's like more capability uh and we can think about capability as like solving problems like capacity to harness energy things like this um right so I think what the classical fear of runaway fast takeoff you know recursive uh self-improving AI recursively self-improving AI um what that looks like is sort of like a you know a delta function like spike on the capabilities landscape and I think like what we're saying mostly uh amounts to uh right okay partially amounts to claiming that uh to have you know sort of this you know very very sharp spike uh has it like a massive associated energetic costs and and that gets cached out in right like frictions with the real world and things like this um we can talk about the details of that um but in general it's just like for every uh increment in the uh sort of like delta that be given uh like region of the capabilities landscape has over other points uh you're like paying a massive uh energetic cost and so when you say I'm worried that AI is going to so it's like steel man what they didn't say in a little bit and and like the general AI uh like risk uh the person who's concerned with AI risk uh more generally um it's that there's going to be uh you know a deformation in the capabilities landscape that is so unprecedented that um the infrastructure of society and civilization will not be able to withstand the exploits uh or the insults that come out of uh the corresponding system and I think this is like a reasonable thing to uh be concerned about and there's like a couple of things to go down uh the list about uh uh uh the first thing is that actually how to be sort these um yeah so I think the first thing is that um every when you when you actually start to write out these scenarios uh like we train an AI the AI has an inner uh so you know this notion of the outer uh optimizer the inner optimizer um uh we have some inner misalignment right there's a hidden goal inside of the the model that uh you know arose during the the normal gradient descent training and that goal is kind of point uh point the model in a direction uh in in sort of goal space that we that we would rather not uh be pointing at um uh and then not only is that inner optimizer there uh or sorry yeah yeah sort of like these these hidden goals there uh uh but those then express themselves during some inference they make contact with uh uh you know external reality in some way and then and so if you start to write out how these these things actually instantiate in the real world what you're doing is you're um you know you know multiply in these like probabilities together and when you do that like your estimates of any given scenario should go down um and i think that like in practice the once you start to get outside of of the system once you so so yeah we can talk about uh you know how uh exploits might leave the lab or whatever um but once you know a proposed exploit leaves the lab the question of whether or not it can be effective and actually breaking down civilization on prestructure societal infrastructure um uh at a scale that is truly existential uh risk um uh it's just it's yeah you have to like to really zoom into that kind of claim you have to like expand it out and i think it gets quite quite complex and uh it's just such a contingent claim to say we have a misaligned AI the world is over uh is like very big and forential leap uh right therefore the world is over another way to put it is like you know i think i think what AI existential risk folks are really scared of is kind of an intelligence monopoly like one intelligence with its own goals that are not necessarily aligned with the rest of the world just being this sort of singularity right just like one corporation if it had too much power in the world would optimize for the benefits the corporation not the rest of the world right uh and that's sort of asymmetry in the capabilities landscape that base is talking about would be kind of like one intelligence is like overcapitalized in terms of how intelligent it is relative to others and that would be a bad scenario because there's not a healthy competitive landscape to keep it in check and i think i think we kind of align uh on that uh uh uh with kind of the EA's and AI existential risk folks and that's why we think that variance and competition is good and that maybe could bring us to um open source versus source models right so there is let's let's assume there is some unknown risk probability right uh right certainly there's a number um and we don't know what it is um but we know these early facts the question is now what what do we do about it right so the main reflex that people have that society has had that culture has had is um hit the breaks um we cannot uh we have to have like we have to sort of sit back think about this problem a lot uh really study it and try to try to solve it um before we go any further but the issue is that we are uh uh uh all of this is not happening in a vacuum it's happening in a competitive landscape it's happening in an adversarial landscape uh at at many scales and um this is not just uh uh you know uh uh right like a uh an argument about China uh you serving the u.s in terms of capabilities uh or something like that i think there are um yeah that it's like multi-scale competition multi-scale adversaries and um it is not free to sit around and wait uh to work on this technology especially given where we are at today given that um so so uh maybe i differ from Beth a little bit on this i'm like pretty bullish uh on the progress that we will make in the next 10 years on this on this problem and i fully expect there to be you know uh like gvt3 uh Sydney Chadgvt like flavor or or or or or level or more events at increasing frequency over over that period um let's uh right so given that the reflexes to to to try to uh hit the breaks um this creates like some disadvantage in a competitive landscape um but it also does this thing right so when you create it actually it increases the likelihood that you get um these um regions of um more uh asymmetric advantage in the capabilities landscape right so when you when you in this happens when you create regulation as well right so when you regulate you reduce the the degree of dissipation of this innovation right so i call this alpha dissipation right you have an innovation and then gradually it disseminates through you know the economy and through civilization uh all the little workers in the economy uh work to you know uh capture the value that can be captured from that innovation and when you create regulation you reduce the the degree of alpha dissipation which reduces the degree to which you flatten the capabilities landscape and when you when you do that you you actually worsen the problem um and it actually is even worse than that because when you don't because yeah because basically when when it comes to AI right the the this sort of innovations that we're dissipating are actually going to help us uh to harden our civilization on societal infrastructure uh in the face of these challenges that will come from having uh you know massive massive amounts of you know intelligence increase uh on earth and so i strongly believe that like what stability is doing uh and and and and and efforts in this direction i strongly believe that they are a force for good they are not increasing uh risk substantially and where they are they are contributing locally to hardening infrastructure which we absolutely must do that's an imperative that we know we have to do um because otherwise we will have like uh you know people getting exploited left and right you know grandma's gonna get a call from uh you know Barack Obama and you know uh he's gonna like ask for her bank account information and she's going to be uh you know fully convinced uh to give it all away uh right like all from that to you know massive uh reward hacking through like end to end you know uh optimized grading optimized generative media like there's so many things that uh we we have to work on we we have Photoshop now right ignore the AI stuff just the fact that no one with their right mind trust in image they see on the internet did we have to upgrade civilizational infrastructure like it just kind of society i think i think i think that is like societal infrastructure right it's like uh you can think about this is maybe it's mimetic infrastructure right um i sort of think about that full stack uh maybe maybe my shorthand doesn't very good for right now uh but uh no i agree right it's like it's sort of like uh in that case what you need to do is uh i i would actually say in some cases no just depending maybe it's a function of age a little bit right um depending depending on the person maybe they they aren't quite sensitive to this uh possibility and uh yeah i think yeah but i think it's a good good point um yeah i mean it's just like you see that the the joe biden paying stuff on twitter and like no one of the very online people they don't even have to be an AI expert thing of that's fake but you show it to a you know i have an older person who might not get exposed that stuff and they might think it's real but also just young people not that long ago right i mean i do think there's been a real upgrade there in terms of i you know i think that the the mimetic infrastructure is a pretty good way to describe it like you could definitely go back and not that many years and people would not really have a reference for the even the possibility that something so compelling could be faked right so the when the prior on that is so low then what else you're gonna do but believe it right i mean the people used to say seeing is believing we're gonna have to retire that saying because it's not really anymore but you know that served us pretty well for an awfully long time and you know i do think there is some i think we're in some ways fortunate that that sort of technology kind of ramped up gradually in a way where we could see it coming in a way where you know we've had the last however many years where people have been kind of saying deep fakes are coming deep fakes are coming oh this is gonna be it's gonna be really crazy when the deep fakes come and now everybody's heard that or at least enough you know enough people have heard it to the point where you know there is some resistance and i hope that we can get you know to a similar place with a lot of the risks of associated with AI like some of them you know maybe outside of what our mimetic infrastructure can handle but some probably are within what our mimetic infrastructure can handle but it does take you know we only run on so many hurts right so we are at so many hurts so it does seem like we need some time to adjust and kind of figure out you know what we can still trust or we're gonna be like pretty quickly totally adrift one of the dangers we're seeing right now the narrative with desaleration is that if the western world adopts kind of desaleration mindset because they're trying to be virtuous and some other adversaries don't necessarily slow down their research then then you have what you know base has been talking about a sort of capability is gradient like a difference in capabilities and that sort of power asymmetry is pretty dangerous right so there's there's no one there's nothing keeping that sort of higher intelligence in check right just like you know in a sense in e-ach we're trying to have kind of antitrust for AI capabilities right like we're trying to maintain a nice healthy adversarial equilibrium and so this is kind of a pattern that has repeated itself you know we've had like ESG efforts in Europe and anti-nuclear sentiment and then that led to you know relying on certain foreign adversaries for energy and was pretty much net negative and you know adversaries of the western world ended up profiting off of what they call useful idiots that adopt ideologies that are useful to their ends right and so I think that AI safety to me walks and talks not all of AI safety of course but some of this like very alarmist faction kind of walks and talks like one of these you know psychological operations that are meant to kind of disarm demoralize and neutralize the west's capabilities and you know isn't it convenient that just when the west is ahead in terms of AI capabilities we decide to slow things down right and let other people catch up because you know we want to be cool at SF House parties and talk about our research I don't know I think like I think like there's been a lot of co-op thing of virtue signaling or what turned out to be nefarious purposes and so you know part of EAC is you know at least being mindful of that that it's kind of anything that like you know uses like it's kind of a mimetic or mind-virus like you know ideology that harnesses people's penchant for virtue signaling to push some agenda forward I just be very resistant like pushing that forward because you don't know who or what is pumping it forward I would say that you know it's not just for an adversary it's like mega-corps right now you know like for example if you're in the lead like opening i and microsoft now's the time for regulatory capture now that you're in the lead it's time for everyone else to slow down right let's capture our lead and so it's not just you know an isrid for an adversary it's like interests of of certain centralized entities and so it's just like I think maintaining a culture of skepticism towards such mimetic viral ideologies that are kind of top down secretly controlled is healthy and important and that's another aspect in which eAC is kind of resisting kind of this ea meme that's kind of taken over the ei world so want to add that um well you know one one way to do that is to maintain a healthy community of developers and not letting a i become super centralized under a very few players that then could be um you know subverted or co-opted because you know if you have one org that controls all Western a i right then then if you you know compromise that org that org you end up capturing the power of that a i and then and then you're in control whereas if you have a very high variance landscape of many millions of neural nets run by you know I don't know some sort of collective of of of of douse or or groups of people uh then then that's really hard to to co-opt and and and and kind of take over um and so it's kind of anti fragile it's more fault tolerant to corruption right that's why bottoms up is you're i mean capitalism tends to win against communism because in general communism is not fault tolerant to corruption of the control nodes of the of the government or mental organism and so we're seeing kind of a similar trend in a i now of centralization and that sets up you know a scenario where some of these orgs could become uh compromised and suddenly the Western a i's can be spewing information that uh benefits whoever and you know subverted them so it's something to be mindful of because it has um happened in the past with all sorts of organizations um but uh yeah i think overall maintaining kind of an open source culture maintaining compute that's decentralized knowledge that decentralized to some extent um uh is going to be super important to maintain like a healthy marketplace of a i's and maintain an uh adversarial equilibrium and our point is that safety is achieved through adversarial equilibrium and that's over centralization and regulatory capture is the enemy of that and ironically more unsafe so how how far do you guys push this acceleration concept so here's just like this may be a terrible thought experiment but let's say we take ourselves back to like 1944-45 and we're in the Manhattan project and it's like guys we figured it out we can make a bomb there's just one problem it might ignite the atmosphere and we need to take a little while and do some calculations on that before we do any like actual physical tests you guys are there at that time would you be like nobody's got time for that you know the Germans are going to beat us if we do anything to slow down so we don't have time you know let's just light the fucker and see what happens because it kind of sounds like what you're saying now with AI and i you know i'm always apologizing for my analogies because i think they mislead often more than they help but that was a moment where it was like breakthrough you know specifically to like energy you know all these new capabilities were sort of there but you know i'm pretty glad they did that calculation and that i would also kind of say like it seems like society has some pretty legitimate interest in regularly regulating like how much fizzle material like any you know particular group or individual can sort of aggregate doesn't feel like a big burden to me that there's somebody now i think we should build nuclear plants to be clear i'm very pro-nuclear uh i want to see a lot of plants come online like it's a it's a real shame that we haven't built any uh but you know that there should be a process that you should like go through some hoops to you know demonstrate your credibility you're like you know your safety standards that you know that everything is up to par like all that seems totally appropriate and i don't want to see us get bogged down but i also don't want to see us just declare you know totally open season for anybody to do whatever they want you know with fizzle material because like that's the fastest way to get some plants online either so you know what would you have done in the Manhattan project in in that time i think these are categorically different things um but maybe you have uh something you might say have done that but i think the comparison between nuclear and ai i think yeah breaks down in in in in a few ways um maybe just really briefly i'll say i i think being able to cause that much of a uh like right so so so nuclear weapons uh they give you the ability to affect the universe uh in a way that when you wield them is sort of constant right it's like uh you know given a certain size of weapon uh you you can just kind of like uh you know root access like vaporize things in in the universe right and so i yeah i think that's pretty different from ai uh and and and hopefully it's clear why i would be happy to go into details about about how i think like a nuclear weapons is a purely destructive technology right and and so like banning nuclear weapons is clear you know removing downside risk even though you know in principle one can argue that it's maintained a sort of peace from mutually assured destruction um but you know let's say you were to say let's ban uranium right as a whole right which is kind of where yeah safety is going with kind of regulating GPUs right well it could be used for bad but it could be used for good right like uh nuclear fission reactors and in fact uh huge damage has been done to our green energy capabilities from the uh nuclear fear mongering right like it's indelible damage to our culture and why people writ you know fear having nuclear reactors anywhere near you know city centers and so on um and so we just got to be mindful when there's like doom orism going about and mimetic you know cultural shifts because they can do like indelible damage and you know there's huge opportunity costs to uh leaving certain high potential technologies behind or decelerating their adoption because there yeah there's huge upside being left on the table and that's something that EA's an AI-access standard to risk folks really underestimate the potential upside of really adopting uh AI tech um you know and scaling it as fast as possible because ultimately you know how many humans even if you're uh you know your cost your loss function is like how many humans are living comfortably right the amount of humans we can support in our civilization depends on how much energy we have and how many people we can feed and and and uh you know how many folks we can support sustainably and as determined by kind of our card of shift scale to some extent and to climb the card of shift scale you need more intelligence because intelligence its purpose is to find and uh and use energy towards sustenance and growth of civilization and so you should be for more intelligence faster because it should allow for more humans uh to exist uh and uh more people to uh live great lives yeah so the the one thing I would add on the nuclear uh Manhattan project who who was it I forget uh it's a Oppenheimer obviously is running it but it's like uh Ed Teller or something's like hey like this might ignite the atmosphere so they went and studied it but it was uh when i was six month study like that was it like there was a bound like figure it out okay it's not gonna do it move ahead and I think that any of the alignment AI safetyism stuff it's a perpetual culture of control it's like let's lock everything down and then we get to make decisions and I think that so so that's where I think that analogy falls a little bit apart in the sense that that goal wasn't going to be stopped but they thought prudent to saying hey like let's just double check the math here or the you know the physics and so I I'm uh in addition to your your other point Bev of then you got to a world where you had fission it it was kind of growing as a as a you know kind of miracle energy in the US and then some 1960s hippies were kind of anti-vietnam war and anti nuclear weapons co-opted the war nuclear to the point where one three mile island happened it hit this political storm and we haven't built any new uh fission power plants in the in the country basically like that you can only expand existing ones and it's like the first one in in like 50 years is just getting finished in Georgia and then they've layered in all these additional regulations when if you just look at like Fukushima for example how many people died from the exposure to the the nuclear disaster that happened there compared to the tidal wave which killed I think it was like 20,000 people just from the ocean and so I think it's like when you start to allow that safety as in culture and in a centralized bureaucratic way the the the civilizational progress that we the potential for it is going to be vastly constrained and I and you can make arguments with the FDA it's like what is it the litamide and it's just like this like crazy aggressive uh pivot because people can get fearful if if the wrong people get into political control and then once it's there and none of the politicians want to touch that bureaucracy and it basically lives forever and so I think that like we have to be really really careful it's like the worst type of regulatory capture ignore Microsoft and OpenAI benefiting from that it's like you put it in a a state bureaucratic body that now some bureaucrat who has no incentive for progress like their whole point is to say no because they get judged on did any accidents happen or bad things happen during your your tenure and so I I'm like the the last thing we want is a big bureaucratic organization top-down managing all this stuff and again I'm broadly on the same side as Dan on almost every argument here I think you know certainly including the the terrible shame of not having built any nuclear plants but it does sound like there's some window of agreement potentially around doing things that would be prudent you know it seems like we all agreed that it was worth doing that study to try to get some confidence that we're not going to ignite the atmosphere with the first nuclear test so is there a version of that in the AI space that you guys can get behind and if so like I think you might actually find more common ground with the AI safety folks than than you think because again at least my perception of them is I don't think they want to do to AI what has been done to nuclear I think they just want to run kind of the equivalence of some of those you know atmosphere ignition studies and get to a point where we can like look inside the AI and be pretty confident that it's not actively deceiving us or that it's not you know hatching gray goo plans or whatever like you know it would be really nice to have that confidence before we like take pre-training up several more orders of magnitude worth of compute what do you guys think is a is a reasonable or prudent course that you know you would say okay well we don't have to go maximum speed all the time like yeah we could take a little detour for this or that that seems prudent is there anything there so this is this is a good segue I just I just think it's important to emphasize actually in general we're very supportive of what I would what I believe is like the majority of the the effort in the direction of of doing a lot work to align models and to make models reliable I think of this right if we forget about the whole history of the field I think you could imagine that we actually you can imagine a world where it's actually quite a boring topic it's called you know matrix multiplication reliability engineering and you know nobody really like looks at it and people just sort of toil away they figure out how to how to do this thing and it's fine I'm like I'm very supportive of of the serious people who are putting a lot of effort into trying to yeah do interpretability trying to create models which are more useful as tools to us in that they are reliable they reliably perform the actions that we request from them and I think it's like incredibly valuable I think it's there's there's you talking about the semantics here but you know uncivilable it is actually just AI capabilities right it's like we just want better models and so I think for us the biggest thing is is actually right so aside from the stuff about the capabilities landscape regulation and the opportunity cost of all these these sorts of you know very strong interventions I think first I think where this started actually is is about the culture right I have personally known people who were very negatively affected by the culture the sort of apocalyptic culture around the sort of less wrong mirror sphere of of this and I mean there you can read about stories about this it's like it's quite sad people don't and so my general belief is that people don't actually respond very well to the pressure of the apocalypse being put on their shoulders and what we want is instead of black-filling people making them feel like there's no hope actually explicitly saying there is no hope and I you know we have no way of solving this problem I think instead it's just like so much more functional to say there is hope we can solve this problem you know what we can quibble about the magnitude of the problems that we face but let's be optimistic about solving it I think there's like a real white pill to be had there and I think it one creates the conditions for people to have better experiences of this whole period of history and two it it increases the chances that we can like sort of work together to solve the problems that we do face and so I have a very optimistic vision of the work that serious people are doing like we could talk about the work that I am through up we could talk about Paul Cristiano's work there's like a whole conjecture right there's a whole constellation of people who are doing very serious work but yeah I believe people and culture work better when when we're optimistic and have hope instead of you know terminal despair yeah no argument for me on that I do think you know even if it just comes down to the very simple question of like motivating the safety researchers you know the some of the recent messaging has been pretty clearly suboptible the whole you know if you're out right telling people like you're doomed you may as well not even try you know go spend more time with your family that doesn't it doesn't seem to lead us toward any you know path towards solution so that seems pretty clear to me I would agree with I think everything that you said in that bit you know it seems like after kind of bouncing off of one another for a little while here you know I wouldn't put myself in in the really either camp of like Doomer or Accelerationist I kind of just describe myself all the time these days as ambivalent you know in the you know classic dictionary sense of like strong but seemingly conflicting feelings I get super excited about how awesome even just current AI can be let alone you know future AI I really do think it's you know endlessly fascinating fun to work with you know it makes my day-to-day workflow much smoother and more enjoyable so I really feel all that stuff and then I also do feel this kind of like but yeah I expand it does seem like we're playing with something that we don't really know how to control we don't really understand it's fundamental nature very well you know just because we know how to make it doesn't mean we know how it really works internally or you know what it might do and yeah those risks are like pretty low but I don't think we even need to think about like the far future you know to to worry that like if something were to go wrong and it were to kill everybody that we know today like that would be you know plenty bad enough that's a Carl Schumann point by the way is a you know long-time kind of thought leader in EA you always is very good about realing these like thought experiments in and just being like let's just talk about the current generation you know that that would be if we wipe ourselves out like forget about you know forgone possible futures so I do really feel both of those things do you guys just don't feel the other side of it like there's no there's no fear there's no like yikes this seems like it could get out of hand especially with you know order of magnitude of pre-training doubling every 18 months or whatever I guess like to to answer this question and circle back to your Manhattan project question which I really liked um I would say that you know the Manhattan project you know lighting up the sky on fire what they did they did physics calculations right and physics is kind of like we as humans we can argue until you know the cows come home but ultimately we must answer to physics right so what does physics tell us about where things are going right and what does physics tell us in terms of the limits of computation and optimization fundamentally you know there is a minimal thermodynamic cost of computing and in fact uh you know optimization of any kind doesn't matter if you do an optimization hierarchy of you know parameters hyper parameters architectures etc like it still costs you energy to get samples and and optimize at all scales um and so I think that you can you could prove mathematically that recursive runaway self optimization is impossible uh physically um you could also prove the gray goo most likely is is not possible either so if you bounded that then okay so those mega infinite downside scenarios are not there anymore okay so that does change the reward calculus right um I mean when it comes to like being extremely cautious before deploying things I mean we've we've changed our entire lives everything is run by an AI algorithm you know you know many of these uh recommendation systems that we use to find uh partners to find uh connection to find friends to find information are all gated by some form of AI algorithm and we've given them control over our lives and they're not aligned in any way shape or form with like what's best for us they're just aligned towards maximizing the profit the corporation that runs them and seems like we're we're beyond that debate now and now we're talking about unaligned AI APIs but we're completely fine with like you know as a tick-tock algorithm melting our brains and letting it just consume attention at infinite them right so so like um yeah I don't know I feel like we've already crossed that threshold of giving the keys away I'm not sure that's a great pitch for the next generation of technologies though like what I don't the leap there is not an obvious one really I think a lot of people are saying wait that's all bad everything you just said is bad why would I want more so here's the white pill so web 2.0 was kind of the attention economy so which which AI neural net gets run on average right which one is quote unquote alive by my definition of a live being like there's a GPU that a GPU instance that's getting paid for and run the selective pressure was towards AI that captures our attention more right and creates maximally distracting content or recommendations because it gets rewarded it gets selected for if it can get more attention from humans get more capital to run more GPUs to run more instance of itself right but now now that we're kind of creating AI that has utility like chat GPT the selective pressure is going to be towards AI neural net instances that have utility to us that we find useful I think that's a much better selective pressure than what we used to have right I don't know if it's web 3 or web 4 now like everything being LLM talking each other the internet of LLM APIs but go web 4 whatever web 4 to me I'm pretty optimistic about it I think like there might be pockets of web 2 like behavior where it tries to distract you but overall it's like the selective pressure on AI right now is going to be what has utility towards us humans human civilization our economy and so on and that is that is a selective pressure where it's going for it's not going towards AI's that can self replicate and take over on our adversarial at least at the moment right so that's why I'm not worried in the near term in the medium term who knows what happens in the long term like I said I think like anchoring on present day humans being preserved forever is not a realistic goal and so I'm open to whatever happens there so I think in the near term it's going to be all right as I said I would you know eacrobatist towards maintaining an adversarial equilibrium so that means open sourcing models keeping a healthy developer economy not centralizing the compute not centralizing the algorithms in the knowledge and kind of AI's will keep other AI's in check just like mega corpse keep other mega corpse in check right so you know Microsoft competing with Google on this AI aspect as is a you know an example of kind of the fight power with power and I think that maintaining this adversarial equilibrium is the way forward and that's only possible if there's not a monopoly on intelligence and so we're fighting uh sort of yeah anti monopolization of intelligence fight and embrace the variance and accelerate so that's the that's the eac way uh so um yeah but again I think we align on many things I think we align on many things um but we just differ in like the way forward to answer the question uh I don't have I don't have a lot of fear I think it's a bit of a choice right uh in in the scheme of things and I think the future can be scary I think it's very possible that we're going to have significant increases in volatility uh you know on this time scale of 10 or 20 years and I think it's important I think it's like very um there's a lot of a lot to be gained from from practicing equity and this kind of environment because uh to stand back and and and and understand things uh without fear uh like without a megalohyde jack being computed in your brain uh right is just like yeah I don't know I find it to be much much more pleasant and uh uh more likely to to lead to a good good decision making and uh yeah yeah I don't I don't I just don't I think this is missing in culture so um I think what we have here is something that counterbalances um uh uh this other trends let me try another one on you just to see what you think so let's say we're all knee-and-r-thals and we're sitting around our you know campfires in uh whatever 10,000 years ago whatever I'm not a great expert on all the dates here but like all of a sudden we've got the uh homo sapiens sapiens you know rolling in from africa and here we've been you know kind of hanging out and here they come and it's like well we could probably get along with them right I mean we could probably coexist and you know is it really you know well but what wait like Beth Bay is like they seem to have some capabilities that we don't have like they can coordinate across bigger groups they seem to be taking over the world well you know let's just practice some equanimity and uh you know hope for the best like we're now for repersent knee-and-r-thal so I guess in some sense you know they left their imprint on the future but I think if you ask the knee-and-r-thals they would be pretty unhappy with the outcome and again I kind of what obviously there's a very different scenarios you know it's not meant to be a tight analogy but I do kind of have this feeling that like we might be sort of acting like the knee-and-r-thals where we're kind of like we can see the writing on the wall we can see that there's some power here that doesn't seem like it's obvious necessarily how we're going to control it we could choose to be optimistic but might that not be like foolish and might we be like dead as a result of you know kind of choosing that optimistic path like it didn't work out very well for the knee-and-r-thals I would I would argue no I there's a couple of things first um okay we can talk about the gap between the analogy and the situation we face but I actually think that no right so I don't I think that there's there's a completion between optimism and sort of like you know starry-eyed like foolhardy that mess right and I think that that's to me it's quite different I think that yeah I guess yeah my personal opinion about this is people operate better when they are like not a megalahide jacked and so uh whatever the problems we face like this is sort of a state of mind that that we would probably want to support in culture and bolster in culture that's that's that's one opinion the thing about the gap between the analogy and the situation we face I think the first obvious thing right is that we that knee-and-r-thals did not build um sapiens right they didn't to craft this they didn't have this sort of like you know homo sapiens 0.01 0.02 right and so on and I think that um the other thing is is yeah again we talked about the sort of like digital to analog interface right um it's not the same sort of situation right so in the sort of competitive landscape of uh you know and-and-r-thals are some sapiens they're sort of very much acting in the same environment and um yeah it's just not the case uh for for us with AI I think there's there's quite a big gap there and and when you start to roll out the the sort of uh uh scenarios that that that correspond to uh I think you end up in the same kind of like where you have like inferential instability but you have this very long chain of contingent uh hypotheses and at the end you end up with like a very unlikely scenario for any single like highly specified scenario that seems true to me but kind of taking the you know integral if you will over that whole distribution of scenarios is where I think we wonder like do we get to 1% chance that we're all gonna die do we get to you know 10% like to me it still seems really hard to make the case that we're not looking at something that's at least cracking north of 1% which for me is enough to take it like extremely seriously I would do I would agree that there is a true number right like that's fine um I don't know what the number is I'm not gonna not gonna venture guess um because I think probably those estimates are just a little bit like pulled out of the air and not like it's sort of like uh maps to like you're feeling about it like but but not very well um I think some people come up with reasoning that they think supports the numbers I don't believe that that reason is usually realistic um but yeah it's not that there is no risk I think it's just um uh you know what kinds of actions can we take that will actually allow us to thrive and flourish uh given that we want to you know uh avert some some risks associated with those so it's not you know don't do safety research uh don't do interpretability research don't don't you know turn off the you know the RLHF right it's I think this this kind of thing is is probably wrong um but taking these sort of irreversible actions that are you know sort of uh a priori seemingly very likely to backfire um like I think it's been floated that we you know uh shut down TSMC like stop stop more's law right um and I think this is pretty it's yeah at the end of the day right like you are always in life facing this this this problem of I might die for you know given that you know just just because there's some baseline risk associated with being alive I think um it's not wrong that the uh risk that we face now is greater than with a risk that we face in the 80s or the 90s or something right with respect to some specific technologies right maybe you know for any given you know you could sum them up and imagine the sort of flow over time but um with respect to AI certainly the risk is like monetarily increased and I think it's like true um but I don't I don't know that we have much of a choice but to sort of deal with the with the with the problem of having the technology here I think building the the world the you know so so this is where he start to get into a vulnerable world hypothesis or territory I I should probably call you out and say I think a lot of you is would explicitly disavow the paper and the conclusions of the paper uh but I I'm yeah I'm a little bit skeptical of that uh disavow right so the the paper says uh right there you know we can imagine that we live uh you know in a in a world where um some you know as we invent technologies uh there's some chance some probability that we don't know that we will pull a technology from you know the the the bag of all possible technologies and that that technology will sort of uh give civilization the the capacity to to really destroy itself um and and of course like maybe maybe uh you know there's also some some uh uh scaling factor of like how easy it is and how much resources you need to do it um and I think the the idea that's like put forward is that uh you know technologies and I think specifically like AI uh that they are so dangerous that the only way to sort of guarantee that you will not end up destroying civilization is to institute a global panopticon with predictive policing and sort of uh just a massive you know uh sort of uh subat surveillance model this uh and I think there's several problems with this the first one is that um I I don't know uh um yeah so we can talk about the humidity in the angle of this um right because there's any such action would be on the basis of you know this very small probability right which we don't even know the error bars for uh which we don't understand very well at all we don't really understand the risk uh uh how it scales at these very tiny levels we know it's probably small um but we don't know exactly who how it changes uh uh around that uh that these various estimates that people have yeah I think there's a question about uh taking a very strong action on such small probably this is like a this is like a Pascal's mugging thing right you like hey all this this cost to try to overt this this very dangerous scenario and um yeah what do you do you may actually create the conditions for for uh you know human civilization to sort of flatline right it's uh uh uh or or worse right um it's very unclear what happens if you successfully create you know a a thousand year panopticon uh but I I don't think it's a stable solution to the problem I think you probably end up either with uh you know uh massive regression in in in civilization or uh over long period or you just end up back where you sort of started with things I don't I don't maybe uh maybe that he has something to say about this I would say like you know if a company has a 1% chance of going to zero or it has um I don't know 20% chance of uh exploding to a trillion dollars in value do you still invest yeah they do right and so like I think like people are trying to figure out do they want to invest in human civilization invest like their emotional capital and their time and energy into advancing civilization and like I think like giving up because there's 1% chance of doom of like that's really hard to predict how that would come about practically um uh versus like you know all the upside that we can have from either emerging or we're leveraging AI to improve our lives and improve civilization like yeah I don't know I I would invest like I guess we're just bullish on um civilization co-evolving with these new technologies that we introduce and you know becoming much better at uh understanding perceiving predicting controlling the world around us and that unlocks whole new set of technologies for us that allow us to grow in scope and scale and quality of life etc and so yeah ultimately like if there's one take-home message is that um I think that uh deceleration and doom rism has a massive unpriced um you know opportunity cost of upside that uh should not be underestimated and that um you know we're proponents for going faster because that's we're going to get faster to this uh upside and um a great future we want to live in. I might uh suggest leaving it there for today that was a pretty good summary kind of concluding point. Thank you so much for having us. Thanks so much. Thank you. It's a good time. link is in our description box.