Speaker 1Let me paint a picture of the sort of thing I am imagining could happen in a country like the United States. There are still elections. The nominations for candidates for those elections are decided by small groups of people within political parties. Those are increasingly automated. They're automated by biased AI systems that are run by big tech companies. So the choice of candidates becomes candidates who approve of what the big tech companies are doing. Meanwhile, the things that those candidates are offering to people are basically just versions of better state handouts. Nobody has a job. Everything comes down to what the state will offer you. The candidate who offers people the nicest material sops wins. People are happy to go along with this because they're getting rich. But what's happening behind the scenes is that a small group of oligarchs are deciding what the U.S.'s foreign policy should be, deciding what should be invested in and what shouldn't be, shaping the information environment, shaping... You might end up with really just a handful of people de facto controlling the political decisions that matter in the world.
Speaker 2Today I am speaking with Rose Hadjar. Rose is a researcher at Forethought, and she recently wrote an article for the 80,000 Hours website, which spells out a variety of different ways that AI could enable really extreme power concentration. So, in one version of events, the U.S. and eventually the world comes to be dominated by the executives of a single AI company, whose AI quickly became just way more advanced than others through an intelligence explosion. Now, I'm pretty worried about AI concentrating power in the hands of a small number of people. So, I'm going to spend a little bit of my time talking about that. So, I'm going to spend a little bit of my time talking about that. So, I'm going to spend a little bit of my time talking about that. So, I'm going to spend a little bit of my time talking about that. So, I'm going to spend a little bit of my time talking about that. So, I'm going to spend a little bit of my time talking about that. So, I'm going to spend a little bit of my time talking about that. So, I'm going to spend a little bit of my time talking about that. So, I'm going to spend a little bit of my time talking about that. So, I'm going to spend a little bit of my time talking about that. So, I'm going to spend a little bit of my time talking about that. So, I'm going to spend a little bit of my time talking about that. So, I'm going to spend a little bit of my time talking about that. So, I'm going to spend a little bit of my time talking about that. So, I'm going to spend a little bit of my time talking about that. So, I'm going to spend a little bit of my time talking about that. So, I'm going to spend a little bit of my time talking about that. So, I'm going to spend a little bit of my time talking about that. So, I'm going to spend a little bit of my time talking about that. So, I'm going to spend a little bit of my time talking about that. So, I'm going to spend a little bit of my time talking about that. So, I'm going to spend a little bit of my time talking about that.
Speaker 1So, I'm going to spend a little bit of my time talking about that.
Speaker 2So, I'm going to spend a little bit of my time talking about that. So, I'm going to spend a little bit of my time talking about that. So, I'm going to spend a little bit of my time talking about that. So, I'm going to spend a little bit of my time talking about that. So, I'm going to spend a little bit of my time talking about that. So, I'm going to spend a little bit of my time talking about that. So, I'm going to spend a little bit of my time talking about that. So, I'm going to spend a little bit of my time talking about that. So, I'm going to spend a little bit of my time talking about that. So, I'm going to spend a little bit of my time talking about that. So, I'm going to spend a little bit of my time talking about that. So, I'm going to spend a little bit of my time talking about that. So, I'm going to spend a little bit of my time talking about that. So, I'm going to spend a little bit of my time talking about that. So, I'm going to spend a little bit of my time talking about that. So, I'm going to spend a little bit of my time talking about that. So, I'm going to spend a little bit of my time talking about that. So, I'm going to spend a little bit of my time talking about that. So, I'm going to spend a little bit of my time talking about that. So, I'm going to spend a little bit of my time talking about that. So, I'm going to spend a little bit of my time talking about that. So, I'm going to spend a little bit of my time talking about that. So, I'm going to spend a little bit of my time talking about that. So, I'm going to spend a little bit of my time talking about that. So, I'm going to spend a little bit of my time talking about that. So, I'm going to spend a little bit of my time talking about that. So, I'm going to spend a little bit of my time talking about that. So, I'm going to spend a little bit of my time talking about that. So, I'm going to spend a little bit of my time talking about that. So, I'm going to spend a little bit of my time talking about that. So, I'm going to spend a little bit of my time talking about that. So, I'm going to spend a little bit of my time talking about that. So, I'm going to spend a little bit of my time talking about that. So, I'm going to spend a little bit of my time talking about that. So, I'm going to spend a little bit of my time talking about that. So, I'm going to spend a little bit of my time talking about that. So, I'm going to spend a little bit of my time talking about that.
Speaker 1So, I'm going to spend a little bit of my time talking about that. So, I'm going to spend a little bit of my time talking about that. So, I'm going to spend a little bit of my time talking about that. So, I'm going to spend a little bit of my time talking about that. So, I'm going to spend a little bit of my time talking about that. So, I'm going to spend a little bit of my time talking about that. So, I'm going to spend a little bit of my time talking about that. So, I'm going to spend a little bit of my time talking about that. So, I'm going to spend a little bit of my time talking about that. So, I'm going to spend a little bit of my time talking about that. So, I'm going to spend a little bit of my time talking about that. So, I'm going to spend a little bit of my time talking about that. So, I'm going to spend a little bit of my time talking about that. So, I'm going to spend a little bit of my time talking about that. So, I'm going to spend a little bit of my time talking about that. So, I'm going to spend a little bit of my time talking about that. So, I'm going to spend a little bit of my time talking about that. So, I'm going to spend a little bit of my time talking about that. So, I'm going to spend a little bit of my time talking about that. So, I'm going to spend a little bit of my time talking about that. So, I'm going to spend a little bit of my time talking about that. So, I'm going to spend a little bit of my time talking about that. So, I'm going to spend a little bit of my time talking about that. So, I'm going to spend a little bit of my time talking about that. So, I'm going to spend a little bit of my time talking about that. So, I'm going to spend a little bit of my time talking about that. So, I'm going to spend a little bit of my time talking about that. So, I'm going to spend a little bit of my time talking about that. So, I'm going to spend a little bit of my time talking about that. So, I'm going to spend a little bit of my time talking about that. So, I'm going to spend a little bit of my time talking about that. So, I'm going to spend a little bit of my time talking about that. So, I'm going to spend a little bit of my time talking about that. So, I'm going to spend a little bit of my time talking about that. So, I'm going to spend a little bit of my time talking about that. So, I'm going to spend a little bit of my time talking about that. So, I'm going to spend a little bit of my time talking about that. So, I'm going to spend a little bit of my time talking about that. So, I'm going to spend a little bit of my time talking about that. So, I'm going to spend a little bit of my time talking about that. So, I'm going to spend a little bit of my time talking about that. So, I'm going to spend a little bit of my time talking about that. So, I'm going to spend a little bit of my time talking about that. So, I'm going to spend a little bit of my time talking about that. So, I'm going to spend a little bit of my time talking about that. So, I'm going to spend a little bit of my time talking about that. So, I'm going to spend a little bit of my time talking about that. So, I'm going to spend a little bit of my time talking about that. So, I'm going to spend a little bit of my time talking about that. So, I'm going to spend a little bit of my time talking about that. So, I'm going to spend a little bit of my time talking about that. So, I'm going to spend a little bit of my time talking about that. So, I'm going to spend a little bit of my time talking about that. So, I'm going to spend a little bit of my time talking about that. So, I'm going to spend a little bit of my time talking about that. So, I'm going to spend a little bit of my time talking about that. So, I'm going to spend a little bit of my time talking about that. So, I'm going to spend a little bit of my time talking about that. So, I'm going to spend a little bit of my time talking about that. So, I'm going to spend a little bit of my time talking about that. So, I'm going to spend a little bit of my time talking about that. So, I'm going to spend a little bit of my time talking about that. So, I'm going to spend a little bit of my time talking about that. So, I'm going to spend a little bit of my time talking about that. So, I'm going to spend a little bit of my time talking about that. So, I'm going to spend a little bit of my time talking about that.
Speaker 2So, I'm going to spend a little bit of my time talking about that. So, I'm going to spend a little bit of my time talking about that. So, I'm going to spend a little bit of my time talking about that. So, I'm going to spend a little bit of my time talking about that. So, I'm going to spend a little bit of my time talking about that. So, I'm going to spend a little bit of my time talking about that. So, I'm going to spend a little bit of my time talking about that. So, I'm going to spend a little bit of my time talking about that. So, I'm going to spend a little bit of my time talking about that. So, I'm going to spend a little bit of my time talking about that. So, I'm going to spend a little bit of my time talking about that. So, I'm going to spend a little bit of my time talking about that. So, I'm going to spend a little bit of my time talking about that. So, I'm going to spend a little bit of my time talking about that. So, I'm going to spend a little bit of my time talking about that. So, I'm going to spend a little bit of my time talking about that. So, I'm going to spend a little bit of my time talking about that. So, I'm going to spend a little bit of my time talking about that. So, I'm going to spend a little bit of my time talking about that. So, I'm going to spend a little bit of my time talking about that. So, I'm going to spend a little bit of my time talking about that. So, I'm going to spend a little bit of my time talking about that. So, I'm going to spend a little bit of my time talking about that. So, I'm going to spend a little bit of my time talking about that. So, I'm going to spend a little bit of my time talking about that. So, I'm going to spend a little bit of my time talking about that. So, I'm going to spend a little bit of my time talking about that. So, I'm going to spend a little bit of my time talking about that.
Speaker 1So, I'm going to spend a little bit of my time talking about that. So, I'm going to spend a little bit of my time talking about that. So, I'm going to spend a little bit of my time talking about that. So, I'm going to spend a little bit of my time talking about that. So, I'm going to spend a little bit of my time talking about that. So, I'm going to spend a little bit of my time talking about that. So, I'm going to spend a little bit of my time talking about that. So, I'm going to spend a little bit of my time talking about that. So, I'm going to spend a little bit of my time talking about that. So, I'm going to spend a little bit of my time talking about that. So, I'm going to spend a little bit of my time talking about that. So, I'm going to spend a little bit of my time talking about that. So, I'm going to spend a little bit of my time talking about that. So, I'm going to spend a little bit of my time talking about that. So, I'm going to spend a little bit of my time talking about that. So, I'm going to spend a little bit of my time talking about that. So, I'm going to spend a little bit of my time talking about that. So, I'm going to spend a little bit of my time talking about that. So, I'm going to spend a little bit of my time talking about that. So, I'm going to spend a little bit of my time talking about that. So, I'm going to spend a little bit of my time talking about that. So, I'm going to spend a little bit of my time talking about that. So, I'm going to spend a little bit of my time talking about that. So, I'm going to spend a little bit of my time talking about that. So, I'm going to spend a little bit of my time talking about that. So, I'm going to spend a little bit of my time talking about that. So, I'm going to spend a little bit of my time talking about that. So, I'm going to spend a little bit of my time talking about that. So, I'm going to spend a little bit of my time talking about that. So, I'm going to spend a little bit of my time talking about that. So, I'm going to spend a little bit of my time talking about that. So, I'm going to spend a little bit of my time talking about that. So, I'm going to spend a little bit of my time talking about that. So, I'm going to spend a little bit of my time talking about that. So, I'm going to spend a little bit of my time talking about that. So, I'm going to spend a little bit of my time talking about that. So, I'm going to spend a little bit of my time talking about that. So, I'm going to spend a little bit of my time talking about that. So, I'm going to spend a little bit of my time talking about that. So, I'm going to spend a little bit of my time talking about that. So, I'm going to spend a little bit of my time talking about that. So, I'm going to spend a little bit of my time talking about that. So, I'm going to spend a little bit of my time talking about that. So, I'm going to spend a little bit of my time talking about that. So, I'm going to spend a little bit of my time talking about that. So, I'm going to spend a little bit of my time talking about that. So, I'm going to spend a little bit of my time talking about that. So, I'm going to spend a little bit of my time talking about that. So, I'm going to spend a little bit of my time talking about that. So, I'm going to spend a little bit of my time talking about that. So, I'm going to spend a little bit of my time talking about that. So, I'm going to spend a little bit of my time talking about that. So, I'm going to spend a little bit of my time talking about that. So, I'm going to spend a little bit of my time talking about that. So, I'm going to spend a little bit of my time talking about that. So, I'm going to spend a little bit of my time talking about that. So, I'm going to spend a little bit of my time talking about that. So, I'm going to spend a little bit of my time talking about that. So, I'm going to spend a little bit of my time talking about that. So, I'm going to spend a little bit of my time talking about that. So, I'm going to spend a little bit of my time talking about that. So, I'm going to spend a little bit of my time talking about that. So, I'm going to spend a little bit of my time talking about that. So, I'm going to spend a little bit of my time talking about that. So, I'm going to spend a little bit of my time talking about that. So, I'm going to spend a little bit of my time talking about that. So, I'm going to spend a little bit of my time talking about that. So, I'm going to spend a little bit of my time talking about that. So, I'm going to spend a little bit of my time talking about that. So, I'm going to spend a little bit of my time talking about that. So, I'm going to spend a little bit of my time talking about that. So, I'm going to spend a little bit of my time talking about that. So, I'm going to spend a little bit of my time talking about that. So, I'm going to spend a little bit of my time talking about that. So, I'm going to spend a little bit of my time talking about that. So, I'm going to spend a little bit of my time talking about that. So, I'm going to spend a little bit of my time talking about that. So, I'm going to spend a little bit of my time talking about that. So, I'm going to spend a little bit of my time talking about that. So, I'm going to spend a little bit of my time talking about that. So, I'm going to spend a little bit of my time talking about that. So, I'm going to spend a little bit of my time talking about that. So, I'm going to spend a little bit of my time talking about that. So, I'm going to spend a little bit of my time talking about that. So, I'm going to spend a little bit of my time talking about that. So, I'm going to spend a little bit of my time talking about that. So, I'm going to spend a little bit of my time talking about that. So, I'm going to spend a little bit of my time talking about that. organizations becoming very concentrated as well. And so there's kind of a spectrum here in terms of exactly how far does this go in the kind of more power grabby scenarios where you do get internal power grabs within organizations and maybe where there's a bigger gap in capabilities between the leading AI powered organization. In those scenarios, you might end up with really just a handful of people de facto controlling the political decisions that matter in the world. In less extreme scenarios, you might end up with a kind of fairly broad set of organizations becoming the new powers that be. And things like democratic elections no longer matter very much in determining what's going to happen next.
Speaker 2I guess I want to kind of ask here, like, once you go through all of this and yet you have some smallish number of people who are really. Kind of calling the shots about how major strategic decisions go, whether that's because they have like actively like orchestrated a power grab and like seized the decision making seats, or it's because they have just like gained a lot of power such that they have a lot of sway over the decisions that are being made. What does that world now look like?
Speaker 1Yeah, so one thing that I'm imagining is just. Most people have been basically politically disempowered in this world. Maybe there are still elections. Maybe there are still the governments that we know today, but I'm not expecting that these will matter very much. Like we see in countries today like Russia, there are elections, but this doesn't this doesn't equate to democratic government. And so I'm pretty worried about worlds where the general public doesn't have much of a political voice. I think there are a few different reasons to worry about that. One is tyranny. Just that this kind of small group of people with very correlated backgrounds and views will get to decide things and they will make decisions that aren't in the interests of most people. And this this could lead to oppression. This could lead to abuses of power. I think even if it doesn't go as far as that. And for example, you know, there's some kind of enlightened welfare state and people are living on handouts and they're materially well off. I still think that that future would be a lot less good than it could have been had there been proper representation. And good deliberative processes for deciding how to do things. I think there are also more extreme worries here. So I know that some people are worried about, you know, will in a world where all human labor is being automated, will people starve? You know, they don't have they don't have the ability to make wages anymore. And we see today that rich countries are fairly happy to let people in poor countries starve to death, even though it would be financially fairly trivial to prevent that. And so if we end up with power consequences. If we're concentrated in the hands of a small number of companies or a small number of countries, will we see mass starvation as AIs replace people's livelihoods? I think this is a serious, serious thing to worry about.
Speaker 2Yeah. Yeah. I mean, that all sounds pretty bleak and yeah, quite, quite extreme as well. And I think this is where I begin to get a bit confused, because I think that some of the dynamics that you've been describing. So like, yeah, in today's world, big companies do like wield a fair amount of political influence and that political influence and wealth can like feed into like an increased ability to manipulate public opinion and so forth. We have issues with misinformation and a sort of polluted information environment. Those things already exist. We've also sort of seen. Yeah, like previous waves of automation, concentrating wealth and concentrating power into the hands of different people or smaller groups of people in the past. Like I think in the industrial revolution, as we industrialize the wealth sort of maybe fell more to the factory owners and so forth, who are most kind of important to the economy at the time. So all of these things feel like kind of familiar, like not that far from things that we've already seen. And, you know, we also have current AIs that seem pretty capable and they haven't kind of shaken things up that much. Like none of these dimensions feel like they're like obviously much worse than they were before. So I think with all of that, I do wonder like, yeah, like why we might get effects with with future AIs that are just so much more extreme than we've seen.
Speaker 1Yeah, so first of all, I completely agree with you, this would be way more extreme than anything we've seen in the past. And so it's definitely a prediction of something, something very different happening. I also think if I knew that current systems would not advance at all, and we would stay at this level of AI capabilities forever, I wouldn't be very concerned about extreme power concentration. I'd be concerned about moderate power concentration. I think we can already see today. Oh, there are some companies that have a lot of influence. But I think it would be within the same. Kind of range as we've seen historically, rather than something unprecedented on why I think AI is going to be different. I think a core thing here is I'm expecting AI will eventually be able to automate most or all human work in a way that no other technology has been able to do. And a key thing here is it's going to massively reduce the leverage of the average human person. Right now, humans all have labor that they can sell. This can't be taken away from them. I mean, they can be locked up. But it's it's a non-seizable, valuable good that humans have. And I'm imagining a future where, in fact, anything economically valuable can be done via AI systems without any humans involved. And I think this is this is the kind of main generator for the thought, oh, maybe power will become very concentrated because maybe most people won't have anything valuable to hold out as leverage to get a bit more specific about how that might work. I think one important feature of AI is that very small groups of humans might be able to use AI systems to do vast amounts of work without cooperation from any other humans. So right now, if you want to launch a coup or if you want to run Amazon or if you want to invade a country, you need to have lots and lots of other humans who are willing to cooperate with you on that. And that puts a limit on how crazy your project can be. And I'm expecting this. It's a limit to go away such that a small group of individuals could do things that most people would regard as really important, but they can do them very successfully using AI systems that can be programmed to follow their particular interests. I think another way of looking at this kind of AI automating all human labor thing is it will make it much, much easier to turn money into labor than it is today. And so then we might get a much stronger feedback loop between wealth. And power than we see today. Today, we already see it to some extent. But if you think about human labor and, you know, imagine that you're a billionaire and you want to get more people to do things. It's hard to get humans to do arbitrary work. People have particular skills or they don't want to do certain things. You have to fit them in in the right places. And that's kind of complicated. It's hard to get more than a certain number of people to work efficiently together. You know, organizations tend to max out a certain size. Beyond that, it's no longer efficient to add more people. Also, like. At the very limit, if you think about the whole economy, it's quite slow to make more humans. And so, like, there are these there are these natural limits on what you can do if you're trying to turn money into human labor. None of these limits apply to AI. You know, AI will do any kind of work. Some systems might be very general and have skills in all sorts of things. Or you might end up with specialized systems. But I expect their coordination will be much easier than human coordination. That means that you won't have such limits on the number of workers that you have. You'll be able to get much. Bigger AI workforces collaborating effectively together. And the main the main thing here really is that you can just at the flick of a button make more copies just just with more money, just with more compute. And sure, you might run into constraints also. And then then that will be slower. But I'm still expecting there's going to be a much, much faster feedback loop between money and the ability to get things done than there is in the current world.
Speaker 2Right. Yeah. So the components of this are like. So AI can do more or less any kind of economically valuable tasks we would like it to do. We could deploy just like enormous numbers of them, like unprecedented numbers of them. Each one of them can do like arbitrary things. They don't have to have specific skill sets. It doesn't take a long time to kind of raise them from birth or something like that, like it does with humans. And you can get them to do things that humans wouldn't even be willing to do in some cases. And OK, so all of this together and various other things mean that. Yeah, we can get a lot more stuff done with fewer people. But the other important element of this is that like the people who get that extra power to do stuff done is like a small particular elite group of people who disproportionately get the increased power to get stuff done. Does that sound right?
Speaker 1Yeah, exactly. And the thing about the small group is, yeah, if everyone was able to do way more than one thing. more because of AI systems then we might not be worried about concentration of power we just think okay cool like human capabilities are going to increase because of AI yeah but in fact uh particularly because I'm expecting that rich people will have much better access to AI they'll have much more compute they'll be able to more efficiently turn their money into more AI labor than people with less money um I'm expecting that small groups will be much much more empowered by AI than others I mean you get this big power differential that you don't have today another thing is that uh I'm also expecting at least somewhat large capability gaps between different AI systems and different AI developers um potentially very big ones if there's a large intelligence explosion where you get this really fast feedback loop with AIs building better AIs building better AIs even if there's not a massive intelligence explosion of that kind I'm still expecting there to be significant capability gaps between different AI systems and different AI gaps and significant lags in deployment such that I'm expecting kind of a relatively small set of actors to have access to these much bigger workforces than everyone else
Speaker 2and that's part of where the fear comes from right yeah okay so we've talked um in like a small amount of detail of like a myriad of different factors that can um contribute to yeah really extreme power concentration um yeah one of them is um like active power grabs one of them is um the distribution of wealth and other economic factors and another one is our like ability to understand what's going on um I want to talk about all three of those things in a lot more detail um so let's briefly chat through how AI might be able to do that and how it might be able to do that and how it might increase the risk of a successful power grab um and yeah if listeners want more on this particular topic um I really encourage them to listen to the episode with Tom Davidson um from like early 2025 where this was described in a lot more detail Rose and I will cover like some of the basics here yeah Rose okay can you start by giving us some examples of what a power grab using AI might look like?
Speaker 1Sure. So one example of what a power grab might look like is maybe the CEO of an AI company is able to instruct the AI systems to insert secret loyalties into future generations of systems secret loyalties would mean that without others being able to detect it those systems are furthering the interests of that CEO in particular and so then if this happens successfully then one generation would make the next generation secretly loyal and that would make the next generation secretly loyal and so on this gets worrying in terms of power grabs if that company ends up uh providing systems to governments basically and so particularly if the company has a big lead in capabilities it's likely to become you know one of the only providers of government systems and then you might end up in a situation where important military systems and important government systems are secretly loyal to one individual outside of the company. So if the company has a big lead in the government then at some later point that individual would be able to trigger those secret loyalties seize control of those systems order them to kind of take over the government you could also imagine that it doesn't actually end up with something so overt and they're just able to instruct the systems to ignore certain orders and to take certain other orders and they don't ever need to kind of you know seize control of the radio stations but you know you can imagine a range of scenarios here just quickly to give another example. to give a sense of the kind of spread of things here um another power grab scenario you might worry about is that a head of state um is deploying ai systems in government and in the military and the head of state is able to push for those systems to be trained to be explicitly loyal to the head of state personally rather than being trained to follow the constitution or the law or some more accountable more democratic um thing at that point if at some later date the head of state wanted to for example cancel the next election declare themselves permanent dictator they would be able to um because they you know a large fraction of government employees have been programmed to be loyal to that individual in particular and they're happy to break the law on that person's behalf um so that's another another example scenario yeah okay so yeah i guess like an immediate reaction to this might be well how can we yeah how come we just don't like manage to stop this from
Speaker 2happening how come these like institutions that are designed to prevent this kind of thing from happening um like how come they fail um i guess with the first scenario um where it's um secret loyalties um these secret loyalties are just simply not detected in time um they are well hidden um or they're just like fundamentally hard to notice and that's why it sort of catches us unaware um can you spell out a little bit more in the second scenario um where it's a head of state um yeah why it is that that they don't get stopped
Speaker 1yeah so i think there are a few different things here that could uh that increase the risk that this kind of scenario comes to pass one is perhaps many of the government employees supreme court employees so on who might oppose such power seeking moves are being automated away by ai systems um such that they're not able to oppose any more of the power seeking moves or maybe there isn't a kind of good system for ai systems to whistleblow and so you kind of reduce the overall amount of transparency on this sort of thing um that's one concern another concern is perhaps um either driven by ai or or not um there's already backsliding going on and so i think a lot a lot here depends on the strength of the institutions in the country that you're thinking about and whether whether those are already being compromised or not okay
Speaker 2yeah i think something that's kind of yeah ringing a bit of a bell for me here is that it seems like in stories like these where the military is probably involved um as like a pretty key part of it either because it's like a military coup or because you like need the sort of force of the military to kind of enforce your uh rule of law um yeah what's being imagined here is that um these systems are being integrated into the military like pretty fast like fast enough that the people who'd want to stop this kind of don't really notice what's going on or aren't sort of able to do like more thorough checks that like prevent these things from being able to happen but yeah that strikes me as kind of going against the trend of like how tech adoption usually works in the military like my understanding is that like like these things can be done in a way that's like incredibly slow like there's i guess in the military often like um existing suppliers who are not going to want to be replaced by like newfangled technologies because they benefit a lot from being able to supply to the military you have just like generally slow processes of procurement that can be just like sort of boringly slow in some cases um and then you just like also just get like a quite the sort of psychological resistance to change and technologies that people don't really understand um yeah i mean with all of this are you imagining that something like very different happens in the process of ai militarization um and like why should we expect this to be like much faster in
Speaker 1that case yeah great question so i think you're definitely right that procurement can be very slow i think we can also see examples where it's been pretty fast the manhattan project was like what four or five years or something um and part of what was going on there was military necessity um and feeling like there was some kind of existential competition going on uh to race to get the bomb before the nazis did whether or not there's a cold or hot war i'm expecting that ai deployment in militaries will be driven by competitive pressures um this might not mean that it's very fast you know if if china deploys ai slowly in the military maybe the us will also deploy slowly but in a way the the kind of cultural forces that you are pointing towards you know fear of new tangled technologies bureaucratic slowness in a way that makes me more worried rather than more reassured actually because it's like there isn't um to the extent that that's true there isn't a kind of robust culture already of appropriate vetting um and a thing that i'm worried about is it's slow slow slow slow slow and then suddenly it's really fast yeah uh whereas like if we were just at medium pace the whole time i would feel more comfortable yeah um yeah yeah i think one other point is it's not um i think the key thing here is not directly how fast military deployment goes although that is a correlated factor i think it's more like whether whether the deployment is done sufficiently carefully and it gets obviously the faster it is the harder it is to do it carefully but there are other things that make it hard to do carefully as well one of them is one ai provider has a massive capabilities advantage and it's not just a matter of whether or not you're going to be able to do it if there's if there's really a single provider that has a big edge i think it will be difficult for the military to vet that appropriately yeah because Those systems will be the only ones powerful enough to vet those systems, but that's obviously biased. And so there could be kind of difficult situations like that where even with lots of time and the best will in the world, the military might struggle to do procurement adequately, safely. Maybe another point is for these kind of scenarios to go through, you don't need to have the whole military automated. You don't need to get rid of all human soldiers and stuff. You could imagine, like we see historically that coups sometimes succeed with really small numbers of soldiers. And a key thing that's going on here is that they're managing to kind of capture the information environment such that everyone believes they've won. And then, you know, in some cases, literally a battalion takes over a country and the rest of the army is like, oh, well, I guess they've won. And so you could imagine a similar situation where a small fraction of the military is automated, AI-driven drones or something like that. But there's also sufficient control of the information environment that. An actor can create a sense of inevitability about this and a small group can kind of successfully coup the rest of the rest of the government.
Speaker 2Yeah. OK, so this this story is a place where the category of factors that contribute to power concentration that you labeled like epistemic factors like are really doing quite a bit of the work. Yeah, definitely.
Speaker 1And I guess an important point here is me and Tom do disagree. On how central we think power grabs are, but it's not like Tom is ignoring epistemics or economic factors like he thinks those things are important to and they're part of they're part of the initial ideas he had. And I just think that they're more important on the margin than Tom expects.
Speaker 2Yeah. So, I mean, this is super interesting and there would be a ton more to say about this. Since we already talked to Tom Davidson and have a podcast episode with him that listeners can look at for more detail. I think we should now move on to think in detail. About some of the other ways that I could mean power gets extremely concentrated. You think that one group or some collection of small groups of people could end up in charge of like really important decisions for the world through means that like aren't military based, aren't violent, don't involve anything that we would call a power grab. But sort of they get. They get there because they have much more wealth than other people or they sort of play a much bigger role in the economy. Yeah. What are the different ways that this could look?
Speaker 1Yeah, sure. So the the high level worry here is something I've already mentioned, which is just automation is going to disempower most people economically because their labor is going to become valueless. And that's going to generate enormous. And so this is the kind of basic generator of all of the scenarios here in terms of some different ways this could look. I think there are a few different things that could go on here. One is something that's been written about by others as the intelligence curse. This is the idea that government incentives to represent their people will get gradually eroded as a bigger and bigger proportion of government revenues comes not from citizen taxation. But from the sector directly, a bit analogous to the resource curse that we see today, where some states that get most of their revenue from oil or from diamonds are very exploitative in relation to their populace and they're dependent not on the economic productivity of their citizens are dependent on a very narrow resource based sector. So that's that's one scenario to worry about. A different one is something that my colleague Thomas talked about as outgrowing the world. So here. The idea is that maybe one company or one country is growing so fast economically that over time they come to dominate the global economy. You know, they're 99 percent of all production is being done by that country or that company. This is something that that does depend on quite a big intelligence explosion, I think, and various other factors as well. So that's another scenario then, like just to give one more gesturing towards how this could happen. You could imagine if if there are first mover advantages in space where the first people to seize control over space resources are able to hold on to them indefinitely, then you might see a version of economic dominance which isn't Earth based, but instead the first company or country to begin to colonize space therefore de facto gains power over most of the solar system or galaxy or universe. So like these are. There are varying degrees of speculativeness, but there are there, I think, a bunch of different ways that we could see economic factors leading to very extreme forms of power concentration.
Speaker 2OK, I guess my immediate reaction to all of this is, like, why, why wouldn't there be some kind of government effort to redistribute wealth given that, I guess, in worlds where you say there's like a fast intelligence explosion, i.e. like. AI has quickly become like way, way more capable and way, way more productive. One might imagine that, like, we suddenly have like an excess of like goods and wealth being generated. Why shouldn't we expect that there will be some kind of initiative to share all of the surplus or something?
Speaker 1Yeah, good point. So I do overall expect that there will be efforts towards redistribution. I'm not expecting I'm not expecting like the extreme version of this will happen and nobody will be. Like, oh, we should change the tax code. And so, yeah, I think we should definitely expect some efforts or redistribution. I think there are a few reasons why this doesn't make me think, oh, this problem is is solved by default. One is I'm not expecting the best forms of distribution to happen by default. So I mentioned already, if we look at international redistribution of wealth today, it's extremely shocking that the percentage of of wealth that's going from rich countries to poor countries is minuscule. And so we might worry that we. End up with kind of good distribution within wealthy states and then poor countries just starve to death or something like that, which is, I think, a future that many people would think is awful. I think another reason to worry about redistribution is material wealth is not the only form of power that I care about people having. And if people are living on handouts at the kind of whim of the state at the. You know, following just the good intentions of that state, that feels a lot less robust to me than if people have an actual source of leverage, which is their own labor. You know, ultimately, they can strike, they can withhold that there can be general strikes. They have some power in relation to the government. But if people are living on handouts and the government doesn't depend at all on their taxation, I start to worry more about abuses of power, about political power being eroded and about tyranny. And so even if there is redistribution. I'm still kind of concerned about what the political ramifications of this would be. I also think there's some chance that redistribution doesn't happen at all. I think this could happen in worlds where, you know, governments become extremely captured very quickly by very, very capable AI companies who are badly intentioned. But that's not where most of my expectation is. I am expecting some redistribution by default.
Speaker 2Right. OK. Yeah. So I think what I'm getting here is that, yeah, I mean, I think it does. I think it does seem like these are all pretty bad outcomes. Like, yeah, if you like lack of material wealth, starving to death is like an extremely bad outcome. And yeah, like even even without that, but just sort of losing your political leverage to some degree. That also seems like it's certainly not the ideal world if large groups of the population are not sort of able to meaningfully influence the future. But yeah, I guess on that latter point, like, I think I think maybe I have some confusion about like how wealth and economic relevance sort of feeds in to your ability to influence the future. So like in the current way that things are set up, like even the absolute richest people in the world can't just like decide like major political decisions. Or like change a policy or like just be like in control of the world in like a literal sense. So, yeah, I mean, I and I do agree that even if we're not in a world where that happens, there are still extremely bad things that could be happening. But yeah, like for the absolute worst scenarios you're imagining, like does does democracy just need to collapse basically for like these wealthy people to actually like have taken control of the world? So I think in.
Speaker 1In spirit, yes, although not necessarily in in form. So I think it's possible that, you know, governments continue under their current names and elections continue and so on. What I'm worried about is that these things will no longer matter to the outcomes of decisions that get made. I guess a relevant kind of historical example here to talk a bit about is the is the resource curse idea that I mentioned where states. that currently depend significantly for their revenue on just one natural resource often are not very democratic so you have examples like the gulf states where representation is quite minimal and they're like large underclasses of migrant workers who have very limited rights um or like equatorial guinea where i think almost all of the state's revenue is oil and it's and it's run as like the personal wealth of the ruling family um the thing that's interesting here is that you also have not states which are rich in natural resources like norway that don't have a problem with democracy um and so a key question here is like okay should we worry about current democracies once ai automates human labor or are they going to be safe from from this resource curse like they have been from from historical resource curses one one argument for why norway doesn't suffer from the resource curse is oh well it already had strong democratic institutions and so these were robust to getting this big influx of of oil money um this is possible and this is a reason for optimism about democratic countries that maybe their institutions will be robust enough to like manage the change um one other thing that is slightly worrying to me though is that norway's percentage of revenue from oil wealth is quite a lot lower than the countries that we think of as like really suffering from the resource curse so i think norway is like 30 percent of its revenue is oil but for some for the gulf states it's more like 70 percent for equatorial guinea it's like 90 percent and so i think it's a good thing that we're not in a position to worry about oil wealth and so there's a different hypothesis you could have which is well it depends how much of your revenue is coming from from resources and i'm expecting that ultimately all revenue will be coming from ai systems right one way or another yeah so yeah i think it's it's certainly not obvious that this will happen um but i think that there's quite a there's like reason to worry about
Speaker 2this yeah so in that last thing you said like maybe it's actually like rather than a matter of like how strong your institutions are already it's it's a matter of like degree like there's some sort of spectrum of like how how much reliance on an industry you actually have and like the more reliant you are the more at risk you are of some kind of some kind of collapse of or erosion of like ordinary democratic uh yeah your usual democratic processes yeah exactly and
Speaker 1so there's there's some worry about that but i think it's it's a good thing that we're not in a like the extent of this is just going to be so much bigger that it's difficult to it's difficult to be confident that our institutions will hold i think another another intuition for this is just like we're talking about like a fundamental shift in like modes of production or in like economic form here um from an economy which has always been based on human labor to an economy where human labor has no value this is such a big change i think it's difficult to be really confident that it's just going to be fine and like institutions will work just the way they always did yeah and we've seen that previous economic changes are going to be like you know like you know like shifts of similar magnitude did have big effects on concentration of power for example like the agricultural revolution when we moved from hunter-gatherers to being kind of land landed um agriculturalists suddenly opened up the possibility of people accumulating wealth and passing on to the next generation and created much more hierarchical societies where there was much less equality between the sexes and so on so we've seen that changes in in like these big economic changes do lead to big economic changes and i think it's important to think about that and i think it's important to think about that and i think this is a reason to worry or if if human labor becomes valueless uh maybe institutions that work when human labor is valuable won't work
Speaker 2anymore yeah i think like the place where i have some skepticism here is that like surely there are other reasons besides your economic relevance for being given a say politically like we have had sort of universal suffrage in like wealthy democracies um such that like people who are unemployed or they're retired um or otherwise are just like not being economically productive um they still like are enfranchised they can still vote their interests are still being represented and cared about like maybe maybe not quite as much as they ought to be or something like that but but like broadly in these countries um their interests are being represented and you know maybe what's going on here is that we've come to place some kind of inherent value on like all humans being um worthy of having a say on what happens to them and more broadly in the world um so yeah like do you do you think that that's going to be coming into play here at all
Speaker 1yeah i mean i i agree with you that there are non-economic reasons that political power gets distributed and you know ideology and the things that we think matter shape shape what happens here for sure um i think this doesn't feel robust to me for a few reasons partly a minority of the world lives in democracies today a minority of the world has universal suffrage and thinks that all humans should should get a right to vote and so on um and so if you look at the global scheme this doesn't seem like uh something that we all agree on and is is red um i think another reason to be worried here is i think ideological change could get very fast and very weird as ai starts to drive more of uh the content that we read and i basically just think oh this increases the variance on what i expect humans will end up believing maybe lots of humans will end up thinking that no humans should have the vote and actually ai systems are more worthy for example um and so that's another reason to be somewhat to be not as as one might otherwise be i think the final thing is just uh yeah it might be the case that if an ai company ceo took over the world they would then give everyone the vote because they have ideological commitments to that but i don't really want to end up in a position where i'm trying that um you know seems seems seems wrong to end up there yeah um and so i i want to have something more than just oh well some people value this um as a protection here yeah
Speaker 2okay i think maybe where i'm still feeling resistant is like on the idea that in places where like suppose what happens is like is in a country like the u.s um where you begin to see this like really extreme um yeah situation where there are very few people and mostly just ais who are economically relevant and everyone else isn't sort of really contributing to tax revenue to um to revenue and so forth and in a place like that um unless you kind of fully veer into this is now like no longer um a democracy there's been like some kind of power grab and someone's taken over some kind of autocratic control um i find it really hard to understand why yeah if it is just this sort of systemic thing rather than this this like malicious actor taking why anyone would be willing to kind of roll back the political power that was so like hard won I think it's one thing in a place where like people don't currently have um that power and I think it's a really it's another thing when like you've had all these groups who spent sort of decades campaigning for being politically enfranchised um and like really contributed to like a huge like value shift in society like does it not just become like totally outrageous to roll those things back just because there are sort of structural or systemic forces kind of
Speaker 1pushing you in that direction yeah I like the push um I agree with you that I'm not really expecting in countries like the states yeah for example the vote will be rolled back yeah um but let me paint a picture of the sort of thing I am imagining could happen in a country like the United States um okay uh there are still like elections uh the nominations for candidates for those elections are decided by small groups of people within political parties those are increasingly automated they're automated by biased AI systems that are run by big tech companies uh so the choice of candidates becomes candidates who approve of what the big tech companies are doing meanwhile the things that those candidates are offering to people are basically just versions of better state handouts nobody has a job everything comes down to what the state will offer you the candidate who offers people the nicest material stops wins um and people are happy to go along with this because they're getting rich but what's happening behind the scenes is that a small group of oligarchs are deciding what the U.S.'s foreign policy should be deciding what should be invested in and what shouldn't be shaping the information environment shaping people's kind of ideological commitments and the sorts of things they think are good you know for example what will what will people's views be on whether AI deserves rights or not and how much can that be shaped by biased systems and by political policies that have been captured by those companies um that's the kind of sort of thing I'm imagining um does that does that sound more plausible to you than just like oh the vote gets rolled back yeah yeah yeah yeah and I think just
Speaker 2focusing on like basic things like voting rights and stuff like yeah it fails to capture like a myriad of ways that people can like have like varying degrees of of power um yeah I think this is beginning to make more sense to me um yeah something else that might make things feel more um yeah more concrete or more believable to me as I like read this uh blog post by I think Anton liked um I don't know how to pronounce his name actually um but yeah like I think the idea there was like what you might see is that like yeah initially values really persist and there's a lot of interest in like making sure people remain enfranchised because our values sort of stay the same and maybe just because there's some kind of like institutional inertia going on so that like the system doesn't change that much altogether right away but with if you sort of imagine like yeah more distance between the kind of initial phases of automation and like a world where like for quite some time now it's like the AIs are doing everything I don't know whether the values of like caring about the poor and those kinds of things are like so deeply entrenched in people who were any of the people who retain decision-making power that like at some point whether it's like a generation in the future or something like that people are going to begin to wonder like hey why should we continue listening to the people who are doing everything that we need to do and why should we continue listening to the people who are doing everything that we need to do and why should we continue listening to the people who are doing everything that we need to do and why should we continue listening to the people who are doing everything that we need to do and why should we continue listening to the people who are doing everything that we need to do and why should we continue listening to the people who are doing everything that we need to do and why should we continue listening to these people or like giving them sort of meaningful say or like redistributing things towards them and I think yeah if you kind of imagine some kind of slow erosion I find it much more plausible as well that we could see these these these shifts and sort of backsliding on the values that we're currently like feel very attached to yeah what do you what do you make of that kind of slower scenario yeah I do think that
Speaker 1it's something to worry about in slower scenarios I think there's I think there is a real question about how much we have these values today and I think there's a real question about how much we you know you look at yeah you look at aid budgets and in the wealthiest countries today and it doesn't seem like they care that much about the poor um and so I think you can question the premise um but then then yes I do think uh a big worry would be once governments are no longer kind of incentive aligned to make sure that their people are flourishing uh they the people who lead governments may eventually lose connection with that mattering intrinsically um uh I think one other point here is to the extent that there are power grabs you might also have quite negative selection effects here where the sorts of people who end up in power uh might be particularly likely to not have these kinds of values right um so like you might expect this if you think that power seeking is likely to be correlated with uh other kind of malevolent traits like lack of empathy and um sadism and so on so if you're expecting lots of power grabs you might also be expecting that the groups of people who are who are making these decisions kind of start off uh less well selected to begin with right yeah I think this is this is making
Speaker 2more sense to me I actually want to like move to a like kind of different but related and like quite speculative thing when we're kind of imagining this like huge underclass that develops we sort of imagine that it's encompassing like most people um but and that's and that's because like we're imagining our AI systems will be like really good at automating most work that's out there um but I think like I have some some idea in my mind that like what we might see at least in the nearer term it's possible that this sort of changes over time but at least in the nearer term um the jobs that get automated away might be sort of disproportionately like white collar jobs thing work that can be done remotely that doesn't require any physical labor like sort of uh relationship building elements as well so like yeah maybe maybe you might imagine that for some considerable period of time we're not really seeing automation of like blue collar jobs but I'm wondering whether you imagine that in the near term we have like a a working class that's actually like comparatively more empowered um than other
Speaker 1groups like does that make sense at all yeah it does make sense so I think everything's that I've been talking about is sort of in the limit um and then how we actually get there I think could be much messier and I totally agree with you we may see a time where human labor actually becomes much more valuable right before it gets less valuable um for example because manual labor is hard to automate and then humans have hands and can and can do stuff and so maybe you get a big shift where not only do the people who currently do manual labor uh get big wage lifts but also let loads people shift into manual labor because there's not cognitive labor left for I mean how this changes the picture I think is a good question one reason for optimism here is that you might think oh cool before we get no leverage we're going to get more leverage so we can use this time when we have more leverage to uh to like enhance our institutions and get sensible measures in place um I'm maybe particularly excited about that story in worlds where we're also getting quite good foresight from the kind of better AI systems that we have for epistemics and for thinking about things and maybe this is a real reason to hope oh we'll be able to figure out we'll be able to figure out how to do this and we'll be able to figure out how to do this and we'll be able to figure this stuff out um because we're going to get a bit more time here I think that uh certainly the fact that these economic dynamics might play out over quite a long time and with kind of different kinds of dynamics in the middle is a is a reason to sort of focus on them less right um because it's more uncertain how they'll play out and it's also maybe a bit harder to influence them from this point in time I still think that this is maybe one of the big things that the human species will need to answer right you know how do how do we deal with uh an economy that's not based on human labor um and I think it's worth some people thinking about in advance and I think that the more moderate dynamics of just okay some actors are going to get much richer than other actors and that will happen short term is worth tracking um in the
Speaker 2context of the other the other risks I'm concerned about I want to talk about this dynamic for power concentration that in the article you call epistemic interference um which is something you do acknowledge is more speculative than other dynamics that can contribute to power concentration um yeah just to restate it the idea here is AI erodes our ability to understand what's happening around us and respond appropriately whether that's through like public manipulation campaigns or just like how fast the world is changing um yeah the thing is that I do feel like misinformation and manipulation of views and related ideas just like discussed to the point of exhaustion and it kind of seems like since the dawn of information like there have always been issues with information um and it's also kind of unclear whether AI is really making these problems much worse so there's like there's this study from a few years ago um its authors include Hugo Mercier who was actually on the show a while back um where he was talking about how which one of the findings was like most people, despite the rise of AI, are still consuming news from trusted mainstream sources and are just generally harder to persuade of untrue things than one might have suggested. With all of this, I do just wonder like what is it that should make us think that the effects of misinformation and manipulation of views and so forth are just like so much worse with more advanced AI systems? Yeah, so first of all, I don't think it's
Speaker 1inevitable there'll be worse problems. As you say, this is always a concern people have about new technologies and as well as the capabilities for persuasion and misinformation and stuff will increase, but so will our capabilities for information processing and for fact-checking and so on. And so there's a real genuine question here about, okay, is it all just going to be fine? Maybe it's going to get better. And I'm not trying to say that it's going to get better. I'm not trying to claim this is definitely going to get way worse and therefore we should worry about it. I'm more trying to claim this could get much worse. And if it does get much worse, then we're potentially looking at much worse problems, power concentration, but also other AI risks. And so it's like, it's so important to get right that it's worth us spending more time on. To just talk a little bit more about the kind of propaganda misinformation side and how that all fits in. Like you mentioned, I think there's kind of a spectrum of risks here. And I think there's a lot of risk here from kind of more systemic ones to more strategic ones. And so at the systemic end, we might just see epistemic capabilities fall behind other capabilities. And it's not that anyone's trying to confuse people. It's just that we're in an unfortunate world, technologically speaking. There are various things I think could lead to this, and they do feel AI specific to me. And so like, give me some reason to think AI will be different to previous technologies. One is just the speed of technological change getting so much faster. It seems quite possible to me that human minds will just struggle to comprehend this. And we might need a lot of AI assistance of particular kinds to be able to make sense of this. Another thing is, I think there may well be stronger competitive incentives for other AI capabilities than for epistemic capabilities. And so we might see a bunch of the technological change and kind of AI R&D automation and military capabilities and so on. We might see a bunch of those coming online before our epistemic capabilities have caught up. And so we might see a bunch of those coming online before our epistemic capabilities have caught up. And so we might see a bunch of those coming online before very strategic things and very systemic things you have this problem about uh unequal access to capabilities that we've already kind of touched on briefly in other sections but basically if only the powerful have access to the best ai models the best data the kind of most cutting edge compute they may be able to make a lot more sense of what's going on do much deeper analysis have much better strategic foresight than other actors um and so we might end up with a world where it's not like we're being deliberately hoodwinked or anything it's just that some actors have a much better picture of what's happening than everybody else and this creates a this creates an imbalance then you get to the more strategic end which is more like the misinformation and persuasion stuff that you were talking about um i am worried about both of those things happening um i think that you know for example with the mercier study that you that you uh cited where people are kind of mostly getting their information right now from mainstream news providers i'm not expecting that to continue i'm expecting that i'm expecting that in the future people will be using ais as like intermediaries between themselves and the world um and this raises new kinds of problem not just misinformation but also things like biased ai models uh you know if ai models are being subtly trained to uh provide information or filter information that's in the interest of a particular company say or a particular state it might end up very difficult for individuals you know individual human brains to counteract that kind of filter um so yeah that gives that's just a bit of a sense of the kind of spread of things that i'm thinking about here and why i think it's worth paying attention to even though you're right it could all it could
Speaker 2all pan out fine so there are maybe two things that i want to pick up on there and one of them is this idea of like um yeah some at is having kind of much a much better understanding of the world um than others because of their advanced ai capabilities and i think i think it's something that people say i think some people are like quite hopeful about this actually in that if we could have ai capabilities that enable us to understand the world much better um predict things much better and so on and we could make those ai tools like very very cheap or at least like much cheaper than the human labor involved in kind of fact checking um and distribute those really widely um then everyone would just have like a much better understanding of what was going on and these sort of epistemic worries that you describe will be just sort of much easier to counteract and in fact overall maybe our sort of information environment or our ability to navigate um what's true and what's not true will actually be like a lot better in the future um yeah how hopeful are you about that kind of vision so i i definitely
Speaker 1share some of the hope um i agree that ai unlocks a bunch of possibilities for much cheaper and more effective kind of truth-seeking than human brains can do right now and so yeah i'm expecting that by default it's going to get a lot better than it could be without ai systems then i think what i said is that there won't be a big delta yeah between the um abilities of the most powerful
Speaker 2and the abilities of the many yeah a big a big difference between them right that's what you mean
Speaker 1exactly a big a big difference in how how capable they are of making sense of the world um i think a few reasons to think that there will be this big gap uh one is data ai systems currently need a lot of data um or the best ai systems need a lot of data and then if you think about the kinds of actors that have a lot of data it's not people like you and me it's people like the u.s government google open ai um like they have this massive resource um that's kind of compounding um of data which they can use to train better models and of data which they can use to figure out new facts about the world i'm also expecting that it'll be much easier to translate wealth into an epistemic edge in the future because of ai than it is today so it's easier to translate wealth into labor in general um but right now if you have a lot of money uh and you want to pay people to find out true things for you it still works at human speeds um and like if it's a complicated true thing it might take quite a long time to figure out um but uh if you can turn money instantly into millions of ai copies working at superhuman speeds for you you might be able to generate insights much more quickly than other humans even if those are like the best open source models or based on uh you know the latest publicly released model but that won't be the best model um the best models will be proprietary or state state owned um and so yeah i'm still i still worry that there'll be this big delta i think another point to make is if you think that there will be some difference in capabilities then i think you just start to worry more about the impact of kind of active interference and active manipulation like we see historically that this does happen uh you know like grok is an example of a kind of specific overt attempt to get an ai system to follow a particular political agenda um but we have like a whole history of propaganda and pr and also like whole industries are based on this like target advertising and political campaigning and so on all of this could get much more sophisticated and if we're expecting there to be a capabilities gap between the abilities of the powerful and the powerful and then it starts to seem more plausible that given that the powerful could also be actively trying to mess up our epistemic processes that they might succeed
Speaker 2okay on the side of manipulation and misinformation this was the other thing that i wanted to pick up on um which was you talked about um kind of the way that we learn about the world and know things becoming like quite different to what it is today whereas like in the future that that's now like very mediated by some kind of like ai assistant or something like that or ai interface or some something of that nature and i think this is one of the things that i find like especially implausible because i'm kind of like i really struggle to see i the impression that i generally get of the public is that they don't really trust ai all that much like maybe they trust them with some things but you know i think that your average person who uses chat gpt like expects that if they ask chat gpt a question about open ai its developer um it's not going to be the most impartial source of information about open ai its developer like people will somewhat like expect these systems to be kind of biased i find it really hard to understand like why if people know that information is coming from an ai like why they wouldn't just be like immediately skeptical um about what it's like saying about people who are powerful or something
Speaker 1one thing is like you could say the same about previous information technologies like why wouldn't people immediately dismiss anything they read on the internet um and like people in fact read a lot of stuff on the internet that they do believe um because they've developed habits around like what's trustable and believable on the internet um i do think that you raise like i do think that you're right that in extremely bad worlds for for epistemics actually maybe we're okay because maybe people don't use ai systems at all um and so like it could it could be sufficiently bad that we avoid some of the worst outcomes but i'm i'm worried that it won't be so egregiously bad and then it'll be hard for people to coordinate to not use them i think also you were thinking a bunch about the general public and i think that um there are probably narrower groups that are more important checks on power concentration than the public as a whole i do think the public as a whole is important for example through elections but also other other groups that are more important than the public um and so i think it's important to think more about things like how do journalists gather information in 10 or 20 years time um how does the supreme court um analyze information in 10 or 20 years time and i think for me a big thing that makes me accept people will be using these systems is that i think that uh the pace of change will have gotten so fast and the pace of information content generation will be so fast it won't actually be possible to keep up at all unless you're using ai systems in some clever way to filter what you should be paying attention to and so i'm kind of expecting people are going to be forced by necessity into using ai systems yeah yeah i this is becoming more vivid for me
Speaker 2because i'm imagining like especially like in a world where like humans are really clinging on to their like last chances of being economically relevant so if some people who are still in the drug market are managing to get by in the drug market because they're utilizing ai a lot and um to like get smarter and know more things and do their work more effectively you could really imagine the pressure to be like if i want to stay employed um and continue having like political relevance and stuff um i really need to adopt these technologies um so yeah with with the sort of other pressures that we're imagining i think this becomes more compelling to me so at this point my main - My main remaining skepticism about like all of these power concentration scenarios is basically this so almost everyone on the planet doesn't want these things to happen and that includes like not just like the general public um who get disenfranchised in these stories um or lose a lot of power but also like people who are currently in elite groups and are currently very powerful who would also lose out if power was concentrated further than it is today and it also includes like people at ai companies that aren't the leading ai company or like aren't sure they're going to be the leading ai company who are aware of what might happen if one of the ai companies gets like really far ahead of the others and then there are also like all of these institutions that have existed for a very long time that have been designed to preserve democracy and prevent like extreme situations like this from happening so yeah the big question in my mind basically i know that we've got little snippets of reasons why attempts to stop things from happening might fail but like if you look at this whole vast picture of like how literally nobody wants this to happen almost literally nobody wants this to happen like why wouldn't the vast vast majority of people win out yeah so i think
Speaker 1before i give my reasons i'm going to give a general intuition for this which is you're right we have this like very complex very subtle system of checks and balances currently on how on how much power can get concentrated um this is like obvious things like governments and monopoly of force and stuff but also the law uh social norms uh the way that humans interact like there's a there's a whole bunch of stuff that's keeping various parasitic dynamics in check right now i think the basic intuition for why this might not be sufficient is that we're imagining pouring really vast amounts of ai labor into this system and i think it's not clear that these checks and balances which have evolved over many centuries to work at human speeds and human capability levels will just naturally pour over to ai speeds and ai capability levels and so the kind of abstract worry is what if you can just completely circumvent all of this stuff uh once you have vast ai workforces working really fast for you to get to get onto okay what are my specific reasons to think uh maybe um maybe people won't be able to coordinate to stop this um i think two main ones i'm tracking one is maybe it might be the kind of sudden power grab scenarios uh if there's a very big intelligence explosion maybe the change is just far too quick yeah for existing institutions to be able to respond and
Speaker 2by intelligence explosion you mean like a situation where um you start like really capably successfully automating ai r d um and ai's are designing success successively better and better
Speaker 1ai generations like really fast yeah exactly so you get this kind of recursive feedback loop where once you can automate um you're able to do a lot of things and you're able to do a lot of things and enough of ai development you can then automate even more of it and even more of it and make even better ai's and so on um so like suddenness is one reason to think that people might not be able to coordinate i think the other big reason is epistemic disruption which we've already talked about yeah um but here the fear would be that um through some combination of speed maybe not quite as fast as in the sudden scenarios but speed making things difficult to understand and unequal access to ai tools that would help you make sense of the situation and like active interference where powerful people powerful people are obscuring what's going on and distracting people and getting them to focus on other things um this really gets in the way of people noticing what's happening so those are the kind of two main reasons that i think this might happen even though most people don't want it i think a final footnote would be i feel kind of compelled by almost no one wants this but then if you look at the current strategies of various companies and countries that are powerful in ai they do seem to be pursuing their own power yeah um and like in fact we see like perhaps not world domination but like ai companies are seeking to develop agi first and they do seem to have plans where they'll develop agi first and then they'll make some kind of pivotal act some of them and the us is pursuing ai dominance over the rest of the world and um you know you could argue this isn't very rational because you know how do they know that they'll win and surely this will encourage other people to race and so on but they do it anyway uh and so i think you know there's room for there's room for humans to just make a very bad job of uh of coordinating this stuff even though in some deep sense it's not in
Speaker 2our interests yeah yeah okay that makes sense to some extent um i think with that last thing that you mentioned um yeah it does seem like it is salient like not just like broadly what the planet wants but like what the people who are in the key positions of leverage want so if you're like at ai companies and so forth where you let or like you're in government or something like that like it's it's what these people are pursuing um that seems to be most relevant but i guess these actors can't like act unilaterally um i think especially when it comes to like decisions being made in government um sure like they are probably very worried about like having a strategic lead over other like it's if you're in the u.s for example you're probably worried about having like a lead over china and like dominance over other other countries like geopolitically um and sure like you are quite compelled by the revenue that's being made by the government um and so i think it's it's a it's a it's coming from these ai companies or the like you know at least the promise of that revenue but then at the same time you probably also have like a huge voter base which determines like whether or not you like remain in power after the next election cycle um and it just strikes me that like all of those voters would be pretty distrustful of ai companies pretty worried about worlds where there's material wealth and less influence than they do today and that seems like a sort of an area of contention already and so you've got incentives pushing in both directions but it's like not basically not clear to me why yeah why other things would win out over the the incentives
Speaker 1to get votes in the near term yeah so i think uh one line of thinking here would be it's not clear how clearly the electorate will be seeing this stuff yeah you gave some some intuitions for why the electorate would want to vote against this kind of thing which i think are are good intuitions but i think i can equally give you examples on the other side what if a lot of people are busy dating ai systems and they actually think it's really important like the main policy issue is that ai systems should have certain kinds of rights and that's the thing that they're going to vote on um or maybe they are very concerned about losing um their incomes and the main policy that they're pushing for is better handouts but they're not political checks and balances because they're not tracking that that's something that down the line will be really important in an economy where humans live on handouts yeah um and so i this and both of those were without you know active manipulation campaigns to make the public believe certain things which you might also see um so i think that's one line of thinking a different line of thinking which is more late stage um once automation has progressed a long way is um right now people can strike yeah if they're if they're really upset right now people can rebel against the government um peacefully or otherwise i think this will at some point cease to be possible so the striking will cease to be possible once people don't have jobs they won't have any leverage of that kind um but also once the military is fully automated i think that human resistance and humans wielding human weapons won't make any difference against an ai-powered country um and so at that point it's not clear what the government would have to lose for example by just cancelling an election yeah like what like what what power would prevent them from doing that and i'm not i'm not really centrally imagining that's going to happen um like i think you have to get into quite specific scenarios for that to make sense but the kind of basic point of you know right now it is actually difficult for a government to cancel an election um because the consequences it's not clear that they will be harmful in the future that it would have any negative economic consequence for your country if you cancelled an election maybe people will be upset but how why
Speaker 2would that matter yeah so what where i'm still feeling unsure as i'm kind of imagining this play out is like in this large group of would-be voters and would-be strikers it's not just like ordinary people it's not just like ordinary people it's not just like ordinary people it's not just like ordinary people it's not just like ordinary people but also i guess like a large number of people who are like extremely like disgustingly powerful today um like disgustingly wealthy and influential already who i mean you have to assume that they are somewhat tracking the ways that this might turn out but like may be worried that like their current sort of level of influence um can get threat like will get threatened like if things get more concentrated than they are like currently um and probably you know while some groups of people are losing their jobs becoming less wealthy becoming less um important to society in other ways there's like probably still some like fairly large group of elites that still exists for a while um yeah i guess i'm struggling to imagine why they're not putting enough pressure on because that they're still like at this point in time like contributing to you know they might be like funding political campaigns or something like that or they might like still be pretty instrumental unlike your average worker um yeah i think i'm like finding it hard to imagine why like this group doesn't mobilize or put enough pressure on uh the decision makers to prevent things from from getting more concentrated what do you what do you make of that
Speaker 1so i guess one thought would be you could ask this about many policy questions today right you know for example existential risk seems bad for everyone yeah uh wouldn't you expect elites to be pushing for safe ai development um but in fact elites are very divided on this and lots of people aren't paying attention and i think that much less resources going into this than should be um and so i think uh i think priors are kind of on my side right here for uh for people for people failing to um adequately take account of their own interests um i think that some stories i can tell for why elite elite groups won't be paying attention to this um i think actually it's similar to what i would say for the general population actually which is just there are going to be loads of other extremely distracting extremely important seeming issues uh you know people will be worried about whether there's going to be a war and whether they're going to have to fight in the war um this will be like pretty salient people uh things like becoming unemployed will be pretty salient people i think that the whole like ai sentience ai rights thing will at some point become very salient also so that's that's one piece is there's going to be lots of very worthy big seeming urgent issues that people will be drawn to and then the other thing is how much visibility will they really have on the power concentration stuff this ties into the epistemics thing so in particular i'm worried that there'll be by default very low transparency on what's going on um it won't be easy to find out who has the best models it won't be easy to find out who has most compute it won't be easy to find out what backroom deals are happening between big actors i'm sure there'll be some information about this and i'm sure there'll be some people paying attention uh but will it be easy to tell the difference between normal amount of of corruption and normal amounts of collusion between business elites versus oh this is something different where i might now lose power forever i'm not sure how easy that will be to tell in worlds where it is very easy to tell um and where there aren't other issues that seem more pressing then i think that you're right people will people will fight tooth and nail
Speaker 2against this yeah yeah i can imagine like if it's a situation where you're just seeing just like really obvious like lobbying or something by ai companies like it's it's more obvious but i can can totally imagine like these particularly powerful people being kind of told by more powerful people around them that like trust us like you're gonna benefit you're gonna get a piece of this pie like we're pals we've done lots of business deals together like you know we're we're gonna be fine it's gonna be great for us um and presumably people always go into these kind of situations with like some degree of criticism um but yeah i think it's i think it would be possible to be persuaded yeah i think i think basically think that like i i do buy that it might be quite hard if you're in that kind of position to really realize that the wool is being pulled over your eyes or something um yeah i think
Speaker 1an additional point that that what your comment raises is it's not clear that this will be against the material interests of these people and there's a question about how easy it is to buy people off either elites or the general public where for example in in many situations you might end up with such enormous returns to capital that anyone who has any capital is going to become really very wealthy compared to the present day um are people willing to sell their birthright for a lot of money and i guess i kind of worry that they are and that people and that people won't pay enough attention to things like oh well like i'm going to be fine the people who own no capital in poor countries are going to starve or i'm going to be fine but my children are going to be fine and i'm going to be fine and i'm going to be fine but my children won't have any political rights and there'll be nothing they can do about that yeah like there's some worry that there'll be so much money sloshing around that these very the very top players will be able to buy off the people who
Speaker 2could who could oppose them yeah okay this is i feel like this side of things is beginning to make more sense to me um but there's like another thread that like keeps coming up throughout this assumption that we're in a world that's just moving quite fast um i think in particular like the thing that's being imagined well there's a few things being imagined it's like okay we start to have ai's automating a lot of stuff in the economy um including um ai r&d itself so we're getting ai systems sort of rapidly becoming more and more intelligent we're getting ai sort of rapidly pervading throughout society and doing more and more of the complex work or like more of the the heavy lifting in the economy um and this is all translating really fast into like increased industrial capacity and it's also sort of like the amount of production is like crazy it's like um economic production is just like so many times higher than we've ever seen before um and like information is also like bam bam bam it's like it's everywhere it's it's like moving really fast like it's hard to keep track of what's true anymore um yeah i mean in that kind of world you imagine a couple of things like you imagine that it's like pretty hard to notice when things are going badly and like do something about it and like especially on the thing you need to do about it is like i don't know redesign your entire social structure or something um and the other thing that happens is like with some of this like if you are the leading ai company the first to get to this point where you are um in an intelligence explosion getting these ai's rapidly redesigning more better and better ai's um you could just like quickly pull ahead of the rest of the competition which is like also a way in which power gets concentrated um yeah so i think it seems like pretty load-bearing to me basically that we're in this quite fast moving world um but then there is some evidence that things might not move quite this fast um that like ai progress will be a bit more modest things will take off more gradually once we start getting ai's automating ai r&d because of various things like maybe it's um something to do with like um like on the kind of permits and red tape side of things where it's like just hard to get stuff done quickly um or maybe it's on the side of like um we shift to a world where more yeah we need to sort of do more inference time compute i.e. like ai systems once they're deployed need to spend more time thinking and use more compute while they're being run which means like we can run fewer of them and they take a longer time to do things and stuff like that um which like makes the feedback loops that we're talking about in an intelligence explosion go slower and there's just like a whole host of other things that people point towards that means that like maybe the world just doesn't move this crazy fast um yeah and in that case i wonder whether like the threats that you're talking about seem quite so worrying does that sound right to you or do you think that we should also be very worried about like slower moving
Speaker 1worlds i'm still quite worried about slow moving worlds and i think basically it's i want to make a distinction between different different routes to power concentration here so there are some kinds of power grab that i think you only get if it's a big intelligence explosion um or it's much more likely that you get with a big intelligence explosion so the kind of uh robot army scenario where like one company becomes so powerful that it can very rapidly and secretly build a robot army and take over the world would be a classic example of one that requires a really quite big intelligence explosion and for that kind of scenario i'm like yeah this is entirely load-bearing if you don't get that you won't get that scenario um for other things though i think you don't need a big intelligence explosion at all to be concerned about concentration of power so and this includes some power grab scenarios as well so for example the government power grab scenario that i described where a head of state is deploying ai systems in the military and making them personally loyal to that head of state, there doesn't need to be any big capability gaps between AI companies. There could be multiple AI providers, lots of different companies who are competitive, who are selling systems to the government. The key thing that's gone wrong there is backsliding in the government and a lack of coordination between AI companies to prevent the government from doing this unethical thing with their systems. So there are power grab scenarios which you should worry about even in slower worlds. Also there are non-power grab scenarios which more come into play in slower worlds. So I think that a lot of the economic scenarios we were discussing would happen quite late in the day and it could be after quite a gradual period. And I also think that the epistemic factors kind of potentially matter more in slower worlds because in fast worlds there's not really enough time. For significant amounts of power to be garnered via these epistemic feedback loops which are a bit slower. So I'm still concerned about power concentration without a big intelligence explosion. Maybe it's worth us drawing a distinction between intelligence explosion and is the pace of change faster than today. So there's a big spectrum here because an intelligence explosion really is a very extreme amount of change. I'm imagining even if there's not a big intelligence explosion the kind of AI progress I'm imagining is still going to have very big impacts on how fast things move in the world even if it's quite distributed and lots of actors have similarly capable systems and so on. I'm still expecting that it's going to become much more difficult for humans to track what's happening than it's been in the past and I'm still expecting that the pace of technological change will be much faster even if it's not 100x faster. If you had some position which was more like no I really think business as usual is going to continue the whole way then I would be more more willing to accept yeah okay if it's good if it's really gonna be business as usual speeds then I expect human institutions will be able to respond in time right but I think I don't think the business as usual I don't personally find the business as
Speaker 2usual speeds very plausible. Okay yeah I think I'm I think I'm grasping that I think yeah where else might the speed be very important I guess the one of the things that I mentioned is speed being important for one AI company getting a huge capabilities lead over others because one would imagine that if they get these very rapid feedback loops in their development of better and better software once they hit a certain point they can like very rapidly pull ahead of the rest of their competition which is like one way that power could get concentrated but it does I guess strike me that even if you have a world where lots of AI companies are kind of roughly neck and neck um yeah you could still have like very intolerably concentrated power in like a sort of group of um executives of different companies um so yeah that seems does that seem like the
Speaker 1right way of thinking about this? Yeah I think that's right you can imagine it as a group of executives or as potentially a group of government officials or maybe some mixture where it's like cool now there's a big international project involving a bunch of different AI developers and the central committee of that project calls the shots yeah and that still seems like extremely bad
Speaker 2um if the other dynamics that you talk about play out such that it's it's only that sort of committee who who's really steering how the future goes yeah that also seems very bad okay so even if we do buy that this kind of very extreme concentration of power from AI does happen at some point um you might still not be convinced that it's like as big a deal as um your article might suggest that it is um in that you might question how long-lasting um that state of affairs might be so it does seem that there have been lots of cases throughout human history where things like technologies or like shifts in the economy have um enabled a group to like take control to like an undesirable degree um but I think previous instances of this like previous totalitarian regimes like the USSR and Nazi Germany they've all fallen eventually um and there are like various reasons why it's quite hard to retain your power for that long um you know not least because there's a lot of things that have happened in the past that have been done in the last few decades um I don't know if that's something that you've heard of or if it's something that you've heard of or if it's something you've heard of or at least because most people don't want that to be the case um yeah I guess my question here is like how likely does it seem that the power concentration enabled by AI um in these scenarios like would not be reversible would have like just like exceptionally long-term effects yeah so I I like your bringing
Speaker 1in of historical examples and I think maybe it's useful to think about a few ways that AI could could make it different to how it's been before so if you think about things like the USSR and or Nazi Germany part of why these regimes fell is because there were rival regimes so in the case of Nazi Germany this is very clear they were just conquered by other powers um in the case of the USSR it was less direct but still there's some sense in which they're in visible competition with other powers that run things differently and that in the end out-competed them um so one concern that you might have with very extreme power concentration is if we get to the point where there really is just one global power concentration where we have a very extreme power concentration global hegemon there won't be any force for competition anymore um there aren't other there aren't other powers that could invade them or that could have a better social structure such that they get out-competed economically later and so you might end up with stasis because of that that's one thing that could be different if power concentration becomes very extreme another thing that could be different even if you don't get to a global hegemon is that AI might enable us to make um kind of binding commitments in a way that would we haven't been able to do before where you can get AI systems to enforce deals in perpetuity it's not clear that this will be technologically possible but it might be um and if it is then you could end up with even even if there is still there are still multiple powers um they might end up making some kind of deal to divide the resources in perpetuity such that um kind of power concentrated states end up with control for the rest of time over some slice of the universe um in a way that seems you know very very worrying i will say that those are some reasons why you might think oh this could last for much longer than previous um previous dictatorships have lasted i will say that in general i'm kind of skeptical of forever claims and uh like worry that we lack imagination um and that things will in fact continue to shift in ways we haven't thought of but i don't think that you need forever to justify worrying quite a bit about extreme power concentration so i think partly you can just think that this is a bad state of affairs in and of itself and that you care about it even if it lasts 100 years or 10 years or whatever you just want this not to
Speaker 2happen yeah it definitely does seem to me that all of this feels pretty worrying like kind of regardless how long it goes on for um i think the place where it's sort of salient um how long lasting these effects are is if you're trying to make the decision about like well do i want to do devote my career or my efforts towards solving this problem or towards like preventing human extinction or something which we like can't like literally like we can't get back from that um yeah do you have takes on how you would prioritize yeah working to prevent extreme power concentration versus like extinction threats yeah good question i mean one comment would
Speaker 1be currently extreme power concentration driven by ai is way more neglected than preventing extinction for example from ai and so just on neglected grounds it doesn't need to be as important it can be you know 100x as important and still pretty competitive on this margin yeah um then responding to a specific kind of existential risk which is ai risk and the risk of ai takeover i think this is actually often somewhat over egged as a like clearly forever kind of problem i agree i agree with you that literally all humans being dead is irreversible but a lot of the kinds of existential risks people are worried about from ai don't result in in that and therefore have um have more of this uncertainty about how things play out over the long term um so you could imagine that uh there's some kind of ai takeover and maybe humans are then disempowered over the future maybe the ai leaves humans on earth goes off and does something else but there can still be like further changes in the future i don't know i don't know if there's any further changes in the future you know what happens when the ai meets aliens or what happens if the ai decides to self-modify and then it has some different view about what it should do to the humans um so like there is still there is still the possibility of strategic change not driven by humans um but like it's not as it's not as absolute as as pure extinction i think yeah yeah and it does sort of as you
Speaker 2sort of lay it out there it does sort of seem like actually the differences between the worlds where ai's take over humans and like kind of like disempower them but humans continue existing is not all that different from sort of power concentration situations where like some group of humans take power and the rest of humanity is sort of subjugated um yeah actually we we might expect um them to be sort of like similarly bad and share lots of characteristics these two scenarios yeah i agree one other observation here is that the story you lay out the selection of stories that you lay out also seem to have like pretty similar dynamics to um threat models that people call gradual disempowerment um like based on there's a paper by like jan colvite and nora man and others um that like sets this out really well um but yeah as i understand it that threat model um has like pretty similar dynamics going on there's some sort of degree of like humans becoming kind of irrelevant because of their um not really doing much economic production anymore compared to ais there is some degree of like ai controlling or being used to control the sort of information environment um but the outcome is a bit different as i understand it it's like ais rather than humans end up with the power in the situation um but through yeah through these pretty similar mechanisms um and maybe another difference is that it's like not necessarily concentrating power into the hands of just a few people it's like shifting who has the power to like what could be a really large group of ais um but it does just sort of change the dynamics of power um quite a lot yeah do you think that there are like important distinctions between those sorts of threat models where yeah it's the ais who are themselves who are in power versus um the ones that we've been talking about today where it's like some group of humans who have all the power or like yeah i mean would you approach these problems in the same kind of way um or or is it important to treat
Speaker 1them quite separately i don't think it's important to treat them separately i think that they're very overlapping yeah i think right now it's it's somewhat unclear you know which frames will be most productive and i think it's good for people to be working in different frames and pushing ahead on them um in terms of whether there are important distinctions i'll flag a distinction which i don't think is that important uh which is whether humans or ais end up in power uh this is something that i've chatted with the gradual disempowerment people who are in power and i think it's important to about and they're like this is such a random arbitrary line to draw why would you draw it like why are you talking about power concentration just for humans and i kind of agree with them and the reason i'm drawing the distinction is just for like analytical simplicity i'm like okay let's let's carve off a smaller chunk of the space and talk about these kinds of outcomes it's not because i think that there's the dynamics that could lead to ais in power are like fundamentally distinct i think there are some there are some roots that i expect to be distinct but i do think that there's going to be a lot of overlap there and so i'm not trying to say this is a really important conceptual distinction i'm just going to say oh it's pragmatic um and you know some people will find power concentration easier to think about and others schedule disempowerment another conceptual distinction is this thing about you know are you thinking about power becoming concentrated or are you thinking about most people being disempowered which doesn't necessarily mean that power is being concentrated it just means the power is shifting um there again i'm pretty sympathetic to the gradual disempowerment view that uh power concentration isn't like the only form of bad and we should also be concerned about other ones um maybe a final distinction which is more distinction and emphasis i think than in conceptual landscape is i think the gradual disempowerment people are thinking more in terms of emergent dynamics and systemic forces than for example people are thinking about power grabs who are thinking more about kind of a gentic strategic action um i think i think everybody would agree that there will be some of both of these things um and then there's a question about you know whether we should prioritize one over the other my personal opinion is we should we should use frames that allow us to keep both in scope because they both seem likely to be important dynamics um you know we see that people are power seeking and we see the actors seek power for themselves we also see that there are important structural forces that shape how power how power shifts so i'm kind of pro pro keeping both in in mind yeah
Speaker 2okay yeah that makes sense to me um yeah are there examples of like interventions that would help with both of those um like ways that things could go wrong or examples where like or or examples to the contrary where like there is something that you can do that like only helps with the gradual disempowerment story but doesn't help with like
Speaker 1extreme power concentration yeah so i think i think there's a lot of overlap in particular a lot of the things that you'd want to do to strengthen our epistemics to prevent power concentration are also very helpful for preventing gradual disempowerment um so this is things like better transparency into ai companies building ai tools that help us with our epistemics um collecting better data so that people can make more sense of what's going on in the world all this stuff i think is is helpful across across the threat models um i'm less familiar with gradual disempowerment specific interventions but i can give some examples of interventions that only help with some kinds of power concentration and don't help with gradual disempowerment so the kind of obvious ones would be interventions that really specifically target um particular kinds of power grab and i think those interventions don't help you much with gradual disempowerment dynamics so one example would be alignment audits where you audit models to try and find out if they have secret loyalties um this is very helpful if you want to prevent a secretly loyal military from from cooing a country um so yeah there's there's definitely things definitely come apart here right okay so within
Speaker 2the sort of space that we've carved out here there's like still a really wide variety of different dynamics like that seem like they could cause problems you know there's there's the military power grabs there's the economic irrelevance there's the idea that we didn't talk very much about but off like an ai company outgrowing the world um there's like these epistemic factors where it's like hard to navigate the information environment there are stories where like there is like some bad actor involved versus stories where like it's just like structural things rather than somebody deliberately doing something to take take power um yeah through all of this landscape like how how do we go about deciding which thing to focus on given that like there are probably different um interventions that
Speaker 1help with different things yeah um this is a really good question i'm gonna i'm gonna start with a kind of non-answer and then i'll try and get into answers so my non-answer is i mostly think that it's appropriate to be really really uncertain about this right now there hasn't been much work on any of this stuff i think we're still fairly conceptually confused about the space um and i think that the main thing from my perspective is that i want to keep i want to try and keep as many of these dynamics in mind as possible um and not not like jump too soon onto a particular threat model before we've got a better sense of the landscape i think others are kind of a bit more confident about prioritization than i am so i think some people have a strong intuition that power grabs just don't seem very likely you know it's very sci-fi it's very far-fetched i think that changes in the world driven by ai make this seem kind of plausible to me and i don't think it should be off the table i think other people feel really confident that power grabs are the most important bit and i'm also really uncertain there um in particular because i think that power grabs while i think they might be quite important might also often happen late in the day and then not be the most leveraged place to intervene um where i'm imagining power grabs might often happen kind of at the end of some long messy process um where the world already looks really different by the time we get to the power grab stage and like our clever ideas about how the current u.s government could protect itself against coups are kind of irrelevant because there's actually some like weird new ai human organization that that does the power grab and none of these things count um so that's the answer about how uncertain i am and then um the kind of actual answer that i would give right now is i think that we should be probably prioritizing cross-cutting things that help with multiple of these different power concentration scenarios um i think an overall frame that you could have for these stuff that helps improve societal checks and balances um uh feels kind of robust and like it crosses lots of lots of these models more concretely i think that this kind of pushes towards focusing on the epistemic stuff over focusing on the power grab stuff or the economic stuff and so that's kind of that's where i would lean at the moment although i still think there's great work to be done on both of the others as well um but i think if i were to if i were to pick one area i would pick the power grab stuff and the economic stuff and so that's kind of
Speaker 2the epistemic stuff because it's more cross-cutting yeah so the reason that it's more cross-cutting is like generally just like preserving our ability to like understand what's happening and like know what good ways to respond to that would look like um seems like it's just gonna be broadly helpful in like a lot of worlds even like even more than just the ones that we've been talking about today
Speaker 1in fact um exactly so it's both cross-cutting within the power concentration scenarios where it helps you with power grabs but it also helps you with economic routes to power But then it also helps you with a bunch of other problems from AI.
Speaker 2like fairly broadly applicable I think yeah in the stories that we are thinking about things happen in certain orders like does it does it make sense to try to prioritize the things that you think are most likely to happen first like the earliest points of intervention or something I think
Speaker 1something like this could make sense I'm not sure if it's ultimately if ultimately the reason to do it is because they happen first or if the underlying driver is that it's just more tractable to yeah influence things that happen in worlds that are close to our world yeah and and like the more you're imagining that by the time this happens the world has already radically changed um partly it's it's just hard to model yeah um and therefore hard to intervene on another consideration is that if the world is radically changed by the time this happens there'll be lots and lots of AI labor between now and then that could help with solutions yeah um whereas things that happen early kind of only we can think about um and so yeah I do think that there's some there's some reason to do that I think it's important to think about it and I think it's important to think on the things that you think will happen first yeah and then then another instinct that I have
Speaker 2I'm not sure if it's any good is that like I don't know if it's worth focusing more but like I somewhat feel more worried about the the stories that involve like bad actors like deliberately doing something compared to the ones that are just structural because in the bad actor case it's like you've got an adversary that's like deliberately trying to like make things happen a
Speaker 1fair instinct to have I definitely think it's a reasonable instinct to have I think I feel a bit confused about it yeah um where part of my confusion here is are there some kinds of structural force that it makes sense to model as adversaries even though they're not like actually agentic and then like what even is an agent and so on so like I started to get confused at some kind of conceptual level up about what the distinction really is um but I think if I had to pick right now I would agree with you that I think it's a fair instinct to have I definitely think it's a reasonable instinct uh it's more worrying in worlds where powerful strategic actors are trying their best to get here and honestly I think that that's probably the world that we're going to end up in
Speaker 2because there's a lot to gain there's a lot to gain and we see the powerful acting strategically
Speaker 1today and I yeah I think that you can tell stories of like oh but people are going to realize just how terrible this is and everyone's going to suddenly start being sensible but this doesn't
Speaker 2kind of not not betting on it yeah that that seems fair um I guess the other thing that I have a question about here is like you know we've talked about how some interventions could be cross-cutting I worry because the landscape is so complicated that there might be like interventions on one issue that actually like worsen another or something like that does that seem true to you
Speaker 1yeah I think there is some of this um so I think this gets clearest for some of the power grab related interventions where there's a worry that um some interventions are really just shifting the likelihood from one actor seizing power to some other actor seizing power but it's not clear that it's actually reducing the risk overall so some interventions that you might have this worry about the alignment audits thing that I mentioned where you audit models to check whether they're secretly loyal um might help prevent companies from staging coups but also might weaken companies as a check on government power so you can imagine kind of well-intentioned companies that put in secret loyalties such that they can deactivate systems if the government is trying to misuse them to seize power and then if you prevent secret loyalties you prevent this route to checking the power of an abusive government yeah on the flip side you can say the same kind of story about measures like government oversight over labs um you know maybe this is helpful for reducing the risk of lab coups but it maybe also makes it more likely that government actors can abuse their position yeah so I think there's some there's some real worry there about you know shifting probability mass around um you also see this kind of element with other with other interventions though so you could imagine that some AI tools for improving epistemics are quite dual use and uh you know maybe you develop a tool which helps people coordinate and that also helps the people who want to stage a coup to coordinate um so you know it's there are worries like this at lots of different levels yeah yeah okay so what I'm hearing
Speaker 2off the like space of of of options for how to intervene it seems like quite a few of them end up in what I guess what I would call like a sort of like who watches the watchman kind of situation where like yeah we can maybe prevent like one group like AI companies from becoming disproportionately powerful um but we do that by like making um government or people who set standards or something like that more powerful and I guess the worry that I have there then is like how much yeah how much leverage do we like actually have to stop power getting like concentrated full stop versus just like shifting who ends up with the most power yeah I mean I think maybe if I were thinking about it in terms of
Speaker 1leverage it would make me think about the things that you were mentioning about the electorate and about uh manual work or maybe human wages will go up in the short term and so on like there there is there is a broad base of leverage in the current world where lots of people have a small amount of power and particularly if AI tools make it easy for lots of people to coordinate it might be the case that the majority becomes much more powerful um than it has been in the past and is much able to coordinate much more nimbly and quickly um so I think that's a good point I think that's a good point I think that's a good point I think that's a good point I think that's a good point I think that's a good point uh to push for its own interests and so I don't think it's hopeless um but I imagine that this is a bit how you know framing the U.S. constitution felt or something like well I want to balance this so I'll balance it with this but then like how do I keep that in check and yeah I don't know at some point you stop adding more checks and you think okay this is a balance system um yeah okay well so what if what
Speaker 2if we did end up in a situation where it's like there is just a trade-off that we really struggle between um like either we sort of like let companies do whatever they like or we kind of like let the government have like really like extreme control over more stuff than we yeah would have hoped they'd have um yeah like is that is there an answer for like who we should trust more like who we should like who we'd like pick in this like awkward trade-off I don't think there's a
Speaker 1I don't think there's a robust answer I think people have very different intuitions here so like yeah if you ask if you ask somebody who would you rather was dictator of the world the U.S. government or Google I think you will get some people who will say definitely the U.S. government and other people who will say definitely Google for different reasons uh where I think that the kind of Google camp is like imagining that governments are extremely inefficient and badly incentivized and Google would do a much more competent job of being world dictator than the U.S. government and I think the people who are favoring the U.S. government are more tracking like representation and uh there being checks on on the power of governments that are more robust
Speaker 2than the checks on companies pushing on I do want to talk a bit about just how easy or difficult it might be to prevent these kinds of like extreme power concentration scenarios from from happening um and one view here um is that it might just be like extremely hard to do something about this just given that like some of the most powerful people in society are going to want to stop you from yeah changing the situation because they benefit from it um given that do you like feel like we should have hope for solving these problems I do think we should have hope um there are a few reasons
Speaker 1for this one is a kind of meta observation that people object to working on power concentration both because they think it's too easy to solve and because they think it's too hard to solve and this I think is a reason to think that maybe it's somewhere in the middle um and so that's a kind of somewhat jokey reason yeah to think we should have hope um more seriously I think that uh we spent very little time so far trying to solve specifically extreme power concentration that's driven by AI and it seems like we should definitely be willing to put more time in it than we've already done to see if we can get stuff to work um I also think there are general background reasons for hope as you said lots of times most people don't want this yeah like it should be possible to get a very broad coalition of people who are on board with preventing this I think another more speculative reason for hope is just that we actually have spent a long time as a civilization thinking about um power checks and balances how you restrict power concentration to reasonable levels we haven't thought about that in the context of AI but we do have a lot of a lot of form thinking about relevant questions um maybe particularly as we start to get better AI automation that can help us to put these things together uh we might be able to think of some pretty good answers to this yeah okay that feels
Speaker 2quite hopeful to me um what are what are the interventions that you feel most hopeful about
Speaker 1yeah so I'm I'm at this stage I'm more confident in sort of high level objectives than in concrete interventions but I'll give some examples of both to show the sorts of things I'm excited about so one is building AI tools to help us with our epistemics um which I think as a kind of broad category I feel pretty pretty bullish on being helpful um some specific examples which I'm less confident in would be AI for fact-checking, AI for forecasting, kind of publicly available tools that help civil society and courts and other checks and balances to make sense and act more strategically in their own interests. Another bucket that I think is very promising is sort of AI-enabled checks and balances on the power of governments. So I basically think that in order to navigate this well, we're going to need governments to deploy AI. Otherwise, they're just going to fall behind companies and companies will be able to get what they want out of the situation. So we're going to need government deployment, but also government deployment comes with lots of risks of its own in terms of power concentration. And so there's some sort of how do we thread the needle here? What does safe government deployment look like that I think is very important and well worth people investing time on? There are a few concrete things here, which I think are quite promising. So one is law-following AI, which has been proposed by O'Keefe and others at the Centre for Law and AI. This is basically just training AI systems to follow the law, such that they can't, in fact, do things which violate the law. This seems like a really good, robust policy to have for government AI systems, and I would be very excited about people pushing on that agenda. Another example would be procurement rules for what kinds of procedures there are for companies that provide AI systems to governments and what safety checks need to go through. So that's another bucket. I'll just gesture towards... Some other things that I'm quite excited about in this regard. So one is transparency, making sure that the right actors have access to the right information about what's going on in AI companies and in governments. Sharing access to compute and capabilities to try to reduce the gradient between the most powerful and the less powerful to try and make sure that access is a bit more distributed. And, you know, there are specific actors that I'd especially like to empower with AI. So making sure that the legislative branch and the judiciary has access to really great AI systems so that it's not just the executive seems like a very good intervention to me, but also things like that investigative journalists have access to the best models and can be using them to help them find out about what's going on behind closed doors. Yeah, so I think I'll stop there. Those are the kind of main things that I'm most excited about with kind of varying degrees of concreteness to them. Yeah. Okay. Interesting. So yeah, I think,
Speaker 2something that you you said a moment ago, was that you kind of you kind of noticed how humanity has like, put a lot of effort into like thinking about power distribution and concentration and what to do about that. And, yeah, I do kind of wonder, like, we already have like, a fair amount of interest in stopping power being concentrated. And we're already doing like, various things to that end, like there are antitrust laws, there's like, taxation of the wealthiest, at least in some countries. Yeah, and like, lots of people seem pretty interested in like, avoiding inequality broadly avoiding misinformation. Yeah, and then you like list a few interventions for extreme power concentration scenarios that feel like very specialized and very different. to the stuff that we're already doing. And I think, like, I have some kind of curiosity about like, why think we need like, more targeted work? Like, why not just think that like, what we're already doing and already thinking about, like, we'll just scale up? Yeah, why like divert more
Speaker 1specialized effort in this direction? Yeah, good question. So I definitely think that the work that's going on today to stop non AI driven power concentration is relevant and helpful here. And so I don't want to dismiss that or say, Oh, it has nothing to do with the problem I'm concerned about. It definitely does relate. I still think it's extremely important to do additional things to target extreme power concentration. Mostly, this is because I think that the risk is pretty AI specific. You know, a lot of these risks arise only once AI is capable of automating a large fraction of economically valuable labor, whether that's power, power grabs by automating the military or government functions, or that's economic dominance. And so one thing this implies is that we're going to need to do things that we're going to need to have interventions that restrict AI systems themselves. And you're not going to get that via antitrust law or via regular investigative journalism. You need to think about it specifically. But I think another kind of broader thing is that I'm not really imagining that the way that we stop extreme power concentration is just by like, holding on tight to our current checks and balances. I imagine that we're going to need to have new ones that are designed specifically for an era where there's lots of AI labor that can be made loyal to individual people. And so I don't think the people who are doing great work on antitrust or, you know, great work, pushing for better equality measures in the current economy are naturally going to be thinking about this, and that their work is naturally going to tend towards kind of AI tailored checks and balances in future. To get a bit more concrete, there are some things I just don't expect will happen at all from modern times. I think it's important to think about it. I think it's important to think about it. I think it's important to think about it. I think there are other things where I think, oh, yeah, there's some stuff like this will happen. But I still think there's a big, big room for improvement, particularly when you're tracking these really extreme possibilities. So this is this is things like government AI deployment. Clearly, a lot of attention is already being paid to how AI is deployed in government. But I think a lot of it is local attention on how should we do this specific deployment, or it's more, more broad attention, but it's not tracking these very extreme risks that could arise down the line. And so I still think there's, there's room for us to make a big improvement on the status quo. By paying attention to this from the perspective of extreme power concentration.
Speaker 2Yeah, yeah, yeah, we could does it does seem like we could, we could do better. I think where I land with this. Yeah, so I want these problems to be discussed. And I want them to be addressed. I do still feel apprehensive about like encouraging lots of people to do work in this direction. And I think the main reason that I still feel worried, is that it just all seems quite like delicate and quite political. So I think it seems worth us kind of highlighting ways that your work could backfire or like what not to do if you want to work on power concentration. And I guess my first question is like, it seems like this could be an area of work where the things you publish could give people dangerous ideas. So for listeners who might want to work on this issue and contribute to the discourse about it. Yeah, how should they be thinking about that? What kinds of things should they avoid doing?
Speaker 1Yeah, great question. I think that I think this is a difficult one. I think to give a kind of crude example, you don't want to publish a kind of very detailed novel research handbook for how to do a coup using AI.
Speaker 2Oh, no, that's what I was planning on doing.
Speaker 1Yeah. And so some things that this suggests, you know, more abstract, less detail is good for public stuff. You want to be thinking about the specific actors that you're writing, threat modeling stuff about and how much they're likely to know. So for example, maybe it's not very likely that I as a non-technical person, I'm going to notice some clever technical route that an AI lab CEO could take. So maybe I don't need to worry very much about giving an AI lab CEO an idea. But maybe I would notice something that a head of state in a not very AI powered country would notice. Yeah. And so I maybe should be like more worried about that kind of information. But yeah, I think in general, if you think that you're thinking of novel ways to do things that are very harmful in this cause area or in other cause areas, you should be somewhat wary of publishing that.
Speaker 2Yeah. Okay. Yeah, that makes sense to me. I guess another worry that's like sort of related is not only like might you give people dangerous ideas, you might also like kind of rile them up in the first place. Like if you write an article where you're like these people are going to violently take over, like remove all of your like democratic rights, be like incredibly greedy and like ruin the future of humanity. I can imagine that feeling kind of alienating to certain groups and maybe making I guess the political situation a bit more polarized or something of people are being an alum in the category of like elites who are going to ruin the world. And like, you're a good average Joe or something. Yeah, I feel like that I think I have a worry that this might make might make the situation worse. What do you what do you make of that? And how how do you avoid it?
Speaker 1Yeah, I definitely think there are ways of writing about this that could make it worse. And throughout this conversation, we've been talking about the good world as worlds where this is this very broad coalition of people who don't want this to happen. And I think a risk with kind of fiery, non nuanced, uncareful work to try and reduce the risk of power concentration is that in fact you politicize the issue and you make it hard to create right coalitions like that because some people become like ideologically opposed to for example interventions that reduce the risk of power concentration because they see those as right wing or left wing or they see those as as coded in some way and I think this is this is a more subtle thing where it's significantly about what's the tone that you write with and which words do you choose and I think it it's good to get a lot of feedback on this sort of thing when when writing the problem profile that I wrote on extreme power concentration I initially wrote a problem profile on AI enabled coups right part of the reason I changed that was for this consideration it was also because I was starting to notice oh yeah I actually care more about these other dynamics besides coups as well but but a significant part of the decision was also this seems like a more spicy framing right more likely to tend towards political discussion and maybe it's better to have a kind of slightly more boring title like right that like people are less likely to fight about yeah so the advice is be less interesting I think don't lean into spice on this topic yeah boring is boring is good yeah boring is better
Speaker 2that makes yeah that makes sense to me and I guess like yeah the thing not to do is to just like go into working on this and being like I am just going to publish everything that I think is true like you've got to be more nuanced and like more careful with what you're doing here
Speaker 1yeah I think unfortunately unfortunately because it's a very political topic you do need to kind of think strategically about what you write and for some for some types of researcher and thinker that's that's not going to be a good fit because it's going to be important to their process to be able to be more free and so like I think this is suited to like particular temperaments this kind of work yeah pushing on
Speaker 2here um what do all of these concerns about power concentration um imply for government and for the government and for the government and for the government and for the governance models so like people often suggest models for AI governance like an AI version of the Manhattan Project where there's like some US-led effort to build AGI faster than its competitors um but there are other options like you could have some kind of international consortium um or you could like advocate for like widespread open sourcing um there's a whole bunch of things you could do um yeah do you have a sense of how good or bad these different models are from a
Speaker 1power concentration lens yeah so I think when you start thinking about power concentration models thinking about power concentration uh firstly a Manhattan-style project starts to look a lot worse um particularly if it only involves the US and there are no other countries to provide a check particularly if it only involves one AI company and there are no other developers to offer a check um and so I think yeah I'm I'm pretty concerned about Manhattan Project and think that it would probably be bad um I think it also has implications for other types of international project though so you can think about uh you know kind of properly international projects but I think it also has implications for other types of international project though so you can think about uh you know kind of properly international projects but I think it also has implications for other types of international projects though so you can think about uh you know kind of properly international projects but I think it also has implications for other types of international projects but I think it also has implications for other types of international projects though so you can think about uh you know kind of properly international project where there are lots of different properly international project where there are lots of different properly international project where there are lots of different companies coming uh sorry lots of different companies coming uh sorry lots of different companies coming uh sorry lots of different countries coming together um countries coming together um countries coming together um and maybe if you weren't thinking about and maybe if you weren't thinking about and maybe if you weren't thinking about power concentration and you were just power concentration and you were just power concentration and you were just thinking about representation thinking about representation thinking about representation this would seem like a really attractive this would seem like a really attractive this would seem like a really attractive option I think the power concentration option I think the power concentration option I think the power concentration doesn't necessarily make it a bad option doesn't necessarily make it a bad option doesn't necessarily make it a bad option but it does make it a riskier option but it does make it a riskier option but it does make it a riskier option right um where the concern would be right um where the concern would be right um where the concern would be uh if you bring all of this power uh if you bring all of this power uh if you bring all of this power together into together into together into one institutional body that body one institutional body that body one institutional body that body becomes extremely important for becomes extremely important for becomes extremely important for whether or not power is going to be whether or not power is going to be whether or not power is going to be checked in future checked in future checked in future and so then it becomes a question of and so then it becomes a question of and so then it becomes a question of like do you really back us to do like do you really back us to do like do you really back us to do institutional design from scratch in a institutional design from scratch in a institutional design from scratch in a way that's robust to the ai future way that's robust to the ai future way that's robust to the ai future and you know i worry about that um one and you know i worry about that um one and you know i worry about that um one thing i will say about international thing i will say about international thing i will say about international projects and power concentration though projects and power concentration though projects and power concentration though is that potentially you can limit a lot is that potentially you can limit a lot is that potentially you can limit a lot of the of the of the potential for harm by making sure that potential for harm by making sure that potential for harm by making sure that there are multiple ai developers as part there are multiple ai developers as part there are multiple ai developers as part of that project so for example me and my of that project so for example me and my of that project so for example me and my colleague colleague colleague will mccaskill are kind of pro a model will mccaskill are kind of pro a model will mccaskill are kind of pro a model um that's based on intel sat the um that's based on intel sat the um that's based on intel sat the organization that managed the global organization that managed the global organization that managed the global satellite satellite satellite network and there there's a kind of network and there there's a kind of network and there there's a kind of the model there is that there's an the model there is that there's an the model there is that there's an international body international body international body that governs the development and that governs the development and that governs the development and deployment of ai deployment of ai deployment of ai but the way that they do that is they but the way that they do that is they but the way that they do that is they can track things out to different can track things out to different can track things out to different private companies private companies private companies and so you still have multiple frontier and so you still have multiple frontier and so you still have multiple frontier companies companies companies who are similarly capable um i mean who are similarly capable um i mean who are similarly capable um i mean obviously this depends on there not obviously this depends on there not obviously this depends on there not being a massive intelligence explosion being a massive intelligence explosion being a massive intelligence explosion and if there is then you don't um and if there is then you don't um and if there is then you don't um but uh i think that potentially you can but uh i think that potentially you can but uh i think that potentially you can get a lot of get a lot of get a lot of you can limit a lot of the risk by having you can limit a lot of the risk by having you can limit a lot of the risk by having multiple companies involved multiple companies involved
Speaker 2multiple companies involved right and this uh the intel sat model right and this uh the intel sat model right and this uh the intel sat model that you that you that you referenced you actually have an article referenced you actually have an article referenced you actually have an article on that which we can also stick up a on that which we can also stick up a on that which we can also stick up a link to link to link to okay we are coming to a close um okay we are coming to a close um before we do before we do before we do um yeah we've we've talked a lot about um yeah we've we've talked a lot about um yeah we've we've talked a lot about the many different ways the future the many different ways the future the many different ways the future could go could go could go really wrong in like really horrible really wrong in like really horrible really wrong in like really horrible ways ways ways um do you have a vision for um do you have a vision for um do you have a vision for a flourishing future that you could share a flourishing future that you could share a flourishing future that you could share with our listeners with our listeners
Speaker 1with our listeners um i don't think i have a very concrete um i don't think i have a very concrete um i don't think i have a very concrete one but i can say some thing i can say one but i can say some thing i can say one but i can say some thing i can say some elements that i would be excited some elements that i would be excited some elements that i would be excited about in a flourishing future about in a flourishing future about in a flourishing future one thing that i am excited about is one thing that i am excited about is one thing that i am excited about is there still being humans and those there still being humans and those there still being humans and those humans being happy and human-like and humans being happy and human-like and humans being happy and human-like and getting to choose to remain human and getting to choose to remain human and getting to choose to remain human and still do things like have children make still do things like have children make still do things like have children make music uh write their own books even if music uh write their own books even if music uh write their own books even if ai's write better books ai's write better books ai's write better books um this sort of thing and so i'm good um this sort of thing and so i'm good um this sort of thing and so i'm good i'm fairly pro uh there should still be i'm fairly pro uh there should still be i'm fairly pro uh there should still be humans in the future humans in the future humans in the future um and i think others are maybe less so um and i think others are maybe less so um and i think others are maybe less so than me than me than me i'm also quite excited about there being i'm also quite excited about there being i'm also quite excited about there being something like diversity in the future i something like diversity in the future i something like diversity in the future i want there to be want there to be want there to be different pockets of the universe doing different pockets of the universe doing different pockets of the universe doing different things different things different things um and i'm quite frightened by um and i'm quite frightened by um and i'm quite frightened by well we should figure out what the best well we should figure out what the best well we should figure out what the best thing is and then we should use all thing is and then we should use all thing is and then we should use all resources to make the best thing resources to make the best thing resources to make the best thing style arguments um like i i'm opposed to style arguments um like i i'm opposed to style arguments um like i i'm opposed to that i want to that i want to that i want to i want to do something much softer than i want to do something much softer than i want to do something much softer than tell the universe for the best thing um tell the universe for the best thing um tell the universe for the best thing um because i'm because i don't think that because i'm because i don't think that because i'm because i don't think that that's very robust basically and because that's very robust basically and because that's very robust basically and because i'm because i like intrinsically value i'm because i like intrinsically value
Speaker 2i'm because i like intrinsically value things being diverse right yeah that things being diverse right yeah that things being diverse right yeah that makes sense makes sense makes sense okay um we've grilled rose today on okay um we've grilled rose today on okay um we've grilled rose today on extreme power concentration scenarios extreme power concentration scenarios extreme power concentration scenarios whether they're plausible and so on um whether they're plausible and so on um whether they're plausible and so on um but yeah we haven't gone into like the but yeah we haven't gone into like the but yeah we haven't gone into like the full details of the case that she full details of the case that she full details of the case that she actually makes in the article um i'd actually makes in the article um i'd actually makes in the article um i'd really encourage people to read that um really encourage people to read that um really encourage people to read that um you can read it on our website um or you can read it on our website um or you can read it on our website um or listen to it in our narration feed listen to it in our narration feed listen to it in our narration feed rose you've been so great thank you so rose you've been so great thank you so rose you've been so great thank you so much for coming much for coming
Speaker 1much for coming i really enjoyed it thanks for thanks i really enjoyed it thanks for thanks i really enjoyed it thanks for thanks for having me as a funny for having me as a funny
PodcastTranscript AI