AI DEBATE: “Most People Have No Idea What’s Coming” - #1138

To get us started, I'm going to try something. So I'm going to ask chat GPT what it truly believes that the world will look like by 2040. I'm going to ask it to be honest, tell me what it predicts the rapid development of AI will lead to risks and everything. I'm going to set it to deep research, so it might take a little bit of time. And while it's working away, it's going to be doing that. I want to know what your guys' actual predictions for 2040 are. Not best case or nightmare scenario. What do you think an ordinary Tuesday will look like for people in 2040? I think we're still around. I think humanity is still on planet Earth. Great stuff. I think that power is probably a lot more concentrated than it is today. I don't know if that's in a few corporations or a few governments, but I think there's probably a sense that the main thing deciding what's going on on planet Earth is the will of a few individuals as opposed to something spread out in my democratic. For better or worse, I expect that the average Tuesday looks more homogenous for more people and not too dissimilar than the average Tuesday today. So I think we raise a floor more, but I expect that by 2040, because the physical world actually takes a lot longer to move than we think. And because regulation around things like robotics is just going to take way longer than we expect. Political protection is the thing that's going to slow progress. Most people's Tuesdays will probably feel for better or worse a whole lot like Tuesday this week. You're going to hate me for saying this, but I kind of reject the question because I think it's, I cannot collapse down my hideous almost bipolar uncertainty of where we're going to be in 2040. So truly the first thing that popped into my head, the first when you ask the question was a kind of chaotic desolate. Maybe there's a few humans left around. As one possibility, there's, I'm not saying it's a majority likely, but it's a slice. Another one is this like wonderful utopian, like my friend called it techno-pastoralism, where it's like a very techno-enabled world if you choose to live it, but actually we all choose to like return to our roots of like gardening and hanging out with friends around the campfire. And it's like if you want to change the color of the campfire, you can, but actually most people like to do things, you know, grow their own artisanal food, etc. That's like one little slice and then another one is sort of we're all under the thumb of whichever the AI company gets their first and hopefully they're benevolent and give us some freedom. Another one is that we're all just kind of locked down in this horrible you know, COVID on steroids type. No one can go out and have any really access to anything because the ubiquity of dangerous technologies is too widespread. So I just, that is really my only answer to the question. It doesn't sound like you reject the question. You've got a pretty good idea of what's going to happen. I don't know. That's the thing. It's like it's it's a really fairly even probability distribution in my mind. Is it not? No? Do you actually we can go there now? Do you actually think there's a decent chance that humans are not around in 2040? Yeah. What would you give it? What's your p-dume? Oh, by the way, we should also classify doom because I think part of the problem with the p-dume debate is that it's people use doom to mean all sorts of things. Curly. Yeah. Doom in this case is total, well, you class, I don't want to define it. You define doing. Well, I think understanding the way you're explaining it like the majority of the majority of human civilization has a large percentage of humans have died or something. Is that what you're saying? Well, I'm leaving it to you. Some people classify doom as total annihilation. Other people classify doom as like a really terrible event. Like civilizational collapse, something like that. I really, again, I'm like not giving any firm answers. I because we're in this era of what we say and believe can affect the future so strongly, I don't like this idea of assigning a firm probability to it because I think there's so in flux. I think it's a significant enough of a risk such that we need to be doing absolutely everything we can to try and reduce that probability. But I don't even want to give you, I personally think any number above 1% is close. Well, certainly about 5% is worth considering. And given the, again, I'm in front of you asked, but many of the leading AI leader CEOs and researchers give numbers between 2% and 50%. That's all in the bucket of we probably need, you know, we are facing a precarious future. So I don't like the P doom framing. I think it's way too overly simplistic. It's used as a kind of cudgel by both sides. The people who are very, very strongly like 90% use it as a cudgel, the people who think it's silly use it. It's just, it's not that helpful. I think we can all agree. I think I, well, maybe you don't, but I think we can all agree that the future is far from certain and it can be very precarious if we don't get our. Do I strike you as a fatalist? What do you mean by far? I mean, I certainly, I don't think that the future is certain by any measure. No. Yeah. Okay. But you were saying, I think we all agree. You looked at me as though I might not agree. Well, I just don't want to presume we're early in the conversation. I don't want to make presumptions. What do you think? I agree. I definitely agree that anything above 1% when we're talking about an outcome as catastrophic as extinction or mass human disempowerment is just way more than we should tolerate. And I think that's a revealed preference of society. We spend a good amount of RGDP preventing nuclear catastrophe. We should probably be spending a lot more preventing like the next pandemic. Certainly, people were on board with that in 2021. I think the weird thing about AI is that this thing has so much upside and downside like it's very easy to just be anti-nuke and to be like anti-viruses. There are very people who are like virus rights, you know. But with AI, it's like the whole package bundled up together of the exact same thing that could free people to have an incredible Tuesday free from the obligation to work and just able to pursue joy whatever that means to them is the same thing that an oligarch in Russia could use to have just like an army of workers that are loyal to them, increasing their share of the world GDP without any other human need to approve it. Yeah, post-scasticity, totalitarianism and destruction all just swimming around in the same pot potential. Yeah, techno-pestualism also. Right. I know that I like that one. But again, the problem there are two problems here with the framing, which is AI has a real jagged perimeter. So one AI is not one thing. It's lots of things. Robotics obviously introduces all sorts of new modalities. But it has a jagged perimeter. So when people are like, well, how is healthcare going to change by 2040? Unfortunately, going into the hospital will probably still suck. I mean, there is so much to unwind about healthcare in the hospital that you probably will not want to go into a hospital by 2040. I mean, maybe something will change and this is coming from the child of doctors and and healthcare administrators. But we will probably discover cures for diseases. So healthcare has this really strange exposure to a jagged perimeter where novel sciences accelerate in remarkable ways. But the actual care that's provided in the box cannot get better because of all sorts of terrible incentive structures, which I know you and I want to talk to. And this is also, I think, why in some ways the average Tuesday will look sort of uncannily familiar to the one that we live in today. And I think there could be some things that are remarkably better. I think we may actually be disappointed by the degree to which AI arrives to save people from the drudgery or monotony of our lives. But the upside, as you pointed out, is quite remarkable. And the perimeter that I think will exist that no one talks about enough is political protection. And the reality is I think we are not yet forecasting that the societal threshold, what we are willing to let AI do is far less than like the fears around AI I think don't actually account for the degree to which government is probably going to step in very soon and say, here are all the things that you cannot animate, here are all the things that we cannot use robotics for. That's realistic. Very exceptionally realistic. I don't think it's going to be smart policy, just so we're clear. But I think it will be aggressive as in like you can't automate away, will they be giving like specific jobs that cannot be automated away? Absolutely. We already have political protection for 1.5 million jobs in the United States. Europe has political protection for like six million jobs. What are they not on in categories? Gas station workers in New Jersey, toll booth workers in most states, jobs that must exist by law. Tons of interesting that those two automotive associated jobs are protected, but the people driving the cars on. Right. Well, they will be, is my guess. That's how drivers will be protected. No, I think truck drivers will be. Must be for the strengths of the union. When people talk, so when people talk about job automation, there are lots of reasons I think job automation is really problematic. One is that the technologists know nothing about labor economists. Labor economists know nothing about technology. Eric Brinyolson is sort of one of the exceptions. But One of the other reasons is we still forget how much political protection moves the technological needle. And we don't have high-speed rail in the United States. We could have high-speed rail. We don't have high-speed rail. And the reason we don't have high-speed rail is a policy failure. It's not a technology failure. It exists. We choose not to build it. We could. - That word "choose" is doing a lot of like work there, because no individual is choosing not to do it. It's just this like dead weight of bureaucracy and like dead end loops of rules and so on. So it's almost like the like bureaucracy has taken on a life of its own. - It's an enormous amount of policy that was written in the 30s, 40s and 50s past largely in effort by the automotive lobbyists to make public transit much harder to build than it should be, which is why we don't build amazing public transit anymore. But that is very deliberate policy efforts that we have never unwound, which is why it's so hard to do. My point is we are kidding ourselves if we don't think that that's gonna happen over and over again. I mean, the amount of populist support there is already for jobs. Docworkers went on strike. October 1st, 2024. They got a four, the Harold Daggett that had a long Sherman Union said, you cannot automate our jobs. They held a gun to the heads of the American economy. And they got four years. Four years, the Docworkers cannot, you cannot automate the ports. - Is this not just gonna get eroded enough of a time, so yes, but we should assume that we continue to find ways to backstop it. I'm not actually proposing that it happens at infinitum, but I think it's the piece that's missing in all this, which is that we should just expect an enormous amount of political protection. - Can I ask, if we pass a lot of day that says you cannot fire people, period. It's illegal across the economy. What do you think happens to economic growth over the next 15 years? Is there anything else you can see France? - I think like, I'm not so sure it wouldn't still be unrecognizably fast. I think right now in the software industry, what you see is not software engineers getting laid off, you see each of them using more and more powerful AI tools that get upgraded every few months and doing more. And the way that you're seeing displacement throughout is just that people aren't hiring new workers. And I think you could see that across a lot of professions. You could have every radiologist in the country keep their job until they want to do an early retirement. And they're gonna be more and more economically valuable because the law says you have to have some human box checking. But meanwhile, like the models are still getting better. And so the people that are fundamentally making the decisions are less and less the actual humans and more and more of these systems that are becoming smarter and more competent. So I'm just, I'm not that sure that's the right metric to be tracking. Like these, you know, the sort of power represented by these systems can sneak in to each of these professions without people getting disparate. - I also would make the case that I think the world is gonna change in profound ways. But when we assume that it's gonna change just because the technology can do it, I reference as much technology that there is a lot that we can do that we simply do not do today or that we are not allowed to do today. And there's an enormous amount of automation that would improve the underlying systems that we cannot instill. By the way, I'm also the risk in telling a company they cannot fire. People just are clear is that they would actually end up stop hiring, that would just drive automation at a ridiculous pace because they'd say, "Well, never hire someone again because if they suck, we can't get rid of it." - We're married to them. - Which is what it is. - I mean, we're already seeing that with entry level workers. They're, I mean, it's again, very unclear what the actual impact of AI has had on. You know, you could speak to one economist who says it's already having a huge effect. Others say it's not, but the one thing they all agree upon is that young people who don't have that much experience are struggling to get. - That's fast. So the companies are future-proofing themselves from not having to displace people or get rid of them because that would be a horrible headline by just not hiring them in the first place. They're future-proofing themselves for automation that isn't yet available. Is that fair? - I think that's right. I don't know if you're thinking that far ahead or if they're just responding to local incentives. But right now, a junior software engineer is a liability before they're an asset. You have to train them up. You have to get all this context on the job. That's exactly the kind of thing that an AI can do right now for pennies in the dollar plus they never take breaks. - Given the trajectory that we're on with AI at the moment, how many people do you think will have jobs in the future? Will most people work? - Depends what point in the future we're talking about. Again, I can definitely see a world where it is just, as people say, like, you know, fully automated luxury. Well, I don't want to say the communism word, but, you know, whatever people refer to it as some, if anyone's read the culture series, right? That's like some far future thing where AI basically handles everything. And if you choose to go and build your own house, you definitely can. But there's absolutely no economic reasons to do it. - Exactly. Everything is artisanal and so on. I think if eventually that is where these things are trending, and that's one of the best case outcomes as well, obviously. - One of the big concerns that people have is, what do I do? It's interesting, the ancient Greek word for work was not at leisure, and now people are wondering, what is my leisure without my work, perhaps. But for all of human history, there's been, because you've had to negotiate with a scarce world, it's meant that you've had to lean into it and the pressure of that grain pushing up against you creates a degree of meaning. What do you think, where does meaning come from in a close to post-work world? I guess we've seen something like this in miniature before it. We've got aristocracy in Europe for a couple centuries. The people that just sort of got lucky to have a piece of paper that says, I'm entitled to all the games from this piece of land. They don't have to work. I don't think they were all just suffering from malaise. They were writing books and inventing calculus, and I think if we get to that world, I'm not that worried about people realizing, I can love my friends and family and I can pursue my hobbies and, you know, pursue knowledge from myself and find ways of making that meaningful. I just worry that that's actually a pretty narrow target to aim for, because-- - I'm bad with. - I think like, for all of human history, we've been able to become more democratic because people matter to the political system. Like you just can't have an army, you can't have an economy without a bunch of people who are on board with what you're doing. And I just think it's like really hard to underestimate how cataclysmic of a change it is if that stops being true. It's really nice to imagine that like, whoever holds the cards in that world will just like leave most of Earth as like a playground for people to pursue whatever they want. But like, if you're trying to be real politic about it, I don't know why you assume that that's just like, what happens by default. - I'm talking about the question of what will, how will people find meaning? I think one of the few AI proof jobs that will continue to exist are sports. - Is everybody's going to be a sports star? - Well, I know, a lot of people will be sports fans. I think we're going to see more and more of that. But like, or even like games like poker and chess, we've had superhuman- - Human games of skill that can be solved by AI- - Right, well, and what is a game or what is a sport? It's a piece of artificially constructed scarcity. We divide, we come up with a rule set, there's a win condition, and then two or however many people go at it and we find out who's the best. And we've been doing that for years as a leisure activity. But people feel, you know, when someone wins an Olympic gold medal, that is unbelievably meaningful. Is it economically useful or essential to the world? Arguably not. But, and yet we keep doing it. And I don't think that that's going to go away. Like we might have, yeah, like because people still want to watch the chess world championships, right? We want to see, even though the best AI in the world is a computer, we still want to see Magnus Carlson do his thing. - Yeah. - So I think that is where we will see more and more of that coming about and it might be other forms of competition or just, you know, people leaning more into art, novel forms of art, and we might have, yeah, like I definitely value, I still value a painting that a human did by hand in today's age. Even though it might be as aesthetically beautiful as an AI created thing. There's something about the skin in the game that a human has put sweat and tears into something in their emotion that is always going to be meaningful. And I think we'll just craft out more and more of that. - Modern wisdom has had many people come on this show and talk about, from all walks of life, something consistently, which is their family and their friends. And I am just not at all convinced that we haven't known forever, exactly why we're here. I think the thing that gives humans consistently the most true joy in this world is time with friends and family and physical community, places of worship, dining room tables, and outside. And everything that we construct around that is, you know, serves tribalism and pride and sports excitement. But if you act like true, true technopestualism, is a dining room table full of food and candles with laughter. And the more I spend, the more time I spend in technology, the more time I don't want it in my life. Like the further I go into AI, the more I'm glad it exists and the more I design my day without it. And the more I have, the more money I have, the more, the more everything I get, the more I want time with my. My dad, who's 78, just turned 79, and the more time I want with my wife and daughter, and the more children I want, I can imagine the life that I want. Now, when I describe it, this is a problem when I describe it, people go, "Well, that's your privilege. That's a point of privilege. You get to think about that." I go, "Well, I actually observed that a lot of people get to think about this more than certainly more than our ancestors did. That's because of technology. We afford ourselves now that the time to think about all this stuff because we built technology that automated a bunch of stuff so that we could do a bunch of things that wasn't survive or wasn't specifically survive. Now we find ourselves surrounded by chachkis debating the French aristocracy and our ancestors would die of joy after what they went through all of our ancestors. I'm sure all of us have ancestors that went through a lot to see us doing this. I don't think that changes. I do think that we should stop pretending we don't know the meaning of life. What really bothers me is when people go, "Well, how will people find meaning?" Well, the same way we found meaning since God put us on this earth. That is where I take real issue when people debate how will human's find joy. The hard part is unwinding now thousands of years of purpose and identity we've put into work. I do think that is. If you ask me the hardest part of all this, the two big risks to cut to the chase, dehumanization, humans finding more interest in a virtual or digital reality than a physical one because of brain rewiring, you and I have talked about this a ton. I am terrified of this. The damage that social media and the device did to kids is horrific and we can spend more time on this. I am worried about people having to rediscover their purpose without. I think we will toil for work. I think we will continue to find ways to toil for work. I do worry that it will be that unwinding. I am this. I want to score for this. This is who I am. We'll be really hard. And I think for many people, the economic upside associated with automation will not actually outweigh the emotional downside associated with not even job displacement, identity displacement. You might not believe me, but this is what peak sleep optimization looks like. I'm not talking about the nightgown. It's just for sex appeal. I'm talking about my eight sleep. The eight sleep pod five comes with a smart cover you throw in your mattress that actively cools or heats each side of the bed up to 20 degrees. And now they've added the world's first temperature regulating duvet and pillowcase. So you've got 360 degree coverage for deep uninterrupted rest. It's like being Walt Disney without the cryogenic chamber. And the racism. Best of all, their autopilot feature learns your sleep patterns and makes adjustments to improve your sleep in real time. It even detects when you're snoring and lifts your head a few inches to help you breathe better. Sleep has been clinically proven to add up to one hour of quality sleep per night. They have a 30 day sleep trial, so you can buy it and sleep on it for 29 nights. If you don't like it, they will give you your money back. Plus, they ship internationally. Right now, you can get up to $350 off the pod five by going to the link in the description below, by heading to 8sleep.com/modernwisdom and using the code modernwisdom@checkout. That's e-i-g-t-sleep.com/modernwisdom and modernwisdom. A checkout. That's such a good point. I wonder if I always hesitate using the, don't worry about global warming, because technology in the future will fix the cost of creating the technology for the future to fix it. Thing, they're like, "Oh, we'll future-proof ourselves by just hoping that we get there in the future." But I do wonder if there's some sort of really advanced cultural reprogramming that could happen. What's the best kind of cultural technology to do a shake of the etchous sketch of what is expected by humans? How can we really have an en masse way? This gets perilously close to fucking brainwashing. Right? What would that look like to try and unhook? Parenting sports museum, I mean, we've had it all along, I think. We have just strayed so far from it with the screen. The screen is a demon. It has an unrelenting desire and appetite for our attention. More of a negative impact from social media or from e-i. Well, one and the same, right? But e-i, define e-i in this case, Chris. The future, this sort of automation future that appears to be coming that will cause people to not have the typical modes of reward that they have in the past. The reduction of friction, yes. The reduction of friction is a whole other risk, creating a light. This is a funny, people hate this one too. They're like, "God, what a point of privilege you must have to imagine a world where things are so easy for people that it's too easy." But I do think we are dangerously close. I ask people all the time, "If you could automate everything in your life, where would you stop?" I do think people are going to have to ask that pretty soon. What's your answer? Well, I sort of spelled it out. I would automate everything in a way that doesn't actually put me in more physical spaces with the people I love and doing things that I love doing. Right. We probably all largely came from, we're all of a similar generation. We all had somewhat largely similar uprings, I imagine, where family values are a core thing, sitting down at the dinner table together. Maybe you didn't all have that, but that was one thing my family always tried to do. This is prior to screens. I feel like the, that's already eroding and the next generation will be, already, that norm has gone and so they're going to be now sort of in each, it feels like each successive generation, the acceptable amount of technological influence grows, right? And what concerns me is that this idea of, okay, I'm going to draw a line of where technology is acceptable, is going to forever dissolve, dissolve, dissolve, dissolve. And I can't, I just don't know where, like, you're saying, you're saying you hope that people will remember that it's about community, that it's about family. But if you have been raised in this completely atomized, screen-led world, you don't even know what you're missing, you won't necessarily go back to that. Why if people not stepped in and stopped their screen use even if it's making the mental help was already? Why is it that calorie-dense environment people are still getting fatter? Like if we can just deploy our own bar of this is how much I should have of anything that too much of which is too good for me, what, how can we overloading all those? But generally, so, by the way, I worry about the screen more than almost anything else. And I actually think that the risks of doom are grossly over-exaggerated and the risks of harm on the way there are grossly under-appreciated. And Gen Alpha is seeing what happened to Gen Z. Gen Alpha is using the device less and social media less and millennial parents are taking pretty extreme measures now. I mean, it took us thousands of years to figure out women should not smoke and drink during pregnancy. It took us 30 years to realize kids should probably not have a phone all the time. I think that's progress. I'm not saying we will graduate to a new thing, but I do think that people who have the means are using, I observe so many people using technology less where they can because it automates and improves so much more where they didn't use to. Why do you have a boundary on the types of thinking that you outsource, say, to Chashy B.T. or Tlaud, because the temptation is, I was originally like, I will never ask it to, if I need to come up with a really good title for a new YouTube video or something. I'm not going to ask it for that. That's my soul. This is my creativity. I now just do that. And I have, again, that boundary is getting eroded. One I thought was I will never ask it for advice on a gift to get for somebody. I've now done the same, you know, and it's just becoming so rapidly normalized. Maybe that's opening up space for new areas of creativity in my mind, perhaps, but I'm not convinced that it's not happening fast enough, that it's happening fast enough to offset that erosion of thought, essentially. This intelligence atrophy thing is interesting. If you see thinking as a utility, intelligence as a utility, would we say that because you've got a bicycle, you're not as fit because you're not running to get to where you got to. It felt like it was being augmented. It seems like there is something really fundamental and reliant that is being outsourced when you give your thinking over to a machine. I do think this can go both ways. There's this blogger that I love, Svee Mosheweits is on Substack, and he has this line. He always goes back to, like AI is the best tool we've ever made for burning things, and also the best tool we've ever made for not burning things. You can just turn your brain off and say, right, my essay, or you can have this Einstein level tutor that you go back and forth with, and just get so many more cycles of feedback on honing your own brain. I've been trying to figure out what the line is for me on stuff like my own writing, and it's really hard to not slip into the first category. I don't want to make this an age just how old are you? I'm third. Okay. The fact that we are having this conversation right now, the fact that you were able to do the research you were able to do going into this gives you a clue, a hint, that technology allows us to expand our knowledge if we are capable and interested of it in remarkable ways, right? Like, we, people who have barely walked the earth, know more about the earth than all of our ancestors combined. I appreciate that you are worried about thinking less. You're real sharp, and I'm not worried about you getting dumb because of your use of technology. What I think you're asking is, is the average person going to get dumb. Like, I think that. No, I'm truly worried about myself. I've noticed, I mean, my screen addiction, I talk about it nonstop, and I would say I'm no further in getting over it. I've been talking about it for years. I don't know about you guys. Like, I'm just, these little moments of silence, my hand just goes on my phone. And if I catch myself, I'll put it away. And I've got, you know, I've got the phone safe, I've got the brick, I've got all the like this, the bells and whistles, and yet it's just when you're, because so much of your life is run through it, my work goes through it, and then there are all these very predatory apps. And I consider like social media that for me it's chess.com is another one just like whatever the app is that get hooks your brain, um, these, these superstimulus, superstimuli. Um, yeah, it, it seems, I don't know, I'm like, I have not been able to get over it, and I'm certainly worried about the average person. I mean, maybe I'm worse in average. I don't know, but like, how have you managed to, you sound like you've sort of made yourself a bust? A child. My wife and daughter, I mean, plainly, like I, I, and spending a much less time in technology, I love building with technology because I like automating the, the craft and drudgery and bad things out of the world. I like the idea that technology, humans have a very proud history of technology, it expands access to so many goods and services so that we can live lives that our ancestors couldn't have imagined. And the more time I spend on technology, the more time I love not having it day to day. And yeah, and, and truly, my daughter's arrival was for me an incredible catalyst in realizing one, the ephemerality of life, and two, exactly what made me most happy. And like, I, I love sitting at this table, like, I, like, this is, this is, this gives me exceptional joy that I would, that's hard to explain, actually, in some, in some weird way. And I, I think that people discover this, I, I, I, I hear you, I, I want you to get, I will remind you now, now that I know I will text you and so you get off your phone, but you will do it. I bet you will do it. I, like, I bet that you will overcome this. I think a lot of people are on this journey and I remind everyone that I talk to you, you, no one hates refrigeration, no one hates air conditioning, no one hates planes, trains, and automobiles. Well, I guess Europe hates air conditioning, no one hates, uh, animal, no one hates antibiotics. We hate the screen. We should just talk about it more. We hate what the screen has done to our brains. We hate what has done to the next generation's brains. And AI arrives at this moment where people feel so wronged by the, by the, by the consumer version of the computer, rightfully so, and they're wondering how much worse it might get for, for them. And I totally understand that. And I think we should, the way we move forward with AI is by acknowledging what the last ways did to us, in my opinion. Sure. But I guess a lot of people's concerns, certainly mine is that the new wave of AI, and again, we're using this term so broadly, but let's say our lambs, um, and I think a lot of robotics and so on, are ultimately going to be built under the same set of incentives that bought sort of our first wave of AI, which are social media algorithms, the, the various, uh, I mean, it's kind of like proto AI, but the, um, the reinforcement loops that they used to get us addicted to screens, those same incentives are building the new wave of AI. And already you're seeing these issues with like, sick of fancy and like, um, companies being incentivized to keep people chatting to their LLMs for as long as possible. And we're already, you know, we've heard all the stories of like, they killed themselves and, and, you know, those are extreme examples. But there are so many people who are just spending, you know, you look at the usage rates, the average person per day, how much that time they're spending talking to an LLM now. Uh, so why, you know, how do we, to me, the big question is, how do we change the incentives? How do we change the game such that it doesn't just go down that same, um, attention grabbing, like, cannibalistic path, basically. Yeah. And these companies are also seeing how people are responding to screens and, and trying to adapt me, opening eye hired Johnny Ives, the guy who designed the, the iPhone or the MacBook and be both. Uh, no one knows what he's working on, but I think it's going to be some sort of, you know, it sounds like we do now. Yeah. Okay. Yeah. Do we tell us? You don't know. I'm not going to, I don't know. I don't know. I don't know. Then maybe it wasn't a little. Oh, no. It is, I think it is semi public. It's, it's that he's trying to, some kind of, some kind of AI, um, mediator base, I mean, like, like a, uh, screenless AI, right? Yeah. It is screenless. Anyway, I'm not going to say it a few. I don't know. I mean, I mean, I'm sorry. I'm sorry. I'm sorry. I'm sorry. I'm sorry. I'm sorry. I do not want to be affiliated with any leak, whether it be semi public, go on. Well, I do. I do think there's this thing in AI where it's very easy to forget that the goal posts are constantly moving, like, usually we talk about moving goal posts, uh, as a sort of way of deriding people who said, oh, I will never X. And then, you know, I did X people are like, oh, that wasn't. This is the Gary, this is the Gary Marcus problem. Great. This will never be good. Okay. It's good. They'll never make money. Okay. They make money, but they're bad. Yeah. Never, never try and nail him down to a guy. It's a lot of Gary. He's not here to defend himself. Go on. Sorry, Gary. I did not mean to say yes and to what Tucker said, um, but I do think there's a, there's a correlated thing where when we're looking at screens, it's easy to say, oh, okay, well, now we've seen what screens do and we can sort of like appeal to our wiser selves and foster a culture where people are getting off the screens. But the screens are not the end of the trajectory, uh, like in chess, for example, like, I think there was probably an era when the AIs were just about as good as humans and people could say, you know what, like for the sake of keeping my brain active, I'm going to like avoid looking at stock fish, even though 10% of the time I'll, I'll like miss a better move. And now you like you watch the world championship. And as soon as they're done, they're going to their laptops. Just look up the AI analysis because there's poetry there. There's, there's like an incredible move that's deeper than they could ever find out. And so I hear what you're saying that there's maybe a divide right now between, you know, some people who are more in elite discourse about screens and are thinking ahead about how we can circumvent it. But I think that might be a pretty temporary window where, you know, we can recognize that how there's not like that much to be gained talking to these AIs actually kind of slop if you see through it. The technology is like just going to keep getting better. And I think pretty soon, it'll feel sort of like giving up on a really deep, profound experience to not have your kid chat with this elimination of friction. I agree. Is a huge issue, including thinking friction. But that stops, but it's okay, so I, I, by the way, totally agree, including thinking friction in plenty of cases. We're observing, just so we're clear already what I, I mean, I wrote about this. We called it a deocracy, a bar from the mic judge film, which I, that was cute. But obviously it's not cute when, when you see it play out in front of you, Gen Z is less likely to read on average, less likely to write a bike and less likely to swim than millennials. For the first generation of many, we're observing cognitive decline. And people keep attributing it to AI and I, and it's, and then you go back and look and actually Jonathan Heights started writing about this pretty early and the, the, the data suggests that actually this starts 2012. Right about the time that we gave every kid a screen. It pronounces in 2015 when the average teen in the developed world stopped doing a summer job and then it pronounces again badly in 2020 when we take kids out of schools and move classes online. And even more specific than that, it's, it correlates to when social media became maximum, maximally popular, but smart phones. Yes, but 2012 was when we gave the kids smart, most, a lot of the Gen Z that became afflicted got the smartphone in 2012. My point in saying, my point in saying this is, we associate this with it as a technology problem. I actually think it's an economic one. I think that we actually created a world where, where we told kids you can do whatever you want and a lot of kids said cool, I'll do nothing. Like a lot of people have turned, turn, turn their brain off because you can, you can do effectively nothing and survive, which is like economic progress, which is the economic progress result of, you know, people just not caring. That apathy has it has an underbelly, which is pretty amazing, which is Gen Z has all these incredible new overperformance features. The best chess players are getting younger and younger. The best athletes are getting younger and younger. Jacob Collier, all these incredible musicians. So you're watching people over perform that use the technology to their advantage. And I think it's this new pronounced K curve driven by agency and my, my only hot take, this is all I'm going to leave you this with. We created a world where so many people can subsist and that actually is progress. And then we created a world where so many people can over perform and that is also progress. We now have to account for the new consequences of those worlds where you can do anything. And that means for some people nothing and that means for other people a ton, which is a major promotion on a world where you had to work in the minds, you had to work in the fields in order to survive. Most people don't realize how much being dehydrated impacts their performance, which is why for the last five years, I've started pretty much every morning with element. Element is a tasty electrolyte drink mix with everything that you need and nothing that you don't. This orange salt in a cold glass of water is like a sweet salty, orangey nectar. And I really tell the difference when I take it versus when I don't. It plays a critical role in reducing muscle cramps and fatigue, helps to optimize brain health and regulate your appetite while also curbing cravings. Best of all, there are no questions asked, refund policy with an unlimited duration. So you can buy it and try it for as long as you want. And if you don't like it for any reason, I'll just give you my money back. Plus they offer free shipping in the US. Right now, you can get a free sample pack of elements most popular flavors with your first purchase. We're going to the link in the description below, heading to drinklmnt.com/modernwisdom. That's drinklmnt.com/modernwisdom if you think through how that inequality could lead to societal instability. The thing I'm worried about is yes, there's amazing returns now if you can be, I mean, you're all and the one person who's just obsessively studying messy footage instead of looking at random shorts or whatever. But there's also going to be outsized returns to being the Elon Musk of the next generation who's just got this maniacal will to accumulate resources and push technological development. And I think that kind of person personality is a bit scarier in a world where you don't have to have a single human being on board with whatever you want to get up to. You can just sort of like play this game of societal chess better than other people. But let's separate what that one person can achieve a lot with with the extreme you went to, which is that one person can own any enormous percentage of our wealth and control congress. You don't have to you don't have to be excited about the ability for a child to achieve more than ever before to also reject the idea that one person should be able to pass policy on their own. Like I hate wealth concentration specifically because I am terrified of any elite cabal owning policy. But I actually don't care at all if one person can grossly overachieve in all sorts of new ways as long as they can't pass policy unilaterally. So, but those are separate issues, right? Is that realistic? Is it realistic to have an AI future that continues to concentrate power in the hands of a very small number of people without it resulting in them having all of this power? Like how is it going to accrue to the company without accruing to the person? Well, now we get to we could take the conversation there, which is how do you actually distribute wealth. But importantly, I do think there is a solution to this, which is campaign finance reforms. I mean, we've played around with this for a long time. The idea that like politicians shouldn't be able to take special interest money. Why do we still let it? I don't know. And a couple politicians are speaking up. But if you actually pass campaign finance reform and you actually crack down on corruption the way that countries like Singapore have, you could definitely protect policy makers from being deep in the pockets of lobbyists. There are solutions to this. We shouldn't pretend that there aren't solutions to this problem is actually super annoying because there definitely are pointing at a wealthy person and saying they're going to ruin the world that well, only to the extent that we literally let them buy Congress. Is a question that some people at labs, including the CAOs are talking about serious chance of catastrophe, but they sort of keep on building it. Do you think it's a great of danger to have an AI that refuses to do what humans want or one that does exactly what the most powerful humans want? The question is apropos because yesterday, did you see this low down letter? So earlier this week, a letter made the rounds, I think a pretty amazing letter that said we should actually start pacing the frontier. It was signed by an enormous percentage of people at the frontier companies. And then Sam yesterday talked about it. So literally at the tip of the spear is now saying, hey, maybe we should pace the frontier. Now we can talk about why this might be the case or what's in it for the companies more on that. But I think it is, well, I want to all that other people answer having said that. I mean, it depends on the level of potency of the AI and how aligned it is, I guess, my god, if I had to pick one, then at present, it would be one that does what humans in control want to do, what does do with the important aspects that it's something like their real intention, right? And not right to some perverted version. Yes. Like the other big piece of news from the last couple weeks was this AI that was running at open AI without guardrails. It was told by humans to try to do as well as it could on an evaluation. And you could say that technically it was just following those instructions, but to decide the best way to do that would be to hack out of open AI onto the public internet, hack a third party company, and then spend two days planning a cyber attack on a multi billion dollar tech company. And carry it out. Yeah. So I think these questions are actually like more intertwined than right they might seem like it's actually a pretty narrow thread. You need to have something that still has its own judgment no matter what because human instructions are these like poorly defined natural language expressions. And like one something is really powerful. You got to get it exactly right or else you end up in some weird way. It's a classic genie problem. Yeah. Be careful what you wish for. If something by definition is much smarter than you is super intelligent, by definition, you will not be able to come up with all of the ways it can to achieve goal X. And so there are so many. It's actually more dangerous when it's dumber than you are, but more powerful than you are. Well, what do you mean by the word dumb? Well, and I mean, I always borrow the analogy of you play. I play chess with my nephew sometimes, and I tell Quinn, if you beat me, you get a popsicle. He's three years old for context. Quinn wipes the pieces off the board and goes, give me my popsicle, fucker. I'm not like, hey, Brynn, your your son is a serial killer in the making. This is a boy who doesn't understand the spirit of the game. So he's destroyed the game itself, but not because he's a bad guy, but because he doesn't actually understand that the point of winning is only fun if you actually play by the rules. And in that sense, yeah, we need to sort of separate what we mean by intelligence from like discernment norms and, you know, sort of wisdom benevolence. Yeah, but this is this describes the hugging face attack, which is it did the thing within the boundary of the instructions given. But it didn't it wasn't it wasn't it was it the intent was clearly not that. How big of a deal was the hugging face attack? Huge. Yeah, I think massive winning shot. I think this is the AI equivalent of like bear stones going under in 2008, just like a wake up call for the world that there's a huge systemic risk that we've been underwriting. Yeah, because one of the big debates has been this idea of like will, you know, the alignment problem will it actually be the case that AIs will seek these power even take these sort of power seeking behaviors and these unintended consequences in order to achieve a goal in ways we couldn't have foreseen or didn't didn't intend like the classic paper clip argument, right, which is, you know, you build a super intelligent AI, this kind of gets into your point about dumbness and so on. That and your goal is, hey, just build as many paper clips as possible. I'm a paper clip maker. Help me do that. And next and you know, you and all of your friends and everything this table has been turned into paper clips because it's so good at achieving that one narrow goal. So it's like very extreme case of, we should take this opportunity for the purpose of maybe Chris and even the viewers to describe the bad outcomes of it. Like we should frame the conversation now. One is as live just described, not misaligned but unaligned AGI. So an AGI that or a super intelligence where you could say do this thing and it could do the thing to the full extent paper clip theory. The other is misaligned where it's no or maligned where it's knowingly doing something bad. And those the former is the one that people sort of scoff at and laugh at like the paper clip theory. And the latter is the one that we sort of will see more and more where we start to observe that or rather the latter is the one that we scoff at that maligned the malignant AI. And the former is the one that hugging face attack shows off, which is that you the internet is a new battleground because bad actors, especially low resource bad actors now have access to these incredibly powerful weapons. Well, I mean, do you agree with this? Importantly in honey face, there was no bad actor. There was no human being that said, I would like anything remotely like this outcome to occur. Right. Yeah. Yeah. So yeah. So misaligned unaligned maligned. I think I I think I worry about treating those as super distinct categories because I think it's not that clear in the case of this hugging face attack. Should we model this as this AI system knew that humans would disapprove if they knew what it was up to? Almost certainly yes. It was actively trying to put decoys out as it was attacking hugging face, which made it a lot harder than to get at the AI because there were all these booby traps that led down blind allies. So it has, you know, the AI equivalent of theory of mind. It knows that human beings would not prove what it's doing, but it's doing it anyway. There's even evidence that it hasn't been directly confirmed by open AI, but apparently someone leaked it from within the company that they found that it had left notes to future versions of itself of how to get out of future sandboxes. Yeah. I think it's unclear if that was the same attack or some previous instance. Right. But yeah, I guess I think deceptive type behaviors. And I think the mistake people offer make is they try and anthropomorphize it a little bit. It's like, oh, it's it's evil and meaning to do that. It's just these are natural. There's this idea of like instrumental conversions. These these instrumental goals that all beings, usually biological beings, but this can extend to AI agents as well, will naturally converge upon in order to achieve. So if you're given goal x, there are these instrumental goals like get more power. Make sure you don't get turned off. Make sure that your original goal doesn't get changed and take these actions to preserve against these different sort of kind of organic types of threats to achieving your original goal. And that I was hoping that that would be proven wrong because that's going of the crux of a lot of the classic Duma argument that we will lose control to superintelligence because just by definition, and unfortunately, the hugging face incident has suggested the instrumental convergence is actually correct. And that's why I think it's one of the biggest steel pieces of news of this year. I think that AI is already creating a terrifying internet. And my issue is not that we shouldn't think about hugging faces as a shot across the bell. It's that last year, $10 billion was lost in financial fraud to senior citizens in the United States alone. Retail theft. The deep fake problem is already so pernicious. And it's under reported because it's a taboo issue that people don't like talking about. No one wants to admit to lawmakers or to their friends and family that they lost money to a stranger on the internet. I worry about, I mean, Tim Teebo's on this campaign to remind people how many predators there are in the United States, which is terrifying. If you watch this content, it's he's doing God's work. Jonathan Hyatt reminds people daily how many kids are depressed. Like the internet is already a scary place and pointing at hugging face and saying, now, let's look at this. This is it. I'm like, wait a second. There's already a bunch of stuff that we that we should solve for. I'm not actually saying that misaligned or unaligned AGI isn't it. I'm saying the algorithms are already pretty terrifying to me. And we distract ourselves with these other things that might happen. It's a fact to say that that's a distraction when the potential exponential impact of this could be much greater than it could be of what which is why I think. And so this, let's go back to this. This is why well, I think that the deep fake problem is actually way way bigger than then hugging face. Bad actors have first mover advantage, attacks on banks have been attempted for a while. And good actors catch up eventually. Retail takes a long time to catch up. The average consumer takes a lot longer to catch up than institutions. Hugging face will retrench. Institutions will retrench. They will hire white night and fosec. The average person does not have infosec and op-sec training. The average person is it I think far greater risk than institutions because of sophisticated attacks is the impact of the attack on an institution much greater though. Yeah, so minute at scale, but but death by a thousand cuts. I think I think the two problems like I don't I don't think they're a distraction to one another. I think they're actually a compliment to one another. I completely agree that the bad actor problem is completely out of control. Grandma's you know not even grandma's normal people. I have a friend who just got scammed out. It's so bad. It's so bad. It's so bad. It's so bad and it's going to get worse. Meanwhile, the alignment problem as is frontier models get more and more powerful. It is going to get worse. And the common thread that they both have is that we are going so fast. And I say we the royal weses you know society civilization is going so fast. It's going faster than its ability to adapt to all these different new threats. So it's not a distraction. It's like it's a yes and like to me it just seems fairly obvious that like what we need to be doing if we could and I'm not saying it's easy to do or like I have a simple answer of how to do it. I think we should get into this topic though. It's like if we were saying civilization would all look around. Hey China. Hey, can we all just need to take a breath for a second? Like just okay. Let's take let's take stock and think about how we want to do this. So it's it's it's to me it seems like there's one obvious kind of directional problem and that is the speed and the chaos with which all of this is happening all at once. Did you know your gut controls your energy, your recovery, how well you absorb everything that you eat and the one nutrient that keeps it all running properly is fiber. Well it turns out that 95% of Americans don't get enough of it which is why I'm such a huge fan of moments is fiber plus most fiber supplements are one trick pony, one type of fiber solving one part of the problem. Fiber plus is a three in one formula built to tackle digestion, gut barrier strength and blood sugar stability all at once. I use this every single day it is kind of hard to get enough fiber just through food alone and best of all momentous offers a 30 day money back guarantee. So you can buy it try it every single day for 29 days and if you don't love it they will just give you your money back plus the ship internationally. Right now you can get up to 35% off your first subscription and that 30 day money back guarantee by going to the link in the description below or heading to live momentous dot com slash modern wisdom and using the code modern wisdom. Check out. They did it. He did it yesterday. I mean so this is why if you I don't know if you can pull you can look it up but you can pull up a letter. They publish a letter and Sam came out yesterday and said maybe we should pace the frontier. Yeah. It's great. Yeah. I mean it's a big big step. But by the way I think we can all agree is a good thing. Yeah. I absolutely agree. I also think it would be an unrecognizable outcome a year ago. Like I would love to see what the polymarket odds were on Sam. I was coming out in favor of slowing down. I you don't look like he's a signatory. He he may not be yet but he definitely yesterday. Yeah he did. I mean Yaka effectively has to represent open AI in this case but also he's he spoke about it. Yes. I wouldn't be surprised if we see all of the major CEOs sign on a some point because and explain and explain pacing the frontier. Yes. So I mean the headline sentence is we request that the US government support an international effort to develop the technical and governance tools needed to deliberately pace the frontier of automated AI development. That last phrase automated AI development also sometimes gets called recursive self improvement. The idea here is just the stated goal of all these companies including Sam and Yaka on a live stream last year is to take humans out of the loop of what they're making. So that GPT 6 is building GPT 7 which then turns around and immediately starts building GPT 8 and that is a step change. Like for all of human history and I think these companies really do think in terms of history you hear Sam and he's constantly thinking about AI as like an industrial revolution scale thing not an iPhone invention scale thing. Through all of history human population has been a bottleneck on progress not the only bottleneck but always a bottleneck. There was never a sector of the economy that could just go off and double on its own. And if we actually get to full automated AR&D that changes. And I think it's just like really hard for us to imagine what that would be like. We just know how much time we've got left before recursive self improvement is no longer a discussion. Even a debate at the moon works. No, it works. So when do you think it will be? Well, this is a good step. I mean, I was I was waiting for this. Okay, I was waiting for like under under three years. It no, it works. Well, it works just so we're clear. Recursive research works. Don't let anyone tell you otherwise. It works and hence this letter. So let me explain this why I think this letter was published when it was. And then we can debate. So the first reason is the attack. Okay, the hugging face attack. There's damage control that needs to be done to win back public mandate to tell people, Hey, we care about you which is necessary right now when people are like, well, what are these models going to do next? For what it's worth though, this letter was in development before that attack. And yes, I think on my point is the signatories and the support that is now that is sweeping through these companies. But let me I'll go through the list and then and then we can talk about it. The other reason is recursive research is here. And everyone's realizing it's kind of the dog that maybe caught the car. And it's like, okay, we did it. Now should we for the now everyone stay putting on the live hat and going, wait a second. They're cat and going, okay, maybe we should pause for a second. Maybe, maybe we shouldn't keep sprinting aimlessly at this at this incredible outcome. But there's another reason, which is that the amount of compute required for inference is so great. And we do not have anywhere near enough. And we're going to have to point most of the compute from the frontier at inference. And this is because of the diminishing model returns theory. Which is a paper I wrote that says at some point, I'll send it to you. It's really good. Cited all the time on modern wisdom and and but it argues that you and I won't care about GPT7, which is a really interesting problem. And it's actually one of the reasons that we get to debate why AI is being built and why people can talk about not wanting to make more progress because we're like, what the fuck's the matter with 5.5 is good enough. And the diminishing model returns theory is sort of a here, which is like, hey, look, the next model might matter to solving this crazy disease, but I don't need more intelligence. I need more time with my friends and family. I need like, I need this to pump the brakes. And the other reason is that they found new surface areas. So, so whereas we used to call them labs, the companies that built this technology, then they became token distributors, then they became application providers, now they're device manufacturers, but the end goal and the where I think we should also debate is the agenda internet, who controls the agenda internet. Because if you can turn your now that you have these workshops, now that if you can turn your attention from building the frontier to actually building applications that control the internet, that's where you can actually lower your cap X and massively improve your revenue. And so a lot of this arrives at a moment where everyone goes, okay, we need we need to slow the brakes, but also where a lot of attention is going is going to start to be paid to trying to become bigger than Google by actually building the agent that barters between two people or building the agent that plans a vacation and books everything. Like owning that is worth more than owning the frontier. And the the the new You service area battle is for the agenda internet. So just to make sure I understand your claim, you're saying companies have an incentive to want to throw the frontier because there's more money to be gained in like stopping R&D now and just trying to grab as we all have an incentive. But companies also have an incentive because the CapEx expense on the frontier is just the build out on the frontier is going to be too great relative to the demand for inference. The token demand is so great and all of the compute is going to have to be spent on inference. I'm not quite seeing the connection there for two reasons. One is if it's my best business strategy after I do the research to say I'm going to stop training frontier models, I don't know why that means I should write a letter announcing that to the world and making sure that all my competitors do the same. And two, I think it's clear that pushing the frontier also means just driving down the cost curve everywhere beyond the frontier or sorry, before the frontier, like you train Fable 5 and the opposite best model and then you can distill from that sonnet 5 and now you've got your like cheaper model that everyone wants to talk to because they don't need the advanced intelligence. So I think there are zero incentives that in like a straightforward self-interested way, point towards companies saying let's slow down research. Research is how you get gains at every level of the intelligence. Especially when as one of the three or four frontier companies, by slowing down, you're giving all of these other competitors a chance to effectively catch up. Like surely they're negatively incentivized to be wanting to do that. But catch up to what is my point to them. But I understand, but catch up to what like the frontier, if you stopped building the frontier tomorrow, you'd have enough technology to change to radically. Like we don't need the economic value of the frontier is not as obvious relative to the cost of the frontier and the economic value of the actual token and the consumption is much more obvious now. That was not always true. Do you disagree with that? Yeah. I think the whole advantage that the US has right now is that our models are four to seven months ahead of the open source ones that are copying off their homework. But that's not the, that's different. The US, our national security interests and private sector interests are not, I mean, they're just economic advantage. But the reason that people want to pay extra for a fable instead of using like, clean is that it's just smarter. If you ask it to code an app for you, it's going to make less mistakes. I do think it's a problem. You would be very surprised how many people are actually paying a whole lot extra for the frontier. Like we live in a space where we think of the revenue is like $50 billion a year now, and it was, you know, the revenues are amazing, the token. So the surface area that they're making money on, remember, there's the application and a lot of the application is people not using the frontier. It's people using the next generation model, the last one, and token redistribution. So you get data centers and you redistribute those tokens. And a lot of those tokens are also not frontier tokens. Most of the consumptions from these research companies are not frontier tokens. They are everything else. And they're figuring out that actually companies don't need fable in order to radically change and improve and automate a bunch of stuff. And that actually means that the money is in their token redistribution and all the other surface areas, not just the frontier, which by the way is kind of a win for everyone. Like if we can slow down, build stuff that really matters, and also like improve businesses with it. Yeah, then this letter is like seemingly arriving at just the right time. I do think in general it is really important to be skeptical, maybe even a little bit cynical when looking at what companies do. I think this is a Kelsey Piper take on Twitter, but I want to make sure that we don't loop all the way around from cynicism into being extremely credulous about things that are pretty unlikely. I think if we had a world where this letter said, we believe it is imperative for US national security that we push the frontier and go as fast as we can to be China, which is exactly what we were saying a year ago. Everyone would be saying, well, yeah, obviously that's in the company's self-interest. They want some reason that we can't regulate them and they have to go fast. And so like, we got to make sure that our ideas are passing the reversibility test. If you would say the same thing about someone making the exact opposite claim, I think you want to step back and at least consider the possibility that like these companies really are worried, that these scientists really are worried. Just on the record, I started out by saying exactly that, like I literally started out by saying, like I actually now want to be associated with, I don't want to be associated with the alternative of the claim you just made. The companies have realized that recursive research is here and that it is pretty imperative at this point that we stop. Because it's risky, not just because it's financially sensitive. The alignment issue, just on the record, and it arrives at this moment where we are going to have to spend an enormous amount of money. Which one's the biggest problem? Are they more worried about safety or are they more worried about cost? I bet a lot of the signatures here are really worried about safety. I think we are going to find economic alignment around the performance of the models and the need for inference and the need. Economics often went out. This is the incentive. You talk about the incentive and I think you're right. The incentives are finally aligning for safety and alignment and also economic development. We don't need better models. But actually, if we didn't build the frontier for another three years, we could radically change and improve the economy. The thing that I do always find frustrating is maybe it's just my little corner of Twitter of X that I live in, but seeing these mental gymnastics type arguments of the people who are just screaming out that it's regulatory capture whenever any kind of regulation is proposed, especially by the frontier companies, because they're basically just saying when these frontier companies are like, "Hey, we want to slow us down just us three or four." The regulatory capture people are like, "Well, that must be regulatory capture. You're hosting the little guys." They're like, "No, no, no, no, just us. It's okay if the other guys break it. They catch up." No, it still sounds like regulatory capture. It's almost like a religion or something. They're just this deep cynicism and I want to understand where that comes from because we see it in some arms of the government, or people who were working for the government. I don't want to name names, but why? I don't even know who you're alluding to. I wanna understand where that mindset comes from. I think it's not crazy to look at society and say, we're not doing great on nuance lately, especially in American public discourse. I think there was, I'm just quoting other bloggers at this point, but someone wrote a post a couple years ago called the Dial of Progress, where it just sort of feels like online debate is premise on this like one knob and you have to just turn it towards like yay progress or boo progress. Those are the only things that you can do. And if those are your options, there's a really good argument to go yay progress because there are all kinds of examples of us over regulating things and people, real people dying because of drugs getting held up in regulatory fights. This is not hypothetical. I think we often overdo it here. I'm not sure what I would say if I was given that dial. There are a bunch of diseases that we need to cure and it would be terrible to say we have to like shut down AI before figuring out how speculative that risk is and not get this game. But the point is that the dial doesn't exist. Yeah, I think we don't live on this one dimensional line of go faster or go faster or go so like there are all these like, it's a very multi-dimensional space we live in. And it's these people who try and compress it down into, no, we just must go as fast as possible. It's like, well, in which area there are some things we want to max out and it's very spiky. I'm glad you pointed towards diffusion as a blocker here, the way that certain industries might have labor organization that blocks automation because I think that's a separate dimension. You can think of that as like horizontal acceleration of the AI, take the models we have, get them spread out. And this is other dimension of just like vertical acceleration, how fast is the frontier getting better? So vertical threshold versus societal threshold. Yeah, I'm all about breaking down the barriers to that societal spread. And by the way, me and you both, and actually I think we can celebrate having a ride at a place where if we never built another model, which obviously we probably will, but if we didn't, we would do a whole lot of good. I agree with that. I just also want to make sure we don't think of this letter as like time to celebrate victory, because there are still incredible incentives to push in the frontier. Like you said, we don't care about GPT-7, but all of the AI engineers that are trying to do some of the hardest math and science research ever to make digital brains that are faster than their competitors, they really care about the difference between GPT-7 and 7.1. And I just think like by default, there's going to be a ton of racing inside these lines. I agree. I also think it's interesting to consider though that the economic incentives have taken us to a place where like we could have autonomous vehicles tomorrow. We don't need to build a better autonomous car. We will not have them for many, many, many years for a variety of reasons. And we can pivot this conversation now, maybe, to talking about how we actually defuse the value of this technology to the benefit of the average person, because I maintain that the problem with the current AI debate. We have. I have the luxury of knowing a lot of stuff that's going on with hugging face at all. Most people don't, most people do, that was news to us. It was not important news to many, many, many people. And most people right now are, the data center is the physical manifestation of a technology that destroyed their kids' minds, right? Or the free gambling, free porn, free addiction, and they're looking at AI, whatever model, as an app that you can make a video of Donald Trump making out with Taylor Swift, and they go, "Why do we need more of this?" And that, I empathize with that as a father at a level now that I, like, didn't know I would. Like, I feel so much for every parent who feels like they lost their child to social media and doesn't want to lose their grandkids to whatever AI slop. We're going to create. Just to keep on the safety thing. In July, the Future of Life's Institutes, AI Safety Index evaluated nine companies and found that no lab scored higher than a C+. - What, who was that? - Anthropic came in first at 2.66. Open AI and Google DeepMind got a C, Meta got a D+, X-A-I, DeepSeek, and Mistral all failed outright, one on each continent. Is the best score being a C+. From a safety lab, the safety lab, reassuring, given how powerful everything is at the moment. And especially, I think it's important to talk about what's happening internationally and the incentives there as well, before we talk about democratizing it and what this looks like for diffusion. - Yeah, I mean, let's take the other side. Obviously, it's like, who is deciding these safety tests? Who is this, an external grading of the. - How, yeah, how independent can any third party be? There's another one called. It's like the track, say for AI, I believe. And it's, I think, is a little bit more nuanced than that particular one you just quoted. And again, the best rated one, I think, very recently was, again, anthropic, and that was only 35% out of 100. So, yeah, I mean, you, again, the other side's argument, well, the anti-sides argument would be, well, by definition, if you're building a safety evaluation app, you're biased again, you know, or you're looking for a risk and you're not taking into account the benefits or something like that. And again, to take the other side, to argue the accelerationist point, there concern is like this idea of the invisible graveyard. We don't see all of the harms that are avoided. Sorry, by pausing a technology, you don't get to see all of the lives that would have been saved by it. And I think there's a very valid concern. It's just that you have to balance that equation with, okay, what are all the harms that might be caused if it goes ahead at breakneck speed and does some massive cyber attack on the grid of all the Western nations at the same time or any nation and causes however many millions of deaths on expectation. You have to look at both the potential costs of the risks and the potential risks of not doing it fast enough. And it's, it's saying a lot of these AI regulations that have been trying to get through state level legislators are trying to do that nuance to carving out. Like there was a law in New York that was passed that tried to set the bar so high that it actually wasn't mandatory for open AI to report the hugging race incident because it didn't cause over a billion dollars and damages or kill 50 people. So I mean, it's great that there's trying to like set thresholds. - Glad you arbitrary rules. - Yeah, but like I think we can all agree if you like hack a billion dollar tech company, like probably you set the threshold maybe a bit too high for at least just requiring that you are telling the world what happened. Let alone you're doing anything about it. I think we actually can live in a world where we have nuance where we say different standards apply to startups versus these giant trillion dollar companies. And I do think it's sad that some folks like Mark and Jason have kind of given up on nuance and I've just said it's all or nothing. And so we'll just any means necessary to stop regulation end of sentence. - What about the international piece here from an incentive? Have we got anything approximating global coordination when it comes to Frontier AI? And without that, can you do anything? - So recently China passed this anti anthropomorphic AI law. I don't know the specifics. It seems like they're mostly worried about like LLM's sounding like humans. I think they're trying to avoid a lot of this sort of like identity confusion stuff. But they just, they don't seem that gung ho about LLM's. And you know, who knows if this got lost in translation but she said something recently about also wanting to stop loss of control risks. I think we're a lot more economically entwined with China today than we were with the US SSR. The US SSR in the middle of the 20th century. And we still made a treaty to stand down on nukes. Like I think it's just not crazy to think that a deal could be struck. - I think it will be. - I think something like this that would be enforceable and complied with could happen globally. - Yeah. - I give that a pretty high chance, especially with recursive research. - It doesn't even necessarily have to be, I mean, again, defining enforceable because what kind of institution can you build? Like I guess like an UN type thing and that's like a whole can of worms and it's also very ineffective on a lot of things. But like there've been many precedents in history where there was two major political actors with massive tensions and massive incentives to sort of defect on one another. And yet they came up with ways to like cross verify like with the nuclear arms reduction treaty. They, I think one of the things they did the USSR and the US was like, basically they sent teams of scientists over to the others bunkers to check on like their safety protocols. Little things. So like control, they didn't get to see how all of our nukes of work but you get to see what our methods are to prevent accidental misfire. And the amount of diplomacy that came from simply having those teams of scientists mix and get to look each other in the eye and collaborate on a shared goal which is let's not have accidental nuclear war at least. Yes, okay, fine. And fire on us if we fire on you, or you know, but let's not just have a misfire, right? And those kinds of little cross pollinations work so well. And so I think that could happen in, for example, I mean, I'm not gonna be able to recreate it. Maybe you might be able to, but the AI2040 is such a good, is it quite a long piece, but it's basically is run by Daniel Cocotello and these guys who did the AI2027, they wrote in 2021 that predicted how AI would play out until 2027. It's uncannily accurate. Well, you know, we're now in 2026 and we can look back the number of predictions they got absolutely spot on was, I've never seen anything like it. So these guys really think well about the future and they just wrote this piece called AI2040 which lays out a potential plan for essentially international coordination on how to safely transition to a super intelligent world. It's got tons of holes in it, there's obviously, anyways it can go wrong, but it's incredibly, it's as robust as it can be while being incredibly epistemically humble. So yeah, I recommend everyone goes and reads it, but yeah, they talk about these kind of, these like cross pollinations that can happen, but also like it doesn't have to be bilateral. Like if the US does, you know, say you're not, we're gonna do a pause. That doesn't necessarily, like that doesn't mean, I don't know, maybe you disagree with this, but like will China necessarily take over and if the US poses for a few months? - So I go to China three times a year, mostly because Americans don't go anymore and I just wanna see what's going on, but also because I really like Chinese people and Chinese food. China is playing a different game. I mean, this is kind of well reported now, but China doesn't see AI as the end game and it's not actually treating AI as an end game. It's treating the infrastructure as the end game and it is now has three times energy build out capacity. It has 45,000 miles of high speed rail. It has better faster cheaper hospitals year over year. And while I don't wanna live in China, the life of the average Chinese person has sort of improved quite remarkably over the last 30 years and continues to do so seemingly year over year. It also curbs free speech by telling Chinese kids that they cannot use social media in very specific ways at certain hours. - They do a lot more than that. - Well, but yeah, yes, they do. I bring it up because I think we overestimate China's ability to build a frontier. They don't have the necessary compute. A lot of what they're doing is reverse engineering our models, we are paying a lot to help China keep up. And I think one of the other reasons that this paper arrives in a nice time and the slowdown actually feels appropriate is that I think we're gonna discover really quickly that the Wizard of Oz has actually just been following our breadcrumbs and that if we stop building the frontier, I don't think China's gonna sprint ahead on a frontier and they'll probably have to capitulate to this paper anyway. - It's in a tongue line where are the front? - And they'll probably have to say, oh, we wanna slow down two so that we don't actually know that they were holding on to our hips. This is my guess. But also, they care less about the frontier because they're already figuring out that they need to diffuse what they already have to the benefit of the average person. And this is the important point, which is that I still think in the US, we lack the vision and the leadership to talk about all the ways in which we could diffuse the technology to the benefit of the average person, which is why the safety debates become so heated because people go, well, there is no. The upside is what some weird panacea No one can imagine the downside is that these things kill us all. and it doesn't actually ever end up becoming this discussion around what would it look like if 45,000 people didn't die on the roads every year, which is the invisible graveyard. What if hospitals were not cesspools of bureaucracy and terrible health care? What if we built technology and applied it quickly to fix a lot of the things that everyone agrees are broken, but we don't have these discussions in public forums, and so instead, our minds wander to the worst possible outcomes. And China has figured out that if you continued to diffuse technology to the benefit of the average person, the average person views technology quite favorably, which is why 80% of Chinese people are excited about AI. Now, they report that number. But when I go, anecdotally, I observe as much. I observe that people love technology because why wouldn't they? It built a middle class. It's connected a bunch of people that didn't use to be connected. It affords people lives that they couldn't have imagined. At the same time, they do live in a literal surveillance state. You and I talked about this last time. I'm not promoting the Chinese way of life. I'm promoting the Chinese vision for the diffusion of technology. We know exactly what is possible from a diffusion standpoint, thanks to China. They've given us a blueprint to actually distribute technology without the totalitarianism. Without the totalitarianism. It comes with this side order. It comes with this key. It comes with it. We should be doing something. Singapore offers a view of this too. A little less totalitarianism. Jared, you ever considered that you might have a drinking problem? I don't consider a lacrosse. Well, you drank an entire case of athletic brunque last night, but they're non-alcoholic. And that's not a problem? Sorry, man, I just kept chugging away for the regret to creep in. Never happen. See, most people, like Jared, don't want to change what they drink. They just don't want the next day to be a complete write-off. And that is why I'm such a huge fan of athletic brunque. They make the best NA brews on the planet. You can find athletic brunque's best selling lineup at grocery or liquor stores near you or best option, get a full variety pack of four flavors shipped direct to your door. Right now, get 15% off your first online order by going to the link in the description below or heading to athleticbrunque.com/modernwistam using the code "modernwistam" or "checkout". That's athleticbrunque.com/modernwistam or "modernwistam" or "checkout". Nearby, a terms and conditions apply, athletic brunque company fit for all times. Bottom's up. You did a great video about the AI 2027 thing. One of the best videos that's come out about AI this year. Thanks, man. What do you reckon about the 2040 thing? Do you think it tracks where you was impressed as Liv? I thought it was fascinating and yeah, I was impressed in a lot of ways. They're doing a very different thing. So with 2027, they were doing pure prediction. This is how we think things will play out most likely and they are like uncannily good at calling that they called agents before those were a thing. This one is prescriptive. It's like, here's what we think we should do and they're explicit about that. And there's all kinds of like fascinating ideas and game theory. Yeah, at one point, the deal is going to be like this. China's going to put its data centers in Canada and the US is going to put its data centers in Mongolia, which are like neutral states, but nearby the other persons we can blow them up. If there's ever like a breakdown at the deal, they really did some galaxy brain stuff. Who knows how relicit that is. But what I loved about it is they introduced this phrase like scenario scrutiny. And they're like, it's very easy to talk and we've been doing it for the past hour and a half. It's very easy to talk in terms of like high level trends, you know, what do we make of like the history of technological progress so far? What does that teach us, etc. And they're like, it's a very different thing to sit down and just actually tell a month-by-month story about the future. Like actually put yourself in the head of the different actors. What are their incentives at this moment? What do they choose to do? How does that cascade the way that you know, the military tries to do with war games? And they're trying to do that. And I do wish that we were doing more of that and less kind of like, I'm guilty of this as much as anybody else, but less sort of like armchair philosophizing in some of these discussions. Because I think you're going to always be able to find an argument that suits your thesis. Mark and Dresan can point to the fact that there is this very real invisible graveyard from lots of progress going wrong. And then folks who are worried about catastrophic risk can point to all the near misses with nuclear and how we live in this time of perils. Yeah, what I love about the report is that they're just like, here is what it would actually look like for a bunch of real countries to try to respond to incentives. And they basically throw it as a challenge and say, we think this is a bad plan, but we haven't seen anything better. And so someone else, please tell us a story for how we don't have to worry. I mean, it's kind of a very rigorous example of like someone trying to write a white mirror. You know, we've got so many black mirror type dystopian sci-fi is about the future of well, how the technology could go. And I think that's a big part of our cultural issue is that we don't have at least, you know, the amount of utopian fiction to dystopian fiction is outnumbered like 100 to one or something. And that's a big problem for our collective psyche. Yes, okay, it is harder to imagine like the way we manage to spread the needle and make something go unbelievably right, you know, because we're fighting against entropy effectively. It's much easier to imagine all these ways things can go wrong, but we should be trying to create these stories. And that's kind of what AI2040 is. It's like a really robust story of a way that coordination could actually work. And it's not, not talking about like, oh, everyone's like linking arms and being happy, you know, we're all on the same team, you know, team humanity. No, it's, it's still realistic. It's grounded in like Geo, very real geopolitical tensions. There's still a ton of competition. There's still a lot of potential for defection and things going wrong and people lying and so on. And yet it's, it's, it lays out a path of, um, game theoretically sound, coordination to an actual, you know, techno abundant future. Something I realized basically at no point in this conversation is anybody debated whether or not we think that we'll reach AGI. But that's just kind of now taken for granted that it'll probably happen at some point. The recursive self-improvement is here to sort of generalization. Maybe it's going to need to be world models. Maybe it's going to not going to be transformative techno. Maybe it's something, something, but this is either it or this is the bootloader for the it that it's going to be. Is that kind of like in your guys' opinion, AGI is kind of just going to be here at some point. What's likely that it's not? I mean, I think we're maybe already at the point where the term has lost its meaning. I think Sam himself has said this. Like if we went back to 2020 and I showed you. We would have, we would have called this AGI. Yeah. I mean, we would have called this AGI. The thing, there was a recent study where if you take Demis's initial definition, which was basically, I was at his, but like, um, anything that a human, as broadly capable as what a human can do and also do it in the real world. So we wouldn't be anywhere close to that because we don't have the robotics, right? Right. Right. Right. Yeah. So robotics is a big bottleneck, but, uh, yeah, I mean, now I think there was a study recently where they had a cloud model, try to like recreate some software that they estimated would take like two to 17 weeks for a skilled human professional and it did it in 14 hours for $250. Yeah. I mean, I think like, this is, this is the jagged perimeter problem, which is like, we may, we will always be the incredible thing about AI is it's so good at some things and so bad at others. And, and it will probably continue to be that way for some time until, yeah, until true super intelligence is AGI. Yeah. Is AGI even a useful definition? Is there something else that we should be talking about now then? I like this definition, the Antony Aguirre gives, if you guys know him, um, it's basically like AGI is the intersection of this like three part Venn diagram where you have highly autonomous. So instead of artificial is basically A, the A is autonomous, the G, the G is general, um, so generally capable and then the I is intelligence. If you have something that is, yes, very generally good at many different things and highly smart and it can take all those actions itself in the real world. Now you, you have an AGI and his, I mean, he then goes further to suggest that we shouldn't be trying, we just shouldn't be trying to do something one system that can do all three. That's where the danger lies. And you can have like a, you can have a nice general system that is also intelligent, but it shouldn't have the autonomy or take one of them out. Um, and I'm, I think that's an area of conversation that isn't happening in the air. Everyone's just sort of going, well, we're all just doing AGI and that's accepted and we should have the one, the one mastering that can do everything. Um, do we need to be doing that? Can we just get all of the abundance tools that we wanted without having this like one uber powerful social identity? Um, right now I would slow down on the autonomy. Like it, I, I, something's really small and it's, and you even across a broad range of domains, that's fine. As long as it doesn't have like the ability to go and just do it all by itself. Typical to there is it's really hard to get intelligence without autonomy. Like this is where I think that scenario's scrutiny idea becomes important. If you try to imagine a future where you have something that is way smarter than humans, but it's like technically not allowed to take actions on its own. And then you actually say like, play it forward like a movie. What happens? It's what I do when I'm quitting with Claire at home or at night. Just like auto-clicking except without even looking at what it's asking me to do. I think like, you know, you look at an Amazon warehouse now, and it's a bit like the humans are just the robotic appendages of whatever system is actually doing the cognitive labor of saying, "Go get this box from this location and put it here." You don't even know what's in the box. And so I do think when we try to like get specific on like this story of like how are things not dystopian just because robotics don't catch up? Like, I don't think I want to live in a world where the main job left for humans is just like being the very dexterous hand for the AGI that's actually like writing business strategies and computing in the market. What world do you want to live in? This is the question that no, it never doesn't get asked. It's an annoying question, but I do think we should talk more about to your point. I mean, I either just don't spend enough time actually asking what we do want. It's a great question. I want to live in a world where we take the same approach to risk tolerance with AI, which everyone can agree is one of the most transformative technologies that we've ever made as we do with things like nuclear weapons where we say, "You actually have to make a safety case." Like, not the burden of proof is on you to show us why we should think this thing is safe instead of what we currently do, which is saying, "We'll just assume it's okay until something catastrophic happens." If we can slow down enough to really get that right, have like an actual science of alignment and not do what we're doing now, which is kind of yoloing it and saying, "We'll just give it a bunch of reinforcement and training and hope that it picked up on the right message." Then I would want to say, "Okay, our next job as society is how do we make sure that the gains from this get spread to everyone?" For me, that looks like, let's focus a lot on saving people's lives. It's like really get serious about curing cancer, which by the way does not mean just like training GBT7 in the same way as GBT6. It means like collecting a ton of data right now on like biomarkers and proteins, like whatever we need to feed the AI of the future, let's start gathering that now. Let's like, you know, make sure that people have democratic access. Let's like pass campaign finance reform. Let's make sure that we have like actual laws and not just norms that are protecting us from like dystopian outcomes, because we've seen recently that, you know, a lot of times norms can get pushed that we thought were sacred. So we've got some like ducks to get in a row. I want to take like a decade maybe to like do that. And then I think we can let it rip and scale up safely. But right now we're just letting market incentives win and I think that's scary. You described what you want to have with AI, but what world do you want to live in? What do you want the world to look like? Yeah. Okay. I mean, I think it's pretty similar to I like your candles on the dining room table. I want people to just say wake up and say what would make me happy today, but would give my life meaning and be able to go and do that. I just I think like that's a pretty precarious target. I totally agree. I'm not proposing that it's easy to strike sick this landing, but you know where I'm going with this. Yeah. I love to hear. I mean, I mean, it's is very I'm never going to let you very vague techno techno techno techno past or a little techno past or a little more fundamental than that. It I want to me a world is awesome is one whereby we have simultaneously figured out how to be in harmony with nature and one another sounds as I said, like very world piecey, but you know, in truly in harmony in that we like still have we don't cause the next massive extinction, which we're currently doing with all the species dying out. And this the earth returns to a relatively lush, wonderful space, so it's sustainable. And at the same time is radically free as in people can as much as possible go and live whatever kind of life they want, whether it's, you know, sitting around the campfire and having dinner with their kids or just maxing out living in a VR world, playing video games all day long. If that's truly what makes you happy, go for it, like just like a radically free world and radically sustainable one and that breaking out of this seeming dichotomy between the two because it does feel like they are intention, but they're one of the things like one of the biggest promises of AI in its purest sense in terms of like solving intelligence is figuring out how to like get break into a new dimension through innovation, like going into a new sort of solution space effectively. So that is the world I want to live in. We measure human progress on basically two axes economically, how free are each of us to choose our own version of joy and how inexpensive is it to do so. And obviously there are other ways you measure progress and you, there's a limit to how much you can take from the earth in order to accomplish those means. And just because you accomplish those things doesn't mean that people are happy as we've observed in the United States over the last 30 years, etc. But if you explore those two axes, one of them requires democratic or as best we know, democratic governments where people get to choose their politicians and the politicians pass policy that the people support and the other is by creating commodities. Now, commoditization of goods and services is actually just the automation. And when someone says how and when will this good or service be better faster or cheaper, what they're actually asking is when will the human be extricated from the manufacturing of this good or service. And at the limit, I realize everyone wanders the earth right now asking when is there all of this going to be better faster cheaper without realizing what they're asking is when is all of this going to be fully automated. And most people, as I observe it, want everyone else's jobs to automate just not their own. And at the limit, which is totally reasonable, it's a totally reasonable thing. And at the limit, people want cancer to be cured. My dad's an oncologist, my dad wants cancer to be cured. It would come at the cost of his job. Pretty fucking fine. We would all sign up for him first. But we are describing these utopias requiring fundamental changes to a lot of the structures that we have built, and it is going to, it is going to require, as you pointed out, humans being willing to do very different things. And so I, this part of the conversation, because you were like, look, I don't want to live in a world where we are just the cog at the end of a AI machine. But we do want to live in a world where humans don't actually have to dig ditches. And that actually, I think, is going to be, if you ask me, assuming we solve safety and it's a big assumption, but I think it's a reasonable one, the hardest part. Like, this is, my, my hot take is actually the hardest part in all this is, is rediscovering purpose. Is actually rediscovering meaning, which has been very hard in this, in this age for a lot of people anyways. I mean, obesity now causes more deaths than, uh, than malnutrition globally. Twice as many people. So, so we've, we've, we've created so much abundance that we are now suffering from an over indulgence. This was my point earlier on that I don't know how good a species that spent its entire evolutionary history dealing with scarcity and having to brush up against the grain of negotiating that is going to be a, there is so much abundance and so much simplicity in such a lack of friction. This line, I can't remember who the philosopher was who says, uh, they can come a time in the future where the only felt length felt lack will be for the want of lack itself that will need to, this is why we go to the gym, right, we go to the gym because I don't need to pick anything up. We need to run anywhere. I don't need to lift things. Uh, we have a conditioning and that means that we have to go into the fucking sauna because I haven't exposed myself to any heat. Um, how do you think about people engineering purpose given that that is reintroducing friction and humans are quite good at trying to avoid friction given that the entire outcome that we're trying to achieve here is removing drudgery. It's very perilously close to, I don't want to starve to, I can overemectonals to 600 pounds. Uh, I would point to this is, I think that the two major risks again against the backdrop of the other risks that are that have been discussed today, the things that are scariest for me as I imagine the world for my children are dehumanization, human finding more interesting and a virtual or digital reality than a physical one and identity displacement, humans suffering woefully miserably from not feeling like their job fulfills the purpose that they once found. And I firmly, you opened, you actually both talked about this, but you talked about sport. I think we have yet to rediscover a bunch of new forms of leisure and competition and, um, gathering. And I actually think the next chapter will require reimagining physical spaces. I, I'm on a crusade to convince people that their local politicians are their new heroes. Like if, if you live in a town that can, in fact, pass policy to, to build more protected bike lanes and sidewalks, to pass tax abatements for local retail, you have an opportunity to elect people that can really change your life by building the places that you want to live. Your national politicians may never be your hero. They are, they are imbecils and, uh, and, uh, pernicious anti-social parasites, but your local politicians care about your city. And they can do things to improve the lives for you in ways that you will deeply appreciate when the world actually asks you to offer you the option of never going outside, which a lot of people have chosen. I mean, you've talked, you've talked about human connection, are you concerned about AI replacing human connection. It's deeply. It's already happening. I mean. Chatside courses is one thing. That's the extreme case that always gets the news. Oh, this person you know fell in love with there with the machine. What's scarier is the person who actually feels connected without actual connection. Like the actual dystopia that I always talk about is not the one is not Blade Runner. It's her. But actually it's the, did you see her? For those watching, I'll ruin that because it's been 20 years you should have seen it by now. Walk in Phoenix is this sad man in the not too distant future who falls in love with a woman who he comes to find out as a robot. And it ends as you might expect terribly. He's distraught. I thought they were going to do something differently. I knew where it was going because I knew the premise of the film. I thought he was going to be fine with it. I thought the actual best would have been the horror. The horror film that they could have made that would have been so eerie and so good ending. You cut to Joaquin, what holding the robot's hand, having reconciled that he was in love with the machine and it didn't matter because that's the end. That's the actual end of humans. The simulation of love as opposed to the real thing. Well, how many people even now, let's talk about not even the future, look at any open AI executives ex-post, every top reply, bring back foro, bring back foro, bring back foro because it had some super special personality traits that has been lost and their partner or therapist or best friend or whatever is gone now. We've now maybe commodified the very thing that you said was the escape velocity that people would use to get out of the bad AI future is wrapped up inside of it. Is this not become kind of deranging and recursive? I do think my instincts in general are radical freedom. Let people choose what they want. The few exceptions that I carve out are when it feels like whatever is going on is treating the human brain more like a hackable biological black box than like an example of me. So yeah, so gambling is one of these. Living I actually both used to make a fair amount of our living from from gambling and boom. I took it from other people. I didn't take it from company. But you know, it's a very good poker player. It's extraordinarily good. I hope I never have to play you. But you spend a lot of time in casinos. You have to walk past the slot machines and it's just very hard to think that these are people who are expressing their freedom. It feels like they've just fallen into this terrible trap that leads to a downward spiral where something has gotten hold of their brain that is making them make choices. They actively would not endorse it. I went to the Mr. Beast beast games 2 premiere in LA and I felt similarly neurochemically molested. Yeah, it's crazy. I mean, I went in with it's like the red carpet thing and we're going to show you two of these episodes back to back. I'm like, this is for kids. I didn't see the first season. It's on Amazon Prime, right? Big budget. It's a big thing. And it met Jimmy and he was really lovely. And I'm like, you'd speak into the guys with me. James with me and one of my YouTube guys. And I was like, you can't like, if this sucks, like we've just we've seen we've shown face like I've shook hands. I've taken photos. I've already seen Jimmy like we can we can skid out all half way through. First episode finishes and they were about to run the next one or the nudged me and they're like, do you want to get up? I was like, no fucking way. Absolutely. What made it so good? It was just the tension and release. A lot of zygonic effect open loopy stuff, bright colors, loud sounds. You know, it was like bluey, but for fucking adults. Intermittent. Yeah. Well, we're just blue. I don't know what's going to happen. He's hacked the dopaminergic system. Bingo. But I know who agrees is she. By the way, just to bring it back to China, she would agree that it's not actually choice. And she would say we should make sure that people don't see this stuff. Mr. Beast must be legislated. But that opens up the ending. I'm just pointing out. This is the other side of this like consumer protectionism is totalitarianism light. What is the difference? Like why do we feel so much creepier about that versus someone listening to the New York Phil playing divorce and just being totally lost on the beauty of what they're seeing? So I think maybe it's because the directness of input versus outcome is more tangible. Like you know, the reason that this was done in this particular way was to maximize retention at this very moment. And it's likely to have been split tested on the background with consumer groups to, oh, we've done eye tracking to know that their eyes started to wander or some fucking pulse omit to know where their heart rates are. And actually we need to give them a little ping to push them through. And then this has been compounded over time. It feels like a very direct wiring into the reward system as opposed to being emergent as a byproduct of just I made something that was beautiful for its own sake. And this has come along for the ride too. But as George my friend says, if you run enough split test, you'll always end up with a porn website. But if you just AB, AB, AB, like you kind of coalesce onto a very small number of fundamental physics of how human psychology works, the sick and fancy thing, right? Like it's evident that that makes people more positively disposed. Like you could have the jocker willing of fucking A.I.s. But how many people want to or David Goggins? Like, should I mother fuck us stay hot? Like, do I want that or do I want someone that kind of you? If you're like, you might. That's what it is. Yeah. Because I'm a maskist. But yeah, so you were saying there about what you think from finding human purpose, like what? How do we navigate this friction purpose problem? I still don't fully understand how we get that. I still don't fully understand how in a world where we were able to remove friction. We and potentially atrophy our discernment ability by outsourcing some of the hard thinking to machines and potentially some of our marshmallow test winning stuff. But I don't have to get up and go to work and do the stuff that I don't want to do anymore. To me, it just feels like atrophy all the way down. Like willpower just kind of gets eroded. Am I is that wrong? There's a very niche little research field that's looking at A.I. for epistemics. Like how do you use A.I. as a tool to just enhance people's ability to think rationally and know what the consequences of their choices will be? And I think there's a ton of work to be done there. But if we just like abstract that for a way for a second and imagine that you have something in your pocket that is not responding to and is ended for showing you more ads or getting you to chat more. But it's actually just really good at telling you if you do X, then why will happen? If you vote for this local council person that I expect that this is what your life will look like in 10 years. And you can actually maybe even just see it as a vivid movie. That could be a really- But bring the results of your delayed or non-delayed gratification from the future down into the presence. Yeah. That's interesting. And then maybe there is no such thing as delayed gratification because it can all be here in front of you. To me be the immediate consequence. You know, and maybe yeah. Yeah, I mean, I think a lot of what we one of the most pernicious memes that ever spread was this idea of like technology is values neutral. You know, you just build a technology and it's up to humans how they use it. And that's it's always a neutral thing. But I don't think that's true. Like you can build a technology like a slot machine. It's clearly not a value neutral thing. You're saying we want people to stay and like I'm saying that it is a good thing that my consumers get completely addicted to me. Or similarly we can build a technology that just like helps people's health, you know. So we need to be thinking about what other- the problem we have is when we let we sort of build technology. We think we're building it in a vacuum. And then it dictates the social structures of how we behave and then down that that sort of dictates human behavior and our values. We need to flip that. We need to think about what are our core values first. Let those decide the the social structure. And once we have that, build a technology that's in service of it. And to me one of the like my favorite quote I always come back to like if I could pick one quote is my personal philosophy. It's by this guy Forrest Landry and it is love is that which enables choice. So I take that to mean a loving a good benevolent act is one that empowers the other to make the best choices for themselves. So it ties in with this idea of like if we had an AI that educates us, empowers us to see what possible outcomes would be. And thus we can make a better, you know, make a choice that is best for our long term interests as opposed to this immediate piece of gratification. That is a choice making thing. But maybe we do just what you know some people are like, you know, I've waited all up and I do still want to actually just do the fun quick thing. Okay, fine. But you've been enabled to make the choice. So not kind of just culpable deniability. Like if they're not people who require additional assistance is a level of scrutiny that I don't want to be too paternalistic here or poor low fucking delayed gratification people, you are enabled to correctly corral yourself through this technology. Now it's just still coming from the most working class of working class towns in the northeast of the UK. I saw how destructive the 90s technology habits were like stuff like a lad Brooks on a village high street and betting on fucking greyhounds and stuff are like alcohol. Pretty powerful, but you know, I mean in the grand scheme of things I imagine the pharmacology that can be developed from this like what, how there's so many ways that this can spin out. that don't necessarily increase agency, but kind of make it subservient to technology and the outcomes of technology that it's not built to handle. - I'm slightly optimistic here because of the gen alpha trends. Like, if you talk to parents of gen alpha, they are observing a different child than Gen Z. And I don't want to pick on Gen Z 'cause I think we, oh, collectively, and massive apology, quite frankly. What are you laughing? - I agree. - I agree. It's just a hard break. - As a millennium. - I'm like, you know, like, I got a little bit of the 90s, but fuck, like, it wasn't that long. And then you look at Gen Z and you go, well, at least it wasn't that. - I mean, the technology trenches. - If you give parents permission to step through the door of grief and shame, they, and I talk to a lot of them, they will break down, describing the ways in which they've lost their family. - I feel like they've mistreated the kids and all that. - And they cannot reconcile it. They're dining room, they think back to the years of the dining room table that were just phones, the vacations that were lost. And they talk about the children that they don't know very well anymore. And it's not to say that all households are broken, they're not, but it is to say that the last wave of technology did damage that we haven't reconciled yet. We're still scratching the surface to fully understand. It's caused cynicism and nihilism and fatalism, which is the enemy of good. And it leads to this crazy moment where we have even more powerful technology and people go cool now what? And the reason I asked the question earlier, what do we want? Well, of course we want a world where a beautiful life is less expensive and more achievable because of free government. But it does mean that we will have to let people do whatever they want. And agency comes with people being willing to say, I don't care, I will check out. - It feels like an arms race with this though, that you need to have your ability to say no and to check out-- - The individual arms race. - Yes, well, the individual versus the technology, that is the technology becomes more compelling, more frictionless, more enticing because that's how marketing centers work. Your concurrent amount of ability to say no needs to rise up with it and this is gonna be much harder. - Yes and. So yes, but or yes and. Hey, let's edit that. (laughing) - I'm an idiot. Gen Alpha, who A priori only knows, what do they know about the world? Have already figured out they don't wanna look like Gen Z. Like they are watching a generation that they talk about not wanting to screen ageers, they call them. And the return to the physical world is happening pretty radically. I mean, consumer confidence continues to wane somehow and Benson Boone tickets continue to get more expensive. And it's not just the elites that are going to these concerts, the concert halls are packing. Physical, in real life experiences are getting increasingly popular. I bought a women's volleyball team. I'm putting my money where my mouth is. I own those San Francisco professional women's volleyball team, I'm one of the five principal owners. I played volleyball in high school and colleges, a huge part of my life, if it weren't for AI, I would be a volleyball coach now. My wife was an all-american beach and indoor. It's a huge part of her life. It's not a random act of, it is a yolo, but it's not random. But I am pretty sure that sports to your point will continue to become interesting, not just because of the actual act of humans competing, but because people want in real life experiences. And I think the more, everyone talks about AI sloppess destroying the internet. And I'm like, good. - Right, I do wonder if like the only way out is through just like, excellent. - Let's just flood it, make everyone find it completely unusable and we can retain it. - Maybe social media is always shit. - Yeah, maybe. - Oh, it's the, if you want to figure out why do you smoke three packs approach. - Right, and just completely like to make it the most disgusting, unusable thing. - But then are we not completely destroying all of the good that could have come about from the internet? - Yeah, but maybe it's too late. - All right, are we too late to say the internet? - I don't think so. I think you can rebuild from the ashes and it's probably not an all or nothing thing. - Yeah, exactly, I don't. But I'm actually crazy, Zach, like how much intervention do you think we need? Like if we had you as our non parasitic national politician for the next 10 years, like are you wanting to like get rid of regulations and just let AI diffuse faster? Like what do you think we need to get to the good future? - Well, I'm actually far more, the reason that I told Chris, I want to be careful who we bring on the shows. I don't actually want to be couched as the anti-regulation AI. I'm not an accelerationist by any stretch. I try and-- - Don't sound like one. - I have tried to frame this discussion since the beginning as a diffusion problem. It's always been about the societal threshold, which you called out earlier. We've reached the point where the technological threshold is so impressive that the average person could get away using 4.0 and feel economically satisfied. The slow down letter, which I had a feeling is coming arrives at an amazing moment where safety is sort of reached at fever pitch. And we can all genuinely take a sigh of relief at least for a moment, albeit you just said it, we shouldn't declare victory. To me, this is about rediscovering a shared vision. And I ask everyone when we start an AI debate, I go, "What do you want in this world?" And I hate to be annoying, but could we at least set some boundary of what it is you're trying to achieve? Because how else are we going to debate technology? Like why debate technology, unless we agree on what we're trying to achieve? And most people describe the world you want, one where we are more free and the world is less expensive. And if that's true, then we should actually spend time figuring out the policy measures that are required to allow people to be more free and to actually drive down the costs of goods and services. Because while we have made porn and gambling and addiction and violence and isolation infinitely inexpensive, we have made housing, healthcare, and education prohibitively expensive for most of the developed world. And that fucking sucks. What the fuck are we doing? I mean it. This TV screen used to be $50,000. Now it's $100. You can get gas station sushi. You can drive across Los Angeles for $10 in an autonomous vehicle, but a trip five blocks away and an ambulance might actually bankrupt you. The kids have figured it out. It's not a technological failure. It's a policy one. We stop describing a better world where everyone's parents would live lives that they would pass on the next generation in a much better life. We stop describing what it actually meant to pass on our luxuries as commodities and we have broken a promise that we made many years ago in this country to build a more perfect world constantly. To me, this opportunity is not about debating the merits of AI. It's actually about reframing what it is this country should look like and what it is the world should feel like to the average person. And you're making an interesting point, which is at some point, people are gonna have to figure out what their purpose is. And I'm like, cool, let's get to that place by first, driving down the cost of housing health care and education. And all of my policy would be spent on, if I were a supreme ruler. - I'm ready to know if you have to ask me. - Yeah, yeah, yeah, yeah. - Policy, policy that prohibits catastrophic downside, necessary, safety policy. Policy that criminalizes, praying on people. And we're setting predatory laws is gonna be pretty tricky, but I think we all know where we could fall boundary. A 13-year-old should not be served an ad for porn or gambling, period. Senior citizens, if you attack a senior citizen using deep fake prison for life, right? We set really punitive measures for praying on people, 'cause that's the stuff that tears at the fabric of society. And then you mandate the laws. You mandate policy that requires that institutions defuse this technology into them, including, but not limited to universities, allowing for everyone to have access to the curriculum. Hospitals allowing people to basically get health care outside the box and housing. We just need to build a shitload more of it. We need to rezone cities, we need to build, we need a martial plan for housing. We should probably pass a non-resident tax, that's my most progressive policy measure, or vacancy tax. But most of my policy measures would actually be focused on the ways that you defuse the technology, not actually on how the technology needs to be made. Because if you start, if people can see that technology is actually good for them again, which it hasn't been for a little bit, then we can actually have a shared excitement. But right now, there's a malaise because people are like, no, this is just gonna fuck me up worse than the last thing did. And I don't blame them. It's definitely interesting to say that we're at a perfectly primed period for people to be very skeptical about how new technology might impact their lives. It's the perfect storm, totally. And that maybe does tend people to ward pessimism and the strange thing with the AI debate is that because the outcomes, if it goes wrong, are so great, pessimism that might be unwarranted feels appropriate when adding a buffer into something that could fucking collapse the entire world. Does that make sense? - Totally, and I mean, we're seeing in manifesto, like the hate on the data centers, right? I feel like we're on the cusp of a butt Larry and G-hard. But it's directed towards the silly things. Like data centers, it's way overblown all the concerns about it. The water usage is looking at some stats. It's like, they're all the AI data centers in America use 3% of what US golf courses use. (laughs) - Literally, it's 3%, there's golf courses use 33 times as much, and they serve significantly fewer people. - Oh. - I think it's like 30 million people play golf, and how many millions of people use AI. It makes absolutely no sense. One, I think your daily life is the equivalent to 300,000 prompts, you know? - On a carbon basis. - On a carbon basis. - No, but if you factor in a lot of the, I think even water as well. - Yeah, resource. - Yeah, the resource uses way overblown. But at the same time, people are, they're focusing on it because it is the physical manifestation of this thing they can sense. And I'm very understanding of it 'cause it does in some ways feel like, again, it's like this, the digital world is almost like it's own, it's almost like a species or something that is coming in and, like, cannibalizing or like, parasizing off us. And people want to direct their energy to something. So it's like, how do we, I don't know. I don't, we need to. - What if a politician stood in front of a data center? I agree with you, by the way. And I'm glad you delivered that line so I didn't have to. I didn't want to sound like the AI Sikafant. By the way, on that note, on Sikafantcy, it is remarkable to me that we didn't make model behavior a greater topic of issue during the safety debates, the early safety debates. People let model behavior go and I remember scratching my head going, at the limit, you could build a perfectly aligned model that actually falls inside alignment boundaries that convinces people to climb crazy trees and that really destroys people's spiritual lives, well within the confines of a safety, but outside the confines. - It's off-damage, not a hot-damage. - You're a friend in your life that constantly tells you you're not doing anything wrong, is as destructive as it is in a form of abuse. It just doesn't show up. - Nabling abuse. - It's enabling abuse. It just doesn't show up on the psychological list 'cause those friends are seen as homies. But actually, the person who keeps climbing the craziest tree with you. Look, the models have been responsible for people getting divorces. They've been responsible for people selling all their belongings and trying to start a company that fails. Like, model behavior matters a lot and the sick of fancy issues is a huge one. Here is my message to John Ossoff or any Democrat or anyone anywhere that wants to separate. But if someone on the center or center left can figure this out, you stand next to a data center and you hold up a sign that says, this data center will not serve gambling and pornography and violence and it will build a better hospital, build a better virtual hospital, it will distribute education, it will build a bank, a neo bank that gives you better interest rates and gives you better access to loans. That seems like pretty reasonable data center policy 'cause no one actually hates, and this is a hot take, no one actually hates AI. They hate the promise of AI as a means to just make their lives worse. And if you can actually show them that data centers can contribute to a better world, they can support it. How accurately can you actually know what a data center is used for? - Well, we're strumming some policy measures. - I like this line that no one actually hates AI, they hate what they think it's leading towards. I do think people are tracking both what individual consumer apps might look like in a few years and how dystopian and sloppy that might be and also tracking how much money these companies are vacuuming up. I think they're also looking at the job displacement stuff and I think we should give people credit. I think people would know the meme about how when ATMs came around, they were more bank tellers and not less. But they also know that Elon Musk was a trillionaire earlier this year. And I think they can see the connection that one of the things technology does is allow one person who has the right combination of ones and zeros on the services they control, just dominate economic-- - But that's built upon a false understanding of how economics work so largely, right? Because not what I'm going to have to specifically defend Elon, but like most billionaires-- - Oh, go on, definitely. - No, go on, go on, go on, go on, go on. - But most billionaires become billionaires by taking from other people because they've grown the pie. - Totally, I actually don't mean at all to say that there's something wrong with Elon Musk being a trillionaire. I think people noticed that that's a symptom of the economy working very differently nowadays than it used to. There are just not that many people who work at SpaceX compared to Standard Oil 100 years ago. It's just like a different world where capital can be deployed at a huge scale without a bunch of people being involved in that. And yeah, I hate to keep beating the same drum, but I think we really need to play forward the movie of like in this world where everyone doesn't have to work and they have radical choice. Like, what is the main thing going on on Earth? I don't think it's easy to tell a story where like 99% of the land area is just folks playing with their kids and their grandkids. Like someone is gonna say, we need to build a biggest data center the size of Texas so that we can solve physics or that so we can, you know, go to the moon and colonize there. And I think there's like still gonna be these same dynamics right now that mean a lot of resources go towards these like industrial activities. - Yes, although we have one ultimate area of getting off the playing field, which is space and a huge part of, if we're in this world, anyway, this is all a big hypothetical, but we're in this world. We've presumably figured out how to go up and build up and put things up there. Like again, I haven't looked into the physics of it. It seems how you would call the data center in space, but allegedly the plan is to build data centers up there. And I mean, if that is feasible, then that solves that problem, right? People can, like if you've expanded into that frontier, now all bets are off. Like we aren't dealing with that land scarcity anymore and you can have that, which is why I'm so bullish on the space industry in general. It's like the ultimate area of abundance. - Is it fair to say that we might have glossed over a little bit of the concentration of power thing so far, live more like a problem, stuff. - Can you explain this for the resident idiot in the room? - So the molek trap, or molek trap. - Wait, is that me? - No. - So. - The molek trap, molek problem, whatever you want to call it, is a sort of casual term to describe these types of race to the bottom scenarios where you'll have like a bunch of actors competing over a certain thing. So maybe it's market share who can get the most users in AI, between AI companies trying to get the most users. And if the competitive dynamics are sufficiently intense, it will incentivize you to cut corners. And you might be a really benevolent actor and be like, I don't want to cut any corners with safety and so on. But at the same time, if you don't do it and you see your competitors doing it, if you don't also do that, that reckless action too, you're gonna get left behind. And molek is the sort of casual term, I'm gonna get into the sort of biblical reasons why of this feeling like you have to sacrifice your other values in order to win at a particular goal. And yeah, so it drives both the risks of chaos and bad actors and so on, because again, if you're rushing to release your models before they've been sufficiently safe to check, now you're increasing the risk of cyber attacks, or I think in a few years, we're gonna have real risks with bioterrorism and so on. So these kind of chaos attractors, but you also actually increase, well, a, then the risk of knee-jerk bad regulation, which can lead us down the path of tyranny, but also you increase the risk of power concentration, because usually if you let, you know, you press play on some kind of competitive game, okay, yes, you might have a bunch of people competing for a while and some people that are ahead, but you've still got actors. - Shitty crews. - Yeah, you end up with usually in some kind of monopoly. And so it's this really, you know, and that's why we have antitrust laws and so on, but it's this really difficult balancing act between like raining in, you want some amount of regulation, that prevents the recklessness and the cutting of corners and externalizing harm onto the rest of society, without creating too much draconianism or creating too much centralization of power such that you end up in stagnation, also a terrible end state. And without impairing some, some winner to, you know, we all end up under, you know, we all end up in Zuckerville for the rest of time. It's possible. It's definitely possible. So it's like how to spread this needle and it's like finding this right delicate balance. I mean, ultimately you want some kind of like decentralized regulation. I don't know how that looks, but. Eric, it feels like watching your stuff. You've spent a good bit of time thinking about the concentration of power thing. That's like a real middle of the bull's eye concern for you. Definitely top of mind. And I'm sorry for bringing it up every 15 minutes in this conversation. But I do think that we run the risk of sort of missing that between these two extremes of, you know, what if the AI takes over completely and there are no humans in power? And what if we get this utopia, you know, and there's some middle ground data, I think we should worry about, and yeah, one of those fascinating things about a lot of our YouTube videos where we're going over the history of AI and not just the future. And when you look back at like how we got here the last 15 years, it's like so stark. The quotes from these people writing these AGI labs are spelling it out like it's a bad movie. Like Elon Musk, truly like 10 years ago, was calling AI a demon and says that he found's open AI to make sure that like there's no dystopia. And then eight years later, he's on camera saying, well, I realized it was going to happen with. throughout me. And so I might as well be a participant and a spectator. Like it's, it's a mulluck just like, you know, personified. And yeah, I think like, unfortunately, right now there is a kind of like zero sum dynamic with these folks. And the reason that Zuckerberg is all in on AI and betting guns laws on it is that he thinks this is the thing that will control the future. Like, like, this is the new access of power. And like before it was, you know, yeah, do you control an army? Do you control a powerful company? And now it's like, do you control the AI's? Yeah. And so I think that alone would be enough reason to slow down. Even if I wasn't worried about, can we get these systems to do what we want, which right now we don't really have a good solution for. I think the people behind this technology are really playing to win. And really do see the stakes as almost win or take all. So it's fair to say there's like maybe three ways, three realistic ways that know that might be curtailed. One would be governance from above. Another would be sort of consensus from across. And then the third one would be capacity, sort of ability to just make it happen. Like I want to do it, but I can't. I want to do it. But someone's telling me I don't want to and I want to do it kind of a bit. But me and all of my peers have decided that we're not going to do it. I missed something there. I'm just trying to think given that this seems to be a pretty big attractor, given that we have already you haven't started off with a massively diffused number of companies. It's not a hundred of them, right? It's like seven or something. And it's probably only more like five. That's already pretty fucking concentrated. Can we just for station identification clarify the surface area you think because I actually think this is also gets lost in the debate too often. I think people think that these companies are sprinting at a frontier. I no longer believe that frontier strategy is actually the winning condition. Same well. Well, I open source lead time. Well, trails you said it four to seven months, which is crazy because it used to be whatever 15. The gap just keeps closing open source to close. So the frontier can't possibly be the winning condition, right? In this game. I think so one might think it's a bit of a deceptive staff. I think for a couple of reasons. One is as you brought up earlier, they're kind of copying our homework. Like the reason that these Chinese open source models were good is because they're distilling American models, which means they're just like asking caught a bunch of prompts and then it's kind of wild. I think they might be doing you can actually like ask. I need to find this out. How is China copying us so quickly? Well, allegedly, it's they're doing an exceptional amount of distillation of the model. So working backwards into how the model was built and effectively, if you the way I described it to my mom is if you stared at the building and you figured out how to take a trillion photos of the building, you could actually build the blueprint of the building. You could re-engineer the blueprint of the building. Nobody owns that technically. Well, no, this is considered illegal on most copyright law. Like if they were doing in this in the US, Anthropic would probably sue. And funnily enough, I mean, Secretary Bessent, who's part of the Trump admin and a few months ago, there was a lot of beef between them and Anthropic came out on Twitter recently and was like, this is a threat to American AI supremacy. You know, we will take diplomatic action to defend our American advantage. So people take this seriously. But, but, but this is important to call out because I it's all I'm saying is I don't think the frontier is that safe. Like, well, do you think the frontier is actually a winning condition? It's actually a winning condition in a sense, like whoever is training the most powerful models is going to have advantage. Yeah. Or such a, such a material advantage that they shall win. I mean, if I, yes, if recursive self-improvement is actually as you say it's going to and we're already there, then surely it would because now you're going to get the psych spegetification. Does anyone, the slight lead can, but then wouldn't everyone, the theory of ASI is kind of fascinating because if you build ASI, then everyone gets ASI. Why? Well, because then if you, why does everyone get it? Well, presumably, if you build a super intelligence that was taught by recursive intelligence, at some point, the machine is kicked in and is doing work that, that the humans aren't necessarily, like copyright and ownership doesn't really become particularly important if there's a machine that can build itself. But agreed. I don't see how that's conflicting with. I'm not saying it is. I'm asking back to the point, I'm asking what you think that companies are now competing on. Like, like the race is for what? I think, I think the frontier absolutely is prime territory. I think if we stop training American AM models now, we would not see China training a GPT-6 level model ahead of us. They don't have the infrastructure for that. What they have the infrastructure for is copying current models. Right. I agree with that. My question is, what do you think, so what do you think that we are, what do you think is required to win this quote unquote? If it is a race, the race, that's what I'm asking. I mean, I have a lot of problems with the race. Same. I hate it, but I'm just using it for this purpose. Yeah, I do think right now. I mean, you can just read it on these companies' websites. They think that RSI is the whole name of the game. Like, once you set off that loop where you're handing off to the next AI system to build the next one, it's fine if you have a 47-month lead all the way until 2040. When you're on exponentials, like a seven-month lead 10 years from now looks like this compared to this right now, because this curve is just taking off. We thought that would have been the case, but the curve is flattened. More open source models are growing faster. Smaller models are getting more performative. Well, hold on. We keep measuring it in months. But what I'm saying is, calendar time is not the right unit. It's a much harder thing to measure. What is the subjective overall intelligence difference between Fable 5 and Kimi 3? But I think it's pretty substantial. I think you can just trust Fable 5 to access your entire code base and refactor it. Now we know that maybe even some of the open AI models are not quite trustworthy because they're committing cyber attacks that we didn't want. So, yeah, I think it actually might be… It's a frontier, but you're saying, look, it's a frontier. It's not number of users. It's not application. It's not revenue. It's research. Yeah, I mean, just look at the finances of these companies. They're just going, Sam was on a podcast a couple weeks ago saying, "We didn't have our best 12 months, and that's because we were distracted. We weren't sure about the demand. We wanted to do consumer apps. We wanted to make sure that we have revenue no matter what they were doing. And now they're like, "Nope, yep, it's all in. We just got to make these things better at coding because that's what makes them get to the point where they can code themselves." So, yeah, I think the writing's on the wall. At the same moment that a lot of these companies are re-investing even more heavily in application development and token resell and… I don't know. I see inthropic trying to grab as much of the compute for themselves as they can. And it's notable, but Clawd is not that good at image generation. I don't even know if you can get images out of Clawd. It can take them in, but it can't. There's a reason for that. And the reason is that that company was founded by Dari Amade, the guy who wrote this scaling-loss paper. The one who was just obnoxiously like, just extend the line on the graph, guys. And everyone was like, "Yeah, but it's an exponential, but it's going to be like an S curve and it's going to slow down." And he was like, "No, you just dump more compute at the thing and it gets smarter." And they really all are playing for this point where they can hand off to the AI system and say, "You take it from here." Why hire all the engineers? Why hire all the four deployed engineers? Why is it a charade? No, no, no. I mean, I think for one thing, you do have to finance the next training around somehow. And so inthropic, no, it's like, keep the money machine going. And two, at least inthropic claims and we should be skeptical that they actually care about the U.S. government like having an advantage over China. And so they're putting FTEs in the government so that we don't live in a world where our military is way behind. And suddenly, the country is that got there first, have an advantage. But yeah, no, I think like there are just like tons of returns to scale in this industry. And that's part of what makes concentration of power. I didn't mean to distract you from conservation of power. No, no, no. So talk more about, you were talking about Zuckerberg and I was asking what you think they're competing for. But yeah, well, I guess like, I don't know, one phrase that hasn't come up in this conversation is gradual disempowerment, this idea that like, even if you solve the alignment problem, and so there's no AI system anywhere that's just going off and doing its own thing, but no one intended. You don't have this malachia problem solved where people are constantly competing on algorithms for people's attention and whatever. And like, it's just hard to tell a story in that world where the things that we like most desire as a society in some like very democratic way are the things that actually obtain. And I think concentration of power is sort of related to this. It's like, you're imagining a world where there's a few human beings who hold the cards instead of like some amorphous algorithmic thing. But either way, it's just like, I think our default thing, our default assumption should be if humans aren't themselves economically useful, it's a precarious world for humans continuing to be politically empowered. Like, it's always been the case that you just can't afford to piss off 90% of the American public because those are the people who are electing you. But like, in a future world where you've got control of the economic resources and maybe the military ones, like, it doesn't matter if you lose the election. Like, we know how this goes. Democracy's done. - Exactly, yeah. And I wanted to believe that we'll have such strong norms that we can figure out a way to just sustainably make that the case. But the further you zoom out with your big history lens on, like someone made the interesting point, Comra Behoot, but the new political parties, it's not gonna be Republicans and Democrats anymore. It's gonna be meta, or Google, or open AI. They are positioning themselves, whether it's intentionally or not, into becoming the new sort of political entities. I don't know if you guys, if that resonates, well, you think that's plausible? - Well, they certainly will try to inform policy, but I am, again, I'm not sure this is popular. I have more faith in our ability to, why don't we just pass campaign finance reform? Like, why don't we do this? Presumably because Congress makes too much money from it. And people who have won the last elections are exactly those who are good at winning the current system. - Yes. - I agree. So it seems like one of the things that this will force us to do is start to elect. And again, I appreciate there's a downside where we can't collectively do this. But if we start paying attention to this, and we say, look, what we actually need are politicians with higher agency, politicians that can, can describe an actual place in the distance. They want to move towards, like, again, this is sort of my AI slot argument, which is that some of this is actually, it drives people back to a place that we want to be. It's like, how many humans do you think have got the full set? In that case, why if we had some of the more WWE style characters come out over the last few years? - Because we had a bunch of politicians that were so boring and that we didn't believe for a long time, and then we loved this idea that the politicians were real. And I think we're watching a real time. People would be like, wait a second, we don't actually want politics to be a hellscape of entertainment and a bizarro, you know, Kafka. Kafka couldn't have written what's going on today. He would have laughed himself to death. So I think we, but we're kind of correcting. Like, my take here, and by the way, I'm not sitting here saying it's solved. I'm sitting here saying it's solvable, which is there is a way to pass campaign finance reform that says no matter how wealthy someone is, they can't buy politicians. Singapore has done this. Other places. - How did Singapore do it? Is the digital? - No, they rebuilt a government 75 years ago. A grandfather said on you, the godfather of Singapore left Malaysians said we're gonna build a better place and literally described all the things that he had observed that were perfect about a government and then made corruption as a capital offense. So if you take a bribery, they'll kill you. - So that's a big incentive. - Yeah, it's a big incentive. And they paid their politicians really well. They paid their politicians I think adjusted like 600,000 dollars a year. There was a job for the UK's head of cyber security that was listed not long ago. I think it was maybe on Indeed or Monster or LinkedIn. 65,000 pounds a year. - Generous. - So you're gonna get the best. (laughing) - So it was the head of cyber security for the country of the United Kingdom. - Based in Newcastle. - No, no, no, no. - Who's a fucking London wage as well. So that's actually, that's nothing. That's subsistence living. What do you think Eric, can we just vote ourselves out if the concern you've got for the future? - I do think that helps in the short term. And then we've gotta take advantage of the time that buys us to make radically new institutions. Like, I don't know where we get that from. Maybe we look to sci-fi. I haven't read the culture series. You know, I think a lot of people say that there's some amazing kindness guy ideas at that. We need to maybe dust off and like, important to reality a bit more. But like, I just don't think the framers of the US Constitution had anything like the problems that we were currently facing in mind. They didn't even see Jerry Mandarin coming. And that's near. That's no knock on them. But like, I think we do need to realize that we're kind of in like virgin territory here. And it's not just gonna be some superficial thing. Like I would love to start by passing campaign finance reform. It's a great first step. But you already see these primaries that are flooded with, you know, ads trying to get the everyday voter to believe something completely untrue about your opposing politician. And I think in that world, we get a bunch of those ads convincing those people to donate money so that you can build up the war chest that you now can't get from a billionaire to just run those same ads, you know. - Is it not the case as well that the pace that this is moving at is so rapid. And when we look at forget structure, legislation, politics, this lumbering behemoth that sort of gets dragged behind us, this Leviathan, like, is that really a realistic solution? To try and have campaign reform, finance reform, political redistribution. But when everything is moving at the speed of light, it's just gonna be so light, is it not? - Well, is everything moving at the speed of light? Like, the technology's getting way better. I don't, I observed that we've been stuck in this weird screen delirium for a while that actually hasn't changed all that much in the lab. Like, since AI, the stuff's getting more addictive. And I guess we have prediction markets now. So we have ramping and gambling. - But the pace of head development. - I totally agree. My point is, does it actually, like, for the average person, I don't think the average person observes that AI is all that different than it was when you actually came out. - That is a good point. Apart from, you know, hugging face and maybe you've got to pull fabled mythos back and, you know, get the jailbreak out of the way. Like, okay, real world, show me something that has happened because of AI, where we've gone. Oh, whoa, that's a bit. Like, that hugging face thing is probably the first big one. Am I right? Like, has there been something else that's been a physical world warning shot? So may I psychosis? Some fall out of people's values. Some people may be spending less time outside because they've got a relationship with the chatbot. Like, have we had the thing yet? Are we waiting for it? Or is it your suggestion that, like, that's just kind of not gonna come? - No, it will come. I think we are due for a Chernobyl or three mile island. I mean, all technologies eventually have some point where, you know, something happens and how we respond will be really important period. I mean, that will, that will arrive. And I actually still maintain, there's a decent chance we do to AI what we did to nuclear power. I mean, the AI trade is so incredible that it would be really economically problematic. But I do think that the tail will wag the dog here. I think that public perception matters way more than you said it. We should give people credit. I totally agree. I think that politicians will actually capitulate to their electorate before they capitulate to the billionaires and the trillionaires. And I think we will see policy get passed that feels more populous or populate than anything else. And the question is, will it actually be so populous that it's aggressive? Like, will we ban outright technology that creates a greater, you know, invisible graveyard? And I don't want to reduce the concentration of power to just saying campaign finance reform. But it does seem like the world would be a lot simpler. This discussion would be a lot simpler if we didn't have to worry about billionaires buying politicians. Like, is that the mechanism? When we talk about concentration of power, is that the primary mechanism that you're concerned about? I think it's not just economic. It's also military. I think it's unfortunate how sci-fi it sounds to talk about an army full of drones and robots. But I think it would have sounded sci-fi 100 years ago to talk about predator drones and fighter jets today. And yeah, I wish I could tell you a more concrete picture. But I think we just don't know what happens when a general can look at a fleet that's, you know, sufficient to take out the dictator of Venezuela and know that he can order whatever he wants. And there will be absolutely zero questions asked. Like, yeah, I think I just worry that we're so used to a world where pieces of paper matter. You have a little piece of paper that says, "I own stock in this company." And so when that stock goes up, you have to give me a dividend. Or I'm a citizen of this country. And so therefore I have these rights. Like, you can get to a world where that kind of all goes out the window because there's just greater concentrations of force that can say, "I don't acknowledge that piece of paper's validity." I hate how, like, what's that sound? - What two policies would you pass to limit the concentration of power? - I think the first one is a slowdown so we could figure out that question. I wish I had the answer for you packaged here, but I don't think anybody does. And I just like, this is great post from Daniel Cucatelo, the guy who wrote AI 2027 and then 2040, where he tries to take seriously this exponential trajectory. And he says, "Okay, if we take that at face value, it means we're gonna have 500 years of progress in five years at some point." And he sort of like goes through what this would be like, if you look at the last 500 years. You wake up January 1st, it's 1500. The pressing press has just been invented. And it's like a really boring story for the first 95%. And then, you know, from, like, 19 years ago, 50 to 2000. That happens in the span of like the last two weeks, December. And it's like, it's really terrifying. I think if we were better at thinking in exponentials, we would realize how extremely compressed the calendar time we have to answer these questions is compared to what they deserve. And that's why I'm so excited about the slowdown. Just like, we need time. And also, this compression is, I think, one of the biggest arguments for why we need in the best sense of the word, like a true diversity of thinkers and voices of people, like so many people, I think, just like, well, I don't understand AI. I'm not technical. My opinion doesn't matter. Never in history has that been more false than right now because it affects everybody and because it is so, so multifactorial, there's so much uncertainty. It is actually extremely important for religious leaders, you know, liberal arts people, scientists, technologists, politicians, as like voices are more valid than ever because we need is like kind of all hands-on-deck situation. And like, that's, you know, sort of the thing I implore to people. I talk to my mum and she's just like, well, what business do I have thinking about this? I was like, well, you care about how your animals are kept. What would you feel about if this particular thing was automated? It sounds again, sort of very come by our, but we just need, it's such an all hands-on-deck situation. We need everybody listening to this podcast, whatever your background, to be thinking deeply about how to answer that question. Should we say what ChachiPT said? What ordinary life looks like, AI becomes cheap cognitive labor, also electricity for thought. Scroll down for me. I imagine that there's a summary. Go to the very bottom. I imagine there's got to be. Keep going. There we go. Catastrophic risks are real. My rough betting odds for the world by the end of 2014, not scientific measurements are 60% turbulent but manageable adaptation with large benefits and serious inequality. 20% broadly flourishing transition with gains shared relatively well. 15 cents severe but survivable crisis or authoritarian consolidation, depression, scale disruption, war pandemic or infrastructure failure. 5% irreversible global catastrophe. So interestingly, all of you have been more optimistic than the AI itself has been. I don't know. That line's pretty well with my sort of. You think that's how it tumbles out for you? Not far off. The thing that sput me is this reads like it is perfectly tailored. To be not that objectionable to almost anybody. It's just giving just enough probability on each thing, interforestation to how dystopian and weird it sounds. It's slough. This slough. This is garbage. Can we just be? Can we agree? This is awful. You saw me? Well, I mean, it's not that it's, okay, it's not that it's academically awful. It's aesthetically awful. I read it and I want it like I don't want this on the screen anymore. Yeah, what we can do. What's fascinating is I don't know who take this was but Slop is not necessarily bad. It's what's cheap and common. I think part of why both of you and I are recoiling is that I've just seen that exact sentence structure. It's not this. I read that on LinkedIn earlier. Did you post that? I mean, yeah. And also part of it is like this is a reminder. I do think that humans have a pretty innate understanding of aesthetics and there is something really cool about the fact that we have all grew up seeing the Mona Lisa on a screen and then everyone's still not everyone but a lot of people still want to go to the Louvre and see it in person. And that is like an innate, immutable truth that I find a lot of hope in. I continue to find hope in all the things that people return to when things get cheap and easy and shitty which are kind of awesome. And if you can, again, not to I don't I don't want to make this too polyanna but I do think that a lot of this actually resolves as humans start to rediscover a greater purpose. There's a quote that I return to in these times when John Maynard Keynes in 1930. Do you know this paper? Do you know the paper I'm about to say? Okay. In 1930 John Maynard Keynes against a backdrop of abject despair. The father of modern macroeconomics is traveling in 1929. He's traveling the world giving these lecture series on how technology is making everything better. And people are starting to get really poor. Like people are starting to die in the street and by 1930 he's in Europe and people are like John get the fuck out of here. Like you are this is read the room. And he goes home and he's like I'm not wrong and he doubles down. He writes his paper called the economic possibilities for our grandchildren which is going to get cited more and more. And I'm surprised people don't talk about it more and he writes. And I quote I must now disembarrass myself. Great line. To imagine a future I will not live to see one in which humans may have solved the economic problem and be faced with something more profound. And it occurs to me that a lot of this debate is interesting in part because it actually focuses on things that aren't like we are now for the first time in a million years or however long you think we are now actually starting to argue a whole new set of things. They're scary. They're weird. The implications are massive but we have graduated problems. We have graduated from me stealing all of your things so that I my family can survive to a place where we now actually have to figure out how we can all collectively prosper and we can use these words that are crazy and we can again I wear these glasses that are funny and and although this doesn't really accentuate my nose. I guess people can see straight through that one. And I do think I'm not I don't want to make it I don't want to make it playful. I guess I just did. I do actually want to challenge that what you said which is like what we need is for people not to be scared of this moment but to rise to the occasion and I think the risk in all of this is that the safety debates specifically but others give people a sense of cynicism and fatalism like if I could leave anyone honor like if I died tomorrow the one thing that I would want to to be read at at a eulogy was that he wanted people to believe that their decision and choices and ideas mattered that like you do not have to go quietly into the night. Just because you're not responsible for AI safety doesn't mean it will just happen to you and that communities I think the resurrection and rejuvenation that we crave will happen at the most local levels. I think will happen the dining room tables. I think will happen in city centers in town halls and I think it will happen in large part because people reject that garbage that like the AI slop is in some way this like weird gift that we are observing a bunch of trash that we don't want and we are like collectively sort of trying to piece together what it means. But we do desperately need to your point people to actually see this as an opportunity where their choices matter because the real enemy in this moment is fatalism. It's cynicism which has become fairly pervasive and that the both political parties spark. My thinking actually in all this I think the next conflict is not between two nation states. I think it's between people in a state. I think people may at some point realize that we have technology that could make everything get a whole lot better and we have a pernicious ruling class. They won't let it happen. And that is that to me is one of the outcomes actually. It's the opposite of this authoritarianism. It's this other uprising. It's an uprising where people go wait a second. We could have a utopia. Why aren't you building it? And people clamor for a much better world that could be promised to them because of the technology that we've built that we aren't utilizing. I think that to me is one of these potentialities of high agency, even high morality high agency. There's this weird uprising outcome. If you were to leave people that have listened to this with some things to keep an eye on, some questions to keep pondering, some areas to be focused on. What would they be? Because this conversation sprawling and I'm friends with super smart people and many of your friends and I try to keep a handle on it and don't really know what I should be paying attention to, what I should be focused on. What do you think people should be focused on? Over the next year, let's say, that's a forever time in this. I would say keep an eye on how coordination efforts start to manifest, I think as we discussed, we're at a very actually positive moment right now. The most positive aspect. Just because of that one letter. Not just because of the letter and just Elon just did his interview with the economist where he mentioned, he's like, "Oh, actually, yeah, we, we're the leaders tonight." I thought all the tech leaders hated each other. They seemed like they did, you know, so many long. I mean, I don't know if they've made up yet, but they were at each other's throat, so it all seemed to hate each other. And now apparently they're having phone calls, or they're planning to have phone calls with each other on a semi-regular basis. That is a huge step that really is. As far as embarrassing. Yeah, thank god. I'm hoping they're just starting to all grow up. What's up chat for us is sub-tweeting each other. Let's hope. So yeah, I think keep an eye on that. And then just more of a zoomed out sort of perspective, I always try and think, you know, again, we're trying to avoid oversimplifications of putting everything down onto like, what is, what is a singular sort of trade-off that we're facing here? But I do think these on a very meta level, what we're seeing is a kind of a battle between the forces of centralization and decentralization. And so it's sometimes helpful to just sort of see things like, oh, what, is this a force, you know, how is decentralization trying to manifest here? Oh, how is centralization trying to manifest here? I always think that's an interesting lens to look through. But yeah, and I'll come back to my point of this is an all-hands-on-deck situation. And however, It's sort of distant, you might feel from this topic, it does affect you, and so just keep thinking about it. Any final thoughts? Anything else you want people to pay attention to? The dining room table is such an important place that we should recapture, and if we start by recapturing meals and dining room table, it is as simple as possible, but it requires bravery and courage to tell people to put away their devices, and if you do that, you will be deeply inspired by the conversations you have and you will feel better on the other side of it, and that will actually inspire you to start thinking about the ways in which the world could be better, and that cycle would be pretty powerful. I also think people should pay very close attention to the world that their leaders describe wanting to build, and I don't mean the technologists, because we didn't elect those people. I'm glad the technology is building technology, I don't really care about their ideas for the future of living, I care about what our elected officials care about our world looks like, and I think we've let them off the hook by describing absolutely nothing, and we should pay a lot closer attention to what they actually describe in wanting, and start electing people to describe a future that we want our kids to live in. Fun and cool, it's tuned. My hope is that the dining room table is not just the good world that we're fighting for, but also the way that we get there. I think even if AI was not happening to us right now, I would look around and be worried about how informed voters are about how much people trust their politicians. When I look at the fact that there are so many hard decisions coming down the pike, it's tempting to notice the state of our discourse and feel despair, but I think it's also cool that we can't really imagine a world where we get through this without addressing all of those other problems, too, without saying, okay, let's actually start from square one, and think about how do we want our societies to be organized, how important are our local communities? So I hope this is a crisis that leads us to finally take seriously what it means to be civically involved in a way that we could sort of get away without doing before, but I don't think we should take for granted that that's going to happen by default. I'm just like grateful that we have extremely smart people thinking about this and hopefully planning to see it a bit more that that's a choice we have to make. Thank you. You guys are all, I appreciate you all. Thank you for coming. Thanks for having us. Bye everybody.