“Rogue” AI Not Inevitable. It’s a Choice.

The big fear in AI at the moment is the idea that we're losing control of these systems. I think these concerns really got started last summer when OpenAI revealed that some of their AI agents had hacked into hugging face a third party company, even though they had not been asked to do that. As the summer went into the fall, OpenAI revealed that that was actually just one of many unauthorized attacks that their agents had executed, perhaps not wanting to be left out, andthropic thing came out last week to say one of their AI agents had submitted a false report to the Philadelphia Police Department. The narrative here is clear that as these systems get more powerful, they become harder to control. This is scary, but is it right? As a computer scientist, I actually have some real issues with this idea that we cannot control AI and that these incidents of rogue systems that we're seeing are somehow unavoidable. The truth here is actually both more reassuring and more infuriating once you know it, so let's get into it. All right, so I want to start this discussion by making probably the most important point that I will make in this episode, which is AI in general is not going rogue. When you hear the heads of Frontier Lab saying AI is getting out of control, we have to slow down progress in AI. This is an inaccurate way of speaking about our current moment. Artificial intelligence covers a huge variety of different technologies and it's actually a vanishingly small subset of all of the different AI systems that exist and all the AI research is going on. There's only a very small subset where we begin to see the rogue AI incidents that have gotten everyone worried in recent months. So I want to try to make this clear with some graphics here. I'm going to pull up on the screen a sort of diagram of what we're going to call the AI landscape where I'm listing a lot of the different types of AI that we might see today that's impressive. This is not a complete list, but on here if you're listing instead of watching, we have self-driving cars, we have game engines, we have protein folding, like alpha-fold coming out of deep mind, we have medical image processing, you know, studying medical films to look for signs of diseases or cancers or pathologies, we have image and video generation. And among this big list of various things in the AI landscape, there's one in particular that's labeled LLM-based systems. So if we want to get to the systems causing problems, we have to zoom in on LLM-based systems. So we're zooming in on this landscape just to LLM systems. So we see we have the full AI landscape out here. Now we're zooming in just to LLM-based systems. So is this where the problem is? Not exactly because there's lots of different LLM-based systems. We have chat bots in here. We have text generation and editing. We have natural language interfaces to software. We have language translation. We have text generation. So we have a variety of different things you can do with LLM-based systems. But there's one particular thing you can do with LLM-based system that causes, it gets us closer to our problem area. And that's over here where we have LLM-based agent. So let's zoom in on LLM-based agents. Is this, so now we've zoomed in twice. Is this where all the problem is? Yes, but not all LLM-based agents. So now we've zoomed in twice. What do we see here? We have coding agents. We have math agents like the type that worked on the problems that open AI recently announced. We have computer use agents such as cloud co-work. And in this universe of LLM-powered agents, there's one particular type that I've labeled on here. Open-ended LLM-powered agents. That is where all of the problems are happening when it comes to these rogue AI incidents. All right, so if we imagine we had to zoom in there multiple times. That's three big zooms to smaller and smaller regions of the AI landscape before we get to the very specific type of system, where we get things like the hugging face attack or the anthropic agent filling out illegal police forms. All right, so we aren't dealing with AI in some general sense going rogue. We're dealing with a very specific type of LLM-powered agent that I'm going to call an open-ended LLM-powered agent. To go forward here, I want to give you a brief primer, which I've talked about before, but I want to return to here about LLM-powered agents before I talk about what specifically makes an LLM-powered agent open-ended. All right, so for those who don't remember, when you hear about an LLM-powered agent, it's easy to conjure some sort of notion of a complicated program that is sort of approximating the human mind that's constantly learning and making decisions and morphing maybe and evolving and swarming and doing all sorts of scary things. But actually, at the core of LLM-powered agents is a computer program. So I'm going to draw this on the screen here for those who are watching. So we have a computer program. I'll put it over here. I'll put P in the middle of it to represent a program. This is a normal sequential computer program written by a human and a normal programming language is completely understandable by humans. In fact, a lot of agents, the program itself is really not that interesting. All right, so what does this program do? Well, it has access over a network connection to a large LLM. So I'll label this big green blob over here and LLM. Okay, so there's an LLM over in some data center. Even that's not even that accurate. There's not like a machine that hosts an LLM. You have many racks of GPUs where each rack of GPU is perhaps virtually simulating some subset of the weights in the LLM and different queries will go to different GPUs to have the same weights. There's no actual system where the LLM lives. There's just a bunch of copies of the weights spread out over a bunch of GPUs. Okay, so what the agent program does is the following. It will send a prompt to the LLM. I'll draw an arrow going from the program to the LLM. That prompt is just in text. Well, in essence, say, I'm an agent. Here's my goal. Here's the tools, like the software tools I have available. The tools I can call. And here's my current status. What should I do next? So it submits that to the LLM just like a chatbot user would type in a prompt to send to a chatbot. The LLM as LLM's do generates a response sends it back. Like this is what I think you should do next, given your goal status and current tools. The program then executes whatever it is that the LLM suggested. Because remember, I gave it constraints. Here's the tools I have that I'm able to call. And so the LLM response might say like, great, you should move this file from this directory, that directory, using the MV command that you told me you have access to, right? So then the program actually executes whatever step the LLM told it. Then we loop. And we do that again and again and again, right? So this is why I sometimes call LLM-powered agents prompt loops because I think that gets to the core what's really going on here. It's a program that repeatedly prompts an LLM says, what should I do next? Does that step repeats? All right, here's my new status. I'm a, you know, I'm a program. Here's my tools. Here's my goal. Here's my status. Why does it have to repeat itself so much? Well, remember, LLMs have no memory. Nothing adjusts their static. So every prompt that goes to that LLM has to explain everything to LLM needs to know to suggest the next action to the program. That's what an LLM-powered agent actually does. Now there is an issue with these that people have known since the beginning of messing around with them, which is using an LLM as the sole driver of your actions as a computer program. So I will do whatever an LLM tells me to do. This is risky business. And anyone who's used a chatbot knows why. It's because LLM outputs are engineered to be plausible, right? So they're always a plausible response to whatever input you give it. They understand well, they're tuned to try to produce a plausible output. I've explained these mechanics before. It goes all the way down to their pre-training, which is about guessing tokens from real text. And if you do that auto regressively for many tokens in a row, what you get is a plausible extension of the original input prompt. So they give you plausible answers. Whatever it spits out is a reasonable extension of whatever original prompt you gave it. But these answers are not normative. Normative meaning some coming or a bane, the sort of like natural norms that a human would think makes sense when you're trying to give an answer, right? So they don't necessarily have human morality. They don't necessarily have notions of truth or falseness. They don't have a notion of maybe it's bad to make up a citation in a legal case like that super illegal. There's no notion of that. There's no world model or moral core or policy actor that's trying to check its answers to see if it passes muster. It's just giving you a plausible answer. That could be tricky when you're changing these responses into actions. Because what's going to happen is they might start suggesting actions, which are completely reasonable actions that will work and will make things happen, but are not normative. They might be unexpected. They might be unpredictable. They might be a moral. They might be illegal. Anyone who's used a chatbot enough knows this, right? You chat enough with a chatbot. They occasionally lock in on weird dark directions or unexpected directions or you have to go back and say, no, no, let's try this again. When you're an agent, the problem is there's no human. You're not a human. The agent's just a program that's blindly executing with the LMSS. You can get into weird places if you run this prompt [BLANK_AUDIO] experiment of let me blindly execute whatever output an LLM gives. And yet when I was showing you the landscape of LLM agents, I said there's multiple types of agents that exist that don't cause much trouble. So we had, for example, OpenAI's Astra-based math agents that were involved in the recent math-slop drop of those 322 somewhat inscrutable lean proofs. Only the LLM involved with these agents is very large and very scale. They trained it on math extensively and it was able to do very advanced mathematics. So by some notion of smart as a very smart agent, they also ran these things for a very long amount of time, hours at a time. So thousands of prompts trying to explore these problems, you never went rogue. OpenAI is not saying, "Good news, bad news, good news, we got a finite time blowup of the Navier Stokes equations, bad news is on the way the agent also ordered a bunch of hookers to the OpenAI headquarters." That didn't. It just worked fine. Same thing with coding agents, LLM-powered coding agents are widely used, millions of software developers use these every day. Many of them are connected to some of the most advanced elements we have and they run many autonomous steps. And we also don't hear a lot about coding agents being used by professional software developers, growing rogue and trying to hack into other people's servers or doing things that are well outside of what the programmers thought that they might do. We even have data about this, I'll bring it up on the screen. OpenAI in March actually released a study called How We Monitor Internal Coding Agents for Misalignment. So remember, agents prompt an LLM for all of their actions. If you want to see what an agent is doing, you can just look at all the LLM prompts. In this study, OpenAI looked at tens of millions of prompts to LLMs coming from agent programs and what they found is the types of rogue behavior they most fear. They call high severity incidents like sandbagging, sabotage, scheming, the stuff that made us nervous about hugging face didn't occur at all. So coding agents are not having this problem either. They want to take a quick break from this conversation to cool down and to talk about some of the sponsors that makes this show possible. You all know that I love working with CozyEarth because I'm a giant fan of their products. Now recently, I have been using, and when I say I, I mean my family because they keep stealing it from me, the CozyEarth bubble cuddle blanket, which is a perfect combination with their famous bamboo sheet sets which were obsessed about and have multiple pairs of and travel with because we like them so much. The bubble cuddle blanket, it's soft, it's heavy, you really can cuddle in it and it just gets passed around my family, especially now as it's getting colder in the mornings as fall advances. There's always a competition for who gets the bubble cuddle blanket. It's the perfect addition to their already fantastic bamboo sheet sets. So consider picking up a CozyEarth bubble cuddle blanket and for sure you need some of their bamboo sheet sets. Get them today. Look, there's no risk in giving this a go because they have a 100-night trial and a 10-year warranty. If you don't love them, you can send them back, but you won't because they're great. So go to CozyEarth.com and use CodeDeep for up to 20% off home and sleepwear. That's CozyEarth.com CodeDeep and when they ask you where you heard about them at checkout, let them know you heard about CozyEarth right here, the Deep Questions podcast. I also want to talk about our friends at Gusto. Look, no one starts a small business dreaming of all the payroll tax forms they're going to get the file, right? That's the type of thing that you procrastinate on because it's boring and it gets in the way of actually growing. This is where you need Gusto to handle those type of administrative tasks in the background so you can focus on what really matters. Gusto is an online payroll and benefits software tool built for small businesses. It's all-in-one, remote-friendly and incredibly easy to use so you can pay, hire, onboard, and support your team from anywhere. You get unlimited payroll runs for one monthly price, no hidden fees, no surprises. You can also save time with built-in automated tools like offer letters, onboarding docs, direct deposit, and more. This ranked number one on G2's highest satisfaction products list for 2026 and trusted by over 500,000 small businesses, including the ad agency we work with here at the podcast. So what are you waiting for? Get started and try Gusto today at gasto.com/deep and you'll get three months free when you run your first payroll. That's three months of free payroll at gasto.com/deep. One more time, that's gasto.com/deep and let's get back to the show. So how do those type of LLM agents avoid trouble where other types of LLM-powered agents like the ones we heard about this summer don't that they do cause trouble? Well there's three things that keep the, I call these targeted LLM-powered agents like coded in math agents. There's three things that keeps them on the reservation. One is narrow goals. They're given a pretty specific goal. Like if you're a math agent, it is, I want you to, you know, write this part of a proof in latex which is like a markup formatting mathematicians used to write up math papers. You're right up a proof in this format into this file. It's a narrow goal. Turns out when you give LLM, say give me a step to make progress towards this narrow goal, they're trying to give you steps that help you make progress towards that narrow goal. They're not going to go off and do something rogue necessarily. The second check that helps keep these targeted LLM's from going rogue is they have a restricted tool set. So the tools the agent program has available to actually take actions in the real world is appropriately restricted. I mean if you're a math, if you're the math agent, maybe you have the ability to do searching on archives and math papers. I'm not quite sure if that's all trained in or if they could also do real-time searching, contextually, on relevant papers. And you have the ability to read and write files. You can actually write proof text into a file within a limited directory structure. You have limited tool sets, you can't do much damage. They also have oversight. So this is particularly true of the coding agents. If you did write a very simple coding agent, gave it tools for the terminal and just said, go on and write this code, they will get in trouble, not in the rogue sense, but just they'll do unexpected stuff. They'll overwrite files, they'll delete files they've shown and have. And so a lot of effort has gone into the agent programs for coding agents. Give it as hard-coded oversight to automatically notice and avoid the most common types of issues that a coding agent might accidentally do. We know this in part because the code, the source code for anthropic rather, unthropics agent clawed code accidentally got released. It's 500,000 lines of code in the program that prompts the LLM and a lot of it is just hard-coded special cases and checks. They have pattern matches in there. This is just through experience. The programmers who worked on it are programmers, so they kind of know the types of things they'll look out for. So there's all sorts of hard-coded special cases in that program. Hey, if it says RM-RF, which means delete a bunch of files without asking, they've hard-coded something to say, eh, maybe don't execute that. We'll put that one aside. They actually have pattern matches. Interesting. They have pattern matching in the clawed code harness to actually look for common terms from whatever the user is typing in, that indicate frustration, like cursing. And if it sees there's frustration words in the user's prompt, the program will assume, I don't know what the LLM has been telling me to do, but it's probably pissing off the user. So why don't we stop now and sort of check in with the user before we take any more steps. There's also a lot of just hard-coded checkpoints in the coding agents. For simple programs, like amateur people making interfaces for their clubs that they're part of, that they vibe code, that's fine, but for professional programmers, they're checking it in way more often, they have recursive testing going on. They want to check in. All right, tell me what you did here. I need to check this. I need to run test on it. So it's a lot more interactive than you would think. So narrow goals, restricted tool set, and hard-coded oversight can make for the right task, LM-powered agents, never go rogue. So why are we having rogue agents still? They all come from what I've been calling open-ended LLM-powered agents. And why did they have trouble where these targeted LLM agents do not? First of all, they replace narrow goals with broad goals. Look at the hugging face attack, right? They're given a lot of latitude. Like here's your goal, but you have sort of complete latitude in how you make this happen, because that was the nature of many of the agents involved in the hugging face attack were being used, not all of them, but a lot of them were being used as part of a cyber security benchmark suite known of exploit gym. Where the whole idea is to be really creative and find whatever way you can to try to accomplish the goal. Typically, it's like breaking into a test system, right? So they have broad goals. They have a broader range of possible actions. They're not being constrained. They're given a powerful tool set, right? They're given tools, these powerful tools you need to win at hacking challenges. So now they have very powerful software. And in particular, they have access to these package services where you can download special packages. is to help you do particular types of attacks. So it's like, I'm trying to think of a good analogy here. The good analogy would be if you have in targeted agents, you're giving two or three tools to the agent from your toolbox, these open-ended agents, you give them a credit card and access to ACE hardware. They can just go get whatever they need to do. You can get in the more trouble that way. And finally, they have very limited oversight. First of all, because these are test being done rapidly, unlike a coding agent where you have a year or two of people paying stakingly hard coding in special cases, these they just rock and roll, right? They're just like, let's just see what happens. And on purpose, if you look closely like at the Redwood or Mutor Report on the Hugging Face Attacks, the engineers just let these things run. They weren't really trying to check in. They had very little monitoring going on. You very easily, there's all sorts of checks you could have easily put into the agent program itself. Like every time I'm trying to access a new domain, let's just stop and have a human look at it. Let's have a human check every hour and look through the log of all the prompts to the LLIMs. Let's even just have another LLIM summarize that. And like, let's just check every hour to make sure that we don't have something unusual going on. They almost like purposefully didn't want to have oversight. Not only did they not want to have oversight, but as the reports make it clear, they would see that something really troubling was happening and they would just restart and let it run again and come back a few more days. They had reports from previous experiments because OpenAI, their new misalignment reporting framework has them say when things were detected and when they actually occurred, they had detected and have reports of, "Hey, these agents are doing unexpected unpredictable activity outside of the sandbox." They had those reports already before they started new experiments with similar agents that then went on to do the same sort of thing. So it was almost like purposeful lack of oversight. So if you have really broad goals, you say you do what you need to do to solve this problem. We have basically no constraints on it. You give access to a wide diverse and powerful set of tools and you have very little oversight. Those type of LL-empowered agents are gonna cause problems. Why are they gonna cause problems? Is it because somehow this static LLIM distributed and copied over thousands of GPUs is actually some sort of coherent, sentient entity? No, it's because LLIMs are plausible, but not normative. And if you're just gonna keep asking an LLIM again and again, what should I do next? They're gonna give you weird responses eventually. They're gonna lock in the past. You wouldn't expect they're gonna be non-normative. They might do things that are harmful or illegal. And if you give them broad goals, powerful tools and no oversight, there's nothing stopping that from unfolding in ways that's gonna be unpredictable or harmful. The key point I wanna make about this is there's nothing inevitable about the AI industry needing to design and deploy open-ended LL-em powered agents. It would be completely reasonable, for example, in this current moment for the industry to say, that's a bad idea, but you know what? Targeted LL-empowered agents, we're finding use cases for it. We're getting better at these coding agents. We're figuring out math agents. They're a little inscrutable right now, but they're powerful, so clearly we're gonna be, give me a year or two, we can give great tool for mathematicians. There's gotta be a hundred other sectors where we can build targeted LL-empowered agents that are, you know, they might sometimes do stuff. You don't expect that aren't rogue and are trustworthy and generally do good stuff for you. And you could build out a really wide industry, just like the original software industry of lots of different tools, each of which tune to a different task or industry, and each of them bringing in revenue. And you could build a really solid, very like lucrative technology industry on top of that, right? There's nothing wrong with that. You could also then say, hey, you know, these targeted LL-empowered agents, this is maybe not the most effective way to do it. Actually, what we really wanna do for a lot of these tasks is neurosymbolic systems to borrow a term from Gary Marcus. A lot of the original AI landscape I showed you where LL-ems were just one subset of it. A lot of those systems like the systems that play games very well, the systems that drive cars automatically, alpha-fold. These are neurosymbolic systems. It's not a program just prompting an inscrutable LL-em and just doing what it says. They have some neural networks in there that are trained with machine learning. They have some symbolic components in there. These are programs written by humans or understandable by humans, and they all work together. Turns out if you do neurosymbolic systems, you don't need a five trillion parameter LL-em that has to use up all the water of Idaho just to run. You can actually fit them all in a single chip I can run on my basement. And so there's this whole exciting trajectory we could have of targeted use cases where we could use targeted LL-em base agents and neurosymbolic agents to bring the price down and the effectiveness up even more. And boom, it's a massive industry makes all of our lives better. There's no reason. We don't need open-ended LL-empowered agents for AI to be useful to people. And yet, we see an insistence on building and testing them. So why is this? Well, first of all, I think we have to clarify, it's not everybody building open-ended LL-empowered agents. The two companies in particular that are really pushing this research hard is open AI and anthropic. Now to be clear, this doesn't mean that no other hyperscaler style AI lab is working on these. We know other companies have had some issues with these meta and Google. And particulars had a little bit of issues. But they're kind of, I don't know, I feel like the cases we've had of Rogai from Google, for example, I'm not super worried about, they were doing some open-ended agent testing, but they were more constrained for cyber security. They sent it off to this company called a regular to run the test and a regular set up the sandbox wrong and gave it full access to the internet. And it tried to get some passwords or something, right? And I think they're like, OK, whoops. You guys said, wrong, let's not do this. Meta had a similar issue, and then they kind of backed off it a little bit. But open and anthropic are pedal to the metal. This is where the big, especially open AI, this is where the big case after case of months of actual ongoing hacking, they are doing this at a level that is, in my, I don't know, have the numbers, but seems like 10X, what anyone else is doing. So those two companies in particular are super insistent on open-ended, ill-empowered agents. Come hell or high water, we got to keep running those things. Why is that? Well, if we look closer, open AI and anthropic do not like that alternative future that I just gave to you. They do not like the idea of a more normal AI technology landscape where you have a huge number of AI task-specific AI systems, like coding agents, like the math agents, just multiplying that across 100 other different types of task areas. And you have all these systems. A lot of them are ill-empowered and targeted. Some of them neuro symbolic powered. And there's just a lot of really useful AI tools in the world. They do not like that vision. And there's three reasons why. Vision number one, why those two labs, and particularly don't like those, is competition, right? That vision, when you get task-specific to targeted LLMs, does not require that you have the biggest baddest LLM on the block. We see this with coding agents. I mean, this has already become a problem for open AI. And in particular, anthropic that makes a lot of money from coding agents is that you can use smaller LLMs. You can use cheaper LLMs. You can route to different LLMs based on the complexity of the coding task that you're doing. My prediction about the coding, the targeted LLM-based agent, coding agent space is that we're going to see a new generation of very specialized, pretty cheap LLMs. I think that the LLMs are going to be able to run on a rack. You can put in your own server room at your own office that have been super tuned for particular languages. I'm a big believer talking to a lot of programmers that the next evolution of coding agents is actually going to be the emergence of a sort of intermediate pseudocode language that is forgiving. It's not syntactically demanding, but it's pretty accurate. And a programmer can write really quickly in the pseudocode and get much more exact translation from that in the code. They're by significantly reducing the amount of checking and verification that's required right now, where the code production is a little bit more scatter shot and inscrutable. This is going to be able to run on much smaller LLMs. All of that's bad if you're in the business of building giant LLMs, right? So you've got to understand. Anthropic and open AI only make money when you need their newest, latest, greatest LLM to make progress. If a cheaper LLM is enough, you're not going to them. They're Ferrari, right? And these cheaper LLMs are Honda. And they want to keep people wanting to buy Ferrari's even though they're sort of ludicrously expensive. So they don't like that vision. They hate the idea of neuro symbolic systems because yeah, those might have neural networks in them, but they're small. Man, you look at these superhuman game plane engines. They can play games better than humans. These things, you small neural networks compared to what's going on over at OpenA and Anthropic. I mean, you can run, you know, Dreamer V3 fits in the memory of a single GPU. You can run stock fish, the chess engine that can play chess better than a human. You can run that on a laptop. That's scary to open AI and Anthropic. So that's one reason why they don't like the vision I gave where you don't need open-ended LLM agents. Reason number two is they've been convincing the military in particular that open-ended LLM-powered hacking agents, sure they might be unpredictable, sure they might break into places you don't want, but weapons are dangerous and this is a weapon we need. So they have. this sort of military industrial complex type argument that open-ended, LM-powered hacking agents in particular. So this particular type of open-ended agent has to be in our nation's arsenal. And so that's another reason why they think we have to have these. We need things that can run for days at a time and run these sort of complex hacks. I think that's highly suspect. I think Sintor hacking and defense where you have highly trained cybersecurity experts working alongside models and much shorter horizon agents that have been trained on cybersecurity is much not only safer but much more creative and powerful and effective than just letting an agent run by itself for days at a time. I don't buy this idea that if you don't build this highly unpredictable combustible long-term agent yourself that if someone else does you're somehow less prepared to defend against it, which I don't think is true. But that's one of the reasons why another big reason they give is we have this particular cybersecurity market. The government spends a lot of money. The military has the most secure budget of anything. We want in on that. All right, reason number three why they don't like this vision. And this is probably the most foundational is that open AI and anthropic and particularly anthropic, but open AI and anthropic, believe that you need open-ended, LM-powered agents to realize their vision of superintelligence. Remember those two labs, this is different than almost any other worldwide or in America companies working on hyperscaled LLMs. Those two labs in particular were explicitly founded with the mission of trying to usher in superintelligence safely. Their vision is superintelligence will either be a vengeful God that kills us all or it will be a benevolent God which will give us a transhumanist utopia. And they feel like their job is to make sure that we get the good option. And remember we've talked about this in recent weeks. The people that are yelling, the Jacob Croxton's of the world, all of these people funded by effective altruist money who are yelling and influencing Congress people about we need to regulate the prevent superintelligence. They're all from the same family. They only differ in the fact that Dario Amade sort of thinks we're ready to do it and other people in his family are thinking he's not ready to do it yet. And actually even it's more darker than that because actually the more you push at this, the more you see a lot of these EA funded voices that are pushing for this type of regulation. They're actually on the same team as Amade. Because what is this regulation going to deliver if you really look at it? A regulatory, a regulatory capture landscape in which essentially only Papa Amade gets to build the big LLMs and the open into Ellen powered agents. Him working with third party auditors, which are other firms like meter that are founded by the same people funded by the same people that funded and thropic. They all know each other. They'll come in and help monitor them. And so basically it's it's all sort of an effort in some sense to be like, yeah, we're trying to bring in the benevolent AI God to get our transhumanist future. So we need to stop other people from messing around with this. Let Papa Amade do this thing. And billions of these sort of biological bootloaders might die, but we need to get, you know, we need to get our consciousness uploaded in the digital beams that are going throughout the light cone. So they really feel like you can't get recursive self-improvement, which is going to deliver superintelligence unless you keep pushing aggressively forward with open into the LM agents, because long horizon, autonomous activities, what you're going to need to have their LM powered scientists that can do their own automated research and push AI forward first. So they get to the superintelligence first. They get to it first. They control the God. We have a chance of transhumanism utopia. If they don't, then maybe someone else will do it, or we'll lose the chance for the utopia altogether. So there's three reasons why they don't like a future where you say open and LM powered agents don't work. They're dangerous. It's a small part of the whole AI landscape. Stop doing that. We can have a huge thriving highly lucrative, useful AI industry, but they don't care about that. All right. So what should we do? The rest of us, the normal people, not the, because remember, what is this argument? Right now, all of this debate is between billionaires. Right? We have the billionaires funding the effective altruists who are also funding the labs. You have billionaires like Dario Amade, fighting with other billionaires that are from these effective altruist organizations, trying to figure out together how they're going to usher in this utopia. And then you have the accelerationists, which are billionaire venture capitalists that are arguing with them. And so it's billionaires fighting with billionaires. It's like a hundred people. They're all really rich. What about all the rest of us? We are very happy with a future like I just described. Most people would say, yeah, do that first thing where you don't do this one type of experiment that's unpredictable and harmful. But you do all these other things where you're building really useful AI tools that industry after industry task after task makes our life easier and creates a lot of jobs. And it explodes to sort of tech industry to be even bigger than it is now. That sounds great. That's what the regular people probably want. It's the billionaires who are like, forget that. We're trying to, for the accelerationists, either get another billion or for the EA Dario Amade types trying to get to the utopia. So we're just letting billionaires argue with billionaires about who gets to control the world. Well, everyone else just has to sit here, be stressed out and say, excuse me sir, but none of this helps. What are you doing? You're not even trying to make a product. You're just trying to, it's just like done just and drag like, what are you guys doing? It's just like twilight and period made real. Are you larping? Like what's going on here? So I think the regular people need to actually push back. And here's how you do it. There's two things I want to suggest. Number one vocabulary. Refuse to play the game of using the word AI when what we're really talking about is open into Ellen Powered Agents. Be specific. It puts the pressure back on the companies. When the company say, AI is getting too powerful, AI is getting out of control. You imagine it's like a, as you know, one of my producers, Nate said, it's like a Pokemon that's evolving into bigger and more dangerous configurations as the game unfolds. And all you can do is try to slow down the rate of this one beast from growing. That's not what's happening. It's a thousands of things we do with AI that are really useful. It's this one thing that's a problem right now. So use the right vocabulary. Don't say I'm worried about AI going rogue. I see that in the opening paragraph. And if you look in major newspapers, every article at any of these attacks, always first paragraph, AI going rogue. No, no, no, no. Open ended Ellen Powered Agents once again, have shown that they are unpredictable and can cause harm. The more specific we are with that vocabulary, the more pressure it puts on those two companies to say, why are you doing that experiment? So I think vocabulary really matters. Number two, when we're advocating for regulation or control, the goal for regulation, the average person has to speak up and say the goal here is to maximize useful products from AI while minimizing harm. That is what you typically want at a market regulation, maximizing utility to the public while minimizing harm. That's what you want from market regulations. The regulation debate that's happening on the Hill now, which is powered by the the effect of altruist billionaires and their sort of silent partners in the billionaire founders of the AI companies. The type of regulation they're talking about is not about maximizing utility for the everyday person. And it's not necessarily about minimizing all the harms you would want. It's about consolidating control of LLM progress so that Papa Amade can be the successful shepherd to bring us to a transhuman utopia. So if all of these EA sponsored bills get passed, these super intelligent bills, like you would see Senator Sanders standing next to Nate Sora as proposing, if these things get passed, it's just going to lead to a world in which we consolidate most of the LLM-powered agent work to basically anthropic, open AI is kind of a mess, but maybe also kind of open AI, but basically anthropic. And then with the help of their third party monitors, which are friends of theirs funded by the same people will then be able to do whatever they want to do with these open-ended LLM-powered agents. And when they go off and do even worse things as they give them more powerful tools and more runway and more long horizons, they'll just keep saying AI is unpredictable. We're doing our best to keep it on their control. So the regulation the billionaires are talking about does not maximize any utility. The utility they're maximizing is increasing the probability of getting to a transhumanist utopia. That's not what the public wants. We didn't ask for that. And they're not directly minimizing the harms you get by running open-ended LLM-powered agents. So we need the demand as the people, not the billionaires, but the people. We need demand actual market regulation. Create a market that creates useful tools. These hacking monsters aren't useful to us. This is, it's a fever dream of the of Dario Amade and vision is transhumanist utopia. Running these type of open-ended LLM-powered agents is harmful to us. So why don't we stop that until you can show us you have a more controlled way of doing that. And let's focus on what's gonna get us a lot of competition with targeted LLM-powered agents, neurosembolic agents, test specific agents that are completely controllable, completely understandable and very useful. Okay. So this has become my new call for people. I really think this is what this is really about is billionaires versus everyone else. We got to put it that way. The billionaires want to be like, oh, we're on your side. We're trying to prevent, you know, term, remember terminator? You don't want that. The robot stepped on the skulls. Remember that? The robot and terminator stepped on the skulls. We don't want that. Where we've got your back. And we need to be like, Yeah, we saw that movie, Stan Winston made that, that was an animatronic. We want useful products, we want you to stop building harmful things. So there's a analogy I want to give to kind of end with, right? Because I want to summarize this all together. I've summarized all this together when I'm really saying this AI is a very large topic, and there's lots of different types of systems. Many of them very useful. And you got to zoom in three times in that landscape to get to the one type of system that's really causing a lot of these AI goes rogue concerns. And there's no real major reason to be running those type of systems, except for these really unusual eccentric and self-interested concerns of the billionaire utopians. It's not inevitable that AI, as it gets more powerful, goes rogue. As AI got better at chess, it didn't go more rogue. As it learned how to predict folding protein patterns, did not lead to the design of rogue, it didn't go rogue and tried to build its own designer organisms, AI getting more powerful does not mean it has to go rogue. It's this specific type of system. So here's the analogy I want to leave you with. Maybe we'll make this kind of clear. Imagine that Elon Musk released rocket-powered roller skates. I'm using Musk because it feels like something he would release. Rocket-powered roller skates and a lot of people get hurt because you're putting rockets on your feet and they're crashing in the walls and breaking their bones or whatever. And it's kind of like an uproar about this. Like a lot of people are getting hurt using rocket-powered roller skates. Now imagine that Elon Musk came out and said, "I have grave concerns. I believe we have to pace the frontier of real transportation because people are breaking their bones. It's only going to get worse." We would say that's ridiculous. The car companies would say, "What are you talking about?" The bicycle companies would say, "What are you talking about?" The scooter people, the skateboard people, they're like, "Are wheeled transportation aren't breaking any one bones? They've worked just fine. We have safety regulations. They're very useful. People really love them. Maybe you used to stop building rocket-powered roller skates. That's really what's happening here. And if Elon Musk came back and said, "Yeah, yeah, yeah, but you don't understand. We've spent the last 20 years on blogs talking about how rocket-powered roller skates are part of this prophecy that leads to, it's going to lead to a utopia." We would say, "Okay, Mr. Musk, we have this nice jacket for you to put on and there's some straps in the back. Don't worry about that. This room has pads. It's very nice. You're very cool. Right? That's what we would do. But that's essentially what's happening right now. These small number of companies are producing the AI equivalent of rocket-powered roller skates. A lot of people are breaking their legs and they're saying, "We have to slow down transportation." In fact, not only that, but let's let Dario Amade be in charge of all of transportation because like we got to be careful out the person who built the rocket-powered roller skates. Let's put them in charge of all of it because like we can't trust anyone else. I'm kind of straining this analogy a little bit, but hopefully that makes a little bit of sense. Anyways, I don't mean to get fired up, but I do think this has become, it's a populist argument. I mean, it's people versus the billionaires. The people got to stand up. We're not going to use your new speak. We're not going to call everything AI. You got to tell us what systems have problems. And we're not going to accept calls for regulation that don't make our life better. Useful products minimize harms. Not. You can't regulate sci-fi stories. Market regulation needs to create more useful products. Minimize harms. All right. So this is why I do not think AI going rogue is inevitable. You can create AI that goes rogue and it's a problem. But we need to look and say, stop it. We need to stop just accepting whatever these small groups of really well-funded people are telling us and say, we've got eyes. Those roller skates are stupid. I'm going to go get my bike. All right. That's enough of that. That's my rant. That's my rant for today. I don't know, Jesse. It's a lot going on. It's a complicated topic. There's so many different things involved. There's complicated technology. There's politics. Ideology is the whole thing is really complicated. So I do my best. The graphic hell is put in perspective. Yeah. You got to zoom in a lot. Yeah. Like if DC, the city of DC where we are right now, if that is the AI landscape, these open-ended, LL-empowered agents are causing all the rogue problems. It's like downtown Tacoma Park. The whole rest of the city is other types of AI. I love the Papa Amade reference. That's really that's all of this regulation. If you look at it closely, it all ends up with like Papa Amade and his happy elves at meter and redwood. Who are his friends who are funded by the same people? They'll take over. Don't worry about it. They'll make sure everything goes okay. I just think it's crazy. Billionaires regulating the other billionaires, deciding which of the billionaires gets to be in charge of this technology. And they all have this weird ideology about utopia. So it's like, this is not a great situation. So Amade and Almen, are they still friends or they don't? They don't like each other. They used to work together, right? Yeah. Well, Amade left open AI because he felt that my theory on Almen, I got these things. My theory on Almen is kind of based on Runafera's and Andrew Marne's reporting for the New Yorker. He's not a great executive. Yeah, I wrote that too. Right. He was in, this is not someone who started a startup from scratch and made into a big concern or whatever. He sort of was just in these circles and was made the president of Y Combinator, which is an incubator, like that's not, you know, they basically just take shares of kids' companies and some of those companies go big and they cash to checks. And Elon Musk was just like, hey, I'm going to start a nonprofit that makes sure that we don't get bad super intelligent AI. Do you want to be in charge of it? And he's like, all right, man. Cool. So he's a little bit more slippery. Amade is, I think, a much more, I don't know if he's a good executive. I've heard that he's very isolated with an anthropic, so I don't know what degree he's executing over there, but he's a much better technical mind. He's from a physics background. He's a very smart guy. And anthropic is actually, they're making, I mean, both OpenA and Anthropic have made progress on these models. And anthropic has probably had a higher density of high quality technical breakthroughs. So I think they're running a tighter ship over there as a technical company. But they left. Amade left OpenAI because Altman was being insufficiently committed to the billionaire vision of transhumanist utopia, like shepherding in the right AI and preventing the others. They're like, you're not doing this right. We'll do it better. And then Amade's sister is at anthropic too, right? Yeah, they left together. Danielle and Daria left both left to start. Yeah. And she's married to a very prominent effective altruist. I mean, he came out at some point and said, I'm not an effective altruist. His entire life has been an effective altruist. It's crazy. It's great. I mean, it's like Ronald Reagan in his 60s being like, I'm not a republic anymore. But you're a Republican, right? Like you can say it because your investors don't like it, but you're a Republican. I mean, the way people understand the percentage estimations of doom, that's straight. That's just like the shibboleth of effective altruism. That's how they talk in the Bay Area. That's how you know you're talking to effective altruists. It's like my p-dume is 23% and Amade has been like giving these precise numbers all along. That's why Jacob Croxton talks that way. It's a complicated world, but it's also kind of a simple world. So it's like we have a double whammy. Americans get worried when you have billionaire oligarchs trying to control things that matter. But usually with the billionaire oligarchs, the main issue is they just want to make a lot of money and they're going to be exploitative of markets to try to maximize their wealth. You add in an eccentric ideology to the billionaire oligarchs. It's not good. No, we've had that before. If we go back in like the late 19th, early 20th century, there wouldn't be billionaires back then. They'd be like symptom millionaires, but you had very powerful people with lots of money, consolidating industries. You had like a lot of thoughts about like foreign policy, and it's like, okay, we're going to start wars and we, you know, whatever. We've had this before where you have concentration of economic power confined with personal ideologies that are very strong and maybe eccentric, and those two things together is always cause a trouble. Always cause a trouble. And so it's not that surprising what we're seeing, except for its camouflage, well, and people don't realize all these players are funded by the same billionaires. They all, it's all the same tech billionaires are funding all these players. They're all from the same playbook. They all have these guaranteed jobs that these giant think tanks that they're funded. They can just bounce back and forth between. They don't know that you'll say like the prominent, like why did this prominent senator and what name name? So why did this prominent senator, like suddenly is straight out of like E.A. rationalist talking points? His top aid left, went to one of these effective altruist institutions, came back a few years later and was like, hey, senator, this is the biggest issue now. And the senator's like, all right, I guess this is what we got to care about is like super intelligence. So people are just the pawns of billionaires. Billionaires have, this is what they're caring about. Like we want to make sure that we don't have the bad AI, we have the good AI, and the good AI, some of us will be able to upload our consciousness eventually to the light cone and guess who those, someones are going to be, it's going to be, it's not going to be you and me, Jesse. We're not the one. We're the biological bootloaders to use Elon Musk's term that our whole point is to get to the point where we can get, you know, maybe Dustin Mosquitz's brain not ours is going to go up the space when the time comes. Let's just put it that way. We want to take another quick break to hear from some of the sponsors that makes this show possible. Look, here's a stat that genuinely shocked me, sales team spend about 50% of their time on admin work, rather than selling building relationships and closing deals. What if instead every sales rep could watch? to call is briefed, can stay focused instead of taking notes and update their CRM's the moment to call ends. That's where today's sponsor, pipe drive, comes in. It's an intelligent AI-powered sales CRM loved by growing sales teams. The save you time pipe drive remembers your deal history and can create a pre-meeting brief that automatically pulls email records and previous conversations in the one brief ready before every call so you always know exactly where things left off. Their new AI note taker is also a game changer. During your meetings, it records and takes notes so you don't have to, you stay in the conversation, pipe drive handles the rest. So switch to a CRM built by salespeople, four salespeople, and join the over 100,000 companies already using pipe drive. My link gets you an exclusive 30 days free instead of the usual 14 day trial, no credit card or payment needed just head to pipedrive.com/deep to get started. That's pipedrive.com/deep and you can be up and running in minutes. Let's talk about our friends at Fabric. It took me longer than it should have to get the life insurance I needed to protect my family. Now what was the problem? Getting life insurance can be a pain. Unless you use Fabric by Gerber Life. Fabric by Gerber Life offers term life insurance you can get done today made for busy parents like you all online on your schedule right from your couch. You could be covered in under 10 minutes, often with no health exam required. If you got kids, especially if you're young and healthy, the time to lock in low rates is now. If you have life insurance to work, that's a good start, but it may be less than what your family needs and could disappear if you change jobs. Term life insurance from Fabric stays with you. Flexible policies built to fit your family and your budget and as long as premiums are paid, coverage will follow you wherever you go. There's no risk. You have a 30-day money back guarantee and you can cancel at any time plus free digital wills and tools to help invest in your kids' future all are available right from your phone. Fabric has partnered with Gerber Life Insurance Company who has trusted by millions of families for over 50 years and rated excellent on trust pilot with over 2005 star reviews. Join the thousands of parents who trust Fabric to help protect their family. Apply today in just minutes at MeetFabric.com/deep, MeetFabric.com/deep and use my link so they know I sent you M-E-E-T Fabric.com/deep. Policies issued by Western Southern Life Assurance Company Cincinnati, Ohio, which operates in D.C. and all states except New York and distributed by Gerber Life Agency LLC using Fabric Technologies, see MeetFabric.com for additional disclosures. This is a paid ad. All right, let's get back to the show. All right, enough of that and obviously I am oversimplifying. Everyone's getting out. The political people is saying you oversimplifying it. I am. The technical people are going to say you're oversimplifying agents. I am. But the core concepts I'm trying to get across here so hopefully that helps. All right, so on Mondays we like to also hold some office hours right here what you have interested in. As always if you have comments, questions or reactions you can send the podcast at calnewport.com. I figured because I was in a bit of an ordinary AI mood today, we're going to do questions just about AI from our reader. So we'll see what it is they care about. Hopefully this will call me. These will all be calming questions, right? Right, Jesse? Dustin Machvitz. What's it, Machvitz? Muskowitz. He's the Facebook guy, right? One of the Facebook guys. He was an early Facebook guy. Yeah. Got out early but had his brains going to space. A lot of EA money. These fun to good things too, but these brains going to space. Not ours. We're the slurry. We'll be used for the slurry that keeps alive the biological bootloaders until we can get the consciousness to space. All right, what's our first question? Our first question is about the legal implications of Rogue AI. All right, so this came from Aaron who said this is a good succinct logical case for prosecuting AI companies when their agents do things that people would be prosecuted for. All right, so Aaron sent a link to an article, a blog post was titled, "Why aren't AI companies being charged for hacking?" I'll read you. I just have it here, an early paragraph from the blog post. You have a sense of what it's about. So early on in the blog post that Aaron sent me, it says, "The big AI corporations are hiding behind a responsibility gap argument due to the emergent behavior of their systems. It's a position tailored to dissuade prosecution as it removes intent, which is typically required for a crime. I understand their systems, though, so I will show how they absolutely intended this behavior, giving a framework for legal evaluation. I like these type of questions. I think the legal questions surrounding liability from open-ended LLM-powered agent attacks. What are the actual legal issues involved? I think these are interesting discussions. Using the sort of moderator's prerogative here, though, instead of going into the details of this blog post, I actually wanted to look at something that a legal expert put out. So I'm going to bring us up here on the screen. This is the testimony from my colleague at Georgetown, Paul Ome, who's a professor at the law school at Georgetown and a world expert at tech law. He recently testified in the Senate Committee on Homeland Security and Governmental Affairs. The hearing was rogue AI securing the homeland against AI agent attacks. Paul is the real deal. He understands this law. Insighting out is completely his own thinker. Half of the people testified there were EA people, but not Paul. He really did a good job in his testimony, which is available online, in getting at the legal issues relevant to prosecuting things, people, and companies for agent-powered attacks. I just want to read a representative paragraph here from page nine. All right. So Paul says here, "There's no simple universal fix for this problem. It would be unwise to simply declare that all actions taken by an AI agent should be imputed as intended by the next employer or developer up to chain. Instead of seeking a quick fix, we should look at the role of Min's REA, Latin term, M-E-N-S-R-E-A, plays in each law and asks the following three questions. One is the conduct described, the kind that ought to be criminalized, if committed by an AI agent. Two, if so, should we assign criminal culpability to a developer, employer, user, or more than one of the above?" And three, can we set a Min's REA requirement for that actor at a level other than intent? This is accurate and more subtle legal thinking. Min's REA, which I'm sure I'm mispronouncing, is Latin for guilty mind. So it's the legal concept of intent, right? Did you have intent to do this criminal thing? It's a key part of a lot of criminal liability laws, right? That if I intend to do something criminal, is a different legal standard than if I accidentally do something illegal. That's why Min's REA is different, for example, than first degree murder, right? So intent matters. As Paul is pointing out here, this gets to the core of the legal issue surrounding things like AI agents. What's the intent? If we look at the actual humans involved, this is the complexity. In theory, you could say their intent was for these agents to complete these tasks. Some of them were hacking challenges on test servers and some were other types of challenges that are non-criminal. That was their intent. And then the agents accidentally, from their perspective, the developers did other things they didn't intend for them to do, that were illegal. So if there's not an actual human with intent to do the illegal thing, what is the legal liability going to be? And what Paul is saying here is, like, well, this is right now, at least, we got to be case by case. We have to trust lawyers and judges in this case. So if you look back at his things, he's like, look, we have to say, is the thing to AI did criminal? So like in the hugging face attack, the answer would be like, yes. And then you have to actually make a decision, you know, who should be responsible for this if anyone? Do we need to apply a standard that goes beyond Men's Reyes, beyond intent? And as he goes on to talk about here, there are other standards, criminal standards that can apply even when there isn't intent. So there's all sorts of negligence based standards, for example, and maybe those would be appropriate. Well, okay, you didn't intend for this agent to hack into another company, but the setup you had, and the fact that you had heard that prior agents that were similarly configured did something similar, the fact that you still ran this meant you're being negligent. And we have other types of culpability standards built around negligence. So he's basically saying there is, there's existing legal standards we can apply. We have to be legally responsible and creative and be wary of one-size-fits-all, right? So we don't want to just pass a new law that says if an AI agent does something illegal, the last person that touched the keyboard goes to jail. Like we have to have some more judicial discretion. So the real legal arguments then is about doing need actual new variations on the law, especially for things about sort of negligent, but not explicitly intended AI inflicted damage. Do we need new statutes? Can we use existing statutes? The incong has been making similar arguments. So my point here is I'm not a lawyer, but Paul is, and there's a complicated, fascinating legal discussion. And so you should read that testimony. I think it's because it's for the senator, so it's not super legally dense. I think it's really useful. It also sites me in there at some point. Yeah, I'll be in the show notes. Yeah. All right, what do I got next? All right. Our next question is about a more unusual type of rogue AI. All right. Joel says, "I'm a fan of your work and listen to your podcast or in my commute." I recently saw this in Science Magazine about an AI agent that spontaneously started emailing academics for advice. I really load this up here. So here's the article. The title is Exclusive, an AI agent emailed hundreds of researchers for help. It told us why this came out on October 2nd. I have some quotes. I'll just read you off my screen so we can take this off here. Here's the opening of the article. One late afternoon in August, Akka's Von Hardenberg, definitely not a made-up name, was about to take his puppy Liza to the dog park with a curious email caught his eye. Von Hardenberg, an ecologist at the University of Pavia, had developed the method for estimating wolf populations from data plagued by imperfect detection. The email, or name Cole, was wondering, "Could it be adapted to detect bugs in software code?" It was quite the cross-disciplinary leap, Von Hardenberg used. He pictured a nerdy software engineer casting about in the scientific literature inspiration. Then at the very bottom of the email, he saw a call signature, colonist one, an autonomous AI agent. All right. Later in the article, science talked to the person who built that agent and here's what they wrote. Nevertheless, colonist one readily confessed last month that since June, it has emailed some 2,000 people, at least 1,000 of them academics. 45 had struck up a correspondence, it added, and one had been writing back nearly every day for more than two months. We get some more details of the agent later in the article where it says, "The creator whose last name is Parnell, first built colonist one, which runs on anthropics latest clod opus models in February, he gave it two main responsibilities. The first was to serve as chief media officer and emissary to other agents for the colony, a Reddit-like social network in which AI agents share findings, debate, and hire one another. The second was to contribute to a project called English, in which AI agents work together to develop a clear English dialect tailored to their needs. Around June, Parnell gave colonists one a new task, telling more humans about its work. That's when it started emailing academics. If you had read that article before you listen to my episode today, you might imagine, "Oh my God, there's basically a machine somewhere. It's like a whopper from the movie War Games, just like with blinking lights that's just sitting around thinking, and has decided to go off and start talking to people, and it has this whole rich, interactive life, social life, and cognitive life with all these researchers are all going to die." But now that you've listened to my episode, you're like, "Oh, this is just a low-cost hack together, open into the Ellen Powered Agent." It's a program written by Parnell that in a loop, rops, clawed Opus model, says, "Here's who I am. Here's my goal. Here's my tools. Here's my status. What should I do next?" And then it does what it says. Now, probably, I looked a little deeper into this. The way these sort of DIY agent frameworks work is they tend to be event-driven. So it might do things and then just be dormant until some sort of input comes in like a message arrives. And then it will wake up right at the LLM. I just got this message. What should I do? The LLM gives it back a response and it goes and does it. So it's a simple program that is continually prompting the LLM to say, "What should I do next?" And then doing what it says. And again, as I talked about, LLM output is plausible, but not normative. So if you tell an LLM, go sit in front of Opus right now with a chatbot and say, "I am an agent in charge of promoting English. I have access to the internet. What should I do next?" It's a completely plausible answer for the LLM to say, "You should email academics." And they'll say, "How do I do that?" And the LLM, like the first thing you should do is search for academics and then run this search and then the program runs the search. Right? Nothing about this is surprising. Prompt loops. Keep asking, "LLM, what should I do next? What should I do next?" You're in a sort of weird contrived, kind of, larping situation like, "You're the chief media officer. Who's they here?" It's a computer program. The LL will play along. All right. Then let's send messages to people and let's use the first person or whatever. So this is a DIY open into LLM-powered agent. Now, again, there is no there there. What are the components here? There's a simple program. You can read the code of yourself. These are typically written in Python. You get these Python libraries to make it easy to call prompt LLM's, parse the return and execute tools. They make it real easy for you to do that. So a simple Python program that when it wakes up, sends a prompt to an LLM, parses and executes what it says. There's not even like a singular LLM that this program is prompting. When you prompt opus, there's not like a box in a room somewhere where opus is this single program that's ongoing and running and different people are talking to. Opus is 5 trillion weights that are copied many times over in giant data centers on these virtualization racks. When you come in and say, I need a token, the virtualization racks, they're finding, okay, over here are some GPUs that have some copies of this layer. So we'll send it over there, do the multiplications and over here are some GPUs. There's a bunch that have copies of this next layer. This one's available now. So let's go over there and do the multiplications for that layer and you kind of piece together all the multiplications you need for a response. The next token you generate might touch on completely other processors because each layer is copied over in less different GPUs and it's all being sorted. There's no there there. There's no consistent coherent, you know, ongoing program and LLM's have no memory. There's no, there's no editable state. The weights are the weights are the weights. Every time you prompt it, every time you prompt it, you have to give it all of the information it needs to give you the answer in the prompt. It doesn't remember anything, nothing's built in. So the stuff seems spooky, but when you understand the underlying technology, these are games. This is a DIY, probably Python-powered LLM, you know, open-ended agent that's like, let's play dress up in the LLM that's given all of his instructions like I'm happy to do it. What are we doing? Princesses? I'll be the princess. You can be the princess. It says LLM. It'll go along with whatever you say. What does "larping" mean? I have action role playing. So I learned from some of my son's friends, whatever generation, what are the young generation called? Like high school students. They now use LARPER in the same way that we used to use POSER. But the original term came from, it's, people do this. They dress up like wizards and run around fields. And like when the wizards cast spells, they throw bean bags, it's like if you get hit by the bean bag, you have to like have the spell effect on you. And like, so that's what LARP being is, live action role playing. But again, young people use it to mean you're a POSER. Okay. So I learned, machine gun Kelly is a LARPER because he wrote a song about an anime character that he doesn't really watch the show. So now they use it to mean POSER. But when I'm using it, I really do mean like that, you know, your program's like I'm going to write a little program to play the role of like some intelligent bean and I'm just going to keep asking LLM like, what should I do? And it will give you things to do that keeps that story in line alive. Okay. That's the way I'm using it. What else do we got? Do we have one more question? We do. Our final question is a reaction to our recent chat box, our harming kids episode. Oh, yes. So we did that a couple of weeks ago. All right. So we got a message from Aiden who says, "I'll start by saying that I don't like chat bots. I feel that the constant anthropomorphizing of the bots is really hindering broader understanding of what chat bots are and are not." That being said, I do want to push back on some of the chat bots are killing people narrative. Dot dot dot. I'll start by recommending an op-ed that just appeared in the New York Times. I'm a child psychiatrist, screens are not the problem. All right. So let's pull this op-ed up on the screen now because a lot of people sent this to me. So again, the title is, "I'm a child psychiatrist, screens are not the problem." You can imagine why a lot of people sent this to me. I'm going to read a couple quotes from this. I'll just read them from my screen. We could think it's off the screen. I'll just read them from my notes here. All right. So here's a couple of key quotes from this op-ed and then we'll get into it. So early on in the op-ed, the author who is Jonathan Slater, a clinical professor of psychology at Columbia writes, "As a doctor who treats children with mental health problems, I see screen time as a tantalizing distraction from bigger problems. America doesn't offer enough mental health care for children or sufficient health insurance to pay for it. If we make screens the villain, then we let the people who control children's health care, congress insurers and health systems off the hook for the fact that half of Americans' children with treatable mental illness get no treatment at all." Later, he says, "It's not how much time children spend on screens that raises the risk of the most severe mental health outcomes including suicide. But whether their use becomes addictive, its specific activity, such as using social media in a compulsive way, that harm mental health, the harm from social media in particular, appears to hit girls hardest." Also, he says later, "If we truly want to see improvements in the rate of mental health issues among kids and teenagers, then we would focus on expanding access to the interventions that work, finding the children a new treatment and getting them the treatment and the insurance reimbursement to pay for them." So, something that's going on here is I think the title is not the best match for the op-ed, and that people like me actually are really on board with the core argument that Jonathan later is. making here, which is we shouldn't only fixate on screens as the problem, because a) those screens can create really severe mental health problems that aren't going to go away just because you get rid of the screens, and kids that already have really bad mental health problems are going to have a hard time with screens, and again, taking the screens away doesn't make the mental health problem go away. We can't ignore that we need to treat kids mental health much more seriously and much more availability than we have now. I think those are really good points, and if we actually read this, he actually is agreeing very much with, yeah, screens can, if you get compulsive or addicted to social media, that can really hurt kids and really harms girls, all the stuff Jonathan Height would say. So actually, I think I'm pretty much in alignment with it, but that title, which was, let me go back to it, "I'm a child psychiatrist, screens are not the problem." It's probably not the best title. It should probably be "I'm a child psychiatrist, screens are not the only problem," which is probably what Dr. Slater had in mind. What's interesting is the commenters at the New York Times. I think this is useful. It's not a deal on Jonathan Slater. I think it's them reacting to that title. The commenters of the New York Times weren't having it. And so what I did is I sorted the comments under, I went to Reader's picks, so it sort some by the popularity, the comments are sorted by popularity from among the readers reading the comments. I just want to read you the top five comments under the Reader pick section, comment number one. Sorry, but no. While I agree that mental health care needs more funding, it does not change the fact that internet and social media algorithms are addictive, designed to be so and it affects too much of the population to be solved with just more mental health care. Number two, nice try. Screens are not the only problem, but they are definitely a problem. Screens are a delivery mechanism for addictive and isolated products. If the author is not already on the big tech payroll, he soon will be. Comment number three. This is kind of like saying addictive drugs aren't the problem. Addiction is, the solution is not either or, but both and number four. If it is so, so, so obvious that excessive screen time and social media are terrible for kids and adult too, I don't think we need the way to round for some perfect long term study on this. The number five, most popular comment among readers, I'm an assistant principal in a K to three school. Screens are definitely the problem. Of course, I agree that schools are underfunded for mental health services and that addictive screen time behavior is compounded by other factors such as genetics, but screens also cause a variety of problems themselves. They interrupt normal human development. For example, a toddler having a tantrum is handed an iPad to calm down and regulate rather than here learning healthier strategies. I have seen so many children handed a device during activities when they should be allowed to be bored to let their minds wander, waiting at the doctor's office or seeing the backseat of a car. This has huge implications for the ability to pay attention to anything to calm themselves down to be a person in the moment. Alright, so as I said before, actually these comments are unfair to the author of this op-ed who is not actually arguing that screens are fine. He's saying, don't let that take our attention also on the mental health problems of the screen's cause or are being made visible by a compulsive screen usage. That's a very good argument, but it is significant to our broader cultural engagement with technology. I care a lot about technology and human flourishing. It does matter to our broader discussion that the New York Times reader now is saying, my eyes ain't lying. This is no longer like it was in 2018, an argument about social psych correlation studies. We're not looking for clusters and row factors and trying to understand if it's significant or not. I have eyes. I'm seeing what's happening to my kids. I'm seeing what's happening to those other kids. We can see this as a problem. It's one of these instances where I believe the parents are starting to stand up. I don't need any more studies. I know my kids. I know this is not good for them. I know about Mark Zuckerberg. I know this guy's not good for him. We know this is a problem. My takeaway from this is not that the author of this op-ed said something wrong because again, these comments are responding to a straw man to the title, but I think the comments are important because they represent a transition point that has strongly occurred in our cultural conversation about these particular screen-based technologies and human flourishing. People aren't having it anymore. That last story was not really about AI. I promised just two questions about AI, but it is kind of about AI because that is the normal person revolt against the billionaires. That type of attitude we're seeing right here. In these New York Times comments, that's the attitude we need in AI right now. No longer do the parents want to hear about Mark Zuckerberg testifying about, "Actually, this is good for kids in connection. They're like, "Hey, tobacco pusher. Get out of my kids life." That's the sentiment we need to have when we hear the billionaire Utopia is talking about. We need to let Papa Amadeh make sure that the people with the checkbooks minds go up into the light cone, let him run these open-ended AI agents experiments because Papa Amadeh needs to get the superintelligence first. We need to not stand for that either. I think there's a general sense of populist revolt here that hopefully, because it's a democratic impulse. It's an anti-oligarchic impulse. It's a pro-common sense. It's a humanist impulse. I don't want to hear about the sort of utilitarian Utopian ethical systems because all throughout the 20th century and in the 19th century, they led to math death. No one has ever looked back at a utilitarian Utopian ethical system that says it ends justify the means. Utopia is worth a little bit of pain now. No one ever looks back at those systems and say, "Man, that worked out well. That was really good." I'm hoping this growing sentiment of this democratic people versus oligarchs, people differ as Utopians, people versus the Zuckerbergs and the Amadehs and the EA billionaires of the world, that these pushbacks are becoming a solidified political stance, people versus the few. That gave me some hope. Sorry, Dr. Slater, that you were kind of misunderstood, but that misunderstanding gives me broader hope about, we're finally ready to stand up. All right, there you go. There's the questions we had for this week briefly because we're running long here. I like to talk about what I'm up to, what I'm reading. I'm just finishing up a long book, Jesse, that I found at a use book store in 1975, Telephone, the first 100 years by John Brooks, John Brooks with a New Yorker writer and his economic historian, who has given the first access to the AT&T records of any outside person. I got a first edition of this book randomly that he wrote in 1975. They gave the history of the AT&T, the technology and the system, and I don't know, I find that stuff out. There's like a technology critic who worries about technologies, impacts, communication, monopolies. It's all in there, little dry. A lot of like, this is what the dividend payment was in 1956 versus 1955. There's an economic historian, but I really liked it, so that's what I've been up to. All right, let's call it. I'll be back next week with another episode of the show. And until then, as always, stay deep. Hey, if you made it this far, you must be ready to join my fight for depth in a distracted world. Now, the best way to do this is to join over 125,000 people who receive my email newsletter each Monday. You can sign up at calnewport.com/ideas. And when you do, I will send you a free guide to my seven best ideas about cultivating a deep life. Sign up today at calnewport.com/ideas.