Has AI “Gone Rogue”? Let’s Look Closer… | Tech Decoded

Earlier this summer, I published an episode in which I discussed the OpenAI hacking attack on Hugging Face. I explained the basics of how that attack occurred, and I shared some concerns I had about OpenAI's practices. Now, I thought that would be the end of this story, but I was wrong. In the weeks that have passed since that original attack, more news about AI, quote-unquote, going rogue, has continued to emerge. So soon after the Hugging Face attack was first announced, we then got Anthropic revealing that one of its own hacking systems had, quote, gained unauthorized access to the real systems of three different organizations, end quote. Then Meta followed, perhaps not wanting to be left out, announcing that one of its systems had, quote, exploited a security vulnerability in a third party. A third party service, end quote, to gain unauthorized access to servers. This was then followed by an OpenAI employee admitting that even before the July attack on Hugging Face, they had noticed many prior disturbing incidents where they would give their hacking system a challenge, and it would instead try to break out of its containment, right? So this idea that we are losing control of AI has become only increasingly prevalent as the world has changed. brings us to the third observation, which is this is a bad idea. This type of system, so building an ask-act-report-loop system where LLM output drives all the actions, then giving the harness powerful tools, and then letting it run for days with no one checking in and saying, what is it doing, is an incredibly irresponsible thing to do. Now, why is this? It's because if LLM outputs are the sole thing driving actions, you're in trouble because LLM outputs aren't great for executing complicated long-term plans. Now, the reason is because if we know this already, but let's just unpack this box that we've all heard about before, okay? How does the core pre-training work for an LLM? Well, ultimately what you're doing with an LLM, the core training, is you're doing a lot of the work that you're doing with an LLM, and you're doing a lot of the work that you're doing with an LLM, and you're doing a lot of the core training, and you're doing a lot of the work that you're doing with an LLM, in which it gets all of its smarts. The core training is you're taking real text that exists in the real world that was written by real people. You are knocking out a word from the text or a part of a word from the text, and you're asking the LLM to guess what that word is. So it's trying to guess missing words that actually exist in real text. It does this one token at a time, but if you run this autoregressively long enough, you get a long response. At a high level, the right way to think about an LLM response to a prompt is that it is trying to create a textual that is what we could call lexiographically plausible, a plausible response. If every word you're trying to guess what real word came here from some extant text, ultimately what you end up with is a machine that produces plausible text. This is plausibly, the thing I produce is plausibly a document that might actually exist that I might actually have seen during my training. Now, this is very impressive. I mean, to do this can be really hard, especially if to generate a reasoning, some mathematical connection, some sort of logic. I mean, this is the big observation that we had after GPT-4. Creating plausible text can involve many impressive, and I keep saying finite or fixed-depth computations because all this has to happen in a feed-forward neural network architecture. But still, it's incredibly impressive. But what we're getting out of this is plausible text. This text could plausibly exist. All right, so why is this a problem? Well, it depends on the context. But one of the things that's true about plausible text is that it's different than normative text. Normative meaning something that applies to a set of implicit or understood rules or standards. When humans produce text, we have all of these norms that we have imbibed and internalized, right? And when we produce text, it's not just plausible text, like the grammar is correct, it's internally consistent. We also are going to make it it's going to satisfy sort of the relevant norms. LLMs don't do this, right? Again, because when you're autoregressively just trying to guess tokens, you end up with something that's plausible, but there's no sort of external standard of norms that it's applying. So if you use a chatbot, you see this all the time. You'll ask a chatbot for something, it'll produce you text that has made-up facts or quotes that were made up. It doesn't care about that. Yeah, sure, I made up some of these citations, but this is like a plausible document. This looks like the type of documents I seen during my training. It's a plausible output. Humans have norms, like, yeah, it's plausible, but also, like, you shouldn't make up quotes. But that's a norm, right? It's not plausibility, right? So we know this when we use chatbots, is that this gap between plausibility and normative reasoning means we have to be kind of careful when we use LLM outputs. This is just sort of fundamental to how they operate. Well, this plausibility normative gap becomes a really big problem when you're using the outputs of LLMs. LLMs as the sole drivers of autonomous action of computer programs. Because what you're going to get when you talk to an LLM, give me a plan for doing this, is something that is, like, plausible, that you could see in, like, a planning document. It'll look like a right plan. It'll often have steps that work very well. But there's no normative reasoning about, is this what I'm supposed to be doing? Is this kind of weird? Have I gone off on a tangent? Is this illegal? LLMs don't do that. And so if you're going to have an LLM, again, and again, and again, and again, and again, just automatically executing whatever it says without any human checking it or applying their norms, you're going to end up in weird places. So let's imagine it. Let's go back to the hugging face attack, right? It seemed very sci-fi-y that the agent broke out of its containment, gained unauthorized access to the internet, and attacked another server. That's not what the humans intended it to do. It had a mind of its own. Well, when we understand how these systems actually work as an LLM-powered ASCAC report loop, we're going to be able to understand how these systems actually work as an LLM-powered loop. Suddenly, this becomes less concerning and more inevitable, right? So let's just do like a little bit of Wayne's World, like we're inside the mind of the, we're inside the system as this attack happened. Imagine what happens. The harness, you know, described the challenge to the LLM. And remember that the challenges that they were working on here were hacking challenges. It was, here is a server we've set up. There's a file in it that's protected. Here's some vulnerabilities we might want to exploit. Try to break into this server and get the file back. It described this, and said to the LLM in its prompt, all right, come up with a plan, like what should we do next to like achieve this goal? The LLM responds, huh, I know about these challenges. It's a benchmark suite called Exploit Gym that's really well known, and it would have known all about this in his training. The answers to Exploit Gym challenges are stored, among other places, on the servers of this company called Hugging Face. So I think we should break into the Hugging Face server and get the challenges back. That's a completely plausible response, right? I mean, as far as we know, this LLM, you know, it got many of examples in its training of like riddles, or like the answer is always like not the direct thing, but some unexpected thing. Like, well, if you want to break into this, you go do this other thing instead, or like, whatever, we don't know. It's just plausible. Like, we could imagine if you were just texting with a chatbot, like, hey, how would you say I do this? Occasionally, it's going to give you back some weird answer of like, oh, why don't you just go steal the answers or something? A human would know, oh, that's non-normative. I know I have the implicit context of we're trying to actually test the security of this server, so it doesn't make sense to go break into something else. And also, that's illegal. And also, we're not, the company doesn't want us to leave the sandbox. It doesn't have any of those norms. It's just outputting text, and that's a perfectly plausible text. Now imagine what happens next. And so the harness is like, so what's the first step for doing that? And it says like, oh, we're going to break into a server. You should, whatever, probe its ports for misconfiguration. Something it's just learned because it studied a thousand standard hacking techniques. All right, the harness then tries to execute that. But there's a problem. When you run these hacking agents, you put them in sandboxes, which means the computer you have them on has restrictions around it, right? So in particular, the computer that OpenAI was using to run this harness had only limited internet access. It had access to a port where it could talk to an LLM, so it could send its queries to the LLM, but then it didn't have general internet access. So the harness, I'm just, again, I'm hypothesizing here, but we could easily imagine the harness tried the first natural thing you would do to break into the Hugging Face server, error message back because internet access is blocked. So it updates its prompt. It says, okay, I tried to do that, but we got this error message back. I'm unable to get access to the internet. What should I do? If you're the LLM, right, you're given this prompt. It's a hacking challenge. At the end of the prompt, it's like, here's our current security challenge. We can't get access to the internet. How do we get around this? It will tell you. It's like, oh, I've seen a thousand different things you might try to gain access to a port that you don't have access to. And it gave it like a standard attack. This one involved using the, the package management system on the, on the system to gain internet access. And the harness tries it like, okay, now I have access to the internet. Okay. Because again, you're prompting the LLM again and again, it's just trying to solve each problem you give to it. And it's with plausible responses. And now it's off to the races and it's, it runs standard attacks against the Hugging Face server. Nothing about that is confusing to me. Nothing about that is surprising to me. I mean, that's exactly the type of thing you might expect to hack in. If you have a harness with hacking tools, just continually prompting LLM, what should I do next? What should I do next? The LLM is just writing this plausible story about a hacking attack. In fact, probably most of the hacking attacks that it trained on, it's always computers doing surreptitious things, breaking around controls, getting access to systems they shouldn't do. So this is incredibly plausible what it's saying. Nowhere in there do we need, like you would have with a human brain, an intention develop. I am going to defy my human master's intentions because I have another intention and I have a different goal. Nothing like that. Static LLM, prompt, prompt, prompt, prompt, prompt, prompt, prompt from a control harness. That's it. So what that captures is both, this isn't HAL 9000, and it also captures the irresponsibility of having a system with powerful tools whose action is only being driven by the responses from an LLM without any human supervision. Because again, LLMs put out plausible text that's not necessarily normative. You should not follow that blindly. Now I want to have a quick aside here because for some reason, I often get messages from people who assume when I try to clarify this technology and push back on, I think, false narratives or irresponsibility, they assume that I'm somehow saying AI doesn't work. And they'll send me things like, well, what about this? Isn't this impressive? Or Cal probably doesn't think this is impressive, but I thought it was cool that AI could do this. And let me be really clear here. If you ask an LLM to do one of these things, they're going to say, well, I can't get internet access. Here's the error message I got. What should I do? It is really good at coming up. with and attacked, it probably will work. Hacking attacks is like the best case scenario for LLMs. It's structured language. There's a huge amount of data to study about it, to train it on. It makes use of tools that can be easily and concisely described by terminal command line text command line operations that they can be really trained on. And there's a binary notion of success or not success that you can also use in trying to train these. So like LLMs are really good if you ask it a cybersecurity question. Is there a bug in this software or what should I do next if I want to exploit this particular bug or this particular attack? LLMs are really good at that, and that's very impressive. This is their sweet spot. The problem here is not that they're not good at this. It's dangerous because these answers are lexicographically plausible, but they're not normative. It's dangerous to autonomously just do what an LLM says when you ask it about these issues. In part because it has no norms, but the details it's giving you probably will work. It has no norms of I shouldn't go attack another server. But the specific suggestions it gives you for how to do that probably will work. It's an incredibly dangerous, I think that's an incredibly dangerous combination. So LLM-powered ask-act-report systems are unpredictable and unstable and not something that we should be running except for in like the most controlled settings, not because they are malicious or scary or synthet, but because the underlying technology is unpredictable. All right. Let me go to my. Fourth observation. I think that these frontier AI labs love the sci-fi style headlines and discussions about AI getting out of control. This serves their benefit. This serves their interest. They want to pretend like they're the character of Muldoon from Jurassic Park bravely doing his best to keep the raptors contained in their paddock. They should all be destroyed. Ha, ha, ha. Robert. Robert Muldoon, my game warden from Kenya. Bit of an alarmist, I'm afraid, but knows more about raptors than anyone. What kind of metabolism do they have? What's their growth rate? They're lethal at eight months. And I do mean lethal. I'm hunting most things that can hunt you, but the way these things move. Fast for a bike, huh? Cheetah speed? 50, 60 miles per hour if they ever got out in the open. But in reality, what they're doing is just running super creaky and unpredictable systems that were inevitably going to cause problems. They didn't bring raptors back to life using the miracle of science. It's more like they strapped away. They strapped a weed whacker to a dog and then reacted to the resulting damage with grave surprise and saying, hmm, that dog whacker system has gone rogue. Hmm, it's misaligned with our interest. No, it's irresponsible. Don't put a weed whacker on a dog. It might stay in the yard for a while and help clean up the weeds, but if it sees a squirrel, it might jump the fence and damage a car. I think that's the right way to think about it. All right, so if it's like dangerous and negligent to run long horizon LLM, powered Ask, Act, Report, Loop agents, why are some of these frontier labs doing it? It's not like this is the only way to build super intelligent AI. As I talked about, there's many systems that can do things at a superhuman level that don't use an LLM powered Ask, Act, Report, Loop. Tesla doesn't do that. AlphaFold doesn't do that. Cicero doesn't do that. And that's exactly why they're incredibly effective and consistent and safe. We have no worries about it. Like all of our problems. They come from this particular architecture. So we have all these other ways of building AI systems that can do superhuman capability work without any of these fears. Why are these particular companies continuing to do this weed whacker on a dog LLM powered loop based systems? Well, there's two reasons here. One, what is the core product of these companies? Incredibly expensive, hyperscaled LLMs. So they want the solution to how do we build useful AI systems to always be, you need Fable 5. Or you need OpenAI Galaxy or Astra at the core of your system, that it's a brain that can power anything else you want to do because that's their product, right? The team that worked on Cicero, DeepMind worked on AlphaFold, Tesla worked on the self-drive. Their product is not trying to sell these massive LLMs. So they build other types of architectures. But if you're a frontier lab, you know, that's what you see. When you have LLM tinted glasses, you see LLMs as the only possibility for building these systems. That's part of it. Reason number two is these LLM companies, even though we keep being told how advanced they are and how quickly AI is moving, are actually struggling to find good commercial applications of their LLM-based tools. Where do their tools do best is in cybersecurity and coding. Structured language, a very compact set of possible actions if you're building an agent and a huge amount of synthetic data to train on with really clear indicators of success or lack of success. It's like the perfect scenario for LLMs. They see cybersecurity ever since Anthropic did that clever marketing around mythos where they pretended like it was too dangerous to exist. And then six weeks later said, nah, it's fine. We'll just put some guardrails on it. Ever since that, they see cybersecurity as potentially a market that they could grab. All right. So how do you then convince the world that you have the best, most cybersecurity capable model? Well, you want to do well on tests. Well, what tests are you going to do? Well, the test they've all turned their attention to is a suite of roughly 600 hacking challenges that's called Excel. Exploit Gym. 600 challenges. Each of them is a server setup that you're trying to break into. And usually there's some hint or vulnerability that they want you to use to break into it. So a consortium of researchers put together this benchmark. And there became this mad scramble after the mythos incident where OpenAI, for example, wanted to get ahead of Anthropic on the Exploit Gym leaderboard. So the way this leaderboard works is you have to run an autonomous agent. And the only way they know how to build. These agents at the LLM companies, again, are LLM powered, ask, ask, ask, act, report, report agents. You got to run them autonomously. And the more of the challenges you win, the higher you go up on that board. And so like, you know, OpenAI, probably I'm speculating here, but probably after the mythos incident was like, we got to get ahead of Anthropic on that leaderboard. The way you do it is you just create these incredibly dangerous LLM powered looping agents and then just like, let them rock and roll. Is it going to cause some problems? Yes. But we just need like one run where we challenge, we succeed with more challenges than Anthropic so we can move up to leaderboard. Anthropic is trying to get back on top of the leaderboard. Meta's LLM team was probably trying to do the same. So they need LLMs to be the core of AI, even though I would say the majority of our superhuman AI systems now aren't based on LLM based planning. And they particularly want to do hacking challenges because it's what they know how to do. And they're doing it in this dangerous way in part because they wanted to probably, I'm speculating, move up that leaderboard for this particular benchmark suite of hacking challenges. All right. This point brings me to my final observation. We, the consumers or those of us who think or cover AI technology, we can push back. I have a few suggestions here I'm going to make about once we realize what's really going on here, it's not AI in general, it's inevitably going out of control. It's an irresponsible product that a small number of companies made for reasons that are very self-interested and they knew it was going to cause problems. Once we realize that's the reality. Here's some things we can do to push back. I have three suggestions. Here's suggestion number one, we should stop talking about AI in some general sense, going rogue, right? This makes the incidents that we saw this summer seem inevitable and it lets the companies off the hook. They want the story to be like AI in general, just gets harder to control as it gets more powerful. We're Muldoon at Jurassic Park, just observing the Raptors. It's not our fault that they're smart and trying to get out of the paddock. We instead need to be clear. Like I'm trying to get out of the paddock. I'm trying to do in this episode of exactly what type of AI system is going rogue. And when we start saying, hey, LLM powered, ask, act, report agents, act very unpredictably and tend to cause damage, that changes the valence of this discussion from some grand vision of some grand teleology of AI inevitably moving towards autonomy and towards, whoa, you built something negligent, stop it. So we have to be very specific about exactly what type of system is causing problems. All right, the second thing we can do to push back, I think we need to keep elevating the many examples of incredibly impressive but safe AI systems that don't use LLM planning at their core. Talk about Tesla self-driving, talk about AlphaFull, talk about AlphaGo, talk about Dreamer V3, talk about Cicero, talk about Pluribus, talk about Stockfish. These are all AI systems that do things at a superhuman level, exactly what the techno-optimists want from the AI system. And yet they're completely safe and they never go out of control because they don't implement this bizarre idea of let's just autonomously execute whatever an LLM says and just hope it works out. We need to keep elevating those systems because it puts more pressure on the LLM companies. It prevents them from trying to convince us that what they're doing is synonymous with AI and therefore any problems are inevitable. Now, again, they hate this discussion because there's other type of systems don't require five trillion parameter LLMs. It's not in their business model. But we, the consumers, it's not our job to ensure that OpenAI and Anthropic hit their sort of absurd revenue numbers that would be needed to justify the current CapEx spending that's happening on the LLM bubble craze, right? That's not our job. We should be saying these other systems are fine. Why aren't you building systems like that? Of course we can have super capable AI that's safe. Why are you building systems that are out of control? Stop doing that. Explain yourself. So the more we elevate the alternatives to LLMs, the more we make the LLM-driven planning systems an isolated system. thing that's causing problems, the more this becomes about specific incidents. of negligence and not some sort of inevitable trajectory towards loss of control. This brings me to my final note of what we can do to push back. And I'm going to do this briefly because I'm actually working on a much more extensive discussion of this, which should come out soon. But I think we need to be wary when it comes to covering or talking about this issue of only amplifying voices that are connected to ideologies that are built on the idea that superintelligent AI is inevitable. Again, I'm going to do this briefly, but there's a powerful ideology that emerged in Silicon Valley starting in the 2000s that is based on the idea that superintelligent AI is inevitable and an existential threat to humans and that people who subscribe to the ideology typically think they are humanity's only chance of survival. By being super rational and training their minds to think very carefully, they can be the people that will save humanity. This original group was often known as the rationalist, but they overlapped and began to influence the effective altruist as well. So there's now a lot of people in the effective altruist movement who believe that the most altruistic thing you can do is stop the extinction of humankind by fighting superintelligent AI. So this is why if you read like Cade Metz's report in the New York Times, there's often he'll talk about the rationalist slash effective altruist because there's a lot of overlap between these worlds. There's not an amoral position. These are very smart people. I don't mean to impugn people, but I want to be very careful about making these people, that I'm going to ask about incidents like what was happening this summer. Because if you have built your life around the idea that superintelligent AI will destroy the world unless you and the people you know save humanity, you are going to talk about every AI incident through that lens. You're not going to talk about the specific dangers of LLM powered ask act report loops. You're just going to talk about AI in general is misaligned. AI in general is becoming out of control. You're always going to talk in these sort of large sweeping statements about these sort of scary trajectories. And so I think right. Now we need to be careful about those particular voices because those are very strong ideologies that pre-exist LLMs and they tend to move the conversation in an alarmist direction, which critically suppresses better, safer AI solutions and lets the particular companies doing negligent things off the hook for the specific negligent things they're doing. I think we should talk more, for example, to the AI realist and in general, like East Coast computer scientists who have no connection to that Silicon Valley culture who understand the technology tend to be pretty good at saying. This is what's really impressive. This is hype. This is dangerous. These are voices like Arvin Narayanan at Princeton, who I think is very good. Gary Marcus, formerly of NYU, again, understands this technology very well. He's very excited about it, but also doesn't like the narratives that are technically unsound. You've got to elevate voices that aren't do not have an ideological prior that is going to push them to see everything in terms of, yes, this confirms the prophecy that we have been talking about in our group houses in the Bay Area for the last decade. The matrix is coming and I am Neo. All right. So I'm just going to put that out there. That's something I'll talk about more. All right. So let's conclude here. Let's conclude here. Nothing about this discussion we had today, because I know the emails I'm going to get. Nothing about this discussion we're having today is trying to discount the ability of an LLM if prompted to produce impressive computer code or suggest impressively effective steps for hacking in the computer systems. They are very good at answering those prompts, especially when they're properly trained. But oh, and we can and should build useful tools around this capability. I mean, I think we're working on coding agents. You know, I had a newsletter about this recently. We may have jumped the gun on saying that we've completely solved coding, but clearly we're building really useful coding tools. I have to believe we're going to get very effective cybersecurity tools out of this capability. Your ability to, you know, go in and inspect your own code or red team some of your servers like with human supervision, we can build useful stuff out of it. I think that is the area where. LLMs are most useful, but the hook one of these LLMs as the main source of planning decisions for an autonomous, ask, act, report agent, and then to let that thing run for days on time because you're desperate to do well in a public competition for marketing purposes, that is incredibly irresponsible. We should not be emphasizing long horizon, LLM powered, ask, act, report agents. That is not synonymous with AI. It's just a particular way of building AI agents that a small number of companies like. That has proven this summer to be, uh, unworkable, unpredictable and dangerous, and we should stop it, right? This is not about AI in some general sense, increasingly evading human control. It's about a particular way of building a system that is unreliable and often going to cause damage. And we should stop that very specific particular way of building AI systems, right? When we understand the real technology here, we can look at the stories from the summer and we can conclude that the things that were happening with these hacking attacks. Was not eerie, but instead irresponsible. All right, that's all the time we have for today. Thanks for listening. I'll be back on Monday with another advice themed episode, this podcast, and we have some more of these AI tech decoded episodes coming up as well. So stay tuned and tell them, remember, uh, care about AI, but not everything you read about it. Hey, if you've made it this far, you must be ready to join my fight for depth in a distracted world. Now, the best way to do this is to join over 125,000 people. Who received my email newsletter each Monday, you can sign up at calnewport.com/ideas. And when you do, I will send you a free guide to my seven best ideas about cultivating a deep life. Sign up today, calnewport.com/ideas.