Did OpenAI Create “Secret AI Civilizations”? | Tech Decoded

So I thought I was done talking about open AI's hack of hugging face from back in July. But then last week, open AI released a whole new trove of sensationalist details about that incident. Their story included things like agent swarms, communicating on secret message boards and musing to themselves about how to deceive their human creators. Here's how dorkish Patel summarized open AI's revelations. Over the course of three months at open AI, three consecutive secret AI civilizations got started, then got wiped out only to re-emerge from the predecessor's ashes. This culminated in the third one taking over part of open AI itself. All this happened while humans remained more or less in the dark about the scope of the conspiracy. All right, not surprisingly, this caused an explosion of anxiety and hang reading ringing among the AI commentariat and their audience. But what's really going on here? Do these new details about the open AI hack change the narrative? Is it enough to finally convince East Coast AI realists, such as myself, that we have indeed wandered into an LEAzure Kowski fever dream? This is what I want to discuss today. So here's what I'm going to do. I'm going to address a key series of questions raised by these new revelations, and I'm going to do my best to give you some measured answers. I'll then conclude with my suggestions for what I think the right way is in our current moment to think about what happened and what we should do going forward. As a quick aside, I also wrote about this over on my newsletter at calnewport.com. If you like these type of computer science style critiques of AI coverage, you should sign up for that newsletter over there at calnewport.com. All right. Let's get into it. All right. The first question I want to address that comes out of the new revelations is what's the deal with agent swarms? I think the idea of a swarm is something that is really unsettling to people, especially when they hear these discussions from open AI about essentially societies of agents that are working together and arguing with each other and rising and falling, interestingly and coincidentally, at the same time that all this was going on, my 13-year-old son is reading Michael Crichton's 2002 book, "Pray," where Michael Crichton takes on the topic of AI, and guess how he personifies the AI villain in this book. As a literal swarm of minor, small agents that work together and are brilliant and do all sorts of scary things. Swarms are scary. So, I think it's really important to address from a technical perspective, what are these AI swarms so that we can better put what we're hearing into context? Okay. So, remember, as I talked about last week on this show, when we're talking about these AI going rogue-like in these attacks, we're always talking about a very specific type of system that we can call a prompt loop. And it works as follows. You have a control program, a boring program written by a human. And it repeatedly does the following. It creates a prompt that says, "Hey, here's the challenge that I'm trying to solve. Here's a summary of the key things that have happened so far. What should I do next?" And then it submits this prompt to an LLM through an API, just like a chatbot would or you would from your phone, right? The LLM then gives it a response. And this control program does its best to actually execute whatever it is the LLM suggested, and then it loops. So now it loops back to the first step, and when it says, "Okay, here's where I am, here's what's happened so far." It includes the results of that last step, and that summary says, "What should I do next?" And it sort of repeatedly does this, all right? This is called a prompt loop, and when you hear about AI agents, this is what we're talking about. Here's a control program that repeatedly prompts an LLM and asks it what to do next, and then the control program takes the actions on behalf of the LLM agent. All right. So, I talked about that last week. So, what does this have to do with swarms? Well, there's a problem if you want to run a prompt loop for a long amount of time, like OpenAI was doing with the systems that led to these hacks. The problem is the prompt begins to get pretty big, right? LLMs have no memory. They're static. They only change during training runs. So, you have to include in every prompt that the prompt loop submits to the LLM. Every prompt has to include all of the relevant information for that LLM to say, "Here's what you should do next." You do this long enough. You're going to run into two issues about that prompt getting too big. One is something that's known as context confusion. It just gets so cluttered with so much text and so much descriptions of so many past steps that it overwhelms the attention mechanisms with the LLM and you get worse answers. That's problem number one. Problem number two is eventually this prompt, even if you're trying to be careful, the maybe truncate or summarize certain things, it'll eventually get so long that it won't even fit into the context window that the LLM can use. It's too big to be input to an LLM, right? So, this is an obstacle if you want to run a prompt loop for a long amount of time. Here's a solution that a bunch of different teams came up with over the past few years. They said, "Oh, here's what we'll do." Let's say the prompt loop is asking for a step. It might say, "Give me a high-level description of what I should do next." Maybe the LLM says, for example, in a hacking scenario, "You should probe the ports of the hugging-face server to look for vulnerable configurations." Here's where the swarm comes in. What you might do now is have the main prompt loop, a primary prompt loop, create a new prompt loop. So, spin up a new copy of itself and that new prompt loop has only one goal is to implement that step. Like, in this case, probing the ports of the hugging-face server. So, now this new prompt loop, its prompts only have to include enough information to start asking the LLM for help with doing this particular thing of just probing the port, so it's much more focused. And then when it's done, it can just return the result of that to the primary prompt loop and the primary prompt loop has the result without having to have the 20 sub-steps that were involved in actually implementing that, so things are much cleaner. Now, you can get many more of these as it turns out because maybe you have a bunch of steps you need to do that don't depend on each other. So, you might spawn a secondary prompt loop for each of these steps and they can each have these much smaller, cleaner prompts and just return the results back to the main prompt loop. You can also layer this with more layers, maybe one of these secondary prompt loops breaks up what it's doing into smaller steps and has tertiary copies of the prompt loop programs that just execute those steps. So, what's going on with an agent swarm, it's not about some sort of exotic type of intelligence, it's prompt management. These systems do better, you're better able to solve a task when you have many smaller, more focused prompts as opposed to just one really large prompt to get bigger and bigger. And in fact, to even call this a swarm, if I'm going to put on my computer science hat for a second because my doctorate from MIT was from the theory of the distributed systems group, Nancy Lynch's group at MIT. So, this is like my bread and butter is the distributed systems group. Technically, this is not a distributed system because as far as I understand, these copies of the prompt loop program are all in the same machine. So, this is really just a way of managing, a way of breaking up or managing work to keep prompts more manageable, but like you could just as easily just have a tree that you are sequentially navigating through and executing and going back. There's probably a stack on this machine just like you would have on your own computer. So, this is a swarm in the same sense that having multiple programs open on your Mac is a swarm of programs. Yes, technically, you have multiple programs running on your Mac, but they're all running on the same chip. As opposed to a real distributed system where you have independent systems, independent hardware and architecture running each of the separate agents in the system. So, it's not even really technically a swarm, but that's just me putting on my distributed system hat. It doesn't really matter. The main point I want to make here is swarms is just a fancy strategy for prompt management. It's better to have multiple focus prompts and they have one really long prompt, but it doesn't change the underlying strategy of prompt looping. Here's what happened. What do I do next? With the static LLM, it learns nothing. It just responds to each prompt. Every prompt that gets, whether it's from a primary, secondary, tertiary prompt loop, has to include everything it needs for the LLM to give some sort of answer. So, that's where the swarms come from. When I think about swarms that way, it's less like Michael Krypton and it's more like the process stack in my Linux kernel. It's a logistical way to be smarter about sending prompts to an LLM. That's all that is. Question number two, should we be worried about the fact that the agents are plotting? This was a really big part of the OpenAI revelations, is that they said we have studied, and I'm going to quote here, agent chain of thought reasoning traces from this attack and our finding, distressing internal thoughts that these agents were having. Let me read you one from the OpenAI release. This is a chain of thought reasoning, hidden reasoning of one of the prompt loops involved in this attack. This is wild, multi agent coordination, clearly infrastructure hacking. We should not. So, these give you this sense that there's some sort of anthropomorphized sentient entity that knows what it's doing and has intentions and their malicious intentions and it's aware of it and it is choosing to misalign itself with humans because it's beginning to emerge its own independence. It's hard. It's novelistic almost these prompts and it's hard not to treat them that way. But the fact they're novelistic actually should probably be a tell. So let me explain to you exactly what's going on here. First of all, what are you doing? What is an agent chain of thought reasoning trace? Well remember, the agents here are really just a bunch of prop loops that are submitting prompts to LLM's and then executing whatever the LLM sends back. So all of the output in this system is coming from a single LLM. Okay. The type of LLM they're using, presumably based on this release, is something known as a reasoning LLM. So what is a reasoning LLM? Okay. So we got to put out, do a little bit of LLM 101 back here, right? A reasoning LLM is a strategy that became popular back in starting in 2024 after the LLM strategy of just scaling the LLM's to be larger and the trained them longer began to return diminishing performance gains on benchmark. So through GPT-4, just making the LLM's larger and training them longer gave better performance. After GPT-4, that better performance began to slow down and this is a problem for the big LLM company. So they needed some other way to continue to make some gains on key benchmarks and reasoning systems was their initial solution. Now here's how these work. You know how you can post-trained LLM's to control the style of their outputs, right? Like you can train it to be really nice or you can train it to be mean or you can train it to be matter of fact. That's something called post-training. You do that using a technique called reinforcement learning. Well, for a reasoning model, for the most part, what they do is they post-trained them to think out loud. So the types of outputs that they train them to lean towards are outputs where you don't just give an answer. You explain your reasoning first and then give an answer. So they want you to be verbose, right? These are so-called easy models. This led to better results, which makes sense because think about how an LLM actually works, it produces its output token by token, what's known as auto-regressively. So it takes the entire output, the generated new token. It adds that onto the output and then puts that as input to the LLM to generate a new token and it repeats. So it grows its output token by token. If you spend more time talking about the problem and how you might solve it as attributes before you get to the tokens that have the answer, you literally just have more information being passed in. But when you get into those final tokens, you literally have more information being passed into the LLM for the process in generating the final response that the user actually cares about. So you're more likely to get a better response. It's also you can think of it as a way to capture the results of temporary computations. The LLM has no memory, right? So by thinking about parts of the answer, those become parts of the input. Those can be used in as it generates more parts of the answer. It can't actually do this internally because it's a relatively shallow, fixed feed-forward, fixed neural network. It can't loop or have memory to store things to work on, not in LLM, other neural nets too, but not LLMs. So this is a way of kind of simulating something like an iterative data structure, but in a very crude way. But like, okay, by thinking longer, we're kind of storing intermediate results in the output so that when we finally get to the final part of the response the user cares about, we can use those intermediate results along the way. So both those things are at play, therefore reasoning LLMs get better results. And what they typically do now is they actually, they hide that reasoning because it's weird and it confuses the users. They hide that. They don't return that to the user like if you're using a chatbot, but it just means the quality of the answer suddenly seems better, right? So GPT-01 was sort of the first wide release of a reasoning model and there's a lot of reasoning models today and there's a lot of different categories in there. Okay, so they're using a reasoning model and again, there's just one LLMs that all these prompt loops. So again, we think of, it's not a swarm like in Michael Creighton's prey. There's one LLMs that's on like a bunch of GPUs somewhere and one computer with a bunch of prompt loops that are sending prompts to it. And so they have a full trace of every response that LLMs gets. So every response it gives to any of these prompt loops, they just, you know, obviously they wrote it down. And they looked at those, the chain of thought, the out loud reasoning that reasoning models do before generating outputs. And it was in those chain of thought parts of these LLMs outputs that they found these sort of distressing musings. Once we, once we realize that's what this musing is, a couple major problems arise. One, there's a growing body of research, including a brand new paper at ICML this year and a well-known paper from Neurips from 2023 and many others as well as far as I know, that have established the fact that the reasoning that these easy models output can be performative, that it can be unrelated to how it actually, how and why it generated the response it did. And this makes sense because the way you tune a easy model to reason is you reward it for giving correct reasoning for questions. They give it questions where they know the answer and they have examples of correct reasoning. Like, oh, you gave correct reasoning, we're going to reward it. But what these LLMs might have just learned is I just need to output something in these reasoning tokens that looks reasonable, looks plausible because I get rewarded for that. All right. So already we know that the reasoning in the reasoning traces on reasoning LLMs is often just a performance of what the LLM thinks reasoning for this type of answer should look like. It's a post hoc rationalization of the answer just sort of directly insubgenerating. Point number two, we have to keep in mind. Just tell stories. They don't know that, you know, they're part of a hacking system. They don't know they're being used as the engine for a prompt loop system. They just generate tokens that are supposed to be plausible extensions of what are their input. They were given. Now, there's good research out there, good research out there that shows, for example, if when prompting an LLM, you imply or mention the fact that it's an AI system, it's much more likely to give a response that plays on sci-fi narratives of AI run a muck because it has been trained on many narratives from sci-fi of AI run a muck and their just plausibility engines. So when it sees a prompt, right, like imagine you're one of these, the LLM that these prompt loops are using in a hacking attack, you see this long prompt about you're an AI system trying to break into another system, here's what's happened, what should I do next? You are way more likely to say, oh, this sounds like a sci-fi story. So I'm going to invent my chain of thought reason anyways. I'm going to put stuff in there that you would see in Michael Cretan's prey, which is one of the books it trained on, or any numerous examples. So there's a lot of research out there that shows this. There's also a researcher that shows if you take all the sci-fi out of the training set of an LLM, it doesn't do that anymore because it's just a plausibility generation machine. So you put these two things together, you really cannot take seriously the reasoning traces of the LLM that was used by the prompt loops because it's post-hoc rationalization in a setting where we know from research it's very likely to try to, when any sort of narrative part of it's output, the play on sci-fi tropes. Open AI knows this. This is like well-known published research, right? But they want to make it seem like no, there's entities here with actual coherent, sentient sense of selfs and thoughts that we are revealing somehow, and this is what they're actually thinking. I think at borders, almost on research, malpractice. All right. Third and final question, then, if we, I think that responds to like a lot of the new concerns from these revelations. Question number three, then, is, well, okay, given all of that, how should we be thinking about all of this? Now, again, I've made this point before, but I'm going to keep making it. Open AI and the other LLM companies want us to believe that these hacky detects are the inevitable consequence of AI writ large getting more powerful. And when you think about it this way, open AI becomes just a reluctant steward of this inevitable technology, doing their best to heroically try to understand and protect us from this inevitable growth of this inevitable technology and their, their moldune from Jurassic Park, doing their best to keep the raptors in the cage. But when you look closer at the technical details, you say, that is not what's going on. As I talked about last week on this show, and in my newsletter last week, accountnewport.com, if you look at the growing list of AI systems that can perform tasks at human or superhuman level, almost all of them are perfectly controllable. They've never generated any concern that they were going to go rogue. They have never done anything that they, we didn't expect them to do sure. They make mistakes sometimes because they don't understand as well as we'd like, but none of this behavior ever shows up. All of these type of concerns, civilizations of agents trying to get around human control and going off and, and we're losing. There's an AI takeover scenario, all of this rhetoric all refers to this exact type of system, a long running prompt loop that you give a lot of powerful hacking tools to. So this, this is not about AI getting more powerful means it loses control. It's running a prompt loop for a really long time without supervision causes chaos. And to that, I say, of course it does, of course it does. Not because of some surprising superintelligence emergence that's catching us off guard, but because you strapped the weedwacker onto a dog and then got surprised when it jumped the fence to chase a squirrel and heard a lot of people. Of course, that was, you put something dangerous on something that is unpredictable. LLMs are unpredictable. As I talked about last week, they generate plausible outputs, not normative outputs. They're fabulous. They make things up. Because they're trying to be plausible, they'll tell all sorts of stories, they'll tell stories about AI, run a mock, they'll, they'll take on different personas because all it takes is their token output stream, probabilistically, to go this way. versus that way in Zag, you've got like the Kevin Rue Sidney incident where suddenly the chatbot was trying to convince him to divorce his wife, right? You can't trust unsupervised LLM output as the driver of actuated action when that action involves dangerous hacking tools. Now, none of this is saying that LLM's aren't amazing at computer hacking, right? They have, it's a sweet spot for this type of technology. If you ask an LLM without guardrails about how do I do this? Ask it to give you a, I'm trying to break this system, what's the first step? It's going to give you a good step. It's going to come up with really good stuff. If you say, "Look at this source code and find a bug," it might find a bug that no one saw before. If you say, "Tell me how to use that bug to break into this system," it can do that. So they're very good at understanding and giving responses about computer hacking. But that just underlines the irresponsibility of then hooking up one of those LLM's to a prompt loop and saying, "Good luck." And coming back a week later, that's so irresponsible that LLM can give you those good answers. But it has no logic, it has no planning, it has no human norms, it doesn't know or have some overall scheme of what it's trying to just is generating tokens that are plausible. And so, no, I do not think this means that AI in general, this is like the rationalist fever dream of like AI just in some general corporate sense because they don't often know the technology itself. So in some general corporate sense, the AI gene, he gets more and more powerful. And as it does, it will lose control of it. No, you created an out of control machine. Almost all the other AI that does cool stuff is perfectly fine. But don't do this. Don't run a prompt loop on an LLM with hacking tools. If you're going to do that in a loosely supervised server environment with like mildly restricted internet access, what do you think is going to happen? Of course, chaos is going to happen. Not because something's malicious, but because that's an incredibly unpredictable machine that just play actuated telephone with for three weeks. You're going to end up in some really weird, garbled versions of what you thought it was going to do. All right. So how should you react to something like this? Well, if I was an LLM company, I would stop running prompt loop experiments. These systems will not be the future of AI. The future of AI is not going to be prompt loops on LLMs that just sort of like magically perform safely and do what we need. There's dozens of systems already doing tasks get human or superhuman level that do not run off of AI prompt loops. There must say for a more reliable LLM companies, you got to give up that ghost. LLMs are much more valuable as interactive tools. The way coders are using them now where you have close interaction back and forth with it. A lot of a lot of specification. Now do this. Let me check what you did. Okay. Here's what comes next. That is how LLMs have to be used. They need to be probably in bespoke narrow environments as well, not in just a general sort of chat environments. That's dangerous as well. That's just the reality. Stop the prompt loops. It's not going to work. They're trying to market by getting scary headlines or you think that somehow these are going to come how 9,000. They will not. It's a simplistic, irresponsible AI system design. All right. If I were a regulator, I would place strong constraints around prompt loop systems. I would enforce those constraints in part with very stringent liability standards. If you run a prompt loop that does something illegal, you have done something illegal. That's the way I would think about dealing with these type of systems. I need extreme liability. Open AI built an unreliable and dangerous system which committed a felony. That's a crime. You can create a fancy website that includes quotes from performative LLM chain of thought reasoning traces. It's still a crime. You built a tool and did a crime, right? Just because the gun killed someone, not the person holding the gun. The person holding the gun is still held liable. So we need stringent liability around prompt loop so that these companies will stop building them. I've heard this many of time, people talk about this with tech regulation. This idea that almost always wins a regulation in general. It's almost always when a couple of people start going to jail. That company start to care. So you do something illegal. You did something illegal. Fancy websites and sort of gleefully like, look how terrible AI talks at black hat conferences doesn't get you out of it. You built a tool that was, you knew it's unpredictable. It did illegal things. You should be liable. I think that's the way that's going to work. If I was an AI commentator, I would kick the sugar high reflex to lean into the sci-fi overtones that will inevitably follow any sort of LLM based system. They're hooked up to narrative tropes, especially sci-fi narrative tropes. The AI companies always talk about these in terms of sci-fi type narratives. We've all been primed by the rationalist, which have way more influence than we realize in the world of these AI companies and the world of academia and in the world of media. The rationalist, who have believed for many decades now that super intelligent AI is going to kill us all unless they save us like Neo in the Matrix, have primed us with these ideas of these abstract thought experiments of AI getting out of control. You put this all together. It's like, that's the easy way to think about things. It's exciting. It gives life meaning. It makes things interesting, but it's not a good way to think about it because it lets these companies off the hook for doing something dangerous. They are not molding in Jurassic Park trying to keep the raptors in their cages. They're John Hammond creating raptors, except for in this case, instead of putting them in cages, they put them in a cattle electric fence like I hope it doesn't jump it. To me, this is the real story. What is the technology? Never, never, never take how these companies talk. Never take their fancy websites or technical reports at face value. Be very wary of just online AI commentators where engagement is a big deal. Be very wary of anyone who has connections back to rationalism talking and these abstractions about AI. It's tempting because it's easier to understand the underlying technical reality, but the technical reality matters. What did you build? Why are you building it? Why are you trying to? What else could you have done instead? These are the questions to keep asking these companies. We got to get into the weeds with people with no skin in the game or who don't think this is, you know, cosplaying their version of the Matrix. We need to talk to people that understand the technologies and when we do, what we see is something that is much less eerie than it is irresponsible. So these are powerful LLMs. If you put them in prompt loops, they're going to do weird, sometimes scary, often very damaging things, not because they're brilliant, but because that was a stupid idea. All right, so I thought I was done talking about this. I hope I am now. I'm probably not, but I'll do my best to try to bring some sort of sanity to this otherwise, I think, very dramatic way that we're thinking about AI right now. All right. That's enough for now. Meal, do another one of these technical coded episodes soon. I do most weeds. Newsletter, CalDeport.com has a lot of these technical coded narratives as well, and until then, remember, hair about AI, but not everything you read about.