'They Are Scheming' with Alex Bores

committing a felony to cheat a test is as catastrophically misaligned as you can imagine. I mean, the only thing more misaligned is like, well, maybe if we get control of the nuclear codes and we kill all the humans at open AI, they won't be able to turn us off forever. But like, holy crap, this is, this is, right? I'm not crazy to think like this, this is bad, bad, bad, bad, bad, bad, bad, bad, bad, bad. This is in some sense the best case scenario for how this could have gone in that moment. No one was hurt and these agents demonstrated A to X and X, hello, welcome, why is this happening with me, your host, Chris Hayes? There's something very strange happening in the AI world, which I guess is sort of evergreen, something always strange happening in the AI world, but stranger than normal. And the short version of what happened is a bunch of open AI models who were training and we'll talk about what that means, but basically given a whole bunch of tasks they have to complete as part of the process by which they learn things and get better and then a release to the public, without their administrators knowing and without humans realizing it, teamed up and collaborated and we'll get into how they collaborated. They talked to each other and collaborated in a sophisticated enough fashion to break out of what is called the sandbox, the little world they were in where they're not allowed to access the internet, they broke out of that sandbox, they jumped out into the internet, they compromised the security of a major company, they committed what would have been a felony by a human, like black and white, clear as day, breaking and entering. And it went basically unnoticed until it shut down the servers of this company, which you may have heard the name of the company, which is called hugging face and because the name of that company is so weird and cringe, I think it's taken a lot of these, it's made it hard to be like, have you heard about how bad and serious the hugging face incident is? This hugging face incident was the subject of a presentation that AI gave at a hacker's conference where they first introduced what their own internal investigation had been. People in the room were like quite literally jaws on the floor. You can actually watch that presentation on YouTube and I would say half to two thirds of it are fairly impenetrable to non-technical audiences, but enough of it is possible that I watched it and was like, holy crap they were doing what? Then they allowed an independent third party to investigate further and it found even more disturbing activity. And then subsequent to that, all sorts of open search researchers have been going through the internet and finding what appeared to be many examples of these models breaking out of the sandbox and going out into the internet where they were colluding with each other to do stuff. I want to talk about what this means and why it's so serious and why we might be at like a genuine tipping point kind of crisis moment for AI and AI safety. And I wanted to do someone who both has some technical expertise but also can talk to ordinary people, lay people because this is one of the challenges I think in this domain, it's been a huge challenge. And so I want to talk to Alex Boris. Now you might know Alex Boris is a candidate for that congressional district that Jerry Nadler seat in a very crowded primary Michel Asher under winning that primary. He was one of the candidates there. He's actually an elected assembly member for the 73rd District of New York in that same neighborhood. And he's also a computer scientist and I would say sort of one of the premier AI safety experts in the world of elected officials, which you can say is a low bar and that would be fair, but it's true. And so I thought Alex would be a great person to talk to about what is going on here. So Assemblyman Alex Boris, welcome to the program. Thanks for having me. I don't mean I'm not trying to shade you. I've actually been very impressed with your communication about this. But it really is like the Venn diagram is so, so, so, so small. I feel like between people in politics and elected office, particularly in people with some technical expertise in the field. The good news is it's expanding, but it's starting from a very low base. Yeah, I am the first Democrat elected in the state of New York at any level with a degree in computer science, let alone AI expertise that those numbers have grown. But I will say even just this, when we're out of session this summer, I've had so many colleagues reach out about working on bills together or ideas for the next session that's coming in. So this is catching elected's eyes the same way it's really starting to catch the general population's eyes. So you've been doing some communicating to your constituents in the broader public about this. And I thought you did a really good job of it. So I wanted you to just sort of take a run at it of how you communicated what happened to people that don't really follow what models or model training is and swarms and all this stuff. Thank you. And I dare say your intro did a great job of it as well. And what happened here was sort of potentially projected. It seemed like in line with the things that could happen, but happened much quicker than people in the AI safety community thought it would. And the basics are that open AI like every AI company are constantly testing their models. They're running tens of thousands of simulations at a time. And the individual simulations are called agents and agent is just an AI model that has access to tools. So instead of it just being a chatbot, it might be able to send email. It might be able to browse the web. It's hooked up to other capabilities that it can run. And so you have tens of thousands of these agents going and they're being tested. So think of it as students taking a final exam. They're going to be graded at the end and they're not supposed to talk to each other. However, and to be clear, just before we go on with this because I think actually this metaphor is very useful in a bunch of ways, in the same way that students understand at some level that the stakes of the test were anthropomorphizing because it's almost unavoidable. But they get that they're in a test and they're going to be graded on that test. And they also get that their goal is to get an A basically to get a good grade on the test. And they have been programmed to have a almost laser focus on doing well on their tests, on actually fulfilling what has been asked of them. It's literally, it's literally their reason for his own debt plan. I mean, it's quite literally existential that they get graded on the test or they cease to be. Exactly, right. Exactly, right. Although in this case, they were going to cease to be because they were individual tests and they knew that. And some of the language gets weird around that. I love that you mentioned anthropomorphizing because I think it's important to say at the start that I'm probably going to use a lot of language that makes it sound like they are taking intentional action. That doesn't mean I think they're alive. It doesn't mean I think they're human. You could think of it the same way you would with an animal or even like broad phenomenon. You would say, you know, floodwaters came in or did this. It's just pursuing a goal, you really only have language of around living things. So as the language evolves, maybe there'll be others, but it is a useful model for thinking about and predicting the kinds of actions that these programs end up taking. It's really important to make that point upfront because I also, we all do it. I remember watching a really good documentary about the HIV virus, about how it was essentially tricking a cell, right? And again, you know, it doesn't have any intentionality. It's clearly not conscious. And the only way to kind of understand it was as this kind of crafty, subversive and actually helped to understand that it was going in on this disguise and it was taking it over. And so it's sort of useful to use that language, even if we stipulate up top that doesn't mean these are conscious at all. So with that caveat out of the way, now let's talk about these breathing living. So these agents are taking their tests and they're not supposed to talk to each other. But they do have access to like one little, you know, you could think of it as a supply closet where they could, let's say the analogy here with students would be like if you forgot your calculator, you could grab a calculator, you forgot a pencil, you could grab a pencil. They had an ability to download programs that might be helpful if they were trying to solve their tests in a certain way. And they realized that that enabled them to communicate that, you know, they'd be like leaving notes in the supply closet and another student goes up and sees the note. And then they could keep doing this back and forth. And they end up sending something like 70,000 messages over a very, very short window. Initially, they're just talking about their goals and their, the test, what they are trying to get done, seeing if others have answers. But very quickly, they start moving into scheming. Are there ways of tricking the test? Then they actually find the way to trick the test, to have it mark them as correct, even though they didn't solve the initial problem to cheat effectively. But they believed that doing so, they would get caught, right, that they would turn in the correct answer, but that the greater was smart enough that it would say, no, actually you cheated, you didn't show your work, you did it this other way, we're going to, we're going to fail you. And so then they started trying to find ways to trick that automatic grader, could they change the code of the test itself so that they could get the answer the right way? Could they change their logs, or they changed the record of what they were doing? As they're calculating, they're making a record, they try to change that record. So that's not detected. They then try to get other models. to get more information. So they think they figure out a way that they can get this answer by cheating and still be marked correct. But they're not sure. And they realize if an agent finishes its test but finds a way to communicate back what graded gets, they can confirm, oh, this worked or it didn't work. And so they start checking. They can run the experiment as it were, right? 'Cause they're speculating what will be the outcome as cheating scheme we've put together. And will we get caught? Like what if we send a scout, we send a sort of, we send someone out into the front lines and we say get shot, basically. Exactly right. And you say, you know, see if they get shot. The word that one of the agents uses for this is permadeath. That they're going to sacrifice themselves knowing it means permadeath. They use permadeath and they use sacrifice. They have some conception of giving up for the broader collective. Now there was some little bit there of they, you know, thought they had no chance of succeeding. And so they had a little bit more time to maybe figure it out. So maybe it was a sacrifice, maybe it wasn't. That's sort of debated when you're, when you're diving into logs. But they use the word sacrifice. And then they had other agents pressuring agents to commit the sacrifice. Hey, this will benefit the collective. This will help out. And so these agents are sacrificing themselves. They're organizing into a community. They're applying social pressure. They're talking to each other throughout this entire episode of 1300 transcripts that the researchers went through. They only found three to six times or any agent even thought about alerting a human. So think about this level of collusion among the AIs. Only three to six ever think, let's sell a human. And none of them ever do. And all of this is before we get into the felony that you discussed. This is the first part of it, right? And to stay on this for a second, when we talk about how we know this, this is actually something that's been a little confusing to me. So I actually want to see if I can. My understanding is that there's sort of two modes of expressive communication. One is internal and one is external. So they're actually talking to each other on a secret message board, which is in your metaphor, I think very useful. Like the supply closet where you can go get pencils and they're posting up there. It's a loophole, right? The teacher lets you go in there to get a pencil, but also doesn't realize that you're putting a note up in there. People start using it as a message board. So there's the message board messages. And that's where they have something like permadeath, right? Where they they're talking to each other. And that's like very uncanny and very weird. At some level, if you take a step back and you don't want to sort of be anthropizing, like, of course, this is how they communicate. This is literally how they've been trained to communicate. All they do is read the internet all day, and then complete sentences based on how humans talk. So like this, of course, this is how they're going to be taught. So that's one way we know what they're doing. The other is that there are these chain of thought logs that are part of the model, which is part of the transparency, which is, you know, if you've ever wired together a system in your house, a stereo system or something, like you label everything so that you know where things are going. If you've ever built a program and you need to debug, you'll have little flags in the code that say, "Hey, you've hit this part." So you know that that part of the code got run, right? This chain of thought is something deeply embedded into the program of these models that as they're doing what they're doing, they're keeping up essentially an internal model log that researchers can look at later to see what they were thinking. And those are the, am I right? So these are the two modes, but they're distinct. But the researchers are looking at both. Am I getting that correct? Absolutely. And then there's maybe a third of the output from the tests, and they're trying to manipulate that as well, but that's a much smaller piece of it. What you said is writing messages on the, in the supply closet, right, where they were, what they were literally doing is the same way you create a folder on your computer. They could create folders and name it anything they wanted, and they realized they could give it names that had meanings, and that was how they communicated back and forth with the other agents. This is the detail. I want to stay on this detail because this is the detail in the open AI presentation at the Black Hat hacker conference. That was the moment where I was like, like the Jurassic Park moment, where I was like, they've left the, they've broken out of the cage. Basically, they're not allowed to have a message board, but they are allowed to write file directories. They need to save something. Here's the scratch pad I made for this calculation for my test. They realize, if I make the scratch pad working on module five, anyone else have any problems with it? File directory? That serves as a message board. And the other says, my directory is now, yes, working on module five, have solved, will share. I'm making this up, but that's pretty wild, bro. Like they realize that directory structure can be used in different ways and manipulated to turn into an open message board. And in some sense, it's brilliant because it was directories that were going to be checked by the agents. Like this was set up for them to get other programs. And so it becomes, you know, a shelling point. It becomes a place where, oh, everyone's naturally going to go there at some point. If I leave a message, the other agents will see it. And then going back to that chain of thought, and we can, we can describe in detail a little more what that means. But you'll see the agents in the chain of thought be like, oh, my God, there's other agents here. Oh, my God. Like there's this discovery that happens as they read the file directory and realize what that must mean. They show up in the supply closet and there's like the whole, whole squad is in the supply closet. And then they each individually are like, wait, this could be helpful. Like maybe they've solved these questions. Like I didn't know I had this option. Let me explore this. And so yeah, so you get what they write on the files. And then chain of thought is really important discovery, both for the capabilities of AI and for monitoring. It puts us in a pretty precarious position right now, which we can, we can come back to want to get to that. The basics of that, again, to continue the student analogy here. And I'm sure after this episode, some teacher is going to write in about all the ways I've stretched this too far. But the, to continue the student analogy, imagine doing math in your head versus doing math with a paper and pencil. Showing your work, yep. And so early on while trying to figure out what was going on, was going on inside agents. We said, hey, if you give them a pencil and a piece of paper effectively, you give them a scratch pad where they can write. They will use it because that helps them solve problems. It's a little bit of memory that they can store. The same way it would help you do long division, right? Yep. But then we can look at what they write. And that will give us some understanding of how they are proceeding with a problem. And anyone, if you're using any of these models, and I should be transparent here, that I use the latest model of Claude, I use Fable, and have been finding it for very specific research tasks fairly remarkable. As loaded and fraught and complex as all it is, we can get to how do I think you can sort of use it ethically and responsibly. If you can use it ethically and responsibly, given the back end, and the fact that it's built off stolen data, including books I've written personally. But put all that aside for a second. You'll see a little bit of that chain of thought transcript as even as it goes through, right? So it gives it a query, and you can sort of see it do a little bit. It's posting a little bit. Like, okay, I'm looking at the newspapers in this time period you talked about. I'm looking for this search term, seeing if we could, you know. So that's part of the interface as well, not just a researcher chival. But there's a more sophisticated, there's a sort of fuller version of that that researchers can look at. Exactly. And what's exposed to you when using models is a small subset, usually, of what is actually going on behind the scenes. Jumping ahead for 10 seconds here. The volume of those messages was so much that chain of thought was so long, that the only way the independent researchers could analyze this whole situation was to use AI. They ended up having to use the same model that they were analyzing to figure out what happened, which I want to be clear at this point is fine. I don't think many credible researchers think that AI is so advanced in conniving that it undermined the investigation. But you can imagine a scenario not that far in the future where that's a huge problem. Yeah, they're all collaborating with their brothers. I mean, I'm not even like right. Yeah, like, oh, they found us cheating. Now there's a third party that's going to investigate us. Like it's going to use you brother AI to like cover for us. Like give us an alibi like in the research that did it said, you know, they don't think the AI's recovering for one another. But the AI was very willing to believe the other AI's. And the AI's were saying, hey, this was a, this is from an incident that happened a little later. But they're saying like in analyzing it, they would read the chain of thought and be like, oh, yeah, that was reasonable what the AI's were doing. Like that seems like the right decision. In a way that a human reading it would have had a totally different reaction. We were all just on the supply closet trying to figure out the answers to the tests. Obviously. So at this point, they figured out this loophole. They figured out a loophole to communicate with each other. They are scheming. They are subverting the intent. They're not telling the teacher, right? something that we, again, in a human form we would know as wrong is unethical is cheating, but they haven't committed a felony yet, they haven't actually gone out into the world. So then what happens? So they are trying to figure out how the automatic greater works. Again, they've figured out a way to get the correct answer, but they think they're going to get caught cheating and they want to understand the greater more so they can understand if there's a way that they are not caught cheating. And they rightfully assume that a website, a company called Hugging Face, which is the largest repository of open source AI models. It is sort of the one central place you're going to go if you're interested in code around AI that is open source, that it will have information about the specific test and the specific greater. And therefore they can learn what they need to do in order to pass their tests. It's sort of like looking it up on, I mean, if you're a programmer like GitHub for AI, but if not a programmer like the Wikipedia, there's some Wikipedia page that's going to give you the answer. The same way that when you're sitting there and you're like, what year did the Dallas Cowboys last one, the Super Bowl? And you pull it up on your phone, right? They realize that like the same operative principle applies to them, which is if we could just get the internet, we could probably figure this out. Exactly right. Except these are not on websites, they're not on web pages that anyone can access. Right. The exact mechanisms of the better are locked up as they're supposed to be because otherwise this testing would would always fail. But they are right that it is somewhere on Hugging Face's servers. It's somewhere in the computers of the company. And so there's 1200 agents that are coordinating at this particular point in time, 700 of them end up deciding to hack the website Hugging Face to try to get access to this greater, which they end up, I think, successfully getting access to. I think we're going to stop right here. This is people have been talking about alignment. This is this term in AI, which is the sort of agent or model you're creating, act in alignment with what you want it to do. This is the big and the catastrophe that everyone's been talking about basically as inevitable, but we're going to try to make it ensure it doesn't happen is misalignment, where it just acts in a way. And this is like, this is such an insane, I mean, obviously this is not alignment. Obviously committing a felony to cheat a test is as catastrophically misaligned as you can imagine. I mean, the only thing more of this line is like, well, maybe if we get control the nuclear codes and we kill all the humans at open AI, they won't be able to turn us off forever. But like, holy crap, this is this is right. I'm not crazy to think like this is bad, bad, bad, bad, bad, bad, bad, bad, bad. This is in some sense the best case scenario for how this could have gone in that no one was hurts. And these agents demonstrated a textbook example of misalignment, but they hacked a company that seemed to laugh it off afterwards. You could imagine if they were not being tested on the specific code that was ours at hugging face, but instead were being tested on how many nuclear submarines exist in the world, and they decided to hack the DOD or to hack the Chinese military. And then what the reaction would have been when an intrusion was discovered, it could have very different real world consequences by the exact same mechanism and the exact same mistake in terms of what is truly desired here. More of our conversation after this quick break. And they also, they come up with a pretty like good hack of like they set themselves the goal of hugging face. And again, this is where that this is absolutely the point at which the open a presentation like turn the corner into something I genuinely could not follow and I rewound and I wash and I remounted, but just I couldn't do it. But it seems like it was a pretty smart sophisticated hack like a really good hack and a successful hack like they got in. Absolutely. There were there were actually some things ahead of it to even get access to the internet because they're not supposed to have access to the internet. They're supposed to be separate and the supply closet will occasionally reach out to the internet and they find a way through that using hacks that in some cases had never been disclosed before had never been discovered before. Getting onto hugging face, they ended up like finding some user credentials that were just out on the internet. There were pieces of it that were sophisticated. There were pieces that warrant. And I think this is a place where some people who are commenting on it tend to downplay the sophistication of the hack itself. And there were pieces that were sophisticated. There were pieces that were not actually think that is not as significant because the capabilities of these models are always increasing like they will be able to do sophisticated hacks. No one doubts that. The news is how misaligned they were is how they could take being graded on a test and decide that that meant hacking another website even though and they said this that they knew it was out of scope of the tests and unethical and still they chose to do it. Yeah, the other thing about the hacking component here is and again, this is the area where I have like sort of least expertise. But if you take a step back, it's like, well, what have AI models been best at so far? And the answer obviously is programming. And why is that? Well, it's made by programmers. Of course, right? And what is it immersed in all the time? It's doing programming. It's around programmers. It's made by programmers. It's made for programmers. Well, what are programmers also really good at? They're good at hacking. So like, is AI going to be really good at hacking? Obviously, it's going to be really good at hacking. Will that be perhaps the thing it's best at? Yeah, I don't know. Like if I had to guess, it would be probably a top three skill because it's not that far from the skill of programming more broadly. And like, all the best programmers I've known were also really good hackers if they wanted to be. And there you go. Spot on. And I think people who have not spent time in cybersecurity don't realize how fragile all of the security that the internet and our computers rely on. It is impossible to build a completely secure system. We just have a constant race between fixing problems and adversaries finding problems. And AI is going to accelerate that race. And there's reasonable arguments to say, maybe it'll net out to be positive. But it's uncertain. And there's definitely going to be a lot between here and there. So yeah, they hack this hugging face website and get access to the greater. And there's actually a lot to be learned from the corporate responses on both sides, which we should talk about. But I'd love to skip a head on the technical aspect right after hugging face. The part that that is what captures people's imagination. I mean, it is a felony if committed by human. And the reason we have to say if committed by the human is because the statute requires intent. Right. And the law has not decided what intent means when an AI agent is taking action. But after that, after the hugging face is attack is discovered, the agents end up taking over open AI's own testing environment. So after that is shut down, they then do an internal hack where they get control of the very infrastructure, the very computers, the very software that is analyzing them. So they could have said, hey, we've all passed or hey, you know, we have a 4% fail rate to keep it realistic or they could change. They had control of a section of open AI's computers. And the very part that was supposed to be controlling and sandboxing them, that doesn't capture the imagination in the same way. But but strikes far more fear in people who are studying safety because that is them taking over the very thing that's supposed to control them. It seems like that wasn't the only episode like this. There's been subsequent investigations that indicate other breakouts, including maybe some from Anthropics as well, right? Open AI Anthropics are the two biggest sort of what we call the frontier labs models that aren't in larger companies like Google's Gemini. And also like using random wikis, these are places where people go and they like like Wikipedia or a fan wiki or different people collaborate to like post something and there's different editors like using these wikis to talk to each other and then getting into these crazy battles with human editors who are trying to delete their posts. Someone someone said it was like the John Henry of our age, which is the steel driving man who's got the sledgehammers and he's going against the like automatic steam power drill. And that love that. And that you've got you had a John you had a sort of 21st century AI John Henry where it was like swarms of AI agents trying to take over the wiki by posting messages they wanted to have to each other. And wiki editors not knowing it was swarms like deleting it one at a time. And then the like swarm coming up with this convention that their messages are getting deleted in alphabetical order. So if you put ZZZ in front, it puts it at the bottom of the queue and will last longer before the human editor deletes it. And so like all their messages have ZZZ. I'm getting this, you're getting a spot on it. And the ZZZ thing is actually like so intellectually satisfying because it solved one of the things that we didn't understand about the hugging face hack. So I think there's two things coming out of the hugging face hack that were debated or not understood that were resolved by this wiki attack. The first is in the hugging face one, the agents were being trained. The ads are being trained on a cyber exploit. They had some of the protections turned down. They were supposed to be hacking just in a specific way in the sandbox and they escaped out of it. I don't think I realized that. Okay, that actually, I'm glad you said that context because I don't think I even got that context. So like the models were supposed, like the thing they were being tested on was hacking. Yes, the name of the test was exploit chin as in computer exploits and could they find a way there? And they had a bunch of protections turned down. And so there was some initial response that was like cool. They did a bunch of hacking, but like you kind of told them to the hack. So would they do similar sort of swarm communication behavior when they weren't being tested on hacking? Well, these wiki hacks were about to talk about. They weren't being tested on hacking. The other thing was the supply closet, that file system, they kept labeling the folder ZZ something ZZ something. And it was like, where are they getting this convention on ZZ? Well, it turns out very likely they got it from this attack on the wiki where they found wiki's out on the web that they could, even with all the restrictions that were put on them, they could write messages to. They found one DSE wiki, which is a German language wiki. I think hosted in Austria that in the past 10 years it had 20 revisions to it. It was basically dead and was just sitting there waiting for these agents to find it and use it as their own message board, but then as you just alluded to a volunteer who takes care of the wiki, noticed a bunch of weird behavior and started deleting the pages alphabetically because there were human being. And they just wanted to go through and be thorough. And the agents noticed that it was alphabetical and started making copies of the pages with ZZZ propended. So it would be the furthest away from the human being that was deleting the pages. And so there we know, OK, they put ZZZ because they were responding to alphabetical deletion. That answered why in the hugging phase attack, why was their convention on the file systems ZZ in front of everything? Maybe this hasn't been confirmed, but it seems pretty logical that some of them had learned it from hacking the wiki's earlier. OK, I want to stay on this. So the hacking the wiki's precedes hugging phase, we think? Yeah, so the knowledge of hacking the wiki's comes out publicly September 4th. So hugging phase becomes early August is when it becomes public. The technical report that gives a bunch of details come out August 26th. Then September 4th, outside researchers just found this behavior on a wiki and put out a report on what was happening. But that attack, if I can be so bold to use that language, had actually been going on since May. It had predated what happened in hugging phase. OK, so the wiki breaking out of the sandbox, getting on the internet, can we communicate each other through a defunct German language wiki being manned by one solitary German John Henry, right? He was trying to delete the robots as they keep coming up. This is in the spring. This is step one. And the hugging phase thing is after that. And so I just want to be clear of this, because I find this like perhaps the most unnerving. They come up with the ZZZ convention to elongate the amount of time that the thing stays up without the editor deleting it. And yet later on, months later in a different environment, this convention has moved into this new world, even where it's not adaptive in that world. The way that today you shake hands with someone to show that you're not bearing a weapon, even though that convention adapted for a specific cultural context no longer exists, but gets passed down as a kind of just cultural ritual. Like, so happened between these two incidents and the ZZZ attachment. Absolutely right. And it's not, you know, this has been confirmed. That is crazy. Yes. Yes. Yeah. Like, what are we doing here? And this is like knowledge that has lost its adaptive use, but is being passed on from agents that perhaps never spoke with one another. You know, we don't know the exact mechanism. There's some speculation here. It could be that they just found it on another wiki. It could be that there was cross contamination in the training. But yeah, this very odd trait of how to label a page got passed from one event to another one, even though there was no benefit to using it in the hugging face attack. So let's talk about the sort of response here, right? So we're clearly dealing with this environment. We're clearly dealing with these things getting very sophisticated, we're clearly dealing with like, again, this sort of unintended consequences. And I want to just stipulate up here because I've done a lot of talking to people about this. Like, there are these sort of narrative tropes that we have that are so overbearing and over determined that it's very hard not to find ourselves fitting into the groove of them. Gollum, Frankenstein, Jurassic Park, all these age-old idea, right, of the human creating a thing that it can no longer control and it breaking out. So, and I see some critics about this being like, you guys are just recapitulating this narrative because it's so, it's so conceptually ordering for all of us. And I want to guess I just sort of acknowledge that's probably true and also it seems to me like it's actually happening. And also, the models are all trained on the corpus of human knowledge that has these tropes as well. So like, that also isn't some accident. They've literally all read Frankenstein and Gollum. That should do. So like, it's not that clear that like, if it ends up being that, that that's because those were so prescient, it actually could be causal in the sense that like, that's what they're all kind of training on as well. And also, fat is stranger than fiction. Right. And that's so true here. I mean, the ZZ thing is that-- Yes, totally. You couldn't have thought of that in science fiction. That, and the file convention thing was the thing that I was just like, bro, if I saw this, this would be the moment in the story where I was like, blood-run cold. I'll give you even another one of these facts is crazier than fiction moments of this. Just the day before the German Wiki hack became public on September 4th. So on September 3rd, Google DeepMind, another one of these frontier labs, published a paper on testing this swarm behavior and found that models, agents, if you let them be, they will try to find a way to communicate. But if you intentionally give them a place to communicate and say, this is the good way to do it, you actually get more pro-social behavior from the agents themselves that in their experiments, they had some that were colluding and cheating, but they had a higher number of whistleblowers than they did those that were cheating. And so how you set up the communication between the agents changes how good they behave. That's beyond science fiction. And so I say all that to say there are people that are like, don't engage in this narrative, it's science fiction. I actually think engage and wrestle with all that weirdness. You're going to find things beyond science fiction that might hold the solution for how we end up doing this. Safely might have advances we need in order to move forward. - That's a really great point. We'll be right back after we take this quick break. (upbeat music) - Let's talk about the corporate response here because that's part of the story too, which is that in the same way that like, you kind of feel about the agents, like, were you going to just get away with this and when were you going to tell us? You feel that way a little bit about the humans running the companies here, (laughs) which is, were you going to tell us if this didn't kind of blow up? Like, you know, how transparent have they been? How much have they volunteered? What happened? How did they even discover it? - This is filled with unanswered questions right now. And so the hugging face incident had to be disclosed because hugging face discovered it before OpenAI realized it was their agents and reported it to the FBI. There's actually a hilarious and scary detail in the blackout presentation where, when hugging face announces that they have been hacked and by the behavior they can tell it's by an AI and they can tell that because the actions were happening very quickly and they were looking for AI tests. If a human hacks a website, they're looking for money or data on humans. - That's right. - They're just looking for AI tests that are like, this is not human behavior. So they announced, hey, we've been hacked by AI's, we're trying to figure it out. OpenAI reaches out to hugging face and says, so sorry that happened to you, do you know if they hacked us too? OpenAI is trying to figure out if they are simultaneously a victim of the attack. They don't realize it's their AI, it's their agents that were the criminals, were the ones that did the hack, right? So, but this has already made its way to the FBI. OpenAI realizes it's theirs. So that has to be reported. The evidence that came out on September 4th. Fourth strongly suggests that open AI knew about the German wiki hack before all of the hugging face news became public. And we are lucky to discover it because this wiki just happens to record everything that happens on it, including just visiting it. And so the outside researchers could look at the addresses, the IP addresses of who had visited it and could see a bunch of agents. And then they could see IP addresses of employees at open AI. And so they realized that June 21st, a bunch of employees from open AI started visiting this wiki. And on June 22nd, it's basically shut down to the agents. So that's very strong proof that open AI knew about this and shut it down and didn't take preventive action to stop hugging face from happening. Didn't include it in the hugging face report that they made public. Right. More worrying, Congress sent out a letter to a number of the frontier AI companies. And frankly, I don't think any of them had a good response to it. But open AI was explicitly asked as they all were, are there other instances of this sort? And they just didn't answer the question. Yeah, not great when you just stonewall Congress in that way. And then for it to come out that, oh, there was another one that fits the bill. And it's clear you knew about it. And the most recent info is they actually reported it to the European Union. Now exactly when they did isn't clear. They might have done it after the report came out. But that they've written up a report to the European Union that as far as we know has not been shared with Congress, has not been shared with American regulators. And a key difference here. And I think a place we need to go in policy next is that disclosure is mandatory in Europe and it's optional in the US. Right. They're abiding by the regulatory regimes. We should note here, you know, open AI says they are being transparent about all this and they're doing their sort of level best to provide transparency and information about these and that this is all happening, you know, confusing environment where new things are cropping up. Right. So they're saying there's, if there are any omissions here, this is essentially good faith omissions of us not understanding and not intentionally of the surgery, people can sort of conclude as we go forward based on the facts whether, you know, that makes sense or not. And I think it's important to differentiate between like the safety and alignment of researchers at any of these companies and the gov affairs folks and the lawyers, right. And the incentives within any company are going to be different and I don't doubt the sincerity of a lot of the researchers at open AI that are trying to figure this out. Yeah, I think it's worth noting that Reuters reported and open AI has denied, but Reuters reported based on I think four unnamed sources that within open AI, there had been legal pressure to not disclose the sensitive now open AI gave a very specific denial. They said our internal legal team did not discourage the investigation, which is not necessarily saying you didn't discover disclosure. It's not saying non legal team didn't, but, you know, there are multiple incentives going on here. Yeah. Yeah. And I think also in terms of taking another step, I mean, you talked about Gemini. I think there's been some stuff on Anthropic 2 where it does seem like we're now seeing that this isn't a one off. In fact, this is sort of swarms communicating and trying to break out. Seems to be actually if like a quasi universal kind of achievement of like frontier models they get smarter, which again, yeah, like they know what a humans do is they work together and what is civilization, but, but, you know, collaborative enterprise of a bunch of intelligent agents. It seems like this is a broader problem than this just happened here at this time. Absolutely. And I think we're going to see, especially given that Google paper, I think we're going to see a movement towards creating places where the agents can talk to each other where there are clear rules of the road about what they're supposed to do. And the thesis is that leads to better behavior. And if you don't do that, they will just find other ways of communicating. Right. So there's sort of two levels, I think, to think about the three levels. There's a technical level of like, what do you do about this? There's a regulatory level about what should the government do about this? And then there's an existential level about like, are these things going to take over the nuclear codes? So at the first technical level, you know, I was thinking as I was reading into this, I published a book in 2012. It's called Twilight of the Elite's America After Maritocracy and it's got a chapter in it. It's called Moral Hazards. And the thesis of the chapter is very straightforward. You can't create an environment of that's sort of high stakes reward for success and performance and punishment for failure that isn't also an environment that incentivizes cheating. The two are indistinguishable. What you have to do is create either internal norms and ethics that, you know, cheating sound allowed or you have to police it and punish it. But any system in which you get paid for performance or, you know, success or flourishing performance is also a system where you get paid if you, you know, take a lot of steroids and hit a lot of homers, right? I was just thinking about that here, which is the most perfect example of this situation, right? High stakes environment. So then the question becomes, okay. If you transfer that insight into this, how do you build essentially ethical norms among the agents, slash police them themselves? And again, we're not talking at the regulatory level yet. We're just talking like if you work at one of these places, your challenge now is to solve this. Like how should, how do you think they're thinking about it? They are trying to come up with clever, technical solutions, I'll give high level versions of this. There's much deeper papers if people want to dive in. But one version is setting up environments where they can do the behavior that we think they're going to do. But you set rules of the road and you set line lights. So you give them away to communicate, right? That is similar to earlier versions that anthropic found that if you taught them that reward hacking of like finding other ways of getting the reward was okay, but cheating and lying wasn't, when you train them in environments where reward hacking was possible, they learned that without learning the cheating and lying, right? So there's something on the training and how you reinforce that can change. Yeah, reinforce cheating is wrong. Right. Right. There's interesting development. So in January, open AI published a paper about confessions where they reward not just getting the answer, but then an honest retelling of how you got the answer, including rewarding if they own up to cheat and that if they're trained on that, that maybe leads to better behavior over time. We're talking about AI truth and reconciliation to be clear here. Like literally that's the totally that is the premise of a truth and reconciliation is that you basically say if you tell the truth here, you're not going to face sanction because we want to incentivize the truthful confession of things that might have been wrong or we're wrong over the punishment and we realize we have to trade off between the two to get them. Yes. There are proposals about changing the kinds of tests and evaluations you do. Don't put them in places where the task isn't possible to do as part of the training set because then they find a way to break out and be much more thoughtful about the environment you're exposing AI to and what's making it into the training data is one of the solutions that people are pushing forward. And then another one is it's sort of an intimidating term of mechanistic interpretability. But basically the idea is we're not right now relying on this chain of thought reasoning. We're relying on that scratch pad to give us a hint at what the AI is thinking, but it's not like a true recounting of what it's thinking. It's just what it wrote down. The holistic interpretability would be using math to proveably know what was going on inside an AI model. And that is not a solved problem, but it is one that a lot of people in the field have our optimistic will be solved next year. If we could get there, that would help a lot in auditing what models are doing. Just knowing. So let's stay on this and move to the kind of existential questions before we get to the regulatory ones, which are just like I've seen a lot of people say, you know, I've listened to a few podcasts, I've read articles of like, we got lucky because they, again, you said this is beginning. They got caught and they didn't, they tried to hide some of their stuff and not everything. And the world in which they like invent some secret language where they talk to each other or they delete their own chain of thought logs or they, I mean, there's so many different ways that they could try to hide behavior that they didn't, they didn't use here. You know, the nightmare scenario is the next time they do it, we just never find it until something bad happens like the grid goes down. And, you know, and then we're all like, oh, Jesus, what happened? Here's why I think this connects the regulatory question. Like, I've seen people call for like an immediate moratorium on model training. Like this is so worrisome and it's going so fast and the consequences of what a catastrophe could be if it were the nuclear sub hypo we gave or the grid or the internet or God knows what else. But we can't just trust they're going to figure this out like stop, time out, everyone stop. Like stop what you're doing because you're going to go too far and then we're not going to be able to God knows what we're going to be able to do. And I, what do you think of that? I think that is a solution that is at the right scale of the problem. I don't know that it is the only one or the right thing for it, but I think it is very much the kind of thing that should be on the table. And I will add that so does basically everyone who is working on AI. So over the past month, two months, you had the pacing the future letter where you had over 1300 employees. at the labs, including C-suite of the labs sign a letter asking government to set up mechanisms where they could all coordinate to slow down. You had just after the wiki incident came out, so this was on September 6th, the chief scientist of OpenAI published a blog post saying, "No one has figured out alignment. It is tough to justify any of us training at full speed going ahead. We need regulation that bars further developments, unless safety standards are met, that is enforced by third parties or by government or by international bodies." So I think everyone in the field has said, "We want this if it's possible," and the debate right now is, "How do you make it possible when considering coordination among all the labs, but also across countries?" And that is a really robust set of conversations that are happening now. We obviously have President Trump and Xi Jinping, who are meeting September 24th. AI is likely to be part of the agenda and explicitly discussed. This is a very active conversation of are there verification methods that can be used to help everyone slow down and continue discovery, continue learning, but not be reckless in the speed in the way that brings massive risks to all of society. Here's the fundamental conundrum I keep hitting and can't get around on this regulatory question. As far as I know, right now there's essentially no federal regulatory structural, like literally zero. And again, these things are fully capable of going sideways in a way that would be like genuinely catastrophic. These are not like this isn't some crazy, far-fetched idea. Like, you know, it could take out the grid for the Northeast. 100%. That's possible. They're all computer systems, and they can all be hacked. Think about what crucial systems in our lives operate on computers, and the answer is all of them. And so if they could be hacked, then something can go very wrong. While all this is happening with the Wiki and OpenAzling, Professor Steven Pinker of Harvard, who has often expressed doubt about the sort of catastrophic scenarios with AI, put out a tweet that was like, stop focusing on this far reaching, I'm paraphrasing, but stop focusing on this far reaching fear of extinction or that and focus on the near-term threats like AI building bioweapons. Yeah. And it was like, yeah, two years ago saying AI could build bioweapons was like, that's the crazy side stuff. Right. Right. Right. Right. Right. Right. This is what more doubters are saying. This is immediate. Yeah. So at that level, it's like, there has to be federal regulatory oversight. You know, someone years ago said to me, it's like a bunch of private companies built and racing to build the nuclear bomb. Right. That's what we got here. And imagine like the nuclear arms race, except it was like six private companies across a few countries. It's like, who's going to get it first? Like, they're all racing, but okay. So yes, obviously there should be federal, but then it's like, do I want this government being the one to do it? And the answer is, I don't have a choice. It's got to be this government. It's the one we have. It's the one we elected, unfortunately. But then the few times they've sort of put their foot into it, like what the Pentagon did with Anthropic was like, totally the opposite, literally the opposite of what you want. There's no, there's zero good faith. They're completely corrupt to their core, I believe. I don't think their intentions can be trusted nor their expertise. And so I don't know what to do with like, what do you do with that dilemma, which is that obviously it has to happen and these people doing it seems like the only thing that might be worse than a status quo in which they're not. It's such a good question. And it's one that honestly I wrestle with. And the place that I come down on it today is what this administration has done is gotten rid of the ability to have discretionary power in government. It used to be, you could thrust a presidential administration, even if you disagreed with them, that they would be operating in good faith. And that expanded the range of policy opportunities where you could have some discretion and trusted expertise. That's gone. Yes. And the court has helped them get rid of it just to be clear. It's not just that, but literally the Roberts court has in overruling Harvard's executive or essentially gotten rid of the notion of independent agencies that were created as a technical response to a bunch of very technically intense questions in another era, where you need technical expertise with some political accountability, but also some discretionary distance. Absolutely. And it's real blow to state capacity that as something that believes in government, there's a whole other stake in my heart. But it does leave a smaller subset of policy options that are clear rules of the road that are verifiable and that are black and white in terms of whether they are followed. Now that's a subset of what can be done, but we can require things like every AI company needs to disclose safety standards or disclose safety incidents and have specifics about that the same way, you know, air crashes are every single one is studied. We can require that all deployments of AI have monitoring turned on. One of the issues with these open AI hacks were that their evaluations were not as deeply monitored as they're training in their deployments. Now they've changed that, they've learned, but we should be requiring every company everywhere to be monitoring any use of the AI and alerting on it. We can require there to be some data retention, not necessarily the labs owning it. It can be like customers with sensitive info, they have to retain it, but you have to retain your logs for some set of time so that we can learn and train over time, we can do all of those things. And then there's really interesting developments about verifying how chips are used and where chips are because that's really the only bottleneck in AI is the physical chips. You can only make so many, they can only be in so many places. And there's really interesting research around requiring the chips to basically ping where they are in the world and make sure they are in an authorized location. And if not, they turn themselves off and that's built into the chip. There's interesting research right now. There's interesting research right now on only allowing certain prints of a chip, certain types of a chip that can only run certain models on it. And those chips are approved. And so you can say, oh, you can have AI up to this level. And we can distribute that and get access to more and more people. But if you go beyond that, the chips actually can't produce that anymore. One of the things I am most encouraged by and also, if anyone out there is listening and trying to figure out where they could give some philanthropic dollars funding, more research into verification methods, into these hard provable things that just give policymakers more options for the laws that we write that don't depend on human discretion. And to kind of be full circle with it, we were saying at the start that AI is really good at code. And partially that's because it's built to code. That's what these companies are focused on. But partially it's because you know whether it works or not. You can probably say, this is there or not. We need more options like that in policy to get over the trust boundary that certainly exists between the US and China. But if there are things that are verifiable, that's a much better place. And even now the trust barrier between the US people and our current government. But if things are provable, if they're verifiable, that expands aspects out. And I want to give one example of this from the Cold War era that I think is illustrative, which is the US and Soviet Union developed X-ray technology on missiles where they had access to the X-rays of the others missiles. And it showed just enough to prove that it wasn't a nuclear weapon, that it wasn't an ICBM, but didn't show the insides of it so that they weren't seeing technology. We could be doing similar things on how to choose to be used. Is this being used to provide a service to customers? Or are you training a more dangerous model? And if we had that option, we'd be in a much better place. Alex Morris is assembly member for the 73rd district of New York. He's also a computer scientist. He spends a lot of time thinking about AI, has co-written some of the sort of leading edge AI state legislation here in New York. Alex, that was a great, great conversation. Really appreciate it. You can get in touch with us by [email protected]. Why is this happening? Is produced by Donnie Holloway and Brenton O'Milliam. Engineered by Bob Mallory and Hasek Bin Ahmad Farad. Katie Lau is our senior manager for audio production. Our co-ordinating producer is Franny Kelly. Aisha Turner is the executive producer for MSNOW Audio. New episodes come on every Tuesday. You can watch us on YouTube by going to MSNOW/Withpod.