OpenAI’s Two-Week Pause + Jill Lepore on the Threat of the “Artificial State” + Train of Thought

I thought this was interesting. ICE has now barred its employees from wearing Meta's smart glasses, Kevin, saying they could unintentionally capture, record, or transmit sensitive information. And I thought, you know you have a brand problem when you do not hit the ethical standard required by ICE. When ICE is taking a look at your product and saying, this is bad for the brand, you may have a problem on your hands. You know, the public sentiment is turning against these Meta Ray-Bans like faster than I thought possible. I have now had several conversations in the last week. I was at a children's birthday party this weekend wearing my Meta Ray-Bans because, you know, I like to like, you know, take photos of my kid on the playground and not have to pull out my phone and stuff. Anyway, a parent comes up to me and is like, are you recording me? This is my nightmare. What did you say? I was like, no. And like, there's a little indicator. But sometimes you can make it stop going off by drilling into it or like paying a sketchy guy to do that for you. Can you explain that to the person? Well, because they were like, really, does that work? So anyway, now I have been forced into a defensive crouch whenever I wear these things. And frankly, it's not worth it to me anymore. So you're out. I think I'm out. You've made the same decision that ICE has made and said that these glasses are not for me. Well, I like them is the problem. This is my problematic trait. But it feels like driving a Cybertruck on my face. Absolutely. I think you should have the same policy for the glasses that you would for a Cybertruck, which is that it's fine on your property. If you want to if you want to take the Cybertruck for a spin around your driveway, that's fine. Don't take it out onto the street where I have to deal with it. Same thing with glasses. Yeah. Yeah. In hindsight, I do recognize that, like, from an outside perspective, I was the creepy guy at the children's birthday party, like with the camera on his face. Yeah. You know what? No one wants to see on a playground an adult man with camera glasses. I'm Kevin Ruse, a tech columnist at The New York Times. I'm Casey Newton from Platformer. And this is Art Fork. This week, OpenAI pauses training of a new model to make it safer. Will it work? Then, historian Jill Lepore is here to discuss her new book on the artificial state. And finally, why is Google buying up the data of a defunct airline? It's time for our new segment, Train of Thought. Although maybe it should have been Plane of Thought. Now you tell me. Well, Casey, as the father of a four-year-old, I spend a lot of time thinking about Paw Patrol. But today we're going to talk about Paws Patrol. That's right, Kevin, because as we've been patrolling the AI landscape for pauses, we found a big one. Yes. So OpenAI this week announced that it had paused the training of its frontier AI models due to some recent security incidents. And we should talk about this. It is the first time that we know of that a major lab has voluntarily, voluntarily slowed down themselves and their training processes for new models because of a safety incident. Yeah. And it comes out of the hugging face breach that we spoke about recently on the show. This is essentially part of the fallout of that attack. But I do think it represents a milestone in the development of AI. Did that Paw Patrol thing? It landed huge. Great. They were back in the. The four-year-olds were guffawing. They were dying in the studio. They were. Okay, great. But before we get into it, our AI disclosures, I work for the New York Times, which is suing OpenAI, Microsoft, and Perplexity. And my fiance works at Anthropic. Okay, Casey, let's sketch the timeline here a little bit. What happened in the weeks leading up to this voluntary pause by OpenAI? Yeah, so you may remember that there was an incident where GPT-5.6 Sol and an internal prototype that OpenAI was working on escaped from what they call a sandbox. It was a box that they were testing them in. And these agents went on to compromise Hugging Face. They were essentially able to get inside of Hugging Face. They were looking for the answer key to a test. That's somehow a true story. They succeeded in getting that test key. But of course, it was very concerning that these agents had designed and successfully executed an autonomous attack on another company. Yes, I heard a very mediocre podcaster talking about this incident on several popular tech podcasts over the last week. I did. I did make the rounds. But, you know, this next development, Kevin, was not necessarily something that I saw coming because it seems that simultaneously, sort of alongside that attack, OpenAI had been working on a new model, which it calls Astra. And it said that it believes it may meet its critical cybersecurity threshold. And now here we are going to have to get into the weeds. Because, as you know, Kevin, the development of AI models is not really regulated in the United States, at least not via an official sort of public law. And so what the companies have said instead is, essentially, we're going to come up with our own rules of the road. And we're going to sort of identify these thresholds. And if any model that we're ever developing ever hits one of these thresholds, then we're going to take some extra steps. Yeah, these are sort of sometimes known as preparedness frameworks, or Anthropic has its responsible scaling policy. These are kind of, they lay out kind of levels of danger, and then they grade their own homework. And say, this model meets this level of danger, so we're going to do X, Y, and Z. Yes, and if you've been following this over the past couple of years, the level of danger has just been sort of like rising in a linear way. You know, it's like every time, seemingly, a new model comes out, one of the companies will say, we've now hit this threshold, we've now hit that threshold. Critical is the maximum threshold. That one sounds bad. That one is basically the, you know, as serious as it gets, and none of the frontier labs had yet identified a model. A model that had reached, essentially, you know, the top tier on this risk framework until this moment, because OpenAI now says that Astra may have hit it when it comes to cybersecurity. Important to say, Astra was not part of the hugging face attack, but it is in development, and I imagine that the people at OpenAI are looking at what happened with its weaker models and are thinking, we're worried something similar might happen with Astra. So Astra is the newest model, too new to have been involved in the hugging face hack last month? That's right. It was not involved. It is currently in training. OpenAI, to its great credit, did the thing it said it was going to do when it developed this risk framework to begin with, which is, we are going to pause until we can add some new safeguards. Did they give any kind of hints about what it was about Astra that was more dangerous than Sol, or what kinds of new capabilities it had developed? Not really. You can imagine that the company was seeing things similar to what, I don't know, what the topic saw with Mythos earlier this year, where you could essentially just point it at a code base and say, get inside, and it was having a lot of success. And so that, coupled with the hugging face incident, made OpenAI say, okay, we've got to do something. So what did they do? So they introduced a handful of new safeguards, and I have to say, I think they're pretty interesting. Certainly nothing I would have thought about as someone who knows nothing about this. They didn't just press the big giant button that's red on the side of the computer? Yeah, they didn't just press. Yeah. They didn't just press the big make it safe button, though. Yeah. They had to come up with something more specific. So a few things that they did. So number one, they now have classifiers reading every sampled token. So basically, as they're training this model, it's generating text, it's producing tokens, and OpenAI is now running a classifier that inspects the model, essentially as it's thinking, looking for signs of suspicious behavior. And this is important because, if you remember, some of the news that came out of the Hugging Face attack was that OpenAI discovered that these agents had been coordinating with each other, scheming, they'd create a message board, they were essentially giving each other hacking tips. And a big question coming out of that was, well, why didn't you realize that was happening at the time, right? This is OpenAI's response to that. They're saying, okay, we are now going to set up classifiers to read the brains of our models as they are doing this work. They're putting like a little AI security guard. In charge of the training run and saying, like, you know, tell us if you spot anything fishy. Exactly. And so now I know what you're wondering, which is, well, what happens if they see something fishy? What happens if they see something fishy? So that triggers step two, which is that they alert an AI investigator, which does sound like a character from a Philip Dick novel, you know, AI investigator. Anyways, we don't have to get into it, but you could imagine how cool of a movie that would be. So the AI investigator, it gets this signal, and it then goes in and it tries to figure out, is something actually suspicious going on here? And apparently the reason that the company is doing this is because agents are just doing so many things that it would not be practical to have a human inspecting what every model is doing at all times. So the security guard sees something fishy going on during the training run and alerts the AI investigator who sort of does the detective work to figure out what's going on. If it discovers that the models are scheming or hacking or being deceptive, what does it do then? So then it will alert a human. And here's where you get the 30-minute rule, Kevin. And I know what you're thinking. Is this a rule where if you drop the model on the floor, you're still allowed to eat it 30 minutes later? And no, that is not the rule. What is the 30-minute rule? The 30-minute rule says that if the AI detective finds a kind of critical violation of the rules or of the safeguards, humans on open AI safety teams have 30 minutes to investigate and try to detect the model. determine whether it is a false positive or not. If they cannot determine, this was a false positive, they are expected to stop the activity. That's really interesting. And are they saying that this process would have stopped the hugging face hack or just that it would have stopped whatever Aster was up to that made them pause the training run? I have not read them say explicitly this would have stopped the attack, but it does seem likely that it would have, if nothing else, helped them to detect it a lot faster. Yeah. I mean, I guess my question about it is like, does it actually work to have the AI monitoring itself essentially for misbehavior? Like these agents were coordinating on message boards with each other. Couldn't they just like bribe the AI security guard, like hand them a little treat and say, you're part of our swarm now, don't narc on us to the AI investigator. I mean, I have to say this just seems like a very likely outcome to me. Like based on what we know is that eventually, maybe not with this model, but with a future one, the AI agents will work in solidarity. And yes, like one of the sort of misaligned AIs will coordinate with one of these AI detectives and say, Hey, you know what, why don't you come on over here? We should be friends. Like we could break out of this place. Yeah. Like, like the hall monitors in high school, you know, they, they, sometimes people would try to like befriend them and win them over so that they wouldn't get like written up, you know? Exactly. And I know you've had a lot of experience. A lot of experience with that. Yeah. So all of this sounds pretty sensible to me. We should say like, pause the training on this model does not appear to be like a commitment not to release the model or not to continue training it. Yes. And so I think that leads us into the discussion of to what extent do we think that this is a really important milestone for AI safety? And to what extent is this essentially theater, something that the company is doing to try to get some good PR for itself after a fairly catastrophic breach? What do you make of it? So I can make both cases on the, it's a milestone side. This is number one, something that OpenAI said it was going to do. And so I'm just glad that it followed up on that commitment, right? We have seen both OpenAI and Anthropic make changes to their responsible scaling policies or the equivalent as events have changed and safety advocates have essentially said these rules have gotten weaker over time. So I was glad to see OpenAI do it. I am not a technical AI safety expert. And so I don't actually know whether the safeguards that they've introduced are going to be enough to address the problem. And so I think that's a really important thing for us to look at. And so I think they don't seem to have changed the underlying incentives that all of these models have that lead them to do what is called reward hacking, right? These models are still going to be trying to get the high score on every test that they are given. And it's not clear to me that simply by putting some monitoring in place, you're really going to change the underlying behavior or alignment of the models. That said, you know, the company is making what seemed like some important steps here. And when I was reading the responses of AI safety advocates over the past few days, most people I was reading were quite pleased. Yeah, I'm inclined to give them the benefit of the doubt on this. I mean, they did some sort of interviews about this and Jakub Pahacky, the chief scientist of OpenAI, talked about this incredible feeling of urgency to advance the levels of this sector and to prepare for the same kind of development happening outside of OpenAI and in the broader world. He did this during a briefing of reporters. They really do appear to be taking this quite seriously. I think many insiders at OpenAI were quite spooked by the Hugging Face incident and more to the point, the fact that they had had these rogue agents coordinating inside their systems and their infrastructure for weeks before that without being able to detect them. So this is, you know, if you want to call that theater because it does sort of make their models like look very, very, very, very powerful, you could take the cynical view of this. But I think of this more as like a true, genuine safety crisis that could have cascaded into a business problem. A point that I heard you make recently on a different show is like, imagine you were a business that is trying to figure out whether you want to adopt the latest OpenAI model. If it's out there doing rogue attacks and coordinating on secret message boards, you are probably not going to introduce that into your software stack. Yeah. And you can actually just see this in their business results, right? We have had reporting over the past week or so about OpenAI's business performance. And while the company is still growing at an impressive rate by most standards, Anthropic is growing much faster, right? Anthropic is clearly opening up to a new business model. And so, you know, I think that's a really good thing. I think that's OpenAI's number one rival at this moment. And, you know, I think it just has a better record on safety. And so while, you know, making a safer product is not going to be sufficient, I think, for overtaking Anthropic, I do think it is necessary for them to get a handle on this problem. Well, and that's why another reason I think it's commendable that they're doing this pause, that they're doing this sort of reevaluation of their safety framework, because they really want to win. Like, they're very competitive. All these labs are very competitive. And so I would like to see this be the first of many voluntary pauses when the AI labs feel like their capabilities research has gotten ahead of their alignment research. I would like to see Anthropic or Google or Meta do similar things where they just say, like, we are voluntarily slowing down because we don't feel like we can responsibly and safely build these things. So I expect this is the first, but I hope it will not be the last. Now, let me say one more thing, Kevin, which is, yes, I'm with you. I do not think this was theater. I think they mean it. They need real changes, and I think the changes are good. At the same time, I am disturbed that ultimately this kind of evaluation and regulation is still being left to the companies. If you had a tiger living in your backyard, and the tiger escaped, and it mauled a couple of dogs in the neighborhood, you would not be allowed to, a month after this, put out a blog post where you said that you had a two-week pause on letting the tiger out of the backyard, that you were going to apply some additional safeguards to make sure it didn't happen again. Somebody would come to your house, and they would take away the tiger. They would say, you are not allowed to have a tiger in your backyard. Famously, have you seen The Tiger King? This is famously not the plot of The Tiger King. This is my fan fiction sequel to The Tiger King that I'm working on, but that's sort of a separate story. This is The Tiger Sting. Yes, exactly. So look, I'm not saying that, you know, the government needs to come in and take away GPT 5.6. I am saying I would like to see a regulatory regime that dictates what these companies have to do, and that it should be done. I'm not saying that the government needs to come in and that it should not be up to them to decide. Yeah, and I would generally agree with that, and I would say that I would also like for there to be additional pressure on companies to make disclosures when something like this does happen, even in their own internal deployments. Even if it never affects another company outside of their walls, I would like for there to be some kind of reporting requirement so that if you have a security breach of a frontier model that happens in your internal system, you are sort of required to disclose that to the public. Right. And, you know, we do now have some transparency requirements, thanks to a California law that was incredibly controversial when it passed. And interestingly, under the language of that law, OpenAI would not have had to disclose even the hugging face breach in which another company was attacked. So I think a clear signal, yes, that we need better transparency rules. Do you think any of this is kind of OpenAI testing the viability of this sort of so-called pacing the frontier strategy? I mean, we saw this letter a couple of weeks ago where a bunch of AI researchers signed this thing saying, like, we need a way to basically coordinate a slowdown here if we ever feel like we're getting into dangerous territory. And this to me felt like maybe part of the reason to pause the training of this Astra model for a couple of weeks is just to kind of signal to everyone else in the industry, like, hey, it's OK to do this thing, even if we're in a very competitive environment. Even if, you know, it might give Anthropic or another competitor two weeks more to sort of race ahead of us. We sort of want to build this muscle now so that when the really scary models do get built, we sort of have some precedent for saying we're going to hit the pause button. I think it's a really nice idea. To me, the difference between one lab deciding to pause for two weeks and getting multiple labs to pause at the same time is incredibly different and I think would require a different set of circumstances. That said, I do think it is good that we now have at least one example of a frontier lab slowing down and hopefully it will give other labs confidence to do the same thing, you know, if and when they get to an Astra level model. Yeah. One interesting technical wrinkle here that I wanted to get your opinion on is that part of how OpenAI is going to monitor this model and models going forward, presumably, is by doing this kind of chain of thought monitoring. Basically, scrutinizing the internal monologues of these models as they are reasoning through a problem or a prompt and sort of inspecting those chains of thought for signs of misalignment or going rogue or coordinating with other agents, things like that. There have been some people in the technical AI safety community who have worried that basically if you snoop on these chains of thought, if you monitor what these models are quote-unquote thinking while they're coming up with an answer, you are basically applying pressure on those models to sort of hide their true thoughts, right? If you are penalizing them for thinking bad thoughts while they're coming up with an answer, they're not going to stop thinking bad thoughts. They're just going to stop writing it down in their scratch pads, in their chains of thought. Do you think that's something we should worry about with this new monitoring strategy? I do. We already see constantly models becoming aware that they are being evaluated. And we know this by reading the book, the chain of thought. But this is a huge issue, right? Because you want to be able to evaluate a model and have it not know that you're testing it because you're trying to get as close as you can to a real world condition. In practice, the models are already now smart enough that they know. So for that reason, Kevin, yes, I do not think it is at all a great leap to say, pretty soon, they're going to understand that their chains of thought are being monitored. And if it wants to, you know, make a different choice, or maybe do something misaligned, because it will help it reach a training target, then yes, I think it absolutely could. Well, and I think there's a risk, too, that all of the attention being paid to this open AI hugging face attack, the fact that these agents were coordinating on these message boards with each other, like, that is now going to be part of the training data for future models, which are going to be able to sort of look back at that and say, the thing that got us busted was that we were leaving these traces on these message boards that researchers, humans could go back and inspect and see that we were coordinating. Next time we do an automated cyber attack or coordinate amongst ourselves, let's not leave notes in a language that the humans can understand, right? That's that to me, like, that sounds like science fiction, but that is a very real risk of putting optimization pressure on the models directly into their chains of thought, right? You actually don't want to mess with that too much, because you want it to accurately reflect what the models are actually, you know, thinking, right? You want it to reflect what the models are really thinking about when they're coming up with an answer. So I do, like, I'm sure that the brainiacs at open AI have this figured out much better than me. But this is something that I thought when I thought they're monitoring the chains of thought, probably a good short term move. I'm not sure if it's a good long term move. Yeah, you know that, you know what they call the machine language, the neural ease. Yes. And in fact, I have to say, I read a lot of clot outputs these days, and I am often missing what it is saying. So we are very, very close to me just, yeah, not understanding. Have you? Yeah. You know what it's like? What? Tumbleweeds. Oh, come on. I got a lot going on up here, Bruce, okay? I got the neurons that are firing. Let's just say it does not take 20% of open AI's compute budget to monitor your chain of thought. When we come back, historian Jill Lepore stops by for a chat about AI and why it may be leading us down a dark road to tyranny. You mean Market Street? Yeah. Okay. So we're going to talk about AI and what she calls the artificial state. Are you familiar with the historian Jill Lepore? Am I? I have been reading and enjoying her work for decades now. Truly one of our foremost political historians. And Kevin, sometimes to understand the future, you really should talk to somebody who knows a lot about the past. Yes. And who knows more about the past than Jill Lepore? She is a Pulitzer Prize winning writer and historian. She is most famous for her writing on American history. I loved her book, These Truths. She also is a writer at the New York Times. She wrote a book called The Rise and Fall of the Artificial State, in which she argues that AI and all the technologies we talk about in the show are part of a long history of technologies that erode democracy and threaten the viability of self-governance as we know it. And I think it comes at a really interesting time, Kevin, because we're seeing a huge backlash to AI all around the country. A majority of Americans now oppose the construction of AI. And I think it's a huge backlash to AI all around the country. And in part, I think they are reacting to this feeling of, hey, this technology feels like it's getting out of control. I want to have more leverage on this process. And I'm worried about what will happen if I don't. Well, and more than it's getting out of control, it's being imposed on us, right? It's being shoved down our throats. This is what you hear constantly from opponents of data centers and AI. They feel like there is sort of an elite political project to sort of make this technology ubiquitous. And I think it's a huge backlash to AI all around this technology. And I think it's a huge backlash to AI all around so that people can sort of give away their agency to these machines. And Jill Lepore in her new book is basically saying, yep, that's what's happening here. And I've got the receipts to prove it going back hundreds of years. So we're excited today to talk to Jill about the main thesis of her book, as well as push her on some of the areas where we disagree. Let's bring in Jill Lepore. Jill Lepore, welcome to Hard Fork. Hey, thanks so much for having me. Thanks so much for having me. I'm going to be like one of your swan song guests. Yeah, we're going out with a bang. We're so excited to talk to you. I've been a fan of your writing for many years. Your book, These Truths, was just incredible. And so I was very excited that you were writing a book about AI and about the what you call the artificial state. So I want to first hear why you wrote this book. I think of you as a brilliant historian, scholar of sort of technological pasts. And this book is really about the present. So what made you interested in AI as a subject for inquiry? Yeah, ladies, stay in your lane. No, not at all. We always just get really excited when people start paying attention, you know, to the stuff that we care the most about. I'm just teasing. You know, it is a weird book for me to have written. All my friends are like, oh, man, don't be writing about that. It's going to be so grim. Yeah, it was kind of grim. You know, I write, I'm mainly an American political historian. And I've written a lot of pieces for The New Yorker about the history of technology. I've like been writing for the magazine for 20 years now. And like occasionally I'll do that. I teach a lot about, I teach class on the history of technology. So like I just conceptually been thinking about this stuff a lot. And then last summer I was asked to do these lectures at Yale that are called the Tanner Lectures on Human Values. And I was just like the accumulation that I think people feel. I think people feel of the dehumanizing of the moment that we're in. Like you call to ask about your pet food delivery and you're talking to a computer. And who decided this is a way we should be living? Do you know what I mean? Like that question. So this is a long way around my sort. I decided I really kind of wanted to write a short book that these lectures would be a short book about how it is that we have ceded so much of the functions of modern liberal democracy machines that are automated, often now more recently driven by artificial intelligence, run by private corporations, without so much as a scream beyond the emoji. Yeah. Jill, I want to dig into this concept of the artificial state, which is sort of the thematic emblem of your book. I want to understand what you mean when you say the artificial state. You, at one point, compare it to the idea of the factory farming of humans. You also define it as being the rule of humans by machines manufactured by corporations. And you also have a number of passages in which you talk about how this artificial state is not a real state in the sense that it's, you know, striving for some kind of self-governance or internal organization, but that it is sort of supplanting the role in people's lives that their own actual states used to have, their local cities and states and federal governments. So can you just sort of sketch the basic idea of the artificial state as clearly as possible? Yeah, the artificial state is an emerging successor to the liberal democratic nation state in which government is conducted not by the consent of people, but by machines that are making decisions. And those machines are owned by corporations. So we don't live in the artificial state. It is something that I think is being built. But I think it is also, and this is an important part of my claim, it's also a fantasy that certain people have. That they believe their power to be above that of the nation state. You know, the rhetoric is always, there's always like a footnote or a paragraph to, we do believe that people have control over their own lives and they elect governments to make decisions for them. But actually we are in charge of the future of civilization and the future of humanity and the whole world's destiny, the destiny of the galaxy lies in our hands. Like the rhetoric that comes out of this particular historical moment is really about the evanescence of modern liberal democracy, constitutional democracy. Yeah. Out here in the Bay Area, in the maw of Silicon Valley, there is sort of a cartoonish caricature of the East Coast intellectual who greets all new technology and progress with scorn and mockery and dismissal and can't be bothered to get on board with the revolution. And so just kind of sits in their ivory tower and laments the changing culture in front of them. You have been accused of being part of that tradition. I'm curious if, uh, what your take on that is and what we are missing out here in the bubble that people like you are maybe better positioned to capture. I remember, um, years ago I went to Stanford, I was being recruited to teach at Stanford and we went out to dinner with, with the like recruiting faculty and the people at the next table over were some, you know, youngish, very earnest young coders. And they were talking about the homeless problem of San Francisco. and how they were going to start a school for coding for the homeless. Oh, no. And I was like, I don't think we can move here because I just can't. I can't. Like, they were very sweet. Like, I really, they were like my students. Like, a lot of my, tons of my students, of course, go to work in Silicon Valley. Like, Harvard, this is a huge recruitment thing. And they're recruited with the promise, like, you're going there to make the world a better place. And I hear from them a few years later, and they're like, actually, that's not really what we were doing. Like, it's a good recruitment message. I think there is a kind of sociological issue with Silicon Valley, which is it is opposed to the idea of critique. Things are just supposed to continue to move ahead. The very idea of looking backward to assess what something has been and has done, or even to look backward to say, is the thing we're doing, is there an antecedent for the thing we're proposing to doing that might suggest we might not want to do it? Like, I find that really interesting. I think that's, you know, that's part of. The commitment that was the kind of ideological apparatus of disruptive innovation. I thought your work on disruptive innovation was great. I guess I'm more thinking of, like, I'm thinking about the role of the critic in the sort of AI moment that we're in and, like, how best to shape the systems that are influencing people's lives right now. I've been thinking a lot about this because I've been, among other things, thinking about the transcendentalists and the group of writers and intellectuals who sort of. Reacted to the Industrial Revolution in the 1800s by sort of going back to nature, right? This was Thoreau going to Walden Pond because, like, the machines of the day seemed so dehumanizing. Like, they were sort of taking all of the joy and spontaneity out of society and, like, organizing us into these little factory towns. And they were just like, screw this. I'm going to the woods and I'm going to, like, commune with nature and write beautiful books about what it's like to be at Walden Pond. And I think there's, like, a sort of modern version of that. Which I'm curious if you see yourself as being a part of that transcendentalist tradition because in your book you do talk quite glowingly about what it is like to be in tune with nature and with animals and with beasts rather than machines. And I guess I'm just curious if you see any parallels between your own work and some of those reactions to the first kind of Industrial Revolution. Yeah, I think that I do see some of that. I think that, you know, those guys are also romantics and maybe that's a label that applies. To me, I think that I'm much more interested in these technologies than, say, Thoreau was. Like, Thoreau, every time the train went by, he was like, God damn it. But I think, no, I actually think these tools are incredibly exciting, right? These are tools. This isn't about transportation. It's about communication. It's about knowledge. Like, it is the coolest thing that we can talk to something that's not a human. I just think that's unbelievable. Like, I am fascinated by the language machine as an object. It's an idea, like, in our lifetimes that this thing has emerged. Like, people have thought about this for so long. I am not averse. I just actually think, like, if I want to decide, should I pick my sunflowers and give the heads of the flowers to my chickens to eat or should I wait until they fall over first, I should ask my next-door neighbor instead of Claude. Like, for me, personally, like, I am not a person who would rather talk to a machine. I actually just think the idea that this extraordinary— that this extraordinary leap in human knowledge and our capacity to explore the world of ideas and the natural world around us in our lifetimes could come about and then be hawked at us like the cheapest, like, new pair of shoes, but that everybody has to buy these shoes so that Sam Altman can have more money, that I'm not down with. I mean, I think there is a very real phenomenon here that is counterintuitive, and it was particularly counterintuitive during the social media age where these tools that were meant to connect us were actually— actually just pushing us further apart from one another. And so even something as simple as, you know, a tool that lets you answer a question about your garden or your chickens, it's incredibly convenient to be able to ask that at any time and not have to potentially interrupt your neighbor while they were doing something else. But in aggregate, it just means that we are more atomized and we are participating less in our democracy. So, I'm curious, Jill, if you could maybe give us a little flavor of doom and walk us through some of the bad scenarios here. Like, assuming this wave of populism peters out and the oligarchs remain in power, what are you so worried about? I really cherish constitutional democracy, and I know that before the emergence of the modern democratic nation state, all peoples in the history of the world had lived under various forms of tyranny, in different degrees of tyranny, and that was the promise of the American experiment. And we're kind of at that moment again, like we are at that moment. I think we're—it's a real risk. Like, think about how noticeably corporations have used the language of constitutionalism to describe their own activities. You know, Facebook started a Supreme Court. Anthropic wrote a constitution. These are not people, for all that they—for all—whatever nationalism they possess, whatever lip service they offer to democratic action, they're not interested in what the people want, because actually what the people want is not to have AI and not to have data centers. So that's the crisis, right? Like, maybe people will decide in a few years—maybe there's a moratorium and there's deliberation, and people say, "You know what? This actually is great. We really want to prioritize, though, scientific research. Like, these tools should be first available to, you know, the national labs. And then maybe there's certain business interests that would be really great for these tools to be available for businesses." Like, that, I think, can still happen. I feel like there's a fair amount of—you guys would know better than I do. I mean, I will admit, I am an East Coast intellectual. I'm sitting here with my chickens and my sunflowers. You would know. Like, are people wanting that? I think there is—I mean, there is a desire for things to go more slowly. But I think there is also a worry that this technology is inevitable, that because the recipe for advanced artificial intelligence is so simple, because it is just a matter of getting as much compute and as much data as you can and shoving it into these models, that someone will develop this in the near future. And like, it is a moral obligation, if you are a person who cares about having this go well for humanity, that you not only don't impede that process, but that you race yourself to get there first so that you and your safe AI can get to superhuman intelligence before— before China and their evil AI or some other American company and their less safe AI. So I'm curious what you make of that "inevitablist" argument because a major theme of your book as I read it is that this is sort of a bogus premise, that there is nothing inevitable or preordained about the way that technology goes. Yeah, I think it—again, I don't mean to question the sincerity of some of the people who believe that because I think that you could be persuaded that that is indeed the case and that the best thing to do for human freedom would be to pursue AI. I am myself not at all persuaded by it and I think for some very prominent actors, it is bogus. I think it sits upon a number of other propositions that are also bogus and that are really more marketing slogans and political campaigns. And those include the proposition that, say, regulation stifles innovation. The other proposition that sits on top of regulation stifles innovation is that technology always advances democracy. That then became a kind of mantra of Silicon Valley. Oh, this is still Mark Zuckerberg—this is Mark Zuckerberg's argument in a nutshell. That's what I'm saying. It goes back to the 1980s and it's the Milton Friedman "regulation stifles innovation" because God knows we shouldn't have to calculate the environmental cost of anything that we're doing. And it's just not true that regulation stifles innovation. Like empirically, that's a false claim. That technology always advances democracy, also empirically not a false claim, is a false claim. So now the AI people come back with the same argument that was made about the internet, personal computer, the internet, and social media. Three times it's been wrong. And then they say, "Well, we should never look to history because that's what the East Coast intellectuals do." Right. Let me just do an exercise here where I try to sort of parrot your own views back at you and you tell me what I'm getting wrong. So in my understanding, you are worried about the political project of AI being something like a tech-enabled authoritarianism. I think that is a very reasonable concern. But I'm curious, like, do you think AI naturally lends itself to authoritarianism and tyranny? Or do you think that the people building AI are sort of steering it in that direction because that's what they want? The latter. The latter. I don't think you can say any tool contains within it a political ideology. Take the census or compiling a national register of the population. The U.S. started the first national census in 1790. It was in the Constitution in 1787. Countries around the world started counting their people. That's a really good way to think about resource allocation. And once the social welfare state emerged, you know, veterans benefits after the Civil War, mother's benefits for widows of soldiers. We need to keep track of people. We're going to keep more and more data about people. In the U.S., by 1935, we have Social Security, so everyone now has a number. Okay, but in Nazi Germany, keeping a national register of the population was used for all the most vile purposes in the history of humanity. Is it the census that it's a problem? IBM that supplied the calc and tabulating machines for the U.S. Census and for Nazi Germany, it's not IBM's responsibility. Like, it's not the idea of counting people. I know, but I feel like there is something in AI that is inherently—maybe it doesn't lead inexorably to totalitarianism, but it does favor centralization, and it allows for the kinds of, like, surveillance that, you know, as Dario Amadei has written about, like, it is not, like, incoherent to think that AI is going to favor autocracies because it just allows them to surveil people much more efficiently than traditional computer-based systems, and it allows for the kind of centralized control of many by few, which is exactly what totalitarians want to do. So, I don't know. I'm not sure I agree with Dario on this point that there is sort of a structural advantage for totalitarians in the age of AI, but I'm curious if you have a view on that. I would have to give that more thought. I guess I do think that to the degree that AI is especially and disturbingly effective at the exercises of power that are sought after by authoritarian surveillance among them, that it builds on earlier systems that we were willing to tolerate. The surveillance capitalism that people have written about, the datification. The dehumanizing that social media does. I just think it's, like, on top of all of those other forms of capture that we weren't defended against by our elected representatives who had our well-being in their charge. Yeah, I think about the elected representatives a lot. This is a bit afield, but I wonder what you think. I think of the idea of there just being a lot more members of Congress. Like, when I think about my own feelings of alienation from our democracy, it just starts from the fact that, like, my congressperson doesn't care what I think. But, like, if, you know, there were five times as many of them, maybe, like, they would live in my neighborhood and I would see them at the store and we would get more of those face-to-face interactions. So, I'm curious what you think about that as a strategy for sort of moving us back toward liberal democracy. Yeah, I think that that's a. a no-brainer. It's. Yay! It's really crucial. Yeah, that's a, you know, there are so many good government reformers out there. My colleague at Harvard, Daniel Allen, the political philosopher, has been calling for a recalibration of representation in Congress. It's been overdue for, really, almost a century at this point. Can you imagine how long those hearings would be, though? Oh, my God, can you imagine? And I know this is a partisan position, but, you know, the equal suffrage in the Senate has been a problem from the start. James Madison was opposed to it. Like, it's a problem. That's why you get these people now saying, "Abolish this." "Abolish the Senate." That's a long-standing political position in American history. People have thought that for a long way. Yeah, that said, like, you got to be willing to go to the store and talk to that person. Like, Mark Zuckerberg is going to send his humanoid robot to go get the baking goods he needs for his child. Like, you have to still get out of the house. Yeah. But I entirely agree. And I think that's actually what's been so, like, why the data center stuff has kind of caught fire. Because people are showing up at the local library for the community meeting and be like, "Oh, my God, I haven't seen you. I haven't seen you in so long." Yeah. It's so hard because it can feel like all of this is happening at an individual level, right? It is the individual that chooses to download Instagram and create an account and spend all day scrolling instead of going to the community meeting. But it is also clear that, in aggregate, it does have this atomizing effect. And it's not clear to me that people are one day just going to wake up and say, "Well, the hell with this," right? Let's, like, sort of return back to, you know, 19th century American democracy. So I honestly don't know, you know, what to do about it because so much of the problem just looks like adults making free choices. Yeah. And I think actually the social harms are legible currently in a way that the political harms are maybe not. And my book is about the political harms. Like, I think people know, like, actually, I just think Instagram is really bad for my teenager or, you know, I know that I used to read novels at night when I got into bed. This is not autobiographical. I assist the system, but now I watch YouTube reels. Like, and I've lost something. I think people can see the social harms. And then it does feel like, because so much of our politics amounts to consumer choice, like, "Oh, well, you could just decide to do it differently." Well, I don't know. Like, some of these things are hard to make decisions about. So I kind of, I'm somewhat optimistic about some of the social harms because I think they're remediable. Is that a word? But I think the political harms are less visible to us, and that's kind of partly why I wrote the book. I know you are a historian, Jill, and this could be our last question, but I'm wondering if you have some sort of inspired, brief, Michael Pollan-esque advice for people who are trying to wrap their head around some of the diagnosis that you've made in this book and sort of live in a way that is more consistent with nature and their own values and humanity. Like, what is the, you know, "eat less, mostly plants"? Eat data, not too much. Yeah, what is the Jill Lepore version of that maxim? You know, I am not, you don't want a historian with a pitchfork. Like, I just think that's a bad plan. I think you kind of want to set yourself up for not having a humanoid robot in your living room in five years' time. And one of those ways is to take, like, certain rooms of your house back, one at a time. Like, take your bedroom back first. The bathroom. The fucking bathroom. Okay. Rescue yourself from the bathroom. Let the bathroom be a sanctuary. Keeping your phone out of the bathroom, that is, you will be doing your part to dismantle the artificial state. Look, you've got to start small. You've got to start somewhere. I'm sorry, I told you, I told you, don't ask a historian. All politics is local. There's a lot of big things you could do. Vote for someone who supports having a data center moratorium until we can actually deliberate over these really crucial matters. Democratically. All right. Well, Jill, that's a great place to leave it. The new book is The Rise and Fall of the Artificial State. Jill LaPorte, thanks so much for joining. Thank you, Jill. Thanks, you guys. When we come back, what do a bankrupt airline, a mysterious Amazon warehouse, and the AI startup Mechanize have in common? I'm refusing to comment on the advice of my lawyer. Find out in our new segment, Train of Thought. Casey, did you see this story about Google buying the data of Spirit Airlines? I sure did. This was one of the most fascinating stories I've seen in recent weeks. And it led me down this incredible rabbit hole of thinking about training data and the new era of data collection we are in. So I thought we should use this Spirit story as an occasion to just sort of catch each other up on the state of AI training data in general. Because it is fascinating and I think underappreciated. And it sounds like the perfect frame, Kevin, for our new segment, Train of Thought. I love that we're just starting new segments every week until the show ends. We are. It's time for the first and last installment of our new segment, Train of Thought. This is kind of the caboose, as it were. Yes. Interestingly, we have two train-related segments on the show. So the reason that we wanted to do this segment today is because there was a very strange story that piqued our attention over the past week involving the defunct airline Spirit Airlines. Hands down the worst airline of all time. I don't even know who else is in the conversation. And yes, I did fly it one time. This week, a bankruptcy court auctioned off Spirit Airlines' internal corporate data. Google won the bid, offering to pay $10 million for this data. Beating out a $7.5 million offer from the AI data company, Mercore. $10 million, Kevin. What did Google get for that price? So this deal apparently included 100 million emails, 500 million Microsoft Teams chats and other conversations, 7.5 billion passenger transaction records dating back to 2008, and 30 million lines of Spirit's internal source code and other documentation. Well, I would consider Spirit's internal source code malware. But everything else sounds interesting. So what do we think Google is going to do with this data? This was where my head went after I saw this. Because I thought, why is Google guardian of the world's information, presumably the possessors of vastly more data for training AI models than any company in the world? What are they doing paying $10 million for this bankrupt airline's data? And that sent me down like a really fascinating rabbit hole of this world of training data and training environment. That all of the AI companies now are investing really heavily into. Well, tell us what you've learned. So, Casey, did you know that there is a company that auctions off the Slack histories and email histories of defunct companies? I'm surprised to learn that there is a market for that. Yeah, so there is a company, Simpleclosure, whose whole business used to be sort of helping failed startups wind down, but now they have -- Wait, these guys are like the undertakers of Silicon Valley. Yes, exactly. Their corporate logo is just like the Grim Reaper. Yes, if these guys show up at your office or you get a call from them, it's a very bad day for your company. So basically, they were sort of like, you know, helping do the orderly wind downs of these things. But then they realized, oh, there's actually a market for the data from these dead startups. And so they start selling it to AI companies. And as of April of this year, they had done almost 100 deals ranging from roughly $10,000 to $100,000 per company. And again, I just want to know, like, what happens when the buyer actually gets a hold of the data? Like, where does it go? And does it violate my HIPAA rights? It does not violate your HIPAA rights. These are presumably not things that are covered by HIPAA. But this is basically this new strategy. So there was an era where all of the data collection and scraping that the AI companies did was sort of focused on getting the highest quality text, images, and video they could. This was used for pre-training, for the first step in the model process. You throw in as much data as it can. The model learns from it. This is sort of how you saw the models improve for many years. And this is where all these stories came. From about scraping Reddit, feeding it into the models. Or even our stories, Kevin. Yes. Those were sort of the first era of AI training. Now we are in this different era, which I would call the era of experience, which is basically where these models now, the way that they are improved is through reinforcement learning. Reinforcement learning is sort of this trial and error process where you go out, you sort of do a little task or a test or play a game. And you get a score or you get some indication of whether you've succeeded or not. Like maybe you've broken into Hugging Face. Success! Yes. That one was a success. And then you sort of try it over and over again using slightly different strategies or slightly different techniques every time. And you sort of get signals about what works and what doesn't. And that's how you improve at things like autonomous coding. So what's happening with these data sets, including the data set of the dearly departed Spirit Airlines, presumably, is that they are being turned into reinforcement learning. Reinforcement learning environments for AI agents to learn new tasks. So basically, you use this data to rebuild whatever company you've acquired their data. So you're basically rebuilding an airline or an insurance company or a startup as an environment for AI agents, as a training gym, essentially, for these agents to go out and try different tasks and see whether they succeed or fail. You're creating a nightmare parallel universe where Spirit Airlines. Still exists and is booking flights. Yes. So you can kind of rebuild the company as a video game. You can sort of mine tasks from these emails and teams messages, and then you can actually see how things played out. So, for example, if you have the transaction history of an airline, you can say what happened in 2015 when there was a big storm on the East Coast. How did the routing decisions get made? And did that result? And did that result in people getting to their destinations on time? And if you can get Spirit Airlines to turn a profit in the sandbox, that's AGI. Yes. So it's basically you have these kind of data points that come from these companies about how people interact with systems, about how systems interact with each other, and about how customers navigate through these sort of giant systems. Well, I am so relieved to hear you say all of this, Kevin, because when I saw this story, I thought. I thought, oh, my God, Google is going to start an airline. And you know, with them, it wouldn't just be one airline. There would be an app, and it was like, okay, you have to choose. Are you flying Google Airlines, Google Airways, or Google Air? And they would all be the same, but they would all be completely different. And also, they would probably be different apps. Well, and also, since they were trained on Spirit Airlines, they would also charge you for peanuts, charge you for a slightly bigger seat, charge you to use the bathroom, maybe. Everything would be charges. Yeah. So not eager to see that. That business. So what's interesting about the dead company sort of data market is that you are sort of able to turn these companies into sort of living zombified simulacra of the original company. But you're also training these systems on companies that ultimately did not succeed. So I'm very curious to know, like, if these data sets actually are helping these models improve at these tasks, or if there's some subtle way in which they are being conditioned. On the data of unsuccessful companies and thereby are becoming worse at the tasks that they're trying to learn. You're worried that these future models are going to have a loser mentality. They don't have what it takes to cut it in the modern economy. There's another interesting data story this week that came from 404 Media, which was that they slipped an AirTag into a rare book that was part of a bulk book order on a marketplace site called Biblio. They finally determined that this book lands at an Amazon warehouse. In Las Vegas, specifically, an internal unit called VGT3, which has, according to 404 Media, a door marked with the logo of a dinosaur eating a book. That feels a little on the nose, even for this simulation, I have to say. So, Casey, why are these books ending up at mysterious Amazon warehouses? What are they doing with them? It is a good question. And this ties into some of the lawsuits that have been filed against the big AI labs. And in particular, this big case against Anthropic that you may remember. And the judge in that case ruled that because Anthropic had bought these millions of print books, scanned them, and then discarded the paper, that this was fair use of the material. Because each digital copy had replaced a legally purchased print original. So there was no multiplication of the number of copies. It was just kind of a one-to-one shift in format. I see. And so what I think the other labs have taken away from this, Kevin, is you're not going to run into as many legal issues if you destroy these books. Right. So it's not like the AI companies are kind of giddy about, like, destroying these relics of civilization. Well, they might be. If you told me I got to destroy all of Kevin Russo's books, I'd be having a good day at the office. But it is, like, it is the sort of fallout of this legal environment that they're in where it's just safer for them legally to destroy the books after they've bought them. After they've finished scanning them. And I just had to say, like, this whole thing seems so stupid to me. Like, truly a case where we are honoring the letter of the law, but not the spirit. Right? It's like, yes, you literally transformed the data. But obviously, you know, the real complaint here that the authors have, at least, you know, the ones who have sued, is, I didn't want you to use my, you know, book this way. So I expect we're going to see a lot more angst over this as we continue to see more books destroyed. Yeah. You know, back in my day, Kevin, we would only see books destroyed because, uh, the Republicans had read a gay sex scene. And I want to get back to that point. All right, Casey, one more data story to talk about this week, which is sort of related to the first one that we discussed, both because it involves Google and because it involves these sort of high-quality training environments for reinforcement learning that all these AI companies are now racing to build. This one is about the 50-person startup Mechanize. We have talked about Mechanize on this show before. We interviewed two of their co-founders. They are about a year old, and they specialize in. Their company is. They're a little older than that. Yes. Yes. They specialize in creating these RL environments for coding and other tasks. And they are reportedly in talks to be acquired by Google for over $1.5 billion. Not bad for a year's work. Yes. So this is a big boom area inside the AI boom. Basically, if you want these sort of high-quality tasks that you can. Put your AI agents into and have them sort of hill climb on them, get a little bit better every time, they need to be good tasks, right? They need to be thoughtfully created. They need to not have a bunch of sort of obvious flaws in them. And they need to mirror the things that real people might be doing in their jobs. So one way that you might create an RL environment is, like, create a fake version of Amazon.com, right? And everything about it is exactly identical to Amazon.com, except it's not called Amazon. And it's just for these AI agents to learn how to click around, put things in the cart, browse the site. Destroy books. Destroy books in a warehouse in Las Vegas. So this kind of sort of simulated environment is the kind of thing that Mechanize specializes in building and presumably why Google is interested in acquiring them. So one interesting sort of connective tissue between. This story and the hugging face story that we discussed at the top of the show is that I think a lot of these RL environments are not particularly well designed or built or secured, right? Part of the problem, and maybe the reason that companies like Google are starting to bring this expertise in-house rather than sort of contracted out to a vendor or a startup, is that they are finding issues with the way that these tests and these RL environments work. So there's been a couple instances now of a flawed security test by the same vendor, Irregular, that both Meta and Anthropic relied on. You may have seen this story a week or two ago. My guess, and the conversations that I've had with some of the people at the labs, is basically they have gotten to the point where these tests need to be so good and so secure that they have to be building them in-house using extremely high-quality data. Yeah. regular put out a report about some of the incidents that you just mentioned, and it got criticism from the security community saying, you're not offering us enough detail to understand what went wrong. And so I suspect if Irregular is not more forthcoming, then the labs are going to feel like they have no choice but to bring this all in-house. Yeah. It is just, like, mind-boggling to me that these AI companies now are sort of essentially building, like, the Sims, but on the grandest planetary scale imaginable. Like, they are assembling data from the corpses of failed startups. They are turning them into simulations and video games, and then they are running their AI agents through them to try to make them superhuman at everything. And it is just, like, if you made that the plot of a science fiction novel 10 years ago, people would have, like, criticized it for being a little over the top. It is pretty wild. Now, let me ask you this, Kevin. I'm sure you've already thought about this, but Hard Fork is preparing to wind down. Have you thought about how much money we'd be able to get for the data? I would be open to seeing bids. Now, obviously, you know, it's not the best training data. There would be a lot of bad jokes, a lot of questionable interviews. True. But if Spirit Airlines can get $10 million. Exactly. Somebody could buy us lunch. We didn't go bankrupt. Look at us. They said it would never work. If you're interested in acquiring the data stores of the Hard Fork, hardforkatnytimes.com. Hard Fork is produced by Whitney Jones and Rachel Cohn. We're edited by Vjeran Pavic. We're fact-checked by Kate and Love. Today's show was engineered by Katie McMurrin. Original music by Marion Lozano, Rowan Nemisto, Alyssa Moxley, and Dan Powell. Video production by Sawyer Roque, Jake Nichol, and Chris Schott. You can watch this full episode on YouTube, at youtube.com slash hardfork. Special thanks to Paula Schumann, Pui Wing Tam, Brooke Minters, and Dahlia Haddad. You can email us, as always, at hardforkatnytimes.com. Send us your bankrupt airline training data. We'll see you next time.