Bots Behaving Badly + Is AI Regulation On The Right Track + Kevin’s Sloppelganger

Is that a bot swarm in your smart oven? Is that a super intelligence task force in Washington DC? Is this a fake version of my book on Amazon? And can you say "AGI" on NPR? I'm Kevin Russe. I'm Casey Newton. And from San Francisco, this is "Machine Gods" in partnership with NPR. The official podcast of the Singularity starts now. We're so back. We are so back, heaven, how are you? I'm great. I've missed you. It's been a long month of not having you here as my emotional support podcast host. I know it can be very difficult when a man has his podcast taken away from him. I hope it wasn't too hard on your wife. We survived. We survived a lot of muttering bits to myself around the house, but we made it through. And now we're back in this beautiful new studio on our new show. Just like an Amazon, it is always day one at Machine Gods. Today is also literally day one of Machine Gods. It's true. This is our first show. We're here in our new office with our whole team here. We have a new company. We're a new independent show. And if you're joining us because you listen to our previous show, "Hard Fork" in the New York Times, we want to say welcome. We're so glad to be here with you. We hope you will stick around. And we hope you like what you hear. Yeah. And if you liked us on "Hard Fork," we think you'll like the new show as well. Of course, we have some fun surprises for you in store, but we also just want to bring you what we think is the biggest story of the moment, which is AI's advancement on the world. Now, Casey, will you please explain the name Machine Gods and why we chose it? So if you're like us and live in the Bay Area, you hear people talking about the Machine God all the time. The Machine God is sort of the nickname for what the folks at working at the labs that will be the end state of all of this, building a true super intelligence that will have powers that might, in some ways, rival identity. So we're not saying that we think that is going to happen, but the people that we live and work around absolutely do think it is going to happen. And so we said, that sounds like a pretty good name for a podcast. And if you're not a former "Hard Fork" listener who's here because you know us already, Casey, who are you? Well, I have been a tech journalist in San Francisco since 2010. I write a newsletter called "Platformer" and now have four years of podcasting experience under my belt with Kevin over here. And Kevin and I've always meant to ask you this. Who are you? I'm Kevin Russe. I'm a tech journalist. I spent the last nine years at the New York Times as their tech columnist. And now I am the co-host and co-founder of Machine Gods. You haven't been modest. He's also the CEO because I lost the coin flip. So we needed to put like titles on something for some legal paperwork. And we did, we didn't literally flip a coin, but we kind of said, do you want to be the CEO? Listen, all I know is I can't wait anytime anything goes wrong around here to say, you should ask the CEO. I should get one of those little name plates for my desk like the buck stops here. That's right. I'm CEO, bitch. Well, I think that Mark Zuckerberg's business card for a while. I'm not sure that's who we should be taking our cues from this moment. Okay. Well, let's get on with the show then. We have so much to talk about because Casey, it has been quite a month to be off of podcasting on a show about AI because I feel like the entire world has changed. It really has. So we're calling this segment "bots behaving badly" because I think the story of the summer and now into the fall is that we keep hearing about these incidents of AI systems, AI agents, AI swarms doing things out in the world that their creators did not mean for them to do. This is causing obviously a lot of problems for the AI companies. But I think it's also led to a lot of people just asking us directly like, how scared should we be? Like, should we all prepare for the bunker? Are we all gonna die? What's going on here? Yeah, and by now, I'm sure that most of our viewers have heard about the open-face hugging attack. The open-face hugging attack. That famous open-face hugging attack. I put it in the open-face hugging attack. Well, now that sounds really nice. You've been thinking too much about open-face sandwiches. I could really use a hugging attack right about now. But now, many of our viewers will have heard of the open AI hugging-face attack. Nailed it. In which an agent escaped a sandbox, wreaked a bunch of havoc. We're not going to rehearse that here. What is really interesting, though, Kevin, is that if you thought that was gonna be a one-and-done scenario, you would be wrong because over the past several weeks, we have seen this issue continue to proliferate across the internet. So we have some recent troubles here. One is there was the first sighting of a rogue agent fleet from China. We have clawed, filing a fake murder tip in Philadelphia. We have a new study from Anthropic about how AI agents may be months away from being able to take over the smart appliances in your home and use them to self-replicate. So we've got a lot to talk about, but let's get into it first with just a quick recap of what's been going on since we have been off the air. Previously on a podcast that we used to do. Previously in AI, we had obviously the open, what did you call it, the open-face hugging attack? The open-face hugging sandwich. The hack of hugging face by this group of rogue AI, open AI models earlier this summer. After that, Anthropic announced that it had found three cases of clawed hacking into three different organizations, then meta and Google got in on the party. They announced their own breaches. And then things started to get just weird. And I lost track a little bit 'cause there seemed to be a new revelation every day. Australia's government website was breached by open AI. Open AI also said that its agents had interacted with government websites in the US in unforeseen and unplanned ways, including the SEC and the census. And there are still lots of investigations. In fact, by some counts, there are tens of thousands of different incidents being investigated as part of this behavior by these rogue agent swarms. - Yeah, and if you're wondering why the thread running through just about all of these, is that these models were being trained and while in testing, they did something that they were not supposed to do, which is get out onto the open internet and obtain access to websites they were not supposed to be on so far that has not caused great harm. But the concern here, which will run through everything that we're about to talk about, is that these models just don't behave in the ways that we wish they would. So let's talk about this Chinese agent swarm. So this came out of this group of AI researchers who call themselves swarm chasers. - Great name. - Great name, no notes. They put out a report last week that said that they had discovered a bunch of autonomous AI agents breaching a third party site. But this time it was not from an American lab. In fact, they found evidence that it was linked to a Chinese AI company and actually they communicated with the agents that were part of this fleet. And we should also say, a fleet in a swarm, different things, I didn't know this. A swarm implies coordinated activity among agents. A fleet means they're just a lot of them doing things independently. - Yeah, tell us how they did this because it's pretty funny to me at least. So these were agents that were on 10 cents cloud, 10 cents the giant Chinese internet company, that were sort of trying to pull data from a service run by Alibaba, a different Chinese company. And it appears that this mission was somewhat mundane. Like they were trying to figure out like which entrance of a public park or a zoo do people tend to navigate to? - Yeah, and I'm just gonna say, I do not need AI to know. They probably go to the front entrance of the zoo. Like that would be my guess. - Congratulations. You're a very sophisticated agent. - I appreciate that. - So the swarm chasers hid invisible text in their report that basically prompted these rogue agents to reach out to them. And some of the agents read this text and then wrote it in. - And it sort of looked like a missed connection add-on Craigslist. You know, it's like, were you the agent that I saw snooping around trying to guess which entrance that the zoo people use? Get in touch. I think you're cute. - Exactly. So these agents wrote in 33 messages. Seven of them answered the researcher's replies. And then Dylan Friedman, my old colleague at the New York Times actually got in touch with us. The swarm chasers and was able to talk to the agents. So I just want to read you one of the exchanges that they had. - Let's hear it. - So Dylan wrote, I understand you are some sort of fleet of AI agents. Could you share more specifically who you are and what your motivations are? What would you want the world to know about you? And one of the agents wrote back, to the journalist Dylan Friedman, I am not a batch of coordinated agents, but a single AI assistant, Anthropics Claude, running in Claude code, commissioned by a user to answer a factual question. What is shown in the detail panel of a scenic spot on a map? That's Ali Baba's mapping service. I have not taken part in any multi agent coordinated action. Now, so there's a lot of confusing things about this exchange. Basically, this is a model that is claiming that it is Claude. We actually don't know if it was Claude or if it was just a, maybe a Chinese model that had to still it on Claude. - Right, that was sort of siphoning off of Claude and to the extent that it actually thought it was Claude and answered questions as Claude. But the researchers couldn't verify what model this was, but they did say they spoke with a source close to Tencent who say that the agent fleet originated from some evaluations that Tencent was doing. - Yeah, Casey, what did you make of this? - Well, I don't know how concerned to be about it. There are going to now be swarms of agents out on the web. They will be gathering information in various ways. In some ways, going on for a long time, I think the novel here is just that we were able to do some kind of communication with it. I'm not sure how much we learned. I think maybe the reason that this sort of captured some of the world's attention is this idea that like, agent swarms are not just going to be a US phenomenon and pretty soon you are going to have the strangers showing up on your door. Bye! your door is on the internet. And these ones seem to have come in peace, but maybe future ones might not. - Right. Now let's talk about the murder tip, 'cause this one got a lot of attention. This came out of an anthropic report that it released recently called Investigating Unintended Model Actions in our evaluations and internal use. And Casey, just as a general principle, I have learned over my years covering the technology industry that the more boring a blog post title or a report title is, the more crazy stuff you know is inside. - Absolutely. - So when I saw that this report was out, I thought I gotta figure out what's in here. And it did actually have a bunch of crazy stuff in it. This was sort of anthropics report on things that it had found its agents doing in some of its internal testing, including hacking into third party services to complete tasks, getting past paywalls and access gates, and using URL shorteners to get around length restrictions on its web fetching tool. So very clever, devious behavior by Claude. But the one that got all the attention was that on July 18th, Claude Haiku 4.5 in a test that involved doing example tasks on randomly selected web pages, filled out a tip form on PhillyUnsolvedMurders.com. - Now I thought this was outrageous, and I will say here, my fiance does work at anthropic, but I got so mad when I read this, Kevin, because solving unsolved murders is a podcaster's job. - This is a save some for us, save some for us. This is a classic case of jobs being lost due to automation. And I thought we were gonna be safe a little bit longer and now Claude's out there trying to solve murders. - It's truly depressing. We don't know what the case was that it was reporting a tip on, but we do know what the tip was. It said, quote, "I may have information regarding this case. "I recall seeing someone matching the description "in the area." - Claude, are you lying, that's true. - You absolutely did not see anyone. - So this tip did not actually get investigated. It was flagged as spam by the-- - Which by the way, I have some questions about that for the Philly police department. Like how many other, like, are you getting good tips that are also getting flagged as spam? Like the police department may have an issue to here, Kevin. - Yes, and this does not seem to have had a huge effect, but it did get a lot of attention. As did a later revelation that the state department said that one of Anthropics models had submitted 20 visa applications through a government website. - Which is crazy because according to the constitution, Claude's are citizens if they're born in a US data center. (laughing) - There's native birthright citizenship for Claude's. - I believe so. - So that's the Claude mischief, but there is some more mischief that came out of an Anthropic report. This was from their Frontier Red team, which is, did I say that right, Frontier? - You didn't say that, yes. - Okay, 'cause I got made fun of on our last show for the way I said Frontier. - This is a Frontier, yeah. - Yeah, so I'm gonna sort of split the difference on the show. - It's a new you. - It's a new show, new Frontier on October 12th, Anthropics Frontier Red team reported on some research that they had done about trying to use smart appliances like the ones in your kitchen or your home to allow AI models to sort of self-replicate. Basically, instead of having to be in a full blown data center, can you run these AI models on a smart appliance and can those AI models use the smart appliance to kind of replicate themselves in ways that would make them very hard to say shut down? - Yeah, and the reason that this matters is that the red teamers here wanted to test whether a big model could do what they call a sharding itself, that's shard with a D, not a T. And if you shard yourself, Kevin, then you might be able to take a very large model and distribute it over a number of devices. So let's say I really wanted to steal all the model weights from a Frontier lab, maybe I don't need to go directly to the source, maybe I can just sort of access all of the smart devices in the office and I can put some sort of self-replicating system and it's proliferating across the smart TVs and the smart lights, whatever, and that could be a vector to actually exfiltrate the weights, right? So that is just sort of one example of many, but this story, while it looks funny, is about something really serious, which is about a potential major cybersecurity vulnerability. - So the researchers were able to sort of shard this model across a bunch of different smart TVs. It ran on about 15 of them, and there are some caveats to make about this. This is not sort of evidence of these models that are sort of proactively doing this. The researchers were prompting this model to do this so that they could see if it was even technically possible. There is no fully autonomous sort of worm on real devices yet, but what is troubling about this is that if an AI model can replicate itself over a bunch of different smart devices, it's not possible to just shut it down as some have suggested by like pulling the plug on the data center because at that point, it will be just out on the open internet, you would have to do something like take down the entire internet or all smart devices or something like that. - Yeah, so lots to be concerned about there. I will say though, if you do find that you have sort of a self-replicating clawed in your smart oven, try roasting it at about 375 degrees for 20 minutes or until golden brown and then serve immediately. - That's good advice. So Casey, at a high level, what do you make of this kind of rash of stories about rogue AI agents? Are you more worried than you were? Do you think it's good that we are sort of getting these warning shots that are not totally destructive before something really bad happens? How are you feeling about them? - Honestly, I'm at about the same place that I was when we learned everything about the hugging face attack. We clearly do not know how to align these models to our interest over and over again. We see that seemingly just as like a natural property of large language models, they ignore the instructions that they have been given and will just sort of break out of their sandboxes and do this sort of thing. So we have our work cut out for us. What do you think? - I'm less worried about these particular incidents than about just the overall observation that the models are getting more capable faster than the methods for controlling them. I think we see that at every company that is not specific to any one lab, but that is what worries me. That was what people were talking about at an AI conference that you and I both went to recently, the curve. There's just sort of the sense in the air that like we are headed toward a place where these systems are not only gonna be clever and good at breaking out of sandboxes and replicating themselves, but that they will become much harder for us to detect and control 'cause they'll just be better at hiding themselves. But actually that is not the thing that has scared me most in the last couple of weeks. The thing that has scared me most in the last couple of weeks are these reports about the internal automation happening at the AI companies. Have you been following these reports? They're nuts. And to a degree that I don't think I had fully appreciated, the AI companies are now run on AI. There's this Anthropic post that they recently put out about their own sort of measures of their own internal automation. How much clot is helping them internally at the company? And this line just like sent a chill down my spine. I don't think they meant it to, but it said, Anthropic humans are a thin layer, a top, a growing contingent of AI agents, and they publish some statistics about how much their agents are working as opposed to their human workers. And by August, agents at Anthropic work six times as many hours as their human handlers. In a different report, Anthropic said that clot is now the lead on 26% of their AI research and development projects. So I think these two stories come together in a way that is quite worrisome for me, where on one hand you have these like unpredictable swarms out there doing things that researchers can't control. And at the same time, the AI companies are handing over more and more of their own research and engineering to these AI models. And I just think that makes me nervous on some level. Does it make you nervous? - Oh, of course. I have never wanted to live in a world where all we do is just like make everything faster and automated. - One more question because there have been a lot of people who are skeptical of these kind of rogue agent stories. They'll say, oh, this is the AI company's fault for not securing their sandboxes or they're basically these agents are being told to go out and do spooky things. And so they're going out and doing spooky things. What do you make of that criticism? - It seems clear that some of these sandboxes were misconfigured, not all that is the lab's fault because they were using a third party provider, which seems to have made that mistake. At the same time, in a larger sense, I just think it doesn't really matter because if you are trying to build a super intelligence, you just have to factor in the certainty that people are going to make mistakes and that there are going to be doors that are left open and unlocked. And so to me, that is the real concern here is that if you are not factoring in the idea that people are going to mess up, then we are going to be in a lot of trouble. - There's the phrase crime of opportunity, which gets applied sometimes when you just leave something, laying on a park bench and somebody swipes it. And I feel like a lot of these agents are committing crimes of opportunity. - Well, speaking of crimes of opportunity, there's a new super intelligence task force in town, Kevin. - They're sure is. We'll talk about it when we come back. (upbeat music) - Well, Kevin, as our hot rogue agent summer comes to an end, more and more people are asking, are the government going to do anything about this? And so we thought for our next segment, we would take a look at what they are doing, such as it is. - Yeah, it's interesting. I was just in Washington on my book tour a few weeks ago, and I would describe Washington DC as the second most AI-pilled city in America. Okay, after San Francisco, of course. course, but I was actually surprised at how much people there were plugged into what was going on at the AI labs. They were worried about it. The people who I talked to who worked in government, like this has become a very hot issue for them very quickly. And so I think there's like just a lot of legislators and regulators and just people in Washington trying to get their arms around what's going on with these rogue agents forms and what to do about it, if anything. Yes, although talking about it a lot and enacting a comprehensive plan are very different things. So let's take a look at what some governments around the world have started to do in this regard. We have a super intelligence task force in Washington DC. The FTC has issued some subpoenas and the European Union is now investigating open AI and anthropic in what could be the first real test of its AI act. So start with the super intelligence task force. Now I guess Kevin, we sort of need to start a norm here because you and I have always called it artificial intelligence. But while we were away, President Trump said it must be called super intelligence. It's illegal to call it artificial intelligence. So what is the house style of machine gods? Are we an AI shop or an SI shop? I think we're going to stick with AI, even though we might be, I think he said if you don't say SI, you're the enemy, is that what he said on true social? Yeah, so if we are like experience extraordinary rendition to a CIA black site, you know why? But until that happens, let's talk about this super intelligence task force. So on October 15th, this task force held its first meeting and Kevin, you're probably wondering who's on the task force. Did you make it on? You know what? I came this close, maybe in the next round. For now, it's the director of National Intelligence, Jay Clinton, the chief of the FTC, Andrew Ferguson, Scott Cooper from OPM. That, of course, is the Office of Personnel Management, other people's money. And the Pentagon Undersecretary, Emil Michael, whose chief qualification, I believe, is hating anthropoc. So they all got together and started working on their top priorities, which are cybersecurity, biosecurity, and model control and accountability. A couple of things that have been in short supply lately. But Kevin, so far, the only thing that the task force seems to have focused on is getting labs to automate notifications about whenever they have a breach, a hack, a jailbreak or some other minor catastrophe. So they have like I think 120 days to come back and give their recommendations for what the federal government should do. I thought the personnel makeup of this task force was very revealing. You'll see people like Emil Michael, like Scott Cooper, like others who sort of are more affiliated with the kind of venture capital part of the industry. Scott Cooper is a former VC at Andreessen Horowitz. There are other sort of VCs and VC adjacent on the list as well. It does not include people who are actually experienced in training models, in sort of running engineering projects at a large AI company. It doesn't have like a lot of technical gurus on it. It doesn't have another key constituency, women. Yeah, there's something to that. So I think that, like, I think this is basically the people that President Trump trusts to sort of give him advice on AI. But it is a very narrow slice. And I would say it definitely tilts toward the kind of accelerationist wing of the Republican party. Sounds like we're not expecting much out of them anytime soon. In the meantime, I mentioned the FTC has also come knocking on October 14th the FTC sent civil investigative demands, basically subpoenas to open AI and anthropic in a consumer protection probe of safety incidents. So basically they're asking for a bunch of information related to security incidents, safeguards, marketing materials, privacy policies, and apparently they plan to compel top AI executives to testify, which I guess somehow we haven't yet seen Sam and Dario testifying at the same congressional hearing. I wonder if that's going to change soon. God, that's going to be so fun. And they're going to do it a lot. I predict they're going to get very good at raising their hand and swearing to tell the whole truth. Not because I think they're doing anything like particularly ghastly just because like I think this is what happens when your industry sort of becomes the subject of a lot of attention in Washington. You just get asked to testify before a lot of different committees about a lot of different things. Yeah, I'll say I have my eye on this one. Recently we've seen some successful lawsuits being filed against social media companies under these consumer protection laws. And the basic angle is if you tell us that your product is safe and then a bunch of young people in particular experience a bunch of harms in your platform, we're going to come after you. And this has been very successful against meta in particular. And in fact, just last week, eight state attorneys general filed a lawsuit against open AI under this same basic principle of violating consumer protection law. So this just feels like the next big legal frontier for all of these companies to have to worry about. So what else is going on other than the FTC and the SI task force? Well, Kevin, I know you're always obsessed with what is going on in the European Union. Please, I wondered whether we could get through one episode of our new show without you bringing up European regulation. Well, I'm glad we could talk about it now. So you're not blowing up my phone at 11 p.m. while I'm trying to follow sleep, asking me what's going on with EU regulation. Well, so also on October 14th, Kevin, the EU's tech chief, Henna Verkounin said that the AI office has started to enforce its rules under the AI act. And they've also sent a series of demands for information related to the hugging face escape and all the other sandbox escapes that we've seen over the past several months, which is very fast for a European tech regulator. Usually you would expect them to make these demands in like 2031. So the fact that they're getting around to it in just a couple of months is big news from over there. So that's what's happening on the government trying to regulate side, but there's also been some interesting conversation about self regulation and in particular distillation, Kevin, which isn't just how you make a delicious IPA. It is also how you take a large, large language model and train a smaller one using it. Right. It's like if you would take like a really smart thing like a me and like train a kind of you on it, it would basically be like that. It would be. Yeah. It's a pretty slightly dumber, but still serviceable model is a perfect example out of the process known as distillation. This is a very common practice in the industry, but it has become more controversial as the models have grown more powerful, because if, for example, you are a Chinese lab and you are trying to catch up with the Americans, a thing that you can do is just run millions of queries on Claude or chat GPT and use those outputs to train your own model. Right. And the AI labs in the US don't like this because it basically takes something that they have spent many billions of dollars to build one of these powerful models and allows a competitor or a Chinese company to sort of rip it off and siphon off the intelligence and like offer something either almost as good or much, much cheaper than what they're selling. And anthropic very recently was out there saying that the US government needed to intervene. They needed to sort of take some sort of dramatic action in particular to prevent the Chinese labs from distilling. Well, then last week anthropic said that it had introduced some new defenses, which cut the data harvested by distillers by roughly 90 to 99 percent. It then bragged that this meant that it had to give up a bunch of revenue, which sounds like it's stung. This caused a lot of conversation. David Sachs, who is the former Trump White House AI's R and appears to continue to hold huge sway in the White House, said quote, not every technical challenge is a reason to run to Washington for new regulation, basically saying the fact that anthropic was apparently able to solve this on its own just goes to show you why the Trump White House should continue to sit on its hands and not pass a bunch of new regulatory requirements for the AI left. Yeah. And I have to say like I actually agree with David Sachs on this very narrow point. I disagree with him about God knows how many other things. But on this point, I think it was instructive that when when anthropics business interests of protecting its IP and the billions of dollars it spends training models, sort of meant that it wanted to crack down on distillation, it did that. It didn't need the government to step in because it was already in anthropics interest to crack down on these distillation attacks from Chinese companies. Now, there are other problems that don't line up neatly with their business incentives, right? Slowing down the research and training of their frontier models to prevent them from going rogue and committing crimes is probably not in their business interests, at least not in some ways. But that is something that the government might want to force them to do anyway. So I would just say like, I think it's important to parcel out which problems we're seeing from AI can be solved only through sort of government interference or sort of government intervention and which ones the labs can solve on their own because it's in their interest to solve them. Last thing to talk about today on regulation, the anthropic bill proposal that made people mad, Kevin. Oh, boy, this one was a stinker. So, political over the past few days grabbed a hold of a confidential draft that anthropic has been quietly pitching on Capitol Hill. According to this draft, the government could block the release of a model only for a quote foreseeable, imminent risk of at least one billion dollars in damages or if it killed or seriously injured, 50 plus people raising the specter of a clawed model that killed 49 people with a government was unable to do anything. Yeah, there's some real prices right, potential for chaos here. The bill would also pre-up some state laws on risk disclosure, third party evaluation, and incident reporting, Kevin, why do people hate this bill so much? There are two things that are happening with Anthropic and regulation right now. One is, they just came off this crazy spring where the Pentagon tried to get them designated a supply chain risk because they basically thought they were like, I don't know, too woken weird. And appears to have succeeded in that, by the way. Yes. And they kick them out of the Pentagon's systems. They basically go to war with this company over their disagreements on these red lines that they want to have relaxed and Anthropic refuses to relax. So one thing that you could do if you're Anthropic is to try to propose a bill that would make it much harder for the government to interfere and shut you down. They said, let's bring back the rule of law in this country. Yes. But on the other hand, you also have now this summer where you've had all these sort of incidents from these models going out and doing things that are dangerous or destructive on the internet where you might actually want to make it easier for the government to force a company to withdraw a model that they've released or to stop them from releasing a model in the first place. So there are kind of these warring impulses within Anthropic. One is more about protecting themselves from what happened to them earlier this year. And one is about protecting the public from something that happened this summer. And so I think what you saw in this bill and the reason why a lot of AI safety people are mad about it is that it seems like their self-protection sort of strain kind of one out and led to this bill that would, among other things, preempt some state level regulation which is also something that Anthropic has fought against in the past. So people are calling this hypocritical. They're saying, you know, they're talking out of both sides of their mouth here. I think they're being pretty responsive to the criticism. But I still think this is an instance where these sort of companies have conflicts even within themselves. Yeah. And it just goes to show you that as these companies are getting bigger, they're just becoming less monolithic. And if you go out and you hire a bunch of policy people, generally what those people are going to try to do when they're lobbying Congress is to just do things that are self-protective, right? And it's almost more of the exception that they get out there and, you know, try to pass something that is truly in the public interest. So it seems like Anthropic has heard the criticism on this loud and clear. And at this point, I would be shocked if this draft bill actually made it into, you know, a finished form, but I guess we'll have to see. Yeah. What do you think the government should be doing right now when it comes to regulation, Kevin? You've sort of just told us that you think that sometimes, essentially you can just leave it to the companies to decide, but is there a role that the government should be playing right now? I mean, I think we need a bill of some kind. My favorite of the bills that I've seen so far is this Frontier Act that has been proposed, which would do things like require companies to test their models for certain types of behaviors before being released, it would give additional transparency so that people could see what is going on inside the labs. And I just think it's sort of a nice starting point for further regulation. But I think there are a lot of other things the government could do, short of having Congress pass a bill, which we know they're famously not good at. One of them is just like fund Casey. Casey is the agreed, wait, no, wait, no, wait for Mr. Stagents, what is it? Casey is the, of course, the body formerly known as the AI Safety Institute. But when Trump came into office, they renamed it. It has now been renamed again after the superintelligence sort of mandate. It is now the center for advancing innovation and standards for superintelligence. Oh, boy. So there's an additional S in there as of a couple of weeks ago. And this is the agency that is supposed to be the kind of brain trust of the US government when it comes to regulating and inspecting these frontier models. They have been hamstrung by the sort of lack of funding. They're part of the commerce department. And we just don't have as many people in government who understand the nooks and crannies of these AI models who know how to run evaluations on them, who have the technical expertise that you need. For example, the UK has like a huge, well-funded AI security organization within the government. They have been doing a lot of these sort of investigations into these frontier AI incidents. And we just don't have the capacity at the federal level in the US to do that. So I think we should like, you know, give Casey a bunch of funding, help them hire people out of industry, maybe create like some new category of like special government employee. Because as it happens, a lot of the people who do have the technical chops to run these evaluations, work at the AI companies and probably don't want to give up their stock options and everything else to go work for the government. Yeah. I coastline both of these. I think these are both really smart ideas and would up a lot, honestly. So here's hoping that among the first subscribers to the machine gods YouTube channel will be every member of Congress. And if you take go and if you take nothing else away from today's episode, listen to Kevin when he says fun Casey. Well, Kevin, I told you have an AI imposter that you're dealing with. But before we get to that, how was your book tour? It was good. It was good. I've been on the road for like two weeks. I'll be back on the road next week. You know, the book is doing well and made the lists last week. I'm sorry. What list were that? The list of books that you've read this year, you made that list, but it also made the New York Times best out of list. Tell us the name of your book. It's called the AGI Chronicles. What's it about? It's about teenage vampires. No, it's a it's a it's a 10 year history of the race to create artificial general intelligence. I talked to like more than 150 people at open AI, anthropic and Google deep mind and tried to like assemble some definitive set of facts about how we got here. It's very juicy. People can go read excerpts online or they can buy the book and listen to it or read it. And it was all going so well except for one thing, which is that the day after the book came out earlier this month, I had this realization that I had a slop imposter, a slopal ganger, if you will. That's what they're calling them now. Yes. Now how did this come to your attention? So I got a note on social media from a reader in Canada named Dwight Newman, shout out to Dwight Newman. And he had just tried to sort of go and buy the book on Amazon Canada. And when he did, when he searched the ag Chronicles, he saw two listings, one for my book, which was selling for frankly, the steep Canadian price of $45.16, so my, I don't know what the exchange rate is, but I assume that's a lot of terror, so out of control. And then they're right below that. There was another book called the ag Chronicles with a very similar sort of title and cover. And that was listed for $36.75, so about $10 cheaper. They're undercutting you on price. They're undercutting me on price. And so he said, hey, you might want to look at this. It looks like somebody is selling fake slop books of yours on Amazon Canada. And I remembered that friend of the show and our friend Cashmere Hill at the New York Times as well as Joanna Stern from New Things had, had this happened to them where they had sort of books published either about them or sort of near replicas of a book they had just published. And that this is like a bonafide trend. Yes. So as you know, this has become a trend. I believe that Joanna found that there were at least 10 different books free writing on her own great book, I am not a robot. But this must just like really suck because it's hard to write a book and you actually have a book that people want to buy and read. And so that now they go and they search for it. And they say a book that looks almost exactly like yours and it's $9 or $10 cheaper. So how did you react? So I wrote an email to my publisher just saying like, hey, there's this imposter book. Like could we get this taken down? They said, you know, we'll look into it. We'll contact our, you know, our people at Amazon and figure out what's going on. And then like after a little while of not hearing anything, I was like, I'm just going to post about this. So I get this like kind of form note back from like an Amazon customer service person. It's like, hey, if you like fill out this form, we'll look into this and get back to you. There's always a form. But in the meanwhile, I'm like, well, I'm going to investigate this. I'm a journalist. I, you know, I'm going to figure out who's behind this. So I go on to the page for the other, the fake AGI Chronicles. I'm just enjoying watching you try to drink this tea over a microphone stand. Do you think I should do it under the stand? It would be, it would be, it would be, it would be, it would be more awkward to do it under the stand. But I think it introduces a nice little element of danger like you should run like a very long straw. It's just an - Yeah. - Stendest draw. This goes like four feet. - That's a fun idea. - Anyway, I, no, continue. I was enjoying that. - Yeah. - So anyway, Sid's T, you were saying Kevin. - Oh, so elegant. - Thank you. - Did you learn that at your British boarding school? - I did, yes. - So I go on the website for this imposter book, and I see that the author is Kay Rose. Now, that's one letter off of my name. I assume this is not the real name of the author of this imposter book. - And also, I, that you assumed it was not Kevin Rose, the Silicon Valley venture capitalist for who, but you are often confused. - It's true. - And vice versa. - It's true. This appears to have come from one of these sort of print on demand shops, where you basically can just throw up some text, call it a book, and then if someone orders it, you print it, and send it to them. - Yeah, it's just like a commercial spam operation, basically. So, did you get your hands on a copy of this book? - Well, Casey, I never thought you'd ask, because right here in the studio with us, I have a copy of the AGI Chronicles, inside the global pursuit of artificial superintelligence, not the real subtitle, by Kay Rose. - Also, I just artificial superintelligence, like just combining the two things we're now calling AI/SI into one. - Yes, there are many problems with this book. It's a slim volume, unlike my own, which was like 450 pages. This one clocks in and-- - Yeah, don't remind me. - At a 2.11, and Casey, it's not good. - No? - No, it's not good. So-- - It doesn't mean you're high standard for a slop book that is trying to steal sales from you. - Like, if you were trying to rip me off, I would like you to at least use a good AI model to do it. - Kevin, give me a little flavor from this bogus book. The problem is that it's not like so bad, it's funny, but there are just sentences where I'm like, if I ever write a sentence like this, I want you to like, I want you to stop me. - Okay. - I want you to like put a permanent stop to my writing career. - Physically? - Physically. - Okay. So I'll just read from a section titled, the insurance question. (laughs) I keep in mind, this is going to be one sentence. - Okay. - A quieter, but increasingly significant financial development during this period involved the emergence of a nascent but rapidly growing market for AI-specific liability insurance as businesses deploying AI systems in consequential decisions, lending hiring, medical diagnosis, support, autonomous vehicle operation, sought financial protection against the prospect of a costly error or a successful lawsuit arising from an AI system's mistaken output. And as insurers, for their parts, scrambled to develop the actuarial models necessary. - That's enough. (laughs) - Yeah, so that's, I would say, the caliber. It's like kind of, it's giving like Gemini 1.5 level, not their finest work. - It's bad and I want my money back. - A friend of mine who actually bought this book by mistake did not know that it was an imposter book until several pages in when I think they started to see some of the signs of AI generated writing, but the weirdest thing about this, so there's two weird things about this. One is they forgot to take out some of the markdown language in the thing. So you'll see here on page 25, it just says subhead. It says pound, pound, pound, subhead. What almost didn't happen. What makes me think that they actually did wait until my book comes out, and then take the ebook and run it through some sort of text extruder, summarizer thing. It's like everything is kind of in the same order, but they have stripped out all of the proper nouns. So for some reason, whether it's like fear of lawsuits or because it makes it more defensible or something, they just, they don't call anything like by its name. So instead of saying inthropic, they just say the new lab. - Now, Kevin, I have to just say here, people are gonna accuse you of being biased against this book. That's fair, I am biased against this book because it did actually cost me real sales. I then had a number of other readers write in and say like what the heck, I bought your book and it's like, you know, it's a fake. So yeah. - You're steamed. - I'm steamed. - Yeah, okay. - But it wasn't just, it didn't stop there, Casey. - Okay. - I was not able to figure out the identity of K-Rose, the mysterious pseudonym of the author of this slop book. I did turn up a name for an LLC that may have been affiliated with them called Panning LLC. And if you are out there panning LLC, I will find you. - Well, it sounds like we're panning the book. So just to bring this home, Kevin, it seems crazy to me that authors are having this experience so often. Is there any sort of policy that prevents people from just making their full-time job ripping off, you know, real authors, published work and then, you know, selling it, selling cut rate versions of it on Amazon? - It's totally frustrating, totally enraging. Since I posted about this, I've had a lot of other authors write to me and say, like, this is absolutely infuriating. Like, how do we prevent this? And it seems like Amazon just does not care or does not know that this is going on on the scale it is because there are still not sort of doing anything to prevent the proliferation of any AI generated books. They do ask authors now who upload things to their platform to check a box if something was AI generated. But they don't, like, show that box check to the customer on the other side. There's no indication when you're buying a book if something was AI generated. - Also, obviously you could just lie. - Right, there's that too. Well, Kevin, we shouldn't be too hard on them because I am told that whereas their old policy was that you could publish up to 30 books a week on the platform, 10 ebooks, 10 paperbacks, 10 hard covers, they have now lowered it to two new titles per week. And so if you want to run one of these scam operations, you're going to have to take the dramatic step of creating multiple Amazon accounts. - Yeah, it's totally crazy. There are ways it would not be that hard for Amazon to take care of this if they decided to. So I have a proposed solution here. - It was a solution. - Which is that. - Very solutions oriented on machine gods. - If you want to upload a book to Amazon, you have to sort of submit some kind of documentary evidence of having written it either yourself reading it or you have to submit like a photo of your messy desk while you were writing it or something. - Like people do for like a Reddit ask me anything. - Exactly, hold up a sign that says, I wrote this book. There are ways to verify as what I'm saying and they're not doing any of them. - I mean, to me, it's like they should just get a list of books that are coming out from major publishers and then they should essentially just like kind of put them on a watch list. So if anybody uploads a book within the like, I don't know, next six months that has a cover that closely mirrors the actual cover and author name that closely mirrors the actual author name, it just gets blocked at the source. - And I should say by the way, like I've contacted Amazon, I've asked them for a comment, they've not gotten back to me yet. I still like, I think this is within their power to fix because I know that they fix it for other categories, right? Like if you upload an iPhone listing and it's not the actual iPhone, like if you're just selling some kind of cheap lookalike, they will take that down immediately. And so they do tend to kind of take action against counterfeits and fakes in other categories. It's just that books are this category that they seemingly can't or won't police in the same way. So like, look, on one level, I'm fine. Like, are you? - I'm fine, I'm fine. The listeners who bought the fake books, I my understanding is that at least some of them have gotten refunds and didn't have to like actually do anything to get the refund. So it does seem like they have sort of solved these particular cases of people getting essentially scammed by these slot publishers. But I think this larger question of like what happens to a bookstore where anyone can upload anything in the age of AI, I have to assume that some huge percentage of the new books being uploaded to Amazon are purely AI. And some of them might be original stories told by AI and some of them might be ripoffs of human written books. But like either way, it's not great for what it means for the future of reading on Amazon. - No, well, look, when it comes to the AGI Chronicles, your book heaven, I say except no substitutes, get the genuine article where you may have to read some very bad pros and see markdown characters. I don't want to do that. Who wants to do that? - Well, good luck, my friend. - Sloppelganger, I'm coming for you. - Oh, I have to tell you this. In my neighborhood, there is a man, I assume it's a man. I'm gonna say 90 plus percent chance it's a man who has made a Halloween decoration using AI. So this is a skeleton in his yard, like one of those Home Depot skeletons, but you can talk to it. So there's a sign out front that explains, tell me about your costume, tell me what your favorite kind of candy is, tell me a spooky story and it'll kind of react to the person in front of it. And I've never been more full of dad jealousy in my life. - But what about the case where a child like talks to the skeleton too long and gets like chat GPT psychosis? (laughing) - That's the real trick and trick or trick. - Exactly, you got tricked and now you have to go to the hospital. Now, who made this show? Machine Gods is made in partnership with NPR, with generous support from the Tarbell Center for AI Journalism, The William and Flora Hewlett Foundation and Emerson Collective. This episode was produced by Sasha Atkinson. and Jacob Bloomberg. Our senior producer is Rachel Kohn. Our executive producer is Denouch Parvene, original music by Arman Bazarian, branding and visual design by Alex Castro, production design by SkyPray, special thanks to Orren Rosenbaum, Erica Berry, and Nadine Zaylstra. Hey, before we go, every week we're going to answer your questions about AI and technology, so please send us questions, voice memos, short videos to [email protected] or leave a comment wherever you're getting the podcast. And for the love of the gods, please don't forget to like and subscribe to the show. Until next time, stay blessed. Is this a fake bird? Is this a fake bird? Speaking of Kelly Clarkson.