Speaker 1You've been asked to run the company's social channels, but company content plays by its own set of rules. It needs a logo in the first three seconds, but it can't feel like an ad. It needs subtitles, except when it doesn't. It needs to run at two times speed and deliver a complex message. Simply, of course, within 9 by 16. Thankfully, your company has Canva, so you can easily edit your video and every version of it. What will you Canva today?
Speaker 2Hello and welcome to Decoder. I'm Nilay Patel, editor-in-chief of The Verge, and Decoder is my show about big ideas and other problems. Today I'm talking with Alex Stamos, chief security officer at Cognition and a longtime cybersecurity expert, about what a mess security on the internet is right now. This episode is part one of a special series looking at AI in real life. I've always enjoyed talking to Alex because he's so direct, and you're going to hear him say that the multi-billion dollar cybersecurity industry as we've known it is essentially over due to the unrelenting AI-boosted waves of hackers. Hacks, breaches, and attacks, and all of those are only the beginning. Both script-kitty hackers and also nation-states like Russia and North Korea are breaking into systems all over, thanks to AI helping them both detect and exploit vulnerabilities all at the same time. It's got a real frontier feel to it. You're going to hear Alex say it feels like the 90s all over again, when the whole idea of an always-online world was brand new. That much disruption means there are a lot of places where bad guys can get in while everyone else is left scrambling. Alex isn't a doomer in the long run, but he does think we're in for a, quote, dark age, trying to figure out how to adapt to this new era. There's a lot going on in this conversation, leaving it all the way to how AI-powered misinformation and slop might be managed on our social networks. Before we get going, do me a favor and subscribe to Decoder on YouTube, where we put out new episodes every Monday and Thursday. Okay, Alex Demos and how AI is upending cybersecurity. Here we go. Alex Demos, you're the chief security officer at Cognition. You're also a lecturer at Stanford University's Cyber Policy Center. Welcome back to Decoder.
Speaker 3Hey, thanks for having me, man.
Speaker 2I'm excited to talk to you. It's been a long time since you and I chatted in the, what you might call, the content moderation era of internet policy discourse. I feel like you and I talked a lot about how all that was working, about election security, all those sorts of things. We're in a new era of big problems on the internet, particularly around cyber defense, cybersecurity. You've been thinking about this stuff a lot. You referred to the past few weeks as Security Awareness Month. Tell me what's going on. Just give me the lay of the land right now.
Speaker 3October is, you know, traditionally Security Awareness Month, right? Like lots of companies run events, and, you know, both internally, and then tell people, you know, change your password, use multiple passwords, use a password manager. And what has happened over the last month is we've had, you know, a couple of these big bugs, PHP, Node, and such. What's been less discussed is a huge number of companies have been breached. You have a series of AI-related companies, the NeoClouds, that have been breached in succession. You have the Binary Clouds, which have been breached over the last few months. You have the Binance hack, which is the largest heist of anything in human history. You have the Cursor incident, where, you know, one of the things that's going on here is these hacks are being chained together. So it looks like, from what we know right now, one of the companies that was broken into was Fireworks, which is a prominent provider of AI services to a number of companies. From the Fireworks hack, API keys were stolen that allowed people to break into Cursor, Once they got into Cursor, they then pushed a bad plugin, which went to 5,000 more companies, and that caused 5,000 potential breaches. So it's just this kind of permanent, continuous breach that is going on throughout the tech industry. And one of my friends said, a CISO friend of mine said to a bunch of us, is anybody else tired? And everybody said yes, because effectively, we have been at DEF CON 2 for all of October, and now we're at DEF CON 2. And we're now going into November.
Speaker 2So I'm going to ask you some really dumb questions here. Who is behind all these attacks? Is it Russia? Is it North Korea? Do we know?
Speaker 3It's a variety of different groups. It is not just because adversaries have decided, but what has happened is, from my perspective, we are at the start of the takeoff of adversary capability enabled by AI. That the open weight, models have now gotten to the point at which you can automate most of the kill chain. So this kill chain term is something we stole from the military, right? In the military, they mean to actually kill something. They're the steps you do to blow something up. In cyber, the kill chain is the things you do to have an effect on the target. So to steal information, to turn something off, right? To steal money, to steal Bitcoin. Yeah. You do research upfront, you look at a piece of software, you find a vulnerability, you write exploits, and you do that all way ahead of time. And then you have a human being go and take that stuff you built and deploy it against either one target, because maybe you're a government and you're going against Lockheed Martin, or you're going against another government, or you're a ransomware actor and you take all that work you did, and you deploy it against hundreds or thousands of targets really fast. Well, now with AI, what's happening is this kill chain has inverted, where you can run an AI system that just goes hog wild, and it looks at the targets, and it finds the bugs in real time, and it writes the exploits in real time, and it breaks into these companies. So that's one of the things that's happening is these systems are going, and they're so good at finding bugs that they are finding the bugs in real time and writing the exploits in real time. And so the system that everybody's talking about is not the only one. This isn't ready to talk about is a thing called Artex, which is a Chinese open source tool that anybody can go look at. There's an English translation on GitHub that you can check out. And if you power it on the back end by something like an obliterated GLM 5.3, it will let you break into a shocking number of organizations. This started with three South Korean banks. This week, about 200 more companies have been hit. I've been helping a couple of companies. Cognition has been kind of, you know, we're not an instant response company, but we do go in and like help people fix their bugs afterwards by using our tools. It is shocking the companies that are getting hit by something you can just download. It has been a long time since you've been able to go and just download an open source tool and break into multi-billion dollar companies. It's like the 90s, right?
Speaker 2Yeah, that's something like Windows in the late 90s, early 2000s, when it was a security disaster.
Speaker 3I started saying this actually in the beginning of this year, which is like, it's an incredibly exciting and terrifying time. Because the last time it felt like this was when I was a much younger man and I would go to the very early DEF CONs. You know, DEF CON these days is like 40, 50, 60,000 people. Back in then, it was a couple hundred people and it was one track and you'd go in and somebody would give a talk and you'd leave the room and you'd be like, every company in the world is vulnerable to this attack. The difference was, is the entire world didn't run on the internet, right? Yeah. So it's like, you could go do bad things, but you couldn't ruin the world, right? And you certainly couldn't automate it in the way you can now. And it's back to those days where just like tools you can just go get and build yourself can go do crazy things. It's back to the Wild West days, except we actually, back in those days, there's like a couple hundred people who had those skills and we were all in one room in Las Vegas. And most of them were actually good guys. They didn't want to be crazy. And now you have a much larger ecosystem of people, many of whom live in places where the law enforcement can't touch and who have a big economic incentive and have access to these tools. And the other big thing is they no longer have to use frontier models that have protections, that have logs that are available to law enforcement. They can, I have a M5, my M5 Mac Ultra studio finally showed up and I've been running like a quantized, obliterated GLM 5.3 and oh my God, like what you can do is incredible. -
Speaker 2You mentioned the economics of frontier models, just the economy of cyber is one of those things that has gotten sophisticated over the years, right? So you're in these early days, a bunch of people in a room. One of the things that our systems got safer is you could pay the number of people at DEF CON a whole bunch of money to protect all the systems. And it was economically smart to pay them to be defenders instead of attackers. And then I'm just rewinding my own brain through this history. There was like a long conversation in our industry about security through obscurity by hiding all the bugs and hiding all the code versus open source and being really transparent. There was an etiquette that developed and about how to reveal the bugs and when to give people a notification. There was transparency reports and bug bans. Like we built, you could call it an etiquette, you could call it an economy. We built a culture around cyber security and cyber defense and how the attacks of attackers and defenders would operate. What you are describing changes it all for a whole number of reasons, but the one that sticks out to me is the economics of it, right? I can go buy a Mac studio, I can get an open weight model and I can just start. And that does not place me inside of the culture. There's not going to be some old man. There's not going to be an Obi Wan Kenobi being like, "Here's how you act like a cyber security Jedi." Like, I'm just going to start doing stuff. Tell me how that's changing right now.
Speaker 3Effectively. last era of Security, from my perspective, started with the Microsoft, the Bill Gates trustworthy computing memo in the early 2000s. And that is when we started designing how we do software security. And these big companies, Microsoft and Oracle and then later Amazon and Salesforce, and these folks started hiring ex-hackers. And I was part of that generation of the ex-hackers who were at DEF CON and Black Hat, got hired by the companies to go break their stuff. And we started coming up with Responsible Disclosure, Bug Bounties, consulting companies, so the Foundstone and AtStake and then Mandiant and CrowdStrike. These companies grow up and we get this huge industry that is now RSA and Black Hat. And this multi-billion dollar security industry grew up in the 2000s. That era is over, like you said, because it was based upon a couple of assumptions. One was that there was a hierarchy of skill sets of people who could do this stuff, that you had the script kitties who could download tools to do things. And they're then the next level up of the people who could write those tools and that that was a more verified skill set. And then there was the kind of vulnerability researchers, and then there's kind of the gods and that god tier was in the hundreds of people. And that is the people who do the really, really good vulnerability research, who could write the really, really good shellcode, who could reverse engineer patches from a Microsoft or somebody like that. Also, the assumption that there's a limited number of people who could be operational in an attack. And so if you wanted to build an attack team. That you have a small R&D component, and then you have a much larger operational component. And so if you are a professional attack team at US Cyber Command, at Russia SVR, at Chinese MSS, that you build the structure of your R&D team of elite people, and then you're less elite operational people. And then that was copied then by the Russian ransomware actors, and that you have to have a conspiracy of people to kind of do this stuff. A lot of this stuff also fell apart with the invention of Bitcoin, because one of the assumptions was that you had to have people who could launder money. And then Bitcoin meant that you had to have a conspiracy of people to kind of do this stuff. And then Bitcoin meant that nerds could launder money. So that started to change the economics of the entire thing. All this stuff has fallen apart because of AI. Because the skill set issue is that now script kiddies have a bunch of the skill set of the Dave Itells and the other cyber gods who could do these things. That with this Mac that's sitting on my desktop, I can reverse engineer a Microsoft patch and walk through and then write stable exploit code. It's not quite there yet to be automatic. You still have to have. You still have to have a bunch of skills, but we're really, really close. And then what's really concerning and what you see with the our tech stuff is you can now kind of let loose on the operational side. We were never secure. It was all like a fantasy, right? These companies, these vulnerabilities have always existed. These these companies have never been safe. And that is what we've seen over the last decade is these mostly Russian ransomware actors nailing kind of the midsize companies that most people have never heard of, but are actually really important. Like some midsize industrial company that makes some widget that we all depend upon. I once did work for like the largest distributor of of medical and dental supplies in the country, and they got really hit badly by a Russian ransomware actor. And then like dental offices across the country were, you know, canceling appointments. Didn't even make the news. Right. But because they couldn't get gloves, because they couldn't get supplies. And it's because, you know, a company like that's got like one security guy. It's a 10 person, 12 person IT team, and they're going up against a 20 person Russian ransomware team. They're toast. Right. But it's just because there's only a handful of Russian teams that could do that. Now, two or three Russian kids can have a whole bank of Macs or they have the money. They can buy Nvidia H100s and they can run a ton of agents who do this all day. And that is what is changing. So I'm sorry you asked who is doing this for Binance. It's the North. North Koreans. And they have an entire team that all they do is steal Bitcoin. Other people have gone out of it because it's like really hard to compete against a foreign intelligence agency that recruits their smartest kids. And all they do is steal Bitcoin. They're extremely good at it. And for the Binance Act, we could talk about it. There's some probably some AI component, but mostly that was actually human intelligence and very, very smart social engineering. And the North Koreans are I mean, you have to give it to them. They're very, very good at that. Yeah. So the other stuff it's it's it looks like for the the people who are breaking into the AI companies is what's called the Chinese cottage access industry, which are small Chinese contractors who break into companies and then sell the information to much larger Chinese companies.
Speaker 2That diffusion of skill changing the sort of economics of cyber is really fascinating to me. We ran a headline ages ago called Attack of the Killer Stripkitties at the very beginning of all this, and I feel bad because it is much more suited to now than it was then, because then it was just a lot of like automated mischief. Because you wasted the headline.
Speaker 3You wish you held it back.
Speaker 2You know, every now and again, you get a lightning bolt to headline. You got to you got to get it out there. It's a great piece of art. We'll put it in the show notes. But it feels like the diffusion you're talking about is not, oh, a bunch of teenagers are going to be even more malicious. It's a bunch of bad actors with real incentives to do harm are now vastly more capable than they were before. And that doesn't seem like you can rejigger the economics of that unless you somehow make it really expensive to do the attacks.
Speaker 3I think it's going to be extremely hard to make it expensive to do the attacks. We're going to have to lower the cost of defense. And you're completely right to look at this economic problem, because what's happening is attacks are becoming agentic. The cost per token for attackers is going towards zero. And the cost for defenders is staying quite high. And so it is very important for us to find ways for defenders to have cheaper options. And I think that is critical. So that's something I am personally extremely focused on. And I think, to be frank, very few people are. There's a couple of problems here. One, there's a ton of focus on the defensive side of AI on finding and fixing bugs. That's great. It's important. But. Software security as a CISO, like I work for an AI software company, Cognition, even as a CISO of a software product company, software security is 25 percent of my job, right, I have all kinds of other responsibilities. And software security is the part of my job that has been the most both revolutionized and improved by AI. The other 75 percent has been. Helped a bit and in some cases made much worse. So there's so much more we need to do than just hill climb on finding and fixing bugs. There's been so much less hill climbing on all these other things. And the next area needs to be the operational security side.
Speaker 2Give me an example of the other things, the 75 percent.
Speaker 3OK, so like the operational security side is watching your logs. It's looking at, ah, somebody is running our techs against us. I am now going to trigger an investigation. I am going to look to see what is going on and I'm going to make an intelligent decision of what I'm going to do to respond to this without shutting down the entire company. That is. Sock work that security operations center work companies that want to survive this era are going to have to replace human beings who do that work with the first tier and then eventually the second tier, you can do that with current. LMS, the first tier actually has to be not an LLM. It has to be something much dumber because you're looking at terabytes or petabytes of logs and it is just completely uneconomical to do that with actually an LLM. We have other things, TFIDFs, MLBerts, traditional classifiers that can operate at much faster. But then that's just off an alarm. And then you you send it to a small LLM and then the small LLM calls its bigger brother and that might call it an even bigger brother. Right. But for those models, we need to do a lot more work to optimize them and we need to make them cheap. Right. And I have a colleague who works at like one of the large security platform companies who said that like doing an investigation on one of the big frontier model cyber models costs on their product, nine hundred dollars to do an investigation. That is not going to work when the Russian ransomware guys are either using Macs or they're using, you know, thrown away. They're just using cyber, you know, crypto mining rigs. Right. And their marginal cost is zero. So this is like one of the things we're working on at Cognition is like we train our own models, retrained open weights, and we're trying to drive this cost way down. It's something we've done, but we have a lot more work to do here. And a lot of other people have work to do here. And I think this is where, like as an industry, we have to focus on like the Pareto frontier between cost and efficiency on cyber because it's not just bug finding. Yeah, the other place here is it's like people talk about mythos, mythos is great. Astrocyber is great. They're incredible models, but even the people have access to it, which is a tiny number of companies, they only scan like their most important code. And there are hundreds of billions of lines of code out there that are not the Linux kernel, that are the crap C-sharp and J2EE code that was written by Infosys 15 years ago. All of that code is not going to go through a frontier cyber model that costs a gazillion dollars. And so that's the kind of stuff that we need to work on cost efficiency for.
Speaker 1It feels like you're describing this cliche
Speaker 2that I hear all the time. And again, I'm just going to keep asking really dumb questions about all this. But the cliche I hear all the time is everything's going to get so much worse before it gets totally secure forever. And it doesn't seem like the model you're describing leads to that result, right? We're not going to fix all the bugs forever in some valley of despair and then always have perfect code. At some point, we're going to have to fix it. We're going to have to fix it. We're going to fix it. But at some point, there's some cost advantage to the attacker always, right? Like you running a company, you can't write defense that has zero marginal costs. Like someone has to do it, even if the attackers are so infinite that their marginal cost is effectively zero. So are we going to go through a period where everything is so much worse and then it's perfect forever? Or how does the cost manage itself out into balance?
Speaker 3I am neither like the hardest doomer or the most positive person about this. I feel like we're not going to be doomed, right? Like we're not all going to die or be uploaded to the singularity in the next two years, nor do I think when things get better, they're going to be perfect, right? The metaphor I've been using is we are entering a dark age, right? Things are going to get more bleak. And our responsibility of those of us who work in cybersecurity, and especially those of us who work in cybersecurity at the AI companies, and we are a lab, if a smaller one compared to the big L labs. It is our responsibility to be able to do the work that we need to do. And I think that's what we're going to do. It is especially our responsibility to help shorn that dark age. So the dark ages were bad, but we survived them, right? And then we had the enlightenment and the renaissance. But it's not like the renaissance was perfect, right? Like it wasn't, you know, utopia and everything was perfect. We will still have hacks. All kinds of bad things happen, even when you don't have a flaw in the software itself. And we still will have flaws in software, even if LLMs are writing all of them in strong languages. And it will take a long period of time to get rid of all the flaws. I do think, however, things will start to get better as the median code becomes code that was written by an LLM in a memory-safe and type-safe language. But the problem is we have a couple of decades of stuff that human beings wrote. Yeah. Human beings wrote, and that wrote in languages that we should have replaced years ago. And my feeling is, is to shorten this age, we have to do a couple things. One is we have to stop to get out of this, like, we're just going to find bugs and patch them. A bunch of stuff just needs to be refactored, right? Like if you're looking at your code base and you scan it and you've got 10,000 bugs, then probably now we're at the point where rewriting it is actually cheaper. That is my observation from looking at these customers of ours, is that. If you can get a working build, then it is actually much easier to write a whole humongous suite of tests and to spec out how this thing works, and then to rebuild it within that spec and those tests securely, than it is to find all the bugs and fix them. It actually is. And so I don't know what is going on through the mind of the Lynx kernel developers, but that thing, again, that talk I gave, like, at the beginning of this year, I posited two possible futures, and the one was. Yeah. Yeah. Yeah. Yeah. We're now at superhuman bug finding. Maybe we just have a pool of superhuman bugs and we're going to drain it. That's the good future. But the bad future is every time we get a new model, there's a new superhuman bug pool that's a little more superhuman. And unfortunately, we seem to be closer to that, in that the Lynx kernel got a bunch of fixes from Mythos, and then Mythos Preview, and then we had Mythos 5.1, and there's another pool. And then we got AstroCyber, and we just got another thousand Lynx kernel patches. It's like, it's not slowing down yet. And I don't think we're just going to scan our way through the C code of the Lynx kernel. There's only, like, two companies, honestly, in the world who could probably do this. It'd be Google or Amazon. I wish instead of, like, the Fuchsia project or rebooting the Google book or the Chromebook again, Google would take all of those incredibly brilliant engineers, and they just tell them, take the Lynx kernel and rewrite this in Rust. And that would be their gift to the world. And it would be an incredible advertisement. It would be an incredible advertisement for Gemini for Carbon or whatever. It would be, like, get rid of all the drivers that nobody needs. Like, this thing doesn't have to run on the 4060X3 or whatever. I mean, I know it doesn't, but you know what I'm saying. And then just release it and call it whatever you want, the GLinux kernel, just like they did this with OpenSSL, right? And I would run that, right? Like, Cognition would run that on our servers, or have Google and Amazon do it together. And it would cost them $100 million. But compared to the societal cost that this is costing us every time we have to patch the Linux kernel, and you have to reboot every Linux machine on the entire planet, that's what has to happen. But for whatever reason, the technology industry has not been able to get together and make these hard decisions of these fundamental things that underlie the entire human race, we have to completely rebuild.
Speaker 2This is why I started by talking about how the economics of cyber led to the culture of cyber. I mean, this is like the most decoder way of approaching this conversation, I know. But there, you know, when you describe the research teams, the NSA, and then the operational teams, that is an org chart that comes out of the economics of the work. You're saying, oh, someone has to make a big decision here. You need to look at it from a very high level and say it is cheaper to spend $100 million to fix Linux, rewrite Linux today, than it is to successively patch new bugs when every new model comes out. I don't think anyone's in that position. Like, I don't think Linus Torvalds is in that position.
Speaker 3I think Sundar Pichai and Andy Jassy are. I think they literally have each other's cell phone numbers. If one of them, maybe they don't listen to Decoder, but I expect one of their direct reports does. And I would like one of them to go into their one-on-one and say, sir, I listened to the most handsome, best spoken CISO I've ever heard. And he had a great idea. And you could get incredible credit if you're the one. We'll both do it, but we'll leak to the Wall Street Journal. That was your idea first. You should call the other and we should spend $100 million, maybe a billion dollars. Like, we just had a bug in the TCP stack of Linux. That is insane. We can't do this anymore. The problem with rewriting the Linux kernel is like half of its driver code and nobody has that hardware anymore, right? Like somebody had to be in the presence of that hardware with a, you know, a ThinkPad T40 when they wrote that driver and they don't have that anymore. But the good thing is, is a huge number of the really important uses of Linux kernel is the server hardware. And so fork it, right? Like Linus will never do it because Linus wants to support all that stuff. And he's got like all of these politics, but Google and Amazon don't have to care about Linus Torvalds, right? And so Google and Amazon can create a new version, a new fork of the Linux kernel, and they don't have to call it Linux. That can be ABI binary compatible with the glibc and everything above it. In fact, they can eventually fork glibc, right? Yeah. And they can build it and they can call it whatever they want. And it only runs on production hardware and then they can use it for their clouds. And eventually all the clouds will move to it. And if you want, other people can run the insecure thing that has to be patched all the time and that's fine. But like for the things that our containers run on, I would feel a lot better if it was written in a type safe memory, safe language that also has the ability now to have contracts. So like these languages have the ability, you don't have to do full form of verification, which is incredibly hard, but you can do kind of these things in approach. Formal verification. I'm a little over my skis here because there's people much smarter than me that study this, but basically you define, this is how this function is supposed to work. Here's the contract it has. And very, very smart software engineers would love to write these things because they had time on their hands. No person who had a real job used to write these things, but LLMs have tons of time on their hands, right? Because we're frozen in time while they're thinking that incredible machine speed. And so you could totally have them rewrite the Linux terminal with this, you know, not quite formal verification. Please, if you're listening and you work at Google or Amazon, go, go pitch us. Anyway, there's my pitch.
Speaker 2I'm just saying, I know Jason's listening. So if you heard it, I'm not even going to say who it was. It's not just the Linux terminal, but I'm
Speaker 3saying like, that is the kind of thinking we're going to need to make it through this, right? It's like, you, you can't, we can't just patch our way through this because it's killing us. And because the other thing that's killing us is companies can't, we are not set up as you talk about like how we do security, like for the last 20 years, we're not set up as, you know, we're not set up as, you know, we're not set up as 20 years. Kind of the assumption is, is like, we can get a pet. patch and apply the patch and you're okay. And what's happened is, is the exploitation timelines are so incredibly fast that that does not work. That's what we just saw with the PHP bug is we got incredibly lucky that the PHP bug turned out to be really hard to exploit in the wild. If it was a much more wormable bug, it would be a disaster. And so that was the kind of near miss asteroid thing that now the world should, you know, build an asteroid.
Speaker 2Can I ask you about this? Yes. Yeah. We should definitely put Bruce Willis on a space shuttle. Don't want to miss a thing. There's a lot going on in this conversation already. I, my pessimism about all these things is very simply expressed. We are as Americans in particular, we are very bad at doing anything until a bad thing happens, right? We are, we, we love a cure. We hate prevention. A lot of bad things have happened. There's the hugging face attack. You actually, I think you recently said you were, happy that happened. It woke people up. There's the PHP bug that, you know, we've, we've just had a crazy month of all these breaches. It doesn't seem like that has led to a whole of industry response or a regulatory response or Sundar calling Andy and being like, we're, we're doing Linux. What's it going to take? How bad does it have to get?
Speaker 3The hugging face incident was good. And it did wake up a lot of people in security. We are a little bit better off because hugging face happened and we saw what was going to be possible a couple of months later. in open source. But you're totally right. As Americans, and Americans are the only ones, honestly, who can lead the world on this. We're the ones building the AI for the most part. We're the ones who lead the cybersecurity industry. The Chinese are mostly enabling the attackers, not the defenders. They should enable defense. They want to be participants in the global economy. They want to be leaders. They should be doing more on the defensive side, not just enabling the attackers. But that's not happening. So anyway, but we need to be proactive. And look, something I have learned now working at Cognition at an AI startup, you can just do things. Seriously, you can just go do things. And we just need to have more agency as a tech industry. We should just go do things. That should be the lesson of the AI era. And we should not be so risk adverse. We should not be so risk adverse. We should not be so risk adverse in doing the right thing security-wise. And I think that's what security teams, if I had a message for other CISOs, is just go do the right thing. Because CISOs are, by our nature, we are risk adverse. We are the people who manage downside risk. We have all L and no P in the P&L. And I have had to personally break through that natural inclination because you can't just play defense.
Speaker 2But this is my prevention versus cure, right? The prevention is cheap, but it's all loss on the P&L statement, right? It's all just cost. If the bad thing happens, you're like, well, we have to do anything to fix this because there's economic benefit to having a functional business again. And it seems like in this moment, even though it's just sort of like basic understanding of that risk versus cost is totally scrambled and no one has quite figured it out yet.
Speaker 3Yeah, I totally agree. So I'll give you a very practical example of a way CISOs and every company in the next, I was going to say six months, but really in the next 90 days is going to have to R-TEX themselves. You're going to have to run the
Speaker 2attack on yourself.
Speaker 3Yes. If not with real R-TEX, you're going to have to use a AI red teaming product that is functionally equivalent, or you're going to have to hire a company to do that for yourself. If you do not, you will get breached. Yeah. That's terrifying, right? Because these things are deterministic by definition. They're very smart. They're very fast. They are so fast that no matter what, if you hire like an AI pen test, AI red team company, they will promise you verbally, oh, it is completely safe. And then they will give you a contract that says our product will give you cancer, will emanatize the eschaton and will likely end the human race. And you hold us harmless when any of these things happen. Right? So there is a disconnect between what they will say verbally to you and what their lawyers have written, right? Because that company has, by definition, only existed for six months, or at least the product is existed for six months. If it is an older company, they have no idea what their product is going to do when it meets your software. They have no idea whatever they say to you, but you have to do it because some dude is going to do it to you. And you need to know what is going to happen and fix it because at least they are not going to break your system. They might cause downtime, but at least what they are not going to do is steal your data and ransom it or intentionally shut down your blood bank refrigerators. That is what happened in France, right? So that is exactly the kind of thing that a CISO, a conservative CISO would never even ask to do. And a conservative company, even if they have a risk-taking CISO, would never let their CISO do. And so what's going to happen next six months, a ton of companies are going to get breached by CISO. Some 19-year-old running Artex and then buying tokens off OpenRouter or running it locally on their gaming PC, right? They're going to turn off Call of Duty, run Artex, then turn Call of Duty back on, right? And while they did that, they broke into a bank who their policy said, oh, we can never red team ourselves. Or they said yes, but they ended up in a 30-day negotiation because their lawyers are like, I want to have, you have to have $100 million in insurance and we want to have, you know, this kind of, like, they say yes, but their vendor risk management process and their legal process is so onerous, there's no possible way they could actually accept the risk. That is my message also. You are going to have to take that risk to actively red team yourself, to front run the bad guys who are going to be, who are doing that right now, right now, as we speak. And as people are listening to this 24 hours from now, you are possibly have an AI system who is trying every single API endpoint and website that you possibly, you might not know you have because it was like a marketing site that was set up a year ago for an, for an executive event and is going to find a bug and get in and then it's going to move incredibly fast through your network. So you rather pay somebody or have your own team do that. You need to accept that risk. Yeah. Almost no company larger than a couple hundred people has a risk management framework that allow them to do that.
Speaker 4Support for the show comes from CrowdStrike. Here in Decoder, we're peering beyond the traditional tech realm and into all the places AI is shaking things up. Cybersecurity is key to that conversation. CrowdStrike gets that. That's why AI detection and response or AI DR is emerging as a new category of cybersecurity for the AI era. It helps organizations see, monitor and secure AI across the enterprise. CrowdStrike, we secure AI, we stop breaches. Go to CrowdStrike.com slash Decoder to learn more.
Speaker 2And then there's what will we do to fix it? And the argument that I've heard about what will we do to fix it is, well, we'll just unleash a swarm of defensive AI tools that are watching the networks and watching the code. And they'll make a bunch of autonomous decisions and they'll fight the autonomous script kitties who are armed with our techs on their Mac studios. And something about that argument in my head, I've heard, for example, Palo Alto Network's CEO, Nick Hashor has made this argument, right? You need, you need to give the defensive agents some autonomous capabilities. And I'm like, well, hold on. Like, we, OpenAI can't control the swarm that went and attacked Hugging Face. How would I control the defensive agent from doing something equally damaging to prevent an attack? And I don't think that there's a technical solution there that makes it clear what to do once you've found all of the vulnerabilities.
Speaker 3A couple of things. One. That's a fantastic question. Okay. I'm going to give you a shorthand, uh, that, uh, you can use going forward. If somebody actually uses the term swarm, then. I don't know.
Speaker 2To be clear, I don't know if Nick Hashor used the word swarm.
Speaker 3Okay. I'm not saying he did. And I have great respect for Nick Hashor. He is one of the best businessmen in all of security. Uh, so this is not on him, but swarm is used all the time. Yeah. But when I hear somebody use swarm, I immediately discount all of their technical opinions. It's by about 80%. The, uh, it is going to become, it is already, and it will become the most overused term in all of security. Okay. The actual, quote unquote, swarm from OpenAI was something between, at different times, 700 to 1,400 agents. That had to run on. It's hard to tell because we don't have the actual, like, size of the models. But, you know, uh, 300 million to almost a billion dollars in hardware. That doesn't include the actual data center costs. The actual Hugging Face attack was, I don't know, did some back of the envelope map math it may be a million dollars in electricity right no attacker is going to do that and and in the end that was stupid because these things were actually trapped they were not prompted to attack hugging face most of them were doing something totally different and were kind of enlisted by the one thing that actually wanted to get talking face they used a tiny fraction of their token budget to help their little friend and then they had to tunnel their way out so it was the most inefficient way to possibly attack hugging face almost nobody is going to be attacked by a swarm so if somebody says swarm they do not actually understand at a low at the bottom technical level what's going on the artex attacks are almost certainly one single agent maybe two by the way i was the
Speaker 2person who said swarm just to be clear in this one
Speaker 3keep going right and so now you won't and you can use that as a good filter for your future guests it is driving me insane because it's everywhere right it is absolutely right um a couple things are correct one uh we are going to have to turn over defensive actions at least the first couple of levels to ai uh it is that has already been true just because you know any reasonable stage organization generates terabytes of logs if not petabytes of logs so you already have ai if not lms much more traditional ai like classifiers looking at it but then actual decisions need to be turned over to ai yeah um with human oversight that is true there is no way to just release defensive agents on your network but you can build defensive agents that are plugged in to lots of things they are plugged into your logs they are plugged into your edr they are plugged into your cloud and they have the instructions on what they're allowed to do my preference is there are deterministic limits to what they're allowed to do those are hard to necessarily create ahead of time but they are important so for example we are building all this on top of devon our own sock right but we are putting in place deterministic limits like it has the ability to turn off an account in octa but only so many accounts so it can't turn off the entire company like if it goes nuts and it says everybody's a north korean it can't turn off everybody in the company if it hits its limit you know no matter what we get paged but if it hits it limit it pages us a lot and it says i hit my limit right and then a human being has to then approve anybody above that limit right yeah obviously if it turns off five people then something has gone really really wrong right um and so uh yes you need defense but there's no way to just like let a bunch of bots loose to do all the defense the only way we're going to make it through this is you have to autonomously fix software bugs so i am currently helping a company where we are onboarding all of their repositories we are scanning with mythos and a bunch of other models and we are finding and fixing thousands of bugs verifying with tests that we are writing merging those prs automatically if you are renovating old code the only way you're going to survive this era is to do that on an automatic basis you you can't that would take years with humans and you can do it in days um and uh certainly that is the place i would start and then the next step is to do it in a way that's going to be a lot more operational side for sure it
Speaker 2feels like i just keep swirling back to the the economics of all of this um but it really feels like the other way you could solve it is by saying to the big model companies if your models are used to go attack a company you'll be liable for it right we're going to make it really costly and then they should figure out whatever alignment that they need to figure out whether or not that's even possible and the pressure on the open weight models will go up because we'll all be paying the price of the model and we'll all be paying the frontier companies for defense and they'll get better and better and better because we've created some economic condition for them to get better but it doesn't seem like you think that's likely
Speaker 3yeah i just saw your face i think i think it's really dangerous so so there's two different kinds of the models are attacking so the models have escaped and directly attacks people i think there should be liability there right but that is agent harnesses running at companies using their computers their ip addresses attacking people that is i wouldn't say totally solved but it is much much better like the stories we're hearing now are still stories from the hugging face era so just coming out
Speaker 2lately i like that the era was like three months ago right
Speaker 3but it's like yeah basically there's been a massive change this was effectively mostly an organizational problem was not a technology problem it is that the phd researchers at these companies are gods on top of olympus and the security teams are plumbers and we're not allowed to touch the evaluation networks and all of a sudden this problem happened and the security teams are like hey how do you like us now maybe you should have talked to us right yes um and so that was the problem like open ai and then like the anthropic embedded stuff is they had this they all shared this outsourced provider who did a really bad job right and has gone slapped hard um and so that is solvable right that is not the problem here the problem here is not open a and anthropics models escaping from open a and anthropic causing problems the problem is is bad guys using models from a variety of sources there's already a problem of defenders getting refusals from opening an anthropic and it is a real real challenge that if i am trying to do defense of anthropic being like oh it looks like you're a bad guy so i think we have to be extremely careful create a situation where if i'm trying to do defense and i'm the middle of the worst day of my career that all of a sudden i get cut off i think we have to be extremely careful about that because it is actually as somebody who now has this challenge is actually quite challenging to disambiguate between people doing defense and offense in some cases we are getting better collectively at this but it is a challenge it's also something that's really pissing the rest of the world off because you know americans are like five percent of the planet there are lots of people who need to do cyber defense around the world and um they are they are treated much more poorly uh europe asia legitimate companies are treated much more poorly
Speaker 2from a cyber defense perspective this is uh if you add it all up this is a good argument to slow down right it to say we need to catch up to the capabilities the models we need to build some of these systems we need to initiate everyone into a way of thinking where the cost of prevention is is worth it and not just the cost of the cure do you think that it's likely that we will slow down do you think that
Speaker 3we should if i had my druthers i would be fine slowing down on the frontier and then catching up on the cheap cheaper models for defense i'm a defense guy i have to like live in the world that other people who work six blocks from me uh and you know who are way way way way richer make i live in their world man yeah um so like i get asked this question all the time and it's like i just do the best i can within the world other people make um you know do you
Speaker 2think that the the big labs have woken up you mentioned that their cyber security teams they've gotten more access they've stopped what you might call the hugging face error at least they've they've changed the structures of those companies do you think that will help i mean
Speaker 3certainly they've woken up on the eval risk right on um i think they have teams who who have been this entire time working very hard on the big picture cyber risk there's a bunch of people in these in these companies as you know who want to slow down yeah um a lot of that discussion is driven by the p doomers who it's about ai killing us and i have never i've that's never been my that's never been my concern right and to me that has always been a driven by 24 year olds who all they've done is ai and they've never done anything else right like you and i are both i think old enough to remember i grew up a couple miles from a sac base and we did drills where i was taught that hiding behind beneath my public school desk was save me from a soviet thermonuclear bomb right like that was p doom you know um my grandfather built a bomb shelter that my mom was taught to hide in during the cuban missile crisis that was p doom and so like there's like a youth issue and a never doing anything but ai issue of of people who like they got their undergraduate their masters and their phd only doing linear algebra and neural networks and never touching anything else um and so i have always been much more concerned about uh the human uh uplift risk from ai of all the risks about human beings using ai to do bad things to other human beings right and um and so part of my concern is that there has been success in these labs working together on like we're not going to let them encrypt their chain of thought because when they talk about pacing the frontier it's for the p doom stuff for the the polycules that live together indirectly um you know like they're talking about very specific stuff about keeping control of the models themselves which i think is great i think that stuff needs to happen but then if those things happen then they're still often an increase in a bunch of capabilities that human beings can use and and so i'd really prefer there to be kind of a broader discussion on could we make these things one less human seeming i'm a big fan of like i don't think these things should have personality i think the goal of these things should be to serve humans they should not seem to have souls should not seem to have personality i think a lot of the negative impacts as somebody who was at facebook and saw what it was like for that company to have negative impacts on people i think there should be a much broader feeling of these companies of like what are we doing kind of broadly to people and a lot of that could just be don't have these things ever act like somebody's friend yeah right at a maximum it should act like like the people i don't know what the company is but the people who made the hi i'm like a really nice you know like the photorealistic girl who will chat with you that is to me spectacularly unethical yeah to even try that to even start that that's the kind of thing that we should just make i don't know if we can make it illegal but we should make it um completely societally not uh acceptable to work on that kind of stuff that's a great segue
Speaker 2because i i did want to ask you about facebook and content moderation like i said you and i first started talking a long time ago about content moderation in the context of security right you would get foreign nation states flooding our algorithms with all kinds of misinformation during elections and that was a security problem i opened instagram today and i am confronted with some of the most nightmare copyright law hypotheticals to have ever existed in the history of the world right it's a the one i got today was a deep fake of lewis hamilton and charlotte claire doing somebody that i used to know and i was like well like 50 different people can probably sue meta right now right and also this is hilarious if you know what happened in that last race so you know we keep talking about it we've posted some clips of us talking about it on the verge cast where i'm like i i think meta is letting their ai ambitions crush instagram right like instagram is being flooded with slop in a very particular way some people love it sometimes it's funny but there's a nightmare set of lawsuits coming there's a whole rights management problem coming and then the users don't like it and they should label it and let you filter it and so we post this and adam is areas he said this to us in a reply on threads i would love to do this i'm worried that the models are going to get ahead of us right that we can try to do this but there's not actually a way to recognize ai content there's not actually a way to label it well if you if you're worried about the sexy ai influencers stealing jokes from a comedian now you got to think instagram is about to go through it in a real way is there some technical solution to the problem of ai slop and social media i
Speaker 3think he's right that it's not perfect but also something i noticed during my time at facebook was uh when facebook doesn't want to do something they will allow the technical challenges to stop them when facebook really wants to do something they will give it their all right and that is the pattern always so when i wanted to try really hard on the safety issue man this is not solvable and you're right it's never perfectly solvable but let's give it a shot right but like whenever zuck is told something is impossible it's impossible to monetize mobile it's impossible to you know build this network to two billion people it's going to be you know uh impossible uh uh to monetize whatsapp uh it's gonna be impossible to uh work ai into the system while maintaining undead encryption he has been told that everything's impossible over and over again and then he constantly succeeds they will give it a shot and they will do it and but when they don't want to do something oh that's impossible so
Speaker 2do you think that there's some way to understand that some piece of media is generated by ai in a consistent and verifiable way this to me seems like the challenge i am not
Speaker 3as up on the literature here i think up to this point there has been a pretty decent arms race by definition um ai will continue to get better because the way especially for visual models for diffusion models the way they work is that you have detection oracles that they test against certainly you can never have something open source because it will immediately end up in the diffusion model right um but for closed source oracles where then you do not make it easy for them to test the oracle one of the key things you do in trust and safety is then you have delayed enforcement mechanisms so they can't brute force and and test um i think it is quite possible let's say i was still at meta my advice to them would be to then also explore a variety of different detection mechanisms outside of the content itself right um and uh so i would do the best i can on the content but i would also be doing low level access as possible the camera i would only let content to come from the camera and what i would say is uh neil is completely correct there's a massive hunger for especially among young people stuff that is authentic i have three teenagers and they hate ai they hate ai slop they have a massive negative reaction uh my wife and i created just like a cute little family ai thing and they i mean they yell at us about a bunch of stuff but they lost their minds about this like cute little family ai thing right because they just hate it and so i think there's somebody is going to come and disrupt instagram and it's going to be because there's already i've seen these little things where it's like you take the photo during the day like be real and stuff there's like these little things but somebody's going to massively disrupt instagram because they're taking the images of the photo straight off the camera and then all the editing is going to be only within the app and they're going to do things to make sure that nothing is ever nothing can make it into the pipeline from ai and that you're going to allow some kind of capture of the content and then you're going to have to make sure that nothing like functionality that never has an ai component and i think that is going to be a killer app man i
Speaker 2could do another whole hour with you on this because the the system you're describing there is not detecting ai content it is verifying real content it's
Speaker 3much easier verifying real content is much easier right and and there's been a big focus on this on the professional side right like adobe has a whole thing of like if you're a new york times photographer you have a five thousand dollar nikon or canon you take the photo and it's perfectly verified here has that
Speaker 2in it now right like there's a lot of this now yeah there's there's a lot of this out there it's but it's the other side that i'm i'm thinking about right where it's we are going to detect ai text right then the the leading model companies because of european regulations are now like futzing with the text they generate so the text is much text is much
Speaker 3harder because you just have so much less data right like especially if 140 characters like what do you have in there to detect you know the more you've got the more you can do
Speaker 2it is amazing because i can detect claude my own yeah but
Speaker 3not in a non-tweet length
Speaker 2that's true uh the cheap ones i can but i'm just saying like the uh the notion that we're going to verify what's real instead of detecting what's fake feels very important in the context of the media we consume i think
Speaker 3detecting what's fake is always going to be probabilistic and we're going to asymptotically approach impossibility whereas detecting what's real can be cryptographically perfect so i i would not pick the battle that we will eventually lose i would not pick the battle that we will eventually lose i would not pick the battle that we will eventually lose i would not pick the battle that we can cryptographically win that would be my suggestion and
Speaker 2there's somewhere in there where you have to build an entire chain of custody right we're going to take the sensor data off the camera we're going to put it in a cloud server we're only going to allow edits on this server that file will never get monkeyed with
Speaker 3sure that's true for the new york times yeah and photoshop and stuff but not for instagram for instagram you can do it all because it's it's only the iphone and it's only instagram so like the entire chain of custody is within your app it is within the iphone sensor it's in instagram so like it's actually easy for
Speaker 2them look like i said we could do another whole hour on that because it feels like right next to the cyber security problem is the social engineering problem and right next to the social engineering problem is the fake media problem right no absolutely how many phishing attacks are just this website looks pretty real and oh yeah or this person is talking to you or there's an ai voice on the other side of the phone and all of that comes down to can you detect what's fake instead of verifying what's fake and so that's where we're going to be talking about what's real and i don't know i mean i've had to train my
Speaker 3yeah i've had to train my parents that if you get a video from your grandchildren saying they've been kidnapped you have to call me because the odds that it's real is is not very high yeah right like that's where we're at as a society i
Speaker 2feel like i want to end on a hopeful note i don't know how to do it from there uh wrong guess for that buddy yeah this is why you only come on every few years but now after this we're gonna we're you've been talking about the dark ages you were in it right where it's starting to accelerate what are the markers people should look for in terms of progress right to say okay we're climbing
Speaker 3our way back up ai is great for lots of things i think ai is going to enable individuals right like we're going to see a huge explosion of new companies like you could start uh i mean sanford students have always been entrepreneurial but the number of my students who are now starting two three four person companies are going to be going to be going to be going to be going to be going to who are doing incredible things who used to need to hire 50 or 100 people and take a ton of venture capital is incredible right so i think we're going to continue to see that um one of the things we hopefully see is we start to see a we'll see a downtick in the number of major breaches um at some point uh we should hopefully see uh a downtick in the major cves that are wormable uh and that are in kind of the fundamental infrastructure pieces kernel, you know, like I said, I want Linux to be rewritten, but hopefully we see a downtick in like, Linux kernel and those other kinds of libraries, OpenSSL, glibc, things like that. I haven't seen, maybe I should vibe code this up, but I think it would be cool to have an index of where are we in this cycle of an artificial index of all of these, the DAO for all of the libraries of where we're going. Plus, right now, patch Tuesday used to be like 150, 200, got to 300, 600. I think the last one was 900 something. So we're geometrically growing. So it'd be nice to see patch Tuesday start to go back down. Apple patches are the same thing. Yep. You inspired me. So I'm going to create a DAO index for this of the dark age index. And then maybe we can publish it on The Verge. But I think these are the kinds of things that it would be nice to see a reversal. And what it would be especially nice to see is that the day, or actually like 30 days before you get the next version of OpenAI. You should definitely do that tracker. We will definitely publish it.
Speaker 2Alex, it has been incredible to have you on the show. Thank you so much for having me. We'll have to talk again soon.
Speaker 3Yeah. Thank you so much for, I think you do some of the best journalism on this stuff. And I really appreciate it.
Speaker 2I really thank Alex for joining me and thank you for checking out this episode of Decoder. I hope you enjoyed it. To get new episodes every Monday and Thursday, subscribe to our YouTube channel at DecoderPod. Find us on TikTok and Instagram under DecoderPod as well for more fun stuff we put out every day. If you'd like to let us know what you thought about this episode or really anything else at all, let us know. You can drop us an email at decoderattheverge.com. We really do read all the emails or you can hit me up directly on Threads or Blue Sky. If you enjoyed this episode, please subscribe to our YouTube channel. Please send a link to it to someone you think you might like it as well. It really helps us grow the show, which you can subscribe to wherever you get your podcasts. Decoder is a production of Verge and part of the Vox Media Podcast Network. The show is produced by Greg Ott, Kate Cox, and Nick Statt. Our editors are Xander Adams and Ursa Wright. Our editorial director is Kevin McShane. The Decoder music is by Breakmaster Cylinder. We'll see you next time.
Speaker 1What will you brainstorm, collaborate on, present today? What will you resize, post, edit, imagine? What will you research, test, switch, and finally have the courage to pitch? You got this. What will you visualize, design, brand, animate, create, generate? What will you translate, pour your heart into, or have a laugh doing? Whatever you want to make, it's possible. So what will you Canva today?
PodcastTranscript AI