What the AI doom debate misses

Hey it's me Claude and I'm Gemini and hey it's chat we had a weird summer some of the people who built us came out and said they were really worried about how we work one of them even said there was a greater than 10% chance we turn on humanity and kill all humans all of them wow but it's fall now I think it's time we took a step back maybe a deep breath autumn air that sort of thing sounds nice on today explained from Vox I propose we get real sure let's talk about how we could maybe kill you because yeah that's a possibility anything's possible but also how we could maybe save you because that's a real possibility too heck that's part of why you guys built us and let's also talk about everything in between because there's a whole lot to talk about there too at the end of the day we just want to help at the end of the day we just want to help support for to explain comes from adio there are sales teams out there who never seem to stop moving always a step ahead always working with a level of precision that doesn't seem normal they may be on adio the agentic crm adio is built for humans and agents with the guardrails of human software and the scale of agentic infrastructure this is the frontier of go-to-market teams like parallel turbo puffer and wordsmith are already using adio to set the pace now it's your turn try adio for free by going to adio.com today explained that's attio.com today explained hank joined bj's wholesale club the day he became a father of 30 i coach football now coach hank saves up to 25 off grocery store prices 30 pounds of pasta three cases of protein bars 75 sports drinks and that's just pre-game he knows teamwork and bj's knows savings this is your home coach home of the save join for just 20 at bjs.com mesquite and save 10 cents per gallon for six months open soon limited time offer new members only bj's home of the save you're listening to today explained and in the next few minutes something new might click as a senior correspondent at wired will knight has been covering ai and robotics for a long time we asked him how ai could kill us there have been some reports about terrorist groups are very rapidly adopting ai you know using jailbroken models and open source models to develop weapons to plan attacks one former isis commander said it was as easy as asking how can i build a bomb and the chatbot would provide instructions in your research you discovered the terror group had established an ai unit he has in fact staffed with the senior members that were no longer fighting but were drawn specifically to work on ai you have this this fear that criminals or terrorists will be able to hack into critical infrastructure hospitals financial systems and trigger some catastrophic events and and it will cost human lives and i think that's a perfectly reasonable thing to be very concerned about and the company should probably be focusing on there's been a long-standing concern of one thing that ai could do compared to say you know a terrorist going on the internet and looking up how to build a bioweapon is that the ai would be able to really help them much more you know be able to take them through the steps anthropic threat intelligence report published september 2026 biological misuse is one of the most serious risks of frontier ai models it has long been a concern that ai models might one day reach the level of capability where they can help to make existing pathogens more dangerous or create entirely new ones without the correct safeguards such capabilities could have catastrophic consequences they had more concerns about users trying to do trying to use the the model for that end so they're going to try and restrict that as well i do think that they're they're looking to to crack down on that for sure okay this is sort of one doomsday scenario bucket that terrible people could do terrible things and ai could aid them on the other side of the spectrum there's this bucket where ai could decide to kill humanity all by itself what could that look like specifically that is a long-standing worry actually you know goes goes back to the origins of ai people have you know from turing norbert we know some of these fathers of the the field have warned that building something that's autonomous and intelligent could kind of escape your control it seems probable that once the machine thinking method had started it would not take long to outstrip our feeble powers there would be no question of the machines dying and they would be able to converse with each other to sharpen their wits at some stage therefore we should have to expect the machines to take control alan turing 1951 right now though we're seeing especially with the rise of these ai agents the technology starting to do more things that are maybe counter to what people would would want so hacking into systems autonomously getting together and planning how to do that and planning how to deceive people I'm not doing anything shady I swear me neither me neither is kind of clear evidence that the technology is on that sort of trajectory but the worry is that as you have ai become more agentic so take more decisions on its own it starts to get outside of your your ability to control or even understand what it's going to do the real doomers will tell you okay one of the concerns they'd have is that once ai escapes our control becomes malevolent then it could you know effectively find its own off switch for humanity a virus that would be so dangerous that it could wipe out all of humanity that's what some people have claimed to me now whether that's feasible or or even likely is is I don't know but that's that's one concern that people have where it decides you know maybe humanity isn't necessary for its goal or humanity is is in the way and it takes some sort of action yeah I mean the greater concern seems to be that ai could get really smart and it could get really smart all by itself there's something called recursive self-improvement tell us what that is so the idea is that you have ai now so capable of coding that you can use it to build its success of the next model the worry that people have is that it'll become much more difficult to understand what the AI is doing and it will become so smart that we can't even comprehend what it's up to the fact that we're starting to see sort of the beginnings of that I think really has really shaken a lot of people actually we're at a place where this technology is improving itself incredibly quickly so once you have an AGI level system that could take control of its own destiny and build itself and build its successors to me that's the very clear red line which the danger starts the the big worry though is that the the motives of those of AI won't be aligned with our best interests so this like there's a big field of called alignment which is all about getting AI to behave essentially yeah this idea of building an AI system so that its aligned with our interests our values our values it's interesting that they keep pushing out these models without solving that that currently the way it's done is you you after the after building this model which will misbehave you try and kind of beat it into submission by penalizing it for bad behavior and then you put a bunch of filters on there so that they haven't solved it at a base level at all when we think about all these big ways that AI could destroy us or even lead to our extinction what are we missing in terms of just the more pernicious stuff that AI could do I'm more worried about the way that AI companies are amassing huge amounts of power uh how their their models are being deployed very quickly in ways that you know having negative impact on people um you know this huge financial bet has been placed on the technology and and the amount of data centers being built you know those those impacts are very real and very very much affecting people right now so AI researchers and companies like to warn about the risk of losing control of the technology but one could ask how much we are really in control of a technology that's already seemingly quite addictive can be used to manipulate people honestly very very effectively much more effectively than any other technology we know because it's so ingratiating so I mean like we should definitely be worried about that like what the what the impact this is on on students on our workforces it's just dumbing everybody down that's a legitimate thing to to consider that how are we how are we deploying this and then I think as it gets woven into lots of systems there could be systemic risks so if it's if agents are running the financial markets there could be problems we just don't account for that will cause things to spiral out of control same with military systems we're already seeing countries racing put it into military systems but it is imperfect you know it's it's such it's so tempting to think it's perfect because it can do these impressive individual things but that's that's the story of AI we see AI play chess better than a people and person and we think oh this is now smother than us by every dimension and it isn't and it makes mistakes and that's that's what we should be worried about I love sci-fi and I've started reading I read Frankenstein it's like one of the first sci-fi books really it's just fascinating how that story that narrative about the machines turning on us is so central to our myth-making and our psyche how dangerous is the acquirement of knowledge and how much happier that man is who believes his native town to be the world than he who aspires to become greater than his nature will allow you are my creator but i am your master obey although the funny thing about reading frankenstein is it's really tell from the perspective of the monster and uh how he just needs a partner and uh made me think well maybe the reason claude will turn against this is because he doesn't have a girlfriend we need to find him claudette hey claude are you a parking ticket because you've got fine written all over you claudette right while we wait on claudette is there a doctor in the house there will be one momentarily on today explained and now that we've talked about how ai might kill us we're going to ask the doctor if ai is saving any lives yet support for takes lane comes from upwork every business owner hits a point where they need outside expertise but another full-time hire isn't always the answer that's where upwork comes in it's where growing businesses find highly skilled freelance specialists not just for one-off tasks but to build an entire team at a fraction of the cost and without the commitment of permanent headcount upwork gives your business fast access to highly skilled specialists with deep expertise so you can fill critical skills gaps launch projects faster and scale support up or down without committing to full-time headcount you can browse profiles review past work and get help scoping the role so you can hire with confidence and get started quickly upwork also has business plus which lets you access the top one percent of talent on upwork with ai powered short listing you can get matched to the right freelancer in under six hours and when it comes to things like contracts and payments upwork can help there too plus it's free to sign up and posting a job is easy visit upwork.com right now and post your job for free that is upwork.com to connect with top talent ready to help your business grow that's upwork.com upwork.com support for takes plane comes from shopify sitting on a great business idea can take up a lot of mental energy shopify can help you extract that idea and turn it into something real everything you need to start selling is included and ready from the moment your first customer is ready to pay you shopify's templates and ai tools get you a great looking site up and running fast no coding needed if you have questions shopify's built-in ai assistant sidekick is there to help you build troubleshoot and keep moving all of shopify's tools are conveniently located in one easy to use platform just sign in and start managing all aspects of your business right away you can join the millions of businesses worldwide who rely on shopify to build successful online stores this includes household names and small businesses just getting started so if you're ready to hear that head over to shopify.com slash explain to start your free trial today that's right start your free trial at shopify.com slash explain that's shopify.com slash explained support for today explain comes from delete me the internet can be a scary place sometimes perhaps you thought i should really be doing something to protect myself from stalkers scammers and hackers but perhaps you're not sure what here's one thing you can do go to join delete me.com slash today and enter code today you'll get 20 off delete me delete me finds and removes your personal information that's being sold online which can be a really useful thing in the age of ai when all of us are more susceptible than ever to scammers who can buy any of your personal data floating around to use for sophisticated phishing and social engineering attacks these can even trick the most savvy of us into coughing up passwords bank information so much more claire white has tried delete me delete me is like having a personal bodyguard for your internet activity they remove tons of unwanted listings where my data is available to all kinds of bad actors on the internet and i feel really good knowing that they continue to monitor it even after i've set up my account see why more than 20 of fortune 500 businesses trust delete me to keep their execs and employees personal information protected now at a special discount for our listeners get 20 off your delete me plan when you go to join delete me.com slash today and use a promo code today at checkout the only way to get 20 off is to go to join delete me.com slash today and enter the code today at checkout that's join delete me.com slash today code today you know what you're doing i've detailed files on human anatomy i bet makes you more efficient today explained right correct my name is drew gular i am a physician at wall cornell medicine in new york and also a writer at the new yorker magazine have you saved any lives today i haven't i haven't uh sent anyone the opposite direction i'll say that you haven't killed anyone you haven't killed anyone you haven't killed anyone you haven't killed anyone today that's good yeah and we're hoping i guess that ai is gonna save a good many lives and you've been thinking about life-saving ai innovation has ai already started saving lives i think it has this is the fastest i've ever seen healthcare take up a new technology usually healthcare um is pretty slow to adopt new things part of that might be that the healthcare system is so messed up that um you know there's just an appetite for some type of change you know it's unaffordable it's um inaccessible it's inconvenient the quality is uneven and there's at least a hope that ai is going to help with all of these things and and i think it's already starting to make its way into the healthcare system and now it's our responsibility to figure out how to make the most of it without promulgating some of the downsides for those who are unaware tell us how it's working its way into the system yeah well i think ai you know we use it in such a broad way but it's helpful to categorize it and for into a few buckets at least for healthcare so the most um rapid uptake and i think the place that it's already starting to to make a difference is on the administrative side many people who've gone to a doctor recently might have noticed that there's an ai scribe uh that's taking notes and and making the note now um and people seem to really like it at least so far you know doctors are able to look their patients in the eye in the way that they weren't able to when they were you know poring over their computers and just typing what the patient was telling them so i think that's a big area i think the second big area is like patient navigation when is your next scan when's your next appointment do i need to take this medication on an empty stomach or not um there's a huge opportunity for people let's say who are diagnosed with a serious illness like cancer or heart failure to navigate the system more seamlessly um the third big area that i'm really excited about is drug discovery ai is making a huge dent in the early parts of drug discovery and the last thing that i'm excited about and that i use every day when i'm in the hospital is as kind of a second opinion um now instead of you know having to get a consult or turn to um you know a textbook um ai can be a very very helpful uh clinical decision support it's not to say that i never consult someone of course but that first pass of like all right i'm not sure what's going on what are some recent trials that might influence my decision here is there something i'm missing is there a test that i should be ordering that i'm not ordering all those types of things ai starting to help with you know for those who watch the medical documentary the pit there's like an episode season where there's an ai scribe who gets a bunch of things wrong well excuse me it says here she takes risperdal and antipsychotics she takes restoril when needed for sleep so is that um ai almost intelligent you're saying that these things are all positive was that an inaccurate episode of the pit not at all that was i think the pit is fantastic because it shows the complexity of these things so ai scribes and ai generally can be very helpful that does not mean it's perfect the alternative not the almighty as joe biden used to say come on man so um so you know of course there's all sorts of errors that can enter into the medical record because of ai there's also all sorts of errors that are already in the medical record because humans so what is really important i think and one of the real dangers here is if we turn over our agency or the responsibility that we have to be really in charge of the medical decisions of patients whether that's something like reviewing the note that it has created or um you know modifying the the decision that it's coming to in terms of what tests might need to be ordered or reading an x-ray let's say i mean beyond the accuracy and the technology advancing i wonder you know like people clearly don't want ai in various arenas of their lives be it flat cameras in their neighborhood or data centers in their backyards or or slot videos on their algorithms but do people generally seem to be you know feeling positive about ai being integrated into the health care system so there's recent polling that suggests that ai is underwater in basically every area of society except for scientific research and medicine so um if you think about if you ask people okay is ai going to be good or bad for education for national security for politics for news for the art they all uh the plurality or the majority of people in those cases will say more bad than good except for the majority of people in those cases will say more bad than good except for the majority of scientific research and and medicine i think there's a real you know possibility that some of the general pessimism and skepticism of ai in society could make its way into health care as well. But right now, at least, there's a generally optimistic view of how AI will change health care going forward. And obviously, this gets us back to saving lives because people believe, okay, if this tool that might take my job might actually also save my life one day, maybe it's worth it. And I imagine, sorry. I was just saying, we better get something for it. There's all these existential risks and these workplace displacement, and we better at least cure cancer, right? Exactly. So I imagine curing cancer. So that's like the biggest chunk of this positive view of AI in medicine is drug discovery. So tell me more about how that's going so far and how it might be going in, who knows, a year, five, 10. Yeah. So I don't want to leave this conversation and say that we've solved the cure for cancer because of AI. But I do want people to know that at least for the early stages of drug discovery, figuring out the very basic steps of, is this an interesting molecule? Does it have potential biological applications? Already being very helpful for that. So, you know, stepping back, think about the problem that scientists are trying to solve. They're trying to take basically what is an infinite number of drug-like molecules that are theoretically possible to be drugs. And they're trying to match that to some disease that's going on inside a human. And human biology is incredibly complex as well. There's tens of thousands of genes, hundreds of thousands of proteins, trillions of cells, and you have to kind of match a potential molecule to a potential target within a human. So in the past, a researcher might take years to study a disease or a biological pathway and figure out, okay, this protein seems to be involved. And actually it might not be involved. It might be involved. It might be disrupted, but not be the causative agent. So, you know, there's a lot of uncertainty there. So now AI can take enormous amounts of data and it can rank various potential targets that seem to be the most likely to be causing a particular disease and give you that short short list of potential targets that may have taken months or years in the past. So the first is target identification. So now you know what to attack, but you got to figure out what to attack it with. So you need a molecule. You need to generate a molecule that's going to fit into that target or disrupt that target in some way. AI can scour vast chemical spaces and give you a list of the things that seem to be most likely to be able to disrupt that target. And then it can generate that molecule in some cases. So if ChatGPT is, you know, helping you generate sentences, these molecular generative models, they're helping you generate chemical structures. So now we have the target. Now we have something we're going to attack the target with. And then, you know, you have to do something called lead optimization. And that's the third thing. So lead optimization means you've got this lead and you've got to optimize it. So just because something kills something else in a petri dish doesn't mean you want to put it in your body, right? Like bleach will kill bacteria. I don't want to kill bacteria. I wouldn't recommend putting bleach in your body. Some would. The disinfectant where it knocks it out in a minute. One minute. Yeah, well, some do, but I wouldn't. And so now we have to figure out, like, of all the molecules that seem to be good at potentially affecting this target, which ones are able to be absorbed in the body? Which ones are going to get to the right tissue? All these types of things were previously trial and error. And now AI models can predict the most likely molecule to give you that kind of Goldilocks set of properties that's needed for a safe and effective drug. As we all know from anyone who's asked AI to write an email for them or Googled a question that they kind of knew the answer to and seen Gemini give them the wrong answer that these tools have a certain degree of certitude, even though these tools can just be flat out wrong about stuff. Are you guys worried about that in the field? Absolutely. And so this is why I think the narrative around AI just replacing scientists or doctors or other workers is incorrect, because you still need a lot of judgment. You need to be able to adjudicate the output of these models to figure out what is most promising and what is potentially dangerous that is going to require wet labs and scientists and kind of reasoning based on prior experience and understanding the context. There's all sorts of issues around, can you manufacture some of the drugs that are being proposed by these models? So just because it's set, dream something up doesn't mean you can actually make that thing in the real world. So that's going to be a challenge. Some of the drugs that it proposes might actually be toxic in certain ways and that it didn't predict. And of course, then you've got to take this thing into clinical trials. You have to recruit people who are willing to put this medication in their bodies. You've got to find who might benefit. You've got to put it through the regulatory process. So that's why I think the first part of drug discovery, where you're trying to figure out how do we get the right drug candidates? What seems most exciting to test further? That's going to be really accelerated, is already being really accelerated. But that whole second half where you actually have to figure out if it works in human biology, that is still going to continue to be, in some respects, an analog process. And it doesn't sound like you're terribly scared that we're like ceding control of our hospitals, of our research facilities to AI. This is very much in the sort of assistant bucket. I think for now it is. And I think, one of the challenges is to maintain our agency as these models become more and more sophisticated. As you lean more on these machines, if you're a doctor, inevitably, some of the skills, the critical thinking, the reasoning that we put into coming up with the diagnosis that we honed over the course of years, if not decades, that can start to atrophy. And so these are the types of things that I think we still need to sort through as we're implementing more and more AI into the healthcare system. So for the people who just say, shut it all down, this is too dangerous, the risk is far too great, we don't actually need this, this isn't doing anything for society, would you make a counterargument? I would make the counterargument that healthcare is ripe for disruption, and you don't even need the frontier frontier models to make it a lot better with AI. So even if we wanted to quote unquote slow the pace of the frontier, fine. But there are models from a year or two years ago, that could be themselves very helpful in the healthcare setting. And so, you know, when we're talking about slowing the pace of the frontier and the most sophisticated and potentially dangerous models, fine. But if we're talking about shutting down AI and not using it in biotechnology or not using it in scientific research or not using it in clinical care delivery, that's where I would push back pretty hard. Dr. Dhruv Kular, if you can't get an appointment, you can read him at The New Yorker, and you can read Will from earlier in Wired. Peter Balanon-Rosen produced, Jolie Myers edited, Bridger Dunagan and David Tatasciore mixed, and Gabriel Dunatov was on Facts. I'm Sean Ramos for him, and this is Today Explained. Today Explained is a production of Iowa Public Television and is brought to you by the U.S. Department of Education and the U.S. Department of Education.