Wanna slow down AI? These biologists can help.

This is Planet Money, from NPR. There's a very strange puzzle at the center of the AI debate right now. If by strange you mean like totally terrifying and apocalyptic. All those things. And that is that many of the people in charge of the AI companies are begging for someone, anyone, to be the adult in the room that tells them to stop speedrunning the apocalypse. Yeah, they're saying like, given that there's even a tiny chance that the products we're building could lead to the end of human life, seems like it might be a good idea for everyone to just pump the brakes a little bit. But at the same time, none of them are taking any meaningful individual steps to slow down. Probably because unless everyone else agrees to get on the same page, they fear falling behind and losing the race. In other words, what we've got on our hands is a collective action problem, where people know they need to come to an agreement, but they don't want to do it because of their self-interest and because they worry the other tech bros might defect or won't play by the same rules. It's a heady time. And there doesn't really seem to be a way out. But sometimes, history offers a way. So we headed to the stacks, over to the Dewey Decimal section for collective action problem, and we found this kind of wildly analogous story of a time when a new technology promised to transform life as we knew it, but also to potentially destroy the world along the way. And the people most invested in that technology got together and puzzled through this question. Could they collectively find a way to stop this runaway technology before it was too late? Or would their fear of missing out make that impossible? Hello and welcome to Planet Money. I'm Alexey Horowitz-Gazi. And I'm Nick Fountain. If the tech barons actually want to meet, actually want to slow things down, may we offer a lesson from biology? That's right. The mitochondria is the powerhouse of the. No, I'm just kidding. Today on the show, we're going to tell the story of the time when scientists, faced with this new technology that they worried could lead to highly contagious diseases, bioweapons, weird eugenics, and a whole host of other problems, figured out how to push past a collective action problem. What's scary about this moment is that there's this society-morphing, world-changing, maybe economy-wrecking technology. And it's not only the Luddites who are saying, let's slow down. It's the people who are leading the charge. But they aren't slowing down because they know it'll be bad for them, individually. And if the story that we're going to tell today is an allegory, they are kind of right. This might be bad for them. When you solve a collective action problem, someone is likely to get hurt. Just ask Janet Mertz. As far as I understand it, this whole story starts with you. Yeah. Are you bummed about that? No. Janet is a big-shot biochemist. In the 1970s, she was a 21-year-old getting a PhD at one of the premier programs in the U.S. at Stanford University, working on research that was revealing the very nature, of how life worked. Did you have ambition of winning a Nobel? We were all thinking about that, back in that era. All right. I'm glad you admitted it. And actually, two of the other students in my cohort went on to get Nobels. Uh-huh. So, it wasn't just my thesis advisor. You know three Nobel Prize winners, personally? Oh, I know a lot more than three. Now, there is a reason so many people Janet knows personally went on to get the big kahuna. And that's because biology and chemistry, they were kind of undergoing a revolution at the time. And Janet was at the cutting edge. Which was all about cutting. Cutting up an organism's DNA and splicing it into another organism's DNA. They called this recombinant DNA. Prior to that, people would make recombinants using natural methods. So, you know, nature all the time is taking genes from. From one organism and getting them into another organism. But it required the birds and the bees up until that point. Yeah, it used to be through breeding. But now, all of a sudden, there was the possibility of copying a gene from a bird and pasting it into a bee. That's right. Yeah. This new technology had enormous promise. It could maybe enable the mass production of insulin in the lab, which up to this point had to be harvested from animals like pigs. Or creating new crops that fixed the. That fixed the soil as they grew. The possibilities were endless. As one scientist put it back then, you could maybe even grow pork chops on trees. Weird, but all right. Janet was working on this miracle technology, developing an experiment that would push it forward. But she wouldn't get to finish this project. In part, because of this guy. Were you the whistleblower that started this all? Yes. How does that feel? It defined the rest of my life. This is Bob. Bob Pollack. In the late 60s, he landed a job working for another Nobel winner, the guy who helped discover the double helix, James Watson. Watson wanted Bob to help him run a lab in some shingled buildings on the shores of Long Island. You were hot stuff. I was hot stuff, yes. The work was hot. The James Watson. Yes. poached you, headhunted you. Well, let's just say I accepted. As part of Bob's job. Bob's job. You get to teach a three-week course. It's a course about animal cell biology that grad students from all over flew in for. Janet was a student in this course. It was a crash course. First day of the course, they're doing introductions. Your name, your major, your thesis project? It's Janet's turn. Hi, my name's Janet Mertz. I'm a first-year grad student in the Stanford Biochemistry Department. And my thesis project is to. Her thesis project is a little jargony, but basically she wanted to better understand how viruses work in the environment. In primates, including people. She was going to try to do that by cloning some recombinant DNA in E. coli. And the particular DNA that she was going to clone contained a virus that just so happened to cause tumors in some animals, like hamsters. And what happened next? Bob Pollock sort of freaked out. Wait a minute, wait a minute. This is a dangerous protein. What if the now-possibly-cancerous E. coli escaped Janet's lab and got into a human? You put it in E. coli. You'll forgive the technical term, but you crap out E. coli all the time. It's in your body. The perfect vector. So do you want to do that? Do you want to put a cancer-causing anything into something that can live in your gut? What if it somehow slips into a grad student's gut? What if it slips out into the wild? You could give the world cancer. This scares the hell out of me. Now, Bob's freakout, it did not end there. He was so worried. He was so worried about this that he decided to basically call the principal's office, to call Janet's boss, a guy named Paul Berg. Give me the exact conversation. Ring, ring, ring, ring. Hello, it's Dr. Berg. Who the f*** are you? No, it wasn't that, actually. No. Bob says that's not the right order. First, he told Dr. Berg how irresponsibly he thought Janet's research was, that they hadn't thought through all the implications, and then Dr. Berg essentially said, who the heck are you? But eventually, Bob kind of convinced Dr. Berg to at least call around and see whether people he actually knew and trusted thought he might indeed be opening up Pandora's Petri dish. Because our feeling was, we could easily arrange this so that it doesn't escape from the lab. But of course, you know, as we know, things occasionally do escape from a lab. You know, so it's not like it's. Was that a Wuhan reference? I think it was. Janet is actually on the fence about whether COVID escaped from the lab, but she says lab leaks do happen. Back then, Janet says they thought the risk of her research destroying the world was teeny, one in a billion. And the potential reward of understanding these viruses better was huge. But as Dr. Berg polled his colleagues about whether the reward actually outweighed the risk, he became convinced. Bob was right. And so he put the kibosh on Janet's experiment. If there was even a small chance that this was going to hurt someone, it. wasn't worth it. And how did Janet feel about this? Well, it's above my pay grade. Kind of, but this is also your best shot at the Nobel, right? Well, the reality in that era, Nobels almost never went to women and almost never went to grad students. It's your best shot of being a superstar biochemist. But anyway. But anyway, it's not like after Janet was told to stand down, biochemistry just. stopped. In fact, it kind of accelerated. There were new advances on the regular, including by Janet. She made some advances in cloning, and she figured out how to splice DNA way more easily. Making recombinant DNA went from being an experiment that, at the time, very few people in the world could do to something a bright high school student could do it. And anybody could make their own chimera. Yeah. This rapid spread. of this new technology could potentially pose a whole host of new problems, like if they could get the right materials, what was going to stop a terrorist from making some superbug? For that matter, what was going to stop some precocious but ultimately incompetent teen from inadvertently creating a superbug? Plus, it raised the ethical conundrum of potential designer babies. In other words, everyone was now racing towards an outcome that could be dangerous in all sorts of ways. It was a collective action problem. They knew that if, collectively, they could stop or slow down and make some tweaks, the world would be safer. But as individuals, for their own livelihoods and careers, the rational thing to do was keep going full steam ahead, so they could win the Nobels and get the tenured-tracked positions. It only made sense for them to stop if everyone else was going to do it, too. Now, there are several ways to try to solve collective action problems. You can get the government involved, of course. You can try to align the incentives in a way that makes people play nice. And you can try to get the government involved, of course. And you can try to align the incentives in a way that makes people play nice. And you can try to align the incentives in a way that can self-govern. Janet's boss, Dr. Berg, was one of the people who took a first stab at that last option, self-governance. Yeah, in 1974, he and a group of the country's top lab heads, basically everyone who had the ability to do this maybe scary research, published a letter in the top science journals saying, OK, this is potentially too dangerous, so we're all agreeing to hit pause on our research until we have a chance to meet up and figure out a safe way forward. They agreed to In other words, let's all stop. This is Michael Rogers. In the mid-70s, he was working as a reporter at Rolling Stone. And he looked the part. He had a leather vest and long hair. But instead of interviewing rock stars, he says his editor forced him to do science stories. And when he read about the moratorium, he knew it was a big deal. That was a first, I think, in the history of science. A first. Good premise for a story. And it involved genetically engineered creatures. Now we've got some fun. Yeah, it had like a double helical twist. Yes, exactly. Which is why in February of 1975, Michael ended up driving to a rustic conference center not too far outside of Monterey, California, called Asilomar to attend the now very relevant Asilomar Conference on Recombinant DNA. This was a room full of nerds. It was a four-day conference. And most of the action took place in one big room, a chapel. Asilomar used to be a young women's Christian association camp, so they had one. Which I always thought was, you know, very meaningful. Here we are debating the fundamentals of life in a chapel next to the ocean where life began. Amazingly, recordings of this meeting still exist. Shout out to the National Academy's archives for those. And listening to the tapes, it becomes clear that the strongest voices in the room were those of the steering committee, the folks who organized the conference, including Dr. Berg, Janet's lab head. Janet herself did not get the invite. No, I was a grad student. They weren't inviting grad students. Plus, of the 140 people who attended, mostly scientists, there were very few women. At that point, most everyone there had abided by the temporary moratorium. They had stopped or slowed their research out of an abundance of caution. And now, seemingly everyone wanted the camp lifted. But how? Therein lay the debate. There were basically two camps. Those aligned with the organizers, who wanted to walk away in just a few days with a new set of guardrails, a consensus on how to do research safely, and those who wanted no restrictions at all on their research. Their basic stance was, anything that slows us down will be onerous, expensive, and will stifle our academic freedom. This camp included big deals like Mr. Double Helix himself, James Watson. Which side was going to win? The fight would play out over these four days, not in an octagonal cage or a petri dish, but behind a lectern up on stage, with some forays from the peanut gallery. But as speaker after speaker came up to that lectern, it became clear that these scientists didn't really want to talk about how to address whatever existential risks to humanity their research might pose. They wanted to talk about their research. Alex Capron, a lawyer and an early pioneer in bioethics who was at the conference, says as a result, the busiest place at the conference was actually the two pay phones. Michael, the Rolling Stone journalist, says just the fact that this conference was happening already seemed to be spreading the information, potentially all around the world. He basically sat in the front row and took pictures of every slide that came up. And he was accompanied by a very slick young man who spoke perfect English and who was immediately identified as KGB. That part's unconfirmed. But it was becoming clear to those in the room, like Michael, that the more they talked about this cool science, the slimmer the odds were that they'd actually enact any safeguards. Days went on. There was more of a leaning toward this is getting too complicated and potentially destructive of the research path. Maybe this whole thing was going to be a mistake. We asked Alex Capron, the lawyer, about this. It is slightly ironic that this meeting that was convened to sort of slow things down or at least do things safely was going to accelerate the whole thing, right? Kind of a super spreader event. Yes. Yes. That was the irony. And my job was to bring the meeting back to how do we behave responsibly. Right. Alex wasn't just some brando lawyer invited to the conference. He was an expert in medical ethics and the law. And he was invited with a very specific task in mind. On the final evening of the conference, the night before organizers planned to hold a vote on how to move forward, he and a couple other lawyers were going to talk about the law and liability. So you were there to scare the living daylights out of these nerds? Yeah. Did you say nerds? I'm not. I don't adopt that phrase. Don't paraphrase me as saying that. These cutting edge scientists. Finally, it was Alex's time to shine. Alex Capron, the next speaker. The organizers were counting on Alex to shift the tide. He walked up onto the stage with some pen and paper notes and started talking about the risks. If and when dangerous vectors escape from control. But Alex didn't just talk about the health risks to the world or the risks for lab workers. He and the other lawyers who spoke that evening also highlighted the legal risks for all the people in the room, the scientists leading those labs. Freedom of thought does not encompass freedom to cause physical injury to others. He then moved to regulation. He said, whether you like it or not, you're going to be regulated. So decide now. Do you want to make the rules for yourselves or do you want other people to make the rules like Congress people who can't tell reticulum from their own rectum? It is the right of the public through the legislature to reach erroneous decisions. Alex finished his speech. And story goes, he and the other lawyers who spoke on that final evening did manage to convince the crowd that this was, in fact, not just a collective action problem. It could also become an individual problem for the people in that room if they were held liable. Yeah, the conference organizers told them that they were happy. Thank you. You got their attention. You succeeded in the mission of scaring the living daylights out of them. Yeah. So now we face the challenge of coming up with something that we can sell. Something they could sell, meaning guardrails that everyone, including the naysayers, could agree to. Michael says if the intention of Alex's talk was to put a fire under everyone's reticulum, it totally worked. That night, there was several groups of people who stayed up most of the night working on the final rules that would be voted on the next day. What happened the next day and what it can teach us about this current march towards either nirvana or extinction? That is after the break. Okay, now, obviously, we're talking about the Asilomar Let's Slow It Down conference because there are a lot of similarities to what's happening with AI right now. People asking, is this new technology so dangerous that we need to stop it or slow it down? And if so, what are some guardrails that everyone would agree to? follow. But there are some fundamental differences between the AI race and the recombinant DNA race. The biggest one? Maybe money. There are trillions of dollars and arguably the entire American economy riding on the AI race. Slowing that down is a tough pill to swallow. Another difference, the pace. Things are moving so fast, these companies cannot blink so much as pause for fear that they'll get left behind. Plus, it's not just companies in the U.S., it's also China in this race, which adds a whole different dimension. So some huge differences, but also maybe, hopefully, some lessons from how that final day of the conference at Asilomar played out. Right. After that all-nighter where the organizing committee hammered out the proposed guardrails, there were only a few hours left before everyone got kicked out of the conference center and had to catch their flights. The hope was that before that, they were going to come to a consensus, all join hands, you know, kumbaya. But there had been dissent all along. This is Mr. And Watson had agreed to the temporary moratorium before. But by the conference, he had changed his tune. Now he was against basically any guardrail. The dangers that are involved are probably no greater than that faced in any hospital where you just sort of get to live with it. And in this case, I think we're going to have to live with it, too, if we want to be in this field. He said, yeah, if we're stupid, we might get sued individually. That's all the policing we need. That sounds very negative and right wing, but I don't see any other way of proceeding. Michael Rogers, the journalist with the leather vest, says some of the loudest voices in the room agree. Let us do science uninhibited. It seemed to me that this might end up doing nothing and that that would be a real disaster in terms of the public. Bunch of scientists gather together. To make rules governing their research and they come home with nothing. What would that mean? That would, first of all, suggest that science shouldn't self-regulate, that if there was enough danger for top researchers to ask for a moratorium, then everyone gets together and says, oh, never mind. That's that really brings into question the whole process. The organizing committee, as mentioned, had a document that they'd worked on all night, which laid out some guardrails for how to lift the moratorium and move on as safely as possible. Guardrail number one, biosecurity measures. They were going to start categorizing experiments based on risk and the risky experiments would have to take place in the most biosecure labs. Guardrail number two was a plan for biological containment. For example, rather than using E. coli that could thrive in any street food, either is good. Scientists would have to use strains that would not survive in the wild. For a show of hands who now support somehow maybe it was the deadline. Maybe everyone wanted to get back to their labs. Those who were opposed to maybe it was the last minute changes they put in to appease the critics. There are substantial agreement here. They successfully self-regulated. It wasn't unanimous, no consensus, but all but a few voted for the guardrails. And part of the agreement included a plan for enforcement, how to make sure everyone would actually comply. And this may be a key takeaway for our tech barons of today, if you're still out there. The scientists at Asilomar self-regulated, but they did not self-police. They handed the policing over to the government. In the US, they gave that power to the National Institutes of Health, who importantly controlled a huge portion of research funding in the US. So even the people who weren't in the room when this agreement was made, like Janet Mertz, whose research started all of this, they had to get on the same page. NIH could say, if you don't follow the guidelines, you don't get our funding. And so out of Asilomar, everyone said, okay, we're on a level playing field. We can't say that with AI because they're going to do IPOs for funding. Right. It's a little different. It's hard to tie anything to money with the AI companies because they don't lack for funding. They don't need the government's money. But there are some proposals for how to regulate AI by at least aligning the incentives. Yeah. There's a camp that says you could impose new taxes on AI companies, and that alone would slow things down. Either tax them according to some unit of work, or according to how much human labor they might be replacing. Basically, you make it more expensive to do things that might cause social harm. There are also technical and governance proposals out there, like building kill switches into hardware, or letting third-party safety automobiles do all the work. But there are also technical and governance proposals out there, like building kill switches into hardware, or letting third-party safety automobiles do all the work. But models before they're released, to name a few. And if the AI bros do want to meet up and talk through all these proposals, try to hash things out, we've come away from our magic school bus bioethics tour with a few more tips and tricks for how they might run such a summit. Number one, limit the scope. At Asilomar, they didn't try to solve all the ethical problems their new technology posed. Like, was it even ethical to do gene editing in the first place? They just focused on safety. Two, when you're trying to talk about this and come to a consensus, keep it in the room. Michael says the two pay phones they had at the conference, those two lines out of the rest of the world, seemed like a danger to their success. The lack of communication, the no tweeting, no blogs, that was crucial. If they'd been tweeting and slacking their CFOs, things could have easily gotten derailed. I think there would have been a lot more confusion. And opposition would have risen up very quickly, probably during the conference itself. Lesson three is kind of weird. Invite the lawyers. That's not the lesson. The lesson is with any collective action problem, it helps to remind people of the personal stakes, personal responsibility, personal liability. That's how you get people to focus. Final takeaway, and this one is maybe more of a warning, there will be losers. This is the hard to stomach truth about most any solution to a collective action problem. Some people will inevitably get the short end of the stick. Did slowing down this research hurt your career? It hurt my career enormously. It hurt lots of people's careers. I had this huge head start over everybody that I lost. I'm probably a little bit bitter about it. Yeah. Does it feel like you sort of paid the price for the rest of the field to take this moment and slow things down and figure out whether or not this kind of path of research was potentially dangerous for the rest of the world? I might have paid the biggest price. If there hadn't been a moratorium, I would have been the first one to clone. Do you feel like it was worth the cost to you? It would have been nice if I hadn't paid such a big cost, but even though I paid a heavy price, I felt it was worth it. For everybody else in the world. Yeah. So, Elon, Dario, Sam, if you guys are still listening, don't forget Mark. Oh, and Mark. And Mark. If you're still out there, still listening, we have a little proposal. Yeah. If you want to meet up in the Monterey Bay, talk these things out. We're happy to help coordinate. Just let us know. Call it A.I. Sillimar. Oh-ho-ho! And while we're on the subject of conferences, if you happen to be anywhere in or near Ireland on November 6th through 8th, Planet Money is going to be there for the Kilkenomics Comedy and Economics Festival. We'll be doing a couple of live shows. Find tickets at the link in the show notes or at kilkenomics.com. Just Google it. It'll autocorrect. Today's episode was produced by Vito Emanuel and edited by Marianne McKeown, who was engineered by Travis Hagen and fact-checked by Ciaro Juarez. Alex Goldmark is our executive producer. Special thanks to the great Dan Gurma, David Marku from the University of Wisconsin-Madison. Go Badgers! And the folks who make the Distillations podcast from the Science History Institute. They have a great audio doc about Asilomar if you're looking for more depth. I'm Nick Fountain. I'm Alexey Horowitz-Gazey. This is NPR. Thanks for listening.