Speaker 1One thing that was reported was the UAE getting hundreds of thousands of next generation Nvidia chips. And that is just an unbelievable amount of computing power. A big thing that the companies will say is if the U.S. doesn't sell it, then China will sell it. So China is waiting, you know, outside the door. If the deal doesn't go through with the U.S., they'll just offer the exact same deal. But China can't even make the chips. Exactly. Exactly. So it's totally disconnected from the reality of what they can do. The UAE is an autocratic country. Political parties are banned. They do, you know, mass trials of dissidents. They persecute the families of dissidents. The economy runs on immigrant labor. In the worst cases, they are essentially forced labor. There's pretty strict rules around, quite strict rules around what the media can say about the royal family. It's a hereditary autocracy with a, you know, royal family that is going to stay in
Speaker 2power. So when I think one of these deals was announced for the data center in the UAE, it was a collaboration, I guess, with OpenAI was involved. They put out a press release saying this is a huge win for democracy. Is this just kind of boundless cynicism or is there something else going on?
Speaker 1It certainly looks cynical to me.
Speaker 2Today, I'm speaking with Helen Toner, the interim executive director of the Center for Security and Emerging Technology, or CSET, based here in Washington, D.C. CSET is one of the most, possibly the most influential think tanks doing analysis of the security implications of advances in artificial intelligence and suggesting potential policy responses. And Helen is also well known for having been one of the board members of the OpenAI nonprofit back in 2023, and having been one of the four board members who voted to remove Sam Altman from his role as CEO. Thanks so much for coming back on the show, Helen. It's great to be back. So you tried to remove Sam Altman from his role as CEO of OpenAI back in 2023. I think people are less confused now than they were back then about why you were interested in doing that or why you and three other board members were interested in doing that now than they were back at that time, when I think people didn't necessarily understand what the motivation was. But I hear from people who, and I imagine you hear from people as well, who say, even if the goal was a noble one, it was handled perhaps somewhat naively or amateurishly. And maybe there wasn't enough forethought given to how staff will react, how investors will react, how the public is going to perceive and understand what's going on. And as a result, you kind of were perhaps caught flat footed and didn't actually manage to remove Altman in the end. I know there's like a lot of confidentiality issues around here, which means that you maybe can't say everything that's on your mind, everything you'd like to say. But yeah, what's your reaction to that?
Speaker 1Yeah. I mean, I don't think I can give a fully satisfactory account of all the decisions we made and why we made them. But two things I will say, one, I totally understand why people come to that conclusion. I really understand why it looks that way from the outside. And I think at this point, I've basically made my peace with the fact that it's going to probably, for people who see it that way, I don't think I'm going to be able to convey sort of why it looks differently from the inside. But what I will say is that, you know, for the board, there have been a lot of, at the time and to some extent since, but especially at the time, you know, a lot of reactions of, oh my God, why didn't the board do X? Why didn't the board think about X? And for pretty much every single X, it was something that we were thinking about. We considered carefully. We had good reasons not to do. And in most cases, I still stand behind those reasons. So, you know, it was a very complicated situation. It was a fast moving situation. I think it's just probably not something that I will ever be able to kind of fully convey my perspective to outsiders, not just because of confidentiality, but also just because there's so many little details of specific things that happened, specific people involved, kind of best guesses and judgments about those and lots of different trade-offs. I mean, I will say for anyone who is sort of interested in this and who hasn't seen, you know, since it happened, hasn't seen some of the more information that came out about kind of how it actually went down. I think that can also help explain some of the decisions that we made. So one place to look there is an interview I did in May of last year on the TED AI show where I talked, you know, wasn't able to share everything, but kind of really tried my best to give a kind of clear description of the basics of what happened. And then there's also been these two books that have come out actually this year, one called The Optimist by Keech Hagee and one called Empire of AI by Karen Howe. And they both include reporting about what happened on the board. I don't want to vouch for every single detail of their reporting, but I think both of them kind of in broad strokes have it just about right. And in particular, Karen's book gets into quite a lot of questions. So I think that's a good place to start. And I think that's a good place to start. And I think that's a good place to start. So for people who are sort of interested and confused and haven't seen that material, that might also kind of help make sense of it.
Speaker 2So the bottom line is, like, maybe you made some mistakes, but if you think it was so obvious what to do, if you think it would have been so easy if only you're in that position, then perhaps you haven't, perhaps you don't fully understand how difficult the position was.
Speaker 1Yeah, I think the big decision, I mean, we made, you know, hundreds of small decisions over the course of that weekend. I think the big decisions I basically stand behind. Yeah. And there's lots of ways that it could have gone worse as well. I guess, yeah, back in,
Speaker 22023, you and the other board members came in for some pretty harsh criticism, I think, particularly from the tech industry, which was really, I guess, blindsided and confused and a bit dismayed by the decision, but by and large. But my impression was that you got a much more sympathetic hearing, I think, in DC among policy folks and also in the press and the media, because they were inclined to, I guess, understand the situation pretty differently. There's maybe a power struggle within a company, although it isn't entirely a company, it's like actually a non-profit. How has that kind of played out for your career over the last couple of years? Do people kind of respect what you tried to do?
Speaker 1Yeah, I definitely see really wide-ranging reactions about what we did. Another big difference I see is between people who have experience in kind of high-profile boardroom decisions and people who don't. But yeah, what you said about kind of tech industry versus more policy folks is definitely true. I think, I mean, to speculate a little bit, I think on this coast, you're in this city, there is maybe, well, for one thing, there's much less of the cult of the founder, which is really incredibly pronounced in Silicon Valley of, you know, the founder or the CEO is a genius and always correct, and the board's job is to sort of stand behind and cheer. And that's just not like a cultural expectation here or an organizational expectation. I think maybe a little bit more inclination to not assume that companies are always doing the right thing, that probably contributes. And yeah, I also think something I really noticed is just different expectations around how much information is being used. And I think that's a really good point. I think is going to be shared out of a sensitive and complicated situation. I think as well, maybe the most important piece about this from my perspective was I was the only person on that board who really had strong professional roots outside of the West Coast. And I think that it's not a coincidence that I've been the person who's been able to sort of talk about this a little more than the others. And I think it contributed a lot to how we were able to make the decision in the first place is, you know, having basically not being vulnerable to having my reputation destroyed in Silicon Valley. I think at this point, I don't know that my, you know, I think my reputation has recovered somewhat, as you said, because of the sort of changes and perceptions of our decision. But I think for many others in and around the company, they are just very vulnerable to sort of perceptions in that one specific community. And so the fact that I had networks and respect and people who know and understand and value my work over here, that weren't going to be subject to that was really important. I mean, I still meet people who know CSET and don't know my name. So they know, you know, they know our organization, they know that we do really high quality work on AI and national security. And that's what they know, which is very different from, you know, on the West Coast, I'm much more likely to meet people who only know about me from the board and have no idea kind of what I spent most of my time doing.
Speaker 2Yeah. So while more people probably have heard of you because of OpenAI, so you were on the board for two years, and it's kind of it's an unpaid role, and I guess a part time role. Yes, definitely. And by far the thing that you've been working the most on since 2019, right, is security, Center for Security and Emerging Technology. What should people know for the purposes of this conversation about what CSET is? And I guess what actually you do?
Speaker 1Yeah, so we were set up, like you said, in 2019. And that was actually last time I came on the show was right as we had gotten set up. And I went back and looked at that interview. And it was interesting, because at the time to sort of describe CSET was basically saying, look, AI, national security, these are big topics, you know, emerging tech and national security, we decided there should be a center like, really important, which at the time was kind of a novel idea. Yeah, right. Exactly. These days, you know, if I, I think, the way to understand CSET is, you know, that is very much still what we do. But to explain a little bit more what makes us special. So we are really focused. We're based within Georgetown University, but we're not academics, you know, we don't do the academic journal thing or don't have tenure track academics. We're very policy focused, very focused on what is going to be relevant and useful for policymakers and other decision makers. And our work, we really strive to be independent, evidence driven and technically informed. So what do you mean by that? Independent, nonpartisan, not coming in with an agenda, not sort of primarily there to do advocacy, but really to just sort of tell it like it is. Evidence driven, data driven where we can be, we have a data science team who do incredible work. We have large data holdings of different kinds. Data on kind of papers and publications, financial flows, job postings. We have an in-house translation team. And so we're really focused on using data and evidence to show what is actually going on out there in the world. And then technically informed, meaning, you know, our work is, we really specialize on a small number of technologies, primarily AI, semiconductors, biotechnology. And that means that we can have people on staff who really understand those technologies. And so we're really focused on that. And so we're really focused on that. And so we're data and China translation capabilities, looking at several thousand contracts from the PLA, the Chinese military on AI and analyzing. In this case, we have more coming out from that data set, but in this case, it was looking at what are the organizations that are working with the PLA and what does that tell us about their China's military civil fusion strategy, which is a strategy they have to try and sort of bring together their public sector defense ecosystem and their private sector. And really interesting findings when you look at this sort of big data set of, I think, almost 3000 tenders around the kinds of new and non-traditional vendors. So universities that haven't traditionally been linked with the defense ecosystem or smaller, newer companies coming up that aren't just the gigantic state-owned enterprises that have traditionally done Chinese defense. So that is the kind of work that we like to do to kind of bring more clarity, shed more light on these issues to help policymakers.
Speaker 2I think one of the big influences that CSAT has had is, I think you were one of the groups that provided a lot of information and analysis data and suggestions that I think culminated in the imposition of export controls on semiconductors and semiconductor manufacturing equipment to China. I guess those, I think that started around 2019 at the very beginning, and then I guess it's gotten like more intense and now it's, I guess, been very topical this year. How do you feel about
Speaker 1that influence with the benefit of hindsight? Yeah. So I guess there's some kind of technical nuances to unpack here. I guess we have time, so I'll unpack them. So CSAT's work, one of the first things that we looked at in 2019 was looking at kind of inputs into AI advancements. And so we did a lot of work on kind of talent. We did some work on data. And then we also looked at, yeah, the compute, the semiconductors as an important component and wanting to understand how does that, you know, how do semiconductors access to semiconductors affect AI progress? And it's really important to distinguish between the two kinds of export, kinds of export controls you mentioned, namely sort of semiconductors, the chips themselves versus semiconductor manufacturing equipment, which is sort of the gigantic pieces of machinery that get put in the fabs, the factories that make the chips. And so our research really focused on the semiconductor manufacturing equipment, that supply chain. And the reason that matters is I think there's actually just a really straightforward, really solid case for why we should be controlling certain kinds of semiconductor manufacturing equipment. So the kind of most famous one for people who follow this space is EUV lithography. So that's extreme ultraviolet lithography. These are the, these machines that are made by this one company in the Netherlands and flown over, you know, to Taiwan and like multiple jumbo jets, because they're these very, very complicated, large machines. And no one else in the world can make them. And there's other examples. So, you know, another sort of choke of this is, you know, the EDA software, electronic design automation, I think is the acronym, which is primarily made by US providers. And the key things about these are, if you're going to control something, you know, if you have, if you're trying to prevent someone from accessing something and you could supply it to them and you want to put a control on it, the question is, what will they do instead? And so if, you know, if you want to prevent a country from having, you know, light bulbs, you're going to have a really hard time because if you prevent them from buying your light bulbs, they're going to be able to buy them anywhere else. So it's kind of pointless. You're just sort of shooting yourself in the foot. And so the key thing about these pieces of semiconductor manufacturing equipment, like lithography machines and EDA software and so on, is if they really are a choke point, meaning there really is only one or a very small number of providers, and it's not going to be easy to sort of start a new company that can replace that, then you can potentially really slow down the China's efforts to build up its own domestic supply chain, which means that they stay dependent on US chips, which I think is just a really good thing. It's strategically very solid. It sort of doesn't actually really antagonize them very much at all or, you know, cause problems for their economy. But it does give the US a significant, and US and allies, a significant strategic advantage. So in 2019, that was what we identified at CSET based on our research, and that has been controlled. It's kind of a shame that semiconductor manufacturing equipment or SME, people in the field will say, those SME controls have kind of like fallen out of the limelight or not been a major focus as there's been so much debate over chips. And so we can talk about chips in a second if you like. But, you know, ideally, the sort of stuff around chips is a little more complicated, a little more trade-offs, a little bit, you know, different theories of change. And ideally, if you were going to dabble in that, you would sort of keep the SME controls as this like very obvious baseline that you're really focusing on. But instead, because of sort of limited resources, limited focus, I think those controls have actually been not implemented particularly well. And so, you know, I think particularly rigorously, while there's been a lot of discussion about the chips instead, which I think is kind of a mistake. You think machines are being kind of smuggled into China? I would have to go look at the details. I think there's like licensing and there have been lots of licenses granted that colleagues of mine who are deeper in the space, you know, sort of saying, why are we granting these licenses? Things like that. So it's not necessarily just smuggling, but also a question of focus on, you know, which actors get access to which things.
Speaker 2I'm surprised you said that you thought the export controls weren't antagonizing China. I would have thought they would feel like they've been put in a pretty vulnerable position security-wise, that maybe their economic ambitions are also being somewhat constrained by this. It's just kind of a reasonably hostile move, you would think, to deny them access to these machines, to the chips as well. Am I understanding wrong?
Speaker 1So again, we should separate out the SME from the chips. I think I was saying the SME is not necessarily very antagonizing. I think the chips question is really interesting. I mean, it's important to look at the backdrop. So I do think that the export controls on semiconductors themselves, China did not like that, obviously. The backdrop, though, you know, they already were saying in, you know, at least 2015, if not before, that they were assuming that the U.S. was going to do this kind of thing, that they were assuming that they needed to indigenize, that the U.S. was a hostile actor, that they shouldn't, that you couldn't trust, and that was kind of a good thing. I think that's a good thing. against them. And then in the first Trump administration, there were a bunch of actions around semiconductors and sort of the tech industry more broadly targeting Huawei as sort of the well known case, but there was also ZTE, another company which was targeted beforehand. And so kind of already then in, I think it was 2018, I want to say, that was sort of confirmation for China of what they had already believed, which was that the US was, you know, going to behave in this way and that they had to indigenize and they had to kind of be looking out for themselves. So I think it's a real open question, like what was the marginal effect of the different controls? And so I would say that the marginal effect of the SME controls was probably not very large. The marginal effect of the chip controls may be a little bit more debatable, certainly some marginal effect. But I think sometimes I hear people in this space who are sort of very focused on AI and not focused on the US-China relationship, acting as though that was like, you know, a bolt from the blue, gigantic hostile action, you know, often nothing was going wrong in the relationship. And I think that's not right. I think it's sort of a little bit trickier to say what was the effect. And, you know, notably China's reaction, I mean, there's now been multiple years of sort of a little bit of back and forth, but certainly at the time, the reaction from China was actually not very retaliatory, was not very sort of escalatory, which I know was another thing people were concerned about was, you know, will this create a huge backlash? So yeah, I mean, I don't want to say... I don't want to say it was all kumbaya and no problem whatsoever. But I also think it's possible to overstate.
Speaker 2A strange thing that's happened this year, there was this big debate in the US about whether the US should be selling or allowing more Nvidia chips to be sold to China. And people would argue, it's bad to do that just for the obvious national security reasons. And other people would argue back, no, actually, it's good, because what we want to do is maintain China's dependence on these chips, rather than basically force them and indeed encourage them to develop their own local chips. semiconductor industry and give that industry a big boost. And then recently, China, I think, made the decision to not allow the NVIDIA chips that the US was debating whether or not to allow into China itself. So in a sense, China has imposed ex-force controls on itself, at least on these particular NVIDIA chips. Can it be the case that it's both a smart move for the US strategically to deny China the chips and a good move for China to deny itself the chips? It seems like it's a reasonably zero-sum situation. So how can it be in both of their interests to do it?
Speaker 1Yeah, lots going on here, I think. So one immediate question is, does China really mean it? Or is it an ambit claim? Is it a bargaining position? I think at this point, most countries around the world know that President Trump loves a deal and loves making gigantic opening bids on deals and then retreating to something more reasonable seeming. So I think it's very possible that this is just mostly signaling from the Chinese government saying, you're going to have to beg us to buy your chips and we don't care. And so far, I think the effect of that rhetoric on the US debate does seem to have been, oh, we should just let them buy. If it is a move by China to be allowed to buy the chips, it seems like so far it's working. So that's one possibility. I think, though, this broader thing and also what you said about how the US debate has shifted really shows how there's been very little clarity and very little agreement on what the goal of these export controls is and what the theory of change is. And I think partly that comes from... The fact that this is a complicated technology and relates to AI and AI is general purpose technology with lots of different components. I think partly it comes from this sort of situation in US policy over the last few years, where one of the few things people could agree on was essentially China bad. And so if your policy was like, oh, this is going to, you know, China bad, and therefore we'll do something that will hurt China, then people could kind of get behind that without necessarily needing to have a very clear sort of position on the nuances of the policy goals and how you would tell if they were succeeding. But I do think the fact that, you know, we've gone from, you know, the initial messaging around this was really about China's use of chips for military applications and for human rights abuses. And, you know, astute observers pretty quickly were like, hang on, military applications, a lot of what you need for that is going to be much smaller chips, you know, on board, like you're having something on a drone or on any kind of equipment, you don't use these gigantic data center style chips, you use something much smaller. So if we're primarily targeting the military, this is kind of a weird way to do it. Likewise, human rights abuses, that's often going to be like, you know, image recognition, speech recognition, other things that you don't need these gigantic clusters for. And so then the rhetoric has sort of shifted around to, well, it's actually about sort of China's ability to build very advanced AI systems. I think that's the most sort of sensible or like that, that one actually makes sense. If that's what you're trying to prevent, then preventing large accumulations of the most advanced chips is a good way to do that, or a reasonable thing to try at least. But then now, yeah, the rhetoric has shifted again to, oh no, like what winning means. And, you know, David Sachs, the White House AI and crypto czar has said this very explicitly, what winning against China means is having more market share in chips. And so, you know, as long as we're selling them chips, we're winning. And that's just sort of a very different theory of change from, you know, winning means that we are using the chips domestically to build something that gives us a strategic advantage that they're not able to build, or, you know, that helps our economy because we have this broad base of computing power that they don't have. And so we have some kind of strategic advantage. So there's really, I think, a lot of confusion and a lot of, almost rug pulling over sort of what exactly are we doing here? And how will we know if it's,
Speaker 2if it's working? Yeah, I've heard this a lot this year, that it's really important for the US strategically to get the rest of the world to start using its AI models and to be part of its like AI stack. To me, I guess it's not intuitively obvious that that is a national security priority, or that it really does make a big difference. Whether, you know, someone in Australia or South Africa or Brazil is using chat GPT or whatever the Chinese model might be. And I mean, a cynic might say, well, this is in the AI industry's interest to convince the government that it's really important that they provide support so they can sell their products more and compete for market share. Why wouldn't they try to persuade people of that if they can get away with it? Does this argument make sense to some extent? Or how strong is the argument? And
Speaker 1how should I think about it? Yeah, I think the version of it that makes sense to me and something that colleagues of mine at CSET wrote about is basically as a soft power play. Meaning that there's just going to be lots of probably, like, if you look back at the history of technologies and sort of who's providing what, there's just a lot of cases where you get some kind of broad, diffuse, soft power benefits from being the provider of a key technology. You know, sort of kind of different example would be like Hollywood and American music. And like, why exactly is it good for America that, you know, everyone around the world enjoys Hollywood movies and listens to American music? Sort of like kind of hard to put your finger on it. But I think it is clearly good. That's, you know, one of the canonical soft power examples. Likewise, I think it does seem right to me that it's better for, going to be better for just a wide range of reasons in terms of sort of influence and standard settings and expectations and relationships if American AI is used around the world. But sort of big caveat there is, I do think that, I think if that is your focus, then it doesn't necessarily make sense to focus on these sort of frontier systems, meaning computing clusters that have, you know, hundreds of thousands of the most advanced chips and are being used to train or run the very, very most advanced models. I think if you actually talk to people who are going around and trying to, you know, sell an AI stack or, you know, get government partnerships with AI stacks, often what people want is sort of more of a turnkey solution of like, okay, well, what is this going to do for me? And there you often don't necessarily need huge amounts of computing power. You don't necessarily need the very most advanced models. Instead, maybe you need some kind of support to figure out how to use a model for some particular use case that's interesting to you or to use it, you know, if we're talking government usage or sort of large enterprise use, the support you maybe need is how does this fit in with our existing kind of IT procurement policies? And like, how do we, what is the UX? How should we build it into our existing offerings for our staff? Or, you know, if it's a government, how should we fit this into our sort of online or digital government initiatives? And that's much more the kind of thing where you need sort of embedded software engineers to go in and understand the problem and sort of help make progress, as opposed to the kind of software engineers that are sort of the kind of thing where you really need hundreds of thousands of chips. So I think the argument is basically reasonable. And I understand why people coming out of the tech world sort of want to say, let's do this full stack, American AI stack approach. But I think it's sometimes used for this, like, and therefore we should let them build their own world-class supercomputers, which I don't think follows. Yeah.
Speaker 2Something that's been even more confusing about this argument to me is I think people will make that argument and then say, and so it's very important that we have leading open source AI developed in the US. But I think if you have a brilliant, like, if meta open source is a really good model, and so anyone anywhere in the world can use it and run it on their own equipment, then they can just fine tune it to have whatever kind of properties, whatever personality, whatever sort of values they like, and then just run it locally on some equipment that's not in the United States. How does this really help the US's national strategic, like national security goals?
Speaker 1Yeah, I don't know. Again, I think from a soft power perspective, it maybe makes more sense. And I also think that often people in an open source ecosystem, people will kind of engage or go back and forth. They don't just sort of grab the open model and then run away and hide in a cave and, you know, do their own thing with it. Instead, there's often more of a rich kind of back and forth and engagement. So I tend to think I am pretty in favor of there being more of an effort to produce really high quality open source models. I signed on to this project called the Atom Project, the American Truly Open Models Project by Nathan Lambert, who's a researcher at AI2 in Seattle. And, you know, I think it's really important that we have a really good relationship with our customers. And the key thing in my mind is actually separating out two different questions, which is, should we be trying to have leading open models versus should we be trying to open source or, you know, open weight the most frontier models? And I think those should be separate questions. So I think that it does make sense for US companies to be trying to offer models that are at least sort of roughly as good as the best open models openly. Um, but I don't think that necessarily means they should be, you know, really racing to open source their kind of, their very best models, um, which I think has
Speaker 2Just because it creates security risk?
Speaker 1Yeah. Yeah. I think my sort of overall stance on open source is to think that, you know, starting point, open source is great. Sharing everything as widely as possible is great and has lots of diffuse benefits that are kind of, um, hard to count up, but really, really meaningful in terms of accountability and access and, um, broad benefit. Um, but the main exception in my mind is these frontier models, the most advanced models, because we just understand them least well, and where they come from. Um, so I think it's really, really important to think about that. And we're least well positioned to sort of mitigate any new risks that they pose. Um, and so just having some amount of, um, the phrase that's sort of gotten a little bit established is precautionary friction, meaning having a little bit of lack, having, is it six months, 12 months, 18 months to test and understand those systems to figure out, you know, what can they do if you really push them to their limits? What kind of unexpected, um, uh, effects emerge and having that happen in a time where you can still sort of pull them back from the API, do some fine tuning, put some new safeguards on. Rather than having it be fully open.
Speaker 2The rhetoric in DC is really around the US being in a race to develop AI and AGI against China. Is China actually racing to build AGI?
Speaker 1It's a great question. Um, we actually recently put out a, a short post on this, um, because there've been these two recent op-eds by one by Eric Schmidt, one by Jack Shanahan, both very smart and well-informed people, um, saying, you know, the US is doing it wrong because we're focusing on AGI and China is doing it right. Cause they're focusing on kind of applications and diffusion. And, um, we put out a post into our researchers, um, Bill Hannis and Tui Mei Chang, who are two of our sort of most senior China researchers at CSET put out a post saying like, well, that's actually not right. Actually, China is doing all of the above. So it's definitely true that China's big push right now, or one of their big government pushes is a, it's called the AI plus plan. Um, and it's AI plus because it's, you know, AI plus some other sector. So AI plus manufacturing, AI plus healthcare, you know, looking for those intersections and those, those useful applications, that is a big focus, but you know, they can walk and chew gum at the same time. And they are continuing to also emphasize, um, AGI, general purpose AI. Um, so some wrinkles here. Um, one is that in Chinese, the word for AGI is the same as the word for general purpose AI. So it's just, it means like general use. It's very unhelpful. So in the US, you know, in, or in the West or in, um, sort of Anglophone discussions, um, we have for a long, long time talked about AGI. And then after chat GPT came out, this sort of idea of general purpose AI kind of got established, which is this more sort of mundane thing that, you know, can be used for many things, but it's not the full on, you know, human level or, you know, building towards super intelligence in Chinese. It's the same word. So it's a little bit unclear. Um, that being said, they do sometimes just use the English letters AGI. Um, and they do sometimes talk about super intelligence. Um, so I think it's very clear that they are right now pushing for both. Um, something that's less clear, um, is the question of, you know, what's the best way to get people to use AGI? Sort of how, uh, I don't know of a better term than like AGI pilled, um, Chinese leadership is meaning like, um, you know, They're really all in on AGI is going to be an enormous deal and it's going to happen soon and the way to do it is to scale up. And the best thing that I've seen on this is actually from Jordan Schneider at China Talk, just has a debate sort of posted on his sub stack. I think if you say like China Talk AGI debate or something, it'll probably come up on Google. And it's basically formulated as a debate between a believer and a skeptic, the believer saying obviously China is very AGI pilled and like here's all the evidence and the skeptic saying obviously they're not. I don't think the – I tend to come down with the skeptic on that debate, meaning I don't see the evidence that they are really gunning for sort of AGI or super intelligence in the same way that sort of the leading U.S. companies are or the leading Chinese companies. I think DeepSeek is certainly very AGI pilled as a company. So there's sort of degrees of what you could mean for is China racing towards AGI. I also don't think the U.S. government right now seems very AGI pilled. And I think that could be for good reasons. Like I think there's a lot of assumptions bundled into what that means that may actually not be correct. So I don't think it's – I don't want to say they're not AGI pilled and that's a mistake. I think there's reasons to have lots of space for different views on that.
Speaker 2So DC has a pretty hawkish attitude towards China, I would say. Do you think this is kind of appropriate given how China has behaved? I think my worry is it's become such an easy thing to say, oh, we've got to do X or Y, you know, things that I liked anyway because of China. And it's kind of uncomfortable, I think, for people to push back and say, is it really necessary to be this antagonistic towards China? When something becomes that degree of conventional wisdom and so easy to say and so awkward to oppose, it can just get – it basically never gets challenged and just gets accepted by everyone despite the arguments perhaps being weaker than they seem. What do you make of that?
Speaker 1Yeah, I mean, I think groupthink is bad. I think people feeling that they need to self-censor is bad. I have good news for you. I think over the past – even just the past three to six months, there, this has been softening a bit. I think President Trump's stance on China is pretty unclear, shall we say, and or maybe somewhat fluid. And that has created some space, I think. Also, certainly in the AI side, you know, the top folks advising him on AI seem much more interested in commercial engagement with China, selling U.S. chips to China. In a way that is not as compatible with sort of the traditionally China hawk stances. So, yeah, I mean, I think it's really valuable if there can actually be many voices in the debate and many, many perspectives. And I think there's been a little bit of a lack over the last few years maybe of creative thinking about what are all the different ways that we could relate to China. There's sort of, you know, it all gets collapsed into hawk versus dove, which is maybe sort of a callback to like 20, 30 years ago, which is sort of a callback to like 20, 30 years ago, which is sort of a callback to like 20, 30 years ago, which is sort of a callback to like 20, 30 years ago, of do we assume that China is on this path to being a responsible stakeholder and a more politically liberal, nice, friendly country? Or do we assume that they're on track for war with the U.S.? And then we have to be maximally hawkish. And, of course, there's a very wide range of possibilities in between there. So I think it's valuable to be able to consider a wider range of options.
Speaker 2Something I've been a bit confused by is it doesn't seem like there's any direct diplomacy going on. So I think it's valuable to be able to consider a wider range of options.
Speaker 1I think you're right, especially at what gets called the track one level, meaning official government-to-government talks. I think there's a few reasons for it. One basic reason is, one sort of AI-specific reason is, I don't think that the groundwork is there in terms of concern about this as a problem or concern about this as a major problem to prioritize. So, you know, what is AGI? What is superintelligence? Are there things we could build ever? Are there things we might build soon? Would that be good? Or bad? And for whom? I think there's just very little, especially within, at the government level, very little agreement on those questions. In contrast to, I think, you know, perspectives in the AI safety community, for example, which is... Or the companies. Or many of the companies, that's right. Which is, well, in the companies, it's, you know, we can build this, we will build it soon. And then in the AI safety community, and it's going to be really bad. And so we should be talking about it. But I think that is a relatively insular set of beliefs still. That's kind of the AI-specific reason that the sort of groundwork is not there. I think it's also really important to look at the broader relationship as well and understand that from the perspective of a diplomat or certainly of, you know, a president or a chairman, this has to slot in with everything else that is going on in the relationship. And so what else is going on in the relationship? Well, one thing is the US and Chinese governments are barely talking at all. So, you know, over the last couple of years, there have been some direct country-to-country talks on some small number of issues, but they have also often been completely suspended. So for a period, I think it was after Nancy Pelosi went to Taiwan, I think just basically all talks were suspended. I mean, isn't it just crazy?
Speaker 2It's like, it's mind-blowing to me. It's just like the two most powerful countries in the world. They've got so many things to talk about.
Speaker 1Yeah. I mean, I think generally my understanding of this is that usually the US is the one that has more appetite to talk. And so China is using, and China knows that. And so that becomes a bargaining chip for China to say, we don't want to talk to you. We're not going to do these military-to-military talks about extremely, you know, sensitive, important issues because we're mad. And if you want us to do them, then you have to give us something in return to come back and join these talks, right?
Speaker 2And I guess you don't want to give in to that.
Speaker 1Well, yeah. I mean, if the whole point is that it's mutually beneficial, then we don't want to be acting as though we're making a concession by, you know, or that they're making a concession by letting us talk. Like that's sort of, you know, there's a lot of context here and a lot of baggage. A couple of other pieces of context and baggage I would give. One is, I think, a fair amount of skepticism among US diplomats. You know, diplomats in general are usually pro-engagement, pro-negotiation, pro-conversation. That's why they become diplomats. I hear among US diplomats, a lot of skepticism about the value of that with China based on their track record of what happens when we do that. A sort of important example is in 2015, President Obama and Chairman Xi had a long discussion about this problem of China spying on US companies and sort of stealing trade secrets from US companies, which is different from, so there's a long established countries spy on each other for sort of strategic reasons. But China was doing something different, namely, corporate espionage, where they kind of take trade secrets, benefit economically, you know, hand things to their companies. Very much not okay, very much out of the norm internationally. In 2015, the end of a long series of discussions, there was a big deal signed between Obama and Xi saying that China was going to stop doing that. And consensus is that they basically started a few months later again. So they stopped very, very briefly and then restarted. That's sort of one emblematic example that's in cyberspace. So a lot of the people who do AI stuff, you know, have previously worked in cyberspace. That's a very salient example for them. So it's, there's a lot of, you know, there's also skepticism about, you know, are they a good negotiating partner? And another example I would add there is, you know, the big example of track one government to government talks on AI was in Geneva last year, this sort of initial foray into this conversation. My sense is that that didn't go terribly, but it also didn't go great. And part of the reason, which again, is sort of emblematic of US-China negotiations more generally, part of the reason was the US sent over some really technical, well-informed, you know, some of their best AI policies to people to have like quite in-depth conversations. And the Chinese sent their America specialists. So they're like US diplomats who knew almost nothing about AI, were just specialists in trying to like, you know, handle the Americans that, you know, the term that sometimes gets used as they sent their barbarian handlers, meaning like they specialize in going out there and, you know, playing nice with the foreigners. And so that again, just, you know, again, I don't think that dialogue went terribly, but it wasn't a good start. And again, it sort of suggests that the groundwork is not there in terms of this actually being a priority, that the Chinese government actually wants in contrast to, you know, the default explanation for why would China agree to talks is because it makes them look like a great power. And, you know, they're up there with the US having bilateral talks on AI. Doesn't that show that they're doing so well at AI? And so that, you know, is also a motivation for them to talk. And if that's why they're there, then it's really not going to be very productive.
Speaker 2I mean, they are a great power and they are doing pretty well. I mean, sure. Yeah.
Speaker 1But if the reason they want to be there is to be able to brag about it, then, you know, you're not going to make much progress on reducing risks.
Speaker 2So you would say it's, it's mostly on China that they haven't, that there aren't more negotiations along these lines. So like some shared responsibility, but like more on China than the US.
Speaker 1I think there's plenty, plenty of fault on China. Yes.
Speaker 2Yeah. I guess let the record show that I'm disappointed in China over that. So you can tell them I'm, I'm, I'm frustrated. Is the US on track to stop its best AI models being snatched by China at the last minute to, you know, have the weights exfiltrated and then just used by the Chinese military?
Speaker 1No.
Speaker 2What should it, what should it do that it's not doing?
Speaker 1I mean, I think one, I think the best hope right now is that to the extent that China perceives the gap between the best open source models and the best closed models to be small, they might just not invest in trying to steal steel weights or steel models. And I think that is a pretty widespread perspective right now. So, so maybe they just won't bother basically. I think that's the best hope. But I don't know, man, cybersecurity is really hard. But I think, isn't there a bit of a contradiction if you're saying we want to,
Speaker 2we're kind of like racing with China. We're trying to get to AGI first, but also it's all going to be so close together. They're not even going to bother to steal the thing that we're making, that we're putting so much effort into. How can these things coexist?
Speaker 1Yeah. I mean, I don't think it's that clear that we're racing with China towards AGI. I don't think it's... I think there's sometimes a set of background assumptions here around even the language of a race. I'm perfectly happy to say that we're competing with China on AI. The thing with a race is there's a finish line and whoever crosses the finish line first wins for real. And if you cross the finish line second, then that's useless and it doesn't matter. And I think sometimes in some AI circles, there's an assumption that AI or AGI really is a race with a clear finish line. And the finish line is whoever builds self-improving AI first. Because then if it's true that once you get to a certain level of AI, then your AI can improve itself or can improve the next generation and you have this kind of compounding improvement, then that could genuinely be a situation where whoever gets to a certain point first then ultimately wins. I don't think it's at all clear that that is actually what it's going to look like because the systems get more and more advanced. They're used in more and more ways. They're sort of diffused through multiple different applications. And in that case, I think we're sort of in this state of ongoing competition with China, but not necessarily a heated race where whoever is like a hair ahead at the very end at the finish line ultimately wins the future or something. Yeah, yeah, yeah. So I think the shape of the competition is actually pretty unclear. And when people treat it as though it is very obviously just this sort of winner-take-all race, I think that is a pretty risky point. Yeah, yeah, yeah. Proposition, because it sort of implies that certain kinds of trade-offs and certain kinds of decisions are obviously a good idea, when in fact, I think it's not at all clear.
Speaker 2So two years ago, my memory is that the attitude was, we've got to tighten up security. We've got to make sure that they can't steal the weights. And it sounds like you're saying there's been a bit of a shift in attitude that people are more fatalistic now.
Speaker 1Well, I mean, I think even at the time, the trouble is that stopping a sophisticated state actor from stealing anything online is really, really hard, or not even online. You know, any kind of digital, infiltrating any kind of digital system, really, really hard. And so I think, you know, the sort of best known work on this is by Rand. They have their sort of security levels that they came up with. I do think that there's plenty of, it's sort of very clear that we should be trying to make it harder for a wider range of actors to steal advanced AI models or otherwise, you know, algorithmic secrets or other proprietary information. Because if you are just nihilistic about it, then maybe it's not just China, but it's also North Korea, and it's also a terrorist group, or it's also, you know, a disaffected young person who uses, you know, not this generation of ChatGPT, but, you know, two generations from now, ChatGPT to carry off a somewhat sophisticated cyber attack. So I do think we should be very much focusing on improving security, but I also think it is going to be a really, really big lift to try and actually make these systems resistant to sophisticated state-based hacking attempts. If that's the case, it does
Speaker 2seem like it really undermines the idea that we have to train the front, the best model in the U.S., because if it matters, if it actually does provide a big strategic advantage, we can fully expect that China will have it pretty soon after it's trained, or we can think that that's pretty likely. I mean, the thing that would remain is you want to ensure that there's more compute in the U.S., that China has kind of denied the ability to do massive amounts of inference, even if they can steal the weights. That could still provide a strategic advantage, but they need to train the thing urgently to get ahead. That argument feels a lot weaker.
Speaker 1Yeah. Again, I think it's this question of like, what is the shape of the competition or what are the sort of the parameters here? And I, maybe another way to say it is, I think if it is purely about who has the most advanced model, then certainly the fact that China will probably be able to steal the most advanced model makes it less important to sort of where it is developed or makes it less beneficial to be racing to develop it first. But as you say, there's also a question of sort of deployment. There's also maybe a question of, you know, if part of why you want the most advanced models to be able to build more advanced models, then maybe you need the computing base for that. And so maybe they could, I think this is reflected in some of the compute modeling in the AI 2027 scenario, for example, it's like China in that scenario, China steals a model, but then they can't actually use it to make as much progress because they don't have as much compute to make the progress. Again, that's baking in a bunch of assumptions about what will progress look like and what kind of advantage will accrue to which kinds of actors with what kind of resources. So I don't think it's like a knockdown argument to say that because China can steal the models, therefore we shouldn't be trying to have the best models. But I do think it should at least temper that claim or should be always in the foreground about to what extent does it make sense to move fast? Especially if we're saying, well, we have to move fast, even if it's risky, because we have to get there first, then it's sort of like, well, which risks and how are you weighing them against the possibility that, you know, your model will
Speaker 2just get exfiltrated? A few years ago, you wrote that you thought China was two to three years behind the frontier on AI. I guess that feels like a lot now. I imagine that the number is lower. What
Speaker 1would you say it is now? Yeah. So this was two years ago at this point in a piece in Foreign Affairs. And the point of that piece was actually to say that we shouldn't use, well, you know, we shouldn't use the concern that China will race ahead as a reason not to regulate our AI sector. I still very much stand behind that. And there's sort of a bunch of other things in the piece that kind of back that up. But yes, the estimate we used in there, which was actually coming from some Chinese investors and folks over there, was two to three years. I think that's clearly not right anymore. At the same time, I mean, so DeepSeek had its big moment in January. I mean, I don't think it's going to happen again. I don't think it's going to happen again. I don't think it's going to happen again. I don't think it's going to happen again. I don't think it's going to happen again. I don't think it's going to happen again. I don't think it's going to happen again. I don't think it's going to happen again. I don't think it's going to happen again. I don't think it's going to happen again. I don't think it's going to happen again. I don't think it's going to happen again. I don't think it's going to happen again. I don't think it's going to happen again. And in the wake of that, I saw some people pointing back at this piece and being like, oh my God, two to three years, what a joke. Like they're right exactly on our heels. There's no gap whatsoever. And I think that is also overstating the case. So I think the smallest gap that we have observed is three months, which is the time between when OpenAI put out O1, its first reasoning model, and when DeepSeek released R1, its first reasoning model. That was very impressive. Like I was surprised that was really well done by DeepSeek. But Miles Brundage, for example, who's a former OpenAI researcher at the time said, look, the way that this works is this is a new paradigm, this reasoning paradigm. And right now they're at the very low compute sort of earliest stages of it. This is going to be the easiest instance to recreate this sort of O1 level, right? And since then we've seen OpenAI scale that approach to create O3 and O4 mini and I think also O4, but I'm not sure what the, is O4 now GPT-5 thinking, maybe unclear. And China has, or DeepSeek, which is, you know, I think essentially the leader in this, in the reasoning space, maybe Alibaba's Quent series has some, something that could compete. They, to the best of my knowledge, they have not previewed anything as good as O3, which confusingly is the second generation of the OpenAI models because O2 is a telecommunications company. They didn't want a trademark dispute. So OpenAI put out their second generation. They previewed it in December and they released it in January. And we're now in September when we're recording this. And as far as I know, no Chinese company has previewed a similarly sophisticated model, which means where now it's something like nine month gap, and that might be longer depending on how long it takes them. And also it's not clear how to account for the O4, which is the third generation OpenAI model, which certainly they don't seem to be competing with. So it's all very murky. It's hard to put an exact timeline on it. I've been saying something more like six to 12 months now. I could imagine that it's going to be more than six to 12 months. I could imagine pushing that out longer depending on how long it takes to have that sort of competitive with O3 performance. And then it's also hard to compare because OpenAI is releasing this as a product, as a sort of full system. So O3, for example, is well known for being really good at tool use. And DeepSeek or Alibaba would put out an open source model, which is a little bit different. So then how do you think about the tool use? So it's very hard to compare. But the short answer would be right now, I would say something like six to 12 months, probably.
Speaker 2It's years ahead of the curve, I would say, in recognizing the national security implications of AI advances. And I guess probably years ahead of the curve in realizing the export controls were going to be a huge deal. Is there anything you're doing now that could be the next thing where that C-set is ahead of the curve where other people haven't appreciated just how important some issue
Speaker 1is? We haven't actually spun up work on this yet, but the one that comes to mind for me is robotics and advanced manufacturing, which I think is sort of sometimes treated as two separate topics, but I think is very closely related, which is basically producing really sophisticated industrial machinery at scale, which could include robots or could include other things. Yeah, this is something I want to learn more about. I want to look into more. I think sometimes the conversation about manufacturing in the US can become too much about jobs, about manufacturing jobs. And that sort of neglects the strategic importance of having really sophisticated, high volume manufacturing capabilities here from a strategic perspective of what we can actually produce domestically or, you know, sort of among our allies and partners. So I think, I don't know, I think that I've been thinking about is this, I forget where I got this term, it's not mine, this idea of an industrial explosion, meaning if AI is actually succeeding, will we see a huge build out of energy, a huge build out of data centers, a huge build out of increasingly advanced sort of robotics and the manufacturing you need to produce them. And I think right now, China is sort of really well poised to make the most of that if things do go in that direction. And so I think some interesting questions around what does the trajectory of that technology, that build out look like, and what would it look like for the US to position itself better?
Speaker 2Despite being pretty bullish about AI, the Trump administration has done a bunch of stuff that I think is kind of inconsistent with wanting to win any sort of AI race. I guess it's made it more difficult for people to immigrate to the US, including skilled workers, and generally just seems kind of hostile to scientists and students coming over. It's been negative on rollout of energy generation, at least at renewable energy generation. I guess the tariffs have made it more difficult for the manufacturing sector in some ways, because it increases the cost of industrial inputs, basically, of the machinery that you might import from overseas. I guess the I guess they had that raid on the Hyundai battery factory. I think like Trump then said that he actually didn't support that. So maybe they kind of rolled that one back. But it feels like there's a bit of a contradiction here. There's a bunch of stuff that you probably would be doing, like encouraging high-skill immigration if you wanted to be, I guess, as far ahead of China as you possibly could be. Things that aren't happening, indeed, it's kind of going the other way. How can you make sense of that?
Speaker 1I think I make sense of it by there being sort of different factions inside the Trump administration, different specific officials with different agendas and different priorities, and them not necessarily coming together into a coherent policy vision. So, for example, I mean, on immigration, that was, you know, one of the topics CSET looked at earliest. And when we were in the space in 2019, there was sort of this common wisdom that high-skill immigration was good economically because it benefited U.S. companies and, you know, helped the economy grow. But it was only a downside. From a national security perspective, because people could, you know, leak information, steal information, or at a minimum, just get educated here and then go back to their home countries and benefit their home countries. And so some of our earliest work was on understanding actually the national security benefits of having high-skill immigrants here, including just the fact that, you know, the U.S.'s high-tech ecosystem is so driven by immigrants. You know, depending on which numbers you're looking at there, you know, at a minimum, something like 30 or 40 percent, sometimes up to, you know, well over half of any given pool, I think. Something like, you know, half of the founders of top startups are foreign-born, things like that. And that makes sense if you look at, you know, if the U.S. is going to compete internationally, we just have a much smaller pool of domestic workers than somewhere like China or somewhere like India. So the fact that we can import them and that we can draw the best talent to this country is a huge asymmetric advantage. But that is just obviously, you know, in contrast with the Trump administration and sort of the Trump, you know, the MAGA movement's perspective on immigration. And so I think that sort of policy wonky set of considerations around pros and cons just gets lost in the sort of broader push to be anti-immigrant. So I do think that, you know, among the policy actions that the Trump administration has taken, the many things that are going to deter high-skill immigrants from coming here and, you know, are up there with the cuts to science funding as some of the most damaging to U.S. competitiveness. Just in terms of being technologically as sophisticated as this country has been in the past. And I don't think that – my understanding of it is not that there is a clear, well-thought-through strategy that explains why that is. It's just these sort of different components of the coalition sort of doing their own thing.
Speaker 2The Center for New American Security and Leonard Haim, the compute governance expert, who was actually on the show two years ago, they've recently been pushing pretty hard this line that the U.S. should be – it should be renting compute to other countries rather than selling the chips itself. Basically just because there's, I guess, a clear security win that if you're just renting access to them, if people are just basically buying access to the compute on the cloud, then you could always, like, cut them off if they're using it for some nefarious purpose or just against U.S. interests for any reason. Do you think that's a good argument?
Speaker 1I think it's pretty good. I think it comes back to this question of what are your objectives and how do you know if you've achieved them? So I think this is quite good if you are trying to – be able to, you know, continue to profit from Chinese companies using AI and you think you're going to be able to sort of monitor and shut off concerning uses, then I think that's great. If instead you're trying to sort of maximally profit from the Chinese market, then probably you do want to sell instead. And if instead you're trying to maximally limit China's access, then, you know, you don't want to rent. So I tend to think that it's a good way of sort of meeting the tradeoffs involved. But, again, I think this debate is so lacking in agreement on objectives and how you know if your policy is working that it's a little bit hard to know if this will hit the spot for kind of different people involved.
Speaker 2What's the best argument against the rent-don't-sell approach? I mean, I think one thing is, you know, if the U.K. just couldn't buy any NVIDIA chips or, you know, even allied countries of the U.S. couldn't buy any chips, then they would feel in a very vulnerable position, basically, because, like, they could always just get cut off. It's not really a security situation that any country would gladly accept. So I suppose that would create a lot of pressure or a lot of enthusiasm perhaps for just, like, sourcing AI chips from anywhere else, including in particular, I guess, China would be the main alternative source. I guess that could be a negative side effect that it might have.
Speaker 1Yeah, I mean, I think with a lot of the chip controls, a huge question is how much does it help the Chinese ecosystem? And that's just, I think, an empirical question that I haven't seen very good answers to. So this is true for the original October 2022 controls is the big downside. The big thing you want to avoid is by preventing them from buying U.S. chips, you supercharge Huawei, you supercharge SMIC, which is their best chip manufacturing company. And I think people who are against the chip controls often assume that preventing them from buying U.S. chips will be a boon to their domestic industry. And people who are for the chip controls often assume that it won't help them that much because, for instance, you know, it was already, and this is true, it was already a huge priority for the Chinese government to be supporting their domestic semiconductor ecosystem. And pouring just hundreds of billions of dollars into trying to subsidize it, and they were already struggling. So I think similarly with the, you know, rent don't sell, it's just an unanswered question of how much does this actually help them indigenize their supply chains? How much, you know, how happy are different countries to just rent instead and say that's totally fine, no problem, versus do they then go looking elsewhere? And if they do go looking elsewhere, does that help or is it just so technologically? It's just so technologically difficult to reproduce this supply chain that it doesn't make much of a difference.
Speaker 2The U.S. has approved the construction, I think, of big data centers with NVIDIA chips in several Gulf countries, I think Saudi and UAE, possibly Qatar as well. What are the pros and cons of that in your mind?
Speaker 1Yeah, I think the deals that you're talking about are, as far as I understand, provisional or sort of early stage announced without the details being hammered out. So I think it will really depend on what the specifics of those deals turns out to be. So the big pro is that these are countries, you know, if you look at the different inputs to AI, if you think computing power is an important input, then what do you need for that? You need chips, you need land, like permitting, the permission to build, and you need energy. And so there's sort of a natural trade that you could do and say, look, we have in the U.S., we have lots of chips. Permitting is a nightmare. Our grid is struggling. But in the Gulf, they have plenty of land. They have plenty of sunlight. They have plenty of oil. And so we bring the chips. We bring the chips over there. They let us build and they give us lots of energy. Great deal. So I think that's sort of the main pro is being able to build out more computing capacity than you otherwise could. I think the cons depend a lot on the specifics of the deals. So there might not be that many cons if it is, you know, I've heard a comparison between U.S. data centers in the Gulf and U.S. military bases in the Gulf. It's like this is a U.S. asset. It is U.S. soil. It is fully under U.S. control. Maybe there's not that many downsides if that's the case. The more that there is, the more that the countries themselves have ownership rights over the chips or usage rights or ability to access the facilities, then there's sort of two big potential downsides. One is the connections with China that these countries have. So this includes doing joint military exercises or, you know, having very tight personal and commercial relationships with Chinese leaders and Chinese more political leaders and business leaders. Meaning, does this help China reverse? Reverse engineer chips have more access to kind of advanced chips in a way we wouldn't want? The other big set of downsides is just specifically around the fact that these countries are autocracies. They are not like nice governments.
Speaker 2It might be worth elaborating a little bit on that. I read a blog post you wrote about this and I was like shocked by the degree of repression that there is in the UAE.
Speaker 1Yeah, yeah. So I think people tend to know that Saudi Arabia is, you know, not a democracy. It's famous for, you know, not letting women drive, famous for hacking apart a journalist in, you know, Jamal Khashoggi in the Saudi embassy, assassinating him in cold blood, which is sort of, you know, emblematic of how they think about free speech and free press. The UAE, though, I think people just have a bit of a sense of like, oh, Dubai and Abu Dhabi are like kind of nice places to visit. You know, it's a bit too hot, but they have big skyscrapers and fun indoor skiing or whatever. But those, you know, the UAE is an autocratic country. I think the score on the Freedom House Democracy Index is something like 18 out of 100, which is really, really low. Political parties are banned. There isn't, there's sort of one body that's half elected and half appointed by the royal family, but it doesn't actually have formal power anyway. So there's kind of elections, but they're sort of fake elections. They do, you know, mass trials of dissidents that are sort of clearly not going along with or clearly not. And due process and actual real rule of law, they persecute the families of dissidents. The economy runs on immigrant labor, which those people have very, very few rights. So at a minimum, they're sort of, you know, non-citizen workers who have very little ability to participate politically. In the worst cases, they are essentially forced labor. So, you know, this is really not a country that respects rule of law or that is interested in, you know, empowerment. If it's population, there's pretty strict rules around or quite strict rules around what the media can say about the royal family. It's a hereditary autocracy, you know, with a royal family that is going to. in power. The shape of these deals is still up in the air. It's not clear exactly who gets what. But if you believe, as some of the leading companies making these deals have said, if you believe that access to compute is going to be a huge determinant of national power in the future, and the deals are structured in such a way that the royal families, the autocratic governments here, get access to essentially world-class supercomputers, then that's pretty concerning because you're handing over a large amount of power to actors whose interests are not in line with the public generally, or even with the US in terms of a long-term strategic outlook where their priority is staying in power essentially. Yeah. So when I think one of these
Speaker 2deals was announced, I think for the data center in the UAE, it was a collaboration, I guess, with OpenAI was involved. And I think they put out a press release saying this is a huge win for democracy and democratic AI or something along those lines. Yeah. How should we interpret that? Is this just kind of boundless cynicism,
Speaker 1or is there something else going on? It certainly looks cynical to me. I think it's really playing fast and loose with what sort of democracy means, what democratic AI means. If you read sort of line by line, they never say that, they never imply that the UAE is a democracy, but they sort of talk about building on democratic rails or promoting democratic AI internationally. I think there's kind of two ways you could interpret this. One is, and I think both could be reasonable in principle, but aren't really present in this deal. So one is that access to AI can be a democratic force for good in general. For example, if OpenAI were to make chat GBT available freely in a country that previously had restrictions on speech or restrictions on access to information, I do think that having an AI system that you can ask questions about, you can do that. And I think that's a great way to interpret this. I think that's a great way to interpret this. I think that's a great way to interpret this. I think that's a great way to interpret this. I think that's a great way to interpret this. I think that's a great way to interpret this. I think that's a great way to interpret this. that you can get information from, that you can learn things from, that you can use to further your own goals. I think that is obviously empowering of the individual in a way that is compatible with democracy. But it doesn't seem to be part of the deal with the UAE. Shortly after that deal was announced, one of their executives was on stage and was asked by a journalist, a US journalist who had lived in the UAE, she said, I've lived in the UAE. There's these red lines of what media can and can't say. Are you going to build those red lines into chat GBT in the UAE? And he was like, no, I'm not going to build those red lines into chat GBT in the UAE. And she was like, oh, we'll see, you know, which is like, not very encouraging. So this sort of first theory of change of giving more people access to AI and that being the sort of like democratic grassroots empowering force doesn't seem like the case. The other one, which I think they are trying to sell more is this idea that American AI is inherently more democratic than Chinese AI. And so anything that furthers American AI or, you know, helps America win against China is good for democracy. And I think, again, that one in principle could make sense and sort of story of, you know, the big pro here is being able to build more data centers in a way that you couldn't do domestically. The problem is, on the one hand, again, if we think that the UAE government is going to be empowered by this deal, that they're going to have access to really sophisticated AI, then why exactly is that better than China having access given that they are so autocratic? But secondly, also, you know, what exactly, what are the parameters of the deal and what did the US have to give up in order to be able to build more data centers in a way that you couldn't do domestically? In order to get this benefit, this strategic benefit. So again, if it's a data center that is essentially like a US military base and fully US owned and controlled and, you know, everything occurring to the US, great. But if, you know, a big thing that the companies will say is if the US doesn't sell it, then China will sell it. So that China is waiting, you know, outside the door. If the deal doesn't go through with the US, they'll just offer the exact same deal. And that's- But China can't even make the chips. Exactly. Exactly. So it's totally disconnected from the reality of what they can do. And actually, this has been a recurring theme. In the debates about chip controls and sort of US versus China capacity is really neglecting the difference between what are the specs that are announced versus what is their actual production capacity and what are their real specs. So you have, you know, so just to close the thought on the UAE deal, one thing that was reported was the UAE getting hundreds of thousands of next generation NVIDIA chips. And that is just an unbelievable amount of computing power that China absolutely cannot match. They are really struggling to meet their own domestic demand. There is no way that China is going to be able to meet their own domestic demand. There is no way that if that deal didn't go through, they could step through the door and offer the UAE the same thing. So I think that's just a really bad motivation. I think in the discussions about chip controls more generally, yeah, it's been really neglected. Like what number of chips are we talking about? What is China's capacity to manufacture them? And when they demonstrate manufacturing, how have they done that? So for example, it's now pretty well demonstrated, pretty clear that one or two, I forget, generations of Huawei's chips, their most advanced chips, they've demonstrated this like pretty impressive performance. And do you know the way they've done it? Have you heard this story? No, I haven't heard this one. Basically, they fooled TSMC into making chips for them. So this is this Taiwanese semiconductor manufacturing company that makes the most advanced chips. So this was after the controls were in place. TSMC was not allowed to manufacture chips for Huawei. Huawei set up a front company, sent in some chip designs that were clearly for Huawei chips, but TSMC just like either turned a blind eye or didn't have the processes in place or something else, and went ahead and manufactured a large number of chips for them. And so they've demonstrated this pretty impressive performance. They gave them dyes, I forget the exact specifics, but basically handed them this giant stockpile of very advanced inputs for their chip making process. And so they are now kind of marketing and selling those chips and acting as though it's sort of their own sophistication,
Speaker 2but it's like, well, no. If anything, it shows the opposite.
Speaker 1Exactly. So yeah, there's often these sort of headlines of, oh my goodness, Huawei announced this chip, Huawei demonstrated this performance, Huawei manufactured this small run of phones with advanced chips in them. And so they're now kind of marketing and selling those chips. That really isn't looking at like, wait, how do they make that? And how many are they going to be able to make in the future? Which is what you need to do to try and understand the actual sort
Speaker 2of effects and trade-offs with these controls. So we almost always assume that frontier model is going to be developed and first deployed in the US or China, that those are the two leading countries. Is it crazy to think that, you know, however you want to define AGI, superintelligence, it could first be trained in Saudi or the UAE? I mean, I think they have a lot of disadvantages, but if they have access to an enormous amount of compute, and these countries, I mean, the benefit they have is they have just enormous stockpiles of cash, basically, that they could throw at this problem. They have a lot more kind of disposable income than a country like the US has. I mean, maybe not as much as the US, but being autocratic, they can direct money much more with like far fewer constraints. Is it possible that
Speaker 1superintelligence could be developed in Saudi first? I think it's possible. I would say the UAE more than Saudi Arabia is my impression. So certainly by all accounts, the UAE government is very, you know, so-called AGI-pilled, very into this idea that AGI is going to be a huge deal and that the way there is scaling up compute and things. So if that's true, then they're an advantage because they already believe that. I think, as with all questions, AI depends on the timelines, depends on how long we're talking. I think in the next couple of years, they would struggle to stand up a really competitive effort. But if we're talking a little bit longer scale, then potentially, I think also they're very well positioned to benefit from the US rejecting, you know, high-skilled, engineers and scientists. They can really offer great compensation packages, great quality of life. And I'm sure that they are trying to do that, if not already doing that. So it's not, you know, it's not what I would bet will happen, you know, that the most advanced models will be developed in the Gulf, but I definitely think it's possible. And certainly, you know, if we continue to make these giant deals and really build up a huge amount of computing power there,
Speaker 2then it becomes even more possible. Do you have any proposals for how we could go through the transition to AGI, potentially superintelligence, without ending up with an absurd amount of power being concentrated in the hands of some companies, possibly some governments, possibly some individuals? Maybe you could explain to people why that feels like it could be a natural outcome.
Speaker 1Yeah, I mean, I think there's a couple of reasons why AI might be very power concentrating. The most obvious one is if it continues to be as capital intensive as it has been recently. So, you know, the clearest trend over the last 10-15 years of AI development is this sort of compute scaling idea of the way to make progress. Yeah, I mean, I think that's a great point. I think that's a great point. I think that's a great point. The top players, so that, you know, there's some kind of very stringent requirements on how to develop AI, and that's very hard to break into, you know, like a regulatory capture. And there's a small number of players where maybe those regulations were introduced with good intentions, but if they end up concentrating power, then that's not great. And I think there's a natural tension here as well between, I think, among some people who are very concerned about existential risk from AI, really bad outcomes, you know, AI safety. There's this sense of, well, it's actually helpful if there's only a smaller number of players, because one, they can coordinate better. So maybe if, you know, racing leads to riskier outcomes, if you just have two top players, they can coordinate more directly than if you have three or four or 10. And also a smaller number of players is going to be easier to, for an outside body to regulate. So if you just have, you know, a small number of country companies, that's going to be easier to regulate. So I think there's often a sense of, well, actually that concentration is valuable. I see the logic there, but the problem is then sort of the question of if you do manage to develop If you do manage to avoid some of those worst case outcomes and then you have this incredibly powerful technology in the hands of a very small number of people, I think just historically that's been really bad. Like it's really bad when you have small groups that are very, very powerful. And typically it doesn't result in good outcomes for the rest of the world and the rest of humanity. I guess that's all preface to your actual question of how do we avoid this? I don't really know. I would love for there to be more work on this. I think a lot of the thinking that has happened about this has been between people who say, well, the risks are really large. And so we have to try for concentration because otherwise we all die. And other people saying that's stupid. We're obviously not going to all die. And so therefore we should just diffuse power maximally. And I think trying to get those conversations, those people to actually engage with each other and say like maybe there is a hard problem here. What if both risks are medium? Then we're in a real tough spot. Or both risks are high, right. Yeah, I think sometimes when I talk about this, people think. Think that I'm like optimistic and I'm like, it's all good. Like it'll be fine. Just let, you know, let there be less power concentration. But I actually think my take is like a more pessimistic take of like, well, I don't think concentrating is the solution. And I don't think maximum diffusion is probably the solution. And so how do we navigate that middle ground? I think it's really hard. I think one answer might be, we might get lucky in terms of how the technology develops. It might be the case that actually things develop relatively gradually. There's time for sort of fast followers to catch up and there can be, you know, relatively broad access to capabilities. And there aren't these like really. Decisive huge civilizational downside risks that you need to manage in a concentrated way. So I think what we might just get lucky, that's sort of my best hope. I think there's also tools that we can, that I would love to see explored more that would target this. So at a very basic level, there's things like AI literacy, meaning like how do we get a larger range of people to understand what is going on with this technology, to be able to engage with it, to be able to think about trade-offs, pros and cons, risks and benefits. How do we think about. Sort of worker empowerment or like the role of workers in including workers at frontier companies in shaping the development of the technology. There's also very like basic things like taxing companies. So if the only problem is just, well, it's a naturally capital-intensive technology and so large actors are going to build it and we don't have to worry about the downside risks, then you have sort of very, very, you know, traditional tools like tax, antitrust. That can. Come in and help try to diffuse that concentration of power as well, but that doesn't solve the safety challenges.
Speaker 2The concentration of power worries also, I think, point to some extent against the export controls or against being very intense on the rent-don't-sell of compute. Because I would guess, you know, if almost all of the compute is concentrated in the United States and, you know, even Europe or Australia or the UK, or basically no countries can get a foot in the door to have significant data centers. That, I guess, leaves them very vulnerable to exploitation by the United States. So it puts the United States in a troublingly powerful position. Do you agree with that?
Speaker 1Yeah, I think that's probably directionally right. I haven't thought about it much, but yeah, interesting point.
Speaker 2You wrote recently about how maybe one of the underlying disagreements in AI that people don't necessarily address directly is between people who favor kind of decentralization and dynamism and those who feel apprehensive about a dynamic, open AI ecosystem. Can you explain that idea? Yeah.
Speaker 1So this was coming from a book that I read late last year by Virginia Postrel called The Future and Its Enemies, which is a great title. It's actually from the 90s. So it's sort of in the era of like environmentalism and sort of cyberspace just becoming a thing. And I found it really helpful because it sort of, for me, helped me put my finger on something that I think weirds people out about AI safety discourse that people in the AI safety community are often not as aware of, which is basically this instinct of if there are these big risks, then there's going to be big risks. And I found it really helpful because it sort of, for me, helped me put my finger on something that I think weirds people out about AI safety discourse that people in the AI safety community are often not as aware of, which is basically this instinct of if there are these big risks, then there's going to be big risks. And I found it really helpful because it sort of, for me, helped me put my finger on something that I think weirds people out about AI safety discourse that people in the AI safety community are often not as aware of, which is basically this instinct of if there are these big risks, then there's going to be big risks. So the solution is, you know, there's this risk that AI will kill all humans or this risk that AI will take over or something like that. And the solution is, well, we have to make sure that the right people build it and that it's a very small group. And I get the logic there, but there's sort of a lack of... And that's actually not my take. My take is more like, you know, we should be more worried. We should be more pessimistic about these factors we have to balance. To put that another way, I think an issue with the... I see the logic when people are very concerned about existential risks from AI say, you know, we're going to have to concentrate in order to manage these risks. But then it's lacking the, oh, no, like, uh-oh. Like, we think we're going to have to concentrate access to this technology in order to manage this very severe risk. That's terrible. Like, oh, how do we handle that? How do we, like, how do we think about all the downsides of that? How do we make sure that it doesn't just end up in massive disempowerment of most of humanity? And I think a lot of people, especially in the tech world, but I think also more generally, subscribe to what this book, Pastoral's book, would call a more dynamist view, meaning that a huge source of success, a huge source of human prosperity, well-being, is having a world that allows for more trial and error. That allows for sort of decentralized experimentation. That allows for things to fail. That is more focused on sort of resilience and recovery from problems than on kind of preventing every problem in the first place. Because if you try to go with preventing every problem, then you end up with a much more authoritarian, centralized, what she calls stasis, kind of stability-focused, control-focused regime. And again, I don't mean to say, therefore, we should just not worry about it and just let people do whatever they want. What I more mean to say is if we really think that these sort of more, you know, quote-unquote stasis solutions of, you know, control and stability are really necessary, then we need to really be apprehensive about that. And we need to be thinking really carefully about, okay, if we really, really think that it's necessary to concentrate power in this way, how do we think about how to deconcentrate it again at the end? Or what are the guardrails that are put in? And how sure are we? Like, you know, how, what evidence can we gather about what the nature of different AI risks is and whether it is worth this trade-off? As opposed to, I think, sort of the vibe of the AI safety community, which can often be like, it's okay, the solution is just concentration and then we'll be fine.
Speaker 2I think that's shifted quite a bit in the last few years. I think it has.
Speaker 1Yeah, I think it has. But I do think it is still a strong undercurrent in a lot of the thinking. I think there's been more discussion of risks of concentration of power and so on. But I think there's still a strong default of, like, the answer is just have one project and then you can manage the project well.
Speaker 2I think this is one of many other kind of assumptions that get baked in. But all the way back, I think, to the very, to, like, the 2000s, when the picture of how this would play out was very rapid recursive self-improvement loop based potentially on not that much compute. Yeah. Thinking that is overwhelmingly the most likely way for things to go. So power will end up concentrated whether you like it or not.
Speaker 3Yeah.
Speaker 2And I mean, it's still possible you could end up with a very strong recursive self-improvement loop, but I think people believe that much less than they did 15 years ago.
Speaker 3Yeah.
Speaker 2But it's so hard to... To go back and remove all of the assumptions that were kind of baked into the language and just how you were thinking about the problem from the beginning. Yeah.
Speaker 1Yeah. I mean, this is like a sort of mini hobby horse of mine for the last couple of years has been, personally, I think something that we really need in this space is people who are coming in who are willing to take these ideas seriously. So willing to think, like, to engage with the idea of AGI, of superintelligence, of existential risk, of human extinction, of, you know, disempowerment, AI takeover, these sort of sci-fi ways. But so people who are willing to engage with that, but who are not coming out of that sort of social and intellectual milieu that I think both you and I come out of, and that I think does contribute to sort of groupthink and sort of bubble dynamics, not bubbles in like, you know, a financial bubble, but like being in a social bubble. And that's actually something that I really value at CSET is being around people who are mostly not from that world and who mostly don't bring in all of those sort of, as you say, those sort of baked in assumptions that are hard to kind of go back and unbake. And so, yeah, personally, I think that in this space, it's just really, really helpful. And I sometimes meet people who are sort of getting interested in these issues. And they're like, well, I haven't been thinking about them for as long as some other people. And I think that can actually be a real advantage because you're bringing fresh eyes, you're bringing sort of fresh perspectives. And the key part for me is, is yet not shying away from being interested in these questions, as opposed to sort of having a particular background in some particular set of answers.
Speaker 2It's it. I feel like we're in a very difficult spot because it's so many of the obvious. Solutions that you might have or approaches you might take to dealing with loss of control do make the concentration of power problem worse and vice versa. And so what policies you favor and disfavor depends quite sensitively on the relative risk of these two things, the relative likelihood of things going negatively in one way versus versus the other way. And at least on the loss of control thing, people disagree so much on the likelihood. People who are similarly informed know about everything there is to know about this. They go all the way from thinking it's a one in a thousand chance to it's a one in two chance, you know, 0.1% likelihood to 50%. Chance that we have some sort of catastrophic loss of control and discussing it leads sometimes to some convergence, but people just have not converged on a, on a common sense of how, how likely this outcome is. Yeah. And so the people who think it's 50% likely that we have some catastrophic loss of control of it, it's understandable that they think, well, we just have to make the best of it. Unfortunately, we have to concentrate it. It's the only way. And the concentration of power stuff is very sad and going to be a difficult issue to deal with, but we have to bear that cost. And the people who think it's one in a thousand are going to say, this is a terrible move that you're making. Cause we're like accepting much more risk. We're creating much more risk than we're actually eliminating.
Speaker 3Yeah.
Speaker 2Um, I mean, I guess they had. idea would be if we could find policy responses that help with both simultaneously or help at least one of them without harming the other. I don't know how, I think people probably have underdone attempts to find like win-wins or at least like win-neutrals, but it's easier said
Speaker 1than done. It's not a simple matter. No, definitely. I mean, I think, yeah, there's sort of a set of policy recommendations that I do think can help us navigate this a little bit. So these are things like, you know, everyone's favorite consensus recommendation is sort of more transparency and more disclosure from the AI companies. I think that is really helpful. I don't know. I think, I guess my stance here is that it's very unlikely that there's these sort of two futures that we imagine and the future like goes down one of those tracks. I think it's very likely there will be unexpected twists and turns, you know, new things that develop that we didn't anticipate. You know, technologies never develop exactly the way that we think they will. So I think our stance here should not just be kind of which of these two camps is right, but like being open to and ready for many, many different possibilities. So yeah, I think transparency is helpful for that in that it lets information propagate about kind of what is actually going on, what is sort of the ground truth of what the systems are being developed, what systems are being developed, what risks they pose and so on. Yeah, this sort of AI literacy point or like technical capacity in government is sort of a similar thing of making sure that you're hiring people or training people so that they, so there's not just a very small group of experts inside the company. And then like sort of resilience focused approaches. So this is sort of cyber defense, biodefense, you know, AI control type approaches, meaning how do you use, how do you invest in building AI systems that can kind of monitor other AI systems and things like that. And one other piece that I think can be helpful as well is really investing in the science of AI. So people sometimes talk about, oh, the government should be funding AI safety because the companies are funding AI capabilities. There's actually a different thing that I'm trying to say with this is, which is how do we, I think we would be in a much better position if we had a more grounded understanding of how AI systems work and like how do they generalize from their training data? How do they, why are they so jagged? Like why do they fail in these weird ways? How could you understand whether a given system is likely to succeed or fail in a given setting without just trying it? Because right now we basically just have to try it. Interpretability is another kind of thing that I think is really important. And I think right now we're really under-investing in the science of AI as opposed to sort of the engineering of like making AI that is better, you know, that can do more stuff. So I think those are like some ideas that I think could be helpful. Sometimes I think the AI safety community can reject anything that is not a full solution. I feel like there's something about the community's like background in, you know, mathematics or computer science or philosophy, where they want like a full, complete solution that you can prove is adequate. And if it's not that they want to reject it. I think that's really mistaken. I think we're going to have to kind of do our best, figure things out as we go. And I think investing resources now to try and position ourselves better so that we can respond better one year from now, five years from now is really valuable. And it's one of the few things that we can do. I think it's also worth thinking through what are sort of more elaborate plans for actual sort of full solutions. But even there, I mean, the idea of a full solution suggests that there is like a clear end state which coming back to this idea of dynamism, I think is the wrong way to think about it. I don't think there is a one best way, you know, an optimal endpoint that we should be aiming for. Instead, I think we need to be trying to get to a world that is dynamic and open and free and empowering, and that that's not going to look like sort of we fixed it. Instead, it's just going to look like we got maybe we got through an acute risk period or something like that. Yeah, the desire for a
Speaker 2full comprehensive solution is another one of these assumptions, I think, that goes back to the Muri vision. I guess what we're talking, I think, in the week of the release of if anyone builds it, everyone dies, where they are. I mean, it's in the title saying, we really do need to solve this completely comprehensively muddling through is that isn't definitely not going to work in their view. And that may yet be vindicated, but I think it's very far from obvious. And on like many plausible views, muddling through is an option, you know, a bunch of like half measures might well work. I guess, I think more and more people have appreciated that in recent years. I think I hear, the people who want the full solution, they're doing it because they think loss of control is overwhelmingly likely, the technical problems are overwhelmingly hard, rather than just because they've made that as an assumption, like an unquestioned assumption. Yeah. I mean, and I
Speaker 1think there's real, like a big spectrum of how competently you muddle through, right? So like, there's sort of, I think, sometimes muddling through, I don't really like that. I don't really love that phrase, because it sort of sounds like, oh, we'll just like take it as it comes and sort of see, and it'll be fine. I think, you know, a different version of muddling through a more competent version is this, you know, this idea of plans are useless, planning is everything of like, how do you set yourself up with lots of different possible levers to pull lots of different things on the shelf that you could take off the shelf if needed, and also set yourself up with the kind of visibility and understanding to be able to tell what paths you seem to be on. Yeah, I think in a sense, that's also muddling through because you're not kind of pulling the trigger on here's our 12 step plan, and we're going to go through it comprehensively. Yeah, I think, but I think there's a big, a big range of what it looks like to muddle through. I mean, arguably, we muddle through with nuclear weapons, but that involved a massive international treaty and a huge monitoring apparatus. And like, that's really valuable. And I think it clearly reduced the number of nukes in
Speaker 2the world and the amount of nuclear risk in the world. I think most of the technical research agendas are either win-win or win-neutral on this. I guess, I mean, one that is a win-win is the ability to inspect the models and detect hidden goals or hidden agendas that might have emerged unintentionally or have been deliberately inserted in there. That is really helpful. I think for avoiding concentration of power, I guess, Tom Davidson in an episode earlier this year explained how this problem of secret loyalties does create a vulnerability for a power grab, basically. But it's also, I mean, the same techniques might well just work for outing any unintentional goals that have ended up getting baked into the model through training that we didn't intend. I guess like model organisms, understanding AI misbehavior, when it generates, when it doesn't, what things you can do to reduce it. I mean, almost across the board, I think funding for all of the those sorts of research agendas does just seem like a real no-brainer in my mind.
Speaker 1Yeah, I think that's right. I do think that there's a different, maybe a different axis where they're sort of like win-neutral as opposed to win-win, which I've been thinking more and more as, I've been thinking of this more and more as really central to disagreements about risk, which is this question of like, will AI just be a tool or will it be something more than a tool? This is sort of part of, but not the full disagreement with the AI as normal technology authors, Arvind Narayanan and Sayash Kapoor, who are both really excellent, and I think are great examples of people who are taking these issues seriously and engaging with them seriously, but coming with a very different set of assumptions, and I think really adding to the debate. And I think, I mean, I think that if AI stays as a tool, continues to just be something that people sort of use to their own ends, and maybe sometimes it breaks or fails, but it doesn't have its own agenda, it doesn't have its own goals, then you could still be very worried. Then it makes sense to be worried about concentration of power. All of the takeover stuff doesn't really come into play. And so I think that is another place where it just, there's different assumptions, different expectations, and different trade-offs look better or worse to different views. But I think that, I think it's been really valuable to have that AI as normal technology view articulated, because I think it is, it's sort of a funny thing where I think they wrote it because they sort of see the super intelligence view as sort of dominant. But in fact, what they articulated was that, you know, if you're a super intelligent, you're not going to be able to do that. But if you're a super intelligent, you're not going to be able to do that. So I think that was a much more like widespread unarticulated view. I think most people when thinking about AI.
Speaker 2Surely, overwhelmingly, that's the most common approach.
Speaker 1Most people, yeah, assume that it's going to be more like a regular technology. It's going to be a tool. We're going to decide how to use it. And this was explicitly...
Speaker 4Someone make the case that that's really what it is.
Speaker 1Yeah, yeah. And this was explicitly part of why they wrote that paper was to say, like, we think this is a common view. We want to put words to it. We want to sort of describe
Speaker 4the components of it. Where do you come down? Is AI a tool or is it a new form of life?
Speaker 1I don't know yet. I think I agree with some of the things they articulated. I think it was really useful to articulate the distinction between capability and power. So an AI system that is able to do something does not mean that it's actually like set up in a way that it can do that thing. And we as humans have a lot of agency in deciding what AI has the power to do. I think sometimes they say at the beginning of the paper that it is both a description of the and a forecast about how it will be handled. And I think those three things, sometimes I agree with them that AI is a certain way. For example, that we are deploying it relatively cautiously at the moment, relatively slowly. And I agree that we probably should continue, especially in high-stakes settings, to deploy it relatively slowly and cautiously. But I maybe disagree that we obviously will. And in particular, I think, and I know that they're working on this actively right now, they excluded military AI from that paper entirely. And they also included a section about, I guess, another debate, which I think is really underexplored, which is how superhuman can AI get in different areas? I would love to see more work on trying to pull apart what are the reasons to believe that there's a very high ceiling above human level at different capabilities. And one thing that Arvind and Sayash say in their paper is that, for example, being faster than humans is not that useful in many settings. It's useful in high-speed trading, but not that many other things. For me, battlefield management is going to be there at speed is absolutely a huge advantage. People in the military talk about the OODA loop, serve, orient, decide, and act loop, and you want to, quote, get inside your opponent's OODA loop, meaning be faster than them, be able to drape around that loop faster. Anyway, so that's one example of an area where I think the AI as normal technology view is not necessarily accounting for the incentives to hand over power, hand over autonomy in order to get benefits like, for example, speed.
Speaker 2You've written that you think the AI security ecosystem, it's often appropriated the language of non-proliferation, I guess, from nuclear weapons, from biological weapons, and you think that might have been a wrong turn, or at least it's narrowed people's scope a bit, and instead we should think about things in terms of adaptation buffers, which is, I think, a term that you introduced? Yeah. Can you explain why you think that might be a mistake and what adaptation buffers are instead?
Speaker 1Yeah, so this is specifically in response to concerns about misuse. So the idea being we're building these more powerful AI systems, they're going to be misusable in particular ways. Two of the most common people bring up are being used to create bioweapons or being used for cyber offense, so hacking, and that the response is the AI systems are going to be so powerful they would enable such bad attacks that we have to do non-proliferation, meaning prevent them from spreading. And I think the challenge with this is it won't work, and it will be very invasive and very sort of authoritarian. And the reason for that is basically because of this. This very clear trend, this sort of maybe somewhat contradictory seeming trend to people who are not following AI closely of, on the one hand, as we're building more advanced systems, the amount of computing power that you need the first time to build an especially advanced system is going up over time. So you need more and more and more compute to build something at the frontier. But as soon as you build something at the frontier, so say, you know, an AI system that is as good as a software engineering intern or an AI system that can get some score on some test or whatever. As soon as we build that for the first time, the amount of computing power and also just broadly technical sophistication that you need to build that is going to fall pretty rapidly over time. So, for example, at the time when the AlexNet paper was published in 2012 by Alex Krujevsky and Ilya Satskeva, Jeff Hinton, that was like very, very hard, very complicated. But now it's incredibly easy, incredibly cheap. Likewise for, you know, a GPT-3 type model at the time, really at the cutting edge, quite expensive, very, very expensive. Very difficult, now very easy to recreate. What that means is for misuse, so for this idea that we'll have AI systems that are very powerful at hacking or very good at helping with bioweapons, maybe initially it will be a small number of companies or, you know, research groups that can build a great model. But a couple of years later, five years later, it's going to be very easy for many groups to do that. And so in order to do non-proliferation, you're going to have to get very, very invasive about, you know, restricting the number of chips that people have access to or monitoring what they're doing with AI systems. And I think it's just really not likely to actually work. Like if you have a determined actor, you know, a bad actor who is trying to misuse these models, they will find a way to do it. And in the meantime, you will be really restricting the ability of many, many normal people to use and build AI systems in ways that would be perfectly fine. And to me, the sort of best alternative that we have is, yeah, I call it a sort of adaptation buffer, basically meaning this period, this buffer period. Between when we know that some capability of concern is going to be very accessible at some point in the future and when it's actually accessible. So I think we are in this period right now for AI systems that can help novices to create known bio threats, if that makes sense. So it's important in the bioweapons conversation, I think sometimes two threats actually get threat models get conflated, one being novices creating known bio threats. So this is like, you know, teenager in their basement creating smallpox kind of thing. That's different from helping experts create much more sophisticated bio threats, which often is going to involve the use of biological design tools or it's sort of a different process. I think we're clearly in the adaptation buffer for that first one, and the thing to do then is to try and put as much in place sort of societal resilience style efforts in place to prevent that capability from having massive negative impacts when it is widely available, as opposed to trying to sort of. Clamp down and prevent it from ever being widely available. So that's that's this idea of the buffer is that the time period between when we can forecast it pretty specifically, not just like a bioweapon, something, something, but something pretty specific and and the time when we can get it. Now, I don't think this is a perfect approach, you know, in particular, if if buffers are short, then maybe we don't have enough time to to get to them. And it's very much focused on misuse, not on kind of more misalignment focused models. But that's kind of the basic idea.
Speaker 2Yeah, I mean, I think people have not always been drawn. To the adaptation buffer framework, and I think it's not for no reason, it's often because they think there aren't any good adaptations, I guess, especially on bio, I think this has been the view that if it's easy for amateurs to create new pandemics, it's just so costly to combat them that we just have to stop them from being created in the first place. I guess I recently did this interview with Andrew Snyder Beatty, where he presented a plan, like, maybe there are adaptations that we could make that would make society very resilient to any new pandemics that someone tried to try to release. It's not it's a heavy lift, though. Yeah. It's challenging, and it's not not not completely obvious that it might work. So, I mean, it's possible that we could go down the adaptation buffer track and then look back and say, actually, that was a that was a bad idea because, in fact, the adaptations were just too difficult and nonproliferation was unfortunately the only option.
Speaker 1I mean, again, I think it's it's like my point here is not nonproliferation doesn't sound fun. My point is, I think nonproliferation won't work anyway. And I do think also I think people yeah, I mean, time will tell. I in this case, I guess I hope that I'm right. Sometimes I'm talking about risks and hoping that I'm wrong, but time will tell. I think people sometimes overstate the inevitability of really severe misuse. And I mean, if you look at when I wrote this post, for example, about adaptation buffers, one of the commenters was saying, like, well, isn't this all hopeless? I remember in college when a professor of mine was telling us about how easy it is to just drive up to a water treatment facility. With a dump truck and dump a bunch of toxic chemicals right after the treatment ends. And someone else was saying to me, you know, that they'd seen, you know, some video in the 2000s of how you can in an airport, how you can make a bomb using materials that you obtain after security. So you like go through security and then make a bomb. But these examples, these examples are both great. Like, they're very encouraging. Like no one, to my knowledge, has done. I don't know about the water treatment. Maybe that's happened. Or if they did it, then they got caught. Right. And it got prevented. So I think sometimes there's a little. A little bit of a neglect of what actually are all the options. What is the toolkit? Including just like the FBI monitoring terrorist groups. Like that's a real thing. Or the CIA as well. Like that's a real thing that happens. Or looking at materials, you know, looking at synthesis of nucleic acids, which is obviously part of the discussion. Looking at, you know, can you just have layered approaches with, you know, the chat GPTs of the world? They're going to refuse to help you. Like they're pretty good, pretty good control on that right now. So that reduces the number of people. And then if you do. Eventually have kind of attackers actually really reducing producing bioweapons. Is it? I don't know. I feel like sometimes in the existential risk community, there's this assumption that it's going to be this absolutely horrific event as opposed to something that is potentially, you know, can be contained earlier. So I don't know. I'm not, you know, not unworried about this. I think it is really worth trying to prevent, trying to mitigate. But I think our at a minimum. If we like any concern that we do feel any risk management, sort of energy and resources that we have for preventing bio attacks should be doing a lot of this resilience focused stuff as opposed to focusing primarily on clamping down on models. Again, just because I think clamping down on models is not going to actually. It's not a long term solution. Right, right. And so saying that adaptation buffers might fail, I agree, but I think saying and therefore we should just focus on preventing model proliferation is the wrong call.
Speaker 2You were recently involved in a CSET report titled AI for Military Decision Making, Harnessing the Advantages and Avoiding the Risks. How are you worried that the military is going to end up deploying AI poorly?
Speaker 1So this report was about a specific kind of AI called decision support systems. And the reason or one reason that we wrote about it was because it's something I think there's sometimes a bit of a gap between kind of external perceptions of military AI and what the military is like actually looking at in practice. And one, for example. I think externally people really focus on lethal autonomous weapons, especially, you know, drones with facial recognition. I think it's like a really compelling use case that the public worries about. But if you look internally, there's many, many other potential use cases that the military is thinking hard about and where, you know, how they adopt them, how quickly they adopt them will matter a lot for what actually ends up happening. And so this paper was actually focused on a specific kind of system called decision support systems. And what? What we wanted to do there was really not get into this kind of binary of, oh, AI in the military is bad and should be banned and we shouldn't do it, or AI in the military is essential and we should just be racing to adopt it as quickly as possible. But instead to say, take a clear-eyed look at what are these systems, why do militaries want to use them, and how can they use them effectively and not get caught up in potential pitfalls of AI systems. So that involves looking at. You know, what are the actual benefits of using decision support systems? It depends a little bit on the specifics of the system. um and uh what are the potential pitfalls so you know do you have a clear understanding of the the scope of a particular decision support system what it was trained for where might it sort of um be out of distribution so you know in a situation it wasn't trained for and fail or um how do you train your operators um so that they can use them effectively so that was kind of the the set of
Speaker 2issues that we were exploring there how much appetite is there for rapidly integrating ai into into the military i interviewed dean ball yesterday and i think i was waiting to him might we see you know premature applications of ai in the military dangerous ones and it was like it's going to be so hard to do it at all uh it's actually probably going to happen very slowly and in a very hodgepodge fashion yeah what do you think yeah i tend to agree i think there is a lot
Speaker 1of appetite um but institutionally the ability to um procure or build ai systems and then roll them out at scale is um tough um we have another paper at cset um that i wasn't involved in um called building the tech coalition which is a case study of a sort of successful adoption of ai and really what you know it's the exception not the rule that they got this system to an operational state where it's actually being used in practice um and the you know the the case study is looking at what are the factors that actually made them able to succeed in that case um and you know one of the the key parts there was having uh the military sometimes talks about having bilingual leaders who are competent both in sort of technology and also military operations and one thing we really identified in that case study was you actually need trilingual leaders who are competent in technology and military operations and also these kind of acquisition procurement kind of questions um contracting getting through all the legal language so it's tough like there's a lot of barriers that the military is not set up the way that it does um research and development is not designed for software um the way that it does testing is not designed for non-deterministic ai systems so i think the the the the the the the the appetite is very much there but um i tend to agree with dean that in practice it's going to be
Speaker 2slow and piecemeal and a slog do you have any read on how worried the military is about ai being backdoored or having secret loyalties or agendas i think they're most worried about that the military
Speaker 1is you know naturally set up to think about adversaries right sabotage exactly so i think they're certainly worried about that in the context of what an adversary whether it's china or um a could do um you know that's certainly reason not to use you know chinese models or to be very cautious i think about um even using us models that are trained on you know the broad internet there's already evidence of um russian groups for example doing um essentially a version of data poisoning um trying to seed online you know data sets with pro-russian views um so i think that's primarily the lens that they're they're thinking about it through do you hear
Speaker 2that i think it's important to keep that in mind i think it's important to keep that in mind i think it's important to keep that in mind i think it's important to keep that in mind i think it's important to keep that in mind i think it's important to keep that in mind i think it's important to keep that in mind i think it's important to keep that in mind i think it's important to keep that in mind should it make should it decide whether to accept orders based on what it thinks the law of war is or should it just follow orders of whoever its operator is i guess each of them has its issues if it's having to make you know if your tank is having to make independent judgments about the law of war uh about military law maybe it's not equipped to do that and that also creates a lot of vulnerabilities i guess that um adversaries could could try to um use that against you on the given whatsoever i think i mean that creates a lot of opportunities i guess for coups where previously you just wouldn't have been able to get human collaborators to go along with it
Speaker 1yeah do you have any thoughts on this i think mostly the this is sort of only like relevant isn't in as much as you're thinking of ai is very much not being a tool so you're thinking of it as being kind of having its own agency um making its own decisions and i think the discussions in military circles are very focused on ai tools um they're just not thinking about independent agents yet i think that's just not really part of the discussion yet because it's unacceptable or because technologically not feasible yet yeah i think it's sort of too sci-fi for the military at this point and also based on sort of where they're at with adoption the level of sophistication of the tools that they are looking at um i will say that you know when these kind of topics come up there is kind of a natural uh like to the extent that ai is operating in a more sort of agentic independent autonomous way that is more equivalent to a human operator um there is a there are some laws for military personnel that you could in theory port over to an ai so for example uh lower level service members are expected to follow the commands of their commanding officers but they are supposed to not if they're if the command is illegal but also things that they do that um uh you know that go poorly that does then reflect back up on the commander so you could there's there's there's ongoing questions about how does accountability and responsibility for the use of ai systems flow back to through the command chain um so that you can kind of if something goes wrong who is held accountable um which works which which can actually work if the ai is primarily a tool and can potentially also um work if it's if it's sort of operating in a less tool-like way but yeah i think those the conversations about ai that is not a tool um are pretty nascent yeah yet to come yeah you recently wrote that we should maybe stop using
Speaker 2the term ai alignment almost at all and instead talk about ai steerability and i tweeted this out recently and it got quite a bit of play it seems like a lot of people are on board with this in principle although i suspect it won't happen because terms are just very sticky it's very if you're used to saying something it's so hard to stop saying it yeah i mean can you explain why ai steerability is better yeah i mean i think it's better in some
Speaker 1ways since publishing the post i've maybe come around to also thinking it's also not quite right um and yeah i would never expect that um i don't think we can get the word alignment out of our vocabulary at this point so i was more making the post for people who are um trying to explain it or you know want an alternate sort of lens on it the basic idea is i think the term alignment um sort of embeds in the word this idea that there is sort of a clear thing to be aligned to so like if we think of alignment we think of like aligning the tires of a car where they're supposed to be straight we think of like you know aligning a picture on a wall where it's supposed to be like you know perpendicular to the whatever um and so i think if you start talking about alignment people's minds just very naturally go to like aligning to what like what is what is the standard here um which i think is sort of is an important question um but is getting ahead of itself where the underlying problem is more one of you know now ai control means something else and i think it's sort of an important question um but i think it's sort of something else but it's more one of controllability or steerability in the sense of can you actually direct the ai can you like build it in such a way that it behaves how you want at all in the first place um so i think steerability you know is the ai steerable conveys this better the challenge with steerability that i've kind of come around to thinking is more of a problem than i than i did when i wrote the post is it it doesn't distinguish between steerability at the development phase of like when you're training the ai when you're setting it up can you steer it to be the kind of ai system that you want um but i think it's more one of those things that i think is sort of versus when it's in operation can you go in and intervene and steer it in in sort of ways that you like so i think some people um who read that post thought that by steerability i was talking about for example when you're with a chatbot and you can like chatting with a chatbot and you can like uh sort of quote unquote steer it to be to role play in one way or another you can it will go with you and it will sort of follow your lead in lots of different ways um which the current chatbots are very good at and which is kind of a different thing than uh uh you know for example ensuring that it can't be jailbroken um which is something that we're less good at so yeah i don't know i think overall i think that the idea of well we actually don't know how to steer ai systems in the first place so we're not very good at it is a more intuitive way to put the problem than uh we don't know how to align them
Speaker 2um yeah but i think no term is perfect my impression is that there's been a big shift in ai discourse in congress over over the last year or two like i guess i just again every week or every week or two i feel like i see you know a new conversation about how to steer ai systems where it feels like you know either members of the house representatives or senators are asking questions that feel like they're almost out of the pages of like some miri report or they have like a very yudkowskian uh energy to them where they're very worried about super intelligence or saying you know you're planning to build super intelligence like how is that going to go i couldn't might we not lose control yeah am i just uh getting a kind of bias selection of of things or or has there been a bit of a or like are at least some people starting to get more open to that to that worry
Speaker 1yeah i think the really huge shift that i saw was after the release of chachi bt so chachi he came out late 2022 and in 2023 and 2024 there was definitely a huge uptick of um uh you know you could say like ai being like the main character on capitol hill or like being you know a huge huge topic of discussion um in a way that really hadn't been before um it is interesting i think you're right that over the last sort of six or twelve months there has been more discussion um of sort of more of the super intelligence type concerns i i don't know exactly why that is i wonder if part of it is just a sort of a feature of the discourse getting a little bit more sophisticated like i do think in 2023 there was just this basic like getting up to speed of like what is ai what is going on with ai how should we think about ai at all in the first place um that a lot of people not just members of congress a lot of members of the public um needed to do and now i mean even things like uh you know i think i think our ability to think and talk about ai has just gotten more sophisticated over the past few years even things like um you know ai 2027 and ai as normal technology being published these two sort of different visions um is very helpful for kind of crystallizing and um making concrete different kinds of ideas and different um kinds of disagreements so i guess that would be my my guess um at the same time i guess we've also seen some of the ceos getting more explicit about the like level of disruption they expect so certainly dario amadei at anthropic has been outspoken about you know job loss and things like that um sam altman at open ai has written you know that they they think they're on a path to super intelligence and so i i think there's probably also a sort of normalization and an ability to ask about these topics in a way that wasn't there when, you know, these sort of high profile figures hadn't kind of talked about it in their own words.
Speaker 2Some people have had the impression that AI progress is slowing down. At least some people have had the impression that GPT-5 was a bit off track. It was kind of disappointing. Do you think it is slowing down?
Speaker 1I don't really buy it. It's really hard to say because our metrics are so bad. We're really bad at measuring progress in AI and knowing what it is that we're even trying to measure. That being said, I sometimes think that, or it often seems to me like people who started paying attention to AI after ChatGPT, their sort of subjective impression of kind of what's going on in AI is like, so nothing was really happening. There's, you know, there's my little chart with an X axis of time and the Y axis of how good is AI. Nothing's really happening. And then suddenly ChatGPT, like big leap. And so for those people, that was kind of, that was pretty dramatic, pretty alarming. And the question was, okay, are we going to see another big leap in the next couple of years? And we haven't. And so for people whose expectations were set up that way, it looks like, oh, okay, we had, it was just this one-off big thing. And like now back to normal, like nothing to see here. I think for people who've been following the space for longer, it's been clearly the sort of pretty steady upward climb of increasing sophistication and increasing ways. And I think if you've been following that trend, that seems to have been continuing. Like basically since 2012, when deep learning really started to work, there haven't really been huge breakthroughs. Like the biggest ones are things like the but that was, you know, that was like a new architecture of deep learning. Like, yes, it was a really big deal, but also was kind of building on like LSTMs and RNNs and this like longer tradition or like, you know, making internet scale pre-training work was like, that's a big deal, but it's also, it's not like a huge breakthrough. It's not a big paradigm shift. And so I think we've kept seeing this level, you know, the reasoning models coming out is another example of like, in one way, very big change in another way, it's sort of just building on the paradigm we had. So my kind of zoomed out, out view is that it looks to me like GPT-5 is sort of continuing on that trend. And I think that trend to be clear is like one that should make our eyes go wide. And like, we should really pay attention to is like, wow, this is a huge amount of technological progress happening in a very short time. It's just not the insanely dramatic level that people might expect if they had sort of just noticed the chat GPT-1. So, and we also don't know for any given step on that trajectory, we don't know sort of what real world effects will result. So I tend to think that it doesn't really seem to be slowing down. It also doesn't seem to be obviously speeding up. So I think also you saw some, you know, some commentary from people at the start of this year, maybe when the reasoning models were sort of being demonstrated and you had that first kind of '01 to '03 jump. I think some people said, oh my goodness, like we're going to have really, really advanced systems by the end of 2025, end of 2026, like even more so. And that hasn't been borne out. So I do think it makes sense to kind of shave some of the sort of fastest progress scenarios off the top of your, you know, confidence interval. But that doesn't mean that things are just sort of plateauing or hitting a wall. Yeah. I think people are getting a little
Speaker 2bit tricked because the releases come so much more frequently now that often every, you know, a release can feel a bit disappointing or only incremental, but you've got to remember the last model came out maybe only three months ago. Yeah. So that's like a big, it used to be like, you know, more periodic releases, the jumps were much bigger, much more stark.
Speaker 1Yeah, I think that's right. People have sort of seen the, you know, there was GPT-4 and then sort of 4.0 and 4.1 and 4.5 and different sort of updates to 4.0 and that's just in the open AI space, let alone the different kind of Google Gemini and the Claude versions. So yeah, it's sort of more of a like drip, drip, drip. Epoch had a really good chart. You've probably seen this of taking some sort of illustrative benchmarks for the GPT-3 to 4 and GPT-4 to 5. And they're different benchmarks because, you know, they basically saturate, but showing like in both cases, there were really significant jumps on sort of key benchmarks of the time. And it's only because, you know, if you're comparing GPT-5 to, you know, whatever came out six months ago or three months ago, that's what makes it look less impressive,
Speaker 2like you said. You wrote in your notes that there's this funny phenomenon that there are kind of multiple different camps that have self-consistent, very different worldviews about AI. And they both feel like they can explain satisfactorily kind of all of the empirical observations that we're making about how things are progressing. Explain how that can be.
Speaker 1Yeah. So this came out of a workshop we ran at CSET, which we're going to have a paper about, hopefully in the next couple of months, on automating AI R&D. So this is sort of ideas related to intelligence explosion or recursive self-improvement or, you know, at what point are the AIs the ones doing the AI research and things maybe take off or maybe don't. And something that was really interesting at that workshop, we had people with pretty different views there, and it was surprisingly difficult to get them to come up with different predictions for what would happen in the lead up to potentially very automated research. And part of why this is, is that these sort of different worldviews, I think, have good ways of explaining contrary evidence and why it's going to converge back to their expectations. So to go one by one, a worldview that is expecting that you're not going to get very superhuman systems, you're not going to be able to fully automate things, you're not going to be able to, you know, have this sort of self-reinforcing dynamics, that worldview expects multiple things at a time. It's not going to be able to fully automate things. It's not going to expect bottlenecks to arise. So if one thing gets accelerated, then it'll get sort of held back by some other factor that can't accelerate the same way. Expecting plateaus on ability. So, you know, maybe the AI system gets much better, but it's, it can only get so much better. You know, it can only hit a certain level of like, you know, biology skill or like coding skill or something like that. And so for that worldview, if you see evidence that things are going quickly, or that things are becoming more automated, your expectation is just, okay, so they're going to hit the limits quicker. So like, sure, if you're going to hit the limits quicker, you're going to hit the limits quicker. It's going faster now, but like, we know that the limits are there. And so it's just going to hit them faster. On the flip side, people who expect there to be these like really dramatic, you know, rates of increase in, in AI automation or AI getting much, you know, far superhuman at different things, when they see sort of bumps along the road or hiccups or difficulties, they are often able to explain them in terms of, oh, this will just lead to more speed later. So like an example that I've seen is if you look at the different kind of agents, have you seen the agent village? I love this thing. So if you look at, so the agent village is this nice demo of different agents being put in a, you know, computer use environment and asked to do some tasks. It's very fun. I highly recommend for anyone who hasn't seen it. But if you look at them, they have a lot of the agents in question, which is, you know, some of the best models, Gemini 2.5, and they must be doing with GPT-5 at this point. And, you know, Claude 4, they struggle with some really basic like computer use tasks, like clicking an interface. And so, you know, if you look at the different kind of agents, figuring out that something isn't working and someone who has a view of, well, everything's going to go very fast, looks at something like that. And instead of saying like, oh, wow, actually the models are less good than I thought. Sometimes they'll say like, well, that just shows that like the UX, the UI that the models are being presented with isn't very good, or they haven't been trained enough on like how to get the pixels right for their clicking or whatever. But once they get that, then we'll have even more speed up. Or similarly, you can see this with compute of like, well, you know, if they're going to be limited by compute, then that just means that once we do build more compute, they're going to really take off. So it's kind of disheartening because it sort of means that you have less, it's actually surprisingly difficult to get some kind of agreement on if there's some potential point where things might get totally insane or might just not.
Speaker 2What would we see in the lead up?
Speaker 1What would we see in the lead up? Are we going to see anything that's actually going to distinguish? So yeah, I think that there are still some ideas, but that it's almost like there's sort of, you know, basins that both views are drawn to. And even if they see a little bit of a roll towards the other base and they just have such a heavy, you know, gravitational pull towards their own view in ways that I think, you know, are fully self-consistent, that it makes it really hard to distinguish.
Speaker 2So I think around October last year, OpenAI said that it was going to maybe kind of spin off its nonprofit and basically disempower the nonprofit, not allow it to have much control over the business anymore. I think in March this year, they did a seeming about phase where they said, no, the nonprofit is going to retain control. But I guess I did an episode around that time with Tyler Whitmer, where we explained that, or at least he explained that he was pretty skeptical. And he thought, in fact, the nonprofit was going to lose substantial control, maybe like almost all significant control. Do you have any thoughts now on OpenAI's restructure efforts and what we should think of them?
Speaker 1Yeah, I mean, I haven't followed this as closely as Tyler has. A couple of thoughts. So I think it was really good that they made that sort of seeming about phase. I think it was clearly the result of pressure from sort of external groups. There's various sort of now like a mini. Ecosystem of OpenAI watchdog groups and also, you know, pressure from the attorneys general, which is really appropriate because the attorneys general are sort of legally the parties that are responsible for ensuring that a nonprofit carries out its nonprofit mission. I think there's still a lot of open questions about what that restructure is going to look like, and I'm not particularly encouraged by the most recent update they've put out about the nonprofit getting a 20 percent stake in the PBC, the for profit. So I think there's very, very, yeah, there's very, very open questions about they say the nonprofit will retain control, but there's a wide range of possibilities for what that might look like. And a key feature of the current setup is that obligations to the nonprofit mission are sort of the primary legal obligation on board members who are ultimately responsible for the operations of the whole company. And I think trying to retain that setup where the nonprofit mission, which is to ensure that AGI benefits all of humanity, that that remains. As sort of the primary legal obligation, as opposed to being, you know, something that a small fraction of shareholders are pursuing or. Or something that, you know, the typical way a PBC, a public benefit corporation works is that the PBC is allowed to pursue both missions. It can maximize profit or it can choose to pursue a socially beneficial mission, which is just much, much, much weaker. So this question of, you know, what does it mean for the nonprofit to retain control? I also think I just really hope that the attorneys general, when they're looking at this, are thinking really hard about how how it can be that so many of the same people are on both sides of this deal. So in particular, you know, the board members, perhaps especially Sam Altman, you know, they're negotiating with themselves essentially for like who is getting what is the nonprofit retaining, who ends up with board seats, who ends up with equity. I think it's a really tough transaction to do at arm's length. And.
Speaker 2Which is legally required, right? It's meant to be an arm's length transaction.
Speaker 1As I understand it. Yeah. And so I just hope that the attorneys general are really looking very closely at what the company is telling them about who was recused and who was involved in decision making and how they're ensuring that the nonprofit's interests are actually truly being represented.
Speaker 2Yeah. I mean, I guess being more familiar with UK charity law, it's kind of mind blowing to me that the kind of the effective trustees of this charitable organization are selling the assets of that organization to themselves. Yeah. I almost don't know what more to say than that. But it's like it's extraordinary to me that you could in any sense be on both sides of a transaction like that. In the UK, I think it would never be permitted.
Speaker 1I mean, we don't know what is going to be permitted yet. They haven't got final approval. So, yeah, we'll wait and see.
Speaker 2An audience wrote in with this question. I've heard Tona described as extremely normal in a complimentary way. Is this true? And if so, how does she manage being normal around so many non-normal people? I'm not sure who the non-normal people are being referenced here. I guess maybe me. I don't know.
Speaker 1It's very challenging being normal around you, Rob. I don't know if I'm normal. I don't think of myself as very normal. I do think of myself as enjoying like translating between groups or something. Like both in a literal way of like I've always loved languages. I speak German at home with my husband, who's German, and our kids. I loved learning Chinese when I lived in Beijing. But also in a like in a more metaphorical way. A metaphorical way translating between, you know, like the weirdo existential risk community in the Bay Area and like the more sort of staid national security community in D.C. I find it really fun to try and understand like what are the different perspectives coming in here? What are the assumptions? What are the social norms? So I think that is more how I would describe it. Definitely there are ways that I feel, yeah, I guess I don't know exactly what is meant by normal versus weird. I certainly identify as a... At least somewhat weird, depending on the social context I'm in.
Speaker 4I guess in the Bay Area, there's like, there's degrees of weird and I think you're not weird by Bay Area setting.
Speaker 1No, that's right. But I really loved living there and being in a social setting where, you know, weirdness was totally accepted and not judged. That's one of the things that I miss about living in the Bay for sure. Actually one other thing that makes me think of is probably people listening who've heard me speak before are confused by my accent, which happens a lot. Maybe related to the translator thing. So I'm originally from Australia, as you know, as are you, and usually I, you know, sound American in, when I'm speaking with Americans, I, you know, basically at some point after living in the US for multiple years, it just felt both sort of more natural and more fun to speak with an American accent. But when I'm around Australians, it feels really artificial to keep speaking in an American way.
Speaker 2So, so are you conscious of your accent changing? Because I think when I talk to Australians, my Australian accent comes out much more like, I've lived in Britain for a long time and I think my Australian accent has really weakened, but I think it probably has come out a little bit more in this conversation than usual, but I'm not aware of that.
Speaker 1I mean, if I listen to it, then I can hear it. It's like partially, I would say it's like somewhat conscious and it depends on the setting that I'm in. But yeah, I mean, I think for some people, some people seem to find it weird or like, I think some people really identify very closely with their accent as like being a key part of who they are and I've just never really felt that way. I mean, maybe it's like as a kid. I enjoyed like putting on, you know, different accents and learning about them. I think also these days, because I am like speaking German every day at home, I'm like code switching anyway. So it's not like there's my real me and then there's, you know, fake something. It really just feels like there's just different ways my voice can sound. But yeah, I definitely get a lot of people being like, wait, what's going on with your
Speaker 2accent? Do people immediately know that you're Australian just in normal life in the US?
Speaker 1If I sound American, then no. No.
Speaker 2Okay. So are people usually, are they usually able to give up their accent to that extent or change their accent to that extent?
Speaker 1I don't know. I mean, when I speak German, people also tell me that I sound German. So yeah, I don't know. I feel like there's a wide range. Like another, you know, a more similar thing that I think quite a lot of people do is Americans from the South will pretty often sound like more Southern around family and friends. And then when they're talking to Northerners, they sort of go towards more of a standard American accent. So I think there's lots of versions that, you know, different. I don't know, communities or cultures like handle it differently.
Speaker 2Yeah. Another audience question was, this field is changing so rapidly, both via technological improvements and via shifting global geopolitical relations that affect the semiconductor supply chain. How do you keep up with all the stuff that's going on?
Speaker 1I mean, it's a challenge. This is my full-time job and it's hard to keep up with things. I do think that, you know, Twitter X is still quite useful as a source of news. For better or worse. I don't know. I have a, have a sub stack, you know, a set of sub stacks that I get in my inbox. I don't know. I don't know. Some of the ones that I get in my inbox that I try to read, some of them, I, you know, try really hard to read every post, other ones I just sort of skim and, and carry on. And you can go to my sub stack, so helentoner.substack.com, there I have a list of the recommendations of different, different sub stacks that I find really useful. If you want a lighter touch way to catch up, then CSET publishes a monthly newsletter about kind of the biggest AI news, which I actually find helpful as sort of a roundup because there's, it's like, so it's like a fire hose day to day. And then once someone that's like, okay, here are actually the big, big stories that, that really matter.
Speaker 2I guess I read the, I think Transformer has a weekly update on Friday. I think I can probably recommend that one if you want like a once weekly sense of what's
Speaker 1going on. Yeah. But I think it's hard. And I think, I mean, something that I've kind of accepted working in the space that I work in is that like, I'm in the intersection of many different fields. So there's AI, there's national security, there's China, there's US politics, US policy more broadly, big tech regulation, platform regulation, social media, economic, you know, things like that. And so I've just accepted that I'm never going to have the, I'm never going to be an expert on every single thing that touches my work. And so it's just always a matter. There's never like perfect. I'm never like, oh, I have to stay up to speed. How am I staying up to speed? Yes or no. It's always just a matter of prioritization and like trying to choose where to put emphasis and where to make sure that I'm sort of spending time, work in progress.
Speaker 2How do you keep up to date or try to have some sense of what's going on in China? It seems particularly hard to gauge.
Speaker 1Yeah. I mean, so a few different answers. I do think the best substack that I know of on this is China Talk by Jordan Schneider, which is really good. CSET actually has a tool called Scout. So if you go to our public data platform, which is called the Emerging Tech Observatory eto.tech, there's a tool called Scout, which does auto translated and summarized, but then sort of human curated news from China, which is very helpful. And then yeah, lots of other sort of miscellaneous sources. I do think it's helpful. One thing I will say, you know, if you're interested in China and AI, I think it's really worth sort of laying some foundations of China and non-AI topics. So whether that's like, you know, reading The Beautiful Country in the Middle Kingdom, which is a sort of comprehensive history of the US-China relationship or learning about the history of the Communist Party or looking at, you know, some of the history of sort of tech development and the economic development in China generally since the 80s. I think it's really valuable. I think people sometimes come in to this topic sort of purely from the AI angle and only know about China. And so I think it's really helpful. And I think it's really helpful. I think it's really helpful. I think it's really helpful. And I think it's really helpful. And I think it's really helpful. And I think it's really helpful. And I think it's really helpful. And I think it's really helpful. And I think it's really helpful. And I think it's really helpful. And I think it's really helpful. And I think it's really helpful.
Speaker 2And I think it's really helpful. And I think it's really helpful. And I think it's really helpful. And I think it's really helpful. And I think it's really helpful. And I think it's really helpful. And I think it's really helpful. And I think it's really helpful. And I think it's really helpful. And I think it's really helpful.
Speaker 1And I think it's really helpful. And I think it's really helpful. And I think it's really helpful. And I think it's really helpful. And I think it's really helpful. And I think it's really helpful. And I think it's really helpful. And I think it's really helpful. And I think it's really helpful. And I think it's really helpful. And I think it's really helpful. And I think it's really helpful. And I think it's really helpful. And I think it's really helpful. And I think it's really helpful. And I think it's really helpful. And I think it's really helpful. And I think it's really helpful. And I think it's really helpful. And I think it's really helpful. And I think it's really helpful. And I think it's really helpful. And I think it's really helpful. And I think it's really helpful. And I think it's really helpful. And I think it's really helpful. And I think it's really helpful. And I think it's really helpful. And I think it's really helpful. And I think it's really helpful. And I think it's really helpful. And I think it's really helpful. And I think it's really helpful. And I think it's really helpful. And I think it's really helpful. And I think it's really helpful. And I think it's really helpful. And I think it's really helpful. And I think it's really helpful. And I think it's really helpful. And I think it's really helpful. And I think it's really helpful. And I think it's really helpful. And I think it's really helpful. And I think it's really helpful. And I think it's really helpful. And I think it's really helpful. And I think it's really helpful. And I think it's really helpful. And I think it's really helpful. And I think it's really helpful. And I think it's really helpful. And I think it's really helpful. And I think it's really helpful. And I think it's really helpful. And I think it's really helpful. And I think it's really helpful. And I think it's really helpful. And I think it's really helpful. And I think it's really helpful. And I think it's really helpful. And I think it's really helpful. And I think it's really helpful. And I think it's really helpful. And I think it's really helpful. a lot of machine translation, but for important documents, it's also very helpful to have a human editor go through and make sure that those key terms are translated appropriately. You know, that all the weird CCP jargon is like, you know, using the correct terms so that you can connect it to past documents. So I think that is, is a big asset as well. And yeah, having a, you know, a team that is focused on AI and emerging tech and not just sort of a small part of a larger think tank, but, but really having the whole organization dedicated to those questions.
Speaker 2Yeah. The conversation is more crowded now. I feel like, unfortunately there's more and more actors who are being paid by vested interests to push a particular line, to push a particular agenda. And I guess, I mean, you can't blame tech companies for having an interest in what's going on in DC and hiring lobbyists to make the case for the things that they want. I mean, I guess one that stands out to me is NVIDIA's lobbying around the export controls. They've paid people to say some sensible things. They've paid people to say some things I think are extremely not sensible and not, not, not always true. Do you feel like the conversation is getting a bit degraded or a little bit corrupted by the fact that there's now people with like clearer agendas, a clearer ideology as well, who are
Speaker 1trying to have their way with things? I mean, I definitely think you're seeing that the, the companies are staffing up and they are becoming increasingly active on AI policy topics. And I think that that has, yeah, has pretty predictable effects in terms of pulling things towards the company's interests. So yes. One counterpoint I would give is, I don't want to, you know, as we record this, this is not yet finalized. But in California, there's this bill called SB 53, which I think is kind of an astonishing triumph of like sensible policymaking over, you know, distorted rhetoric. This is the successor bill to SB 1047, which was enormously controversial last year, subject of huge amounts of rage and fear mongering and hate. And what happened in the aftermath of 1047 was that Governor Gavin Newsom vetoed it, but set up a commission to sort of study what would be the appropriate way to regulate if, you know, and how concerned to be about the kinds of issues that SB 1047 was designed to help with and created an expert commission. The expert commission wrote a report that was really high quality. And Scott Wiener, the California Senator who, you know, put up SB 1047 last year, put up a new bill that was drawing heavily on this report and has gotten a lot of support and a lot of, a lot more consensus. And it's passed through the legislature. And as far as I know, Governor Newsom is sort of indicating that he will sign it. Don't want to get ahead of the game there. But it's been massively less controversial. It's been massively less controversial. And I think will be certainly a smaller step than SB 1047 would have been in both good and bad ways. I think the potential ways would go wrong are much lower, as well as, you know, potential benefits of 1047 had really worked out the way that its supporters hoped it would also lower. But I don't know, I put that out there as an example of, I think there is much more lobbying in the space, there is much more sort of corporate influence, and that is probably going to stick around. But there are still ways that kind of good policy can be made. And I mean, I do also think of that as a big value add that CSAT can provide is trying to, you know, often, policymakers talk about, you know, you know, you know, you're Policy makers talk to companies because the companies can tell them, you know, what is real and what is realistic and how the technology actually works. And you don't want to, you know, do a Europe GDPR of creating all these rules that have all these unintended side effects because, you know, they're disconnected from how the technology actually works. So it makes sense that they want to talk to industry lobbyists if they're going to have sort of that real talk for them. And that is a place that CSET can step in as well and organizations like us and have that technical depth to be able to go in and say, here's what's realistic. Here's, you know, potential unintended side effects, but not be sort of ultimately promoting the company's interests instead, you know, having a broader public interest in mind.
Speaker 2Do you have to worry about CSET's research being influenced by the opinions of its funders and maybe wanting to pander to them too much?
Speaker 1I mean, I think any organization that, you know, every organization is. Has to pay attention to that. Yeah, it's dependent on something. We work really hard to bring in funding that is sort of compatible with our model of following the evidence where it goes, being independent, not pursuing a certain agenda. So, you know, when we take industry funding, which was a very small amount of our budget, we make sure that that is really sort of cordoned off in a way that makes it very difficult for them to, you know, certainly as with any funder, totally out of limits, off limits to. Have that influence any of our findings were especially cautious about that with industry, but also with, you know, philanthropic funders, you know, we have a large amount of funding from open philanthropy and we also work really hard to make sure that our research and findings are not just trying to show whatever we think that they would want us to show, but instead is what we think is true. And, you know, that's really what has built our reputation as well. So at this point, something we hear regularly from funders is that is why they want to support us, because they know that we are going to do that. We are going to make sure that we are not, you know, going. Where our sort of, you know, financial backers, you know, might be inclined to go, but we're going to really be independent and tell it like we see it. And so that is a value add in itself.
Speaker 2Yeah, I think people can often tell when you're being paid to push a particular line because it's like the conclusion will always be the same, no matter what the incoming evidence is or what the updates are. And I think people tune out.
Speaker 1Yeah, yeah, I think that's right. And I think, I mean, I also think even if it's not a financial thing, I think there can be ways in which in AI debates. For example, if people have the feeling that your conclusion is going to be risks are really high, we need really, you know, severe interventions. If they feel like that's going to be your bottom line in every case, on every paper.
Speaker 2And you're not curious to change your mind or learn something new.
Speaker 1Yeah, yeah. Then they also tune you out. And I think that is also, you know, a strength that we bring is not doing our work that way.
Speaker 2Are there any particular talent bottlenecks in DC? Or like, what are the skills or knowledge that you could bring into the conversation? Have people work at CSET or wherever else that would improve people's understanding? That would actually like push things in a, in a, in a like smarter direction?
Speaker 1Yeah, I think there's all kinds of things. I think having a basic understanding of AI as a technology, which doesn't necessarily mean, you know, a computer science PhD, I think, but having, um, being comfortable enough with calculus and linear algebra and a bit of programming to, you know, have played around with models yourself and have a sort of basic intuition for how they work and then bringing some other kind of expertise, like really any kind of expertise. I think something that is a bit overrated in AI is. Purely computer scientists coming in, um, and combining that with, you know, and, and learning the policy side because AI touches so many different fields. I think if you're an economist, if you're a historian, if you're an anthropologist, if you're a, you know, an accountant, if you're just, just so many ways, if you're a teacher, um, I do think there's, there's a lot of space for very wide range of different kinds of expertise. I do think it makes sense in as much as you can to try and get that sort of basic understanding of AI, largely. So it doesn't seem like magic, um, trying to get deep enough on the technology. So that you have a rough sense of how it works. Um, but beyond that, I think there's, there's a huge range of, of needs.
Speaker 2What's something that you think a lot of listeners might get wrong about the AI policy discourse in DC?
Speaker 1I think one thing could be thinking that policymakers are really, um, uninformed or sort of stupid or like, um, on AI issues. I mean, certainly different policymakers interact with these topics differently.
Speaker 2But I think is, is this a widespread perception, do you think?
Speaker 1I think a lot of people think that policymakers have no idea, um, and are really ignorant about tech topics. Um, I think it's really important to remember two things, you know, both just how busy policymakers are and how, depending again on the policymaker, how many topics they might need to, to cover and how many issues, including often just like truly urgent, you know, crises they need to handle. And so, uh, if they like don't know something or misunderstand something, it doesn't mean they're stupid. It means that, you know, they're busy. Um, and I think another thing that people underestimate is how, um, how insular some of the conversations and some of the sets of ideas are where from the perspective of many people in DC, they've just never really encountered an argument about, for example, you know, why export controls would be good. Um, or, you know, why we should expect AI to develop in a certain way. Cause so many of these conversations happen in these pretty self-contained, um, social groups or sort of online networks. Um, so I tend to think there's a huge amount. A huge amount of value in trying to create more like, uh, accessible, no prerequisites required introductions to concepts or explainers of key concepts. We do quite a lot of explainers at CSET, um, as well that bring people into these sort of key ideas and let them sort of understand them and, um, play with them in their own head, as opposed to just having them be discussed sort of, uh, you know, verbally in conversations among, among friends.
Speaker 2I guess you engage a lot and I think very productively with people like across the full spectrum from being, I guess, very optimistic about AI. To being very doomy, very, very, I'm troubled about the, the future. What sort of, is there any particular evidence you think that could help to settle in your mind or give you a big update as to like what, how worried to be?
Speaker 1I think probably no one single piece of evidence, but, um, different things that we could see over the next few years. I do think this, um, yeah, I guess two things we've, we've touched on, but to say them again, um, one is this question. Of how much is AI going to be a tool versus how much is it going to be really sort of a gentic autonomous out there kind of doing its own thing. Um, that's something I expect we'll get evidence on. I think this year the, the updates have been mostly that it's turning out to be more challenging than people thought. expected to have AI agents that can even do pretty basic things like booking flights reliably. I'm sure they'll get there, but it's turned out to be a little bit more challenging than some of the most bullish people are expecting. But I think the more that we see AI systems that can really go out and do in practice on a regular basis, sort of longer time horizon tasks, or also like messier tasks. So things that aren't as sort of cleanly boxed and specified as like, here is a nicely chunked out thing for you to do, but instead here's a, you know, a squishier thing. Can you, can you make some progress on it? That's one big piece. And then another piece is this question of how super intelligent things can get. And I think, I think there's starting to be a little bit more discourse on this, a little bit more looking for sort of evidence or breaking down different types of capabilities of how far above human level could you get on a given thing. I would love to see more of that because I think, yeah, I think, I think assuming that things will plateau at human level seems clearly wrong, but also the sort of, you know, Bostrom is in super intelligence. He talks about these like superpowers of like the AI will have superpowers at all of these different things. And that also seems pretty unlikely to me. And so trying to figure out kind of where in the middle we will land seems really important. So is CSAT hiring for any roles that
Speaker 2people in the audience, if they want to dive into this world, or maybe already like someone in this world, but are considering moving to CSAT or DC should think about? So depending exactly when this
Speaker 1goes live, we're hoping to have a post up pretty soon for a research fellow or senior fellow. So this would be a, so focusing on frontier AI. So this would be a role to lead research at CSAT would need someone with a graduate degree, a few years of research experience at a minimum, maybe, you know, multiple years, maybe some government experience for a senior fellow position. I think this will be a really exciting role. We'd love to see lots of really great applicants. If that role is closed, or also if it's, you know, not the right role for you, then we'll likely also be hiring for additional research roles, data roles, sort of communications and external affairs roles, potentially operations roles. Sort of through 2026. So definitely keep an eye on our careers page, sign up for our newsletter. Stay, stay in the loop.
Speaker 2Is CSAT growing in general?
Speaker 1I think we're roughly happy with our size. So we were, I think the, in 2019, the interview was about the 30 person research group in DC. We're now more like 50. We may fluctuate a little bit around that. But I think pretty happy with the size. It's a nice, I think a big part of CSAT's model is being able to sort of have a cohesive team culture where people are collaborating across teams. And there's sort of a sense of, as an organization, we're able to have a cohesive team culture where people are collaborating across teams. And there's a sense of, you know, as an organization, what are our sort of key priorities, which gets difficult if you're, if you're scaling up much beyond that, that level.
Speaker 2The last, the last six years since Conversation in 2019 have been pretty eventful. I can, I can only imagine the next six years are going to be as eventful or more so.
Speaker 1And talk to you in 2031.
Speaker 2Hopefully we get to talk before 2031. But yeah, my guest today has been Helen Turner. Thanks so much for coming on the 80,000 Hours Podcast, Helen.
Speaker 1Thanks, Rob. you
PodcastTranscript AI