Is It Already Too Late to Control AI? (Anthropic Co-Founder, Jack Clark)

Thanks for listening to the Restus Politics, sign up to the Restus Politics Plus to enjoy ad-free listening and receive a weekly newsletter. Join our members' chat room again early access to live show tickets. Just go to therestuspolitics.com. That's therestuspolitics.com. The world we're in is one where AI companies are building the equivalent of nuclear power plants and we just had our first case where we upgrade the plant and nuclear bomb also falls out of it. You say, "Uh oh, this has some implications." Are we in a world in which we're essentially saying that a bunch of voluntary good guys build these amazing weapons and then they just choose not to release them but nobody can tell them what to do. We're entering an era where you actually need to do serious coordination including between governments to keep having all gas no breaks. It's not sustainable to keep us going. We have to figure out that coordination both within the industry and maybe more importantly and more challengingly with China. If everyone gets access to it, you're now rolling like a whole bunch of loaded dice with with the lives of people around the planet. We have to somehow force a larger conversation on what do we want to do with this technology and how do we want it to be developed. This episode is presented by IG. So whether it's the vetting of senior political appointments or it's costly policy, you terms the last few months of UK politics have been a masterclass and what happens when you don't do your own work. And the consequences, as we've seen, ripple out far beyond the initial decision. And the same rules apply when choosing your investment platform because picking the wrong one can hit you directly and in a very measurable way. Some investment platforms charge hundreds of pounds more a year than others losing you money that never got the chance to grow. With IG, there are no annual fees, meaning more of your money stays invested from day one. And on top of that, you pay zero commission on stocks, shares and ETFs. So there are no hidden charges quietly eating into your returns. And even better, IG has been trusted by British investors for over 50 years. So you know you're in good hands. Search IG.com to find out more. IG. Trade. Invest. Progress. Your capital is at risk and other fees may apply. [Music] Hello, this week on leading, we're bringing you an interview with Jack Clark, the co-founder of Anthropic, which people will know from Claude, one of the great AI companies. That's the world, one of the fastest growing companies in the world. I'm one of the most significant voices and leaders in AI today, Jack's Brit. Now, as some of you might know, Matt Clifford and I have been doing a mini series for our Trip+ members on AI over the past few months. We've got huge feedback. I still think AI is the single biggest issue in all politics anywhere in the world, more important than immigration, housing, cost of living or anything. And I'm hoping to draw you into that conversation in this conversation with Jack because I think it's a great way of engaging with some of the big questions in AI, safety, jobs, employment, the future shape of the economy, and how on earth people think about a technology that could transform the world in really powerful ways for good or ill. So, I hope you enjoy it, Rory and Matt talking to Jack Clark. Welcome to the rest of this politics AI with me, Rory Stewart, me Matt Clifford. And we are very lucky to have with us today Jack Clark from Anthropic. Thanks for having me. Anthropic is one of the very largest fastest growing companies in the world, oriented around AI. The product which many people will have engaged with is Claude, but they've also released Mythos, they've had a huge fight with the Department of Defense, Strict Department of War. Jack is one of the co-founders along with Daria Amode and we are very lucky to have him with us today. And rather unusually in the world of Titans of Silicon Valley is his himself British. So welcome and thank you for joining us. Absolutely, it's great to be on a British podcast. Thank you. Can we start by just take us back to your Britishness? I mean, how British are you? Where did you grow up? I didn't think he was you that would be in the purity test. I think my credentials are pretty good here. I grew up in Brighton. My mother was a nurse. My dad was a grumpy guy from South Shields who had become a copywriter and went to state school, primary secondary college. And then I went to the University of East Anglia. So I've been English and being reigned on for my formative years. And what did you study? I studied literature and creative writing, which was a bit of a curveball because in school I did very well on the sciences and I got degrades in English. So obviously I was like, you know what this is telling me? I'm going to make my fortune with creative writing and studying literature. But I'd always been a huge reader. And I was very interested in going and getting an education, which was actually very much oriented around self-study and reading and grappling with ideas through the form of fiction, which is my main hobby and always has been. And when is this when you go into university? This would be 2006 or so. So let's get a bit of a sense as well. So this is the moment at which I get Twitter and Facebook are just getting off the ground. The iPhones just about to be launched. So you're right there and sort of going to university at a moment where those sort of things are explaining. And at some point you become kind of interested in technology. You don't become casual issue group. And instead you set off in another path. So I'd been reading science fiction from when I was about the age of 11 or so. And I'd always been very interested in studying different parts of science from, you know, how Ancolonies works to, I briefly wanted to be a town planner so reading about cities and trying to model out cities with very basic computer programs to fractals and everything else. And I had this memory of when I was about 14 writing a story in which Ants simulated in a supercomputer eventually tried to bootstrap fairway out of a simulation by building like an AI to break from out. And then I read a story by the writer Greg Egan, who can say, sounds remarkably like, which is about crabs in a computer building technology to break from out. So I was like, damn it, he's done it too. But it was because I was, I was just already at a young age very obsessed with technology and what it meant. What I spent my time reading and writing about at university was what kind of stories can be told grappling with the implications of technology. And how do you make it interesting? Because technology breaks a lot of narrative a bunch of ways. Do you expect it? Because you became a journalist straight out of that. Yeah. You mentioned creative writing as a hobby and for those of us who read your newsletter, we kind of get to experience this every, every week. But was that always going to be the side hustle and the journalism was the core? I had no illusions about my chance of making it as a fiction writer of short stories in the mid 2000s. I wanted to be a journalist because I was obsessed with technology and studying it and writing about it. I felt like the most important story in the world was what was happening, even back then, with things like data centers and the build out of them. I became a technology journalist in 2009. And I remember, you know, I was reading papers about databases that were being used by Google to train machine learning models. I remember talking to the people that made that, that software that you click on that shows you're not a bot on Google where it says click over bicycles. I interviewed them in 2009 and I said, this is very interesting. When are you getting bought by Google? And they were bought by Google three months later. So I felt like I was on the money. So one of the things you did as a journalist is you infued another great celebrity Brit in the world of AI. You interviewed Demis the Sabbath. Yes. Yes. And you got a bit of a scoop. Tell us a little bit about what he said to you then. Yeah. So I moved to Silicon Valley in 2013 to write about artificial intelligence. I'd become obsessed of it. It was clear to me that this was where the research was happening. In Silicon Valley, I started attending conferences, reading about neural networks, reading about deep learning. And pretty soon after that, you run into someone like Demis Hersalves who at that time had a small startup called DeepMind that no one had heard of. But shortly after DeepMind was bought by Google in I suppose 2014 or so. I had gone to a conference in New York where Demis was. And I introduced myself by explaining to Demis. I'd read a lot of his research papers and sort of quoting them to him, which I think led to him tolerating my my interest as a journalist. And I pitched him and I did a lengthy interview a few months later. And I asked him what he thought should ultimately be done with AI technology if we succeeded in building the sorts of powerful things which are then being imagined. And he said he thought it should be sort of managed by or supervised via the United Nations. But on record in a Bloomberg Business Week story from around that time. It's interesting. I was saying this just before, but I think it is worth saying that arguably the two most successful Brits in the history of Silicon Valley, a Jack Clark, co-founder of the fastest-going company of all time, and Mike Moritz, the Sequoia Capital Partner and founder of Google, among many others. He's not in Silicon Valley, though. He's not the road. He's not abandoned. I'm like, you Jack, you know. And they're both you and my Moritz trends are journalists. Is that coincidence? There's not you as like a humanity's graduate. And just desperate to find out. This would be what my colleagues in Silicon Valley call "Cope." No, I like to think that having a journalist mindset means that you ask a bunch of unusual questions. You're trained to ask questions. You're trained to be skeptical. I've always found that a lot of the work that I've done, which involves supervising technical teams to ask like off-the-wall questions, has been heavily informed by my backgrounds as a journalist where you're just trying to figure out what are the interesting questions to be asked. You know, I in the early days of anthropic I asked, could AI manufacture bio weapons and we set up a team to do that? And now I'm running a team of economists asking, well, what impact will AI or potentially very powerful AI systems have on the economy? I think just by asking those questions, you might get some good answers. Did it feel weird to make the jump when you jumped into working open AI? I feel quite natural by that point. - Open AI is the big competitive anthropic that Chris had chat GBT and that's run by a man called Sam, sorry, back every year. - Oh, I felt completely bizarre. You joined Open AI and I remember talking to Greg and Ilya on the first day and we were figuring out what the whole job was and soon after joining Open AI, I said to Greg Brockman, I figured I'd just start going to DC and doing policy. It was like very supportive of that. And so I got into playing to DC, not knowing how to do policy, but figuring out that it was going to go in the office. - I think was Open AI then? - I was among the first 25 or 30 employees. Like Dario had joined a month or two before me, fire call correctly. He was on my interview loop when I interviewed there. - And what did it feel like then? What was the company like? What drew you to it? - I'd been reading about all of these AI research papers and when Open AI was announced, it's hard for me to describe just how unbelievably stacked that team was and how notorious they were. Not notorious to anyone who hadn't been following it. I was saying to my colleagues at Bloomberg, I was like, "Good Lord, they got Elias Sutskaber." They were all staring at me. I was like, "Oh, and Dario and Moote, and no one." Greg Brockman, no one knows these people. All they knew was like Sam Altman, who was somewhat famous. - And tell us a little bit about those three people so we get a sense of them. Because some of them had actually established themselves doing very remarkable research as academics. - Yes, you know, Elias Sutskaber had been on the team up in Toronto with Jeff Hinton, another famous Brit, who had cracked the ImageNet challenge and Image Recognition Challenge in 2012 by training a system using neural networks on graphical processing units made by NVIDIA, which subsequently became quite important. - This is something Matt told us about in the where we've been doing this day, I started it together, where he explained this lovely moment where suddenly this is incredible improvement in performance. - I at the time had been making charts as a journalist, plotting performance on machine learning benchmarks. And I remember putting the chart in and being like, "Oh my goodness, it's happening. "I have to move to Silicon Valley right now." So, Ilya, notorious for that had written many foundational papers at Google already. Dario had studied some of the early scaling work on things like speech recognition initially at BIDU and then he'd worked at Google and written a paper called Concrete Problems in AI Safety, grappling with future safety issues of powerful AI systems in 2015. And it's hard for me to express how like completely berserk, it seems to be writing papers about the safety issues of powerful AI systems when all you have a computer vision systems, but just barely works some of the time. And then there were Greg Bocchman who was known as just notoriously one of the best most talented engineers in the world. He'd helped found Stripe, which is an amazingly like fast growing big startup now. And there were many, many other names like that. So I, from my perspective, I was like, this is the team, it has a mandate to do try and make powerful AI systems for a benefit of humanity. If there was ever a moment to stack all my chips on AI, it was then. So I resigned from Bloomberg and I, I got many emails that day and comments from people I'd looked up to saying I was making the worst mistake of my life. Except at another level, there must be a lot of people for a jealousy because presumably Silicon Valley is also full of tens of thousands of young men and some young women who are desperate to get in on the ground floor of what's going to be the next turning dollar company. I think even at the time it was not that well known, there were a bunch of adorable misfits and weirdos who believed in what was going to happen with AI, but it was not even that high status or well known in Silicon Valley initially. And actually you were competing against DeepMind, which was already very established. So I was questioning-- - You were already bought by Google by that, right? - Yeah, how are you going to possibly be, you know, build another lab when Google has DeepMind? - And where are you going to get the money from? Presumably becomes a question quite quickly. Once it becomes obvious that these things are quite expensive. - Exactly. You know, OpenAI was announced of a $1 billion funding amount from commitments from Elon Musk and others. And so it was serious about the capital requirements. Now then I get there and you realize there isn't a plan. You're like, what's the plan to build artificial general intelligence? And no one knows because no one has an idea of what to do at the time. You don't have language models. You were running an amazing series of experiments of the company, which were basically all-oriented around ambition. Can we train an AI system that could beat humans at a complex video game? Can we train an AI system to operate a robot hand? Can we train an AI system to end up being able to generate text? Now that last one turned out to be very important. But all of the other projects were important for having an organization that was continually doing unbelievably ambitious things that seemed like outrageously bold goals. And through trying to work on those goals, you built bigger systems where the dev had been built to work for problem. You developed infrastructure that hadn't existed before. You encountered bugs at the frontier of scale, which no one had encountered. And you also started to build up this information of, huh, every time we dump more compute into these systems on these outrageous challenges, they get better. So you started to develop this important intuition that actually we could scale for performance of AI systems, not just through having clever ideas as human researchers, but allocating more compute to the training of them, which turned out to be one of the most important insights for what subsequently has happened. And it's worth maybe just spending a minute on something you already alluded to, which is that, you know, Dario, when he went to work at OpenAI, I had already done this work on concrete problems in AI safety. And actually, at the time, if we think about the context in which OpenAI was founded, it was actually quite oriented around at least safety from the perspective of things like concentration of power and other risks, right? I mean, we've recently, because of this epic court battle between Elon Musk and OpenAI, I've seen a lot of the, you know, founding emails and documents. And there's this big thing of, we don't want Demis to control this technology. And today, of course, anthropic, I think even now, describe yourself as an AI safety research company. Do you want to talk a little bit about the kind of importance of this idea in the founding journey of both OpenAI and Anthropic? Taking safety of AI systems seriously requires you to hold in your head how powerful they might become in the future. And I think that this notion of the technology not being as it is today, but moving very quickly to become a lot more advanced has been both important for motivating the research agendas, because you build a different research agenda if you believe something is about to happen in the future, some change up in scale. But also, safety has ended up being extremely coupled to our ability to deploy this technology. You know, as the technology gets more powerful, deployment into society is gated by, oh, well, if it's good at hacking, how do you make sure that you can deploy the good parts of coding, but not hacking? If it's good at biology research, how do you stop the proliferation of bio-epping capabilities, but how do you allow useful biology to happen? These are very subtle questions that drive very, very complicated research agendas. And so I think thinking about safety allows you to think about what I think of as the most ambitious technological optimist version of the tech, which is where you've been able to deploy it very, very broadly. Can I just develop Matt's point, what one more stage? So from the outside, it feels a little bit as though OpenAI is set up because Elon Musk is scared by a conversation he has with Larry Page and Demis for Sabbath, and he thinks they aren't concerned enough about safety. And he sets it up for Sam Orton. Then the story emerges that Elon Musk is concerned about what Sam Orton's doing. Then another story emerges that Daria, Mude, your friend, and you, in fact, leave OpenAI because you're concerned about safety. So now we have a situation in which it feels as though half a dozen of the most famous names in AI have gone through a journey where each one of them is pointing to another saying, we think your reckless are not safe enough. I would think of it more that the great thing about the world is you get to run a range of different experiments. I view all of these as organizational experiments. There's an organization that was already within one of the large tech companies trying to build AI. There was an organization that stars as a non-profits and then converts itself into sort of a for-profit for capital purposes, which is OpenAI, but is trying to build it. And then after a few years of sitting around working together, I think we realized that we had our own vision for how to build a safety-focused organization. And we viewed it very much as the time to build, run another experiment in Mr. Maine is now, because back in 2020, you can get a sense of how expensive for whole projects of building AI is about to be. Opening I stars in 2016 of a billion dollars, you're sitting there saying, oh, it's four years later, we're going to need a ton of money to do this. So you had to do it then. But I wouldn't claim that it's because you're claiming on our end, oh, this approach is definitely wrong. It's more, we have an approach that we think is going to be subtly different and we want to run the experiment in the most pure way possible. Just a challenge for a second. One of the things we keep hearing is safety. I mean, certainly it seems as though that's what is driving Elon Musk in 2016. He's worried that M.S. Services systems not safe enough. And then there seems to be an anxiety that open AI isn't safe enough. I mean, so somewhere here, and this is one of the things that I noticed when I'm talking to the founders, these companies, and I'm saying, why are you taking these risks or why are you traveling at this speed? Often the answer is, well, I have to get there before this other person. And the other person was off from one of their friends 10 years ago. And I've got to get there before them because they're very dangerous. And unless my models there are ahead of them, we're going to be in trouble. And then it moves on to, and anyway, the US has got there to get ahead of China because they're also very dangerous. So there's a sense that we're in a kind of America's cut race where we're speeding along and the storm's coming in and the winds and the sails. [BLANK_AUDIO] Everyone's saying, "Well, I can fix the safety, but I can't fix it in a way that's going to slow up my ship, because I've got to get there ahead of them." I mean, I'd say there was an era where you wanted to run a range of experiments on different organizational designs, different research agendas, but that era has happened. I think we're entering an era where you actually need to do serious coordination, including between governments, it's great that China and the US actually recently discussed AI at the high level summit, but also you're going to have to bring companies and governments and other parts of society together to talk about this, because the the era to keep having all gas no brakes is probably probably entering the rear view mirror. It's not sustainable to keep this going, and we have to somehow force a larger conversation on what do we want to do with this technology and how do we want it to be developed. So if you were writing the report card on the world for how we're doing on this, where would you get it? So for listeners, Jack and I have known it for quite a long time from before Anthropic, but we start really getting to know each other better when I was working in UK government, which obviously doing his role in Anthropic, and we worked together on creating the UK AC and the first AI safety summit, and these were sort of early efforts to try and dip our toes into this coordination. That was nearly three years ago. Lots has changed. How do you think we're doing? Well, now we have the UK AC evaluating both MIFOS and GPT 5.5 from two separate companies on cyber security challenges, which the UK AC has built in partnership with the UK intelligence community. They explain what an AC is an AC is the AI security institute formally the AI safety institute, but they changed the name right. OK, on we go. And the AC has built cyber security tests, which neither of our companies has seen. We, our technical staff believe in the legitimacy of the test AC has built because we trust for the talent there, which is extraordinary. And the AC has built a test that governments around the world can trust for cyber risks because it wasn't built by a company. It is them impartially testing our systems. If you said to me in 2020, we're going to build an entirely new function within the UK government that does frontier testing of AI systems for most powerful AI systems that will ever be built. And they will invent their own tests that won't come from the companies. And these tests will be better than the ones for companies build. I would have said that's impossible. There is no way the government can build that function. And yet it has a malfunction is being replicated in countries around the world. It's been replicated in the US. They could do with more money in the US but we'll get from that. So obviously you were right at the core of building this thing. How do we answer Jack's challenge here? It seems a bit weird, doesn't it? Given these companies have literally as spending hundreds of billions of dollars and are able to pay single programmers $200 million a year. How does a government put together something that can actually run independent tests that actually do any good? Because I can imagine the American companies swaggering around and saying, "Forgot it, you're never going to have the money, you're never going to have the talent. This is never going to work." One, and some of them did not jack, but I remember a very well known person in AI saying to me, "I just can't believe that you'll be better than ours at machine learning evaluations." I remember saying, "No, but I really hope we're better at bio weapons." And so joking aside, that is part of the answer I think. To Jack's credit, I think he saw this very early that both from a legitimacy perspective and a capacity perspective, there are some things that only governments can do. The reason I slightly messed up my life in 2023 to get involved in this is that I really believe that it was both first order good. It's just a good thing for the world that these things AI security institute exists and can do this work. But also that the second order of effective building state capacity in the UK to have a lot of AI experts in government actually understanding what's happening would have a bunch of second order benefits for the UK. Now, to your point, could, you know, anthropic open AI, Google pay such people more to do absolutely. But I think if you believe, as I do, and I think Jack does, that this is going to be the most important technology of our lifetimes, probably ever, then I think you need to believe from a pure democratic legitimacy point of view, that it can't be something that's done to governments. And how do the governments get the money to pay people in a competitive rate if they can all go off and work at Silicon Valley for much more money? Well, I'm two things I'd say one, it's extraordinary how quickly the amounts of money have changed. You know, I remember securing the first 100 million for AC or what became AC in 2023. And it seems like a costly amount of money at the time and obviously now it's it's not. But I do think and you know, this is slightly off topic by I think it's worth saying. I do think that to me one of the lessons is that when governments actually have a real clear mission with a degree of urgency and top down support from the Prime Minister, both Rishi Sunak and followed up by Kirstaama. Actually, there are a lot of people, even people whose opportunity cost is earning huge amounts of sums in big tech firms that are willing to take up the challenge. So I'm not saying hiring AC has always been easy, obviously don't worry there anymore, but I definitely think the caliber of talent that you've been able to get through the mission has been very impressive. The problem that we've got is that AC is not the United Nations of which Demis for service dreamt in this interview that he gave to you. It's effectively voluntary, right? And it's the UK government and the UK government is not the US government, it's not the Chinese government. One of the things you're right that when you're being optimistic, you say, isn't it lovely that US and China discussed AI safety? Actually, if you look at that summit, what's shocking is how little Xi Jinping and Donald Trump discussed AI safety, that's 50% of the world's economy. Those are the two countries that have all the foundation models that matter in the whole world sitting in them. And they're not really doing it. And so it doesn't, I think ultimately really matter if plucky little Britain gets on and pushes the habit of stuff for 50% of the global economy is not playing. So, let's just start with the question, how do you trust a plane that has taken off in another country with a different government system and different regulatory system to your own? Because you have common standards on things like aerospace safety and testing authorities which exist in each of these different countries or sometimes of a regional level. And planes are able to take off in countries, including countries which are at war with one another sometimes and land in each other's countries because you have reciprocal technocratic standards organizations that are actually facilitating some shared data. And so it's very important to be able to maintain some shared notion of safety. Now the things like the UK AC which have been replicated in the U.S. China has its own efforts here, other countries have their own, is exactly the beginnings of what you need for some of this notion of what standards look like. Now is that going to cover everything? No, for some of the larger risks you're going to need bilateral or multilateral agreements between countries and high level diplomatic discussions. So give us examples of some of those bigger risks that would need something bigger. I would say the question of what you do about proliferation of national security capabilities that touch on cyber or bio or like nuclear is exactly the kind of thing that traditionally has been the domain of nations trying to talk with one another about non proliferation regimes, safety regimes, testing regimes. So that is clearly an area where we're going to need to have some agreements with teeth eventually. So let's just dig into one of them which which may raise bio weapons. Yes, explain a little bit to the ordinary intelligent listen to our podcast. What are the potential threats of bio weapons and why you would need a particular structure to deal with the best way to think of it is that biology is inherently. So it's a, it's the science of what what our bodies are made of now the experts that can build things like vaccine candidates also have the same expertise needed to build things that could cause terrible havoc in the realm of realm of viruses. Why don't they well the world doesn't for world doesn't incentivize for this world doesn't want this to happen and also there are things like biological weapons treaties this has been a topic of discussion among governments for a long time and also the basic goodness of people there aren't that many people want to just visit harm on on others there are some but not many. And those that do typically don't have the capability those that well the number of people this gets to the point right the number of people want to visit harm on others in the world is relatively small for this kind of of horrendous act and they typically aren't trained biologists and if they are carrying out acts of terrorism or usually doesn't requires you to not be alone wolf it requires some amount of coordination. Now the risks of AI systems are AI systems are universal educators and if you take either an individual or small set of people will want to commit some act of bioterrorism and they have the ability to access a universal educator which is versed in every aspect of biology then suddenly those people have been accelerated and they've been accelerated without paying the coordination costs or conspiracy costs for typically allow us to find groups of maniacs in the world. Additionally at the state level states are controlled and vestumained by many agreements between states to restrict this area because there is no interest in things that have the potential for vast collateral uncontrollable damage states prefer military capabilities that can be a lot more targeted a bio weapon is almost the definition of a thing which is very hard to target and has huge spill over effects. So for bio you need to basically solve for two challenges one how do you make it hard for individual actors to access knowledge like. street level AI that allows some access to knowledge that would allow them to cause harm. And I think that problem has been worked on by industry and government for many years now, so far with some effectiveness. The harder challenge in our future though is what happens when we have capabilities that are the same as or better than the best single group of biologists in the world, which is what we have right now with systems capable of cyberoffence. And then the question becomes okay, how do you think about access at the state level or the corporate level for these capabilities which are now transformative at the high end as well? Should we switch to cyber then? Because it's very live and obviously anthropic has been in the news on this topic. So do you want to do 30 seconds on mythos for the 2% of listeners who haven't seen it all over news? So anthropic trained a model recently called mythos. It is a standard AI model that uses standard techniques, same as the ones that the other frontier labs do. And it's very, very good at a range of skills, including aspects of cyberoffence and cyber defense. And it's a general model. So it also would be quite good at writing Shakespeare's sonnets. It's great at creative writing, it's great at coding, it's great at biology, but it's similar to a few months ago where AI systems got good enough at coding but suddenly loads of programmers started using them. It went through some hard to predict points of being sufficiently good at cyber, but it gets interesting from the point of experts. And again, just to explain to you, by cyber, you mean the capacity to launch cyber attacks? I mean, cybercrime. I mean, the capacity to find bugs in software like Firefox or Windows or your iPhone and hack into it, which is then the key ingredient to cause to carry out hacks or cybercrime. Can you say a little bit? Because obviously we're already here in this world. This is not no longer a thing that might happen in future. It's already here. And anthropics already had to make some decisions about how to deal with this. And one of them is this sort of structured access. I don't know if that's how you describe it. We are running an experiment right now into how do you take systems that have this capability and you try to diffuse them into the world in a way that is is defense dominant. Because what we know now that this system exists is okay. At some point in the future, it will be systems like this will proliferate. Many people will train them. The kind of water level of hacking skill in the world will have risen generically as a consequence of these systems. How do you deal with that environment? Just to understand, presumably there are two different things. One of them is that the thing will be much better at finding the bug or the little backdoor and your system. The second thing is that it presumably could mount many, many more attacks very, very quickly and respond very quickly to the defense. So instead of some dude sitting in their backyard trying something, failing and then trying again, this thing could be generating tens of thousands of these attacks and learning all the time. So there is an amazing opportunity here, right. You have systems that now you can turn on to the world's most important software and ask it to find bugs in it. And indeed it is finding thousands of bugs in wide-to-use software at a rate far faster than what these organizations, because we shared it with third parties, like JP Morgan, Microsoft others have found them a past. Okay, now again, sorry, I'm keeping track of you, but I just to bring the audience along or trying to bring you. We were told two years ago, don't worry about this because exactly the same moment as the cyber attack capacity increases, the cyber defense capacity will increase. I told you that. Well, I don't, I don't know, but I mean, these are people that you know well, and in fact some of them you were talking about, well, before we came into the room, but I don't want to drop some in it, but I literally was with them two and a half years ago. I raised exactly this problem and I was told very, very straightly by these people don't worry about it because cyber defense will increase just the same speed of cyber attack. Well, I mean, it's the issue with, say, cyber attacks or, say, biological weapon attacks is the defender has to be right all the time, the attacker only has to be right once. So these things don't have a relationship of being like symmetric, at all. They're extremely asymmetric. And so a lot of what we're trying to do is figure out, okay, as these capabilities arrive, how do you give defenders an advantage? The main thing you can do is give them time, which means finding ways to release these systems to what I think of as a, you start with a small circle of organizations and then you try to learn how to expand the circle over time. We are at our small circle of organizations today with Glass-Wing or a Vax-Ester Mythos. The goal is expand it over time, such that they can use this to raise the kind of defensive posture of the world and also get intuitions for how you can use AI systems to change cyber defense. And it feels like Mythos, even if from a capability perspective, it's another point on a relatively smooth curve. I think that for large organizations, across some threshold of relevance where I think even just in the last few weeks, far fewer leaders of large organizations within the public sector in the private sector like Gebritz-Altec-Bro-Height piece. It feels like it's been a helpful thing for getting the world to take AI seriously. Maybe the way to think about it is your CTO has to care about coding capabilities and but your software engineers are getting accelerated. But your CTO General Council and Board needs to know about cyber attacks and vulnerabilities. And so Mythos raises to all of these organizations. AI just got real in a domain which you all care about beyond the technologists. Okay, quick break and then back for more. So can we just go back a minute and talk about you talk rightly about this sort of this capability will broadly diffuse. Right now, Anthropic has it, arguably open air something close to it. Maybe Google does, but these are very small and as you say you've been able to do the structured access program you've chosen who gets it. As you say, everything we know about AI progress today suggests that that won't be true for very long. What's your current taste? What would be true for it? We would be true for very long that tiny handful of companies have models that capable. You're assuming that quite quickly in six months time Chinese companies will catch up or whatever. Somewhere between zero and 12 months, I think we can expect, but I'd love you to push back if you don't agree, an open source equivalent capability that's likely from China but you know, possibly somewhere else. One, do you think that's true? And two, like given what you said about, you know, attackers only have to get lucky ones, defense have to get lucky every time. What do you expect the real world consequences of that to look like say a year out? Yes, I think that in the order of a year, another model will arrive that proliferates generally absent. That is also a policy choice. So it could be the case for government like the Chinese government says we shouldn't proliferate an open way model that's capable of cyber hacking, but very incentives and disincentives on the side. What happens to the world? Well, you will see likely a rise in some amount of hacks. You will also see a step change in how organizations approach computer security. I think the place that you end up with is one where computer security looks more like the white blood cells in your body where you have many, many AI systems running. As Roy said, all the time at speed, patrolling your organization in the software and continually finding and fixing bugs and it will be a new, more robust way to do computer security than before. Just as when our own immune system encounters a new virus and we have no defenses for it, some people get sick. The same will be true in the cyber environment. We will see bad hacks likely proliferate due to AI and the world will go through some period of adjustment. I do think that on the other side of this, you end up with likely a more robust world from a cyber capability when we've had nothing. Let me be challenging for a second and unfair maybe. I guess the anxiety, if you were listening to this, is that maybe a little bit too confident and optimistic. You've sort of described a scenario which is a bit bumpy, but we come out the other end and things are better. Somebody who's listening to that might say, well, isn't the scenario where actually that could be pretty catastrophic? That period you've just described before the white blood cells get going and some companies fail to adapt, some do might actually be a description of AI models unleashing tens of thousands of unbelievably aggressive effective cyber attacks which could do shattering damage to the global economy. Some might say that this is a scenario that you could encounter as well. Some of this is a choice, it is a choice not just on the part of companies, but how seriously governments take this and how aggressively governments and companies work together to go around critical infrastructure and other providers and securities. Okay, my follow-up question. Yes, your follow-up challenge. My follow-up challenge. So you basically, the narrative we got is you chose, you realized you had this thing. You chose not to really say thank you, right? But again, listening to that, it's a bit like, well, these guys kind of built a nuclear bomb and they decided that was a bit dangerous. So they decided on their own volition, not to let anyone have it. That's been worrying because that implies that isn't at the moment a government regulator or somebody who told you you can't release this. If we're relying on, because now we have to gamble that it's not just you're being good guys. Apparently, the guys running Gemini have to be good guys. The guys running open AI have to be good guys. The guys running GROC have to be good guys. I mean, are we in a world in which we're essentially saying that a bunch of voluntary good guys build these amazing weapons and then they just choose not to release them, but nobody can tell them what to do. I think the world we're in is one where AI companies are building the equivalent of nuclear. power plants and we just had our first case where we upgrade the plant and nuclear bomb also falls out of it. You say, "Uh oh, this has some implications." Now as a society do we want there to be more nuclear power? Of course we do. Do we also want to like manage for risk of the nuclear power? Of course. Do you want to deal with a potential proliferation problem of nuclear power plants spitting out nuclear bombs? Absolutely. Do you need laws for that? Yes, of course you do. Like I'm not sitting here saying, "Leave it to industry." No, the situation I've described is like extremely unusual and I think the thing which we're trying to do with MIFOS is we're trying to tell the whole story, which is, "Hey, good news. It keeps being the case for those we make these AI systems better. They are more capable at many of the things that we want them to be capable at, like discovering vulnerabilities in code, advancing science, advancing our ability to push forward healthcare." And also it turns out that as you make from better at this, dual use capabilities are now showing up for the geostrategic. This means we can't treat it like a normal technology. We were going to have to change our mindset. By sharing it with a set of companies and with organizations like the AC, we've made it so that you don't need to trust the claims of the originating company. You can ask them or you can ask for AC. And that has generated, I think, the best information for the world for this as legitimate. And it gives us time to work out what we need to do. And of course, as part of that, you should have some regulation that says, you don't get to choose whether or not to release a new kilobon. But you also don't want to have a regulation that prevents the new kilopower part, because then you're going to be where where the world found itself after overreacting to new kilopower issues in the 70s, where we just stopped building it in large swaths of the world and lost out. Let's stop the ante a little bit on that. So, well, you recently wrote what I think is brilliant essay that everyone should read on this idea of recursive self-improvement. So you'll finesse it better than this. But effectively, AI that can improve itself on that, you should get some sort of intelligence explosion. We can get into whether that's the right way to start. Just to explain again, for the audience, what are you going to explain? So effectively, right now, AI progress is bottlenecked on jacks, very smart colleagues and their counterparts. They need to keep coming up with good ideas and they need to keep improving the machine. If jacks were in that's great essay about the timelines to a machine improving itself, which would remove at least one bottleneck in that. When it's improving itself, it would be as though the machine had 100,000 of the best software engineers in the world working 24/7, and suddenly it would be able to improve much more quickly. And so to jacks, I think really productive analogy of building power plants that spit out nuclear bombs, I think you can up the ante and say, that is the next level of that. Then you'll be building power plants that build themselves. So occasionally spitting out nuclear bombs. As suppose like some people would say, well, Rory said, is there a, we want to make a choice? There's regulation. You say, yeah, we should make a choice. I suppose like one more profound warrobe, do we have a choice? Not in the sense that, in some technical sense, of course, we have a choice. But there is this sort of, some of my much more left-leaning friends would talk about the sort of technocaptalism. Actually, there is no choice here. We kind of have to do this at some level. We are just to do what? We have to allow this to happen. Because the technocaptalist want us to allow. So I think there is a point of view that actually the choice runs counter to all our incentives. You've already brought one of the arguments that she's like, if we don't do it, someone else will do it. That's true both within countries, between competitors and it's true across countries. Imagine that we self-deny the right to build nuclear power plants that build other nuclear power plants. That's been on nuclear bombs. China will do that. Russia will do that. And so there is this sense of like, I'm really interested in like, where is that choice? Now, I actually think that we are not star incentives and that virtue is a thing and we can make choices. But what do you say to people that say, it sounds like you're describing something where actually there is no choice. We have to do this. Well, there's a choice which is basically on a spectrum between like maximizing individual sovereignty and maximizing like safety or what might critically be called paternalism where, you know, maximal safety is no one gets access to it other than maybe just for government or like for safe organizations and maximum, you know, liberty is everyone gets access to it. We trust in the ability of people to like experiment with this. Now obviously both of these are ridiculous like positions like if basically no one gets access to it, you get essentially none of the benefits new centralized power into like one or two entities or a small handful. There must be paths through this that look more like gradations of access to the technology that we allow to come to. But you're saying essentially that we have to figure out that coordination both within the industry and maybe more importantly and to Rurie's previous skepticism more challengingly with China. And I think I'll get it wrong. I think you said you thought there was a 60 percent chance by the end of 2028. Yeah. So we're talking, you know, two and a half years to coordinate, you know, the within industry thing is quite challenging. But with China on not only that we do this, but how we do it in turn. Paradoxically, I think coordinating with China is easier than coordinating between these bunch of people sitting in Silicon Valley. I think that I'm less worried about China than I am about how on earth you get. I think countries have the enlightened view that they're around for a long time. And there's an interest in reducing chaos. And I think industry has the incentives of we might be around for a very short time and we're in intense amounts of competition. So I agree with Rurie, but actually, I think it maybe it's easier to I think like well, I think the Chinese Communist Party is relatively rational. I think if China is convinced that this thing represents some existential threat to the Chinese Communist Party, to humanity, to etc. And there's six months behind they have every incentive actually. Absolutely. With some regulation. But it takes two stangu. The people that I find much more difficult to understand coming on board is I don't really wake up in the morning and think, Elon Musk, Mark Zuckerberg, Sam Altman, these are the guys that are really going to argue that around. I was actually thinking of you last week when when when Sam Altman and Elon Musk beautifully turned up to court to give evidence in this thing. And I think like one thing, Chad, I'd love to know whether you given you some of your recent experiences with you agree with this in the abstract you might want to talk not want to talk about specifics. But actually one encouraging thought on that is actually American companies shop to court and they sort of broadly do what judges say. And it's a low bar. They do what their governments tell them to do. And actually if anything I think what we've learned over the last six months is that governments will be very assertive. Well, the US government will be very assertive. It has a ton of levers at its disposal to make companies do what they want to do. And the idea that the US government couldn't force coordination between these three or four private companies. I think that seems very unlikely to me. My big big worry is maybe the CCP if feeling it's dealing with someone that's coming to the table will make a deal. But actually coordinating those two countries to do a deal is not. Yeah, I would say let's just look at the general landscape. All of the frontier AI companies ended up doing bio weapon classifiers. All of them have ended up sharing lots of details of one another. Now it's not been made mandatory. But I think that's almost beside the point because you can show the industry like self coordinated onto hey let's not randomly proliferate things like bio weapon risks into the. That feels like a low bar. It's a low call. Yeah, but we passed the low bar. Yeah, we didn't proliferate bio weapons and we show up to court. Okay, like that's two wins will take it. Now, that to me is at least a proof that you can do this basic form of coordination. It does require two to tango. It requires there to be greater political will to do something tougher on on regulation. We have a regime of transparency and transparency reporting right now, which is almost like tell us about your manufacturing processes and details of the labels you've put on your AI systems. Clearly it'll go further and in the same way we have aircraft automotive and food testing for safety before you ship it to consumers or I'm a recent dad. Lots of by kids toys get tested effectively to make sure that when my kid inevitably eats it, it's not fully covered in that. I'm worried about this, because actually the story of that regulation was terrible. There were a lot of kids getting poisoned with lead in their toys before that happened. And the story of all regulation has this property, but we have surely we can learn that that is the fact you say about is a reason for optimism, right? Or pessimism because, unfortunately, the examples you've given are much, much more slow moving and the tolerance rate for failure is much higher. So food safety, you know, humans being eating food for 150,000 years, they're current species, right? And we've learned a lot about what kills us and what doesn't and a batch kills someone here and eventually we get up full food safety. But you're talking about a technology which in two and a half years can be spitting out by a weapons, nuclear weapons, etc. So and again, getting to the aviation safety is a slightly different story. I was with aviation safety people this morning. China is completely obsessed with planes not crashing, which is one of the reasons why technological development in China is really slow. I mean, one of the reasons why it's going to take eight years to build a jet engine and then another seven years to road it out is they're terrified about planes falling out of the sky. We're not talking about that kind of industry. You not encourage their worry by the how active the interest of the US government is in this from a national security perspective? I'm, well, okay, let me, and we can come into that in a second because I think that actually raises another type of question. But let me, let me just sort of put out my thing, right? Okay. What I worry about is that if in two years time something really catastrophic happened, I don't know. Critical national infrastructure goes to the US. I got a GI, mad, tens of thousands of mad cyber attacks, bio weapons released, etc. And somebody replayed these conversations that we're having. They would not be that impressed because the gap between the catastrophe and the sort of reassuring stories about, well, the US government's taking our security seriously and the companies have voluntarily done stuff on bio weapons and, you know, we all know we need to get to regulation. It doesn't feel like there's quite a sense. I mean, I had this recently with one of the big companies saying to me after an hour and a half argument, "Listen, I agree with you. We should be regulated, but you've got to regulate us all of us together. If I were you, I'd be going out there and doing the regulation, but you can't expect us to do it because we're going to race with all the others." But what I don't see on the other side is where are people generating all the details of what it is that they want to test these companies on regulate them on and for often what I hear is the company's playing devil's advocate and saying, "Well, come on, you want to regulate us. Tell us what you want to regulate. You're not telling us what you want to specify, right?" I disagree quite strongly. I think it is the ability of a choice of companies to do stuff beyond what is mandated by regulation. We do this already today where we commit to a range of safety testing of our own products and we publish it. And we say, "We think that ultimately this should be what regulation should look like, but we can generate information ourselves at all of the frontier companies for a people that care deeply about the safety of their systems. I mean, how could you not when you're building it? It's not. You don't get to stare directly into the heart of the ultimate cyber-hacking machine and say, "Oh, well, this will be fine." "Well, come on, come on, come on, come on, come on, come on, we've watched Oppenheimer, right?" There's a moment where those guys think there is a non-trivial possibility that when they trigger the first atom bomb, they could get a chain reaction that blows up the whole universe. And they do it. I mean, we know that about human nature. We know that humans can be very worried about things. We know that 30% of the engineers in Google could be very anxious about these things, but we also know from things that have gone wrong in the past that companies can make catastrophic mistakes, notwithstanding the goodwill and the seriousness of the engineers involved. What does the future looking back want? It wants you to have some kind of mandated safety testing thing which everyone has to do and everyone has to go through. And it wants you to have some notion of sharing details about these risks with society. And it wants you to do something where society pre-positions to get advantages or deal with risks or to say no if the risks seem intolerable relative to the gain. All of these things have the shape of beginning to happen now. And we as a company are in like vocal, often support, so vocal in fact that it sometimes causes us issues with other other parties in this space. And I believe that governments will will ultimately act. It will just feel, it will feel down to the wire because my sense and you to experts in this for governments, it takes a lot to move them to action and it takes a lot of evidence before a crisis for them to do anything before a crisis. Do you think this applies? We've talked a lot about what I would call national security type risk today for understandable reasons. For the rest of this many say we spoke a lot about the economic impacts. Obviously I think your new role anthropic or new ish role at least in part is very focused on this. Well first maybe just headline, what do you think? There's a lot of talk about jobs apocalypse. There's a lot of talk about differential impacts within and across countries. Where do you think we are on that? And then I'd love to dive into a little bit like what do you think governments can do on that? We're somewhat where we were with AI and national security a few years ago. There's an instinct that things are about to happen but are important. There is no real measurement or testing infrastructure built within governments to do this. And the companies have only just begun including with the anthropic index to share information. But all of those of the ingredients from which you can build a telemetry system to basically tie to make causal claims of if AI company does X, why happens in the economy? We absolutely have the ability to generate that data to do that across companies and governments. It should be regulated but companies share information. Tell us a bit about this index. So the anthropic economic index looks at all of the ways that AI is being used by our customers in a privacy preserving way. And it joins that with what are called own job classifications which things like the Bureau of Labor Statistics use to classify changes happening in the economy. And what this allows you to do is look at the economic activity happening on the AI company platform and join it with the same economic data used to reason about the economy writ large. If we have any hope of being able to make strong claims about the impact of AI on the economy versus CEOs laying off people and saying it's due to AI but rather it was due to COVID over hiring, which is a sin then he commit. Then we need to set up these kind of data sharing systems that absolutely can be done and is being done now. I'm in here in England just meeting with people with the new AI and economic institute and the UK government which aims to do just that. But we don't know what the shape of a future economy is. All I can tell you is is I can't reconcile the capabilities of these systems with the economy staying as it is today. Like clearly like massive changes will happen. Everyone but tries to predict this tends to be wrong in like ways for the community can outrageous in the future. If you forced me to predict it I'd say clearly you get productivity multipliers on things with AI touches. Clearly you get the emergence of new companies that are able to do a lot more with way fewer people relative to previous generation companies. And probably you are going to have some issue with early just out of school hiring because those of the people that have almost the least set of skills for the most replaceable by AI systems. Beyond that it's very hard for me to say what the shape of the future economy with AI is because I don't know if the productivity multipliers compounded or so create new industries. I don't know what the shape of like whether these new firms that are doing all of this business generation if they proliferate in a much larger number than normal business formation. But you don't think that your vision of RSI this recursive self improvement is incompatible with the world of human labor or you do. I think that under something like RSI the economy grows so much that it's like human sit on top of an economy that's hundreds of times larger. Remember one today you know a lot of economic doctrine is that what you end up doing is you end up validating and verifying the outputs of automated processes. That's what happens in in large chunks of the world around us and things like manufacturing and pricing risk and figuring out as people how you make agreements based on the risk of what you do C.V. insurance markets. Seafing's like bond markets which essentially model risk at the country level. I think that there will be ample employment for people in new jobs and specialisms we can't imagine that sit on top of this much larger economy. But on routes about you're going to see like massive, massive changes in the structure of the economy and in jobs. But it's very hard to predict what those changes will be. I think you can just bet there will be massive changes with confidence. Connecting those two stories together. In fact three stories, mythos, risks and jobs. One of the things that worries me and you were talking about the US government regulating is that the US government could wake up in a couple of years time and say mythos 15 we believe financial security reasons is too dangerous to release outside the United States America could launch these horrible cyber attacks. Which point these frontier A models become proprietary within the United States. A European's can't access the latest cutting edge frontier models. And then there's a huge sucking sound as all the economic value is sucked out of Europe towards the United States where these AI native companies, your trillion dollar company with three employees are set up in the US on the basis of US frontier models. And then you make a lot of money and we sit around in Europe hoping that you're going to be feel that you've made so much money that eventually you're going to in the way that I'm sure Donald Trump would love to share generously with the rest of the world the proceeds of your wealth. Let's do a deal. Yeah, I think this is not impossible scenario. It's a very worrying one. I think there's a couple of things that we need to need to work on. One is it's again hard to me to reconcile the shape of this future economy with I guess current ways that we try to like tax or control corporations, especially AI corporations. The picture looks more to me like well these companies including us are going to have computers all over the world. The computers are going to be where lots of economic activity is taking place. There must be some way to more directly target that in terms of targeted forms of taxation. And I'm not a tax expert. I'm not claiming I have the answers. I'm just saying a basic intuition is that's that's where the thing changing your economy is and it's it has a body and it's geographically distributed outside the US surely you can target that. Sounds like a good argument for investing in domestic compute. It is a hugely good argument in investing in domestic compute infrastructure. The second part is with with technologies that end up getting classified for military use or being deems to be relevant to national security whereas always this tension of sovereignty and building and other things and I have been saying to governments around the world since I started working in AI policy in 2016 or 17. You guys should build a big computer. We're building big computers and it's giving us like outrageous amounts of leverage. have you considered building a big computer? I think for choice of building a big computer, which you could do AI systems on, is still there for the sort of European community, plus England, it is a choice. And the problem with this choice is for the numbers get more outrageous each year. So the best time to start it was last year, the second best time is now, a bad time is next year, and you see the picture. I think that has to be a part of this. What if you learnt over the course of the years debating this stuff about what is productive and what isn't productive in these conversations? One of the things that worries me, sometimes if I talk about safety to some of you who's been doing this for 10 years, is you've heard it all before. Your mind will be closed from certain kinds of arguments. And you know, somebody will say, you know, how about the buyer weapon that's going to blow out the world? And you're going to be like, "Oh, forget the saying not again. I've been talking about this for 10 years." And here's my answer. Where do you begin to feel that conversation is a bit dead and inert? And where do you feel the really life questions are? I think positive questions are okay. What are the exact regulations that you need to do? We had a productive one here where you were sort of pushing me and saying, "Well, that doesn't really cut it. What is actually necessary? I think it's useful to just get people to label specifics. I think it's also good to actually talk about feelings. This might be the fact I've lived in California for too long but I now say stuff like this. You know, as someone working on this technology, of course I'm very excited by it. And I work on it because fundamentally I think humanity has a huge range of challenges ahead of it this century and getting through them requires us to figure out smarter ways to generate power, various science breakthroughs, a huge range of medical treatments. AI can absolutely help us do that. But I'm scared of it. I am scared of the technology that I'm building. And I'm scared of how it is governed less than like the toys I buy for my kids or the food I buy from the supermarket. It seems like an insane situation to me. And I think actually getting people to work on this to say how they feel is good because it makes us accountable for it. Like I'm saying I'm scared and worried about this because it makes me even more accountable to solve that. You know, sort of fit for everyone else as well. Well, maybe there's just a very last one given that you mentioned your kids. Number one question I got after doing this mini series with my friend last year was I was the optimist by the way on the show you might be surprised to hear. You know, what do you think about what do you mean for your kids? What do you do for your kids? Like how you think about educating them? What do you want them to do? How do you think about that? I mean, I think for the sort of child like wonder and curiosity I have a three year old that children have is remarkable. And it reminded me of when I had that as a child on how school methodically beat the curiosity out of me as much as hard as it could. And the thing that AI gives you is a machine that is leveraged directly by one's intuition and curiosity. And basically having areas of like huge passion and having curiosity about the world are things that are massively leveraged by AI technology and having, you know, wrote skills or things like specific career plans or things which are almost disadvantaged by it. So I'm trying to teach my kid, I mean, you can't really teach a three year old anything but in a few years, encouraging this culture of curiosity, encouraging obsession because the way that I've been most oriented to this technology is I have this passion project of writing newsletters and writing fiction within it and actually having a personal creative practice and hobby has been one of the best ways for me to both use AI and ways that kind of delight and excite and empower me. But also one of the best ways for me to feel calibrated about how good it is because I have something I deeply understand. And occasionally now I'm like, oh, it finally wrote a good word and it finally wrote a good bit of dialogue for a story. Okay, you know, it's good to get calibrated. And I think it just reminds you of the amazing excitement where, you know, in a very realistic sense, we have taught sand to think, bizarre, bizarre stuff is a for and I think being awake to that is important as well. Thank you very much. We love the teaching sand to thank you. Thank you. Thank you. And look forward to speaking again soon. We will be talking soon anyway in other setups, but thank you. Thank you. Well, we hope you enjoyed that so much to get into so many other questions we could have pushed harder. And I think we need to keep pushing harder because Jack as one of the co founders of one of the biggest companies world is an example of an individual with incredible personal influence and power that could totally upend our economies on our security, our public service. And in fact, most of the future of humanity and human society, it's a very few people and we really need to make sure they're thinking clearly and honestly. So we hope you enjoyed that and look forward to hearing what you made of it. And don't worry, I will be back next week.