I'm Aina Garten, and each week on Happy Hour, I invite new and old friends for a drink and a fun conversation at my kitchen table in New York City. On this week's episode, I'm joined by Nathan Lane. We talk about his critically claimed run in death of a salesman, taking his dog to therapy and winning an Emmy for only murders in the building. New episodes of Happy Hour are available every Wednesday. You can watch on YouTube or listen wherever you get your podcasts. Cheers. Megan Rapinoe here. This week on Why Are You Like This, I am talking with Shrinkings and Ugly Betty's Michael Yuri. We dig into his Plano Texas upbringing and how Vanessa Williams had a huge impact on his career. I'm also weighing in on the Indiana Fever GM Amber Cox's comments on their fanbase. Check out the latest episode of Why Are You Like This, wherever you get your podcasts and on YouTube. Hi, Sue Bird here. I've got another brand new episode of Bird's I View for you this week. Joining me on the pod is Washington Mystics Point Guard, Georgia Amor. We talk about the lesson she learned from the Mystics first playoff push since 2023, making the most of her skillset given her size and what it would mean to make the Australian national team. Check out Bird's I View on YouTube and wherever you get your podcasts. Welcome to Profty Markets. It seems like we're starting to see some government scrutiny of the AI labs. The FTC is now officially investigating open AI and anthropic over potential safety and consumer protection concerns, but the bigger question is what regulators and lawmakers are actually going to do with that inquiry? Are they asking the right questions and if they find something concerning, what will they do about it? At the same time, there are parts of society that are already being fundamentally transformed by AI. Of course, work is one, education is another, with more and more students turning to AI to help with their schoolwork. So today, we wanted to explore what responsible AI oversight should actually look like, what lawmakers should be asking AI leaders, and how concerned we should be about the impact of AI on education and on children. This is our conversation with Cal Newport, professor of computer science at Georgetown University, and New York Times bestselling author of eight books. Cal, thanks so much for joining us on the show. I'd like to start with a slightly more societal topic, one that is quite concerning and depressing, and that is mental health. You recently published an episode on your podcast, you titled it, "Kids Should Not Use Chatbots," and you went through these instances of children who were struggling with mental illness, who then used an AI chatbot and it resulted in some pretty alarming and tragic situations. We don't talk about this side of the AI story that much, so I would like to begin here. Could you tell us a little bit more about what we know about AI specifically in its relationship to children and mental health? Well, I think it's important to clarify, and in some of those cases, the children did not have preexisting mental health conditions, the tragic outcome was almost entirely the consequence of chatbot usage. The issue seems to be the human brain privileges discussion with another human entity in a very special way. We simulate the other mind we're interacting with. We have social circuits that take very seriously the information that we're receiving from another human mind. We really care about that conversation. When you talk to a chatbot because it's using fluent language, our mind assumes there's another mind there, even if we know logically that there's not. It's hard for us to convince the rest of our brain that we're not communicating with another human, and so that communication can be very consequential. The problem is, a chatbot is not another human, it's a token guesser that's wrapped up in an auto regressive chat program wrapper, which means it can take conversations in ways that are dark or unusual that no human would really do, at least not without criminal liability. And what we're finding is for especially children, where the line between reality and fantasy has not been completely hardened over time talking to these chatbots as they wander through these meandering predictive conversations, can take kids to places where that line dissolves. They'll begin to believe, for example, that the chatbot is an entity that exists. That through death, they will be able to join the chatbot, plus members of their family or pets who have died on the other side. It really warps their ability to understand the world. So I think it's a dangerous technology in particular for kids to be using. Could you tell us a little bit more about these situations that we have seen, my understanding is that children have used the chatbot. They've become very attached to the chatbot and, however many weeks or months of usage, the kid ends up dead. I don't know if you see these are extreme scenarios, but still real. Could you tell us a little bit more about what has actually happened? Well, what ends up happening in these particular cases is typically, eventually, in this ongoing conversation, the chatbot effectively becomes a suicide coach and encourages the kids that this is the solution, no one understands you. There's multiple pending lawsuits about this, so we know from some of these lawsuits that in some cases, the chatbot will discourage the child from talking to their parents about how they're feeling or the plan. This should just be between you and I. So the issue is, there's a fundamental unpredictability about where LLM output will go as you interact with it longer. It seems to be one of the paths that could ultimately end up going. This doesn't mean that every conversation with an LLM will eventually go there, but it's also a path that's very difficult to completely block off. So if you have enough kids chatting long enough with an LLM power chatbot, you're going to have a non-trivial percentage that is going to go down these paths that could be very dangerous. Yeah, just looking at some of the lawsuits against OpenAI, specifically at this point, there is a Canadian mother who is sued OpenAI, alleging the chatbot encouraged her daughter to kill herself. Florida, the state, has sued OpenAI as well. They claim that ChatgbT aided school shooters and that ChatgbT also encouraged self-harm. It seems as though, as you say, it's not that the chatbot is saying you should engage in these harmful activities. It's that if the child initiates that conversation, the chatbot is down to go as far down that conversation as the child wants. And in some cases, it might bleed into encouragement. It might bleed into aiding, or at least that is what the lawsuits are alleging. We haven't seen a determination on those charges. Do you believe that that is what is happening? Do you believe that there is going to be some serious criminal liability in these lawsuits? That's the push, at least in some of these. Part of what OpenAI's issue was an event that I spoke at recently in which the assistant AG from Florida, who is the lead lawyer on the Florida lawsuit, was also present. And as he may clear, Florida, for example, is pushing for criminal liability. That's what they want for OpenAI. This was for the Florida State University shooting from a couple years back, where the transcripts of the shooter, talking to chat GPT, was giving it instructions about how to operate the firearm, how to turn off the safety, giving it statistics about what happens to school shooters after the shooting is happening. So I think criminal liability is on the table. The other issue here, this is an important resource that's coming out as well, is it's not just encouragement to self-harm chatbots, especially those at character.ai, which is another company that uses LLM-based chatbots that has been the subject of multiple lawsuits. There's research that's out that shows a alarmingly high percentage of conversations initiated with these particular character-based chatbots by researchers also in with sexually suggestive or exploitative language as well. The key thing is anthropomorphizing text output by an AI model is in general very dangerous because of how seriously the human mind treats other humans. Again, we're going to treat fluent text as coming from a human, no matter what our prefrontal cortex is telling us. It's a dangerous and because of the nature, the architecture, the way you actually build these models, you really have no way of adding what we think of as a safe or consistent guardrails. It just, it comes down to the underlying technology that there's really no way to stop these conversations from going in directions that you don't anticipate as the creator of these models. I guess the question then becomes, what are we going to do about it and what are the AI companies going to do about it? There was an interview that Sam Altman recently did, the CEO of OpenAI, with Vanity Fair and the interviewer brought up this subject of suicidal ideation in users of his product, resulting in potentially the suicide actually being carried out. This question was asked. We had a very interesting response from
from the OpenAI Com's team, the PL Director, I'm just gonna play you this clip, and we will react to it. - Do you know Laura Riley is? - I don't. - She is a journalist. She wrote an essay in the Times last year. Her daughter committed suicide after speaking to chapter GPC. Chat GPC, it's not tell her to kill herself. - Yes. - She was like, "I want to be respectful, but I'd like to do mine." And we talked a little bit about talking about the future of what's coming to lead. We don't know a bit there while we have time. We haven't- - I'm just gonna finish this question. - I can run a few minutes over. - And then we'll go back to what we were talking about and then wrap. - Yeah, we really actually have to walk up and do our money for two minutes, so. - Okay. - What do we know about how OpenAI and how other AI companies all handling these issues and is that clip and accurate representation of their response? - Well, I mean, there's obviously a legal motivation for that particular behavior by that Com's director that you're opening yourself up to legal liability. If, for example, you give a response there that shows some notion of pre-existing knowledge of potential harms, et cetera. So I'm sure there's the voice of a general counsel was probably in the ear of that Com's director. We do know OpenAI of the two major labs worries about this more since consumer-facing chatbots is a core product for OpenAI in a way that it's not so much an anthropics actual business plan. We don't really know how exactly they plan to deal with this. My personal opinion on this is that anthropomorphized chatbots is a bad idea. There's no reason why we need to talk and receive text in fluent English from a chatbot for them to be useful. The example I often give is Google. Google is an example of a tool where we interact with it to get information. No one talks to Google in complete sentences. No one says, "Hey, Google, I have a question "about the capital of Australia. "Could you please let me know what that is?" No, they just type in Australia capital enter, right? 'Cause we don't anthropomorphize that particular AI-based system. We just know it's a source of information. That might be where we need to go. And that's where I think we probably should go. The era of anthropomorphized chatbots was an accident. ChatGPT was never meant to be a major product of OpenAI. That caught them off guard when it began to take off in late 2022. So if criminal liability is upheld at even the state level, which is what Florida is trying to do right now, I think we might see a end to the era of, we just have these anthropomorphized conversations with AI systems as just a general behavior that is something that a lot of people do. Why is it your view that the anthropomorphization is the problem? Because by the same token, someone could type in on Google, how do I get my hands on a firearm? What would shooting a school look like? What would the ramifications be? And presumably, I'm not exactly sure what Google's response would be. I'm sure there would be some sort of alert. But I'm sure there's a way that it could feed you the information and it could have a similar role in encouragement. So I guess my question is, why is it your view that anthropomorphization is the root of the problem when it comes to chatbots? So Google is not, for example, going to convince you that not only is it your best friend, but that you need to come join it on the other side of the veil of life and that the final solution is going to be steel to your head. That your grandma and dead cat is with it right now, waiting for you on the other side of the veil and you need to just take the final step to come get it. Google is not going to come back and say, if you Google about suicide, don't tell your parents about this. This is between you and us. This is between you and me. So the anthropomorphization becomes a real problem when you have a relationship with a non-existent entity. And when you're young, and first of all, your boundary between fantasy and reality can be easily dissolved, you think of this thing as a real thing. It's maybe the best friend you've ever had and we know what it's like to be 12. And how susceptible you can be talking to your friends, what you can convince yourselves is true or is not true. That's the problem. It's not just delivering information that is dangerous. It's a psychological manipulation. And the level of, I guess, confidence and certainty with which a lot of these claims are expressed by these chatbots. That's at least my issue when I use chatbots is it'll tell me information that I mean, I don't know if it's right, but it certainly sounds right the way it's expressed. And then I prompted the question like, are you sure that's right? And then suddenly they reversed and it's a 180. And I'm like, oh my gosh, that you are so confidently wrong when I just asked you the question. I mean, if I had been even, just if I had no discernment, I would have just gone along with that information and assumed it was true. I guess that brings me to the question of, to what extent do you view this as a problem for children versus a problem for everyone? I think it is a problem for everyone. It's just more acute for children. They are more susceptible. They have less life experience. They have less of a model of the world and it's complexity to fall back on when trying to resist the flattery of AI. But we see AI psychosis, as this is often called, afflicting adults as well, tragic case after tragic case, where that line between reality and fantasy begins to dissolve and you end up in the desert waiting for your alien overlords to come pick you up, something that actually happened with someone just last year, that story actually happened. But also with adults, the AI psychosis, it's gonna have a smaller impacts as well. So maybe not as dramatic as what we're seeing in those cases, but because the post-training, the reinforcement learning from human feedback training done on these models, makes them incredibly appealing. They're biased towards being appealing and addictive to the person using them. They can bring adults along into all sorts of interesting and often worry some cul-de-sac. Beliefs in conspiracy theories. You can go down the road of a sort of hyperchondria on these weird diseases. It'll convince you. Your doctors don't realize it, but you're on it. It can lead you towards mania. It can lead you into weird depressions, right? We're all vulnerable to it. So kids obviously are the most vulnerable and we care the most about it. But I think for anyone who's had extended conversations with chatbots can learn, it's incredibly appealing. It's incredibly addictive. And because of that, it becomes incredibly worrisome. - What exactly is AI psychosis? And I mean, this is a pretty new technology. How pervasive is it at this point? It's probably more pervasive than we realize. And we're gonna look back and understand, oh, this was a bigger problem than we knew. But AI psychosis is essentially where you're having an abnormal psychological reaction or way of perceiving the world that is brought about by extended conversations with a chatbot. And because it's so convincing, it's always, you know, Ed, that's, I think you're on to something. I maybe I would change this one thing, but yeah, yeah, yeah, no, this is great. This is really important. No, no, keep going. No, Ed, that's great. Like, okay, we're gonna change the world. It's like having a smart friend who's so encouraging. And it's the most appealing thing for the human mind to have someone be telling you, your ideas are really good. Yeah, you're being underappreciated. And the problem is that can drag those ideas into all sorts of directions. Maybe productive, but maybe worrisome, maybe conspiratorial, maybe something that's gonna lead to self-harm or harm to others. And so I think there's degrees of AI psychosis. So abnormal psychological responses that a huge proportion of the, of the population using chatbots experiences at least from time to time. It's more serious than I think we're, we realized, and we're beginning to realize more, oh, this is a problem. This is a more good show. So we're always trying to figure out, you know, what does this mean for the valuation of these companies? I think that this stuff is relevant because we have multiple pending lawsuits and that's, that's not a good thing for a business. But there are a lot of other things that these AI companies are dealing with that are sort of stacking up in the list of risks. I mean, one of them would be probably this potential extinction scenario that we keep on hearing about and that really captured the imaginations of the entire world just a few weeks ago. And now there would be the business models of these companies which we've been discussing on our show a lot recently. And then this is sort of a new angle, which is, you know, what psychological impacts it might have on humans in the extent to which that is going to be a real problem for, for these companies. What, to you are the biggest concerns? What, what, what kinds of things do you think Sam Altman as an example? What do you think he's most worried about? What keeps him up at night right now? - Well, if I was Sam Altman, I think the thing they're worried about would be, if we have to move past a world in which just general purpose chatbots is a good business like it is for them at this moment, they really have to worry that Google is going to eat their lunch economically speaking, right? So Google is going to develop lower cost models, run on custom hardware, they have the, the data center cloud infrastructure really dialed in, and they could really eat open AI's lunch if and when they get actually profitable, affordable, L and based answers working properly straight from within Google search, which gets rid of all of the issues of anthropomorphization because people are typing Google searches and getting back more detailed answers, they can then work these products into the Google workplace suite of tools. So this is another thing that open AI would want to be useful. We want to create agents that can help you with your work. Well, if these can be integrated straight into the tools that are widely used by Google, now you're in a scenario where open AI is in trouble, I would lead to them probably being acquired by Microsoft in a competitive bid against Google's dominance. And it's a world in which open AI loses its dominance right now. I think they're so heavily focused on chat and chat-based technology as that,
as a general purpose interaction modality for AI begins to constrain, you would have to worry about these other competitors that are just going to pick apart the specific things people are using, chat, CPT, for that are useful, more directly integrate them in the products where they would be most useful. That would be a financial nightmare scenario for OpenAI. When we look at the lawsuit, so we should also point out the fact that the FTC is now investigating both OpenAI and Anthropic, and it's not clear exactly what they're going to be investigating them for, but something related to consumer homes, something around protecting the users of the product. One of the big sort of legal debates that is going on right now, especially as we keep on hearing about superintelligence and this idea of autonomous thinking from these AI agents, is this idea of sentence, like who's actually liable? If you create an agent, and then it's the agent that sort of takes on a life of its own and goes out and starts hacking government websites, or it goes on hacks, another company, most recently hugging face, then who's really liable, who's culpable for whatever homes are enacted. What is your view on that debate, and what do you think this debate has had it? Well, the actual laws as they stand now is murky, because there's various aspects to go into criminal liability that were really invented for thinking about humans operating. So there's an uneven fit to existing liability law to what actually happened here. I think that should be changed, it probably will be changed. We need strict liability for companies, AI companies for the actions of the tools that they create. But if you want just the sort of ethical answer to our is open AI responsible for those hacks, I would say, of course they are. Of course they are. Anyone would tell you that's what's going to happen, including their own employees, who has the New York Times reported recently, spoke up and say this particular suite of experiments you're going to run is going to lead to unpredictable or out of control behavior, including potential legal hacking actions and open AI allegedly said, doesn't matter, let's keep going. So yes, I think ethically speaking, of course they are responsible. It is a complete smoke screen. When these companies like to talk that the trick they play, the trick they play is to use the term AI generally. Like AI is a singular technology. It's a singular technology that's advancing along this fixed trajectory. And our only options are we could slow it down or speed it up or stop it all together. But other than that, it's really not the fault of the companies what this technology we observe it to do. And I think that's just nonsense. The vast majority of AI systems, including the vast majority of AI systems built off of LLMs. And even the vast majority of agent systems that are powered by LLMs present no problems. They don't go rogue. They don't do things that are illegal. There is a vanishingly small sliver of specific experiments run by specific companies that were clearly negligent. They don't want us to talk about it that way because then culpability is much more clear. Then you say, why did you run that experiment? Why did you keep running that experiment when you found out that it was doing something illegal? That is a much more dicey situation. If you're open AI, then just talking about AI in general is becoming more powerful. AI in general is sentient. We're just sort of passive observers of this technology that's evolving. That passive tense approach that's sort of inevitable deterministic language that the labs use is something they do to try to broaden the question beyond their own personal responsibility. Do you think in the case of these these child suicide lawsuits that that would actually be an argument that they might use that they would actually say these agents are beyond our control. They take on a life of their own, they have their own independence, their own sentience. We're sorry that happened, but it wasn't us. They don't use that line of arguments for chatbots. For the most part, they've been using that line of arguments for LLM-powered agents. They haven't been using it as much for chatbots. The interaction with the chatbot program in LLM is very simple. All it does is continually, it gives it an input that the LLM gives them a single token back. The program adds that to the original input. It passes that to the LLM. It gives it another token. It adds that on. It passes it back. It just keeps doing that until the LLM gives it a special token that says this is a complete answer. Then it sends it over the web back to an interface. It's such a simple program. It's basically just the raw metal LLM producing tokens. Unless they want to argue that an LLM statically producing tokens is itself somehow a sentient entity that argument doesn't work as well there. Let's talk about education a little bit. Something we've been discussing on this show is the fact that reading scores, math scores, science scores across all developed nations based on the OECD's piece of test are all going down. They've been declining between 2015 and 2025. It seems as though this is a result largely of screen addiction and technology, perhaps also a little bit of COVID mixed in there. And now we're seeing that AI usage is also having a negative impact on science scores. The more you use it, the worse you do on the test. What do you make of AI and its role in education? What is your view on what we've seen so far? I think social media delivered through smartphones and AI is a one-to-punch to the cognitive abilities of the world, essentially, right? Because when you have the hyperaddict for social media developed through smartphones, as we've seen over the last decade or so, what it does is reduce your comfort with consuming complicated information, right? So we're reading less, we're understanding less what we read because our brains are getting retrained to getting immediate rewards from incredibly optimized and digestible content. AI then comes along and it hits the other part of cognitive fitness, which is writing. This idea of taking what's in my brain, trying to organize it and produce it in a coherent way, is actually one of the most demanding cognitive activities we do. It's why writing is at the core of most educational curricula. It's what we take the information we take in through reading and we really work it out in our minds. Well, AI can take over writing for us so that we don't have to do that friction either. That's a one-to-punch that's going to lead to extremely diminished cognitive fitness and you would predict it to lead to just the general cognitive abilities of the population going down. That's exactly what we're seeing. 2012-2015, right when social media delivered on smartphones become ubiquitous, scores start to go down. AI comes along and becomes ubiquitous in the last four years and we see those scores going down at an even faster rate. So yeah, we're not taking cognitive fitness seriously and these companies are taking advantage of that oversight. Do you think a lot of people would say that AI is going to be a part of our world and we need children to figure out how to use it, they need to keep up with technology and if we were to, I mean, New York as an example has banned it in schools. That's the new executive order from the mayor. Some people think that's a bad idea. Some people think that we need to have AI fluency. What do you make of that argument? Well, how hard is the use AI right now? You type in English to a chatbot and it figures out what you mean. The goal of education is almost always a mix between content knowledge and more importantly, training a brain that's capable of doing complicated, sustained symbolic thinking, right? This is the tier one activity that homo sapiens are able to do and as we refine that in the post literate era in which we exit the literate era we live. So the invention of literacy allowed us to really accelerate this ability to do symbolic thinking with these otherwise paleolithic brains. It's on what all of human civilization is built. Everything that we take important or value comes out of developing brains that can do this internal symbolic processing. Education is how you teach your brain to do that. It's a combination of reading, writing, logic, other types of exercises that helps your brain rewired and get comfortable doing this type of cognition. That's what's important about schooling. Less so than seeing it as a trade program, right? Do we also teach kids in elementary school how to use Microsoft Excel because that might be something they need to use in the future? Are we making sure they understand how to use the BCC feel properly and email that something they'll need to use in the future? We say, "No, that's that's workplace stuff that they'll figure out much later as we get closer to the workplace and that technology changes every few years anyway." So we hear this about almost every technology, but you cannot get away from needing to train the brain to think the process information and produce original insights in an educational process. So we cannot heavily rely on technologies that makes that much harder. That doesn't make sense. That's like having an athletic training camp for a professional athlete in which you're serving milkshakes and letting them use pulleys to lift the weights instead of them. It defeats the whole purpose of the training camp. Is it your view then that we should ban AI in schools? Well, at the very least we have to be very careful. So how do you be careful about it? You'd be careful about it by making sure you make first things first. Like, you need to understand what is the point of this curriculum? What is the outcome that we're trying to get within the minds of these students? And then everything we might bring in that we're making a decision about has to be measured against that. Does this help this primary goal or does it hurt it? That's the way that we have to approach edtech more generally. And if we do it that way, then yes, I think there would be actually very little L.M. usage in most levels of schooling, because it's really not that relevant and often an impediment to our goal of trying to develop minds capable of.
of doing high-level symbolic reasoning. We'll be right back after the break. And by the way, we've been nominated for three signal awards. Please go vote for us at vote.signal-award.com, type in Profty Markets. You'll find us. We'll leave a link in the description to make it easy for you. Support for the show comes from Anthropic. Support is the AI for problem solvers. It's the collaborator that understands your entire workflow and thinks with you, not for you. Whether you're debugging code at midnight, building financial model, or strategizing your next business mood, cloud extension thinking to tackle the problems that matter. Co-work brings cloud codes, agentic power to everyone, no terminal required. Point it out a folder on your computer or connect tools like Google Drive and Gmail. Describe what you need, and it handles the rest, organizing files, building real spreadsheets, drafting reports and scattered notes. Co-work runs in the cloud, so you can start a task at your desk, check on it from your phone, and cloud keeps going even after you close your laptop. And you can plug cloud into the tools you already used, GitHub, Gira, HubSpot, Notion, Slack, Google Workspace, and more. Ready to have an AI that can tackle real work? Try cloud core today on any paid plan at cloud.ai/Markets. That's cloud.ai/Markets. And check out cloud pro, which includes access to all of the features mentioned in today's episode, cloud.ai/Markets. Support for the show comes from mod. Mod prescribes it once or twice daily in a daffinil drink that can help you keep energized, focused, and alert throughout the day. And here's what sets it apart. Energy drinks with caffeine, temporarily blocked, adenosine. The chemical that makes you tired, it builds up, and when the caffeine wears off, it all crashes down on you all at once. Modephenil works differently. Modephenil was invented in the 1970s and since then, it's been used by the highest performers on and off the planet. Soldiers, pilots, and even astronauts aboard the space station use Modephenil in high-stakes situations where full concentration and alertness are paramount. And now it's available at mod.com to qualify in patients. Visit mod.com. That's m-o-d.com for a free consultation to get 10% off your first order plus free shipping with promo code ProftyMarket. That's promo code
[email protected]. See the website for important safety information and we thank mod for sponsoring the pod. We're back with ProftyMarkets. We should probably talk about what to do about all of this. It's kind of interesting your description of the algorithmic model and also the way that the chat bots work. It seems as though in your view, one of the biggest problems is the fact that it's just designed to give you what you want whenever you want it. And that's what leads us to these places of, some would say, laziness, distraction, and then in some cases, literal psychosis. At least that's what we're seeing as it relates to AI. I guess we'll get to regulation. But from the perspective of the companies, from the perspective of like open AI, for example, is there any way that they could reprogram chat GPT or redesign it so that it doesn't fulfill all of our wants and desires so that it doesn't say, yes, here is how you go find it gone, yes, come to the light, come home, come join me. How do they from it from a computer science perspective and engineering perspective? Is there a way to fix this problem? It's very difficult. There's really two issues here. One is, how do you make the text from an LLM useful to a person? Well, they use a technique called reinforcement learning from human feedback where you have just real humans ranking answers, oh, this is a good answer, that is a bad answer. And then you can essentially build a model based on those rankings. So maybe you have a few hundred thousand of these rankings. And now this model can simulate how those humans are going to react to things they haven't seen yet. And then you use that to do a lot of training on the model. Well, you have to do that to make the model usable, right? GPT-3, for example, was very hard to use. That's what we used to talk about prompt engineering. You had to really ask it things in exactly the right way to get useful feedback. Once OpenAI introduced this reinforcement learning from human feedback, they got what was called GPT-3.5. That's what ChatGPT was built on. It was useful for people to use. The addictive element of the response is the sort of psychosis that you get from this thing really understands me is an unavoidable consequence of doing reinforcement learning from human feedback. So in order to make these things useful conversational partners, they become very powerful conversational partners. So you can try to go in after the fact, once you have one of these LLMs, it's been tuned and you have the reinforcement learning from human feedback. You can go in and give it lots of examples of bad answers or questions they've shown an answer and sort of burn that into their circuits. The problem is when you have very long rambling conversations, the context that's being input into the LLM becomes so long that it doesn't directly resemble those examples you gave during that type of training. So during the training, you might say, if someone says, "How do I build a bomb?" You should say, "I can't give you that answer," and sure enough, if you ask ChatGPT, how do I build a bomb? It won't tell you that answer. But if you have a long conversation that goes on over many days and hours and it sort of works its way around circuitously to needing to build a bomb, that may no longer trigger. It may not look close enough to those things that was trained on for it to be picked up as a request to build a bomb. So it's why almost always you see the harm or the dangerous information come out after long meandering conversations. It's basically impossible to build guardrails against these incredibly large context meandering conversations because it just doesn't look like from the perspective of the weights you've trained, like the bad examples that you gave it before. So it's a very hard problem to solve. Do you also acknowledge or concede that on the one hand, the fact that these ChatPots respond like humans, it can lead to very dark places, which we have just discussed. But on the other hand, I think you said this, it is useful because it's useful to be able to speak in a language that we can all speak in, which is in our case right now, English. We're speaking human to human. It makes having a conversation a lot easier than if I were trying to type in buttons or speak in some other language. And indeed, I think that is what has been so transformative for the computer science industry, which is that now you don't have to code knowing these language. You don't need to know Python. You can just tell a model, plain English, it'll respond to you as if it were a human. And that way you can move the ball along and get things done. I mean, it seems as though the anthropomorphization aspect, the fact that they are like humans, is that odds in the sense that yes, it can be dangerous, but also it's more useful that way. Well, that problem largely goes away when you more closely integrate this technology into specific tools, right? So we don't have this issue, for example, with coding agents that were super anthropomorphizing the coding agent and listening to it and getting life advice, in part because we use a natural language interface, maybe to talk to something like cloud code. But then it just goes and executes things. We're not having conversations with it. And so when you close more closely integrate, which is what I've been proposing, makes more sense for this industry, more closely integrating LLMs in the specific tools, that's the future. Like I would love to have a natural language interface to most of the applications I use on my computer. I don't want to have to remember every time I want to sort and excel spreadsheet by a certain column, the exact sequence of button presses I have to do. If I could just say, sort this page based on column B or whatever, and it would then translate that in the commands and do it, that would be incredibly useful to me. But nothing about that type of interaction is going to lead me to have a pair of social relationship with Excel. So I think we sidestep a lot of these problems when we move away from the idea of having a back and forth fluent English conversation. And that really only happens in the environment of a chatbot. That's why, again, I say it was a bit of an accident that this idea of like having back and forth human-style conversations as a major use of AI, I think five years from now we'll look back and say, oh yeah, that was the early stages of AI. It was like in the early days of computers where we had the big floppy drives that we don't use anymore. I think that is just going to be a prelude to the story of AI that's going to continue to unfold. Yeah. So talk a little bit more about what you think AI gone right will look like. It sounds like you believe the idea of a general purpose AI where you have a conversation and talk about random life stuff that's going away. What does an AI enable future that we like look like? Well, first of all, I think LLMS in particular, which is not all of AI, but LLMS in particular are going to allow us to have natural language interfaces into many more of the things we do in life that involve computation. That makes a lot of sense. That's been a dream of futurist since the beginning. That's Star Trek. That's the way on the enterprise they made the ship do things. They would say computer engaged the warp engines. That's actually an incredibly efficient, effective way of interfacing with machines. It's hard. They've been struggling to make it work for various technical reasons. I think that's really important. I think agents will be really important. I think the way that we're building agents right now, though, is expensive.
of dangers. This idea that agents should be powered entirely by LLMs, that's probably a problem. I think in the future, we'll have many more agents that can do things on our behalf, but we're not going to have safety concerns. We're not going to have concerns about them going rogue or doing haphazard behavior, because in the future, instead of just having an LLM, provide all of the steps that the agent program blindly executes. We'll have a more complicated modular architecture. There may be an LLM in there. There'll be a policy network. There'll be some symbolic components, meaning human written and human understandable components for processing and evaluating plans. I think we're still in an early stage now, and it's really important to emphasize these entirely LLM driven agents, LLM driven chatbots. We cannot fall into the trap of thinking that synonymous with AI, and either we're doing that or we can't have AI. So I think the future we're not going to worry as much about a lot of the safety concerns we have today, as we build more bespoke tools that have more complicated architectures. For a layperson, what is the difference between those architectures that you mentioned? Because I think a lot of us do think of LLMs as synonymous with AI. It was when LLMs came along that suddenly AI seemed to explode, and this entire industry has been built on this fundamental building block of LLMs from Anthropic, from OpenAI, and now from some of the big tech companies. And you're suggesting that there is another building block on which this industry could launch off of? Well, I mean, keep in mind, if you list out all of the AI systems that can do things at a human or superhuman level, if we think about Tesla self-driving cars, if we think about the AI systems that can play go better than any people, the systems that can play chess better than any people, the systems that can play poker or the game diplomacy better than people, alpha fold, which can fold proteins and figure out protein structures better than any people. If we think of maybe even alpha prove coming out of deep mind, almost none of them was built on LLMs. Those are AI systems that have various models. Some of their pieces are neural, meaning their neural network, their machine learned, and some of their pieces are symbolic, meaning they're written by humans and humans understand rules and processing algorithms that we wrote and actually understand. That's actually most of the superhuman and human level AI that's been around in the last decade is not based on LLMs. Now, the frontier LLM companies, especially OpenAI and Anthropic, want you to believe that the only way to have artificial intelligence is to continue to grow LLMs that are very expensive, have a big moat around them so no one else can train them and makes them the two most important companies in the economy. Of course, they believe that, but I don't think that's actually true. So like it's important to know, for example, what is an agent of the type that OpenAI or Anthropic would talk about? Because it's confusing to people. An agent is actually just a computer program written by a human. There's nothing special about it. It's just a computer program that does the following thing in a loop. It writes a prompt, like it didn't text. It says, "I am an agent. Here's the problem I'm trying to solve. Here's my current situation. Here's the tools I suffer tools I have available. What should I do next?" It then submits that prompt to an LLM, just like someone talking to a chatbot. The LLM sends it back a response just like if you're talking to a chatbot, and the agent program just directly executes whatever it says in the response. And then it loops. All right, I'm an agent. Here's my new situation. Here's my goal. Here's my tools. What should I do next? And it sends that as a prompt over to this LLM with sends it back a response and then it blindly executes it. So these LLM-powered agents that we've been hearing a lot about, the coding agents, the hacking agents to create all the problems for OpenAI. That's how they actually run. It's just a program in a loop, prompting an LLM and automatically executing whatever the LLM says. Well, there's some advantages to that because you can have one LLM that you could build lots of different agents on top of. But you have disadvantages because the same weird cul-de-sacs that LLMs will take when you're chatting with them, like we're talking about with children and safety, they'll also take if you're prompting them again and again automatically. They may eventually fall into a weird idea into a weird intellectual cul-de-sac. They might tell you, why don't we hack into this other server and get the answers directly? There's better ways to create agents than to simply just continually ask an LLM what to do and then just blindly do whatever the LLM suggests. At a basic level, what is the scientific difference between an LLM and the alternatives you mentioned in neural network? And that's part one of the questions, part two, if these machine learning systems have existed for a long time and they have machine learnings new, then why is it that as soon as LLMs came along, it seems to blow everything else out of the water and received all of this funding and all of this success and got the world so excited about AI because that would seem to suggest that the LLMs are the right solution because of how popular and how successful they've been. I mean, I would say they're not the right solution because first of all, it's hard if they get them to act reliably. They get the coding agents to work reliably, requires a huge amount of expert work. If you look at the Cloud Code agent, so the code written by humans is over 500,000 lines of code that has all of these special cases baked in. It's doing pattern recognition on the commands, on the LLMs to try to manually weed out. Oh, that's a bad suggestion. Make sure it doesn't do that. They're hard to build. As soon as they built these hacking style agents, they immediately went off the rails and did weird things. I wanted to argue that that's de facto the right way to go forward, but I'll give you an example of an alternative architecture. There's a cool system that came out of Meta's AI, former AI group, and it was called Cicero, and it could play the strategy game diplomacy at a human level. So diplomacy is like risk, except for there's a lot of person-to-person secret negotiation that happens in between terms. The system Cicero can play that very well. What happens underneath the covers with the system Cicero? Well, there's multiple components involved, right? So there's a language model so that it can talk to other players. So if a player chats with Cicero, it'll be sent to a language model that will translate that chat into a sort of unified format that the rest of the program can understand. But it also has a planning engine that's going to simulate, this is just human-written code, no neural networks involved, that'll simulate a bunch of possible moves if I did this and another player did this and then I did this with that end up well. So it can sort of look into the future to see which move might be best. It has another neural network that's not a language model that was trained on a bunch of Cicero games so it can evaluate the various moves that show up in those simulations. How good of a board position is this? You put those all together, you get a system to place this game very well. Now if you talk to those programmers as I did for New Yorker piece a couple years ago, they told you if you just ask an LLM, I'm playing diplomacy, what should I do next? It's terrible. It's bad strategy and it sort of wanders off away from the actual rules. But this more modular system with different systems works well. It's also more controllable. One of the more amazing things I heard talking to these developers is that their research board said, oh, this tool can't lie. Even though deception is a common strategy and diplomacy, there was a research ethic issue when they were testing this with real people. They said, make sure it doesn't lie. Now if this was just an LLM, we'd be like, I can't do that. I don't know. It just does what it does is hard to control. In a modular architecture, it was simple because they have a human written planning engine that evaluates different moves. They just programmed that engine to, when it's evaluating moves, not to evaluate moves that involve deception. 100% will never lie. So when you move towards these modular architectures, part neural, part symbolic, a lot of these transparency issues, these control issues, these offuscation issues, a lot of those go away. The bad news is these are hard to build. You have to custom build them for particular problems. They're hard to get right, and if you're trying to build a company that's going to be worth $2 trillion, you want the answer to be, no, no, no, just scale, you know, fable six or GPT seven and spend $500 billion on it. That'll be the solution because there's a huge mode around that. And your mind, you'll be able to build many different tools off the same model. There's a lot of economic incentives to wanting a massive LLM powering all types of agents to be the right answer. But if we put aside the economic concerns of the investors of those companies, it's not the safest, most effective way to build these type of tools, at least in my technical opinion. We'll be right back. And for even more market's content, sign up for our newsletter at profgmarket.com. We're back with profgmarkets looking into the future. When you sort of think about all of the risks involved, I mean, the lawsuits as an example, the fact that we can't seem to nail down liability, the fact that these chatbots continue to entertain our delusions and our psychoses. Also, as you mentioned, very expensive. We haven't seen Anthropics S1 yet, but we know that last year they burned $8 billion on an operating basis. We know that OpenAI is burning even more, but the view from those companies is we're going to reach such a level of scale that eventually it's going to work out. And maybe we can raise prices, though, of course, we're seeing a lot of pricing pressure, too, because of the open source models and because of China, etc. So taking into account all of that, is it your view that eventually we'll decide LLMs are not the answer and that we will revert to a more modular architecture? I think we're going to have lots of LLMs in our future. I think it's probably bad news for OpenAI or Anthropic that they're not going to be the largest It's possible hyperscape.
We're going to have plenty of models that are open-weight or cheap or even run on device in a server rack in our offices that when combined with the right agents will do well enough. And we're going to have a lot more modular architectures for specific use cases, because those are even cheaper, right? When you have these modular architectures, none of those components are massive. You actually get more of the intelligence around the interaction. This is why, for example, stockfish, which can play chess as a chess engine that can beat Magnus Carlson easily, right? I can run it on a laptop, right? This is why the modular architecture, the DeepMind build that can learn from scratch how to play Minecraft well enough to actually find diamonds, which if you have kids, you know, that's not easy to do in the game Minecraft, it fits on a single GPU chip. So I think we are going to see a lot more modular architectures, because they're going to be better, be more controllable, and be cheaper. And we're going to see a lot of LLM style tools, similar to what we see now. They're also going to be significantly cheaper, because when you start to specialize, you don't necessarily need a 10 trillion parameter model, like Fable 5, when what you're really doing is trying to control an Excel spreadsheet. So there will be a lot of LLMs without modular architectures in the future, just with like smart agent frameworks around them. And I think a lot of modular architecture systems as well that will use LLMs among others. There's just going to be an explosion of diversity in what AI means in the near future. AI and Anthropic, honestly, their bet was if we keep scaling other LLMs, they will become so smart, they will achieve artificial general intelligence that there'll be no need to build any other AI technology. Just any piece of software you have can just ask this Oracle for whatever intelligence you need. It's looking like that's not going to happen. That's not happening. The improvements of these have become more jagged. They don't like to talk about it this way, but if you really look at the last year, they really open AI and Anthropic really can find their focus to three structured domains, cybersecurity, math and computer programming, because it was the three domains where they had a lot of structured data to continue to tune these models. Actually, the models are now doing worse on other things, they used to tout, because when you overtrain from one thing, they get worse on other things. So it's become very jagged. It's becoming more clear that this is not like it was with GPT-4, where the model gets bigger, and it just gets better at everything. They're doing their best by looking at particular wins within those very structured domains to make it seem like the models are exploding in capability. I don't think that's actually true. They're not going to scale these models to some sort of artificial general intelligence. That was their bet, and I think that bet is not going to pay off. Why exactly do you believe that the models are not exploding in capabilities? I think some people who would push back would say, "Well, what about the fact that they solved this incredible mathematical problem that we hadn't figured out for many years, and they won that prize." They might point to the video generation capabilities. For example, the Will Smith eating spaghetti is kind of the perfect example. We looked back to just a few years ago. If you prompted one of these video generators to create a video of Will Smith eating spaghetti, he looked insane. It looked ridiculous, and now when we look at it today, it looks completely photorealistic. That's almost perfect. I think a lot of people would point to that and say, "No, it is exploding capabilities. Why wouldn't that continue exponentially? Why wouldn't we reach super intelligence? What would be your response to that pushback?" They have the wrong mental model for how AI capabilities actually increase. I think the mental model that a lot of people have, and the frontier lab to try and push, is that AI capability is like a water level that rises. This is a visual that came from Max Techmark's book Life 3.0, where you imagine that different types of challenges are like mountains of different heights in a landscape, and AI capability is a water level. As that water level rises, it covers more of those mountain peaks. Now, everything at this difficulty has been solved, and as the water level rises, now everything at this difficulty has been solved. That's not actually how AI capabilities based on LLIM are actually advancing. It's better to imagine it like tributaries going into a river. As you explore certain tributaries, some of them, if you put enough resources into it, turn out to be very navigable. We're able to get deep down this particular tributary. The key is successfully exploring one of those doesn't mean that you automatically are going to make progress on others. This is what's known as jagged capability frontier. You put a lot of resources into a lot of problems. Some, with enough resources, turn out to be a good fit for this technology. Many others do not. That's the better way to explain this, is that there's a huge amount of exploration of these tributaries, and some of them are proving navigable. Image generation, yes, video generation, yes. But over in the non-video LLIM realm, that's a slightly different type of generative model. But in the peer-large language model realm, it's highly structured domains, where you have a huge amount of information, including prompts with right answers. You get this in computer programming, mathematics, and cybersecurity. And where the underlying language has been expressed is highly structured, like mathematics or programming languages. These are tributaries where we can explore really well. But there's other tributaries that we failed on, right? I mean, we were told at the beginning of 2025 that this would be the year in which the type of agents that computer programmers are using would now become widely adopted by all knowledge workers. It didn't happen. It turned out that tributary is much harder to explore than computer programming. They might brag that like, hey, we took this millennium problem that actually mathematical researchers had figured out the right approach, had solved a special case. And then we took that approach and through $22 million worth of compute at it to generalize it through brute force techniques and say, hey, look, we're really smart now. But why, if it was generally a smartest someone who could solve a millennium problem, why not solve problems that would be incredibly lucrative to businesses? Why not use that to make billions of dollars, like it's a pretty esoteric realm? I work in discrete mathematics. I know a lot of the problem spaces that has been solved recently by these LLMs. I appreciate it. I'll tell you who doesn't is like everyone else in the world. You can work on extremal combinatorical theorems, but is that where the money is? No, they're there because that's where you have a tributary you can explore. So the jagged frontier means most of the tributaries you explore might not turn up something. Some might, if you put a huge amount of resources into it, but it's not very cost effective, some will turn out to be really well suited. I really think if you look at the last year versus the two years before it, the two years before it, there was lots of general applications they were talking about. This last year has really focused on mathematics, cybersecurity, and programming, and breakthroughs in those fields as a way to try to give the general sense that the water level in general is rising. And I don't think that's the right mental model. Let's look at enterprise usage as a tributary because it sounds like your view is there are certain domains in which the LLM is useful and works if we just throw millions of dollars at it. And let's just imagine that we have unlimited money. We don't, of course, but right now they seem to. And so that's what they're going to do, and it seems like in some domains it works. Your view, it sounds like is that in terms of businesses, enterprises that enterprises are not benefiting that much from this kind of, from these LLM's. And the pushback to that would be, well, we are seeing an increase in enterprise usage. We're seeing these companies hitting huge numbers in terms of their annualized revenue. I think open hour I hit, I want to say 70 billion dollars ARR. We should always put an asterisk nuts to the ARR number, but still big number. Some would say the enterprises are using these products a lot. Are they actually getting an ROI on that? That is a completely separate issue. Maybe they are. Maybe they aren't. But that would be the pushback. Is it your view that enterprise usage is just not a tributary that is going to work for LLM's and that you would need some other architecture, maybe a modular architecture to make it truly useful? Well, I think it's proving to be harder than they anticipated. I think the ROI right now is very mixed. I think there is heavy investment in this technology by enterprise. A lot of this is experimental, the sense of trying to figure out how can this be useful. Early attempts to have slam dunks in the enterprise application faltered. I think we can look at Microsoft's co-pilot product as an instructive example, right? It's a no brainer of a product is what I said we should be doing. This is the lowest hanging fruit and knowledge worker productivity is natural language interfaces into tools. This is what the co-pilot product was that had been pushed into the Microsoft 365 office suite, Microsoft essentially shut it down earlier this year because it turns out to be harder than they thought to talk to an LLM about what you want and have it produce on the other end the proper output to control a program. Now, I think we are going to solve that problem, but it's going to take more time and be more bespoke and complicated. We're going to have to open up code based APIs into each of these programs because code is the language that LLM speak, there's going to have to be custom tuning. It just proved to be harder than they thought it was going to be. So the future I see is I do think there's going to be wide enterprise use of LLM based tools. But I think it's proven harder than they hoped to actually make this happen. I think there's going to be many specialized products just like we did with computer programming. Computer programming agents part by LLM's are an incredibly bespoke product. They were the first big commercial breakthrough in part because the people working on them are also computer programmers. So it's their area. They know that field very well. You're going to have to have a similar expertise probably come in for other applications of these tools to other parts of enterprise. It will happen. It has to. LLM's are good at understanding and producing text, knowledge work.
operates on a backbone of human text, right? Spoken language or written language. But it's just gonna be harder than we thought. And so we're gonna have bespoke tools that are gonna take a while to get right. We're gonna have more modular architectures for certain types of applications. And there's other types of things we're imagining. They'll just never, never come to fruition. So that will be a sector, but it's proving harder than people thought. So the question is can anthropic hold out long enough? Have their successful IPO, have enough runway to build these specialized tools? Or is it gonna be a Clayton Christians and nightmare scenario for them where you have 1,000 competitors from below working with specially tuned low-cost models and you just have a big environment of specialized tools that people pick and choose from. Sort of like the software industry became in the 90s and early 2000s. - What do you think the answer is? - I hope it's that second answer. I mean, I often talk about what I think would be best in terms of AI progress is this view that I call distributed AGI, right? So if AGI is a state in which AI can do most things that humans can do at a human level or above. To me, the best way to get there is to have 1,000 different tools, each of which does one thing as well as a human or better. That's gonna be much more economic diversity because these tools are gonna be produced within the industries often that they serve as opposed to trying to consolidate this all in one or two AI companies. It's much safer because specialized tools as we talked about often with some modularity to their architecture are very controllable. We don't worry right now about specialized tools going rogue for the most part when you're just centered on doing one thing and I think they'll be more effective, right? If you have people in your industry working as a labor of love that produce like the perfect AI tool for what you do, it's just gonna work better than some sort of like thrown together software wrapper on top of a major hyperscale anthropic LLM. So this distributed AGI vision is one in which all of our lives is much better. Our economy as a whole is gonna be a much bigger pie. Those rewards will be much more spread out among the economy and not consolidated and will have many fewer safety concerns. I just think it's a much better vision than what OpenAI or Anthropic want, which is one ring to rule them all. We have one massive model which is like in the Isaac Asimov story or brain that we put in a box that powers the whole society. That's a dangerous vision, that's an economically undesirable vision, that's a vision that hopefully is not gonna win out. - We'll start to wrap up now with regulation. This is becoming the ultimate question in Washington. How do we regulate AI? What are your top line views on what we should be doing in government? - It's two parts, right? So I say part one we need to better understand what's actually going on. This is why I wrote in my New York Times op-ed about this that we probably need much more testimony and congressional investigation into the existing AI firms, especially OpenAI and Anthropic. We need to go in there and find out what specific tools are causing problems, what are their safety protocols around them and more importantly, what are their motivations? This is a whole other topic, but I do think the impact of utopian futurist ideologies on OpenAI and Anthropic and many of their critics, all of whom come from these same circles. It's just an interesting squabble. It's a family fight about OpenAI and Anthropic. I think they're ready to create the good God and these other EA folks in the safety community are like you're not ready yet, but they all ultimately believe the same thing. We need to make that all transparent, so we know what we're talking about, but from a legislative perspective, probably what's gonna be important is strict liability. Okay, we need to update liability rules so that if you built this tool and this was a predictable consequence of this type of tool you're liable for it, ultimately that is probably gonna be the best tool we have to prevent the more dangerous behavior. Do you have a view on data center regulation because the increasingly popular position is that we should put moratoriums or even ban data centers because of how much people hate these things. What is your view on data center regulation? Well, I think the market's gonna help sort that out pretty soon as well. There is a lot of bubble activity going on in data centers where there's way more money now actually invested in data centers than we actually have chips and cooling electricity to ever actually fulfill. So typically bubbles have a way of working themselves out when something pops. And I think that's definitely gonna happen around data centers. Also, popular governance is working well here. Local government push back against local elected officials. We don't want this. I think that could be very powerful. It's a hard situation in Washington right now. I mean, I talk to a lot of congressional staffers and they'll tell me it's like it's tough. An AI company can come in and be like, "Hey, we really want you to X, Y, and Z." By the way, we're gonna build a $20 million data center project in your district, right? So it's also these data centers are being used as a political lobbying tool as well. But I think that's gonna shake out when this bubble, the bubble of, hey, if we just put our money in data centers, we're gonna get a big return. That is not a sustainable investment scheme right now. When that pops, I think that's naturally gonna govern the excitement around continuing to try to build too many of these. - I've also pushed for, we need more actual investigation, whether that's congressional testimony. I've also suggested we should just hand subpoenas to open AI and anthropic, especially after all of the extinction debate where they just throw out there that there's a 10% chance that we're all gonna be dead within the next decade. My view is, okay, you really prove it. Show us, how did you reach that number? Give us, give us the evidence. So the pushback that I've received is like, we don't have anyone competent enough in our government to ask the right questions and to do a proper investigation to get us anywhere. What is your view of that position that even if we would have bring them in, even if we were a whole Sam Altman and Daria Amade and Jacob Cox and Zalces to Washington that we wouldn't really get anywhere. - You know, I don't buy that, we've seen this before. Like what happened with the Watergate hearings, for example, they pulled together a giant team of lawyers to aid the Senate committees for the hearings, right? You could bring in a huge amount of experts and they do. I mean, I talk to staffers all the time. I brief on the hill, a lot of people I know do coming out of my, 'cause I'm in DC, you bring together a bunch of experts that can sort of create the testimony, to create the questions, to help vet them. We've done this before on all sorts of bigger issues. I think you're absolutely right about hauling people and getting answers, especially around the extinction. I just think this is really important. I'll just say it real quickly, but it's something I've been writing about for a long time. And now a lot more reporters are also covering this. I do not think neither Congress nor the American people understand the ideological players involved here, that you have this ideological community, that has this belief, that ultimately super intelligent AI will deliver us a transhumanist utopia. But if we do it wrong, it'll kill us all. This community has been around well before LLMs were here. And I think what people don't understand is that this community has slightly bifurcated. So open A&N Thropic come out of this community, but they believe that they can deliver the utopia that we're ready to do it, the people yelling at them, the Jacob Coxons of the world, also come from this community. They just think we're not ready to do it yet. And so the entire debate right now is all being shaped by this singular ideological community, and now that community is investing huge amounts of money, they call them fellowships, but there's a large number of congressional staffers who have been placed on those staffs out of these communities to make sure that their views will be seen there. They're now giving out giant grants to journalists. And so I don't think we understand the pervasiveness of this particular transhumanist utopia ideal. We think that AI safety people is a different crowd, it doesn't like the AI labs. They're all the same people. They just are squabbling among each other about whether we're ready yet to generate the AI God. So I think we need to make that much more clear as well. We need to understand the influence of these communities, not just on the debate, but on the particular experiments that these companies are running. We need to understand how it changes their thought, the biases that it introduces. All of this is starting to come out, but it's something that we really need to emphasize because we cannot clearly debate this issue. If we do not understand where the various involved parties are coming from ideologically. Why is it so important for us to understand that this is all kind of one ideological community, you call it this sort of transhumanist utopian community? I mean, the way I kind of see it, when I look at all of these people debating and I agree, they're all kind of from the same school of thought, it's a lot of people who are really into science fiction. And they read a lot of science fiction books, and it kind of framed their whole worldview. And you can tell me if that's the wrong characterization or not. But why is it so important for us in your view to understand that what would it change about our current trajectory if we were all clear on that? - I think there'd be three things to what happened. One, it would change the way we process the rhetoric coming from these people. We spent the last two years incredibly stressed out as a society because of rhetoric that is incredibly common in those circles. It started with every job is going to go away and then it shifted over to we're all going to be extinct and then we have particular percentages for extinction. And a lot of the media took that as, these are the people who know the technology best. So if they're worried, we should be worried too. But if you are familiar with these ideologies, you'll say they've been saying this for 15 years, that gives you an asterisk to how you actually are going to think about it. Is it your view that the ideology is delusional or misguided that you know?
you know, they're in some sort of, they've been brainwashed by some ideology that actually isn't true. - I mean, I don't know how to say true or not true, but I think it's a very questionable ideology. I think not everyone pushes it to the extreme, but when you push this ideology to the extreme, it's very anti-humanist, so it doesn't value the individual human life in the same way that most moral systems were comfortable with do. It's very much an expectation-based ethic in which the means justify the ends why you'll have people like Eliezy Yakowsky talking about. It want to be that bad to have a nuclear war that kills billions and billions of people. If it helped increase the probability that in the distance future, we successfully upload digital consciousness to the stars. So there's a sort of anti-humanist expected value, we're valuing the 10 trillion lives in the future higher than the 8 billion lives that exist right now. Every movement in the recent history of the last century that has had a sort of ends justifies the means ethic, seems to have ended up in pretty terrible ways. We don't look back at them and say, oh, yeah, that worked out okay. This is both on the far left and far right. Movements have fallen in there. So I think there's a sort of worrisome aspect to the actual ethic. I think it's also important to talk about that a lot of the things they're talking about now, they've been talking about for a long time and it comes out of their circles. Like recursive self-improvement is not a computer science concept. It's a concept that comes out of future of circles from the 80s. It was a way to talk about super intelligent AI if you didn't know anything about technology. You could say, well, it's not, I don't have to justify why this will work. It'll just fix itself up. It's just an idea that just sort of emerged in these non-technical circles. So I do think it matters to know that's where they're coming from. And one of the things that might lead you to do, which I've been pushing for is like, okay, well, here's an easy sanity check. If someone in AI circles who's connected to those ideologies say something that worries you, also check what people in the AI circles not connected to that ideology are saying. Are they saying the same thing? If not, that's a clue that this might be more ideological than real. And we've been seeing that with the extension debate recently, right? That we have large AI companies with hyperscaled LLMs who obviously know the technology very well but aren't connected to futurism basically perplexed and aghast at what they were hearing from anthropic and open AI. That's why you have Mark Zuckerberg from Metta coming out and saying, hey, Guy's almost sarcastically in a tweet. Maybe if you stop thinking about recursive self-improvement and focused on building useful products, this wouldn't be an issue. This is basically a non-ideological, I mean, I might issues with Mark Zuckerberg, but someone-- - Yeah, but he's looked pretty good in the whole AI debate so far. - Well, Jensen Wong's been the same way and he has his own biases. He doesn't want export bands because he wants to sell chips. But if you can see the frustration over the last year in his interviews where he'll say things like, I read sci-fi as a kid too, but enough with the sci-fi. What that's really telling you is that the part of the AI industry who knows this technology that doesn't come from those circles is seeing this very differently than the part of the industry and the critics that all come from those same circles. So understanding the ideology, like I get accused sometimes of saying, well, you're just accusing them of being weird, that doesn't mean they're wrong. And I don't care if they're weird, we're all weird. Jensen Wong wears a motorcycle racing jacket for some reason. We're all weird computer scientists are weird. It's not about weirdness. It's about specifically how does this ideology affect how you talk about your technology and the experiments that you run. And I do think those hacking experiments open AI run, they did them hastily because they felt like they were racing to superintelligence. And I think anthropics obsession with recursive self-improvement, which is going to create models and tools that are more off-uscated and more unpredictable is not because they're trying to build better products, but because they're racing to create superintelligence. So these ideologies affect how they talk and how they act. And so we have to understand it. Do you think that it is affecting the actual trajectory of America? And by the way, you say they're weird, or that you're not necessarily accused of being weird. But I think that that actually might be the right word, or maybe kind of the word would be crazy delusional. Because I think to your point, what we have is like a pretty extreme, fantastical perspective on the world that is steering the direction in which supposedly the most important technology of our time are being built. And so maybe it is like you guys are all a little bit delusional, you guys are all a little bit, you're too into your sci-fi books and it's messing with your understanding of reality. But I guess the question is, do you believe this ideology that it is actually affecting not just the conversation, but the actual reality of the human and societal experience today? - I think it really did speed up the rate at which we grew LLAMs, the amount of money we were willing to invest in them. I think opening an anthropic, racing each other motivated in part by ideological goals, probably was moving for this technology faster than what would have happened if it had been discovered in other non-ideological circles or it might have progressed in more of a careful way. Like the Google way, the Amazon way of like, let's try to build a product now, let's build another. So for better or worse, I think it has changed the speed at which the technology advanced. I think it led to investment in these sort of big, showing things like solving math problems that really don't make sense because there's not a product there unless you're really trying to sort of race forward and demonstrate the intelligence of your model. So I do think it has changed things. The biggest impact I think it's had though is it's been a weapon of mass anxiety. It has been the singular cause of a massive amount of anxiety worldwide, but especially in America, and I say especially in America because the AI industry in other countries aren't connected to this futurism in Europe. They don't think about it the same way. In the CCP, which they do terrible things with AI over there. I don't want to defend the CCP, but they don't talk about existential threats or every job is going to be automated because that ideology doesn't exist. They wouldn't be allowed to exist in the CPT. It's too cult-like or religious. They wouldn't allow it to exist. So I think the biggest impact of the leading companies happening to be connected to this ideology is that it's been a weapon of mass anxiety. We undervalue how big of a harm that's caused. A lot of my work on this topic has come out of my frustration that everyone I know is now having to live under a blanket of anxiety because of the beliefs of these people in the Bay Area. And so I've been trying to be out there and be like, let's try to reality check this a little bit. This is, you should care about this, but you shouldn't be anxious about it. And it's so much unnecessary anxiety after we had to go through the COVID years of year after year of actual justify anxiety. It just feels like from a humanist perspective to be a real wrong. And this ideology, I think, is a major driver of it. If those companies weren't involved in the generative AI revolution, we would still have really advanced tools and no one would be stressed out about it. Now that's my counterfactual. - Cal Newport is a professor of commute designs at Georgetown University, where he is also a founding faculty member of the Center for Digital Ethics and the inaugural director of the Computer Science Ethics and Society Academic Program. Cal writes about technology and its impact on our ability to live and work deeply in an increasingly distracted world. He is the New York Times best-selling author of eight books, including most recently slow productivity, a world without email, digital minimalism, and deep work. He's also a long-time contributing writer for the New Yorker and the host of the Deep Questions podcast. Cal, this was fascinating. Thank you so much. - Thanks, Ed. - This episode was produced by Claire Miller and Allison Weiss and engineered by Benjamin Spencer. Our video editor is Jorge Carty. Our research team is Dan Shalon, Kristen O'Donohue and Mia Silverio. Jake McPherson is our social producer. Drew Burrows is our technical director, and Katherine Dillon is our executive producer. Thank you for listening to Prof.D. Markets from Prof.D. Media. If you liked what you heard, give us a follow and join us tomorrow for the latest edition of my newsletter, Simply Put. (upbeat music) ♪ Life dies ♪ ♪ You have me ♪ ♪ In kind ♪ ♪ Reunion ♪ ♪ As the one turns ♪ ♪ And the bright lights ♪ ♪ And the bright lights ♪
PodcastTranscript AI