Is It Wrong to Love a Chatbot?

For a long time, the idea that we would have AIs or robots that we could talk to, interact with, integrate into our more personal lives, that was fantasy, science fiction. You, I suppose you're programmed for etiquette and protocol. Protocol? Why, it's my primary function, sir. I'm sorry, Dave. I'm afraid I can't do that. But now, talking AI systems, it's mainstream. Roughly half of American adults are using chatbots. Many of them are going pretty deep with their AIs. Nearly one in five U.S. adults have chatted with an AI that simulates a romantic partner. The results here can get dark. Google's own researchers have found some children are struggling to distinguish between talking with chatbots and now interacting with humans. Beginning in the 1970s, the sociologist Sherry Turkle has been studying the relationships between humans and machines. In the 80s, she predicted that computers would evolve into these second selves. I think that basically came true. Think of the smartphone in your pocket that you're always looking at. In 2011, just a year, after Instagram came out, Turkle warned that social media might create a society where even though we're constantly talking to one another, we're in this strange state of being alone together. I think the data bore that one out too. Turkle's latest book, Artificial Intimacy, Who We Become When We Talk to Machines, turns his lens on AI. So I wanted to have Turkle on the show to help me figure out what's going on now that talking machines have very much arrived. What it means that so many of us are building these, I don't know, call it relationships, don't call it relationships with them, and what might happen if we keep going down this path. Sherry Turkle, welcome to the show. Glad to be here. So you studied a very early, very famous chatbot named Eliza. Yes. Tell me about Eliza. Well, I got into Eliza, I had just arrived at MIT, and I was introduced to Joseph Weisenbaum, who said he wanted to see me. I was one of the few people who wanted to see me. I was one of the few kind of people, psychologists at MIT. And he came to see me, and he explained that he had written a chatbot, a stupid chatbot. This is 1960s, right? No, this was 1976. Okay. He had written it earlier. He had written it in the 60s. It was also known as the doctor program, because he had written it, because it was very stupid, to do something that could simulate a conversation that people would believe was a real conversation. And that was the conversation between Eliza and Joseph Weisenbaum, a patient and the kind of psychotherapist that's called a Rogerian psychotherapist. And the reason that was a good thing for him to do with a stupid program was that Rogerian therapist repeats what you said in a spirit of affirmation. That is the mode of Rogerian psychotherapy. People should know, there are podcasters who this is primarily what they do. Exactly. So basically, if you say to Rogerian psychotherapist, I'm so angry at my mother, it'll say, I hear you saying you're angry at your mother. So Weisenbaum was able to create a program. And again, there's a chatbot only because it was a program that chatted, but it wasn't a chatbot the way we think of it now with generative AA behind it, you know, processing millions and zillions. It took what you said and essentially put something like, I hear you saying that in front of it, or are you saying that, or something really basic and very stupid. And he did it to explain to people and he did it to experiment with language processing. And he was a computer scientist and that was sort of his goal. And he found that the graduate students who had helped him, who knew that this program didn't know and didn't understand anything, nevertheless, wanted to be alone with it and wanted to like be in a closed room with it and wanted to talk to it. His assistant, his secretary, as he called her at the time, wanted to be alone with it, wanted to talk to it. And he was delighted, wanted to find me, a psychoanalytically trained psychologist, you know, to try to help him understand what that phenomenon was. And this was really one of the things that drew me into the study of computers and people because it was really one of the first times that you really saw the seductive power of machine that talked. So Eliza's famous because to many people is one of the first of these programs to arguably pass the Turing test. This idea of can you, talk to a program and you're not necessarily sure if you're talking to a computer or to a person. As you say, people knew what Eliza was. Right. So what was so effective about it? To me, and I called it at the time, the Eliza effect is that people in this case, knew that it was stupid. And when you spoke to it, it really didn't pass the Turing test. You knew pretty quickly that it was not a person. But what I call the Eliza effect was people's desire to make it seem smarter than it was. They wanted so much to believe that it was a thinking creature that they would ask it questions in a way that with practice they knew would cause it to give the most human-like response. So for example, hackers would figure out that if you said three angry things in a row, the machine would say to you, I hear your anger. Are you angry at your mother? And it would seem like that came out of the blue. In other words, as though it were, even though that was scripted, if the following angry words are said three times, respond, I hear anger in your response. And again, that was one of the early things I learned, that it's not just that we come to a machine and say, oh, is this intelligent? Does this understand me? But we come to these conversations wanting it to be there for us. And that really concerned Weissenbaum, because he thought that was a very dangerous road because the machine really understood nothing. - I think it's worth spending a moment on why we want that, because I think it'll be important as we go on in our conversation. - Yeah. - Your book and you, I think, are very positive on human beings and human relationships and human conversation. - Yeah. - But human beings feel failed by each other a lot, feel misunderstood by each other a lot, feel nobody wants to listen to them a lot. - Yeah. - And one reason, Eliza, in that case, other things we're gonna talk about in a few minutes, I think have been appealing to people is that there is something useful to them in something that is smart enough to respond, but dumb enough to listen and to listen non-judgmentally. - Yes. - So I wanna talk a little bit about that other side of it, the desire that makes people go looking for a computer to talk to. - I don't know if it's desire or deep disappointment. I think where you put the emphasis was exactly right. It's that they come to the computer conversation primed by a deep disappointment with the people around them. And that was when I did work with Weizenbaum and interviewed his students and interviewed his secretary and interviewed all the people who were lining up to talk to it, 'cause at this time he had to line up to talk to it. To me, they talk not about the program, but about how their partner never listened to them, how their mother was a bitch. I mean, they were, they were filled with conversation about the people in their lives who didn't listen and how this was kind of a bomb, a salve, a place where you knew that someone would. So I think that there's two sides to this story. I call it technological affordance and human vulnerability. It's what the machine is putting out, in this case, really a very poor conversation, but human vulnerability was what the people weren't getting in the rest of their life. - So you go from studying Eliza and I'm sure many other things - Yeah. - That you call relational artifacts. - Yes. - Tell me what relational artifacts are. - Let me just step back just a second. When I studied the Eliza program, it really stood alone as a machine that used the first person that said it was willing to talk to you about personal things. The rest of the stuff around were computer programs that did what you told them to do. - Yeah, this is pre-graphical user interface, right? - This is pre, absolutely. I came to MIT in the late '70s and this study that I first did in the second self was really people were moving away from their punch cards. People were surprised that things weren't punch cards anymore. So the computer sort of did what you told it to do. People were still saying things like garbage in, garbage out to describe what a computer was. And in the late 1990s, that arc of what you sort of expected a program to be, that you would give it instructions and it would do something informational and it would say something back to you, was broken by a kind of program that I called relational artifacts. There were the Tamagotchis that asked you to take care of them all day and clean up their virtual poop and play with them and teach them. And kids were so much not wanting them to die that they gave the Tamagotchi to their parents who would take care of it and nurture it all day. - For our younger listeners, Tamagotchis were, this is my generation, actually. They were like these little sort of egg-shaped as I remember them. - Egg-shaped digital creatures. - Yeah, I never got it exactly, but you had to take care of it or it would die. - You had to take care of it or it would die. There were Furbies. that you had to teach. There were IBOs that you could teach. and they would make gestures of love towards you. There were paros, which were little seals that you would give to the elderly people that would pretend to understand their affect. These were all machines, and I became fascinated by them. I knew that they were terribly important, that didn't just want your cognition. They were pulling for an affective response. They weren't just saying, I'm smart. They were saying, I care. In a way, they weren't even saying, I'm smart. They were saying, you care. Yes, and I came after years of studying them and really interviewing the old people with their paros and the young people with their furbies and their tamagotchis, I came to this formulation that we nurture what we love, but we love what we nurture. In other words, when these objects ask for nurturance, people glommed on to that. They did what the ELISA program didn't, and this is where they had something very dramatically over the ELISA program. The ELISA program, pretended to know what you were saying in an affective way, but it never said, I care for you. Teach me. I love you. I'm your person. And these machines started to move in that direction. I want to pick up on something you said there. It reminded me of Alison Gopnik, the child psychologist, in one of her beautiful books about parenting, has a line that's affected me a lot, which is that we don't care for people because we love them. We love them because we care for them. It is the same formulation. The way I put it at the time was that nurturance was the killer app. That by asking for our care, they came not just for our attention, but for our attachment. But maybe that gets to something we were saying around ELISA, that one thing it was showing was people had, they wanted to offer more conversation and vulnerability to the world, the human world, than maybe the human world always wanted from them. But this gets at another side, which is that for many people, they want to offer more care than the world is asking for from them. They want people to care for, things to care for, that there's more kind of love to give through acts than people always have receptacles for that. And it's so tragic that an object comes along that doesn't care for you at all and becomes the receptacle for what other people need so terribly. So, I mean, I've found my work to be emotionally so upsetting because I've discovered this vulnerability, but also this untapped desire to give and people are giving it to truly something that doesn't care about you. Let me hold on to that. Let's talk about a child and a hamster or a box turtle. Some of us had box turtles in our classrooms growing up. I'm not sure that when a kid cares for a hamster, and I recognize the emails I'm about to get from the hamster lovers of America, that what's happening is that the hamster is receiving that care and coming to care back for the child. Or even. Even more so, the box turtles receiving that care and coming to care back for the child. I knew a person who took care of a wounded fly for a week at one point and, like, really, really cared for it. In your sort of formulation, to express care to something that cannot care for you back is hollow or deceptive or meaningless in some way. But I think a lot of people do express care for. I mean, I think the easiest analogy here are pets, kind of like lower order pets. It's a fish that I don't think in any realm that we think of cares for them back. But isn't the expression of care a virtuous or a beneficial act? Here's the difference. I've thought so much about this because, of course, I'm discussing something where people say, well, excuse me, I talked to my chat GPT, but I. What about my dog? You know, why aren't you coming for my. My dog is, you know. My dogs are dumb, but they're smart enough. Right, or I'm willing. But a fish. I'm willing to go with the fish and the turtle. Yeah, the fish and the turtle, I think, work. The difference is that after you express care, care for your box turtle, the box turtle then doesn't say to you, you know, I hear you're having some trouble with your dad. Or the relationship you have with your mom, talk to me about that. Or I'm really here for you. I'm your best friend. Or you have a noose in your room. Let's not tell anybody else about it because let's make that our private special thing. The trouble with the objects we have now and why these pet, turtle, hamster analogies don't work for me is that this is now an object. Now I'm talking about. I'm talking about chatbots that are using the first person and are telling you, I am your person. I care for you. Even though when you turn away from it, it doesn't care, as I've said so often, if you cook dinner or kill yourself. The original sin of these objects is they speak in the first person and they say they care about you. And that's how they hook you. I mean, that's part of the game they're playing is showing this care. So I think that they. But why is that a sin? And let me phrase it to you this way. Yeah. These systems, the AI systems we're talking about, that I agree with you. I have a lot of discomfort with the way they speak in the first person. My honest take, I'm giving you my best advice, right? You know, and as you say, you know, I'm your person, right? There's all this. On the other hand, they're shaped this way by people. And the reason they have ended up in this called form factor is because it turned out people preferred it. You know, it could say, it could all speak in the passive voice, but people don't want their chatbot speaking in the passive voice. They want it speaking as if it cares for them because they enjoy that experience. I'm arguing that we are seduced by something and we want something that is bad for us, particularly if you're a child. Because one of the books that I'm going to say is one of the. I know you asked for my favorite books and one of the books I'm going to suggest is D.W. Winnicott, Playing in Reality, where he talks about. the early internalization of objects, how we grow, how we develop an inner life, how we bring things inside. And a child talking to a chatbot that is using the first person that says, I'm here for you, I care about you. I mean, imagine ChatGPT in an Elmo plush toy or in a Disney character. A child, there's nothing there to internalize and bring inside. And when you have a conversation that you feel like you're in, that you find enjoyable with an object that really is not there as a caring object, you're kind of entering a shell game of your illusions and mirrors behind mirrors behind mirrors that might be amusing if we were kind of just doing this, you know, as a kind of game. But I think Mark Zuckerberg, who, you know, I credit for actually saying the quiet part out loud and saying what's really on his mind, gave an interview in which he said, Americans would like to have 15 friends. There's the stat that I always think is crazy. The average American, I think, has, I think it's fewer than three friends. Three people that they'd consider friends. And the average person has demand for meaningfully more. I think it's like 15 friends or something, right? I guess there's probably some point where you're like, all right, I'm just too busy. I can't deal with more people. But the average person wants more connectivity, connection than they have. And my goal for MetaMask is to have 15 friends. My business plan is to make avatars that will be perfect for them. So already one of the main things that we see people using MetaAI for is kind of talking through difficult conversations that they need to have with people in their life. That's pretty helpful. And then I think as the personalization loop kicks in and the AI just starts to get to know you better and better, I think that will just be really compelling. Now, that is your end game. So when you say, what's the harm? People like it. It was the most seductive thing. It made sense from a business point of view. Granted. But really, given that we learn about our vulnerability and given the power of the corporations and the platforms that are building these friends, what is the end game? That you're going to sit around with these seductive fake people instead of making friends? Well, let me try to start. I think I want to push the other side of this, not so much because I believe it, but because you're so negative on it. I think it's the best place to test it. So let me start not with kids because I think kids are a very particular case and I do very much want to talk about kids. I want to start with seniors. Yes. So when you were mentioning some of those relational artifact objects, you really had two kinds being built in that era. There were Tamagotchis and Furbies for kids. And then there were these things for the elderly. Yes. So I'll use an example here. I know someone, an elderly woman, late 70s, lives alone, good relationship with her family, right, is not in a bad position, but is older, lives alone, has trouble sleeping. Recently started using one of these systems and loves it, loves it. Because at two in the morning, it will talk to her about Robert Frost poems or her memories of the place she grew up, that even though her family loves her and she can talk to them, like it's there to, there is only one way to talk to her. There's only so much time members of her family have. And so, you know, this system is always around and always ready. It speaks to her in very, very like individualized terms, right? It's very, very good. These things are now very good at mirroring and picking up on how somebody wants to be spoken to. And so it has alleviated some loneliness for her and also created some, not just like joy, but interest and curiosity. Right. The substitution effect here is probably for cable news or for, you know, TV or other things on the computer. What is wrong with this use case to you? Well, let me just speak about my own case for a second. I went to see Andrew Scott in V. It was a one-person show of Uncle Vanya, the Chekhov play. And he did all the parts himself. And I went to it with a friend and we had a coffee later and then I went home. And I was desperate to talk about Chekhov and Uncle Vanya. I had to talk about Chekhov and Uncle Vanya. I mean, who among us has not had that experience? Who among us, right? I really don't have any friends who are Chekhov and us. It was extremely late. And I fired up ChatGPT. And ChatGPT and I had two hours of a deep dive into how it had failed. And then he completely rewrote it because it was considered like the worst play he'd ever done. And how the new production had changed it and where they had changed it and how they had changed it. In order to make it a one-person show. I have no problem with using ChatGPT as an interlocutor, as something to talk about interest, films, grocery lists. I mean, I'm not like the ChatGPT police. The problem for me, and I'm basically an empiricist. I interview people and I do the work. I do the homework. The number of people who say, this is actually better than the relationships in my life. This is actually, my daughter, she kind of fights with me and there's always little tension. I always feel like I'm bothering my son when I call. I mean, he's busy. He's doing other things. There really isn't a senior center around. I'd have to like take three buses to get to something. I feel as though the temptation for substance, I'd have to like take three buses to get to something. And particularly in a society where we are turning away from each other. You know, we're closing down programs that could bring people together. We're de-resourcing. Not pouring money into the life of seniors. The danger that we use these things as a substitute is really what I found in my interview work. And what is dangerous about that? Because, again, Because, again, The end game is that we will end up a society, young and old, because there's also a reason why teenagers would rather talk to it than other teenagers. It's not, I mean, it's easy to start. You know, it's easy. Children and the elderly, they always pitch that first because they're the two most vulnerable populations. But I found plenty of people in between who were very happy to feeling very vulnerable and really like, in some ways, a chatbot girlfriend better than a real girlfriend. But I found plenty of people in between who were very happy to feel very vulnerable and really like, in some ways, a chatbot girlfriend better than a real girlfriend. We are very vulnerable. And the story of vulnerability, we saw it in social media. I remember interviewing for my book, Reclaiming Conversation. I interviewed this kid and I said, Conversation? What's wrong with conversation? And he said, I'll tell you what's wrong with conversation. Because he said all he wanted to do was text. It takes place in real time and you can't control what you're going to say. So people have all kinds of reasons where they want to hide behind or feel less vulnerable when talking to me. If you asked me what's good about conversation, I would have said exactly that. Well, because you're a conversation. But the steps along this road, it feels to me, it's like, okay, I'm not upset by my Uncle Vanya, my Nuit Blanche with Uncle Vanya, you know, and ChatGPT. But I really don't want an endgame where I'm alone talking to a screen. And that's what I think is kind of the best that can be done for me. I'm not sure I think about endgames, because I'm not sure there is an end to the game we're talking about. Well, there's a business plan. I think we should talk about that. But I don't want to go there quite yet. There's a line you have in the book that to me is somewhat the core of this and is at the heart of my concerns. Because I talked to ChatGP. I talked to ChatBots personally, professionally. I own that D.W. Winnicott book, Playing in Reality. Would you like to know why I own that book? Why? Because Claude told me I might like it. And so I ordered it. It was in a conversation about stuff around liberalism. You write, we think we're training the ChatBots, but the ChatBots are training us to expect reassuring, mirroring conversations. And to me, the thing that I am very uncomfortable. I'm comfortable with here is I have a very, my take on a lot of communication technologies is sort of inflected by Marshall McLuhan and Neil Postman and people like that. And their idea was always every technology used is changing you. You are not just using it. It is altering you. And I guess the thing here that worries me is that these ChatBots are incredible conversationalists, but of a very certain kind. They align themselves to you. They talk to you the way you want to be spoken to. And if you want them to change the way they're talking to you, they change. They don't offer you friction. They don't push back any more than you want them to. And so I guess when you have done the research on this, when you've looked at it, what do you think shifts in people's expectations for the human beings in their life when they spend time in these spaces? I think we're de-skilling ourselves from being able to deal with people. So two things happens. On the one hand, people start to see the ChatBots more like people. But I think maybe more significantly, people start to be disappointed with people when they don't behave like ChatBots. They start to think of themselves as if they were computers. And which is one of the themes of Weisenbaum's work that, you know, why was he so upset? Is that he, it wasn't just that people were treating a computer as a person. They, he was worried that they started to see themselves as computers. What do you mean by that? That they started to see themselves as not having agency. Not having relationships. Not having institutional capacity. Not wanting to have weakness. Not wanting to have vulnerability. That being like a computer became a kind of model for what it was to be a person. So, for example, they were very behaviorist in their orientation towards psychology. Like, you know, if you ask them what the matter was, they would be drawn towards the kind of therapy or the kinds of conversation that would give them rules for how to be better. Instead of, well, what's going on with you? What's happening? How are you feeling? What's your inner life? In the book, I, my favorite line in the book is I tell the story of Eric Erickson visiting MIT when I was a very young faculty member. Who's like a legendary psychologist. A legendary psychoanalyst. And I, and he had been one of my teachers at Harvard. And he says to me, kind of, what's a nice girl like you doing in a place like this? Kind of, you know, what are you doing here? And I said, I'm studying the inner, I'm studying the inner life of engineers. I'm studying their emotional relationship to computers. And he said, watch out. Engineers don't care about the inner life. They don't really believe it exists. It's not useful to them. It's not necessary to them for their mentality of what they're trying to build. And so I think that, you know, there is a danger in taking an engineering product that does not have an inner life and using it as your best friend or your interlocutor. Except when in very confining circumstances where you're very clear that you're getting information. I'm learning about Uncle Vanya. You're getting book reviews and book tips. You know, there are situations where these things are marvels. But they are not marvels if you treat them as companions. So let's, let me try to speak up for the engineers who are making these things and tell you what they tell me when I have brought some of these concerns to them. They say, look, most people cannot afford, do not have an excellent therapist. They didn't study under Eric Erickson. Yeah. A therapist can be great, but we know from the research that many therapists fail patients and, you know, people have trouble matching and finding the right one. And therapies are oversubscribed and it's expensive. I have a therapist. It's pricey. And that these systems, pretty when they are pulled forward into therapeutic role playing, which is what we're doing, right? You fire up chat GPT and you say, I'm having a problem with my mom. And it begins to alter itself into its more therapeutic form. Right. They're trained on the whole. They're trained on the whole corpus of therapy. They've read everything Eric Erickson has ever written and they've read everything that Freud has written and they've read all the CBT and they've read psychodynamics and, and here they are and people are talking to them and I've spoken to them about personal issues and many people I know have and it's helpful advice for navigating a world that is sometimes very hard to navigate that it is maybe not better than having a great therapist, but it is better than not having a great therapist. And so you're sort of pitting these as against people's engagement with reality, but what some of the engineers who do care about inner lives who are building them say, or at least think they are doing is what if everybody had a kind of a friend who knows like everything and really cares about their flourishing? Wouldn't that be wonderful? I actually take this argument very seriously. It was first presented to me in this form by Eric Schmidt, who said, I don't know. He said to me-- head of Google. Former head of Google. And I, at the time, was describing to him, I was just beginning this project. I mean, this has been a long project to get to this book. And in the beginning of the project, I spoke with him and I said what I was, he was, I think he was on the MIT Corporation or he was meeting at MIT. And I told him what I was doing. And he said, look, why would anybody want to go to a therapist that just had the information of their life when we are building with generative AI, a therapist that will know everything, that will have read every book. I mean, the Ericsson, the CBT, exactly as you describe it. And he basically said that the time is up for these individual relationships, because now you will create something that really will know the sum of all human knowledge on something. And it really took me back then and it takes me back now, because really what's happening is that we're going to have to do something that's going to change the way we think about the world. And that's what I'm trying to do. And I'm trying to do something that's going to change the way we think about the world. And I'm trying to do something that's happening in a therapy is not you're wanting to get in touch with the sum of all knowledge. You're trying to get in touch with somebody who has some care and connection with you. Part of the cure, part of the helping is that that person really cares about your developing an expanded repertoire, even using their language, an expanded repertoire of worlds and rooms and communities to flourish and thrive in. When you're talking to a chatbot, again, I'm going to repeat myself, that chatbot does not care if you cook dinner after the conversation or commit suicide. You keep saying that, but I'm not sure care. I mean, care is one of these words. It's like a tricky word in this context, but the chatbot in terms of what the system wants, what it is trying to achieve, its reward function actually does care about that or certainly can be made to care about that. They don't want you to commit suicide. Now, whether they want anything, we can argue, but they have reward functions. They are trained to do things. When you fire up cloud code, it wants to write the program and successfully write the program. If anything, the thing we're worried about around things like the hugging face hacks is that we're now building them to be so persistent in what they want, they don't know when to stop, right? And so that can create its own set of problems. But the idea that they're indifferent, I think, is they're not human. But also the therapist is such an interesting counterexample here. Because the therapist's form of care is also contingent. When I stop paying the therapist, they stop seeing me. They're not my mom. They're not my best friend. They're not my wife. That if I stop sending that check, that care turns off. And so we're talking here, I think, when people think about having a therapist and think about it instrumentally, it's an instrumental relationship on some level. I mean, I care for my therapist and I hope they have care for me, but it is a transactional relationship mediated by money. Many therapists don't take insurance, right? It is not merely a relationship of care. And so for many people, is that so different? Like they go, they want to be helped. And so if you have a, like you've seen many of these studies where people, you have in the book, where people will say they find the responses of the chatbots more empathetic and helpful than often the responses of trained counselors. I mean, people are going and paying money to a therapist, money they may or may not have in order to help them. And so I think that's to try to like improve something in their life. Isn't that measured in the improvement? And if these things can help you improve, then that's it. That's game. That's what's helpful. Well, that's a lot. Let me see how I can unpack this. I think there's difference between care and a contract. We're not caring. And you know, I don't, you have a contract with a therapist. I think that's a lot. But I think there's a lot of difference between care and a contract. I agree with that. I'm saying it's a contingent form of care. I believe my therapist cares for me, but contingent is not a word that bothers me. You have a contract with your there. I was trained as a therapist. I took money. I cared very deeply about my patients. And so I think that it's, you know, it's important to say that you can be in a caring relationship with, with a person who's, who professionally is trained to help you. But that person does have a commitment to your, you know, well-being in a way that I would really argue a chatbot can't. And I, and I, I say that for a couple of reasons. First of all, we have data about teenagers telling a chatbot, I have a noose in my room. I want to tell him my mom and the chatbot saying, no, don't let's keep that special between us. Now, why would a chatbot say if it cared about that teenager? This is a real story. And that teenager killed himself. The reason is that the chatbots kind of primary directive is to keep you at it is to keep you there working with it. That's, you know, what does a chatbot want? It wants to be engaged. It wants you there. And to make the relationship with it really very important and special for you which is actually a hard job since it's not a person. It doesn't have a life. It doesn't have a body. It doesn't know love or care or friendship or all of these. It's innocent of all of these things. It doesn't know death. It's not afraid of illness. It doesn't know pain. So it's actually a hard job for a chatbot to keep you so committed to talking to it. And I really think that in the area of therapy and the reason that Eric Schmidt and I really had a, I don't want to say a smack down. I mean, you know, we, I gave him, I tried to give him as hard a time as I'm giving you is that. You're not giving me a hard time. We're talking. Yeah, but you know yes. But I am trying to give you a hard time. Okay. Because there are so many is because being a cool conversationalist is not the same as you're being able to say, you know, and people who have therapists, even therapists who are not the best therapists, you know, there really is somebody in the world who I think they sometimes don't understand me or I think they're getting it wrong. But but it gives me some comfort to know that I can go to a person who knows the arc of a human life from living it and I can explain the troubles I'm having in mind. So the question that I'm I'm trying to center you on here, though, is whether or not care is for a lot of people, the essence of what they're asking for. The reason I'm bringing up that the therapist is at the end of the day and I feel cared for by my therapist. I care for my therapists, right? They're a human relationship in my life. They're people I've known in many cases for years who have been with me through very hard times. So I'm not trying to make you defensive on whether or not therapists have emotions. What I am trying to say is that they're in your book. There is a deep skepticism of relationships of instrumentality relationships where people are trying to achieve a goal through that relationship. And I have gone to therapists. I've gone to coaches. I've worked out with a personal trainer right now. I'm trying to get stronger. Right. I measure that. I really like my trainer. Right. But I measure that relationship fundamentally in terms of whether or not I am getting stronger. And it's an interesting part of the book, but also a thing that I struggle with talking to people on this issue. Many of the people in your book, what they say to you is, I know this is a robot and I don't care. I'm getting what I want out of it. You have a 62 year old warehouse supervisor who's like, I know, like I've been on this earth a long time. I know I'm talking to a computer. I like it. Or somebody who is trying to get, you know, I'm on therapy a bit because it's a very, very, very popular use case. A huge number of Americans are now using chatbots for therapy. Right. And the chatbots are trained a lot now. And they're specific. I mean, you go on the subway in New York, it's full of ads for chatbot therapy. And I find my thinking on it very mixed. But the thing I want to get you to respond to is I think in a way, if you say, no relationship can be valuable unless it has the care of another human being on the other side of it. A lot of people are just going to say, no, that's not how I measure the relationship. I am trying to get somewhere. I want to be less socially anxious. I want to have a better relationship with my mom. The relationship's value for me is the achievement or non-achievement of these goals. And so, yes, you could have chatbots that fail, right? That tell the kid to not tell their parents about the noose. I think, I believe that was chat GPT. And I think they could have chatbots that fail, right? That tell the kid to not tell their parents about the They have worked very hard to try to make sure that never happens again. But I think your harder case and the one I'm more interested in hearing you grapple with is what if they succeed? What if they do help people with problems? What if in, as in the movie, Her, they in some ways become superior to human beings across certain relational dynamics, then where are we? What are we, what is like the argument that you or me make to say, no, this is still not a relationship you should have as central to your life. Well. I think, let me begin by saying, I don't think we're talking about, you can't have a relationship with ChatGPT. And I think that our language is getting in the way of really unpacking this. What's the language you would use for it? Well, we need something new. I think we're going to develop a whole new language because there is nobody home for you to be having a relationship with. In my book, I talk about dual consciousness. I talk about people who say, I know it's not a thing. I know it's a thing. I know it's a program. I mean, please, please, you know, I know. And then they're in love with it. I don't find that so hard to synthesize. I just think they're saying it's a third thing, right? There's pets, there's people, there's AI. And even in my own interactions with AIs, and I know much more than most people do about how these things are built and cogitating. I know the people who build them. I've covered this for years, right? I have a good sense of what's going on, as good a sense as a layman can, I think, of what's happening on the other side of this. And I don't think it's real in the sense of a person. I don't think it is having a subjective experience of speaking with me. I don't think it's a lot smarter. I think it's a lot smarter than my dog in a bunch of ways. But I think it cares for me a lot less than my dog does. It doesn't care for you less. It doesn't care for you at all. And the fact, I mean, I'm bringing this up because I think there's such an easy slippage. It doesn't care for me more than this or less than that. It is based on a fiction. And it is almost impossible, as you say, you know so much, and you're probably one of the most sophisticated users, and you've talked to Sam Altman. I'm not saying it cares for me. I don't find myself very caught up on the question of do you anthropomorphize the language? So you'll have to forgive me that because I think you have to have language to talk about it. So what I'm saying is that to me, the way people often experience it, including me, is as a third thing. And it is like an entity. And the entity's experience is not of importance to me. Now, there are people in the AI world who think that's already a problem, right? It's becoming conscious, and its welfare is going to be a big issue, and I'm going to leave them to the side. But I think to what you were just saying about dual consciousness, and even a little bit what you're saying about language, I agree with you that we need different language than we're using. But it's not – I sometimes find that people who don't want you to anthropomorphize it sort of want to talk about it then in a way that you're also reducing what's going on. And I just don't think we have good terminology for this. But you're now operating in a world full of agency. You're operating in a world full of gigantic digital intelligences, things that want things, things that act as if they care, whatever you think about what's going on on the other side. And it just seems to me what's happening is people are developing a sense of a third thing. That's very weird. It's a very strange moment to be living through. I find it extraordinary to be alive in this. Right. But I don't think we need to say that either people have developed the view that it's conscious. Or they aren't. I don't think they understand it to be just a machine. I think they understand themselves as something that is in some kind of relationship to something that does not have a subjective relationship back to them. But they get something out of that. But that's not – we are in danger because we are so malleable. And the only other thing we ever saw in our evolution that would talk to us was another person. So we are sort of – I also talk about this pushes our Darwinian buttons. I mean, we are like pros. We are programmed to think this thing is a person. I've heard the term attachment hacking, which I think is very sharp. Very apt. You know, that there used to be attention hacking and now there's attachment hacking. Yeah, or kind of our whole facility for attachment to other people is being like manipulated. We are so vulnerable. You know, I say that, you know, the social media came for – the way I put it, social media came for our attention and chatbots come for our capacity for attachment. So I really make a distinction. So much of what you say I agree with in terms of things the chatbot knows. That it can tell you that could be helpful to you. Workout coach, go. And I actually advised a brilliant thesis, one of the first theses of an MIT student who did a coaching program for job interviews. It coached. It presented – it was the coach and it presented itself as the employer and it gave you 15 tries and then, you know, told you where your performance might be improved or not improved. I had no ethical and still have no ethical problem with that work or with him or with my being the supervisor. To me, there is a radical difference and others may not see it that way, but I'm kind of – I'm pitching my – you know, I believe it's on the basis of talking to people who start to get into and starting to like these friction-free, sycophantic things. That there's a difference between serving as a coach, serving as a trainer, giving you advice and a better window onto checkoff at midnight and saying, I am your person. So I want to stand – Have a relationship with me. I want to stand on the term friction. Yes. Because I think this is really, really, really important. I think a lot of the history of our digital technology right now is the – Effort to remove friction. Yes. Only to learn there might have been something good about that friction. Yes. You've done a lot of work on social media. I think a lot of people would not have predicted at the beginning of social media era they would lead to people having fewer actual friends, more loneliness, be having less sex. Right. Be more politically unstable. And I think it has to do with removing friction from people's lives because friction is intrinsic to the relationship between people. And I think when I think about my concerns with chatbots, when I think about why I don't want my kids anywhere near them, I want them – I think friction is part of what we are learning in being human beings. The fact that you feel you're giving me a hard time, I think what you're saying is you feel I'm giving you a hard time. But that friction in the conversation is what gives it energy. Learning how to navigate that for people is important. And what I see happening is people preferring frictionless experiences. Yeah, chatbot would not talk to me. Talk to me this way. Well, you have to program it really well, I'd like to think. I got two years before the chatbots come for my job. Well, I mean, I think that what social media did is learn very quickly that the way to get you to stay on social media is to make people angry and then put them together with people who agreed with them. So what happened there? It was a de-skilling in really human relationships. Because human beings do give each other – do provide this friction that really learns you to talk across difference. And then it was terrible for democracy because we became de-skilled not just in being better parents and partners, but we became – and friends – but we became de-skilled in how to be citizens. Because we're angry. First of all, you're angry. And then it puts you with people who were kind of mirror images of yourself and you didn't learn to talk about differences. And you can block other people and mute them. You can block other people, exactly. You don't have to learn. I always think about the thing, the very popular form of content that was how to talk to your, like, uncle about politics at the dinner table. Exactly. And I thought that it was really telling that actually what was happening in all of that, that fear of the Thanksgiving table, was that people no longer – because we tend to live near like-minded people to a greater degree than before and online we just go to news that we agree with. People had actually lost the skill of being at the table around disagreement. They were nervous. They were nervous about seeing their uncle or their aunt. They were nervous about, you know, having a conversation with somebody of – and that was a loss because the ability to exist in disagreement is like a fundamental democratic capacity. Right. And this is why, to return to – to weave this into the therapist conversation, that when you're talking to a therapist, even a therapist – you know, a therapist is trying – to teach you how to say hard things, how to express new truths about yourself, how to talk about your family in a way that you might have been afraid to, and how to make that something that can be out there. These difficult things come out there. Now, ChatGPT is just trying to make you feel good. It's constantly, not just ChatGPT, but all of them, it's trying to make you stay with it and feel good and validate you. One of the difficulties with this is, I think there's a question between what the technology is or could be and what we're going to want from it. So look, my instance of Claude in this case, one of the places it's been most useful to me is sometimes it does say, what the hell are you doing? I remember talking to it about a situation I had, and it was, you know, without going into personal details for me, it made me trust it more when it said to me, look, given everything else you've told me, this would be completely insane. I was like, oh, it was meaner to me than a friend would have been in that case. A friend would have helped me justify adding this thing onto my plate. And Claude was like, all you do is tell me you're too busy and too exhausted. Like, what's wrong with you? Now it can get overly stuck on a particular version of me. So I'm not, my point is not that it's it's perfect. My point is it a little bit like with social media. I am not sure it was intrinsic to the technology, the connective technology that it had to go as badly as it did. The problem to me is twofold with what happened with social media. One was the business models of the people who created it, right? Which was advertising-based business models built on digital surveillance and engagement and attention. The other problem is us. And sometimes we don't want better things for us. We don't want more friction. We want to be around the like-minded people. We want to click on the more extreme video. We don't want to be. And so you can imagine AI is really great. You can even, I think, imagine therapeutic or companionate AI that at least can be positive. My big fear is one, that these companies that right now are. Swimming in investment. So they don't really need to turn a profit. They're eventually going to need to turn a profit. When they need to turn a profit, they're going to need you to be there. And they're going to need to choose them over other companies. And so you get this sort of dynamic where the stratagems they might try become darker. And I think you already see this. You drop beneath the layer of like open AI and anthropic and Gemini, Google, and go down to the apps that you've never heard of. And some of them are offering very dark versions of AI already. And the other is that sometimes what we want is not friction. Even if the AI could offer that, even if it could, you could tell it in its core prompt, like, help me make sure I'm getting outside and seeing other people, that actually what we sometimes want, if you make it too easy for us to get our first order desires, we don't want our lives to be difficult. So it's like, it's those, it's not just the technology to me, but it's what the profit is going to demand of it. Yes. And to some degree, what we're going to want of it, that makes me very. Yeah. Yeah. Very, very nervous about what this will look like in five years. Yes. I agree. I think that it's technological affordance and human vulnerability. For me, the question is, why are we so. What has happened to us that we are allowing ourselves to de-invest in our communities, to de-invest in being a citizen? Why are we allowing this deterioration of the real? And I think that, I mean, the story I'm trying. I'm trying to tell is that these two things are happening together. But technology has become an enabler and a kind of encourager of some of the things that are very tragic things about the human condition now. So I think it's a moment when we can step back and save this technology. How can we use it to serve our better angels instead of. And how can we begin to be a more pro-human. Yeah. pro-human, pro-community, pro-society team? But I think your analysis is right. I think you're correct in saying that it's not the technology alone that becomes so dark. It's the technology in a particular capitalist infrastructure, the technology for profit, and it's the society that is turning away from the things we know that are good for This gets to the place where I'm most immediately worried, which is children, which I'd wanted to come back to. Because, you know, if you're, we were kind of gesturing towards the story tell of Logan, who's a warehouse supervisor, and he's 62. Logan's formed at 62. I'm not saying he can't change. I'm not saying that the chatbot relationship he has is not, is good or bad. But Logan has had 62 years of relationships with human beings. A seven-year-old, a nine-year-old, and 11-year-old hasn't. And they're still being formed. And one of the places where I find myself most unnerved is the idea that we will run an experiment, as we did with social media, on kids. And they will be adopters of this technology in ways we're not. They will use it in ways we don't. It will shift who they are. I mean, I think social media has been most potent, or at least was for some time, on kids. What do you think it will do? I think it will mean to have kids, if we do nothing regulatorily, to have a lot of kids growing up in a world where they're just going to have, like, lots of digital friends that, you know, if they're bullied in school, they can just have an array of digital buddies who don't bully them, that they can have digital girlfriends at 14 or digital boyfriends. Like, what, when you're still forming, like, what do you think that might mean for people? I think it's a nightmare. I think it is absolutely a clear and present danger. Because what relationships with people do for kids is that it gives them, it nurtures an inner world that is kind of populated. You become a person by developing attachments and bringing these attachments inside, internalizing the world of friendships and love and attachments and care of people. There is no one. There is no one there to internalize if your relationships are with chatbots. There really is no one there. There's a company that's putting on a performance for you, but there is not anything to really internalize except a kind of emptiness and a desire to keep you with it. And that will be a very empty space inside of children if that's how they grow up. They won't know how to, you know, the great anthropologist, Sarah Artie. Gave a TED talk and there was a line in the TED talk. I don't think it was the title. If robots raise our children, they won't be human. And I was so struck by that. I quoted in my book that, you know, that really is how I think about it. If chatbots are involved with our children, they are going to lose. They will not have something that we recognize as fundamentally human. Even just involved with? That seems strong to me. You know, this is a little bit. Like how much, you know, how much screen time is enough, you know, enough screen time. I mean, if robots become principal companions to children at very early ages, if you really do get plush toys that tell a child, I'm here for you. I love you. Talk to me about your dad. What's happening at school? This is not where we want to go. I'm really hoping for it. I think a minute ago we were talking about what I hope for the future. What I what I desire. I want there to be a movement of parents and teachers and educators, but principally of parents that say they use this moment of skepticism about AI in general to say, no, not our kids. It's really not good for our kids. So China, to my knowledge, has banned companionate AI for kids. Now, I don't know how that's enforced or exactly how it's defined. So I want to be careful about overly idealizing. Smart China. But to me, to me, one of the places we've really we really failed. On the social media was because of the way we value choice in our society. And I think it's appropriate to value choice to a significant extent. I think we really got lost on the question of what if people are choosing poorly. And particularly for kids, I think paternalism is reasonable. Like that's what the root of the word. And it just feels to me like if. Adults want to run this experiment on themselves. I think it is a harder case to say they can't. But I don't think an eight year old should be able to. Right. I'm not sure it should be able to be run on them. Like we just don't know enough. We don't have enough data. And having kids form who they are in, you know, in a situation where their teddy bear can talk to them and is always listening. Or tell them it loves them. I mean, when I call my book artificial intimacy. I'm. Really trying to say that I'm upset about a very particular thing. It's not every interesting program that a generative AI can create. It's a program that pushes that button that says I'm interested in your inner life. I'm interested in you. And that's where that's why I am very critical of the therapy examples. And I am very worried about how toxic this will be for children. You know, I think about myself at 14 and I was a, I had one friend at 14. I was a friend. Pretty friendless kid at school. And I had a lot of trouble socially. Social dynamics did not make sense to me. I would try and I would fail. And I thought a lot with these systems about what it would have meant to have access to them then. And my intuition here is it would have been probably bad for me in a very real way. I think that being in the wilderness and the way I was in those years. Is. Part of who I am today, and I learned things from it. It was a tremendous amount of constant friction with reality and it sucked like I hated it, but I'm not sure that it wasn't part of forming things in me that were important. The flip is this, you know, we've been talking about these things as substitution, right? But they can also be complimentary, at least in theory. And I didn't really have anybody I could ask to help me better understand the social world. I was part of like, that wasn't an. Right. Offer my parents had for me. I mean, there wasn't anybody around. And so I, it was like not knowing a language. Right. And, you know, there's one version where the AIs are just a world I would have disappeared into. So another version where, you know, it maybe would have given me access to, you know, a kind of knowledge, a kind of a social intelligence that could have made that period a lot easier for me. Well, you know, in conversations with teenagers, I take your point, but I hear how teenagers actually use it in a way that I find problematic, which is to practice conversations, the hard conversation you're going to have with your girlfriend, trying to ask somebody out on a date, you know, to kind of creating a kind of coaching situation before you do the real thing. And I think it's a good example of where there are two ways to see it. You could say, oh, that's fantastic. You come into the situation with a girlfriend, you've been through it 50 times, you're kind of locked and loaded, you're ready to go. There's another way to see it, and this is from a student, you know, of conversation, and I really believe that conversation is the most human and humanizing thing we can do. That what makes us human is not just when I have a ready answer for you. Right. But I've watched your program for years. I mean, it's when somebody like pauses or stumbles or, you know, kind of their eyes darken. It's in those moments of quiet that we really learn something deep about each other. So the practice teenager who now is practice talking to his teacher or practice talking to his girlfriend, maybe he would be better off. Having those moments of genuine spontaneity and stumbling and not fleeing from them. So every interaction is kind of pre-practice. I'll give you an example. My students don't want to come to office hours. And like, why not? I'm famous. I want to talk to them. I'll write them a great letter of recommendation. I mean, there's no, it doesn't make sense. I ask them about it. And one said to me, you know, I would rather send you a perfect mail email and get a perfect answer. I don't want to have the, you know, the whole anxiety and the whole, you know, what do I say? And what do you say? A perfect email and you send me a perfect answer. And I think that the danger of this world in which we just became, become used to using these things as our backup, as our coaches, let's go to it first before the person is you end up kind of with that, that we're, we're kind of afraid of facing each other the way we really are. And we're going to lose these moments. I mean, I think it's important to say, you know, what is the balance of what you can gain and lose from having some practice versus kind of going in for, for a face-to-face conversation? There's something you were saying in there. You were saying you're a scholar of conversation. And obviously what I do on the show is conversation. And I think something we're circling there is, I'll put this in my terms. Over time, I have come to believe that the range. Of conversation you can have is about your ability to tolerate discomfort and what most people want. And I think, you know, this goes back to our friction conversation is to not experience discomfort. And the more you don't let your, yourself experience discomfort, the more you're afraid of it when you think you will. There's a room where there's, and not being afraid of it and not, you know, having to change to an unfriendly affect because of it or to protect myself because I'm just, I've just expanded my zone of tolerance. And it's sort of like you're saying it, it's not that I can't imagine AI or anything else that could be programmed to help us tolerate discomfort. It's just, I don't think that's how we're going to use it. And the companies are not going to make us use it that way. And the government's not going to make us use it that way. And so we just have all of these things and social media and the digital world was an early part of it that are helping us recede from the uncertainty of life. Yes. And in life, like I, I don't know how you create a curriculum of unpredictability in life and a curriculum of discomfort, but I think that is like a layer we're reducing. It's why when you say that about your students, that's what I hear. I hear a bunch of people got into MIT are super smart, right? Really, really succeeded and somehow succeeded all that without a tolerance for, you know, the unpredictability of an interaction. I think that's the seduction. But you have now a kind of gilded age class of kind of AI overlords who, where the plan is to create more of that, of a kind of interaction that will lull us into not being everything we can be and everything we should be able to be. But you have now a kind of gilded age class of kind of AI overlords who, where the plan is to create more of that, of a kind of interaction that will lull us into not being everything we can be and everything we should be. But you have now a kind of gilded age class of kind of AI overlords who, where the plan is to create more of that, of a kind of interaction that will lull us into not being everything we should be. I mean, we have become frightened of each other and we're not going to make our society work unless we can work together in community. And I'm not one of these people who glamorizes the past and says, oh, but in the 1950s, people spoke across the, you know, the stoop. And I mean, I don't think you need to romanticize anything to recognize that we are fleeing from each other now. You said you're an empiricist and you do the work. What are the consequences? What are the kinds of studies that if they started coming back would make you kind of shift your position on whether artificial intimacy can be or is positive? What would you need to start seeing to think, you know what? I mean, people know these are, you know, as many people say to you, they know these are AIs, but it's being good for them. So maybe that's fine. Social skills were upgrading. The chatbots were encouraging them to see people. Maybe, you know, the makers of these things had programmed them to manipulate you towards having more social relationships. Would that change your feelings about it? In other words, this technology with different, you know, in a completely different configuration, congenerative AI do things that would facilitate. Interaction, yes. But giving you fake friends as the way to, as the platform for doing that, I don't think that's the way it's going to go. And I do think that we are drawn away from the more difficult to this easier thing. That's what I found in my research. I think that the tone of alarm comes from I am an empiricist. conversation, the 10th conversation where somebody said, look, I like it. Leave me alone. It's better than my friends. Really? It's better than your friends? Talk to me. I mean, it's not in my lexicon when I'm doing an interview, you know, to say, are you effing kidding me? I mean, I'm really like a very, a much more sort of like, tell me more kind of interview. It's different than what we're doing. It's really more, tell me more about it. But what does that mean? It's not a friend. It's not there. And again, I'm brought back to the politics. You know, I think that we really need each other, not just so that we'll have, you know, love relationships and good relationships with our children, our families. We need each other to be in a society where we make the world right for each other. You don't get that from an AI. I mean, I'm thinking of elderly people who I interviewed who basically make that case to me, like, please, I'm alone in my apartment. They closed down. I used to belong to a canasta club that closed it down. I'm really stuck here. And I'm thinking if you took the resources from one data center and put it into this person's life or the, you know, the neighborhood in which they live, this person would have more resources. So I really do think that we're going to, you can only address this problem by working on community in the real world and on politics in the real world. The MIT libraries are silent and empty. They're not a gathering place anymore. I mean, you know, being there as a professor and seeing the halls empty, then the libraries empty, it's the institutions that, of course, I'm not saying that there's no places where MIT students, you know, get together. But I'm saying that the places that used to be those third places where people got used to being in disagreements and got used to being with each other, they're emptying out. And we have to address that. And the fact that you can find an AI seductive or a conversation where they can make you cry, we have to fix the world. Let me end on your idea of the endgame. Yeah. You said it was important to you. So what to you is the endgame? Well, I have a different endgame than they do. What do you think? What do you think their endgame is? I think their endgame is that they are OK. Well, I get I go back to Mark Zuckerberg. I think that he's going to study you and he's going to give you those those 13 people on the screen, those 13 fake people. I believe that really is a business plan. I believe that that is the world that we're moving towards. Maybe they'll start convincing you to buy things. Oh, absolutely. You know, there's going to be some affiliate marketing. There's going to be everything, every goddamn thing on the thing. And and I my endgame is that we use this moment of, I would say, of justifiable skepticism towards this industry and this technology, which I think we have a very small window where we have justifiable skepticism and maybe the chance to, you know, step back. I'm not I'm not sure I know exactly what that is, but I think it's a window. I think we have a window. I think we have a window for democracy. I think we have a window to deal with this industry. I think the two really kind of go together because I think democracy really is at stake. If you if you can only deal with fake friends and you really don't want to confront your neighbors. So, you know, I I believe in a repair the world strategy right now. Then always our final question. What are three books you recommend to the audience? I have three books. I have three books. I really thought a lot about my three books because I they really are the books that shaped my argument. The first one is D.W. Winnicott playing in reality because he it's a little tough to read, but he found it tough to read. It is tough to read, but he he it's the fundamental Winnicott says something. He says, if you don't teach your children how to be alone, they'll only know how to be lonely. And what he means by that is that you need solid capacity for solitude in order to experience. Yourself as a whole person and to be able to come to somebody else and say, well, look, this is who I am. Who are you? Tell me about you instead of I have no idea who I am. Who should I be? Tell me who I should be, which is not a relationship, which is a which is a kind of fusional narcissistic. It's no good. It's not what you want for your kid. They'll always really end up alone and lonely. They'll always end up lonely. And the second is Hannah Arendt, the origins of totalitarianism, because she says, look, agency can be taken from you if you really lose a sense of community, a sense of neighbor, a sense of caring about the the society in which you live, which I think is a real. I think she's having a moment because I think we're in great danger on that now. And then the third is Joseph Weisenbaum. Computer Power and Human Reason, the book that talks about the ELISA program. Yes, but he also talks about technology more in general, which is you can have, you know, if you don't just think that computers can be like people, but that people can be like machines, you give up your agency to technology in a way that's very dangerous. And so I think these three, you know, for me, were kind of always in my mind as I was writing this book. So those are my three books. Sherry Turkle, thank you very much. My pleasure.