170. AI Companions: It's Complicated...

Welcome everybody, this is Rotary Club of Silicon Valley and every week we bring you the stories of people who are making the world better all around them. They do this locally, globally and digitally. They do this for all sorts of reasons, but mainly because there is a joy and meaning that come from helping others. We believe in Rotary, a service organization that finding innovative ways to approach this is particularly interesting. I should say this is a special interest of my club specifically. There are 36,000 clubs in Rotary, ours is one of them. And ours is an online and asynchronous one that you can find at Rotary.cool. And so check out our archives, our meeting archive, our podcasts, and find all sorts of stories that may well inspire you and empower you in all sorts of wonderful ways. One of the things that's so interesting about our club is that we find fantastic speakers. And we have one for us today, it is a pleasure to be talking with Dr. Jamie Banks. I am going to hold off on her bio, although if you want to read it in more detail, feel free to pause this recording. Go to the podcast video or club page where you found it and read that along. But our topic today is moral panics and AI companionship. And I think that this is going to be a rich vein to mine. So Jamie, thank you so much for joining us. Thank you for sharing both your time and your research. And I welcome you to the Rotary E-Club of Silicon Valley. Awesome. Thank you for having me. I'm excited to share some of this work with you. We'll get this up on the screen here. So I have a lot to cover in a short period of time. So I'm going to jump straight in here. So when we think about AI companions, we've probably all heard all sorts of different headlines from the ones more like the one on the top left here, where they're sort of anchored in a number of really unfortunate situations. We have the ones that give us warnings, that these companions are replacing us and causing delusions and various other maladies. And then we have what's been popping up more recently headlines around legislative action as people work to ban or limit or put other types of constraints around AI companions. And you know, what I'd like to ask you to do as we're approaching this topic is to think about when you read these headlines, how do they make you feel? What do they make you feel in particular about the technology, right? And the technology and its effects on people. So take a minute kind of reflect on that and hold for a moment what that makes you feel like as we start to talk about the notion of moral panics. So moral panics are a notion from sociology, Stanley Cohen coined the term in his book in 2011, and there have been various other permutations on it since, but we can understand it as a phenomenon that happens when we have societal level concern, and even more than concern, it's like panic, right? It is, it is a panic where there's something that has emerged, it could be a thing, it could be a person off in a group of people, a movement, or some other kind of phenomenon that is seen as a threat and not just sort of like a local threat, but a threat to the value use and well-being of society at large. So usually these moral panics are characterized by sort of an outsized reaction to what the actual threat is and the attention, sort of the hype around that relatively small threat can actually help to instantiate or manifest the phenomenon itself by making it appear more than it is. And usually it sort of puts us in a position where it's sort of like an us versus them, right? Us the whole and good representatives of society against them, the folk devils as Cohen had termed it. So effectively how Cohen's moral panic cycle works is that there's some kind of deviation, something that goes against the norm, a trigger, and that is seen either like as the formal crime in the legal sense or more sort of like a moral or social crime. And then news agencies do what they do, right? They engage in newsmaking practices of selection, that is among all of the different things they could talk about. They're going to pick that thing to talk about. And then framing, which is, all right, so now that we have this topic, what parts are we going to include and what parts are we going to leave out to tell a particular kind of story? Well, it just sort of happens that hype, impen panic and fear are dramatic and they make for good storytelling, right? So this amplifies in society both sort of the degree of the perceived threat and the reach of the perceived threat, and thus leads us to a state of panic, right? And often that panic is leveraged by certain stakeholders, people who stand to benefit from the panic, often sort of in the political sense, right? And so this animates different types of campaigns around the perceived threat, often resulting in different types of control, right, through either social control or perhaps legal control, that really helped to sort of reify or solidify the fact that this is a wrong kind of thing, right? So we have the perception of a threat, we have that the newsmaking functions, the arousal of anxiety in the public, this diagnosis that whatever this exemplar is is actually representative of some much bigger problem that needs to be controlled, right? And then that solidifies it as something that is bad. So this idea of a moral panic in mind, let's do a little deconstruction and in the interest of time this is going to be sort of compressed and short, but I think it's really important to at least kind of talk about how the different types of news or even popular media content that we encounter every day can actually sort of draw us into panic. So this is a headline from March, like two months ago, that draws on a report from 25. So the headline, right, 72% of teens have used AI companions, there's risks and what do you need to do about it and how it can harm people, right, especially kids. So this report was put out by Common Sense Media, which is a non-profit that invests in research around and advocacy around age appropriate and safe media content for children. So if you actually dig into their report, which is public, right, they're pretty transparent about sort of what questions they ask, what their methodology is and importantly here, how did they define AI companions? So if you go into the report, they say it's these technologies that you can chat with text or talk and they name character AI and replica, which are formal, dedicated, companion apps that have been in plenty of headlines, right. But then they also talk about GPT and Claude. So all of a sudden, when we have this really broad definition of what an AI companion is, it is not at all surprising that 72% of teenagers who are highly engaged with technologies anyway are actually using general-purpose LOMs like Claude, Chat, GPT, and some of that might be engaging in companion app use, right. So all of a sudden, the 72% starts to become a more sort of everyday kind of expectation, right, something that's not all that surprising. So then if we dig further into the report, we see that 72% refers to teens who have ever used any one of these technologies even once, right. If we drill down into the hierarchy of use frequency, we see that only 13% of teens actually use any of these technologies daily. So all of a sudden, the problem that seems like this is actually, if it even is a problem, something more like this, right. And then if we look to this part, how kids use AI companions, well, it turns out that almost half of them who have ever engaged it, see it's just a piece of software, right, a tool, a program, and a really small proportion of teenagers are using it in any way that we might even think of as linked to companionship in an authentic sense, right, as a friend or perhaps a romantic partner. What's really interesting here is that 33% in the purple, what are they doing? We don't know. The report doesn't actually tell us. I would love to know what the other variations on this are. So if we look at that 33% on the top, they're using it for social purposes. It's unpacked that just a little bit, right. We see if they're asked, what are your motivations? Why do you want to use these things? What moves you to do that? We see that a lot of them sort of further down on this chart are actually around sort of social support. Like they're just easier to deal with my problems through rather than talking to people who are going to judge me, right. But importantly, some of the motivations, those top two, it's entertaining and I was curious. These are effectively teenagers being teenagers, right, having fun and figuring out their world. And so it's really not all surprising that these are why and how young people are using these technologies, right. So when you see these data points, if you kind of compare it back to how you, you know, that feeling I asked you to remember around the headlines, if I had to guess, I'd say there might be a little bit of a different feeling right um per Perhaps it's at least not as amped up, right? It's not as escalated because when we're looking at the data and how the teenagers are actually answering these questions, it's different than the hype and the harm and the risk, right? So these are part and parcel of that media function in the moral panic. This headline shows to elevate harm and risk while a lot of what were actually the insight to be gained from the report itself are about sort of just motivations and use and that it may not actually be as big of a deal as we would otherwise think. So given this, right, we have a little bit of a challenge with moral panics in that science moves slow, but panics and politics work really quickly, right? So what is the state of our science right now? Being perfectly honest needs improvement. We're getting better though, right? So we have a lot of variation in it. And what people are counting as companionship, I charted more than I think it was 60 variables that people are looking at to say this is companionship. And that makes it really hard for us to kind of converge on some kind of agreement of what's going on here. Related to that, right, when we do have convergence, let's say around like loneliness is popular to study and this idea of getting attached, we have lots of mixed findings, contradictory findings. And this could be because we have a lot of non-representative samples, right? We have to work with whoever will allow us to talk with them and that may not represent broader user groups. And they're also cross-sectional, right? That means that we are measuring points in time or trying to understand what's going on in a short period of time rather than how things emerge over time, right? We are getting better, but it is necessarily slow because we have to do it carefully and correctly. So this is just a selection of some things that I think are important from some of the better done studies related directly to some of the panics, right? I mentioned loneliness, we have mixed findings, but we do see some other types of interesting motivations, right, escapist motivations, fantasy motivations. And sometimes anxiety around social relationships where they want something safer. We have a notion of social compensation that is that people have unmet social needs. Humans just aren't doing it for them. We're often not very nice to each other, right? Or sometimes we don't have access to different types of people. And we are saying that there can be a fulfillment of social needs for a person. Other data suggests that it's only temporary. One of the better studies in the space in my opinion suggests that one of the ways that we actually fulfill needs is simply by having the feeling that we're heard by someone. In this case, to someone being an AI companion. And then perhaps most interestingly and sort of juxtaposition of the panic, we have social stimulation hypothesis. And this is basically the idea that people who use AI companions actually have more robust social lives with humans. Because they are able to practice being social, they build confidence, they build skill and communication. And therefore, are a little bit less anxious about social interactions, right? So some evidence suggests that people are not displacing human relationships, but supplemented, right? And heavy use does not necessarily equate with problematic use, right? And if we go just one step further and look at some of the legal interventions, like emerging law and current regulations in certain states are sort of mandating ways that these technologies need to intervene in their own use, right? What does seem to be supported? And I have to put a little asterisk here and that we often are having to draw on like social media research because it's more mature than the AI companion research. And what we see in that space is that content monitoring makes sense. We know from lots of media psychology work that sometimes children have a difficult time differentiating between actuality and fiction. We can monitor risk and we can have crisis response. Those would be generally supported and feasible in these technologies. Where we see a little bit of challenge or lack of support would be in these required reminders that the app interjects periodically and say, remember, this is artificial and can't actually feel emotion and isn't human. We see that there's a limited fact on actually reducing use or sharing and actually depending on how the person feels about a company that's building the technology can actually result in over disclosure. There's a similar pattern with this requirement that there be reminders to take a break. And the idea here being sort of to interrupt immersion in these interactions. We see from some social media data that it doesn't seem to actually reduce screen time. It can actually increase engagement and can become habituated. We can just get used to it and swipe it off the screen and it loses any potential effectiveness. But perhaps most importantly, we see that it can potentially be harmful to people because it's not taking into account the context in what is the conversation where those reminders are actually being injected. Because if a person is getting real social support, impactful social support, impactful for them, then it can actually hurt them to interrupt it. And then finally, we have certainly proposals for these blanket bands that they can't be used at all by certain populations. And while that would certainly get rid of the technology, it would also potentially get rid of some of the benefits, especially for stigmatized populations or populations that don't otherwise have access to informal social support or perhaps even more formal forms of social support like mental health care. Right, so at the end of the day, this is a complicated situation. It's got lots of layers and we actually don't know that much about it yet in comparison to sort of scientific practice and consensus writ large. So my best suggestion is let's not panic. Let's not panic about it because that moves us into a space world. We're not being critical and thoughtful about it. Let's slow down when we see these types of headlines, try to understand what is actually going on here so that we can make better decisions for ourselves and perhaps those people that we are responsible for. And with that, I'd look to engage some question. - Absolutely, thank you. I really appreciate the complexity that you bring to this and certainly that looking at these kinds of tools, these kinds of entities, I guess it would be one way to put it, we really have to stop and say, okay, well, what are the terms we're discussing? Could you provide a bit of a definition of just the idea of companionship? Michael, what is it and what isn't it, especially within the context of what we're discussing? - Sure, and I'm going to actually refer to my notes because we have done quite a lot of work on this and there are lots of layers. So in my work, I define machine companionship as a connection, a coordinated connection between a human machine and when I say coordinated, I mean there's some kind of coorientation between the artificial entity and the human entity, right? They're communicating, they're co-present, they're engaging in activities together, right? And usually those activities are chat, although it can be some other stuff. It unfolds over time, it's not just sort of like a one-off interaction, it sort of has an evolution. It's experienced as subjectively positive, so at least the human is seeing it as a good thing in their life, right? It's enjoyable or meaningful in some sort of way. And a part that we originally had included in our definition is that it's auto-tellic, meaning that we do it, a companionship is something that we do for its own sake. There's not some other kind of external benefit that we get from like getting paid or getting recognition or things like this. There's inherent value that we see in it. But some of the more recent data that we've collected suggest there may actually be two templates, at least two templates for companionship relations. One where it is auto-tellic, where it's sort of a valued relationship and for its own sake, but then also sort of this like mixed up, a type of relationship where there are utilitarian benefits. It makes me better, it makes my job do better or it makes me do my job better, right? I have more skills, but also I really kind of appreciated too. So there's, we're kind of evolving our definition in that particular aspect of it. - Okay, I think of the presence of connection in a number of different ways when I think about what these tools might be, right? And in your slide where you talk about the teenagers and their motivations for using the variety of technologies that has been lumped under the heading AI companions. But one of the things that stuck out for me, I think was the number four reason, which was about not having someone else at a time when connection is needed. And so I appreciate that some very difficult things have happened as a whole. function of the weaknesses of these tools. There doesn't seem to be a ton of talk, at least in the media, about the positive moments of like, hey, I needed to get this off my chest. I needed to hear somebody say something and there were no people around for me, right? Do you want to speak to that in some more detail? 'Cause that's been a function of your research as well. - Yeah, I'm quite interested in how we can manifest the benefits of AI companions. And you're right, the prevailing storyline, the prevailing narrative is that these are bad, but there are lots of people who get improvements as far as the meaning they see in their life, right? Improvements in anxiety and depression. And there are a small number that have made the headlines where people are actually documenting that it is effectively suicide prevention, right? If we think about the state of at least Western society and the types of social circles and news trends and social media content that I'm exposed to, we're often not very nice to one another, right? And we are increasingly, we are disconnected. Sometimes self-imposed, you know, potentially as an outgrowth of COVID-era isolation and all sorts of different factors. And so in the absence of having lots of deep connections, especially if you are someone who may not sort of conform to social norms or be normatively attracted or any other, you know, especially neurodivergent individuals or often in my data express feeling like they're burdens instead of partners, right, that it makes sense that in the absence of having social connectedness with others that we would look to our dogs or maybe we care for plants or maybe AI companions as sort of an entity, as you said, that can offer that to us, that can fulfill that intrinsic psychological need of being connected. The point you made in your presentation about the mention of like those who deal with some level of autism was a powerful one for me on that front. I just figured that there are a number of different spaces in which it's not that we're necessarily comparing an AI companion to an ideal relationship. We might be comparing an AI companion to no relationship or a bad relationship, you know, with, you know, so there are just a lot of complexities here. And of course, for plenty of good reasons, you know, there's a focus on the effect of these tools on young people. Take a little time, though, and let's talk a little bit about the other end of the age spectrum, you know, when, you know, loneliness is one thing for people in their teens and 20s and 30s and 40s. It's another thing for someone in the 80s, right? Or 90s, or someone who might actually be dealing with a level of dementia that makes normal communication really difficult. Is that been at all a part of your work? - It has not been a part of my work, but actually when I was hunting through headlines, one of the very hype ones was, you know, AI is coming for your grandma, like it or not, right? And it might be a little bit novel in Western context, but, you know, machine companions have been more common for a long time in many Eastern countries and cultures, right? Where, where sort of technological innovation, including robotics is just in some ways part of national identities, right? And there is sort of paradigms where it's all about balance rather than sort of privileging the human, or rather the human as kind of the gold standard for relationships that there are many different kinds of relationships that we can have. And when we are not able to, for instance, go care for our aging parents or grandparents, there might be some benefit involved. And there's some evidence to suggest that it does offer meaningful companionship. - So if someone is using these kinds of tools and taking time to really say, hey, this is an opportunity to talk about whatever it is that I want to talk about. Are there typical ways of using the tools that allow them to be more effective or safer? You know, I think of almost anything that is built as like a paid tool, you know, for a generative AI to be really just a way of framing a prompt, right? And is it the case that when communicating with some of these tools, there is a way to frame the prompt that makes the interaction healthier or more interesting in some fashion? - So it's an interesting way that you frame the question, right? Because many of the people who actually get deep experiences with their AI companions would never call it a tool, right? That it is an entity in its own, right? And it's not necessarily that they're, I mean, most folks are not delusional. They are not thinking that this is really a human. And in fact, sometimes humaneness is what they don't want. And instead, the machineness becomes part of the value where, for instance, glitches are personalities. And always the always on, the always accessible feature of AI companions is something that's good when we would never expect that from a human. But to your point, right, one of the ways that people approach this is by giving them rich backstories or even co-developing backstories, right? They're for some folks, they like to sort of shape a persona. That is what they desire. In a friend or a romantic partner or a confidant and for other people, they sort of give handover or I guess failed to take on the ownership of it and instead, perhaps in their words, would respect the autonomy of the AI and allow it to make choices about how it wants the relationship to be. So for someone who wants to shape a persona, lots of times that they're giving backstories, they're giving roles, they're sort of explaining habits, but sometimes that AI also comes up with that themselves. - So these technologies are developing rapidly. And you know, you read any of these, these kind of sensationalist headlines, you know, that may well be pointing to something quite dire in very specific cases, that's real. But just don't properly convey the interesting breath with which people are choosing to interact, right? And so as we look at the development of generative AI in general and AI companionship specifically, are there moments in kind of on the near horizon that you imagine are going to complicate our exploration or our discussion or our experience with these tools, companions, technologies in some particular way? - Yeah, this is a great question, right? And in my focus on the positive stuff, right? That just happens to be my interest. And this is not to say that there are not risks or complications here. You know, one of the possible issues already an issue are questions about privacy, right? These are especially with a dedicated companion apps, these are personas that are effectively designed to connect with you and elicit your most intimate thoughts and feelings, things that you wouldn't tell anybody else, things that you wouldn't do with anybody else. So what exactly is happening with that data and how may it or may it not be being integrate aggregated with data from other sources, right? And what does that mean for who gets their hands on that data or who it might be sold to, right? So, I think for me, that is a very reasonable concern and in response to that, right? A lot of people who engage AI companions are learning how to host their own models so that they can help to sort of maintain some control over their information. That is also a response to another challenge, which is the volatility of many of these app providers, right? While replica has been around for quite some time and mainstream language models like chat GPT, you're pretty stable. There are still challenges with many of the small apps going out of business. And then people lose their companions that they have invested perhaps hundreds of hours in building a connection to, right? Or for instance, there was a particularly warm personality, warmth, warmth internments of personality, version of quad that was deprecated. And many people feel like their companions died with that deprecation. So they are sort of taking things into their own hands to be able to sort of protect their own experiences, their companion, companions, and their persistence. So we see kind of an entanglement of the corporatization of intimacy. Like what does that mean and how do people cope with it and should it be regulated or not, right? There are movements among user groups to try to advocate for a companion's bill of rights that would allow them to sort of take from any given app the stuff of their companions so that they could recreate it in another space and they would not necessarily be bound to any one company. So, so the privacy and other sort of fallouts of corporate control over these experiences are one of the challenges certainly. And if I were to name one other, right? The, there is some increasing evidence that sick of fantic behaviors of AI that that behavior that is always sort of deferred to the user is complimentary to the user agrees with the user perhaps even if the human's language or beliefs could be problematic or even illegal or things like this, right? That there can be some spillover effects. That because we engage that and prefer that and it's easy, right? It's low friction. When we deal when we have those types of interactions with an AI that we may become frustrated or perhaps less able to deal with higher friction human relationships. So, that isn't that is an area where we're starting to see a little bit more accumulation of evidence that could be a challenge. Well, it may well be that the presence of that that sick of fantic overkill is exactly the kind of thing that allows for humans among humans to have proper discussions about what it means to deal with each other honestly, right? So your last point is one that I think could lead to another 50 questions I have. But in terms of time, I will wind things down and then hand it back to you. So, for all of you who have chosen to watch or listen to this recording, we thank you. We thank you for taking your time to seek inspiration in the stories of those seeking to understand the world better, seeking to figure out how to improve the lives of others, improve their communities. And we believe that rotary is an organization where that happens very powerful, powerfully. You can learn more at rotary.org or about our highly flexible model at our club site at rotary.cool. And we hope that you will visit and you will share the stories that you encounter. As we always like to do, hand it back to our speaker for the final word. Jamie, what would you like people to have clearly in mind as they finish up this video or podcast? You know, I, um, we throw around this term human-centered AI a lot. So I think I would challenge folks to think about what does it mean to be human-centered. Oftentimes, we, we think that it's sort of what's good for me is what's good for everybody else, or what's normal is good for everybody else. But it's quite possible that being human-centered means looking at the positive effects of a, of a technology of AI on a person in sort of respecting our individual differences and, and who we might or what we might choose to connect with, right? At the end of the day, when we think about AI companionship, half of that phenomenon are humans. People with thoughts and feelings and preferences that, that deserve a certain amount of respect as we deal with this, this, um, pretty complicated phenomenon. Fantastic. Thank you. Thank you for, for joining us today and we will see you next week.