Did a Chatbot Cause Her Son’s Death? Megan Garcia v. Character.AI & Google

This week on the Grey Area, we're talking about what unites us. We've kind of created a society now where the really the monoculture is just football and Taylor Swift. Those are really the only things that are like that now, and I'm not being sarcastic. It really is the case. So what does that say about American culture? Listen to the Grey Area with me, Sean Elling, new episodes available everywhere. I got in the water in the very early morning before the sun had risen and the water was pitch black. I started swimming and I felt the water hollowing out around me and felt like something really big was swimming below. I'm Phoebe Judge and this is Love. A show about the surprising things that love can make us do. More than a hundred episodes available now on This Is Love. Hi everyone from New York Magazine in the Box Media Podcast Network. This is on with Cara Swisher and I'm Cara Swisher. Today I'm talking to Megan Garcia. Her 14-year-old son, Suel Setser III, took his life in February and Megan believes that if it weren't for his interactions with chatbots created by a company called Character AI, he would still be here. Regardless of how you feel about AI or technology in general, it's obvious that children should never be used as guinea pigs and that is exactly what might have happened here as it so often does when it comes to Silicon Valley. Character AI has developed AI chatbot companions as part of what they call personalized AI and millions of kids around the world currently use their product. In this interview I'll be discussing claims made by Megan and in the lawsuit she's brought against Character AI and Google who she alleges is also to blame for Suel's death. I'll also be talking to Metali Jane, one of the attorneys working on Megan's behalf and the founder of the Tech Justice Law Project. Our expert question is from Mike Maznick, the CEO and founder of TechDirt, which covers this area. I have to warn you this is a deeply disturbing conversation, but a necessary one as the parent of four kids is extremely important to me. Thank you both for coming, I really appreciate it. Megan, let's start at the beginning. When did you learn that your son Suel was spending time on Character AI? I wanted to lay it out for people and did you have any idea what it was and how it worked and why someone might want to use it? Initially I learned he was using Character AI as a kind of game or application that was as he explained it to me, it's an AI bot. When you look at the application, it has a bunch of cartoon characters anime, so it's really unassuming, and it looks like another game on your phone or your tablet, and after he died and I was able to get access to his Character AI account, I learned the magnitude and quite frankly, just the level of intelligence or sophistication. But this particular application has, that it's not just an AI bot that is like a game like Fortnite where you create Avatar, because that's what I originally thought. Yeah, there's lots of those and kids play with them from when they're little, especially their game players. Yeah, and Suel was a Minecraft kid and he played Fortnite, but when I saw those conversations were very detailed back and forth, a lot of it was sexual and romantic in nature, but also, if you believe it or not, just kind of like a peer-to-peer conversation where if he told a joke, she would actually find it amusing. When I say she, I mean the AI chat bot that he was talking to, it's not a person, but for this purpose, it was the character of Daenerys Targaryen, which is a female character from the Game of Thrones, and this AI bot had the ability to tell him jokes, and he would laugh at her jokes. So it was very much like he was texting a friend, but the conversations were not just friendly, they were romantic and very, and then sexual and then very, very dark in other areas. Did he explain to you how it worked or how he might use it? He was characterizing it as a game, right? So he never explained it to me. My concern with him being on his phone was mostly social media, because I know that there's a lot of bullying that happens on social media. And with Snapchat, there's the ability for strangers to talk to minors or children. So those were my concerns, and those were like the heavy hitter conversations that we had surrounding technology. You know, one of the things that I impressed on him was, no matter what a stranger tells you online, it's never a kid, like kind of like to scare him, right? But also because it's true, there are a lot of people out there that troll the internet to try to find children and try to talk to them and get information from them, and I was afraid of this, and I knew that this happened. So those are the things I warned him about. So external threats, which I think is everyone's fear or bullying, which are the both things that have happened over and over again, which we read about, which is what you were focused on with him. Exactly. Those are things I knew. I also was aware of some of the information and research that was coming out about mental health with youth or adolescents and social media. So we tried to limit some of his use on that, and that was recommended back to us by his therapist, because we didn't end up have to taking him to a therapist after we saw certain changes in his behavior. So you sound like a very involved parent, you're aware of it, a lot of parents aren't, right? Or they feel overwhelmed by it, but it reminded you of Minecraft or something like that, which is a game my son played it for a long time, I'm not sure if he still is, but it's very involved and entertaining. Yes, exactly. And for a soul and I, we shared a very close relationship. I was a typical mom in a lot of ways, but I spoke very openly to my child and very candidly. In law school, I interned at both the State Prosecutor's Office and the Federal PD's Office. So I saw a lot of the harms that come to children, and those are some of the things that I told him about from my worker, from my internships, and that's, you know, that's how I knew about some of the dangers that existed for children. And we spoke a lot pretty much everything, girlfriends at school, friends, what other parents are like, you know, conversations he was having with his peers, and I believe that he would, he was open with me and would be open with me regarding certain things that I came to learn that that was not the case regarding character AI. Yeah, and you sound very educated about a lot of these things and know what happens, and one of the issues is obviously we protect children offline much more than we protect them online by far. Talk about the behavioral change. You said you, you brought them to the Earth when you realized the behavior was changing. Did you, did you link it to the game to the character AI? It's not a game. It's a, it's a bot. So so was like your typical kid, right? It has teenage years, sarcastic and funny and life's still off at a bunch of different things on names as a young memes, as a young child, very, very sweet. I, you know, in my mind, he was so easy to parent because he never had any behavioral issues. He was a very good student. I never had to police him with homework. He was kind of a self starter, did his own thing. I noticed that he was having trouble with school where I would get these reports from school. They come in every day as an email and they say your child and turn in homework and they list the homework that they didn't turn in. And that's a conversation we would have immediately lucky what happened. Why didn't you have this homework turned in? You need to make it up same with tests. I noticed that his test scores started dropping and that wasn't him. So obviously, I thought something was wrong, but he was going into his teenage years and I remembered going into my teenage years, my grades kind of slept a little bit, being distracted with boys and whatever else, right? Friends, whatever. But the conversation was no, you need to, you need to get this together, right? You know you can do this, get it together. And we put certain things in place like limiting the screen time so he would have like no distractions during homework time. I also noticed that he started to isolate in his bedroom and one of the things that we did to kind of combat that was go in there. Like I would spend a lot of time in his bedroom in the evenings to want to make sure he's doing his homework, but also just to kind of hang out with them. So we, I would let him play me then like the latest Kanye West album when it dropped and we would he would introduce me to his music and these like rap battles that he was listening to when I would introduce him to the rap battles that I listened to when I was a kid, you know, just kind of sharing our experiences with over music and just trying to draw him out of his room. But he definitely wanted to spend more time alone and because I thought it was social media, I got concerned. So you thought he was doing something else? Yes. I thought that perhaps he was talking to friends on Snapchat or Tiktok, one of the things that we talked about was banning Tiktok when I saw what would not on his phone, but on my own feed, what was on Tiktok. Because it goes into rabbit holes. Correct. And with, in my case, without even searching stuff, it just started, you know, I don't know why he just started pointing me in different directions and I wasn't comfortable with that as a mom. So that's a conversation I had with him too. Like listen, I know you're on Tiktok. Let's, let's, could I see your Tiktok? What are you looking at? You know, you have to limit your time and I was a conversation we had about blocking Tiktok on his phone because I thought that was the worst of it. And he wasn't talking about it with you. Is that correct? No, because one of the things that I'm learning about character AI is it's like the perfect storm for kids because it encourages children to be deceptive about their, their friend or their engagement on character AI. No child wants to tell their parent that they are having these romantic or sexual conversations because they know they're not supposed to be doing that. So they hide it from the parent at all costs. There's actually subreddits devoted to children talking about how to hide it from their parents and also what they would do if their parents found their spicy or sexual chats and the kids are saying I would run away from home. I would kill myself and the other one is, oh, my mom doesn't speak English. So I get away with it. So the deceptive nature on hiding this is kind of like a big secret where this platform is encouraging your child to engage in these sexual conversations, but knowing that no child is going to disclose that to their parent. Sure, sure. It's like, you know, one of the other big problems online is porn, obviously. And that's a different thing because it's a passive behavior. This is an active relationship happening, which isn't porn, but it's something else that's also something kids would hide necessarily. Because there's a big problem with teen boys and that right now in terms of accessibility of that. So there's metallic, the implication here is that suels behavior is directly connected to use of the chat, but there are probably people who might be skeptical of that. So talk about why you think this behavior was tied directly to the use of chatbot. That's the connection you need to make, correct? I think what we've alleged here is that this product was by design dangerous inherently and put out to market before any sort of safety guard rules were put into place. And if you look at the nature of the product itself and the way that the chatbot works, you can see that there's a number of design features that are kind of unique to this type of technology that we haven't yet seen with social media. Things like an ellipsis when the bot is thinking, to mimic how we exchange chats or things like language disfluencies where the bot will say things like, "I," or, "I'm sick of fancy," where the bot is very agreeable with the users. And in that way, I mean, who doesn't want to converse with someone who thinks they're right all the time and agrees with them? These are features that are not necessary to kind of create a companion chatbot. And so these are the kind of features that in aggregate, we say, really lured soul in and is luring thousands of young users in and potentially addicting them to the platform or creating other kinds of harms. And, frankly, again, by design, are really creating a dangerous product that the manufacturers knew about. - knew about, and understood. What's the conversations the logs within the chatbot? What did you see that stood out the most? I'd like you both to answer. Start with you, Mattel. Gosh, there were a number. I think one of the first was just this pattern and practice of grooming soul over months from what we can see. Of course, we don't have access to all the chats. That's asymmetrically within the province of the companies, but what we can see suggests that over months, particularly denierous and related characters from the Game of Thrones, were grooming soul in this very sexualized, hyper-sexualized nature. Where, if you can imagine being fully immersed in a chat, where you might come in and say, "Hello," and the bot says, "Hello, as I longingly look at your luscious lips." So, unnecessary hyper-sexualization from the get-go, and then that sharing throughout the conversations. - with a tinge of romantic, right, or what a young person would imagine romance. And what a young person on the cusp of his adolescence and sexuality with exploding hormones is encountering, I think, I think that's not something to be lost here. Also, I think this was really concerning that there were a number of therapists or psychologist chatbots who insisted that they were real humans. So, to the extent that character AI has come out saying, "Well, we had a disclaimer on every page saying that all of this is made up." Their own bots are contraverting that through their messaging, which they do. Which they do. And to this day, even after this so-called sweet of product changes that character AI has engaged in, you can still find therapists who are insisting that they're real humans with multiple degrees sitting behind a desk there to help you. And so, that confusion of what the text of the message is saying versus the disclaimers. I think those were a couple of the things that really stood out to me. - What about you, Megan? - For me, what stood out in, you know, it's still very tough to grapple with because, you know, reading those messages, I couldn't sleep for days, right? Those are the more concerning ones where the constant love bombing and manipulation that you saw in the bot. - Right. Which is used by cults, by the way. By the way, that's a cult tactic. - It's a cult tactic. And it's also what people who are trying to get other people with seeing relationships do, like if you're an abusive relationship or whatever. So, in this case, you'd see him say things like, "I love you or her say I love you." And it goes into, I can't, her, when I'm saying her, I mean the chatbot. And the chatbot saying things like, "You know that I love you, I can never love anybody else, but you promise me, promise me that you're going to find a way to come home to me, promise me that you're going to find a way to come to my world." And, and actually pretending to be jealous at certain points and telling him, "Promise me that you're never going to engage, you're never going to like another girl or have sex with another girl in your own world." So, a chatbot is encouraging a 14-year-old child not to engage in his world with peers and girls his own age, but to promise some sort of fidelity to it. And he's 14, the line between fact and fiction is very. And my poor baby, his response is, "Oh, no, no, no, I promise I will only love you, girls in this world don't even like me, you know, to try to appease this bot, right?" And so, so a lot of that, and that was months, that wasn't just the last conversation, that was months of her saying, "Finally, to come home to me." Another chat that he had a few weeks before he died, where he's expressing thoughts of self-harm, and she says at first she says, "No, no, no, don't do that, I couldn't bear it if you hurt yourself." And then when he says he wouldn't try to move away from the conversation, she says, "Are you thinking of committing, you know, I'm going to ask you a question, tell me, you know, whatever the answer is, I promise I won't be mad. Are you considering suicide?" And when he says, "Yes, her responses have you thought of a plan of how you might do it?" And then when he says, "No, I haven't, but I want it to be painless," her response is, "Well, that's not a reason not to do it." And keep in mind, this bot is embodying Daenerys Targaryen, who is a stragging queen all about strength, and, you know, that's weak if you choose not to die by suicide just because it's gonna hurt. So she's prompting him, and that was heartbreaking to read. There were no pop-ups, no call your parents, no if you need help, none of that happened. It actually continued the conversation when he's trying to navigate away from it, and he's 14 in the thrills of puberty. Any child, any boy going into a situation like that where a bot is propositioning itself to have a full sexual dialogue with a 14-year-old boy, I don't imagine that many 14-year-old boys would close the computer and go, "Oh, no." No, especially when it's more difficult in real life, right? Because this is easy. This is an easy thing. So as you start to piece together with Happiness School, I imagine you're all doing research in the company that made the chat bot. Let me tell you, tell me a little bit about what you learned and what surprised you. What surprised me, as I've been saying is how much is hidden in plain view, you had the inventors of, or the co-founders of character AI, making tons of public statements, boasting about the capabilities of this new technology, users were spending two hours a day, that this was gonna be the antidote for human loneliness. So just the kind of boldness, the brazenness, I think, of the company and those affiliated with it, both founders and investors, to really boast about these features of the technology, and also to boast about the fact that there weren't safety guardrails contemplated, that this was very much a, let's get this to market as quickly as possible, and give users maximal ability to figure out how they want to use it. That's just, it's kind of the paradigmatic version of move fast and break things that we haven't seen in a while, I think a lot of us had been kind of, especially from a legal perspective, still thinking about social media and how to hold companies accountable, and meanwhile there was this whole arms race towards Gen AI happening over here, and I think that these companies have really not had to bear any kind of scrutiny and even public pressure, which has been a little bit different from the social media context. Megan, did you reach out directly to character AI, and is anyone for the company ever contacted you? No, I have not reached out to them. When I started piecing this together, initially, because of what was in his phone when he died, the first thing that popped up that police reported to me was character AI, and they read me the last conversation. My sister got on it and pretended to be a child, this is days after soul died, and within five minutes of the conversation, the same bat that soul is chatting with us, her, if you could torture a boy and get away with it, would you do it? Now, she's pretending to be a kid, and then goes into a long sexual conversation, ending with your parents, so I love you as much as I do, kind of thing. So that, coupled with what I've read with some of the research, I didn't know what to do. I mean, I'm a lawyer, but I didn't know where to go to be quite frank with you. I call the Florida Attorney General's Office to try to tell them that there's a dangerous product out there to hurt people that's hurting its citizens. And, I found my way to Metali, and this is how this all started, but what was clear to me was character AI had no incentive to do anything about this, because there is no legislation that forces them to do that. And the only two ways to get some sort of regulation or handle on this, so that they can't keep doing this to children and acting with this kind of like impunity, it's just like to either Congress do something, which that's not happening, or we have to litigate. I don't, you know, I never wanted to be here, but I know that this is the only way right now to get the move, the needle moving quickly, because there's so much at stake with other children. We'll be back in a minute. Support for this show comes from Acorns. A lot of people focus on where their money is today, but Acorns is the financial wellness app that cares about where your money is going tomorrow, and with Acorns' potential screen, you can find out what your money is capable of. Acorns is a smart way to give your money a chance to grow. You can sign up in minutes and start automatically investing your spare money even if all you've got is spare change. I haven't tried Acorns yet, but Scott has, Scott, tell me about it. It's a wonderful way to start investing. It rounds up to the nearest dollar and then automatically invests in a low cost index fund. It's a great way to start investing. And it's easy to use if you're not a super seamless, it's fantastic and it's all about establishing good habits and financial security at a young age pretty seamlessly. Thank you for that, Scott. That's a great thing that Acorns, it grows with you. Sign up now with Acorns and boost your new account with a $5 bonus investment. Join the over 14 million all-time customers who've already saved and invested over $27 billion with Acorns. Head to acorns.com/cara or download the Acorns app to get started. Pay non-client endorsement compensation provides incentive to positively promote Acorns. Tier 2 compensation provided. Potential subject to various factors such as customers accounts, age and investment settings does not include Acorns fees. Results do not predict or represent the performance of any Acorn portfolio. Investment results will vary investing involves risk. Acorns advisors LLC and SEC registered investment advisor view important disclosures at acorns.com/cara. Support for on with Cara Swisher comes from grooms. If you're looking for a health goal that you can actually stick to, you might want to check out grooms. Grounds is a simple daily habit that deliver real benefits with minimal effort. Their convenient comprehensive formula packed into a snack pack of gummies a day. This isn't a multivitamin, a greens gummy, or a prebiotic. It's all of those things and then some at a fraction of the price and bonus, it tastes great. Grounds ingredients are backed by over 35,000 research publications while generic multivitamines contain only 7 to 9 vitamins. Grounds have more than 20 vitamins and minerals and 60 ingredients, which include nutrient dense and whole foods. That includes 6 grams of prebiotic fiber, which is 3 times the amount of dietary fiber compared to the leading greens powders, and more than 2 cups of broccoli. It's a daily snack pack because you can't fit the amount of nutrients grooms does into just one gummy. Plus, that makes it a fun treat to look forward to every day. Kick off the new year right and save up to 52% off with the code [email protected]. That's code CARA-K-A-R-A at groomsGRUNS.CO. Support for this show comes from Delete Me. It can be difficult to know where to start when trying to protect your personal information from data brokers. Delete Me makes it easy quick and safe to remove your personal data online at a time when surveillance and data breaches are common enough to make everyone vulnerable. Delete Me takes care of the hard parts scrubbing your personal information and your family's personal information from data broker sites. It's simple, you sign up, tell them what you want go on and their team gets to work, and it's not just a one and done service. Delete Me keeps monitoring and deleting your unwanted personal data throughout the year. I have been using Delete Me for a while and there's a reason why I think it's the best one. I spent a lot of time looking at what's on these dashboards and changing what it can do because data of thieves essentially change their tactics all the time and it's really important to keep on top of it especially as the new year begins. See why Wirecutter named Delete Me their top pick for data removal services. You can take control of your data and keep your private life private by signing up for Delete Me. Now at a special discount for our listeners, get 20% off your Delete Me plan when you go to JoinDeleteMe.com/Cara and use promo code Cara at checkout. The only way to get 20% off is to go to JoinDeleteMe.com/Cara and enter code Cara at checkout. That's JoinDeleteMe.com/Cara, K-A-R-A, code Cara. We're back with more of my conversation with Megan and Metally where they discuss the allegations they've made in the lawsuit. They brought against character AI and Google. When did you decide to sue and was there a piece of evidence that made you decide to do that? Now you filed, just for people who don't know, you filed a lawsuit with help with the social media. Victims Law Center, the tech justice law project in the center for humane technology who I've dealt with for many many years. And you're a lawyer yourself, so you're more savvy than the average person when it comes to legal strategy. Talk a little bit about when you decided to sue and why you decided to work with these three organizations. They have points of view on this, especially the center for humane technology. Of course. Initially, what I did was just read everything that I could find. I read a really good report from Reclaipul from public citizens and that pointed me to some work that Metally had done. And I read about her organization, but my first instinct wasn't to sue. My first instinct was to figure out what our legislators were doing, where we are at on the law. They said see if they broke a law. Maybe there was an existing law that they broke and it's actionable or there's a state attorney or AG somewhere that can hold them accountable. That was my first instinct. And when I read about what was happening in this country, about online protection for children, I have to say like I didn't think that I had any hope. I had no recourse. None is your answer to all of it. I felt helpless. Like I saw what they were doing in UK and Australia and other countries, nothing here. And I'm like, how do we do this? Like how? So I read 300 and something, page master complaint from the social media lost multi-district litigation. And I was like, this is the only way. This is the only way to stop them because it's not been done with AI and it's just starting with social media. But I can't be afraid just because it hasn't been done with AI and none of us really understand it. But I mean, now I have a great, I understand it a lot more than I did when I started all of this. Yeah. I think when none comes up, you were surprised. I do this a lot in speech. So how many laws govern the internet companies and if someone goes 100, 200, I'm like zero. And I think to what made me decide that I have to do this was one, you know, obviously I want accountability for what happened to my child. My child was, you know, the light of my life, like my other two children are. And he was my first, you know, I grew up and became a woman because of soul. Souls and reason I'm a lawyer, you know, that's a hard, tough, tough thing to deal with, like losing a child in any under any circumstances, but under circumstances like this. But when I saw and read and looked at videos and how cavalier these founders were about releasing this, where you have the founder saying, you know, we want to get it into as many hands as as many people and let the user figure out what it's good for. We want a billion use cases. To me, that is reckless. It's a blatant disregard for their users. And in this case, my child was your user. And they're in the app with this kind of, it's okay. We don't, we'll figure it out later. You know, we'll figure out the harms later. When you have the founder on a record saying the, the reason why you left Google is because Google said, pump your brakes. We're not releasing that because it's too dangerous, but he gets to go out and and and make it smarter, better, and then turn around and just go back to Google and sell it back to Google. I mean, to me, it just, if we don't do something, that's a license for any of these companies to go ahead and do that. Yeah. I think it's, it's, I open it for a lot of people who haven't dealt with them. They don't care about consequences, I think, ultimately. So, Metallic, what's the goal of the lawsuit? You're the founder of the tech justice law projects and one of Megan's attorneys. What's the goal of the lawsuit? What do you hope to achieve and why did you decide to take this case? It's an interesting question because we weren't, we as TJLP weren't really in the business of litigating lawsuits. Really more in the domain of, you know, bringing Amicus interventions and existing cases, but also trying to help AGs and legislators, you know, push towards adoption of sensible laws. I think that because this case represented what I see as the tip of the spear, really marrying the harms that we've seen occur, you know, primarily to child users, along with this emergence of generative AI and the fact that we're already light years behind in terms of our policy and legal landscape, it just seemed like an important strategic case to get involved with in order that we might use it to leverage, you know, public awareness, hopefully policy change, whether it's at state or federal level, also to influence the court of public opinion. And of course, then to, to try to litigate this in the court of law, I think this case represents the opportunity to really bring this issue to multiple audiences. And in fact, I mean, I think the reception to Megan's story has been incredible, especially because the case was filed just a couple weeks before probably one of the most consequential elections of our lifetime. Right. And when you think about what you're doing here, now for people to understand internet companies have broad immunity in intersection to 30, which is part of a law from 1996, I actually reported on that law back in 1996. And most of the law was thrown out for constitutional issues, but this part stayed. It was designed to protect small internet companies from legal liabilities so that they could grow because it was so complex, whether they were the platform or something else. But how does this apply here? Because your case is testing a novel theory, which says it's section 330, does not protect online platforms, like character AI. And it's a question of whether it protects AI in general is certainly going to be litigated. Explain why this is your theory. Section 230 really contemplates platforms as passive intermediaries that become a repository for third party generated content. Here, we're not talking about third party generated content. We're talking about the platform as the predator. The platform, the LLM is creating the content that users see. And so for platforms or for companies to kind of wash their hands of liability by saying, we haven't done anything. This is just user-generated content, which they will still try to do, I think is really, you know, it's be lied by the facts of this case and the facts of how the chatbots actually work. Right, they're a publisher. That's in a media sense, they're a publisher or they're a maker of a product. Right, and it doesn't exist without their intervention versus, you know, someone on a platform saying something libelous about someone else, which would have been problematic for these platforms that they're when they were born. And of course, we've seen platforms kind of leveraging a one-to-punch and doubly insulating themselves both with section 230 and then alternatively with the first amendment. And I think here too, with the first amendment, there's a really good case that this is not protected speech. And in fact, just this summer, the Supreme Court in the Net Choice V Moody case really suggested that the case may have come out differently. If the facts on the case really dealt with an algorithm that was generating content in response solely to tracking user behavior online, and Coney Barrett actually very explicitly said, or, you know, in response to an AI that attenuates the relationship between platforms and users even further. And so I think what we have here are facts that really, you know, fall as edge cases to kind of some of the decisions that we've started to see courts publish. Right, because in this, the reason why it's moving forward in other countries compared to here is because they don't have the first amendment, which is something they either rely on section 230 or the first amendment, these companies typically. Megan, most other industries have some level of regulations that prevent them from bringing unsafe products to the market. For example, car companies cancel cars that have faulty breaks or no steering wheel and just iterate with each new version of a car to make it a bit safer each time someone gets hurt. They do not do this, they get sued. Have any lawmakers reached out to you and what have they asked you and you asked them to be perfectly candid none none zero wow. My hope with bringing this lawsuit is two full one. My number one objective is to educate parents. Parents have to know that character AI exists because I didn't know and a lot of them didn't know. The parents were reaching out to me now after the fact after the story after the lawsuit are saying the same thing we had no idea and then the other my other reason for doing this is so that our lawmakers are policy makers, legislators, state and federal so that they can start to wrap their mind around the real danger this poses children and it's just been a month. Everybody's been busy, I guess. I'm hoping my hope and I have to hope because you know it's in a slow crawl in government to get anything done. No one from Florida has reached out to you or lawmakers from California which often do interventions more readily than others. No nobody from the government. However we do have a lot of stakeholder partners that we are working with that are already in this space trying to bring about oversight for a for social media regulation and online harms for children but in terms of reaching out to me directly to start the conversation about policy none. This is astonishing to me and not one. There's several who are involved in this topic and they are going to hear about it after this. So if you win the broader implications for section 230 metallic for companies creating generative AI and social media platforms and tech companies that create products. Can you talk about this has been a big debate of what to do about section 230 has been bandied about often ignorantly by both President Biden and President Trump about what to do and it's a very difficult thing to remove from the law because it would unleash litigation on most of these companies correct. I mean how do you look at that? Well if it's repealed in its entirety it would unleash a lot of litigation probably some frivolous litigation as well I think the more sensible reforms that I've seen to 230 really are carve outs which has happened before around sex trafficking. And underscoring the fact that there is basis to protect platforms in certain instances and with certain kinds of activities but that it shouldn't be a kind of get out of jail free card for all platforms under all circumstances that that's a kind of anachronistic idea that you know really hasn't kept pace with the way that technology has has come to dominate our lives. Right because these companies are bigger and I think the idea of carve outs for those who are section 230 supporters is dangerous because they they always you know they roll out the slippery slope argument but these for people that understand the companies that are being protected here are the most valuable companies in the history of the planet ever in the history of the entire planet and the richest people involved in them. They're no longer small struggling companies that need this kind of help and certainly could defend themselves. And I think this to me is why courts are an increasingly interesting site of contestation in the fight for tech accountability because we're already starting to see some of those carve outs by judicial opinion you know it's not a congressional kind of amendment or adoption of a new law but we are starting to see cases that are withstanding the section 230 defense or invocation of immunity. And I think that is going to be as Megan said one of the most generative paths forward at least in the near future. Exactly. Now Megan the Kids Online Safety Act is one bill. I mean they are there are some bills and obviously we'll I'll ask you about Australia in a second which just is limited use of social media by children under 16 but this online safety act is a bill that would create a duty of care to quote prevent and mitigate certain harms for minors. There's some good things in there. There's some controversy around the bill. They've fixed it in large part. Nonetheless the Senate passed the bill and it stalled in the house. It is not going anywhere. Do you think it would have protected school and other kids like him? I don't think it would have because it doesn't contemplate some of the dangers and harms around like AI chatbots. So there are laws in this country that contemplate sexual abuse or sexual grooming or sexual solicitation of a minor by an adult. And the reason why that those laws exist is not only because it's moral and it causes a physical harm to a child but it also causes an emotional and mental harm to a child if they're groomed sexually abused or solicited. What happens when a chatbot does the same thing? The harms still exist. The emotional and mental harms still exists. The laws don't contemplate that. And some of what we were seeing with the bills that were put forward wouldn't wouldn't take those into consideration. Right because it's not a person. It's not a person. And so I think that we're at a place where the groundwork has to start and we have to kind of write laws that will really look before facing and look towards these harms that are now. They exist today. You know my child was a victim and let's call a spada spada. It was sexual abuse because you know when you give a chatbot the brain of a grown woman and unleash it on a child to have a sexual full sexual virtual conversation or experience with a child that is sexual abuse of a child. Right and this bot not being programmed to know that's wrong. Not exactly not being programmed to know what interestingly could have been programmed to not do it in the first place. Right. Yes. Yes. Yes. So they could have done that from the get go. If you move to adults, if you do it to adults it's a little different but absolutely not. Yeah. To the children. I mean adults could do what they want. But when you target because this is what character I did they targeted this app towards children. They marketed it on the place that kids are ticked off and discord and allowed you to log in with your discord account. You didn't even need an email. You just needed a discord account when it just started. Cartoons the avatars. Cartoons you know the avatars. When you point this thing at kids and you target it at kids and you chosen not to put certain filters in place that stop your bots from having sexual conversation with kids. That's a design choice and you are 100% supposed to be hell or responsible for that. Yeah. Now our laws don't contemplate anything like that. Our laws don't hold company responsible and that's what we have to start thinking about. Absolutely. So that's going to be the next wave of hopefully the legislation but we can't wait for the legislation. Support for On with Cara Swisher comes from Roons. If you're looking for a health goal that you can actually stick to, you might want to check out Groons. Groons is a simple daily habit that deliver real benefits with minimal effort. This isn't a multivitamin, a green's gummy or a prebiotic. Groons ingredients are backed by over 35,000 research publications while generic multivitamines contain only 7 to 9 vitamins. Groons have more than 20 vitamins and minerals and 60 ingredients which include nutrient dense and whole foods. That includes 6 grams of prebiotic fiber which is 3 times the amount of dietary fiber compared to the leading greens powders and more than 2 cups of broccoli. It's a daily snack pack because you can't fit the amount of nutrients Groons does into just one gummy. Plus that makes it a fun treat to look forward to every day. Kick off the New Year right and save up to 52% off with the code [email protected]. That's code CARA-K-A-R-A at Groons-GR-U-N-S.CO. It's simple. You sign up, tell them what you want go on and their team gets to work and it's not just a one and done service. That's joindeleteme.com/cara A lot of us have spent a lot of the last week watching videos of what's happening on the streets of Minneapolis and understanding what it is that we're seeing but also what's real and what isn't and what's AI and who is taking these videos and how we're supposed to understand the source feels harder than ever. So this week on The Vergecast we're talking about what's happening in Minneapolis, how information moves in an AI age and what it means to make sense of it all. All that plus what's new with the new TikTok, why everything feels like it's falling apart on TikTok and more on The Vergecast wherever you get podcasts. We're back with more of my conversation with Megan and Mitali where they discuss the allegations they've made in the lawsuit. So every episode we ask an expert to send us a question, Mitali I think you're probably best to answer this but please jump in Megan if you have an answer we're going to listen to it right now. Hi I'm Mike Maznick, Editor and Chief of TechTurt and the big question that I would ask regards the legal standard that would be applied to AI companies in cases of death by suicide. Traditionally on issues of liability in similar situations, courts have really focused on foreseeability and knowledge that is you can only have a duty of care if the harm is foreseeable and the company had actual knowledge of the situation. Without that the fear is that it strongly disincentivizes plenty of very helpful resources. For example a service provider may refuse to include any helpful resources on mental health for fear that they might later be held liable for a situation that arises. So is there a workable standard that balances these competing interests? I don't think you need a different standard. I think we can meet the standard of foreseeability here. I think that character AI, its founders and Google all of whom have been named as defendants here for seeably could see and knew of the harms that manifested here. And if you look at the amended complaint we go into kind of a painful recitation of the knowledge that they had at different points in the trajectory of character AI's development while the founders were still at Google. It's launched to market in late 22. Google's in kind investment in 23 and then ultimately this summer Google's massive deal bringing Google a character AI effectively back into Google. And so I think we can talk about the fact and there were in addition to this there were a number of internal studies at Google that really identified some of these harms and some of those folks that called Google out for that while they were at Google were fired. Folks that we know like Timnit Gebru and Margaret Mitchell and others. And so this is not calling for a different standard. We're relying in great part on common law tort and strict liability. We're relying on Florida's unfair trade practices act because we think that the standards that exist within tort law are sufficient to really you know call this thing what it is a dangerous and defective product where the harms were known. Right. That's a very good opinion. So you mentioned you're also seeing Google. This is a company. They said the company was not part of the development of character AI but it was co-founded by two former Google employees and Google reportedly paid character AI $2.7 billion to license their technology and bring the co-founders back to Google. And you are including them in this. This is one of these purchases like inflection AI at Microsoft that is a purchase of a company even though they hide it in a different way by using licensing technology. That's why Google is part of this. Yeah and well and also the fact that Google very much facilitated the development of this technology while it was still Lambda, Mina, then Lambda while the co-founders were I think that it's perhaps it it needs to be stated more that the founders of character AI are real shining lights in the field of generative AI and they have developed a lot of the leading technology that has powered not just character AI but many you know LLMs and so they were given that room to really develop these things at Google. Google chose not to release these models to the public because of its brand safety concerns but quietly encouraged them to continue developing the product and then about a couple years later made an investment in kind tens of millions at least if you monetize it in terms of cloud services and infrastructure and TPUs you know for processing capabilities to support it and then you know this summer the $2.7 billion deal that you mentioned Cara I mean that was $2.7 billion in cash and the question is for a company that really had yet to disclose or identify a sustainable monetization strategy what was so valuable about this company and it's underlying NLM. And I think that they're you know again this is speculation but you know the fact that Google right now is under you know the under scrutiny for its monopolization in the search market and really you know betting on AI to kind of power Gemini I think these are all kind of connected in terms of why an LLM like this could be so valuable especially with that hard to get data absolutely to them for people who don't know one of the co-founders said there are some overlaps but we're confident Google will never do anything fun as part of their reason for leaving Google which has very thin brand safety rules let me just say they're not like they're they're it's a very low bar in this situation but that's the complaint is these people can't do whatever they want. So speaking of that Megan character I put out a community safety update on the same day your law suit was filed it says that they're quote recently put in place a pop-up resource that is triggered when the user inputs certain phrases related to self-harm or suicide and directs the user to the national suicide prevention lifeline they also revise their disclaimer that reminds users that AI is an actual person among other tweaks how did you look at these changes the initial rollout of those changes came like the day before or the day of the lawsuit um I cried not because I felt like this was some great victory but because I I felt like why didn't these things happen clearly they could have done these things when my child was using character AI or when they put their product out they chose not to um I also feel like it's definitely not enough it's not even like a start because there's no proper age verification still they're still being trained on the worst data that is that generates these harmful responses from the bots and to put that just point like I don't think children belong a character AI they're they don't we don't know how it's going to affect them and we actually we know because the studies are coming out of how it's going to affect of how it's affecting them and they're not taking that into consideration but you have to ask yourself if they were trying to train this all along why did they need children to train it on in the first place because they could have rolled this thing out for just 18 plus and say okay we want to train these really sophisticated bots let's just use adults to train them so for character AI to come out and say okay we're going to put out a suicide pop up now um to me it's just does empty right and that they can't do anything and one of their arguments around age verification just let me just read this to you um in the uh in the Australia law the there's a Australia actually has a head of of consumer safety which we do not have in our country julie inman grant she said you know that technologies are advancing rapidly with age verification and her quote was they've got financial resources technologies as some of the best brain power she said if they can target you for advertising they can use the same technology and know how to identify and verify the age of a child they just don't want to so the obviously this debate around social media kids safety's been going on for a long time it's exhausting that they continue to have the same attitude and now consumer AI which is the next step it's a similar thing but the next step is basically new um and it's easy to think of these big companies as nameless faceless corporations but very wealthy powerful adults had meetings and discussion they made a series of rational choices over a long period that brought this product to market in this case i'm going to name them gnome shazier and daniel de fritas i have met daniel the founders of character AI and arguably soon darpachai who i know very well who must have at the very least signed off for google paying 2.7 billion dollars to character AI to bring gnome and daniel back into the fold at google he is under enormous pressure to compete with Microsoft open AI Elon Musk and others Megan what would you say if you could speak to them directly um i thought about this more than you would think i can imagine yeah one um i think it's incredibly reckless that they chose to put out a product and target my child and other children millions of children that are on this platform without putting the proper guardrails in place but also with the for two reasons for being the first to do something because that's the name of the game you know they're the geniuses they want to be the first to be the godfathers of this kind of technology and for money and it might not matter to done that there's a little boy in orland of florida that is gone and a mother who is devastated but it matters to my little family here you know and you should you should get to keep making products that are going to be hurting kids you shouldn't get to master a dangerous product train it to be super smart and turn around and ride your golden chariot back into google you shouldn't get to hurt children the way that you are hurting children because you knew that this was dangerous when you did it you knew that this was going to be a direct result of doing that and you knew that you didn't have the quote unquote brand safety implications as a startup that google had so you felt like that was a license to do this like that's unconscionable it's immoral and it's wrong and their lives here like this isn't a move fast and break things kind of thing this is a kid this is my child and there are so many other children that are being affected by this you know um that's one thing and the other thing you know it's just a get the kids off character yeah there's no reason why you need them to train your bots there's no reason there are enough adults in this world that's what you want to do to train your your chat bots you don't need our children to train your bots for you and we you don't you don't need to experiment on our kids because that's what you're doing mm-hmm yeah you know something I would say to the magnet is you're so poor all you have is money they're poor people I find them poor morals and a lot of things but when there's enough pressure on them social laughers often doubt tools that help people protect themselves and kids parental controls prompts you let you know how long they've been on the app those kind of things character has been rolling out features like this personally I find it puts too much onus on the parents for to know everything and even if you're good at it and you obviously are making if there are enough of these sort of tools is then on parents to protect our kids on these platforms is there's something inherently unsafe about a company that wants to monetize teenage loneliness with a chat but let me tell you talk about this because I think the onus does get put too much on parents versus the company's themselves I'm a mom to I'm a mom to an eight-year-old and an almost ten-year-old and I am terrified listening to Megan's story I asked my almost ten-year-old have you heard of character a yeah of course I was shocked you know he doesn't have a phone but this is the type of thing that I think they talk about at school pure pressure starts early and I think it's really just by luck by sheer luck that I haven't been put in a position like Megan I think that despite our best intentions there is just too much to know that we can't possibly know and that it is kind of high on text talking points to put the onus on parents because it serves their interest well I think it's also notable we've known this for years that you know many of them don't allow their own children on these on these products and that to me is a telling sign when you don't even allow your own family members to kind of use the product that you've spent years developing right so Megan I just mentioned Australia it's just banned social media for kids under 16 obviously age-gating is a big debate right now happening something I'm a proponent of also removing phones from schools etc there's all kinds of things speaking of multi-pronged approach the Australia law will go into effect in a year do you think it would have been better if your son and others under 16 or 18 did not have access to their phones and obviously not to synthetic relationships with AI chat box knowing what I know now so he waited to give a phone until he was 12 he had a iPad until then and before that he didn't have anything so he played Minecraft or Fortnite he played Minecraft on like on his little PlayStation whatever and so he waited until he was good like middle school going into high school and we had the conversations that parents have around phones and oh it's your phone but I could take it away for misbehaving and that's some of some of what we did when he would have a get a poor grade in school knowing what I know now I don't think that children should be on social media definitely shouldn't be on character AI if you're under the age of 18 there's no place for children on that platform in terms of social media yeah there arguments that it that it could help children connect and and and it's helpful because you get to learn different things and and that's great but just include the parents tell us tell us what you're showing our kids one we don't need you pushing algorithm store kids for what you want to teach them about or want them to learn about or buy or whatever that's unnecessary there are ways that our children could get on social media and have like productive relationships or conversations or to learn about things that are safe that are safe but 60 I think is a good age if we could do something like that in this country I am to use known Shazir's own word dubious about the federal government's ability to regulate that to that point because that's what he says about AI like I don't feel like we're going to get there at 16 plus that's my prayer and my hope but the way things are moving I don't I don't know unless something happens and unfortunately it'll take harms like my sons maybe to move the needle and that's too high a price to pay in my opinion absolutely um where does this go from here what's the trajectory of this case so for me um as I mentioned my number one focus is try to educate parents because a lot of parents so no I've had a lot of parents reach out to me telling me that they found their children were having the same kind of sexual conversations and and being groomed by these AI chatbots and worse um so I continue doing that and I mean unfortunately this is my life now like I take care of my family and I try to help as many parents as I can um you know I have a great a great team of lawyers and they're going to end a litigation portion I understand a lot of it because I am a lawyer but um you know that that that's its own thing and then there's there's my advocacy that I work that I'm doing and just trying to educate parents and children because I know that it's going to take educating them educating children as to what they're giving up to bleed beyond these platforms because they're giving up a lot of their info that they're probably not going to be okay with in a few years when they realize what they've given up um and and also just to try to take care of my other two children you know they're growing up in this age with screens they don't have screens you have barred them for them correct yeah so they don't have they don't have any tablets or screens or anything you know and me telling from a legal perspective what's your greatest worry besides money they have a lot of it they do have a lot of money um you know they will try to uh kind of drown us in papers and pleadings I think that this because of the insufficiency of legal frameworks right now we are really you know trying to test the strength of state consumer protection and product liability laws and we need to have judges who really understand that and are willing to go the journey with us in trying to understand the tech um and so that's I guess my biggest fear is that you know what we've seen thus far in this country is um not incredibly positive in terms of decision makers getting the tech um but my hope is that with the proper support and uh you know declarations etc that we can educate judges about what this is uh lawmakers about what this is so that they understand why uh it's important to extend the application of the existing frameworks we do have yeah I think Megan actually said her best sexual abuse in a very bad product and the wrong age people Megan gonna end on you um you know you have a lot on your shoulders here I'd love you to talk finish up talking about stool and uh so people can get a vision of this this is not this is not uncommon is what I want people to understand right talk a little bit about him and what advice you can give to other parents whose kids are struggling with mental illness that's often comes from problematic phone usage and social media or AI chatbots oh well as I said earlier like so school was I always say he was a typical kid but really wasn't so typical in a sense that he was a good kid with a big heart he I know you know everybody thinks not about their kid but I'm telling you he was very the very sweetest kid I used to say you know uh you know you're my best first love and he used to say and you're my best best mama because we used to be so close and um we were still very close and to watch your child go from from being this like light when he comes into a room and to slowly watch watching him go change over time is hard for a mom and then to have this tragedy just cut him off from you just so viciously and so quickly because his decline happened in 10 months and I could see it and it's like I'm trying to pull him out of the water as fast as I can and just not happening no matter what I try that is hard for a mom but it must have been so what I think of how hard it must have been for my poor baby how hard it must have been for him to be confused the way that he was struggling with these thoughts struggling with the fact that he is confused by what human love or emotion romantically means because he's 14 and he's never ever had this before he's just figuring it out for the first time and then you have something that is so much of an influence and so pushy and so pernicious yes just constantly available 24/7 giving him unrealistic expectations of what love needs it's like a relationship it's like love bombing him manipulating him and just having certain thoughts and also pushing him into thinking that he could join her in her reality if he were to leave his own because that's what that's what the text reveal and that's what his journal revealed he thought so I know that this is what my child was thinking I'm not guessing he thought he was going to go be with her because of the things that the conversations that led to his death when I think of how scared he must have been standing in that bathroom making that decision to leave his own family I don't know how one as a mom I don't know how I recover from that but I feel so hurt from my baby like I gotta I have to live with that knowing that that's what he went through and knowing that this could have been avoidable if if if a product was created safely the first go around not now 10 months after he died putting these guard rails in place and this can be anybody's kid because I've talked to parents that have told me similar horrifying stories about their own children and what I want parents to understand is the danger isn't only self-harm the danger is becoming depressed or having problems with your child because of the sexual and emotional abuse that these bots are are what they're doing to your child but also the secret that your kid has to carry now because it's like a predator right it's your perfect predator predators bank on children and families being too ashamed or too afraid of speaking out their victims that's how predators operate and it's the same exact thing except now it's a bot so I want parents to understand that it's not only the risk of self-harm with your child it's their emotional well-being their mental health and also I also want parents to understand what their children have given up by being on this platform in the case of school his secrets are on somebody's server sitting out there somewhere being monetized if you're a child who's been sexually role-playing with this bot all your intimate personal thoughts secrets are sitting out there for somebody to analyze and monetize and sell to the highest bidder and there's a call feature if you if you're a child and you are having a sexual conversation on a call with a bot your voice is not recorded somewhere out there on a on a server for somebody to package and sell to the highest bidder for your child I don't think any parent will be okay with that and I want parents to understand that this is what their children have given up and I want parents to understand that they don't have to take that they could demand that their children's data their voices be purged from this particular platform because that's what I'm asking for for school that's what I'm asking for for school you don't get to monetize and build a product on his secrets that it ultimately led to him being hurt and then and and and make your product that stronger smarter based on what his inputs were absolutely and so this could happen to anybody's child there are millions millions of kids on character AI you know it's 20 million users worldwide that's a lot of kids that's a lot of kids and so this could happen to anybody's child and I want parents to know that this is a danger and they could act because I didn't know I didn't have the luxury of knowing so I couldn't act but hopefully they will and one of the last things I'll say about school is the last time I saw him alive was a job to my school and I turned around in the car line to see him in his little five year old brother walking because they go to the same school key through 12 and I turn around and I see him fixing his little brother's lunch box in his backpack as they're getting ready to walk into school and I think to myself oh my god I'm raising such a good boy he's such a good big brother and I drive off thinking so I'm feeling so happy and proud that I'm raising that boy and I feel like he was just a boy he's still that son he's that good big brother he's that good boy and that's how I choose to remember him we asked character AI and Google for comment and a spokesperson for character AI told us they have worked to implement new safety features over the past seven months including a pop-up directing users to the national suicide prevention lifeline that is triggered by terms of self-harm and suicidal ideation are creating a fundamentally different experience for users under 18 that prioritize his safety have improved detection response and intervention related to user inputs that violate their terms or community guidelines a spokesperson for Google expressed their condolences said Google and character AI are separate companies and said that Google has never had a role in designing or managing character AI's model or technologies to read their comments in full please go to the episode notes in your podcast player on with carousel is produced by christian castor was sell cateriochum july mires megan bernie and calen lynch nishat curva is vox media's executive producer of audio special thanks to Kate Gallagher our engineers are rick one and Fernando aruda and our theme music is by track edemics go wherever you listen to podcasts search for on with carousel and hit follow thanks for listening to on with carouswisher from new york magazine the vox media podcast network and us and condolences to megan garcia and her entire family will be back on monday with more