The Tommy Robinson Podcast #4 | Pauline Hanson: 30 Years of Fighting the Establishment

Matkustat turvallisi mielin tänä kesänä. Klarna Premium ja Max jäsenyden tarjoavat kattavan matkaisuajan sekä arvokkaat edut jokaiselle reissuille. Max jäsenena voit perua matkasi mistä tahansa syystä täysin kuolelta, liityny ja saat 30% alennuksen kolmeltä ensimmäiseltä kuukaudelta. Lataa klarnasovellus tai käy osoitteessa Klarna Piste-Kom. Klarna jäsenyden tarjotaan kuukausilisemaksulla edut vaihtelevat jäsenydenukan, voit peruuttaa milloin tahansa klarnasovelluksessa, Merouta mistä tahansa syistä palvelun tarjoaa Klan, - eikä se ole vakutus. Vakuutus- ja turva edut - tarjoaa "X Cover Coverage Injust Europe" PV. Tarjois on voimassa vain uusille preenium ja maxkäyttäjille - ja vanheneen kormastymenes ensimmäinen 7-12/26. Ehtoja sovelleaan. -Is this media fake? Is this media of me? -Ain toinen, että systein puhut ja testausiaa - ja asalus, ja seurin ja perävelun tarjoa. that's gonna protect them individually versus. means something about mitigation in a specific product. Today, we're talking about the people' who create AI - generated sexual content and whether prominent technical proposals - can actually prevent AI systems from generating exploitative content. household/audience/kovs I wonder if you might want to start by telling us a little bit about - the research that you've been doing. I know that most conversations about AI generated imagery begin with a focus on detection. But you've been arguing that that really misses some central facts. I'd love to hear you just start by telling the listeners how you think about this problem. So when it comes to AI generation of non-consensual intimate imagery, sometimes called revenge point, one of the main harms is on the use of someone's likeness without their consent to create sexual content of them. And in particular as a computer scientist, I was looking at different kind of claims that companies were starting to make. For example, like Dolly has a claim about using advanced techniques to prevent - photorealist-ject generations of real people, CID-AI, which is a website for hosting different kind of model objects that got - found to have a lot that make specific people, started making some claims that you could opt into preventing - models from being shared that could generate your likeness. And as a computer scientist that made me wonder, can we actually do that in practice? And what are the flaws that are going to kind of pop up if we try to automate that type of protection? We actually just saw this from Mehta. We saw some interesting decisions about not only opting in, but in fact encouraging. I got a push notification from Mehta AI, telling me I could go create content of people just by adding their Instagram handle, and then it would go and it would generate content using the person's public Instagram account as the base layer. And it was a very interesting product decision, I thought, in light of the potential for abuse and curious what you thought about that call. Yeah, I share your concerns about that choice, particularly given that we've seen it with GROC. There's been a lot of instances of notification or at least sexualization of individuals by using a very similar affordance where you like, at the individuals account, it's just kind of a reply to content that they've shared. And so when I saw the Mehta release, I was kind of immediately thinking about that kind of situation and was concerned that people may not expect that their content can be modified. And I think this goes beyond notification, right, even for various job or reputation concerns, people may have. And actually in particular, we're seeing people being asked to kind of set social media to public, right, for various kind of immigration or border reasons. And so that may put people in a really challenging position. The other thing I would say is, you know, even after, say, XAI has mentioned trying to implement new technical controls, we're still seeing many cases of notification and in a report that will have coming out soon with the Center for Democracy and Technology. We took a look at all of the kind of technical ways that foundation model providers are trying to prevent this generation. And we really don't have reliable means of doing that right now. And so, you know, while I'm sure that Mehta's intent is not to allow for that, there's definitely a risk with a feature like this. One of the things that I have followed in your work over the years is that you write a lot about how sometimes abuse isn't necessarily focused on the substance of the content, but whether or not the person consented to its creation or distribution. Right, and that might mean that there is a new image that the person actively made themselves and chose to upload voluntarily, or a person who is, as you know, not nude, but put into, you know, a potentially dicey image situation. And that did not consent to that and doesn't have the same access to having it addressed because the image generator will in fact generate it. So, can you tell us a little bit about how you see the consent problem and its differences from the content classification problem? I think the idea with consent that we've seen even before use of AI for non-consensual intimate imagery was so large, we had done work with, as you said, victims who had not taken content and maybe had it made of them using secret cameras, other approaches, as well as folks who had taken content of themselves either for recreation or for commercial purposes. And what we saw was for folks who are sharing this kind of content commercially, some people viewed the resharing of their content or now the editing of that content as like a commercial threat, like theft, like basically you've taken something that I make commercially and you're trying to make money off of it yourself. Whereas others saw it as both theft and a kind of sexual harm, privacy harm, victimization in the same way that folks who were not commercial actors would have felt about their content being disseminated or themselves being depicted. And I think that's one of the reasons why the idea of likeness or being identifiable as a real person and whether or not that person got the opportunity to consent is so compelling to me in this space. It's also compelling to me, given kind of how we think about pornography in the past and people being able to provide consent forms and that being kind of a huge distinction or like point of process at least in the US. And so I think for both of those reasons, it resonates with me. So just to be clear, NCII and why it's not called revenge porn is because the idea anymore, that's kind of like it's like a, you know, not a politically correct term among those who kind of study it anymore. And I've always understood that to be kind of true because the idea of pornography is that one does it consensually and inherently like pornography is kind of thought of as consensual. Whereas NCII, non-consensual intimate images is essentially like inherently the opposite of that. And so it does and also because revenge is not, some often we find it in revenge situations but revenge isn't the only way that like non-consensual images show, intimate images show up. So I just, is that correct? Is that kind of the definitions that we're operating under? Yeah, absolutely. Yeah, so I want to kind of pull out a string here, which I think is super interesting from a legal perspective, which is this question of focusing on consent. So from like a legal perspective, I think there is something really interesting about kind of this model that you bring up. One is kind of a theft of property. So to speak, like a theft of something that you can commercialize or something like that, like a content kind of theft. And then there is, you know, this privacy-based regime. So those kind of like end up in different buckets in the law, I would say. But in both, there's like an evidence issue, right? Trying to show or demonstrate that consent has been given either explicitly or implicitly and kind of to track that. One of the ways I know platforms have dealt with this in the non-consensual image context is to just believe the person who says, "This is my image." And I didn't consent to this. And so please take it down. Is that as feasible? Going forward, did you do any kind of research on that? Do you see just kind of a belief in terms of just how platforms should react to this? Like is that actually going to be an actively useful framework going forward? Yeah, that's a great question. So I think there are a few issues here. So one is once the content is created, what do we do about it? Right? And so if that content is shared on a platform that has a reporting stream and people are able to report, then certainly kind of attestation is one approach. I think the one concern we sometimes have with attestation is that there have been cases of like widespread sort of activist-based reporting of, say, sex workers content for take down. And so that becomes like a question of how you balance that. But typically there are other signals when that's happening where you may be able to kind of balance things out. So I think for content that is, up, that's like a reasonable approach. I think the problem though becomes that this content often spreads to platforms that are specifically for hosting non-consensual imagery or are just not interested in doing takedowns, for example, thinking places like Forchan or a bit more hidden. The other issue that can come up is people discovering this content has been made in the first place and that's a big issue for the AI generation of this kind of imagery is how are you going to go and detect that you're depicted. And that kind of comes back to this question of like, can computational approaches do detection of you or any real person in generated media in a reliable way? And that's a different question than the one we usually ask, which Renee brought up, which is like, is this media fake? Is different than is this media of me? And that's where I don't think the existing kind of report an attestation piece solves the kind of search problem or prevention of the content being made. So technically speaking then, when we think about the technology for detecting consent, is that even, you know, it sounds even weird to say that or how do you credential an image is in a way that is like machine readable? We can do this, of course, right? This is come up a bit. But we're seeing quite a lot of platforms struggling with labeling regimes, with detection regimes. There's a lot of different things that they're being asked to build simultaneously. How should we think about essentially credentialing content as consented imagery that is not NCII? Great question. So I think about this in these kind of two components. The first is how do I recognize that this image that has been generated or posted is an image of you? And how do I do that in a way that is privacy preserving preferably? And then the second piece is how do I know whether that is an image that you consented to being created? So for the first piece of like, is this an image of you? One of the first things that may come to mind for computer scientists is facial recognition, which are right, the technologies when you go through TSA now they often want you to stand in front of a camera. They're using what's called one-to-n facial recognition where they're searching for you in their database of faces. And essentially what they're doing is they're comparing the image of your face that they just took doing a similarity score check against the other images in the database and presumably your face or name should come up as the most similar one. When it comes to generated images a few things change. The first thing is that the underlying distribution of pixels, so all the little points that makes up the picture, and that distribution is different in a generated image than in a photograph. And there actually has not been pretty much any research on how well facial recognition algorithms will identify individuals in generated photos. And the other thing that comes up is that facial recognition was originally created for these kinds of border type cases where you want to be like perfectly sure this is the person that you think it is. When it comes to something like NCII I think the concept is a bit fuzzier, right, in terms of like, can a reasonable person recognize the individual in the image? And so how well something like facial recognition will map to human perception is an open question, especially because it depends like, is this you think it's you in the photo? Is it someone who knows you well in real life? Things that's you in the photo is it just like your online fans think it's you all of those can add noise to this kind of machine process. So that's one aspect. I think in terms of like attaching consent credentials to an image, certainly there are different approaches like we've seen for AI labeling to do kind of cryptographic watermarking that attaches content to an image. All of that relies on the platform where the content is being posted, keeping that metadata and respecting it. And so that is like an open question that one might have after the generation phase. So I think this is really fascinating that you did all of this kind of research between the various different types of ways that we move from something to TSA where you say one to end like, I think that like, you know, most people are so creeped out by things like TSA and it's incredible quick speed and it's incredible accuracy mostly because they think that like the TSA is running your face pass a database of all known human beings and the planet and it's not it's like the flight list like the manifest, right? Like it's not in any way that like kind of which I think is actually kind of a super important point to make. And so I think that there's this really fascinating delta between our ability scientifically and through qualitative, I don't know, I guess you could say proof for mathematical proof whether something is AI generated or whether it's real, but actually that doesn't end up mattering for the vast majority of things. So like authenticity in both one sense is so crucial to certain types of like facial recognition and serpent types of like of AI generation stuff, but it's actually not on point to like kind of this problem with non-concentral intimate images because the damage is really just the idea that it's you, right? That's a reputational damage, it's a normative damage, it's kind of a breaking like, you know, being shamed or kind of, you know, embarrassed by something that you're caught undoing or that is revealed about you. How do you kind of capture that sociological component of this? I mean, you're a computer scientist, so I'm just really curious, you live in the mathematical proof, the authenticity proofs of like these things. So how does one measure, how if you, you know, if you're designing a study, how does one measure that other component? So one of the things that we do here is we often try to look at fields that measure human perception. So take psychology, for example, and there's a well-studied task called the familiar facial recognition tasks that's used to diagnose things like pro-apollegia. So it's been studied quite a lot and basically the task is I'm going to show you an image, I'm going to ask you what is the name of the person in the image, and you're going to reply either with the name or with identifying details like, oh, this is the person who played Wolverine in the most recent movie. And so that's an example of a task that we use to see like can someone recognize and then we control for all sorts of confounds, things like, are you of the same gender and ethnicity as the person who we're asking you about, are you familiar with them in real life or not, etc., etc. So we do that kind of work to capture human perception and compare it with computational metrics. But that doesn't get at the deeper part of what you're talking about, which is things like even if this is a cartoon of me, is that something that would cause harm and how do we think about that from like a recognition standpoint? Because certainly I can recognize a cartoon of Albert Einstein, even if it's not photorealistic. And so in those cases, we do one of two things. One is we create surveys that we call vignette surveys. So we basically create a scenario for someone to imagine themselves in. So for example, we would say imagine that a stranger created a video showing you as a cartoon having sex. Would you find that acceptable or not acceptable? And then we have like a rating scale, right? And serving methodologists like these kinds of scenarios because they kind of approximate people's feelings. They're not precisely real, but they're closer. That's one option. The other option is we interview those who run like support organizations for victim survivors or work with them to speak with victim survivors themselves to understand case studies that are a bit more nuanced like cases where there's cartoons, cases where things are not quite as clear cut as maybe we think about in a kind of like photorealistic notification case. You've also done a lot of work looking at the motivations of the creators. You had a really fascinating paper where you talked about how people who create these images think about their role and what they're doing, particularly in cases where they make them for themselves and don't share them. And you have a lot of really interesting nuanced dynamics to your thinking about the creation process. I wonder if you could share a little bit about that with the audience. Yeah, absolutely. And so we started studying communities of folks who use AI to create sexual content because we were curious the extent to which those communities were creating norms that sort of separated out non-consensual creation. We are also curious how they thought about issues like accidentally creating NCII, for example, if a particular model has been really heavily trained on a particular celebrity individual, it's possible you would write sort of a generic prompt and get back someone who looks a whole lot like a celebrity who you know. And so we started going into these online communities, some of which have hundreds of thousands of members to understand like how are people getting into this? What are they making? Is it abusive or not? How are they doing governments? What are their norms? And that helps us both see the type of content being created. It helps us understand what are the pipelines for creation? Because I think we all talk a lot about like notification websites, right? I've upload an image, I get back one that's undressed. But there's a whole wide world out there of folks using what we call open-weight models, things that have been uploaded on hugging face elsewhere. And they are either modifying them or using them just as they are to create their own content either locally or in the cloud. And I think before we did this project, the sort of prevailing wisdom was like, oh, you need some computer science background, maybe even some machine learning training to do this. But what we actually find in these communities is they're actually serving as upskilling centers. So they're kind of training each other if a model is too big to run on someone's computer, then they're gonna compress it down so that they can run it. And in fact, one of our participants said, I'm actually not interested in making sexual content. I have this other content I wanna make, but the most active technical support I could find was in these communities for making sexual content. And of course, not all of that content is abusive, but of the 28 people we did interview across a couple of different communities, three of them talked to us about making non-consensual intimate imagery. And several other people talked about getting requests for making it that they turned down. And the moderators who we discussed with to greater or lesser degrees, were trying to moderate this type of behavior, but they were actually struggling a bit because people come to these communities for kind of an unsensory place for sexual expression, something they didn't feel like was easy to find places to talk about sexual content creation or preferences. And so when there are kind of rules put in place, sometimes the moderators get pushed back. That's like, excuse me, you're not supposed to kind of yuck my yum, don't tell me what to do. And so they actually said that they found two things useful to justify why they had rules. The first was the terms of service for the platform that they were hosting the community on. So the terms of service have some mention of non-consensual intimate imagery and because people so want to be in the community, if they bring up, hey, the community could go down, if you share this or if you take these requests, that really helped. And the second was the new take it down act, the idea that there might be legal consequences for sharing really like motivated the moderators and was also a good justification to the community. That's interesting that you bring up these notions of perceptions of censorship versus permissiveness within the community. I know there's also an emerging legal debate over whether generative AI outputs receive first amendment protection, either through the rights of users to see content or the rights of developers to build models and the sort of expressiveness of codes long running debate. But this is obviously unsettled law. We're starting to see it play out in a few different cases. There was the Garcia versus character technologies one. But right now, I think there's an interesting question. I am not a constitutional lawyer. Kate can jump in and pick this up. But one thing that I think about a lot is, does that conversation leave out the person who is being created? When we talk about the prompters' expressive interest, the model companies editorial or design interest, where does that leave the person whose likeness is being rendered without consent? Yeah, I guess from my side, this is something I think about a lot because in computer science, the main-- one of the main privacy threats that people address with computer science research is something called membership inference. And so the idea of this is like, there are some images used in the training data. Can I attack the model and recreate the exact image from the training data, or can I extract it? And this is an important problem for various reasons. But I often talk to colleagues about my frustration that is not the whole problem because my ability to reconstruct the exact source photo is maybe not my concern. When it's a harm around my likeness, the issue is, can you reconstruct an identifiable version of me? And one of the challenges we've had is with membership inference, you can define this mathematically because that training data object existed at some point so we can write some math about what we're trying to do. With the likeness, we need some sort of mathematical definition of what is a likeness. And that's so much more as we talked about a human perception kind of thing. Can we achieve that? I'm not terribly convinced we can. And yet, it's an important problem to work on to evaluate for, et cetera. So that's where I see this coming up. OK, so I love this question. I think that this is where a list is amazing research and a lot of the work that you do, Renee, and some of the stuff that we've talked about offline and off the podcast about deepfakes and the implications and what the legal answers to Gen AI is going to be. I think that this is where this is going to maybe be the four words decided because I think that these are such obvious places where vulnerable individuals can really be exploited. But also, it is the place where people who have the most economic stakes can be exploited. And the combination of those things motivates people. Well, yeah, this was, I think, met a pulled there weird tag in the Instagram person. You want to make a picture of thing because one of the creator guilds protested as I recall, right? Complained about it. Yeah, so I don't want to kind of go all the way back to kind of, you know, to bore everyone in this pod who did not turn in for Kate Klonk's history of IP law and internet history. But this is essentially what you saw in kind of, in my version of events, this is what you saw with the early internet that we're kind of speed running through as an legal issue now for Gen AI, which is that you have property-based rights being the most enforceable and the most economically viable rights that you can enforce in courts. And so you see them at the four and you see huge interest groups going to bat and taking down the technologies or trying to take out the technologies that do this. And so obviously I'm thinking of because this was like happening when I was growing up in a teenager. And so I was like, on both the side of like this, this group of lawsuits ending, right? As I got to law school and being a person who like downloaded illegally tons of stuff from Napster, obviously like the Napster because I appeared appear sharing kind of issues where you had huge interest groups like the RIA and the Motion Pictures Association of America kind of decided to bring suit because their actual content was getting ripped off. Now that is a solely copyright kind of, like squarely an IP type of right, right? And that was the question there for a lot of people was like, do we change how we distribute IP rights and what we think of as IP and how we do this in the age of the internet? That's a little different when you have a generative AI, which is both mathematically from like the authenticity point that like Elissa was talking about before, mathematically new material that happens to have a likeness. Like it is pixel by pixel it, completely different image, then a photo of Renee. You know, if you had a generated image of Renee that looked exactly like Renee looks right now in her Georgetown office, like, you know, and then you had this image of Renee like that we asked the machine to produce whatever model we wanted to use. And if it was like almost one to one, they would still be different images from a, from it, like I guess like a granular mathematical perspective. But, and this is kind of what I think is so interesting I was getting at before phenomenologically, the picture plays the same role so, it's sociologically, it pays the same role. It doesn't actually matter. And so the effects of the picture, the harm that it creates, right? Doesn't end up totally mattering. And this is I think the thrust of essentially what the debate is going to be. Like do we end up going down? And I mean when we say like the law, does the law end up going down this really particularized root of like mathematical authenticity? Or is it going to kind of done this humanized root of like how it categorizes the harm and like whether or not the generation of that harm exists through AI or something else. Like we are going to recognize this harm. And that has different first amendment balances. So if you tend to recognize generative AI as like where you were a proponent of recognizing that a gen AIs having its own first amendment rights because it is generating something completely new and technically authentically different, you're just going to be able to block a lot of, not all of, but potentially block a lot of the regulation and potentially litigation, regulation and litigation that is going to kind of be filed for like against against some of these models and companies that are running the models for the outputs that they produce that are exactly the type of outputs that a list of work is kind of talking about. If you decide to take a more kind of harm based approach, a more dignitary privacy based approach, where it doesn't really matter. Like where the image comes from, just that it exists and it can be traced to the generation from like either a model or a human. Then you have like a whole different kettle of fish and kind of a whole different set of legal solutions that you're going to put in place there. So I don't want to kind of take this over with a legal discussion, but I do think that this ends up being kind of the payout of a lot of Alyssa's work. If you go to like put it to policy, and I think it's super interesting just because it reveals the limits of like what authenticity gets us and it reveals the power of things like likeness, right? likeness being something that no one has been able to perfectly quantify or understand in either cognitive neuroscience or IP or whatever for years. I mean like literally, you know, how trademark likeness is decided is through survey. It's through the exact type of research and kind of like qualitative and quantitative. Nikon on hienotat en ne Source Scientistophylle. Geonit mitro ollaan k Wie barempia, koska ne sisälti sellaisia tekstytemiseen äänäaresille tasolla. Ja se metillä ne ei ole haskella ja ehkägevalle usea van超älichisesti adjustingista, joon tämänehdypsä laski terveimmäisellisellinen tämän täysin. Maks jäsenyden tarjoavat katta van matkaisuajan sekä arvokkaat edut jokaiselle reissuille. Maks jäsenenä voit perua matkaisin mistä tahansa syystä täysin kuolelta, liityny ja saat 30 % alennuksen kolmeeltä ensimmäiseltä kuukauden. Lataa kloarnasovellus tai käy osoitteessa Kloarna Pistekomma. Kloarna jäsenyden tarjotaan kuukausilisemaksulla edut vaihtelevat jäsenyden mukaan, voit peruuttaa milloin tahansa kloarnasovelluksessa, veruuta mistä tahansa syystä palvelun tarjoaa Kloarna, ei käs ole vakuutus, vakuutus ja turva edut tarjoa X-cover Cover Genius Europe BV. Tarjoos on voimassa vain uusille preiämijumia, maks käyttäjille ja vanhenee kormastyminen ensimmäinen seitsemättä kaksi tuottaa 206. Ehtoja sovelletaan. -I think this question of what should the regulated act be? I'm kind of curious to hear your thoughts on this elissa. Is it the creation? Is it the distribution? Is it the implied threat maybe of putting a person into a very particular type of sexual scenario? Is it profiting from the ecosystem? Where do you see the lines around policy on this front? Or maybe regulatory laws, better term than policy for this one? -So in this space, I'm often kind of torn between my background in computer security and focus on kind of anti-sensorship, protection of people's communications and how we can best address harm. So I think certainly regulation that addresses sort of the pipeline that funds this would be most effective if done in a way that is careful about addressing non-consensual intimate imagery specifically and that sexual content in general. And I think that's always kind of attention at least I've seen from doing work on the security side in this space is just when companies say react to a regulation by going really broad in their prohibitions. Sometimes we find that just kind of pushes everything to the corners of the internet where we have the least ability to kind of take control. So I always think about it in terms of a balance there. And because this is dual use technology, particularly that used to kind of create things. I think one of the biggest areas for computer scientists to do research is how do I identify when something sort of purpose built for creation of non-consensual intimate imagery whether that's a website with very clear kind of keywords and functionality or a model that's very clearly kind of trained or fine tuned on particular individuals. And there are folks doing research in both of those directions. I think addressing those tools and both who owns and profits from them as well as who uses them is important. The other thing that I think about maybe more so from corporate responsibility, I think, or defer to you all on regulation is kind of we have seen the use of mainstream platforms like Google, Facebook, etc. as single sign-on providers. So you know, you click and then you can easily sign into the platform for various notification services. And when that happens, it gives this kind of like acceptability stamp to the platform. Even if it's not the intent of the single sign-on providers, same when we see those platforms like running ads or having applications and their app stores that can do this or are dedicated that way. And so I think there's a lot of kind of corporate governance points in terms of just like keeping an eye on where your brand is popping up and what you're promoting. And we've seen lip service for doing that. We've also seen some failures after the lip service. And I think a lot of those failures are because this is a dual use kind of thing. You have to really like test the actual application. Like you need a we usually like generate an image with a face covered. We try to do some double checks, but it's not identifiable. And we use that to test what the system can do. And I think that often that's kind of not what's happening, just like the form is being read by whomever is reading the form or whatever is reading the form. And that's where we've seen a lot of failure. So I think whether that's a regulation thing that gets addressed or just sort of a corporate responsibility thing, that would be important. Yeah, so I want to actually kind of cut in because I know Renee might have thought some the same question. And she's also done a ton of work. You know, Renee, you've also done like so much work on kind of how all these different systems are running, kind of capturing the idea of how they each respond to different types of prompts. How GROC is responding to prompts, how meta is responding to these same types of prompts. I'm curious how your research you would describe it as different or the same as some of the stuff that Alyssa's done. And also if it has pointed you towards different policy kind of suggestions because I do think that Alyssa's point, which is like it kind of depends on like what it is exactly that the model or the actor or the whatever is doing. Like it is one thing if it is like a jerk ex boyfriend using GROC to like to do this thing. And it's another if you're talking about the model itself. And then there's another thing if you're talking about kind of like literally some type of like six guys in a trench code in Russia deciding to kind of run a shop that does this for individuals off of GROC, right? Or something like to that extent. And so kind of I'm just like wondering like both of you have had kind of these qualitative conversations and interviews with people and done a lot of the qualitative experimental work yourself. So anyways, I just kind of will start with like a brine I'd love to hear that. Yeah, it's a real big challenge. I used to think that corporate responsibility was was something where you would just never see this from large foundation models, including GROC because they would just prevent it. And then we had that debacle with donut glaze right and on a on Elon's on Elon's platform where for I think probably most of the country was aware of what happened with that. But just to restate it in case they're not it was basically that people began to realize that spicy GROC would generate all sorts of images of people in very compromising situations. And with X it's interesting because you could just ask GROC to generate the image right under the person. So this was an extraordinary vector for her ass man and many of those images got millions and millions of views. Ashley St. Clair and influencer who also had a child with the Elon Musk is currently suing him. I don't think we have really seen that case move very much. I haven't seen any updates on it in a while. But you know she sued because of what happened. I think he's fighting about venue at the moment. This is where this beginning to realize that corporate responsibility isn't going to hold. You know, you just see these these rather extraordinary things where like morals go out the window and I just want to say for for listeners that the donut glaze thing is you would essentially ask. It was a way of routing around things that were. So I just want to say like because I do not think the most people are going to be for that are not very online are going to be familiar with this kind of absolutely horrid specifically for the generation of child sexual explicit material. That like how bad this was you could instead of asking for a pornographic a specific type of like thing to be all over somebody. You could ask for a donacle is to be all over someone which was the approximate look and style of like what that thing would be to create kind of a very explicit image that could be taken into various contacts and just like you know having a child covered in this or anything else. It was very it was a very kind of huge controversy in the world of people who follow this type of thing. Unfortunately, I don't think it actually kind of made the mainstream news at all. I mean, both because it's incredibly explicit and because I don't know people tend to not care about these things of the margins and like Elon Musk has such a high threshold for the thing like of all of the absurd things that he does it's like half of them don't even rank anymore. This one was interesting because it did lead to immediate regulatory action in Europe right so you had a lot of immediate calls for you know various types of the regulators in Europe requested specific data from X that sort of pushed it into the news a little bit more than I think it otherwise would have been in part because you began to see coverage of Elon saying this is censorship you know being asked to provide data on exploitation on his platform was censorship. Requests that you know people pointing out that maybe this is actually a terrible thing you know we're recast a censorship actually wrote an article about this for law for anybody who wants to track it down will put it in the show notes because you can kind of see the specifics of this particular story. But I think the challenge on the regulatory front is not wanting to cross bounds into free expression as we've talked about you know where where those lines. You mentioned the child exploitation content that's much more where I have done my work not in adults and C.I. so much as where there is no consent there is no consenting person in that in that exchange right the child is either being exploited or an image. That is potentially derived from models that have a perception or inadvertently retrained on that stuff that's where that's where you start to see that happen so there are some very granular. -Laws that come into play. when the child content comes the focus of the thing that are a little bit different than the potential that it might be a voluntary content creation and uploading. So I think that's where for me a little bit of my work has been focused much more on the former. I think I want to talk about the technological barriers to creation though because this was the other thing. If you have corporate responsibility and the corporate responsibility relates to them putting guardrails on their model there is this dynamic where people will go and do things with open source models and now with cloud code it will very you know just tell you how to set up an open source model in your computer. This used to be something where it was a little bit more gated by competence and now anybody can do it. So each theoretical bound related to ethics or user competence those those have all been sort of eroded. So I'm curious to listen when we think about this creation stack or this distribution stack or the payment stack for sites that are that are making money off of this. How do you think about the different roles that each of them play and what would you like to see as policies for for that stack? So I think of this as kind of a landscape right and as you said we have so the tip of the iceberg is like the the notification websites and services like I do a website I upload an image I click some buttons I get something back. I think those are in some ways the easiest to some degree we take down the website it's pretty clear what the website is now is it going to pop back up a million times yes and so that gets to what underpins that website and what underpins that website as far as we can tell from the research we have done in the communities that I mentioned where some people were building these kind of tools as well as looking at like how two guides that are hosted across the web is that they're using as you said an open weight model they have kind of a system prompt and they've built this such that people can put in whatever they want but they've kind of specialized it to perform well at at notification and so one of the things we find in the report that I mentioned will have coming out is there's no mention of open weight model providers doing any kind of monitoring of downstream use of their services and in fact if you look at like stability AI which builds stable diffusion which is one of the open weight models that's used very heavily on the website called civet AI that I mentioned that has been used for modifications of people you'll see that I believe they reported zero reports to neck-mek during a particular one year period during the same period open AI for example had far far more reports nicknick being the center that runs basically the database for photodyne and other things that automatically use hashes to take down child sexual abuse material and things like that yes and so you know as nicknick themselves who says like the number of reports maybe tells you more about how well you're monitoring than it does about how much abuse is happening and when researchers have done research they've found several different open weight models being used quite prevalently and in ways that are observable because they're kind of publicly on the web so that's kind of one piece of the puzzle the other piece of the puzzle that you're getting at Renee is can we do anything in the model itself like such that the model couldn't be used in this way and what we find is there's like three different approaches you could take at the model level technically right now none of them work very well so one set of approaches people have proposed like what if I remove certain kinds of content from the training data will that prevent that idea so let's say I take all the images of children out of the training data can the model now not make a child plus sexual content otherwise known as child sexual abuse material if you could do this perfectly then it might help in a closed weight model so in one way that you can't modify but even if you could do it perfectly it wouldn't help in an open weight because you can just reintroduce the concept and we can't do it perfectly so in our own research we find it kind of makes it go from three props worth of difficult to 12 prompts worth of difficult in order to generate just because you still have a few examples there is a bunch of research on ways to kind of make this harder do anti-tampering kind of prevent fine tuning none of it's ready for deployment we don't see any platforms using it and so most of the protections that we see from foundation model providers are for the products that they build themselves on their foundation models and they're really those like input output filters like let's try to check that your prompt is okay let's try to check that your output is okay and as we've discussed those are circumventable they're also using AI models to do that filtering and AI models are unreliable so right now it's a very kind of gappy ecosystem the last thing I wanted to mention was we've talked about kind of like that you response in terms of regulating and I know there's been this proposed update to the the AI risk act in terms of kind of saying you can't have a model that can do this unless you've deployed certain safeguards and I think one of the things we found in our research is you hear a lot about the safeguards that foundation model providers have implemented but we find very little transparency information on how well those safeguards are evaluated and even when we interview foundation model providers we find a lot of kind of nebulousness about the kind of threat model that they're trying to address so are they only worried about direct generation of an image or as you mentioned are they worried about code being generated that can go and be used to make the image are they worried about instructions are they worried about API users etc all of that's a little fuzzy the answers to questions we've had about likenesses and sort of what fits the definition of NCII have been fuzzy so I think there's a lot of attention they would need to be paid to like what qualifies something as a sufficient safeguard and how do we set people's expectations appropriately for how much that's going to protect them individually versus kind of means something about mitigation in a specific product yeah so this is actually kind of getting at something that I want to ask both of you which is that like to kind of bucket a little bit the different ways that we've talked about this I didn't tell me if for both of you if this scene's right or this seems wrong like I feel like we're kind of talking around this idea that there's different ways of attacking this problem it's kind of like the post-talk kind of legal reaction litigation type of way to kind of create a post-talk reaction and kind of a tort based like kind of solution set for an individual that is harmed by this type of stuff and ideally kind of like a disincentive to do it in the future for other people for tort fusers who might do this type of thing again to other people there so obviously we don't talk about this but criminal liability who created around that to a certain extent like the NCII in state kind of laws that have been created over the last few years and then there's I think you just said it really well Alyssa there is like the and and Renee kind of gesture at it when she was talking about like the model based versus the open source versus the guardrails versus the regulatory like world of things that we could open up there's like how do you get at this problem when one information is free you can kind of pass all of this along these models are free like is there it seems like whack a mole on steroids yet you just can't possibly kind of do this and then there has been you know for better or worse all of this call for transparency around these things which lets out from a security standpoint a lot of the secrets that we're like learning and refining to like make these products and models safer so one of those speaking of the donut glaze kind of thing to go back to that was something that I had heard about a year and a half ago when I was interviewing people at open AI about like their how they're what their red teaming process look like and that was up popular knowledge of the time and I remember being like wow that's horrifying and also like oh that's not gonna stay a secret just open AI's red team for a very long time and of course it didn't and so I'm kind of just like very interested from both of you going forward you have things like red teams which for those unfamiliar or red teams are essentially like specialty people who are either in specialty areas like you're either a nuclear development or poison control or you're in like you know some version of you know I don't know some version of harm prevention and you get asked by the models to essentially run a bunch of bad actor questions the red hat like kind of idea like you put on a bad actor hat and you test the machine with as much stuff as you can throw on it and that is kind of then garb rails and safety mechanisms are in theory brought in by the company like that is a corporate responsibility idea that we could put into into regulation and force them to have that and be responsive to it but that doesn't get rid of the whack of more problem and maybe you know keeps going so like Renee you asked this question of Alyssa but I'm want to ask it if you and then like Alyssa I kind of want to see if you just agree with that framework that I presented from your research Renee do you think that the where do you think that like the choke point of this is is it all of the above I think there's different voluntary things that the different actors can do immediately red teaming by the way it's not clear from for outsiders what the legal frameworks are right there's certain things that you know you you can't do and reasonably you can't do you know when Meta makes an announcement about its thing and you want to know if you can you know if if it will generate for under 18 accounts that's not something that you can really do ethically as an outside tester. So these questions around what is the formal legal framework for authorized AI safety research, which should be for the people within the companies, which should be outside of the companies for maybe specifically designated individuals. I think it's very hard to know how to do that 'cause just saying I was red teaming is not a legal defense. As far as the responsibilities in the chain, so first of all, I don't have very much faith at open source communities are not, you know, there are these subsets that are actively there for just this sort of content, right? That is the motivation. So with that, then I think you look at questions around distribution. I've seen law enforcement go after certain types of monetization sites. Sometimes that's hard because they live outside of the country. Bellincat did a really amazing investigation into one of the sites. The name is escaping me right now, maybe Alyssa remembers it. But sort of trying to track down where this operator was and then trying to see what sort of legal rules might apply to that person where they are, you have to think about that issue. There's the hosting and cloud services have something of a role to play here, but then people get very uncomfortable about the idea that they're scanning. And again, this doesn't adequately address issues of consent and whether a person has that, you know, those images up in their drive because they just, there's, they made them, they're an adult and this is fine. So you do have a whole lot of different component parts here. And a lot of it is going to be this patchwork of, there will be I think the mainstream companies that will adhere to laws and then be good corporate citizens. And then you'll have these sort of niche outsiders that are going to continue to, to keep that market active. - Yeah, no, I do agree Renee. And one thing that I might add, and I think this relates to the kind of, you know, should we require red teaming as well as to the continued proliferation is, red teaming can vary in quality. And it can vary in like the extent to which it covers the threat models, right? So like part of why we do the research with perpetrators is so that we can make sure when we're doing tests that we're representing how those people are actually doing things, sometimes that's not happening. And so I think the question kind of becomes, if we find that we can't currently safeguard a particular system or, you know, the odds of harm are, you know, ex-sad of why, how do we sort of trade off what we wanna release or allow kind of under like an EU AI risk act type concept versus what we're able to protect. Like if we're only able to protect non-person generations of video, how does that kind of change what we wanna release? And I don't think I alone have the answer, but it becomes kind of a societal question. And so I think the thing I worry about is we're often looking for like a technical bandaid where they're just may not be one or there isn't one yet. And that becomes kind of a question of like, do we try to not let new stuff out? Is that feasible? Do we try to put more resources into deterrence messaging, primary prevention, doing like social norm is shaping? Those are things that I worry about. - I'd love to maybe close with something like, do you think there is one intervention that is technically possible now, but largely neglected by companies that you would like to see implemented? - Yes. (laughing) - Talk us about it. I would love to see implemented more widely is deterrence messaging. So this is something there's like three studies out there looking at how deterrence messaging can, which are basically messages that try to say, hey, the request you have appears to be for content that might be illegal to share, let's say, under the Take a Down Act. These have been used for child sexual abuse material before. There's been research on using them for non-consensual intimacy. The research shows a lot of promise when they're deployed. However, none of the seven foundation model providers we looked at, and none of the ones we interviewed were doing deterrence messaging for NCII to had it on the roadmap, but nobody else did. And I think there's a lot of research to be done in terms of not just deterrence messages for generation, but also like for bystanders, how do we think about that within these communities where people are making sexual content that's not abusive, how do we think about it? But those kind of social norm shaping efforts can be very promising. Yeah, and I should maybe be clear. When I say deterrence messaging, I mean, when you attempt to generate content on a particular, say, foundation models website, or what have you not necessarily when you're sharing content in encrypted messaging, I would not prefer that. So I think what this conversation has really made clear is that AI generated sexual abuse is not just a problem of bad content slipping through imperfect filters. It's really a problem of systems that are built without consent at the center, trying to tack it on now, legal rules that often intervene after the harm is done, and then safety regimes that put an awful lot of burden on the target as opposed to the creators sometimes. So I think the, you've really brought out a lot of the very nuanced challenging issues here. I think maybe some people who are new to the conversation have always thought that, oh, well, those just ban it, and then it'll stop. So I appreciate you walking through the nuance with us. I think the questions are not just whether a model can generate this material, but who is responsible for thinking about potential guardrails, what kind of research is necessary to test the risk, and then how we protect expression without turning content creation into a shield for exploitation. So I want to thank you for joining us, and you have a CDT report that you mentioned. We will put that in the show notes, and yeah, I really appreciate you being here. Thank you to you both for a great conversation. (upbeat music) The Lockhear podcast is produced by the Lockhear Institute. If you want to support the show and listen ad-free, you can become a Lockhear Material supporter at LockhearMedia.org/Support. Supporters also get access to special events and other bonus content you don't share anywhere else. If you enjoy the podcast, please rate and review us wherever you listen. It really does help. And be sure to check out our other shows, including Rational Security, Allies, the Aftermath, and Escalation, our latest Lockhear Presents podcast series about the Worm Ukraine. You can also find all of our written work at LockhearMedia.org. The podcast is edited by Jen Potcher. Our theme song is from Alibi Music. And as always, thanks for listening. (upbeat music) (upbeat music) (upbeat music)