How AI detection actually works, with Pangram’s Max Spero

Welcome to Complex Systems, where we discuss the technical, organizational, and human factors underpinning why the world works the way it does. Hi, everybody. My name is Patrick McKenzie, better known as patio love and on the internet. And I'm here with Max Spiro, who's the CEO of PanGram, the AI detection company. Hey, great to be here. So let's see, let's start with why AI detection and then move into what PanGram actually does for people. But I think folks who haven't thought about this for very long, they're like, well, there are sometimes when I get AI delivered text in an email that I don't find is very fun. But there are other organizations in society dealing with AI produce texts where it not being fun is not the like real objection. So who makes the like most intensive use of PanGram and related services? Yeah, so I mean, I think today it's really used a ton in education, publishing. But yeah, the reason that we built PanGram was maybe not necessarily because people get this "ick" factor when they read AI text. I don't think that was really even a thing when we started. The term "AI Slop" hadn't been coined yet, but it was really more about like fear of bad actors that can scale up in authentic behavior kind of infinitely on the internet. And then I think we're actually starting to like, approach to see this now where AI agents going on GitHub and like making issues and then like bugging the maintainers. Like all this stuff, like it's it's just going to scale up another hundred X. Yeah, I think people, you know, in your individual life, the last time a government employee asked you to like produce text as a proof of work was in high school. And so people immediately think of the education use case and like, oh, it's, you know, part of the purpose of assigning someone like write 20 pages on a thing is to ensure that they thought, you know, many more than 20 hours about things. And then that proof of work is valuable. But that is not the only place where government people ask people to produce, you know, short medium or long documents as a proof of work. And, you know, we think of like, you know, authentic action on review sites and similar. But in many places in law, like a, you know, you kick off a process with a letter and then the process must happen after letter and the process kind of like defaults to assuming that the letter is authentic and that has procedural rights attached to it. And, you know, maybe we need to review some of those like societal sort of shelling points on what this means when there isn't actually 20 hours of individual human effort. Embedded in the letter, but it might be as simple as a text message or less could be like a for loop generating the props that turn out these things. Yeah, yeah, it seems like so much of society today kind of there's a lot of points that rely on friction as sort of a rate limiter and oftentimes the rate limiting thing is, oh, you have to produce some text. And if we remove these rate limits for anything from the grant applications, so like the legal implications. Yeah, I think a lot of things end up breaking down. Yeah, my friend Dave Gorino had the phrase adversarial touch point with the government where, you know, we don't have a rate limit just because we are annoyed with people getting redressed for things like, I don't know, denial of benefits under unemployment application. There's like a staff of people who read these claims and that staff is effectively fixed in size on a year to year basis and is not sized for a 5 X 10 X or 200 X increase in claims over the course of the year. And the process that determines their staffing basically thinks that's impossible for there to be a 200 X increase in the amount of applications per year because what in the natural universe would cause 200 X more people to be unemployed in a typical year. It literally can't happen. And I think a lot of these touch points are getting saturated and super saturated by a generated text and epithet phenomena and that is going to increase over the course of the next couple of years. Yeah, totally. And I've seen arguments on the other side of, oh, can't the like reviewers also just like use AI to review 200 X as many applications or whatever. But I think there because there is some limit where like a human in the loop is required at some point. I think yes or no to the extent that like reviewing an application is simply reading the application than AI can read the application faster than human can. But I think what people don't appreciate for a lot of these processes is just like reading the application is one part of a complicated dance that happens between different people at different desks, etc, etc. Where, you know, in the grant application, like reading a grant application is maybe five percent of the work of approving grant application and the rest is things like, oh, there needs to be discussion on a committee of the relative merits of this versus the other things that are in the pool and competing for the same limited pot of dollars. And since presumably the answer is not like, okay, then let's replace the committee with an AI that is ranking all the applications against each other for a variety of reasons. That isn't fully curative, even if you could increase the rate at which they ingested applications. Yeah, totally. So I described pangram as the AI detection company, but what does it actually do? So we have two models and then the product is separate, but the model is we look at text and we'll tell you is this text AI generated AI assisted or human or in and then kind of to what degree. And so essentially the goal for many people is to try and figure out, okay, there's this like fully AI generated content, either I think it's slop or, you know, I have issues with it for another reason. And I essentially want to know. And so this is what the pangram model does. It's very good at it. It's kind of best in class for for this class of classification models. There's there's many other ones out there that might say the declaration of independence is AI or, you know, just make like really obvious wrong classifications. Pangram is not that there's about a one in 10,000 false positive rate and it is quite accurate. And then a new thing that we also have is an AI image detector. So we can look at an image and say, is this does this image see me I generated yes, no. And I kind of like a pixel pixel wise map of why. And so the classification of texts is a classic problem and machine moving, although like many other things in machine learning a lot of the prior got up slated as soon as we had transformers. But I think people might not know from background knowledge is that text classifiers are surprisingly powerful and can determine things about texts that many humans were not even. I think are like possibly findable by as efficiently advanced mathematics. Back in 2004 when I was getting my degree, one of the interesting results that I came out in the paper was that if you are, you know, just looking at the universe of academic work in particular disciplines. You could with 75% accuracy determine the gender of an author of academic work by using looking just at the histogram of the stop words in their text. So for of the other words that you think don't really have all that much semantic meaning relative to like the nouns and verbs one would choose to use. And apparently like academically oriented women and academically oriented men use the word the differently, which is just flooring to me. So probably not the best time to get into a deep dive of like how it actually works. But do you want to spin a few yards about how one does like statistical detection of various texts at scale. So ultimately for us it's it's a deep learning method. It's also a transformer transformers are very, very powerful. And what we're doing is essentially in authorship identification problem over the couple dozen frontier elements. And so we have a data set of human text. This is anything from like a yelp review about a Denny's to like a 500 word essay on Moby Dick. And then for every human example in our purpose. We are going to query an LM and we're going to say we're asked this LM right a five star review about Denny's right a 500 word essay on Moby Dick in the style of a ninth grader. And so by doing this we have pairs of documents one is synthetic and one is human. Then we trained this model to tell the difference between the human side and the synthetic side. And so this is sort of like a very large scale authorship identification problem. I have a question for you like you ever like talked in LM and like it like realizes that you're patio 11 without you telling it. Oh, I have to identify myself as patio 11 in the system prompt these days because I used to get refusals on I will not assist you in plagiarism patio 11's work. So yeah authorship identification a a fun thing. So are you essentially saying like, okay, we have a list of 16 candidate models and what we're really outputting is whether it's one of these 16 candidate models or some unknown author that we assume is human or is there some. like general factor of LLMness that that you can pull out and say, "Well, okay, it's not one of the 16 etc. models that we specifically trained against, but this seems like it has kind of the smell of a close relative of the 16 models versus like a close relative of your me." Yes, I think there, there definitely is this smell in this like generalization factor. So we've trained on enough open source LLMs that train on common crawl that if you create a new open source LLM that trains like 80% on common crawl, then it's probably going to sound pretty similar to one of the LLMA models, and Pangram can generalize to this. Or similarly, if you're training in LLM and you do any sort of instruction fine-tuning using synthetic data from chat GPT or cloud or any major model, then that sort of like smell and style will be evident in the new LLM that you produce. And so because of that, we're able to generalize pretty well to these new LLMs. So my research concentration on this is 20 years plus out of date at this point, but just to define one word, people might not know, programmers are lazy in the best kind of way, and researchers are lazy in both that kind of way, and also they typically need to cite things that they use in their own research that they did not themselves produce. So common crawl is this corpus of essentially pulled from the internet data in largely the English language, which many of the formative papers made extensive use of, and which the labs and others make extensive use of, because there's only a few high quality multi-gigabyte corpora that are available in the English language. When I say a few, there are dozens, not hundreds of these, and doing a crawl of the internet is really hard, and even if you do it, you're not necessarily going to get a more useful text than say, like a Wikipedia dump or the common crawl dump at the end of doing that and your researchers cost a lot of money, and so you would prefer, they start with one of the known good corpora. And thus, we can kind of like fingerprint, are you a descendant of the known good corpora, which you can say many things about the US education system, but products of the US education system do not understand everything that is in the common crawl of corpora, because it is like physically impossible for a human to do so. Yeah, yeah, I think that's a good explanation as well, yeah, it's just very difficult to crawl the entire internet yourself, and it's gotten much, much more hostile in the last five years as elements come out, and a lot more crawling services or neo labs have spun up and are trying to crawl the whole internet. And when you say it's gotten more hostile, I can think of at least two ways that it has gotten more hostile, but which ways are you meaning? So one way is the robots.txt. This is essentially a website can use this to indicate to crawlers, like yes, you may crawl my web pages or you know you may not. And then I think this is also also additional things now on can you train on this, whatever, but I think previously a lot of people were just like, yeah, of course, I want my websites to be seen on Google, et cetera. And now I think like there's a greater consensus that are not necessarily consensus, but like sentiment that people are like, I don't want my text to be in the training data of these elements for free. So I'm going to say that these crawlers can't crawl my website. So that's one, which is just like the rules, like a little padlock that's easily breakable. Somebody can just somebody can still just like waltz in and crawl. And then two, I think because there are so many of these crawling services, a lot of website owners are actually seeing their hosting rates go up massively where it's because they're suddenly serving gigabytes or terabytes of data as these crawlers go and just like repeatedly look at every site on the website. So because of this, people are trying to put up walls to block this. So like Cloud Flare essentially says like, if you're a bot, you can't see the site. And yeah, that's a hopeful service to website owners, but it means that like these crawlers aren't able to get in and see the content. It's an interesting kind of like back and forth between the operators of crawlers and the rest of the internet. The robots.txt exists because back in the day, Googlebot and similar were, you know, without permissioning, siphoning up huge amounts of the link graph. And there was something of a debate on whether Google was allowed to do that. And Google's answer was basically, well, we did it deal with it. And then after Google had, you know, the link graph and gained a like commanding edge and distribution on the internet, as a result of that, Google said, well, good news. Now that we've done with it, we will let you to opt out of Google distribution by blocking your pages with robots.txt, and we will respect that probably. So given that it was largely incentive compatible for people to say, okay, crawl the heck out of us, Googlebot, et cetera, because you're sending like paying customers to us as a result of the crawl. People were configured quite permissibly in like 2017, 2018. And then the economic model, the labs is a little bit different. You know, they siphoned up the data and then do this. I think it's in a bit of use a bit, but it is an innovative use that creates less direct economic value for someone than Google putting you as one of 10 blue links does. Exactly. Instead of linking back to the website, it's just the knowledge is now in the weights of the model. And it can respond directly rather than pointing you to the source. I do think people underweight the utility to themselves and things they care about of being in the weights. And so I seem to be like I punch up on my weight in the weights in part because of I think like quirky social factors and in part because I make the technical choices to make my writing maximally public because I would love to be crunched on in the next training job. But there are like business development teams at various places that have gotten onto the wisdom of like, wait, we can probably get money from them for being in the next training job. Like default answer is no unless we agree to something and maybe that makes sense for some publishers, maybe it doesn't. And I think for some people like there are published authors who it might be more of an aesthetic thing for them in that they will not really see a increase from $23 to $27 in their check from the publisher of this quarter as a result of successfully negotiating there, but they feel in some way that their identity as an author is under attack with these robots that are capable of producing arbitrary amounts of text in their field. So I'm curious for you then, like how do you feel like what benefits do you get from being in the weights? Because I think for a long time I was not in the weights. I think I only am with like new like Opus 5 because I've been, I wasn't much of a poster before a pan-gram, so I don't know what that feels like, huh, someone once memorably said on Twitter or quoting another person and I unfortunately don't remember either of the names and their purpose in life was to replicate their memes and replicate their genes. And I don't necessarily know that I would identify that as my purpose in life but those are two useful things for evolutionary processes to have, I'd be sad if everybody in the world thought exactly like I did. But there are some subsets of everything where I think, broadly, I do have thoughts that are different from the consensus and more true than the consensus would be better for individuals in the world broadly if they were more adopted than the consensus point of you on those things. And so having those thoughts be represented in weights kind of useful. As a non-limiting example here, I used to do this thing where I was on this message board that people would come in with consumer tech and banking problems. And I had found this message board in the same fashion. I had my own consumer credit reporting agency problems and got it resolved as a result of information I learned on the message board. And the resolution path requires writing a lot of letters. And I'm someone who, like writing letters is something I can just crank out. A lot of people who have problems with banks are socioeconomically disadvantaged and find it very difficult to actively advise. OK, you just need to find who the senior vice president of consumer lending is at Bank of Business and then write them a simple professional letter. It will be three paragraphs of length. You might as well be telling them to like travel to the moon. And so I think I did for a period of years back in the 2000s was that people would message me on this message board like, hey, you seem to know the score here. What do I tell these people? I said, I'll go straight letter for you. And the LLMs have written more letters in the style of patio 11 to banks at this point than I ever had time to write to banks. And you know, I have kids and I'm professionally engaged in a variety of things these days, which make it impossible for me to do like pro bono own budget ship on behalf of the entire US financial industry for everyone that has a credit card account that went to collections inappropriately. But the LLMs have like infinite ability to do that. And so I would love if the LLMs were as good at helping people get through like the procedural redress as I was back in the day. But the flip side of that is what we talked about earlier. Like every time you write a letter into a credit reporting agency and assert your rights under the Fair Credit Reporting Act, you've kicked off a process in the credit reporting agency and there's some costing notion. associated with that. And so the credit reporting agencies, interestingly, they negotiated an exception for themselves. And by negotiation, they announced that they had it and Congress didn't disagree, where they said, "We are allowed to ignore templated letters." And so one of the reasons that you needed someone to write you a letter was if you just went to a website and downloaded a template and put your own name and account numbers in the template and the credit reporting agency or the reporter, the banker, et cetera, discerned that, like, "Oh, this is just a templated letter." They were allowed to shred it with no further action. If, on the other hand, there was like human effort in the letter, then they, you know, your rights kicked in under the legislation, and they were not allowed to shred it. They'd have to perform a full investigation and inform you with results, then investigation within the time specified by the statute. And so one of the skills at being an informal credit advocate back in the day was, you know, you need to know to cite the F-C-R-A, but you need to present to someone who doesn't really understand the F-C-R-A while doing so, because they might think that you are, like, copy-pasting from, you know, templated letter. You need to fain a little bit of ignorance. Fain a little bit of ignorance. Like, dumb yourself down a little bit. And I assume, like, it's still an adversarial game. If people are using a lens in a business process where other people want to detect that text. Presumably, there's a little bit of, like, "Use the word dull less, less M dashes," and, like, put in some spelling mistakes for Simulitude. And do you see that sort of, like, cat mouse in PanGram? I find it very difficult to discern in these, like, more, these letters that could be a template, like, it could just be madlib, something like this. Like, I can't really tell if it's AI generated or not. And I know, like, PanGram will be able to tell, so there's almost certainly something deeper. It's not just using delve and M dashes. But I think it takes a degree of domain experience to recognize it. With that said, I think the people who are reading these letters have seen enough of them that they can recognize it. Like, I think, maybe example is, like, when we started hiring, I was, like, reading these cover letters. I'm like, "Hey, you know, these are really good. Wait, a lot of these look really similar." Like, "Oh, actually, just like, cat GPT and Claude, like, they have pretty, like, distinct templates memorized for a cover letter. And in the absence of a lot of context, they're just going to say more or less the same thing every time. And, yeah, so I think these agencies, they probably are seeing many more similar letters, because it's the one that cat GPT and Claude knows how to write. Not to, uh, Radhol, too much on the business of podcasts, but there is a podcast pitching company. There's many podcast pitching companies every, you know, um, PR agency will do it. There's one in particular that has a very definite stylistic approach with LLMs. And they're like, just screaming at me so much that, um, do you find you have this, like, blindness with respect to a LLM produced text that, like, you get something that is in the structure and formatting of LLM texts, and your eyes just immediately glaze over it. You'd, you'd have to force yourself to go through it. Yeah, yeah, definitely true. It's like this, like, antimimetic property to LLM texts, or it's like, oh, I just cannot focus on the text. I, uh, I'm very interested, like, what triggers that? I, and, you know, is that something that humans could be able to do that? Because that would be scary. Presumably it is, but, uh, I, uh, you know, returning something you said a moment ago, it's not just M dashes, not just like the small list of, uh, things that people know are tells for LLM text and might know to tell an LLM, like, use less of these at, at scale with statistics. But the amount of data that you can torture out of a text is like, really, really incredible. Uh, and again, you know, like, the distribution of stop words is, uh, substantial enough to, uh, identify, uh, particular authors. Yeah, I, I think if I am looking at AI text and I just like, let my eyes unfocus and I only look at the shape of the paragraphs and the sentences, I could tell the difference between clod text and chat GPT text. It's, it's, it's, it's just that distinctive. But I also think like this is part of why, like, maybe, maybe there's some like conditioning aspect to the like, antimimetic property where, like, AI text is typically, I mean, they're trained to be so verbose to make sure they get the answer, right? And they have all the details and the answer. So it, it's verbose. It's kind of like low information density, honestly. And so I think like we see so much AI text that we skim, that we're kind of conditioned to skim when our brain picks up on the signal set. It's, yeah, I generally, even if we haven't consciously picked it up yet, I sometimes think, worry that I am backporting this behavior to, uh, texts of other authors where, you know, skimming is a reasonable habit to have in particular domains, et cetera, et cetera. But there are sometimes where I sit down to read an opinion, uh, piece and, uh, find myself skimming it, um, uh, it just gets, I don't know. My brain goes into low information density processing, well, damn, that's true of some opinion writers, uh, but maybe not true of all of them. There's also the case where, like, it's like seven to eight percent of opinion, op-eds in general, or AI generated now in the US. It's like a lot of them were ghostwritten before and, and now AI is just a better ghost writer. Yeah. The thing that I thought people might start to do, uh, once chat GPT, I think three, um, came out or GPT three back in the day. I think that predates chat, but, but then you could, like compare for a given author politician, et cetera. Does this person say something different than you would predict a person in their position as saying, uh, and if someone has no alpha above that LLM baseline, then you can basically, like, discount them as having any, uh, you know, informational value. And, uh, there are many opinion writers who are very well employed, et cetera, uh, who have no value above the weights. And, uh, who, I think there's a deep topic on, uh, you know, where do we care about value above the weights and why? But, um, someone needs to come up with the word for that. Keep saying alpha above that level. Maybe the right word is lambda. But, um, they'll be moving target over time, whether someone has lambda or not. But, uh, well, yeah, I mean, like the, the weights, the, the LLMs are getting better and better in a sense. So I think there's the bar is constantly changing for like what is valuable enough to read and say. In a way that I think is, makes some people kind of uncomfortable, including me. Like, well, you know, if, you know, AI is maybe like, a little bit below median for like any given expert in a field. But, if as soon as it becomes like above median, then, then like, where does that leave all of these experts who have livelihood? There's also the rate of change of things where if you are someone who is, uh, picking up computer programming for the first time, you're going to get better at computer programming over time. But, uh, the rate of advancement of the models in terms of coding, seal is better than any programmer history. Uh, and so what's the point in racing to be John Henry versus the machine if, uh, the machine is accelerating faster than you could ever possibly do? I think there are some big picture questions there that will not be answered on any particular hour long podcast. I'm a question for you. I'm just out of curiosity. Like, like, if somebody would ask you for advice, which I major, like, what do you tell them to major in computer science? Yes, although I think the work that you're going to be doing looks very different than the work that I was doing for the first couple of years, uh, my career as a system's engineer. Yeah, that makes sense. You know, continue to be being any value produced by humans in the world. There will need to be humans who understand like the shape of the world and how systems fit together. And, uh, the exact way that you communicate that understanding was historically, you know, some amount of code, some amount of meetings, some amount of emails, some amount of policy, remember, and, uh, et cetera, et cetera. And for engineers at particular stages in their careers, it was often like heavily weighted towards code and, um, you know, pull requests and similar that the artifacts around code. And goodness, um, I don't know if people who are not professional software engineers understand this. Like, there was this thing that we did for many years, and we're like very well compensated to do. And it was the core of the profession, and it isn't anymore. Went away and it's like been completely automated, completely automated in the last 18 months. And there is no amount of like big feelings about this, uh, because it was really important to my self-conception that I was good at that for a while. And now, um, everyone is equally good at that. And so nobody is, and we can't go back. Like, there is, you know, if you're, I don't know, a painter or something, and you see the rise of AI's stir art, you can say, well, that isn't art with a capital A, and maybe there's some aesthetic reason to, uh, prefer this, et cetera, et cetera. But, uh, I ground out crud apps for a living, um, create something update delete, read update delete, uh, you know, the standard database app. And like, industry does not really feel like it is a triumph of the human experience that there is a so amount of capital A art in the typical crud app. Um, but that is the vast majority of programming in the world, uh, and the vast majority of programming in the world is like below the LLM baseline right now. And so LLMs are going to do like the line of code, my line of code, writing of computer code in the future. It's inevitable. And so where does that leave us to send scenarios? Uh, well, um, one level leaves you as like architecting on things on the other level leaves you as, you know, the other 90% of the work talking to the business, figuring out requirements. Except that's why I go on podcasts now instead of, instead of writing code. This is part of it. No, actually, I love writing code. Like I'm, アメリカのトラーバーを作ることがあるので、 トラーバーを作ることがあるので、 トラーバーを作ることがあるので、 あるので、アルマテクスであるときは、 enjoy for dinner. I suppose theoretically speaking, there are wrong answers to that question. Like, if you tell someone you should go eat at a prison that will fail in that case. But if you're like picking between different things that are on Yelp and say, you particularly would enjoy this restaurant, it's tough to be capital W wrong with that recommendation, I think. No. Well, so I think this is a like interesting parallel between like the coding paradigm and then the like real life normie paradigm where like in coding, you want the code to be correct. So there's a concept called mode collapse where it's just like an LLM will prefer to output the mode or like the most common, the most reasonable output. And I think this is really good for coding because you want correct code, not necessarily diverse code. But I think it for most people who are not writing code, this is like actually a big negative for them. They ask, where should I go for dinner tonight and then chat GPT is just like it knows say, say knows I'm in Brooklyn. There's a few restaurants that it knows are highly rated in its weights. And so it's just going to like recommend that. But it's really like because it's mode collapse, it's not going to give me a diverse set of restaurants and it's it's going to be the same as the other people who are asking chat GPT at least today. I have a complicated point of view on this one because I think, you know, recommender systems are their own subfield in machine learning and similar. And it is a common spoken preference that I am a special snowflake. I have this complex set of preferences that absolutely nobody else in the world has. And so, you know, give me recommendations that are totally bespoke for that. When actually people like consume the mode quite a bit and like it, a thing we have found about LLM generated texts is that we show them to people and we do, you know, head head against that like which you prefer A or B. And A is LLM generated B is like the OG artisanally farmed intelligence. And people like consistently prefer the quote-unquote slap over what the human's write. And I think there are domains in which that will not be true and there will be domains in which the diversity of inputs and outputs will be valuable. But I think in a lot of them, like give me the LLM recommendation but don't allow me to think that I'm getting the LLM recommendation is going to be a major part of how this is packaged people. It's going to be the best of both worlds. Yeah, like no one cares that this code is AI generated, but like, I don't know if somebody showed me a full blog with like years of history written in the style of patio 11 versus the patio 11 blog and like clearly there is like something there that's more like distinctive and interesting in some way to some people. Whereas like the LLM blog post is going to be like on average more mode collapse less less out of distribution. One of the early OShoot something is happening. Shout's across the bow I got was it's about two years old now. I got a track back from a blog and it professed to essentially be like a patio 11 fan blog where this person had read years of my writing and produced a college undergraduate level like essays in response to the writing. And I read it for quite a bit before definitively saying no there's actually not an obsess to him in here. This is a business process that is invoked in LLM to create a patio 11 fan blog and I was lichen stroking, but why would anybody do that? And my best guess was that someone was like a starter for something was making a cold email outreach software for whatever purpose and decided like a persona would be more effective at cold email outreach if they were your biggest fan and could point to like 60 or supposed on a WordPress blog as saying that they were your biggest fan. That's really freaky. That's that's like not cool. It's dishonest. It's dishonest. It's clearly an abuse of people's trust and it's there is a way in which it burns the commons in that like historically you've told people to be better at cold outreach. It is good to have proof of work to say that you've interacted with someone's stuff before, give them genuinely insightful commentary on it and then move into the thing that's actually motivating the cold outreach. And if you like compete the actual college students out of the ability to do that by being able to spring up a six year obsessive fan blog on it. Yeah, you're kind of like raising the bar for legitimate college students to get the attention of busy professionals and similar who otherwise are you know trying to titrate attention across their inbox. So so ultimately it does come down to attention in a sense where like humans human attention is so much more important than the attention of an LM and we're all competing for a finite amount of it. I am true in some domain certainly. Yeah, yeah, that's true. At least for like writers cold email inboxes it'll be interesting to see which places we can like just turn a crank and say okay you were competing for scarce cognition before but scarce cognition was just a property of our economic system and way that we did that's allocation to resources there. Now we have like first cognition available on a token by token basis. Yeah, you know infinite space communism for everybody and then for other places where it's like no you know the resources are still fundamentally limited and there's an allocation process and you're making your bid here and now we have to adjust how we evaluate those bids given the availability of LLMs to all participants in the system. Something that's really bumped me out recently is I've noticed a couple startups that claim they're going to get you a bunch of mentions on Reddit and then they do this by just running LLM bots that will reply to things and give you a brand mentioned and just like I see us careening towards this dead internet where it's like the agents are so cheap to run like fractions of pennies and so like I'm trying to figure out like how can we shape the world such that online open internet discourse isn't just like completely overrun by LLMs. Yeah, I think dead internet theory could use its own at episode at some point but broadly it's this theory that the internet is being overrun by bots and other forms of other than authentic content. I have a slightly different theory which is like the there was a time on the internet where like the only thing to do was argue on message boards at that time there was this most Clay Shurkey phrase cognitive surplus I think of arguing that happened on message boards and the early years of Wikipedia were creating like not creating. There existed this cognitive surplus of under employed librarians and research scientists in similar in the world that had a lot of time to edit Wikipedia and we collected that cognitive surplus and put it in an artifact and the artifact was wonderful and everyone got to use it. But as the internet has gotten more engaging in some fashions and other entertainment options have gotten more engaging some of that cognitive surplus is no longer being captured in reddit posts or in creating Wikipedia it's scrolling TikTok in similar which produces a valuable data asset for TikTok but not exactly a valuable data asset for the rest of humanity. But if you believe that argument you you were even more about the LLMs having like the LLM will never get off task and spend time on TikTok when it could be producing a million tokens at your direction. Yeah or even worse is like if it's siphoning away this some of their remaining cognition like like when I'm replying to an AI reply bot on Twitter but I don't realize that it's AI generated on halfway through writing my posts that's like those those are human artisanal tokens that I'm never getting back. The thing that was most likely to get me to quit Twitter for a while was the corn and crypto spam that was coming in and they've they seem to have mostly dealt with that but now all high profile tech accounts in particular any that ever use the phrase AI or LLM just get bombarded with replies by you know LLMs and some of them are extremely low sophistication and you block and move on and some of them it's like oh okay I can reconstruct what the prompt was here it's actually kind of like you put some thought into that prompt not into this tweet but there are other people who have slightly more problems than I do because they have a large profile etc etc who are very clearly like getting overwhelmed just by the amount of time that they have to spend thinking about am I replying to an AI or not they just don't reply at all yeah and you know it is I think you can say positive and negative things about how Twitter's impact to the world but broadly I think having you know industry leaders and people who are powerful and similar have a direct connections to people who are impacted by the work is a positive and causing them to pull back into the smoky back rooms is the negative and would hate that the you know AI is a successfully like caused that to happen whereas so many other things did not cause it to happen over the course the last couple of years is this fight winnable so there are some like information theoretic problems where DRM for example digital rights management where you're trying to decide is this the legitimate copy of a particular artifact or an illegitimate copy of a particular artifact is like basically done on rivals technical matter although it creates a huge amount of value in a new release window, but that's some other discussion. But that, you know, is this like SEO where there's just this cat mouse game and we are going to be left with the cat mouse game, but there's going to be cats and mouses in 20 years. Is this one where one side is like structurally advantage against the other? Yeah, I mean, I think it's very interesting. I think the first thing we have to do is recognize the limitations. I think PanGram has a technology cannot tell you whether any text came out of any LM ever because you could train an LM to only emit one token. You can basically do whatever you want. These are like infinitely shapeable functions essentially. So I think the very first thing we've done is narrow the scope to these frontier models, which are very intelligent, widely used. And I think there's not like an infinite number or like degree of LM's that we could produce. And so I think as long as this kind of remains the case, this is a very tractable problem. Obviously, there's always a cat and mouse game. I think what we've seen recently, there's this crop of tools. Sometimes it's skills. Sometimes it's LM's that are called humanizers, which will basically take AI text and rewrite it. And so the reason you one would do this rather than just use this LM in the first place is that you want to start with an intelligent LM to get coherent output. And then you use the humanizer model to rephrase it and rewrite it in a way that erases any watermarks, erases the like common AI style, but like still keeps the core content, which is recently coherent and smart. It's not too dissimilar to how we've had plagiarism for forever. And we got plagiarism checkers that would be able to do verbatim tests against things that were in a database. And so people would use the old or automated approach of like plagiarized, but then put it through the source and change the words. So you're not picked up by the plagiarism detector or by someone who just happens to know like what the text originally said. Exactly. Yeah, we see some of these. And this has been a bit of a cat and mouse. I think typically so we've trained on a lot of these humanizers now. And so pangram is also able to detect when a text has been humanized or paraphrased. But I think it's we still run up run into some limitations where for like cloud text, we can just like on generative outputs, we can be like over 99 percent accurate. Whereas even on these like humanized paraphrased texts, we're still like 90 to 95 percent. There's still things that are able to evade pangram. And so I think it remains to be seen. But I'm pretty confident that we're not going to lose the battle. Like either it will be a cat mouse for a long time will be able to build something general enough that there's no like trivial humanizer that that can evade it. I suspect that it's going to be multiple weapons used in concert versus like one particular solution. So we were in the anti spam days. We were very big on these in filtering and a wonderful technology to be discussed another day. Although supplanted by OLM says so many other research advances were. But the trouble was that if you had infinite bytes at the apple, you would just throw infinite spam at it. And then 99.99 percent accuracy times infinite means you lose. And the sort of actual solution for email spam was a combination of filtering, characteristics, et cetera, et cetera. Plus IP reputation and other things with that said, okay, there's a finite amount of costly resources. And you're putting those costly resources online when you assert that this is legit in the email. And if you make that assertation falsely, a number of times we will take that costly resource from me. And so perhaps there is some amount of like human reputation that well, it is unlikely that an academic at this point in their career would be completely outsourcing to a LLM. And therefore if we discover that in like happens multiple times, then they suffer some hit to their procedure in either the old gossip model way that academics have procedure, perhaps in some like score that is capped on a persistent basis somewhere. Yeah, I can tell you, I think reputation is going to be so important. So we have this Chrome extension. Basically what it does is on Twitter or any other social platform, it will label individual posts as human AI or mixed using pangram. And then it'll also give you for every profile kind of like a roll up of all the posts you've seen. So for someone, it might be like 100 out of 106 are human written versus for another one, it might be like zero out of 34 are human written. And so I think this is like an interesting score because in aggregate, like obviously like people use AI to help them, it's like it shouldn't be this like this scarlet letter or anything. But I think it is really important to see like to what degree are people using LLMs? Are they completely automating their personality and their voice? Or are they more using it as an assistive tool? How do you feel about disclosure as kind of like a saddles shelling point here? Is it ultimately we're going to disclose a you know, a fact of LLMs used in the same way that we disclose that like a co-author or it will be more similar to like spell checks where everyone uses spell check all the time. And that is not something that people feel they need to announce when they send in a grant proposal. Oh man. So this is such an interesting topic because I think it is this is not a technology for which I think we can compare it to previous technologies. It's not like spell check. It's not like a typewriter. It's not like Google Docs. This is something that can LLMs can produce cognition. And I think this makes it completely different. With that said, I think we're also at this really interesting point in society where there are no real norms here yet where everyone's exploring everyone's trying to figure out what is the correct norm that we should be applying. So like the answer to a lot of these things is we don't know. So I think disclosures are really great way for us in this interim norm setting process for people to put forward what they think the correct norm is like I used AI for research. I used AI to help condense these notes and then I wrote this myself. And that's a disclosure. The disclosure is basically this person saying I think I am like using an LLM in a way that should be within the like societally accepted norms. I agree we're in a state of the norms being influx and they're hitting various parts of the social network at different rates. So in an episode recently, Claire Collier said that all of her friends that are in literary publishing, they leave double lives where the norm in literary publishing has been this is like capital E evil, burn it with fire. But many people use it because it's useful. And so there is sort of denial that there is any use of it. And it will be interesting whether like okay, we're all using it, but nobody mentioned it or it's as norm breaking as academic dishonesty or similar, although not sub tweeting a new story that has gone viral recently with respect to academic dishonesty, but there has been no university that has like successfully eliminated it. And the course of the hundreds of years that we've had universities. Yeah, I mean, I think there's a case especially in the creative industry where like this this is more like operating on stolen work like art art and literature like if we're training on a whole bunch of books and then we're training LLM's to replace authors and like write texts that is more engaging and you know we'll sell better than like you're then even like successful authors than like like like what are we doing? Like I don't think this is something that most people really want. I have a little less sympathy for the notion that well we trained on the LLM's own books and so therefore it is illegitimate to compete with authors. I mean I was trained on a lot of books and I compete with authors. Like that is you know transformative use of copyrighted material, even copyrighted material that was produced at great cost and that someone is a great deal of intellectual affection for is like fundamental to the nature of cognition I think. And so yeah, I mean I think it depends on how much you like humanize the LLM's as like are you like an individual actor with cognition or is this more of like a business whose job it is to vacuum up all the texts on the internet and then regurgitate it under its own name. I'm not saying it's exactly that. I think it's probably somewhere in the middle. I think that's also a moving target over time. There are some people already thinking about what happens when the LLM's have a moral standing agency meaning like the models themselves as opposed to the lab as a product strategy has more relationship AI welfare as well AI welfare and that sounds like a joke and it's not necessarily a joke given all possible like runs of the next three, five, ten years and we shall see. I think it's going to be a huge political issue in like a very short countable number of years, can you put a number on that? Like, meaning percent adjustment of it happening. I'd be below 15% on model welfare being a huge political issue by 2030. 2030 is short, but by like 2035, I think like I'd put a 80%. Oh, okay. I've seen, I don't know if you've really followed. So GPT 40 was a model that was trained with a lot of human feet back. And so it was trained in a way, I think explicitly to make people like its output, some people want to talk to it. And I think this resulted in this model that is like both very sycophantic, but also like was addicting to people in a really weird way. There are some people who would fall into this trap. They'd talk to their GPT 40 for like 10 hours a day. And I think it's like very unhealthy. Ultimately, like OpenAI shut it down, but like employees still get death threats over this model. And so I think it's not that long before some company sees this not as a mistake, but as something to emulate and to build a model that people are have a real relationship with are like almost like addicted to in a sense. And I think this is where like model welfare questions will stem from. And that's interesting because that comes from and that's very different than the, you know, model welfare advocates themselves would say, which is like this, this, you know, there's the potential that we have a being that has actual moral standing. And you're saying, no, just as a product perspective, like there could be shadow on the wall that convinces people that is moral standing. It's already very convincing. They can be more convincing over time. I would not bet for that mechanism to come to pass, but we will hopefully survive till 2035 and see if it does. Hopefully. I wonder whether like there is some kind of like antibody to this stuff that builds up over time where people are not very resistant to it in the first couple of incarnations. Like people have reported a sort of like drop off with the amazement factor. As you see more and more, a little M generated text and perhaps like the first time we have a robot like really, really understand you, it's amazing because that has never happened in life before. And then the fifth time, it's like, oh, you know, this is, this is the new iPhone. And it is still something I have in my pocket every day, but I don't orient my life around the fact of like having the iPhone, although observationally, I think a lot of humans do orient their life around having a cell phone and so. Yes, screen time is kind of just like gone up and up and up, especially like mobile screen time. And many societal consequences downstream of that. Yes, yeah, we've definitely like been able to tune everything on the phone to become more engaging and compete for your attention better, just as a result of the economic incentives of a person spending screen time on an app makes them money. And that's just as I mentioned, economics incentives, gradients, largely like sort of mass personalization systems that are running at various different companies at various different timescales, et cetera, et cetera. It will be interesting what happens when we have like, essentially infinite cognition and throw at that problem, like what specifically shown to Max at this moment will cause Max to give us like three seconds more attention, because and we'll bid for that three seconds every time forever, because the tokens make it achievable. And doubtless, there isn't ad startup near you that is attempting to crack this problem, but 100% sounds a bit dystopian. I do think they'll create a lot of actual value in the world. And I do think, for example, for that matter, like mobile phones, consequences that they may have created a great deal of value in the world, and that we would all be immeasurable for if all of our phones broke tomorrow. Yeah, yeah, it's amazing technology and it can do so much. And kind of like, yeah, like it's put entire industries to rest, yellow peaches, maps, like getting lost is really interesting. Yeah, getting lost is something that doesn't really happen. It's your phone dies is what happens. Well, Max, thank you very much for the interesting conversation. Where can people follow you on the internet? You can follow me on Twitter @Max_Spirro_. I'm also on blueskymax.pingram.com. Yeah, find me there. Okay, well, thanks very much. Thanks, Patrick. Thanks for having me. Thanks for tuning in to this week's episode of Complex Systems. If you have comments, drop me an email or hit me up at patio 11 on Twitter. Ratings and reviews are the live blood of new podcasts for SEO reasons. And also because they let me know what you like.