RSS 46: AI Is Not Search. Here's What It Actually Is with Britney Muller

[MUSIC] So being a know-it-all used to be considered a bad thing. But in business, it's everything. Because right now, most businesses, they only use 20% of their data. Unless you have HubSpot, where data that's buried in emails and call logs, meeting notes, they become insights that help you grow your business. Because when you know more, you grow more. See, being a know-it-all isn't so bad. Visit HubSpot.com today to learn more. [BLANK_AUDIO] Millions of people every single day are logging into Claude, Chatchy-P-T, perplexity, in a wide range of other LLMs. Sometimes they're looking for products, sometimes they're looking to achieve a task. And the person I speak to in this upcoming episode has deep knowledge. After years and decades of studying tools like Chatchy-P-T, Groc, Plac- Brittany, thank you so much for making the time. Really appreciate it. I was trying to figure out how long ago we met. I think we're hitting the decade mark. I think we've been OGs here long enough to go back like 10 years may. It's wild. It's crazy. >> Do you believe that? >> I do not. >> It is insane. There's been a lot of change since those early days of wonderful halls of Mascond and all of that good stuff. But I'd love to dive into some of the big changes that have happened right now in what you're seeing in the market. There's a lot of hype around AI, there's a lot of hype around GEO, etc. What is the thing that you think right now? People are just getting wrong about the entire shift in search and discovery. >> Yeah. I think the biggest thing that people are missing here is that it's a completely different technology. It behaves and operates completely differently than traditional search engines. We're trying to stuff AI into this old SEO mental model that worked back when we started in this industry. And those things like ranking factors, the ability to reverse engineer how it's producing something that doesn't work with LLM. And so I think people are getting it wrong in that way for lack of a new mental model or understanding about the tech. >> That's cool. Let's go a little bit deeper into that. So what do you think does that mean? Explain it to me like I'm 16. What do you mean by there's a disconnect in terms of understanding the tech and how it's actually so different? >> Yeah, I mean, I think at face value, it behaves very much like a chat interface. Right? And because of that, because it's confidently amalgamating answers and outputs to us, we give it a lot more weight than I feel the systems like deserve. Right? It feels very intelligent. It feels very confident knowing. And it behaves a little bit like search. Right? You go, we can type in more nuanced queries, more natural language, information to get some answers. But I think what we're missing here is that there's two things happening. There's the base LLM model that is basically a frozen snapshot in time, trained once a year if that. So it's already outdated by the time it comes online. And in order to supplement and kind of keep it updated on information, these companies connect the LLM's to real-time search results. So whether that be Google or Bing, it's retrieving these background queries and contextual information to stay relevant, to stay updated, to hopefully provide you with a little bit better output and performance as far as the LLM is concerned. That's two totally different things that are taking place. And so I think it's important to know the difference, know when that search is being deployed, and how much of a fact that has on the actual out delivered outputs. But then just to contextually know the difference between like that core LLM and a search engine, you know LLM's, they're fed all of the text on the internet. They don't even get fed the URLs of the text that they're trained on. Like they have no concept of where the stuff even comes from. They have no concept of like any kind of domain authority or trust signals. None of that. It's all equally the same. And what gets represented is it essentially magnifies the training data, magnifies the most common or popular brands and outputs. It doesn't really represent outliers or people in other parts of the world that well. So it's something to consider. And then when it is connected to real time search results, that is still something we can influence in different ways. But even that is so interesting. You know, SEO will continue to be important because of that delivery method to LLM's. But we're seeing this like slight step change in how people are approaching it. You know, there's an AI startup that I know about through an advisor I work with. And they just started, right? They just kind of came about. They have no chance to rank for some of these desirable terms. Right, right. They shell out huge money to have advertisement on one of the most commonly surfaced domains for this particular query. And this, I think it was specifically through chat GPT or whatever. It was commonly pulling this one you know. They didn't care about it link. They didn't care about CTA. They just needed the brand mention so that they could be baked into that answer. And it worked. Like those are kind of like the step changes that I think need to be considering more. And so things like that are super interesting. I mean, you've been all over this forever. Like the social media plays and the fact that I can't compete with people, you know, human people and stories and anecdotes. And so that's why you know, Google immediately partnered with Reddit a couple years ago, $60 million dollars a year for all their data. We continue to see them kind of shoving social media posts into the serps because again, it gives users that option. Like here's a succinct kind of a malgamated AI answer. Or here's, you know, real testimonies from real people across the internet. And I've seen people play that really well too, where they know. And you know this like Google's transcribing those videos. And so you can be really happy about what you say. How you open a video so that that's all fed in and you get surfaced among these AI answers. And so those kind of, that's the methodology that I think is really interesting right now. But again, just to be aware of those two things taking place is so important. So something that everyone is trying to figure out is how can you show up in the LLM's and what should we actually do as marketers and brands. And I'm sure as someone who has deep experience in the world of LLM's AI, like you were really early, definitely probably, I'd say definitely one of the first three people in this SEO world to like go all into the AI, machine learning world and stuff like that. People come to you asking this question probably a lot. How would you recommend organizations who want to influence and want to show up in the LLM's, be thinking about their own marketing efforts? Like what should they be thinking about? Yeah, yeah, it's funny. I mean like 10 years ago, I built my first language model and I was teaching at Roads. And I was so proud when it started to spell words correctly because it was character-based. And now it's word-based. Now we've jumped way ahead and it's so exciting, seen like that evolution. But the process is the same. It's a word-predicting machine. And so the more, and it is a bit of a popularity contest at its core. Or you are represented in the training data among contextualized relevance topics and information that you care about that you want to show up for, the better you'll do. That's super important. You know, you could show up a bunch in the training data but be related or convoluted with some other industry and that could actually hinder you. Right. You want to contextually show in places online where these conversations are taking place about your product or your services. And then aside from that, so that's like the core LLM just showing up without any sort of rag retrieval on the generation through search engines. That's just- What does rag mean? Some people are like, wait, let me google that. What's rag? Yeah, so retrieval augmented generation. So rag can essentially reference a set of documents to better update and improve the output of the core LLM model. What we know today's LLMs to be doing is they're accessing search results. They're doing rag through real-time search results. And it's not transparent always as to what those background queries are where it's collecting information. Also, it has to meet some criteria in order to even deploy that. Anthropic, they had documentation leak like a year and a half ago is fascinating. And it was essentially these like rules and guidelines around when to deploy a background search and when not to. Right. If you ask, you know, "Claude, what's the capital of France? It doesn't need to go to search results." Right. It knows that and so there are some like guidelines around when and why you should deploy that. It's important to know that that is what's going on and so again like traditional SEO is going to be more important than ever and it's a it's a brand-mentioned game you know in addition to having kind of control over that messaging and showing up in search. It's being present on those other players that are showing up in that retrieval augmented generated way so that you're part of the conversation but the end of the day it's still a little bit of a game of a magic eight ball you know it's a possibility based so it's like people are trying to measure the eight ball people are trying to influence the eight ball you can do whatever you know you can do all of these things and still not show up but you're essentially increasing the likelihood or the probability of you being surfaced in this one. So are you saying there's no guarantees? There's absolutely no guarantees and there's also like there's no reverse engineering the LLMs or AI is like not how these things work it drives me crazy like yeah the the LLMs they don't even have access to their own internal thinking they don't even know why they generated a you know a 404 URL for you you know they have no right they just regurgitated this information and so that's really important to be aware and to also like set those expectations with clients right because a lot of GEO people are kind of over promising these deliverables or metrics when it's really I feel like that's a really dangerous zone because you at the end of the day you don't have direct influence over that you can do everything that you can but yeah you just gotta be careful how you're talking about how you're setting those expectations. What should be the level setting like instead of going down this path of the traditional way in applying it to LLMs like what do you advise people say when there is a bit of innate ball like what would be the framing that you would advise they start to embrace instead of the current model. As far as tracking goes? As far as communicating it to clients like instead of saying yeah we're gonna reverse engineer the LLMs and then we're going to give you some insights into how many citations we think we can influence based off of this like is there a better way that you've cracked to say like this is actually if marketers had to take a Hippocratic oath around how we show up this is how we should be communicating more regularly with our clients like what does that look like? I love that I love that I think I wish we did after doing oath I think we should. I think we should that should absolutely be a thing because it is just crazy stuff that's going on cash grabs cash grabs right and people are losing their minds and selling their souls for this stuff it's like it's disgusting no I completely agree and I think part of the conversation should really revolve around like honest and transparent expectations and AI literacy and and not even just telling them about how different the technology is show them right show them do the same prompt multiple times in the same interface and view how differently it can you know surface answers and explain the probabilistic nature of these models and how it's not you know there's no ranking system that doesn't exist what exists are these background kind of probabilistic scores for you know what they surface will never have access to that information today's measurement modes and like tools are extremely extremely crude metrics based on synthetic pretend prompts we think people are doing we only have that right then it's like what did the magic eight ball say when they ran the test so what I always recommend to to clients and people talking to clients as well and a lot of people aren't gonna like this but is to build your own internal tool it is easy now than ever to go to cloud code it will save you so much money so much headache build your own custom internal tracking tool that you can run at whatever frequency you want you have full control and flexibility it is pennies on the dollar the API you can even use I love the the bad they call it like batch API with okay I where you can run tens of thousands of queries I'll do it at night for different clients and yeah so cheap it's crazy but it takes a little bit longer so you can set things up like that but that's really the approach I would take instead of you know paying ridiculous amounts of money for these tools and you don't know how often they're run you don't know you know how what exact API they're using or how it's prompting in the background I mean there's just so many unknowns and for a like barely directional metric I just think we have to be thoughtful so would you say more businesses and organizations should be thinking about how they can create internal tools that essentially would run the same questions to chat you be to your cloud perplexity etc on their behalf with prompts that they upload at a fraction of the price of some of the off-the-shelf tools so they can get the same level of details because you don't believe that the data that those tools are even getting is close to as good as what they could get if they just ran their own tools is that fair 100% 100% and then you're able to run those prompts maybe a dozen times a day right that seemed to get a better feel generally what might be the of a range for the the percentage that you show up that's how these things work so I feel like putting the power in the user's hands and again it's easier now than ever to spin things up like this and I think when talking to clients I was just reminded of this brilliant analogy one of my students Ryan Shelley shared with me of like traditional SEO is like having your product on the store self and showing up in an LLM or an AI is like being recommended by an employee right right you know you can do only you can influence it in different ways you can hope that they recommend you but at the end of the day it's up to the employee who they decide to recommend to you know win and wear so when it comes to marketers we've got we're going through as marketers one of the coolest times to be a marketer you have tons of opportunity in the sense that there's a lot of change happening but with the change comes the necessity and the capabilities to now be able to build things like you just described I think something that you've done for the community which is cool is build orange labs where people can actually get coached and guided around how they could build some tools like what we just discussed can you tell me a little bit about what orange labs is and how it's facilitating this augmentation of marketers with AI yeah yeah I'd love to it's it's funny because it really emerged from a clear need beyond the course that I teach so I teach actionable AI for marketers and it's the only kind of small cohort based live where we're hands-on and you're building a project throughout the four weeks that solves a problem that you really have so it's highly collaborative it's highly hands-on we learn by doing and I have kept in contact with tons of graduates beyond the course and something just kind of continue to emerge in these like monthly meetups was just a demand in a space to continue the conversations right different kinds of support are needed after that kind of foundational knowledge is set and so I wanted to I really want to give them that and I tried giving it to them on Slack and it didn't work very well and it was disorganized and so now we've really kind of re-homed everything in orange labs where we do regular workshops tutorials networking events people are getting real-time support you know they just you jump in with a problem and this morning someone asked something about creating product descriptions with this given data of a large website how what's the best way to do it people are jumping in with prompts that they could start with right jumping so support and troubleshooting and there's no dumb questions you know I think something that I personally have struggled with in the actual you know the AI world is lots of barriers to entry right set up to kind of exclude people that don't look a certain way and don't have a particular academic background I just don't think that's the case and I think it's important to make sure that we kind of have this space where everyone feels comfortable and welcome and able to contribute and learn and fail that's the biggest thing is like break things try something like that is the path to AI success and so continuing to like kind of cheer one another on and support those efforts is so so important so I heard you speak recently and you gave examples of some of the cool things that folks at orange labs have built would you be able to share with the listeners an example or two of things that your like community members have been building out of some of the collaborations within the community and just give us a glimpse into what might be possible for some marketer that's listening to this and thinking I want to do it I don't know how to start what could I even create that would be a value for my career yeah I would love to I love that question so so many of the graduates are on fire I mean Federico go past calls like an AI legend in and of its own, but he has basically created this workflow that uses Phantom Buster to identify people that are engaging with competitive posts. So he's a one-man band. He started this company called Wordcrafter. It creates AI-generated content briefs based on what's showing up in the search results. So based on intent and all this information, it kind of curates that for you. And he saw Byward doing that. And so he thought, "Okay, I want to see who's interacting, who's engaging with Byward?" And how could I capture that information? He collects it through Phantom Buster. But then he has one of the most incredible prompts I've ever seen that I share at Brighton and in the talk is he collects all that in a Google sheet and then using GPT for sheets. He works with an LOM to feed it, their name, their title, just basically whatever's on their LinkedIn. So it has all this visual information, also contextual about what comment they left, how they engaged with this post. Yes, and if you do it like recently, you can really tap into, "Oh, this person just said this thing," but it crafts these drafts to reach out to people based on what they're doing with competitive content on LinkedIn. So it was just genius. I took that workflow and I ran an experiment for Brighton. I got over a percent response rate. 80 percent. So wild. So wild. So like, yeah, that is just kind of nuts. That is nuts. Maddie Osman, she's so brilliant, she's a graduate. She has this brilliant zap that she set up that, you know, she realized like I get so many emails a day. And we have all of these link building opportunities. We've got, you know, these horror requests, we just kind of ignore, largely ignore. And they, there could be cool opportunities in there for clients. And so she set up a zap that collects all the horror queries, evaluates them based on client criteria, topics, industries, kind of, kind of validates which ones might be a value. And then, and then takes it to the next level where it even drafts like based on everything the blogsmith has ever written about this topic. Here's how we, we might respond to something and it creates a draft for her to respond. And then she gets, it's wild and she gets all this through her slack. So she'll get like an alert. Here's who it's from. Here's the deadline. Here's a graphic response and a direct link to that horror to respond. I mean, it's just genius. But I think that is genius. It's so cool. Here, where these, you know, examples really stem from is these people, these professionals have brilliantly centered the work in the efforts over a clear problem. Identify a really, really clear specific problem. They break it down into specific steps with that engineering mindset. And then they discover and use what technology could help support this. They don't start with the technology. That's where people get stood overwhelmed. It's a lot, but people that start with like really, really clear problems. One last example I think is so cool that just came about is Martin Cantola is his amaniac right now. He has spun up slowly and like truly insane things that the rest of us at Orange Labs were like, how are you, this is insane. He uses base 44. Okay. First he goes to Claude and he has Claude break up of something he wants to build into specific steps for base 44. So he has a base. Okay. So what's base 44? Base 44 is like a stronger, reflet lovable. Okay. Cool. Got it. You can do more. It's more powerful. Cool. Basically. And so he's doing all sorts of crazy things over on base 44 and something that I think was just so brilliant that he built recently for his sales staff. He's CEO of apartment SEO. So he has a team that talks to the front apartments and renters and whatever, but he connected this real time voice monitoring model that was open source for emotion, for context, for wow. But he has specifically crafted it to evaluate a live sales call based on their team's specific goals. And in real time, Ross, you're going to love this in real time. But like tell his sales guys, hey, you're talking too much. No way. That's what it is. Yes. Or it will say, Cheat, like ask this follow up or like it just supports them in real time on specific things. That is like the future of this stuff in my opinion is like taking really powerful models, but customizing them and fine to them for your specific industry, for your specific tasks. That's where he's kind of workflows really take off. And they don't want to be this complicated, you know, a lot of people are helping simple stuff. And that is incredibly valuable and saving them 20 minutes a day, 45 minutes a day. That stuff adds up to. And so with students or people that haven't used this tech before, I'm always really urging them to start super silly small. And it's not so do I need to be an expert to join Orange Lives like if I'm somebody who's fresh off of the streets and first, I mean my first job, I just graduated. But I know that in this market where the jobs are all saying I need AI, if I want to have a job in the future, like am I the right person to apply for Orange Lives or is this from people who are a little bit more senior and experts? No. So it's really for, it's specifically kind of designed right now for marketers early in their career that are interested in learning more about it. Love the examples. Our marketing related. But anyone can join, we do see a higher engagement and success rate of people who experience the course first and then you actually get free access to Orange Lives for a year. And where would they find the course the course needs? So there's kind of two funnels right now you can sign up. I feel like Maven has a better overview because they've captured all the data for years now. I'm on my 11th cohort there. I know it's crazy. And then there's also a sign up on Orange Lives. So yeah, that's it. It's like a fun way to get people through, but you don't, it's not necessary. We're trying to develop it so it can just be easy. Nice. I'll add some links to the show notes as well. So folks can check it out. One question. Marketing against the grain hosted by Kip Bodner and Karen Flanagan is brought to you by the HubSwap podcast network, the audio destination for business professionals. If you want to know what's happening in marketing, what's ahead or in how you can actually lead the way as a marketing professional, this is a podcast for you. The host Kip and Karen, they share their marketing expertise. They give unfiltered details, the truth, and they help you better understand how you can do more with less. Most recently I tapped into an episode that where they had someone on who talked about how they run a zero person content marketing agency as themselves, as somebody who runs a content agency. I got a lot of insights into this episode that I could apply to the business that I run. Folks, if you are not listening to this podcast, it's one that you do not want to miss. They have recently been dropping a ton of great insights around AI, how you can use things like Claude and more. Listen to marketing against the grain wherever you get your podcasts. We're talking lots about the marketing side. Humans are like onions. That's a shrek quote, but we also have the personal layer. Have you seen or have you used AI personally and been able to unlock some cool opportunities, unlocks from a life efficiencies? Are you using it in that way or are you exclusively using this for work? I use it for all sorts of things, Ross. I've not ever shared this story before. This is so perfect. Before the call, I gave you a little tour of what's here. You're not going to believe this. We were kind of in a bidding war for this house. My husband, I loved it. We had actually never seen it. But we were sitting there and I go, "Should we write them a quick letter?" He goes, "Yeah, we should do it." I didn't have much time. It was kind of really rushed. I had chat GPT helped me. I fed all this information about what we liked, the trees, all this stuff. We sent in this letter, it got in just a little time in the sellers. It's how they made their decision to sell it to us. I feel like we got this from a chat GPT letter. That's crazy. That is crazy. That is crazy. I use it in all sorts of little ways like that. When it first came out, there was a company that owed me quite a bit of money and they were just begging their feet. My voice, my writing style was not working. It wasn't working. It wasn't working. Working for it was not working. I went to chat GPT and I said, "I need you to respond to this as if I have counseled a lawyer or you are my lawyer." I love it. Get this done. I was paid within 30 minutes. That's wild. This is like, we need power to the people. We need to, we need to, in libraries everywhere, people not needing medical bills. True. Trying to get food stamps. That's where this stuff needs to be. I think about all the time. So much opportunity. I think that's one of the things that's always interesting. It's great for the marketing and stuff, but you can do so much as well on the personal side. If you can arm and educate more people about it. about the tools that can be a life changer for many. I think that's cool. All right, let's put the capital that's had back on for a second. So SEO, AEO, GEO, LLM, SEO, AI, SEO, EI, EI, EI, whatever you want to call it, what does the future look like for this space? - What does the future look like for this space from which perspective? - Let's go with all of them. - Oh gosh, okay. I'm really sort of, I want to say fearful, but wary that a lot of like Google's AI mode patent will come to fruition because it's kind of slowly already has where they alluded to just not showing organic results, right? It will just look and resemble kind of that chat GPT interface and there might be like a small text of like expand, you know, here for organic or whatever. I feel like that is-- - That's in the patents. - Yeah, I can say I will link after this. It's insane. - Love to check it out. - It's insane. - Yeah. - So you're saying the 10 blue links are gone? - Yeah. - Wow, fascinating. - Yeah, I think they're gonna look a lot different. They could just look like this little AI mode, kind of like what you're talking about. - What does that mean for SEOs? So now I'm putting it in that context. Like when that type of a shift takes place, is SEO dead? - I don't think it's dead, I really don't, because Google, for example-- - It's on my screen. - And AI answers needs this pipeline. They have to figure that out. This is kind of, I think this is a Google problem to solve the blip. How are you gonna keep this kind of content lifecycle alive when you're reaching off of everyone's content, you deliver the answer. You know, you said you were gonna organize the world's information and now you're kind of authoring it. That's a huge jump. That's a huge, huge jump. And there's a lot of loose ends that still have to be figured out in order for this all to pan out in that way. And they also have a ton to lose, you know? - Right, right. - A lot of trust built up in Google that could be lost through this process. So they have to be careful. I don't think SEOs dead. I think the pie's gonna get a lot smaller. I think we're not gonna go back to pre-AIO traffic levels. Like that doesn't exist. The can is open, if you will. This is out and not going back in. But I think marketers are gonna have to get a whole hell of a lot savier in off-site efforts. They're gonna have to get a lot better at social and meeting the audience where they are off-site in those efforts and really, really adding some of that zero click value, you know? - Right. - And then I think from like a company perspective, you know, I see this is a huge infrastructure play for Google. - Right. - They are bringing Gemini into platforms everywhere. You calendar, Gmail, sheets, maps now. Like the friction is gonna continue to go down for us to use and leverage this in really like efficient, helpful ways. And I think that's Google's play long term. And then they'll, you know, we're the training data right now. - True. - And so at a certain point in time, we've gained all of these benefits using the technology in these platforms in these ways for efficiency gains for you name it. And then my guess is they're likely to pull that back and say, okay, now you have to pay X a month to retain this. After they've figured it all out with us again, and they're training data. So that's super interesting. But yeah, again, I mean, from a marketing perspective, it's getting a lot better in offsite. And understanding like where your customers are having these conversations and where they're taking place. - What do you think happens to all the tools, the platforms, the tracking tools, et cetera? Like do you think we get to a world where everyone has their own and they just build it by code style like you described? - No, it's a good question. I feel like that's definitely, you know, a trend. I wonder like long term, is that helpful, you know? In what ways are we, I think in this existence, just the AI industry in general, but really exists in what we're talking about. Like there's a major measurement problem. Don't have data on how people are using or searching these. We have no visibility into the background queries, right? We might be able to get a little bit here and there, depending on if they're using particular APIs or, you know, things recent, like, want to go to grounding queries, but that's a stretch too. We don't even know where those came from. So there's huge, huge disconnects in the metrics that we're receiving and trying to make sense of and it's just the mess quite frankly. And so I think we're almost better off evaluating things from what's driving traffic today, what's currently - What? - And doubling down on that. And maybe experimenting with the new platform for three months. Does it work? Yes or no, pivot accordingly. And I think we don't do that enough, you know? Like that's not something that we're sort of like trained or pot to do in that way. And again, you were ahead of this forever with the Reddit stuff and the social stuff. - Appreciate it. - So I think that, yeah, that will continue to be huge throughout this. - When you think of some of the players, we've got Google, ChatGPT, Perplexity, Claw, Microsoft, all of these things going on. Does anyone stand out to you as going down a path that might not necessarily lead to the best outcome for civilization people in general? Like where do you see gaps in the way that this whole world is going right now? - Yeah, I mean, I've got major concerns about, you know, the people making decisions about major AI tech at the moment are in the hands of a very few set of people in positions of power that are not thinking bigger picture or looking out for the best interest of others. A lot of them, quite frankly, are like plain pretend tech gods, you know, and they're using and leveraging AI hype to their advantage to even further establish these positions of power to accomplish all these things, to accomplish more funding, more customers. They have found a content and attention flywheel that works really well in open AI, does this more nefarious and better than anyone where they will say and announce crazy stuff, crazy stuff that immediately gets picked up by all this media attention. You, everyone, all of us are talking about users are talking about their game signups, nothing comes out of it. - Right, right. - And we have announced dozens of things at this point that do not come online, where is it, where is it? So it's just, no one's being held accountable, which was incredibly frustrating, and I was just talking to someone this morning, like I think us as users have a lot more power collectively than we think. A lot of these people and companies and positions of power, they're strategically like PR positioning this stuff as evolutionary, as inevitable. It's not, it's a few people. In small tiny rooms, making these decisions for all of us. And it doesn't, it's crazy. That does not have to be a thing, and we should be pushing back a lot more than we are today, and that accountability is so huge. Yeah, it's, - Is there specific things that you think there should be more pushback against that is not happening today in the market? - Yeah, I mean, what the hell, like these companies don't get to live behind, be protected behind this experimental label. Like who, you know, who's accountable when Chattcheap T convinces a kid to kill himself? Or, you know, gives bad medical information. Or, you know, there's financial advice, like all of these things are so incredibly dangerous, and the sick, a fan, the nature of the systems, double down on that, right? They're trying to please you, and they are reinforcement, they're getting better by RLHF, reinforcement learning through human feedback, which, just to please us. It's like, it's your biasing the system. And so instead of like setting us on the right track, that we, it's so easy to lead these things, in whatever direction we want. And that's why you see all these AI girlfriends and boyfriend, you know, they tell us what we want to hear. It's like, it's wild, it's wild. So that's scary. - Yeah. One of the cool things is that I think it was probably like six years ago when I first heard you talk about the bias within the LLM's and what's going to come. And it's coming slash it's here, like you were early with a lot of it. Do you think there's a path ahead where the power structures shift, where the people's voice can influence this? Or do you think it's too far gone? Like where do you think we are? - So I think a lot of the nefarious efforts behind this technology go completely undetected, completely undetected. And this is by design. This is completely designed. There's an acronym called Test Grill. That was invented by Tim Neat, Gibru, incredible, accessible paper on the topic. She also has a video on YouTube. And it breaks down like this transhumanism, this altruistic, you know, AI efforts. And they're under the veil of altruism and like doing good, but at the end of the day, it's a lot of, again, these guys playing techno gods. And we can turn ourselves into computers and live forever and intelligence is the most important thing we need to all point to and breed for. >> For sure. >> And it goes back to really gross, scary, racist, eugenics practices, which unfortunately is a huge undercurrent of AI that a lot of people don't know about. IQ test and IQ came from this eugenics practice to really exclude groups of people. But this is still continuing today under new names. >> Right. >> Literally, right under our noses. There's an institute in London that does a lot of transhuman stuff that is still active. And their history is disturbing and it's scary. But a lot of the Peter Teals of the world and these people kind of moving all the per strings, this is what they really really believe. And it's not talked about enough. Again, it sneaks under all of us because it sounds positive and then you pull a little bit deeper and I like, wait, what? So general awareness about those things would be incredibly valuable for us as a society and moving forward. And that I really hope to continue shedding light on that. >> Do you think curriculums will start to kind of educate around this stuff? Is that the play that we might see in the future? Is it kind of like in the hands of the users and the people to kind of even teach their own and educate their friends? Is it like would the curriculum ever be influenced? Or are we talking like five years, ten years out from now before? >> These big AI companies, they already have their hands in the curriculum. >> True. >> That's true, yeah? >> That is a huge push, huge push. Open AI was all over that three years ago. So unfortunately, they even at that level are getting across their own messages. >> Your messaging, yeah. >> And they don't want to reveal how the hot dog is made. And I think that's important part, people need to be learning about is what goes into building these systems, how offshore workers are being paid next to nothing to consume the most horrific things you've ever imagined in your life. They go on to have lots of mental problems right. So there's just so many gross things that get contracted out to build these models. And then again, the models, the end of the day, they still magnify the biases. They still magnify all these issues. They're still incredibly problematic, but they're just a little bit more consumable for public. >> So it's not just a complete computer on the back end of these things. There's a reviewer, a human somewhere in the world who might be seeing things, what do you mean? >> So a big part of, there's always been a couple bottlenecks in AI, right? It used to be compute. Now we have lots of compute power that's not a problem. It used to be training data. That was a huge problem. And then the advent of ImageNet and all of the hand labeled images. Lots of humans have had to label videos, images, or these training purposes. I mean, think moderation on social networks and how that works. There's an incredible documentary called The Cleaners. >> Okay. >> And just the trailer alone, you get across what I'm referring to of people having to watch this horrific information and make kind of editorial decisions about we should shut down this live over here. But this is all training data. This all gets fed into systems. And so we're learning more and more about these facilities and about these people that undergo horrific treatment. They make nothing. They're stuck in this place watching absolutely awful, gory things. And these big tech companies, they know it. They know that that is what powers a bunch of their systems. But it's through multi-contractors. So they're kind of trying to keep their hands off. >> Hands off, it's not their employees. >> Yep. >> Absolutely wild. >> That is wild. I want to shift gears a little bit. So there's a whole wave of conversation around agentech, this, agentech, that, agent, this. What is, what does this whole move towards agentech things mean? Like what are they talking about? Are they building little agents? What is an agent? Could you give us a breakdown on how folks should be thinking of agentech, anything? Or is it just smoke and mirrors? >> Yeah, great question. And this is evolved recently. So, you asked me this, I don't know, a year ago, smoke and mirrors. It's agents, agents a year ago was the new buzzword. And they were literally not doing anything that we couldn't write a Python program for. Like what are we doing? What the hell? So, that was very underwhelming and a lot of us were confused by that word and how it is exploded. Today they are starting to incorporate really interesting capabilities that we really haven't seen before. So for example, Claude Crone, insane. >> Right. >> Insane. >> Right. >> You can literally set up like a Google Doc, having it do very specific things for you. And it will go through your instructions and open up new tabs. >> Wow. >> And you can watch it do this. And the way that it does it is genius. Like I, my head exploded the first time I saw this because I have always thought, you know, these are language models. You have to feed it the whole website code and it has to dig through the JavaScript and voila. No. >> Yes. >> This agent, specifically, is opening new tabs. It's a waiting for the page to load and it's just taking a screenshot. How genius? >> No. >> Because it wants to see client side. You know, I don't eat mess and guess with the code. I just want to see it. So they're using image recognition models. >> Right. >> That's what I'm talking about. >> Of these language models to do insane things, to see what's on the screen, to see what it is that you're talking about, okay, what button would that be, what form field I can fill that out. And then they do it. I mean, that, that, we've never seen this before. And what we do it all through your Chrome and I'd recommend, you know, set up a fresh Chrome account so it's not connected. >> Yeah. >> All your stuff just in case. >> That's smart move. >> Yeah. >> You did, good advice right there. >> Yeah. >> I'm really extra careful with these because it's funny like the, that this kind of like agent stuff, it's exciting. People are going like all in on it. But at the end of the day, we're still struggling to design adversarial networks that can identify bad actors or bad prompts from real ones. >> Right. >> The image models are not fact machine. They're not fact models. >> True. >> Real world information models, right? They're just trained on all of the internet's text. It's very different. So we're going to have to figure out like better ways of navigating this, which I'm sure, you know, people are working on, but very, very interesting. And then it's like, well, what happens to the internet is everything a mark down file for these agents, you know, really, really interesting. Watch your thoughts on the future of browsing experience. Like, when I think of AI, and I think of myself as a bit of a nerd, so I know I'm early with things and the world won't think the way that I think is normal. But I would 100% take a picture of my closet and say, hey, Chachi Petit, hey, Claude, hey perplexity, this is how I dress. Go find me. All of the things that I should be wearing in the winter, buy me some new sweaters, buy me some new shirts, go on eBay, go on Amazon, I'm giving you a two grand budget to go buy these things. And I don't need to see anything. I'm accepting you to buy them, just make sure they're in a large, and I'm chilling. And then I'll get them at my house, but I'm not able to have, I'm not going to be influenced by search as a human. My LLM might be. I'm not going to be influenced by an ad that remarquets me because I'm not browsing around to different sites and reading blogs and stuff like that. What are your thoughts on the future of the, am I too far? Am I like, yeah, Ross, people will do that in like 20 years or 30 years, but most people will never do that. Am I off or what's your assessment on that type of thinking? Am I too far gone? Is it rooted in reality? Would you let your wardrobe be bought by LLM or am I crazy? I love where your head's at, Ross. I think this is a good and valuable thought experiment for sure. And I love the direction. Because I think lots of people would kind of consider something like this. But knowing what I know about neural networks and these models. You're saying I'm going to get a whack outfit? I'm think you might get one item that costs $2,000. You have to think of it as a text. At least a tiny little bit more guidance, I think. Or I do wonder at the point where advertisements start to happen, which I already have on chat GPT in OpenA. True. How is that influencing things? It's good. There was a crazy, it wasn't a leak, but I was a psycho and I zoomed in and chopped up the video and got all the still frames of the whiteboard of a Google. Sergey Brin came to this hackathon at Google. And it was this guy wearing a shirt of like a top. I think that's how you can find the video. It's just stupid, but it was him and then it was over. I don't know. They thought it was funny. And then it's over to Sergei and the whiteboard was so interesting. And I spent hours assuming, because I'm like, what was so important? This was when ChatGPT came out. I was like, what was so important? He came in on a day off. True. And what was behind him was this thinking system about contextual information about the user. So what do you know about the user? They're searching for party planning XYZ. It was a birthday party of some event. But then it had these clouds of thought of items to purchase all at once from different suppliers for a party. Maybe this was on Walmart. Like table blah blah, utensil, whatever. Had a bunch of items. That made sense for this woman who was querying about, again, it was like some sort of party. I just thought that was so, so interesting where they're trying to make that leap and kind of predict what you might want to purchase based on these types of queries. And they've been doing this for a long time. And collecting all this information. And I also remember I was at a hackathon in Seattle for Google Voice. And we got, we were some of the first users to see, what was it called? It was like a, the backend for developers to use it and build apps. It was like this chat something thing. But anyways, I was like digging through the documentation for this API. And I asked the instructor, I was like, it says here like brand recognition because it was connected to Google Home and the camera. And I was like, are we able to recognize the brands in someone's home and they didn't answer me. But I've never forgotten that. I've always been like, that was crazy. And then I had a Google Home in Seattle. And I remember like opening the serial box. Like I would pay attention to like what? I was wearing. It's like, it's, you know, you know, there's just so much information that we can gather now about people and buying habits. And so maybe it's more to your point Ross. Maybe it collects all of this. It knows your buying habits, what styles you go, what are your size. I don't know. And it can make you a great job. That's wild. Yeah. That is wild. So you've had a fascinating career. And some of the folks listening to this are going to want to be Brittany when they grow up. How would, what advice would you give to someone who's in the early few years of their career? Get on a trajectory like you've had. What advice would you give them to be successful in this industry and in marketing and AI, etc? I think Tinkerine has literally driven most of my career. Like having a very competitive, just like nature. I always want to like figure things out. I like to take it as a part, you know. And so SEO is so good for me early on because I was like, I was a little sponge. I just wanted to figure out how to compete and how to rank for this and testing things that was so fun. And then once I got a taste of like, oh my gosh, machine learning, neural networks, K nearest neighbor, like linear regression, I never went back. I was like, this is what I think you're going. And so I think that curiosity and I think building silly things like I see Simone Gertz does this really well on her on her YouTube where she builds an engineer's robotic things. And I think that serves people in their career so well because it's like portfolio items you can reference. You can hold your confidence. You're testing something really silly, right? If it breaks, it's not going to, you know, upstart. Carbon issue. Yeah. So I've tested so many silly ridiculous models or cron jobs and Python programs on really ridiculous things that solve stupid things, stupid problems. And I think that is kind of the sweet spot where you again, learn by doing. And so breaking things is great, you know, like right. And now if you're early in your career and you're curious about this stuff, the fact that you have a 24/7 supports system with something like quad, you kind of have no excuse, you know, if you've ever been curious about building something about a particular, you know, area of industry or whatnot, you can really get savvy in terms of learning about it, but then also executing tools to kind of continue to absorb it. You know, Claude could build you like a Reddit scraper that evaluates, you know, who the top companies are in conversations that include this keyword. What kind of sentiment are around that? What topics? Questions. I mean, you can build incredible things. And I think there's so much data at our fingertips today that with the admin of AI and tools like Claude, especially as content marketers and this is something I love chatting with you about, is like, what can we be doing with that data? You know? And like, Kristen Tinsky type stuff where they were taking like the DUI, creating that map and doing really, really brilliant things. And that's how your brain works. And I think, I think the most powerful kind of up and coming marketers are going to be able to wield that in really valuable ways and do it in a way where we talk about brand mentions or the new backlinks with all of this AI search. Yeah. But they're going to figure out ways to engineer these campaigns where they get other people to do the talking for them. They get other people to do the brand mentions for them. Right. So it's like, double down on experiments, create things that are fun, quirky, meme related, just have some fun on the internet and tinker and build. Apply that same thinking to taking data and building things and trying to create interesting and unique stories. See what type of buzz and hype you can get. Do a deep dive, deep analysis into trends associated with elephants in Africa versus Asia and understand their biometrics or whatever, try to save the elephants, create some type of AI to help you with that. And then talk about that and get some buzz and that's kind of the playbook. Am I going down the right path? That's absolutely the playbook. Yeah. That's cool. I like it. All your interests, you know? And then on top of that, if you're not sure how to build these things, you can go into Orange Lives and it will give you the multi-pad. Yeah. You're right on that. That's for your support staff. Exactly. I like it. That's awesome. Here's another question. So what's the big bet that you are making right now for yourself on the industry or the industry? What's the big bet? Like when you look at, all right, right now over the next 12 months, I'm going to put my chips in on this. This is where I'm spending time because I believe this is very important. This is going to be some of the most important work that I do in my life. What is that today? Look like for you. That's such a good question. I think it really revolves around AI literacy and empowering people with AI skills and knowledge and tools to get started. Again, by design, the AI industry is really, really hard to break into. Even if I like an informational standpoint, a lot of the courses and books available are very tech, mathematical, heavy, which is fine. But for a marketer, you don't need to know Transformers' architecture to use an LLM in a value-driven way. So, it's about finding that balance of knowing enough to know what the capabilities and limitations are, but then also knowing how to best apply it in ways that actually make a difference for you. I love that. If folks want to learn more about you, your work, etc. Where should they go? LinkedIn. I love it. I think you got me on this LinkedIn kick. Honestly, a couple years ago. I love it. I love it. It's great. Yeah. So, trying to share more stuff there and connect with more people there. That's awesome. Alright. So, marketers are listening to this episode. If there is nothing else that they can do, what's the action that they need to take in the next 48 hours that they can move with and go from here with? Honestly, to show up. I think showing up is the hardest part. I tell orange labs, members and students, give yourself five minutes a day. Show up for five minutes. Try it. Just test and engage with one of these models in a new and different way to learn something, to apply something. And I think that truly is the hardest first step. It's just setting aside the time to experiment and be easy on yourself. You don't have to create this perfect end result. I always challenge people if you want to solve something with AI, make a super ugly V1. That's your goal. Now, do not try to make it look nice or have all the bells and whistles. Ugly V1 is your goal. It's just like be easier on yourself and show up and just create the time. I love it. Brittany, thank you so much for showing up not only for this podcast, but showing up for the industry all the time, especially for all of the folks over the years who you've impacted. It's been amazing to watch. It's been amazing to see your journey. I am so grateful to be your friend and love seeing what you do for the industry, for the culture, for the market at large. So, thanks you and keep doing your thing.