#308: AI Changes Everything! Analytical Fundamentals Still Apply!

Welcome to the Analytics Power Hour, analytics topics covered conversationally and sometimes with explicit language. Hi everyone, welcome to the Analytics Power Hour and this is episode 308. I was having a conversation with coworker the other day and they mentioned that with everything happening in AI, sometimes it feels like the skills and experience they had no longer applied. AI is definitely disruptive, but do we have to abandon the fundamental analytics skills we've got just because AI can do a lot of the work and as we all adopt AI and our businesses, how should leaders think about measuring its impact? It feels challenging, maybe we've got some skills that could help along the way. Alright, well let's talk about it. First let me introduce my co-host Julie Hoyer, how are you doing? I am doing great, I'm very excited to chat about this today. Me too. And Tim Wilson, I know you're chopping at the bits for this convo. I just, I'm sending an agent in my place so this is the last time he's here as a human person. That's right. Muse AI, the latest and greatest, I don't have Michael Helblings and I'm so excited. Our guest today is Daniella Matino, she is a marketing science senior strategist at LinkedIn. Prior to that, she's held marketing and data science roles at companies like Snap, Metta and PepsiCo. And today she is our guest. Welcome to the show Daniella. Thank you. What a great intro. Ah well. AI helped. AI helped. No, it did not. I wrote that myself. AI does not touch the intro. We tried writing with AI and it didn't sound right. Okay. Daniella, to get this conversation started and thanks for, we're excited to have you. Maybe just talk a little bit about what you're seeing in terms of how AI is affecting analytics, it rolls and in the orgs that you're a part of LinkedIn, especially or other places. Where are you seeing it? Have impact. Impact people's roles and changing things. Yeah, happy to talk about overall what I'm seeing in the different industry, in the different industries and organizations. Overall, as Tim was saying, speaking a lot to agents, I feel like this is something that we weren't doing at all a few years ago, starting to do any sort of work asking AI, what do you think about these or give me some prompts has completely changed when it comes to the more analytical technical part of of the things. A lot of the coding that I was doing in the past would be from scratch. These days, we are starting with agents or anything that has sort of built in and we are coming in and in handsening it or making it a little bit faster. So overall, a lot of efficiency, a lot of expectations on delivering more faster. That's what I'm seeing. As I talk to colleagues, friends, is a lot about doing more in less time just because we have these army of agents that are helping us delivering a lot of other things that we couldn't do in a short period of time before. So can I ask about the more because I was with my oldest son who's a software engineer recently and he's developing so like more for a software engineer is he's going through tickets and tasks and he's evolved the way he works and he is prompting and having code written and then goes through a code review and code gets pushed out. So it's like more features and bug fixes when it comes to analytics, like something that and I also like literally today experienced this, like the more being produced by analysts and I've seen this happen in other places, it's like, oh, I can just generate a 29 page report. I can just generate more or we were at a conference earlier this year and somebody was talking about, look, I've now managed to, it used to be so ponderous for me to produce this weekly report but now I can do it, I can do 10 weekly reports and they're 10 times as long and I mean there's the macro view of people saying I'm people, everything's but writing stuff that's longer because they're using AI and that feels like more of the analog to the analyst challenge is it's like, well, it feels better because it's more output and it's more thorough and I explored more avenues, somebody's got to receive that and process it. So I'm curious if you've got thoughts on on that. Yes, I have a lot of thoughts. I have this theory that no one is reading anything anymore because we are delivering to your point team these huge reports that are so thorough but no one has the time to actually go through it. So when we talk about delivering more and to your point on the analytics and the analyst role is really to cutting the chase meaning to cut everything that is not needed. Working a lot with sales in my past 10 years of roles, sales needs something that is very direct, a 10 page report is never going to make it through. So my role in any of the organization is as I use AI more and more is leverage it to produce it overall and then I come in and really understand what are the things, the 3, 4, 5 things, the 3, 4, 5 numbers that are going to be used for punching for sales for clients. A very interesting experience that I had just recently I was working on a hackathon with some ladies and we were basically using Glot to do a lot of the coding. We go through the coding review sessions and we get to the point what is happening like it did all this code but it doesn't make any sense. We can build a dashboard in one hour but to your point team we are looking at numbers well, these assumptions they don't make sense. So baby sitting the code that is being produced, the numbers that are being extrapolated by from the data that we use in the initial stages is a very, very critical and important point. That's what I'm coming to realize. Moving back a little bit to maybe how this is changing people's roles, we've talked about this a little bit previously but the idea of instead of staring at the blank screen and putting in the assumptions in your code that you guys felt was right, having to like babysit the AI as you call it and the coding and looking at code and then having to like re-engineer it and decipher it and then find the logic to like edit it and shift it. We've talked before how that feels like very different like centers of your brain, right? Different skills, it feels different when you're doing the work. Do you feel like that aspect of utilizing AI has really shifted people's roles or it's kind of just a byproduct and like an obstacle? Oh yeah, 100%. I find myself way more of a storyteller today than I was before because the hard skills of coding and I went through interview process not too long ago and I did maybe a third of technical coding challenges that I used to do when I was interviewing before and a lot of the focus is being on case studying, tell me the story behind the data, tell me what's happening, tell me how is this impacting the business. So a lot of analysts that perhaps in the past would do very well because they were strong in SQL, they were strong in Python. These days they are finding this situation that oh now I have to kind of tell the numbers to people and I feel like it's shifting a lot towards facing other people and telling the story. Things that agents are still not doing but maybe in the future they will be doing. There was something else you said, I don't want to put words in your mouth so I want to put words in your mouth and see whether it was accurate because you started to say that there's more, I think you said something along the lines of there's more like facing the prompt and like doing ideation up front with AI, did I hear that right? I don't know if I was wanting to hear you say like no there's like upfront thought that AI can help you with before you dive into having the agents go and crank on data. Yeah, so I usually like to work in, I would say two ways and yes, to your point team. Sometimes in the past I would be working on a project and had a blank sheet of paper and so no ideas talking about content and talking about code. These days I can prompt AI to start sort of the wheel and start things running and so that's one way of working. One thing that I'm finding more effective this way is being very concrete. into one, two, three ideas that I give to AI before I start on a project. And I say this, after a year and a half sort of using AI, constantly I'm seeing that there is so much hallucination. There is so much content that it can produce for me that right now I'm like, well, I have to be very precise in what I need. So I don't get the 10 page report instead of a one page report. But I guess is there a flip side of that that is just using the AI as a-- and I feel like this seems to be coming up in more and more conversations of like, well, this is like a non-judgmental, wildly sycophantic, somewhat knowledgeable sounding board to just be asking it like, well, could it be this or could it be that? And guiding it to say, but give me-- giving the reasons is a bad idea. I found myself doing that where it's like, oh, this sounds great. I'm like, OK, tell me all the reasons this is terrible. And in the specific case, it went through and said, well, these are where the holes could be blown in it. And then it ended by saying, but you made me say something negative about your thought process. I don't really buy into those. I was like, OK, well, that seemed like a useful-- it sort of gave me to consider that added perspective, because human nature is to get caught on one path and saying, OK, I've got something that's not going to get annoyed with me, continually asking you to force me off that path to broaden my thinking. Michael, I believe November 10 may be a significant holiday. Ooh, I'm afraid to ask. What's tape really, which is basically GTM nerd Christmas school, that might not be the official description, but tape's first virtual event does bring together technical marketers, analysts, and service-side tracking practitioners for 18 live sessions, including previous podcast guests like Seymour Hava and Julianna Jackson, which is already a pretty convincing argument. Yeah, they've also got speakers from Google, Jellyfish, and others. And you can switch between two tracks, covering hands-on product implementation and actionable strategy. State and Seymour Hava will be revealing their roadmap for a new generation of Google tag-manager templates developed with community input. Yeah, and the whole thing is online and completely free. So clear your calendar for November 10. Click the link in the show description and attend State Relay for free on November 10. Michael, everyone says AI needs more context, agreed. And what I do is I just upload the company handbook, six strategy decks, 14 called transcripts, a glossary, and a carol from finance. Please don't upload carol. But she knows things Tim. Well, Prism's context notes give you a slightly better approach. You can feed the agent business knowledge, playbooks, transcripts, glossaries, PDFs, whatever helps to understand how your organization actually works. OK, and not to be disagreeable, but six months later, half of that is wrong. Disagreeables kind of your thing. But that's the good part. In Prism, context notes can refresh instantly or on a schedule. Oh, so the AI doesn't keep confidently applying last year's rules like a middle manager who missed the reorg? Exactly. You can even give Prism instructions for how each note should be updated. Oh, that's big, because stale context is just bad data wearing a tiny knowledge management hat. A knowledge management hat that doesn't fit. And since that context is managed centrally, your users get better answers without everyone rebuilding the same background knowledge in their own AI chats. Yeah, let's copy paste. Let's tribal lower, fewer conversations that begin. OK, first you need to understand something weird about our business. Click the link in the show description and sign up for the Ask Why wait list. Use code APH, and they'll move you right to the top of that list. [MUSIC PLAYING] Keep the context fresh, and Carol can stay in finance. Yeah, that's a great point. And also the way I use it, as I produce something for content, for code, for work, a lot of the perspective, I want to at the end of producing something that I'm confident about is, what are some of the holes? What are some of the things that I might get asked about? So I can prepare up front for it. So that's one thing that I have been trying to use it more. So I can feel stronger as I present, as I deliver, because I have thought through it with AI. [LAUGHTER] Yeah, I've noticed sometimes in working with analysts on things, you have sort of a disconnect of between the person and what the AI produced. And I am really adamant with our team on, you have to be out in front, not the AI out in front. Because if someone is in a meeting asking a question, they're not asking the AI, they're asking you. And you need to be able to answer. You can't just be like, OK, let me put your question into the AI, and then find out what the answer is. And I feel like people are getting that little backwards right now, sometimes because they're doing so much that are working AI, they're kind of letting the details slip past them a little, and then they don't know the details of what they're talking about. So it's a tricky thing to balance, because AI is helping us be efficient, it's writing a ton of SQL, and running queries, and generating reports. But then if you're not looking at it yourself and being really careful, then you're sort of at the cart and the horse get flipped around, I think, sometimes. Oh, yeah, 100%. I mean, all the-- I have this theory that a PhD in philosophy these days is going to be so valuable, because that critical thinking that AI is sort of in a way taking away from us, because we are asking AI to think for us rather than to Tim's point, OK, tell me what I didn't think about. So big believer that that critical thinking piece is somewhat getting lost in some roles, in some conversations. I see AI, and I was actually having a conversation earlier for another hackathon. See AI a lot on the automation, tasking sort of situations that we encounter, book me a meeting for days, or purchase these for me, that I purchase on a recurrent basis. That's where I see that there is a lot of potential, building agents that can really do recurrent tasks for us. I see it with a lot of potential, not only on the workforce, but also for our personal lives. Yeah. You mentioning critical thinking being a skill that it's taking from us and people probably need to be a lot more careful about keeping that as something they're doing ahead of the prompting, like you were saying, are there other big buckets or big skills that you would say are fundamental to analytics that you wouldn't sub in AI to do for you yet? Understanding to business. Starting as I work closely with clients, with advertisers, focusing on what AI cannot guess. Because again, we can all, at this point, right, SQL Python are with AI, but understanding the business, what is the client journey that each company is optimizing for is something that perhaps a nuance that AI is not going to be able to give us. Stakeholder management as well is something that, yeah, I can write emails for us, but is the type of skills that I'm finding more and more interesting? The fact that it's just so important, even more important than it was before. It does like a deep understanding of the data, even though we can use AI to write the SQL or the Python in R, is there a part of it? I worry about the risk that just like if you just turn somebody loose with a traditional BI platform five years ago, they could just start willingly dragging and dropping stuff and interpreting things, but they didn't really understand how leads related to contacts, related to opportunities. And I think there are a lot of people trying to solve for this, but it's the same. Oh, I generally understand the data, but do I really, really understand the data? Do I understand the ramifications of how these systems talk to each other that AI is not going to magically generate a joinable link between two data sets that don't have that? Like, is there still a need for that? Because that goes with working with the stakeholders to say, how are we going to problem solve? I need to understand the data as well. 100% and I see it a lot on the fancy shadow. outputs that we can get from ChattuPT Cloud, all the different platforms. And to your point, when the underlying data is not in good quality, when the governance, the integrity, just the quality is not passing the minimums, then the output we're going to get is not going to be great. Yes, we have a good looking dashboard that is not really usable because the data that is displaying is incorrect or to your point, KPIs, they don't quite make sense because perhaps certain tables are not displaying the data that they were supposed to, naming conventions were not implemented up front. That's a dirty word that no one wants to do basically. Though, if we want to continue building AI workflows, it's the type of work that needs to be done before. Michael, you'd played around with some called skills around different types of tasks. Part of that, where I'm still trying to struggling is struggling with our Python or SQL and really having to grapple with syntax at the same time, I'm trying to incrementally build up the data that I want to look at. That can be annoying, but it also sort of forces me to kind of look at the data and understand the data and, you know, periodically do some checks to say these aggregations make sense, which some of that feels like it can be forced steps by the AI to say, before you do any of this, you need to do these five steps of EDA and raise a big flag if there's something that looks off, you know, in that. But I can't figure out, it feels like you're continuing to chase this bouncing ball that is preventing us from the critical thinking or understanding. I don't know, I'm not articulating this well at all. I don't know if it has happened to you all using, I was using Claude a few weeks ago for a personal project and we build a whole pipeline, we model and then the output was terrible because I wasn't babysitting it at all. And I mentioned to Claude, okay, this is this is wrong, this doesn't make any sense. And he gave me an attitude like, oh, but I build this for the past hour and you didn't tell me anything. So that was on you. I'm trying, I'm seeing it more and more. Oh, but you did not see these either. So that's why I build the wrong thing. Oh, man, this has got too much train from developers like, what do you mean? You didn't give me that requirement. That's on you, on you, the human. It was making me think to him that exact piece of like, I feel like analysts classically have been able to go talk to the business like you were saying Daniela and understand like, what are they really trying to ask? And then it's looking at the data that exists and knowing the nuance of what assumptions are truly on each of these, you know, values, data points, whatever, because of how it's collected. And there's this idea of what's missing in the data that there's no way AI could pick that up, right? Like if they don't understand that, you know, real classic, like go to your website and QA it and we're not getting data consistently firing on a certain user activity, right? That is like critical to answering this business question. I don't know if AI would easily pick that up. I'm sure there's agents and things to do it, but like more broadly, just that concept of like what is missing between like you were saying business needs and all the technical data setup to actually get it to an answer. Like that is a big jump that I feel like we've talked about for years and years and years about the fallacy of like just collect more data, right? Tim, like it's another flavor of that. I feel like AI has made people kind of forget that again. And now it's coming more top of mind for people as they're like, oh, it's hallucinating. Oh, it can't do this. It can't do that. And it's like, oh, well, because your data wasn't ready. Like you were saying it was the table stakes still weren't taking care of, but I do feel like for analysts, it's the ongoing thing of being able to see what can AI not see or know in the data. And like that that jump I feel like is still super critical. Is that the discipline of like whether it's a context layer or whether it's making sure it's in memory of the analyst saying every time one of these hiccups or gotchas or don't do this, the same thing where I got the 29 page dump that was not a narrative. There was a separate nine page like decoder document that talked about, oh, and you can't do this and don't compare this and don't use this and don't use that. Now the funny thing I thought, okay, well, that would be a great thing to like load in. The comical thing is the 29 page document violated those things like multiple times without an explanation. I don't think the person who was doing it was necessarily super cutting edge with how they did it. But like that feels like there's a part that it's like the stuff that we were used to having in our head. And I feel like there's a lot of discussion around this of also how do you, how do you share that data knowledge across the company that, you know, Joe in sales like really knows the nuances and the ins of out of this system. How do you not only do an initial picking of their brain and capture it in a way that's usable, but you've got to then maintain it. And the maintenance of that. And then you got to make sure that the AI is always always using it referencing it. And then how do you catch like, I don't know, I feel like people naively know AI hallucinates, but they probably rarely catch it. And then it's like how do you make sure you catch it if you have what you're saying to them like that nine page document. But then you're producing things off of it all the time. Like what are the statistics? How often is a hallucination slipping in that they weren't holding that true? I mean, it's hard like how would you put guardrails around that and catch that? We're doomed. Give up. Yeah, that's that's interesting. That's what I'm seeing with the expectations of delivering things a little bit faster because we have AI's to support agents to support. We have AI that can help us shorten the process to deliver anything that is is needed. But what I'm noticing is that there are maybe less checks happening. Is AI indeed taking this inconsturation? Is this updating on a recurrent basis as expected? These are a few things that I'm starting to notice. How much can we rely on? Well, the agent is going to do it for me versus well, maybe that was a break. And the agent is actually not doing it for me. I mean, didn't it be fair? Humans who are doing stuff, we would make mistakes. We'd miss them. So we weren't hallucinating. We were just screwing up. What's the distinction between a. But we also as analysts knew that we really needed to own it and we knew that one mistake was costly as 25 successfully delivered things, whereas hallucinations, I feel like we feel less accountability for them. What it feels like to me when I'm working with AI is I'm managing a very smart but junior analyst. And there, so my job is to give them as much context at the beginning. Here's what we're trying to do. Here's what it should look like. Here's what good looks like at the other end. Here's how we're going to ruin this process. Here's how we're going to evaluate. So you go through this process, but you're setting it all up as a picture ahead of time. So it has a framework. And then it's running through the different steps. And then I come back through and say, okay, now here's data that I did. And we're going to compare results. And we're going to look at it together and we're going to see if whether or not you got it right. And then it's like, oh, I did. I made a mistake here. I'll fix that. And so you go up to go through. So this is the part where it did yellow. I feel like I'm not sure if I'm saving time sometimes using Claude because I'm having to go through sort of these iterative rounds of sort of like checking the quality of the data, those kinds of things. And so like managing the initial and ongoing context is one thing, not just for you and the agent, but also you and the rest of the team. And then making sure that the process is working correctly through the through the end of it. So the output is actually of the quality you want. But you have to have like good quality because I think you're right, Julie, like stuff slips through. And there's little mistakes all the time. Like very, the agents are the state of the art models are quite good now. They're very, very good. I love using them. They still constantly make little errors over and over and over again. Like they write SQL that doesn't quite work in this environment. They do little things like this. And a lot of times they catch their own error. And they're like, Oh, I made a little mistake. I'm going to rewrite that. But it's still costing you time. I mean, you know, it's fine. Like I sit clothed on a task and then I go scroll LinkedIn or something for them now. I'm just kidding. (laughing) - Look at that. - Do other pandering. - And your other managerial tasks, quote unquote. You know, or you just fire up another session and you know, work in parallels, you know, we've all got that. But that's what I feel like. I feel like I'm managing a junior sort of inexperienced analysts most of the time. - Yeah, Michael, I actually don't have the data on this. From what I'm getting, I have a lot of students reaching out to me from my university and telling me they are graduating and it's really hard for them to find jobs. And again, I don't have the data here, but my understanding of what I'm seeing sort of in the industry is a lot of the initial tasks that were done in the past by very junior analysts are now being done in a way. With AI. So that's why it's a little bit harder for juniors to come in to the workforce and start adding value because we are training agents to do that lay of the land initial work. That is not quite working as you were saying, Michael, but a lot of the initial repetitive tasks that we would have before an intern doing, it seems like it's being taken care by AI. This is interesting and I don't know how this is gonna sound. Maybe it sounds a little silly. But with that being said, Michael, when you were giving that example, I'll send my agent off and it'll keep trying and reworking. It was making me think of the time you saved, like letting an agent go do something. How much money might it cost you like token spend, right? Like to let the AI keep spinning and making errors and spinning and going and then Daniella, you saying that I'm like, I'm probably, I mean, but I don't know, like I would love to know this. Would you end up paying more for an AI agent to spin on something or get more senior resources to train the AI to do the intern thing or just pay the intern to come learn the skills and do it? Like I do wonder what, how that would shake out because you start to hear more and more people be like, oh, didn't realize my agent was off trying to like poke a hole into another system and I got a $30,000 bill. Like I've heard more and more of those examples. - Just a note for the listener, like the bubble the AI bubble may have burst in between when we recorded and when this happened and this is gonna sound very dated. It's like, yes, the model is broken. - Yeah, that's definitely something that I'm seeing a bit more on the token usage and limitations that I wasn't seeing before or maybe less people were playing with it. But now it's a lot, oh yes, we want everyone to use AI and that's being reinforced everywhere. But also, okay, this is actually costing some money. And I say it as in a personal level, I still cannot fully understand how much it takes me to do certain tasks, how many tokens is costing me to do certain tasks. So it's something that I'm growing into it and understanding, oh, there is an opportunity here. And certain things to you all points, I mean, I rather just do them or pay maybe a junior analyst because it's not worth an India amount of tokens that is costing to just kind of go over and over and reiterate something that if you explain to someone, it's maybe easier to do it and cheaper. - There was something else, Michael, you said that I'm curious, everyone's thoughts. And as it happens today, I was doing a podcast test that happens every two weeks and it's a fairly involved one and it's all in code and it runs all day 'cause it every time it runs and it is wrote and rep repetitive. But it's cruced something up, sometimes it catches it, sometimes it'll just get hung, sometimes it'll do something that has been work and fine. And so every time I go through it, I might keep track of all the speed bumps and let's update the tasks so that we don't run into those again, so that's one part of it, that's a springboard of like even a repetitive task, pretty tedious but it takes eight hours of duration for me as opposed to me just doing it in like a focused tedious one hour. But it also drops in like all this task switching. Like I have pushed it to say, what do we need to do to not have me have to click you to give you permissions for different things? Like can we figure all that out of front? And then there are things where I do need to be involved but I don't know when that's gonna happen. So whenever the little ding goes off, I turn and I'm like, well now's where I need my human interaction and I think about people talking about like spitting off all these agents and it feels like chaotic. Like oh, I'm doing five analyses simultaneously. I'm like, are you monitoring them? - No, you're not. - Well, yeah. (laughing) So like, I mean, the tab, they being able to do more but it feels like, unless you have a lot of discipline to say I'm gonna shut off all the notifications, you're just constantly getting bombarded. It's like the next version of email or Slack or whatever. It's now that junior analysts who's sitting too cubicles over saying, hey, what do you think of this? Hey, what about this? - That's so good. The head just pops up over the wall. - Hey. - Yeah, hey. - The hand. - Bye. - And no man, you can be rude to it and say, stop it. And it'll, at one point it told me it was like, I'm gonna wait for like 30 minutes and then I'll check back in on that. And like two hours later, I'm like, son of a bitch. Like you didn't, I was like, I thought you were gonna wait. It's like, yeah, I shouldn't have told you that I could do that when I can't. I'm like, it writes that stuff down to him and keeps the record. (laughing) So I don't know, is there a piece of that that's another like concern of like, are we even doing good work or are we at a point where we have this thing that's interrupting us constantly? Anyone? (laughing) - Yeah. Well, Kim, I'm assuming like the things you're doing are requiring it to do computer use. Like it has to go and navigate to specific websites and click on things and stuff. - Yeah, I mean, some of it is that stuff. And I've said, can we promote all of it? - Yeah, it's gonna take forever. Like is this just AI is not good at that yet for a lot of stuff? - If there was APIs or CLIs. - Oh, no, a lot of times it's just me giving it permissions. But yeah, are we logged into this? I'm like, but I'm like, you should tell me at the beginning to go and log into these three sites. So those are available. - Well, you should just have it write a kickoff script that does all that for you. - That sounds like a engineering thing, Tim. - Oh yeah, yeah. I'm, if there's anyone listening to this thinking, I've got it figured out. Yeah, we've made that clear that I do not so. Can I ask this Tim, like, is there any part of that tedious one hour that, well, I probably know the answer, but like, let's pretend we're talking about other tasks maybe. I always come back to like, can you make AI do it sure? But to your point, like, is it actually a task that is like fulfilling, enjoyable, needs you in the loop so much that like that is actually the valuable hour of a human doing it rather than all this pain of letting the AI do it for eight hours? I don't know. I feel like sometimes there is this push to be like, well, if AI could do it, like, make them do it. Everybody in the organization, use AI somehow, find some way to use it. Like, you were saying Daniel, like, I just get to the point where I'm like, should I really be using AI for the token spend and the pain setting it up and telling the silly junior analyst, AI, how to do it? Like, I kind of just want to sit and drink my cup of coffee and do the clicking myself. But it's interesting of like, how do you draw that line? 'Cause I also have a whole thought that I'll wait on about like, who that efficiencies actually then benefiting from AI. - So I just want to quickly say, this is episode 308 that the joy of that, like evaporated around like, episode 120. So no, fuck no. I do not want to keep. But I do think there's a distinction between having it do the agentic, probabilistic figure stuff out versus using it for coding something that then becomes deterministic that has been, like I've gotten my new ATRW Consulting.com, like I vibe coded like websites where it's like, it's there. It's coded and that was like, I built something and the end was a deterministic code base. Now, how that relates to analytics, how much is like, I'm building a pipeline that I can then refine and tweak and then I've got the SQL that's reusable. Like to me, there seems like there's a distinction between the vibe coding to build reusable code or applications, which I think there's, that still brings me joy with the million little projects I've done on that front. The, I'm going to get this junior analyst, like it's a junior analyst who I'm going to throw out a window 'cause I'm like, oh my God, follow the damn checklist. Like you, the sponsors goes below the copy, not above, like it says it right there. So, boy, I don't know about recording this at the end of this day, but. - So, Daniela, how do we fix this? - Yeah. - Oh my gosh, what a roll. (laughing) - No, but I think, I think we're all touching on things where we all think AI has this idea. this real big positive potential. We're also running into the opportunity cost of how to integrate it effectively in our day to day work, such that it produces the results that are required of the roles that we have in our jobs. And the imperative that this must be something that is disruptive and is changing everything. So we must also do it. And I think it's put a lot of us in sort of like a weird headspace in our jobs and careers in the last couple of years. I don't know, Daniela, like from your perspective, like in working with teams and coaching people, whatever you're in your roles, how are you guiding people into how to think about the integrating their work and AI? That's just like a more eloquent way to say, so how do we fix all this? Yes, the silly question, and then you're like, what if I give more of like a chat GPT really long coherent, but it's the same question. You're absolutely right, Tim, you hit on the key issue. Well, the thing is, I don't have the answer, no one has the answer. And what I'm seeing, and also because I work in tech, we are in this industry that people just expect you to know the answer. Well, you guys are driving the thing. And very often we are all just trying to figure it out. That's my take. And one thing I think that tech does very well is the ability to learn very quick and also course correct, very quick. So as we integrate different AI tools to different companies that I have worked at, very quickly we start sort of mapping what we are using for what? For dashboarding, for coding, for narrative producing internally for video editing. So having that sort of framework that enable us to know what to use for, also a lot of the sharing coming from the teams that are closer to clients or the more technical teams, a lot of the sharing is happening in these early stages of, hey, we are discovering this. And this is how we use it and how we can also teach other teams. Those are things that I have been seeing for myself. And I go back to this as I have been a power user for now a year and a half understanding that more is not necessarily better. That's what I'm coming. We have to touch on these a few times. I was very bullish on implementing, implementing, implementing. And now I'm coming to realize, well, maybe there is a lot of opportunity in slowing a little bit more and doing extremely quality deliveries rather than just quantity because I'm coming to see, well, all these agents, all these models can build way more, but it's not having the impact that it used to have when I was doing it, everything from scratch. So kind of going back or taking a step back on, okay, I'm using it is helping me to produce, but I still have to put the 10 hours of deep thinking, this is how I build the story. This is how I use the three, four numbers that are really going to break through that maybe my agent in marketing science cannot really get it without having met the client. And if, you know, I have a call that is being transcribed so on and so forth, there is still, I mean, hopefully my brain is being used for something. That's a really good point about the type of industry you're in and how that might actually, because you were talking about sort of in your role and sort of tech, it's like very fast moving, quick decisions, knowledge and learning is super fast. Like I have some clients that are in more regulated industries and like there's things you can't touch with AI, like they just, like, sorry, no note taking apps on calls, like that's not allowed. Like, so it's like, okay, we have to put those tools to the side and you do it the old school way. And it's kind of interesting because you do, there's different flavors or layers depending on what type of company you work with or what industry you work in. So I think for listeners, that's also maybe like a little bit of encouragement, like depending on the type of company, like you might not be lagging, you might be right where you need to be for where you work and what your company is. And if you feel like you're going too slow, then go find a new job in tech and go fast. Well, there are a lot of things that, you know, from the industry. But if you feel like, oh, maybe my industry, my company is not there yet, is the opportunity for you to be the driver because I was in the past that company is that weren't maybe in that front end. And I was the one, let's do it, let's implement it. And all of a sudden, I become became the person, oh, I'm going to go and ask Danielle about this because she's implementing it and maybe, oh, her decks look amazing. And I'm like, yeah, they all, yeah, a lot of them for me. I kind of heard something a little different, which now seems like totally obvious. Like everyone should be exploring, but we should be going in saying, I'm going to try to do this with AI, but logically, it seems like it might work, but I'm doing two things. I'm trying to complete the task and I'm trying to learn if this type of task is a good fit. And that little kind of meta-analysis level of, for you personally, anyone, I'm not speaking to that, what are the types of things you've tried? Are you trying different things? Are you assessing after you've done them, like trying to figure out, hey, that kind of worked. And I think here's why it worked and here I could work better. Or some of the examples I think, Julie, you were thinking of like, what the hell, like, sure, I used AI to accomplish that, but it took me longer and it cost the company extra money because of token usage. And you know what? It's not just the next model is going to solve that. Maybe that's actually something I should keep doing. Like, I haven't, I haven't stepped back and said, let me make my catalog of the things I've done and tried to classify them and then assess, like, man, it was amazing at this, including totally new things like ideation around an analysis, which I think it does quite well. It all makes me think too of how are we really defining the efficiency play? Like, I know a lot of people talk about it just as like, time saved and there's a lot of assumptions behind like, what time saved means and the, you know, value of that. But exactly what you were saying is if the end quality is not as good or it's not as influential what I'm delivering, even though I did 10 of them, it's just, I feel like a lot of times people aren't taking into consideration all of those measures to really say, was I more efficient or maybe that dips more into effectiveness? Like, maybe they're not looking at both efficiency and effectiveness when using AI and it would be a really interesting exercise to your point to him to really be like, how would I measure those things? And if I look at all the things I've done with AI, like, which ones were truly like worth it and successful? Then if you find the ones that were really fun, but they actually were a complete waste of time, you're like, yeah, but it was, yeah, but it was so cool. So it's, it's a really interesting point and it's something that I was seeing a lot, maybe what a year ago when AI was really hyping, like we were seeing in a lot of the calls, quarterly calls when public companies have to report, we were seeing it X millions, billions in AI, X prompts being, being digested, number of people that we were hiring, these were a lot of like very concrete numbers that I just, from my perspective, companies maybe were having a lot of pressure in delivering, oh, we are a AI driven company. So we were seeing a lot of these things. And to your point, Julie, right now that we have, I don't want to, I don't know if we have passed completely that phase, but now that AI is being integrated into the majority of companies is more of, oh, what are my, what am I actually delivering or what is the revenue that I'm actually driving because I was able to use AI because at the end of the day, if I go and tell a client, well, I did this in two days instead of a week because of AI, the client doesn't care, right? That doesn't really matter or, you know, at the end of the day, what the client wants is a good, a good output, a good result that is proving revenue for whatever I'm trying to, or whatever I'm trying to deliver. So AI really is the underlying sort of army that is helping us to be a little bit more efficient, but in a way to deliver revenue, at least I'm close, I'm close to sales in general. So I always think about, how can I drive more dollars? Well, and then being more efficient, we have to remember that it's also our job to stay influential, right, with data. So that's the balancing act that doesn't show up in the numbers a lot of times. But that, that, you say, stay influential, I think, I think they're, we're not going to. A.I. your way out of not delivering value like I think when analysts kind of ring their hands and nash their teeth and say I don't have a seat at the at the board, you know, table and then they're like, now I'm going to use AI. It's like, well, okay, that's going to help you produce more. Maybe it's going to help you be more consistent. But it's like, you still need to figure out like, what is the path to get there? Like, I definitely worry about that, that it's like, well, here's the thing. It's going to solve. There's fundamental relationship and ways of working and thoughtfulness, things that AI is just inherently not going to solve. Unless you recognize it and maybe AI can coach you and help you learn it. But I think that's another, I'm thinking about we saw Sandberg, a past coworker of ours, present at Marketing Analytics Summit. And I've continued to talk to her in part of what she's in her team are using AI for is different mechanisms for delivering information. So even if fundamentally the information they're delivering is the same, they're spinning maybe the same amount of time, maybe a little more, maybe a little less than saying, but I'm going to tailor to the audience. Maybe I'm going to adjust the, you know, people in the past, when he said, I only have time to produce the deck. So that deck has to be the pre read, the leave behind what I present. But now I can produce the full detailed report and I can produce the executive summary and I can generate, you know, an audio podcast that's three minutes that delivers it and really thinks about how am I serving my business partner more effectively, even though potentially the raw, what is being delivered hasn't changed is helping deliver it more effectively to the audience, which is like a really like thoughtful way to think about it. And that I think she's had a lot of tried this, it didn't work, tried this, it did work, which goes back to the point of saying, I'm not just going to turn it on and it's going to work. Well, a lot of iterations what I'm seeing with content and this is something that a few months ago, conversations with colleagues with friends, well, we can just now do all these videos, all these posts with AI, it's, you know, quantities is amazing. We can all become influencers. And, you know, a few months down the road is like, well, no one has become an influencer because we all just talking the same and speaking, you know, using the same words because everything is being produced with AI. So that level of how do we differentiate to your point team? How are we touching the audience the way we need? How are we shaping it? That's the layer that in my perspective, AI is just not there yet. And when hopefully our brains and our creativity is being supernated to really stand out in a way. This one, those philosophy PhDs are using AI to read the Nietzsche and produce their thesis and then they actually get a philosophy PhD, but they don't have the critical thinking. All right. This is great. Daniel, I thank you so much for sharing your experience and expertise with us and our listeners. This is one an awesome, good discussion, very fun to kind of open this up. And I think it's very applicable to everyone's jobs right now. So, yeah, but we do have to start to wrap up. Unfortunately, the time flies by. All right. One of the things we like to do is go around the horn and share a last call, something that might be of interest to our audience. Daniela, you're a guest. Do you have a last call you'd like to share? Yeah. I was on a call yesterday with IAB, the advertising bureau, and we were talking about agentic commerce. So, very, very curious as we were talking right in the beginning of this call, what are the automation tasks that we're going to be seeing? And I think a lot of the shopping online, especially the recurrent shopping, is going to completely change the way we shop, the way we use LLMs to inform us and to basically place everything for us and order it. I think it's completely going to change the way we shop online, and I'm excited about it. Is that I was having this? I got a, sorry, Michael, but I'm like, this is like getting people to part with their dollars. Like in when people who are used to like hallucinations, some bad judgment calls, because I feel like they're in the digital analytics world, there's been a whole lot of like, oh, this is coming. Like the agents are going to be sent to get shopping. And I'm like, like, how much is the IAB interview thinking that that's going to be, I may send an agent out to find a lot of options, but I don't know that I'm going to cut it loose to make the purchase decision. Yeah, that's what we are seeing, but not yet for to pull the trigger, but a lot of the search, a lot of the search is happening as we were doing it, having one person or one agent that is basically pulling everything for us based on what they know about us. I think the next stage to your point, it will be interesting to see how long is going to take. I just see like retailers who are saying, we've, we've got to be supporting supporting agentic commerce. And I'm like, is it the purchase you're trying to support or is it you're just trying to surface the shopping versus the purchasing? Hey, I don't know. Door dash already has a command line interface. So say everybody's doing it or it's starting to the door. All right. That's the whole other podcast episode. Tim, why don't you share your last call, Tim? Okay. Well, mine is a podcast. It's an old podcast that's new again. And I am super excited. But I'm also like a political junkie. But the the 538 crew of Nate Silver and Claire Malone and Galen Druck, they're back. So like 538 kind of did. It's bouncing around through ESPN and the New York Times and was gone. But they there's now they're back on the still counting podcast. They're with the crooked media network. So it's, I mean, it's clearly political stuff, but really, really sharp data mines. And I enjoying the first few episodes quite a bit. So basically, 538 podcast is back, but now it's called still counting. And it's in a different world. Mine is actually very applicable to today's episode. It was a Cassie Cosercove newsletter. We love those. And the title was a really interesting statistic. And it's chock full of them. But it's all about how 97% of S&P 500 filings mention AI. But you wouldn't believe the percentage that actually like had written out budgets for them. And even fewer actually describing like how they govern them. And so she goes through some of these crazy statistics. So highly recommend looking at it. And then at the end, she's kind of closing it up with this idea of like the winning companies of using AI are probably going to be the ones who trust their systems the least. And I know that's been just a big theme of like you can't just hope that they're doing the right thing. And like that sense of goodwill towards them is really kind of been eaten away through a lot of experiences the past year, year and a half. So it was it was a good quick read. What about you, Michael? Well, somewhat related to this episode, there was a website that came across recently that I thought was clever. And of course that the vibe coded website called the archive of incorrect AI predictions. So it has this whole archive of people making predictions about AI that have been wrong throughout history. So way back in the past coming forward in time. And you can look at it in different ways. It's kind of fun. So yeah, it's a fun little website. It's a I like the name of it. It's boys who cried AI. It's the name of the website. So anyways, that's a that's a fun one. All right. So as you've been listening, you've probably have thoughts of your own. And we'd love to hear your comments and suggestions. Leave us a comment. The best way to do that is on the measure Slack chat group or our LinkedIn page or via email and contact at analyticsr.io. And we'd love to hear from you. Also please wherever you listen, leave a rating and review on the podcast that helps us out. And if you would like a sticker for your laptop or your mug or water bottle, we all send you one. And you can do that on analyticsr.io as well. And once again, Danielle, thank you so much for taking the time being here today. It's been an awesome conversation. Very timely. I think everyone's work is feeling this impact. So thank you for coming and sharing some time with us today. Thank you for having me. Awesome. And I think I speak for both of my co-host Julie and Tim when I say no matter where you are in AI adoption, keep analyzing. Thanks for listening. Let's keep the conversation going with your comments, suggestions and questions on Twitter at at analytics hour on the web at analyticsour.io, our LinkedIn group and the measured chat Slack group. Music for the podcast by Josh Groherst. Well smart guys. wanted to fit in so they made up a term called analytics. Analytics don't work. Do the analytics say go for it no matter who's going for it? So if you and I were in the field, the analytics say go for it. It's the stupidest, lazyest, lamest thing I've ever heard for reasoning in competition. Being in detail and being like, "Oof, maybe I would've said it so much better." What's been doing that throughout the course of the planning? I'm like, "Yes." Maybe. That's an interesting choice you guys made with how to do it this time. I think what you should do is actually take that audio snippet and send it to Val and see if that's accurate. Well, all through the planning, as she was trying to watch from afar and be hands-off, I would often get the, I was the vent, the outlet for her, "Oh." [Music] Rock flag and philosophy majors!