GenAI Best Practices: What Early Adopters Have Learned

[MUSIC] The Data Chief is a podcast for data and analytics leaders to share their personal stories and insights on technology, culture, and leadership. [MUSIC] Welcome to the Data Chief. Welcome back to the Data Chief. This season has been all about navigating the hype, the fear, and the truly transformative power of a gentick AI. We've talked to incredible leaders, early adopters who are setting the standard for how to drive real business value with AI. In this episode, we have pulled together the most impactful moments into five best practices to guide you in this disruptive time. First, pick the right use case. Next, invest in your people. Third, foster AI literacy. Fourth, do so ethically. Finally, embed AI where people work. Hmm, how did I not mention by the best tech? Of course, but perhaps it's also because the LLMs, the tech are changing so quickly. The AI orchestration layer is being built as we speak. Avaluating whom you partner with then is a matter of trust. Not just functionality. Put that on your RFP. So let's dive into the lessons learned from the front lines of AI. The Data Chief is brought to you by ThoughtSpot, the agentic analytics platform company that is redefining how the world interacts with data. With ThoughtSpots Intuitive Natural Language Search and Agentic AI, every user can access transparent, governed, AI-powered insights when and where they meet them. Learn why industry leaders like Cisco, Toyota, Hyatt and Capital One rely on ThoughtSpot. Experience the new era of analytics at ThoughtSpot.com. Anand Eyer, the Chief Digital Officer at EcoLab, spoke of the ultimate measure of success for any AI initiative. If you can't tie it back to the dollar, it's not going to survive. In a world full of hype and fomo, Anand reminds us that our true North must always be measurable and attributable to business impact, whether it's top-line revenue growth or cost savings. Here's Anand on why you must always start with value. When you think about something that's got so much hype as AI, it's very easy to get lost with that hype. There's a lot of pressure from boards and from executive leadership, CEO down, to be able to use AI to solve every single problem whether it is an AI problem or not. How can we solve this problem with AI? A few months ago, the year ago, Shoeb was all about Genie and Chat GPT. Now it's all about, "I can use Agentic AI and use agents and so on." There's also this huge fear of missing out. If I don't start AI initiatives right away and show progress, then I'm missing out. I'm going to be seen as not progressive enough or not, cutting edge enough. Which is all good. It's all good in terms of the focus and the dollars and everything else. One thing that people sometimes forget is that end of the day, it's all about, "Are we either saving money or making money?" Are you able to show that in the bottom line of the top line in a measurable way? There's got to be a straight line between the initiative and where it's actually impacting the top line of the bottom line. It was all the things around soft benefits and type 2 or type 3 benefits are all right. But end of the day, it has to start showing up as these are type 1 settings or a actual attributable increase in the top line, which then impacts the bottom line. So those are things that the CFO and the CEO ultimately would want to see. But the key is to be able to really identify initiatives that you can actually draw a straight line between the initiative and where it's impacting from a dollar standpoint. I always want to bring it back to what specific problem are we trying to solve. And how will we measure the value? In some cases, it's a very netless value that you've got to be more efficient. Well, what does that actually mean? Does that mean? So you can say, so we hear about this copilot and that copilot and it can give you a 5% 10% efficiency. Does that mean that we can do this with 10% fewer people and we can redeploy those people to other initiatives? But it doesn't mean that then what does actually mean when you say you're 10% more efficient? Right? So if you have a team of people performing a certain function, 100 people, and you're saying it's going to be 20% more efficient because of use of AI or chat GPT or whatever else, then are we able to say that we can redeploy 20 people to some other function that we wanted to use? Or does it mean that we still have the same 100 people and they just have bits and pieces of time, let's say we can't really do something with it? And that's the case, the latter is the case and there's really no straight line between the initiative and any dollars that is impacting. So that's where my head's at, right? But if you want to have a seat at the table, you've got to be able to talk in terms of what the value is in terms of dollars. But the point is if you cannot point out how a specific AI initiative is either impacting the top line or the bottom line or as GNA, then it's going to be very difficult to have a sustained conversation with any C level leader and have that impact. Now that you have picked the right use case based on business value and data readiness, you must bring people along on the journey. One leader described to me that in earlier innovation waves, we might invest dollar for dollar in tech and people. With generative and agentec AI, they believe we should 3X our people investment. So ask yourself, how much is your company investing in AI versus people change management? Karen Strupp, the chief digital officer at Wex, explains why we must focus more on the people. What are some of the lessons that you've learned in navigating this ambiguous time and still ensuring trust? First, I think ambiguity is absolutely the right word to focus on because I think ambiguity is paralyzing. People don't know how to navigate through it. They don't know where to start. They don't know what it means for their jobs, what it means for the future. And so I think our role as leaders fundamentally is to help build confidence and help people navigate through that ambiguity so that companies can learn and they can unlock the power that's being generated today. So I would say three key lessons. One is helping manage people and their feelings and their confidence. Two is around clarity, over certainty, and the third is around speed to learning. So just to unpack those briefly, I was just talking about is I don't think people fear change as much as they fear that unknown. They don't know what's going to happen. They don't know what their job is. And so part of what we're doing at Wex is really trying to involve people in the journey. So it's not done to them, but it's done with our employees and with our customers. And I think that really helps alleviate or subside some of that fear. A second is clarity, over certainty. I can't tell you when jobs are going to change exactly, but I can tell you that they are going to change. And so what we're trying to do is take this ambiguous problem and narrow it down to bite-sized chunks for a lack of a technical word. Little pieces where people can experiment and learn. And what I see is those eyes light at moments where you go and skeptical and be like, "Oh, I don't think an agentex solution can actually do something to solve this problem too." Actually, it is solving the problem. And if they can do that, what then could it do this? And you're bringing people on the journey, but you're doing it in small steps versus saying, "Oh, I want you to reduce cost by 10% across the kept me." And the final part to me is speed to learning, which is not new in the AI journey, but again, as one of those two isms, I think it's better to get started because it's going to happen. And I can promise you one of the things that's certain is it's going to be messy. It's not going to be linear path. And so if you have a hypothesis-based approach, if you say, "I want to experiment. I want to see what I learn in the next two weeks or the next month." Then I really think that the focus on learning helps accelerate the impact on the company and on customers overall. So I love those three key takeaways, and especially that you started with the people impact. I think that is the right priority and bringing people along on the journey. Addressing their fears is probably job one. One that many leaders underestimate the importance of. So as you think about your role as Chief Digital Officer and the changing roles of Chief Data and Analytics or Chief Data and AI Officers, and I'll go back about 10 years ago when I had just joined Gartner, and that was all digital transformation, bestselling book, "Digital to the Core." How do you see the role of the Chief Digital Officer having changed in your time in this role, should this be the next level for aspiring Chief Data and Analytics Officers? It's a great question. And in many ways, I think the word digital transformation, that sounds pesse and dated. So I actually don't use digital transformation anymore. It's really just about transformation. But let me actually start with what I think is not changing. When I alluded to this earlier, but I think there are a lot of fundamental parts of the job that is really around driving transformation and those key tenets are actually the same, regardless of what the technology is. Yet, as we talked about, you want to stay focused on the problem. What is the big unsolved problem? What's the bold vision? What does great look like? And then how do you unlock that into steps that you can sequentially solve and decide if you want to pivot or persevere or stop altogether? But really, that fundamental approach to driving transformation remains the same. That said, I do think there are some material differences. One, I don't think there's any chief, anything officer that should not be considering AI today. I think if you're not considering AI, you are at the risk of being disrupted because you're not going to be learning at the pace with the rest of the industry. And there's someone out there looking for a better way. Two, I think our job is to balance the incremental versus the disruptive, the transformational, the revolutionary change. And it's really hard to figure out how you want to use AI in incremental ways versus thinking about how your fundamental offering or your customers or your business model could be disrupted or needs to evolve because of the potential of AI. But I think our job is to carve out time, to think about both time horizons. And so finally, I would say that our job is to create the conditions going back to the people side, the conditions to help lead through this change, conditions that give people the confidence and the safety, the psychological safety to experiment, to try something, to focus on what worked and what didn't work, learn, let's not make the same mistakes multiple times, but have a really rapid learning cycle. And I think it's more important than ever that our job is around creating that space to learn and evolve. I think if I were to summarize this, the real power is when you unlock the creativity and the how can we mindset of your employees and your customers. And that's a cultural change. That's really about unlocking the power of our people. One set of skills that all people need to hone is data and AI literacy. This is true for everyone from the back room to the board room. As both a society and business environment, we are not doing well here. A recent survey from data to the people found that 44% of employees cannot independently analyze data. And perhaps worse is the Allen Institute found that 84% of Americans fail basic AI literacy. One company that has taken a novel approach to building data and AI literacy is Lloyd's banking group. Josh Cunningham, group head of data and AI culture, explains. As you think about how you enable business people to get more confident and comfortable with data and AI, what are some of the specific techniques that you've used to help them build their confidence here? So I think in an independent level, it's trying your best to meet people where they are and make you real for them. So finding a way to anchor the learning to something that's relevant to the day-to-day role is always going to make it land better. So we kind of try and take a bit of a two-pronged approach. So one would be Lloyd's massive organization, 65,000 colleagues. So we do produce data and AI literacy in a learning and training on mass. So here's what data literacy is, here's what AI is, here's the kind of things that everyone is going to need to know. But all we also do is we pay that with, in addition, more prispoke learning. So we partner with each of the business units or it could be with each of their business platforms to say, what does the day-to-day look like within your specific area? And what kind of things do you need your colleagues to know within the context of their role? And actually, we've found that to be a really productive way of thinking about it because it means that each of the business units will spin up an issue, James Anchor, to what's important to them. So for example, our consumer lending business have said, we want, so we've set a vision within Lloyd's to become the most data-literate bank. And now our consumer lending business are really leaning into the foundation. We want to be the most data-literate business unit within Lloyd's. So actually, their focus is their senior management layer to start with and we're working with them to my part. How do you get a curriculum that actually is going to really resonate with their day-to-day roles? But then, equally, over in our insurance pet pensions and investments business, and they're focusing on all colleagues within their day-to-day roles as well. So how might we tailor learning that is specifically relevant to the insurance business, for example, and what do those colleagues do? So I think it's so important to actually have that broad offering that any colleague can consume, but then have the laser shop focus on at a local level as well. Yeah. And I love the competitiveness between the different spokes or business units, the insurance versus consumer lending. How are you measuring success? So measurement, then. I probably spotlight a couple of things. So one would be, if I think about our day-to-day colleagues, so we've created, if I was to step back, we've created a job family framework for day-to-day colleagues. And I would say pretty reasonably well aligned to what you senior would recognise in other organisations. So we've broken the topic down into 11 job families. There's about 52 role types within that. Each job family has a set of core skills. So roughly 25 with 5 priority. And then against each of those skills, we've used the Dreyfress model, so 1 to 5 in terms of a confidence level. That kind of sets the stage. Now each year we do an annual skills capture where colleagues can fill in, at what level they think they are, must have 360 view of all the day-to-day, by their peers, their line management, etc. So that gives us quite a rich status set as a bit of a baseline to say in aggregate, here's the skills that we have in the organisation at work skill levels. And here's where we might have particular gaps. And that steers us in terms of how might we build learning initiatives there, where we'll really drive a difference, align to our business goals, where we drive a difference in terms of where we're trying to achieve. And we do that on our regular cycle, which means that actually we can track aggregate trends in terms of what skills are improving, what we need to focus on, etc. So that would be measurement of day-to-day colleagues, that's our biggest baseline. In terms of business colleagues, then if I was to think about the skills I've left there, so if I was to think about our day-to-day literacy framework, so we've developed a framework that consists of 5 personas. So we've always tried to break it down in terms of the 5 different overall where people might be on the day-to-day literacy journey. And that's things like so you can go from day-to-begin air through to day-to-inthusiasm, through to day-to-day explorer, through to story-talented citizen. So that's very much almost a bit of an evolving maturity life cycle of 5 different personas. Now, the way we run our day-to-day literacy industries is we then work with parts of the business, work with colleagues to conduct surveys that will map either at an individual colleague level or a business unit level, where we think people are, and they can contribute to this, and they can self-serve it, etc. So again, my question is a bit of a baseline. And then what we do is we target the sooner. So to bring that to life with as an example, we might say, well, for this specific art of the business, we've completed a survey of 15% of the population and most people are in the data beginner phase, and that would be things like, you know, I'm a little bit uncomfortable about day-to-day, I'm not quite sure what tools to use or to necessarily understand some of the key concepts. And develop a roadmap and a vision that says, on average, in aggregate, we want to get people to a data explorer, which might be starting to, you know, comfortable with Excel, starting to use visualization tools, starting to lean into co-pilot, some AI tools, etc. And so I think that kind of measurement, mapping the form, mapping the two, putting some metrics on top of them, and being able to track progress over time. That's how, on both fronts, with similar, different mechanisms where keeping ourselves honest in terms of the progress from making. Right, so I think that's a really good solid set of measurements and progress on the improvements in data and AI culture as well as data and AI literacy or fluency. As you also mapped the reason for all of this. all of this to the overall business strategy at Lloyd's Banking Group. Have you correlated any of this to the hard business benefits like increased market share, increased customer and PS scores, things like that? Or is that a bridge too far? - I think so there's probably not a hard through line we can point to that says as a result of X, this significant board level metric has changed. Well, what we've tried to do, those we've tried to establish, so what is here that is particularly important to us then? So if I wanted a clock back to when I started my journey at Lloyd's, for example, at a board level, it was recognized that we needed to do more in terms of increase in our data and AI maturity. That was a big focus and we had various external, large scale consensus, the organizations who helped us to benchmark. So we set that as a strategic priority, recognizing that an increase in maturity actually would help us with a number of other board metrics that we kind of joined the top team too. And then at that board level two years later, which was a year ago, we remeshed that maturity and we'd made really quite significant progress. So I would say less so we're able to say this specific metric is as a direct result of this training course, but actually what we do is we work with each of the business units to bake into their business road maps and their data strategies the layer on top. What is it that's going to move the delve here? What are your priorities? And data literacy in every part of the business is going to look different. So what we might need to improve or progress within a finance function, might actually be really quite different to what you might want to progress in an operations function in consumer lending, for example. Kind of thing is important to tailor it in terms of what is going to be important to that specific business. - Right, makes sense. Thank you, Josh. And now I'm looking at your background. So if anyone sees the video, you'll see this fun background, data in AI summer school, and Lloyd's recently won an award for your education efforts. Tell us about the summer school. - So summer school, anyone who's spoken to me in the last probably two months, well, this is all I talk about, right? And so summer school is self-dynamized. If I was to step back maybe to last year. So it's an initiative actually that we've run for a number of years. I think about four years overall. Last year was where we really restarted to the scale. I mean, it's something that we run over the summer as you'd expect, for a full two months. It's fully virtually enabled. So the other really great benefit we have again, is that we run something called flexible summer, which broadly means that during the summer months, actually colleagues can broadly work from anywhere, right? So no necessarily, I need to go into the office, actually work at home, work from wherever it's used. Now we've paid our summer school initiative, linked into that by saying, "So for those two months, we would love to offer significant learning opportunities that are open to all colleagues." So last year we ran a few two months. I think we ran about 160 sessions. We had more than 42,000 signups across the organization for those sessions. And it covers, as you'd imagine, from 160, a really diverse set of topics, really unpacking the day from AI, specialisms. Now, this year we've set us out of the challenge of going even bigger, even though there's a pretty massive number. So on the agenda and it kicks off the form, the first to July, we have more than 200 live sessions. They're going to be hosted by internals and externals. So external is new. We're bringing in industry experts, partners, et cetera. And we're really seeing already the whole business leaning in. So the way that we've designed it in Taylor, is yes, state to an AI. There will be some stuff in there for practitioners. And we will have technical sessions, we'll have code and workshops, we'll have even virtual hack of horns and we'll get some industry technical experts to come in and talk about some of these really detailed concepts. But actually, we are also running an internal cons and marketing campaign to make sure colleagues understand that this initiative is for them, whatever their job is. So it doesn't matter whether you've never touched an AI tool at all, it doesn't matter if you think data's not for me. I've got no idea what it's all about. We have in a re-back to basic sessions that will go through data strategy. I'll talk about Gen AI for beginners. He'll talk about some myth-fasting about AI. We'll just mix in lecturers and some thought leaders. So really is something for everyone, trying to take the whole colleague base on that journey. And so as you can tell, this is something that I'm super excited about. I love the idea of a data escape room. For you data geeks out there, I bet it rivals the idea of laser-tack. As we embrace AI in all its facets, it's essential to design with ethics in mind, from idea to production to scaling the systems. We need to red team unintentional harms and where agents may go rogue. Noelle Russell is the CEO of the AI Leadership Institute and the author of the Mustread book, Scaling Responsible AI. She offers a framework for doing this, the poet framework. We can talk agenteic, but I think more important if we are going to have these AI agents, we have to build them with certain guardrails. And you have some frameworks that you use to build responsible AI systems. So maybe take us through your poet framework. What are the elements of that? What's important here? Absolutely. And I built this to kind of take our-- I don't know if you recently saw there was an article that was released, actually it was an academic paper, some research that had recently done, called Your Brain on Chat GBT. Oh, yeah, from MIT. Yeah, MIT did this research. Yeah, of course you would. Yes, you would know. It's in your backyard. But this paper, and it's not new research, this research has been done by Metta in their predictive text organization. So we know that this is true, that there's a bit of atrophy that can occur when you use this technology inappropriately. And what that means is that we've got a really right now we're telling everyone, hey, use this to get off the blank page. But I have always been a bit of a-- I don't want to call myself a naysayer, but just like a optimistic rationalist to say maybe we don't use it to get off the page. Maybe we use it once we have an idea to accelerate that idea. And that's kind of where the poet framework came from, is that we still come up with those synapse that fire in our brain. We still want that to happen. We want creativity to happen. We want innovative thinking to happen. But then how do I go from enthusiasm of that idea to actually doing it? AI can help in the doing. And so poet stands for, I'll just give you-- I'll go through each of them pretty quickly. But the first one is precision. Today we have more data than ever about almost everything from our customers to our employees to the vendor relationships that we have. And I remember back when I used to work before I got into AI, I was in big data. I worked for Pivotal, which is a big data company. And I remember saying to people, like, to these clients, I'm like, just save everything. Because one day a magical unicorn called the data scientist is going to come and make sense of all of it. And neither does it say, like, there aren't enough of us now. But there are these applied AI models. So precision means I need to know first what data I have, what data I need, what problems, your point earlier, why, what problems do I want to solve, and then very clearly identify how I can use that to create a very precise solution. So in the future, it's not one big AI in this guy. One big button we're going to press. It's millions of little AI systems. And it's going to be derived from a precise problem that you're trying to solve. The O is about optimization. And that kind of takes the other side, right? Rather than trying to define a new solution, which a lot of my clients want to do, they just want to boil the ocean with all things AI. And I'm like, or we could just look at how your humans are performing now and make them better at what they do. Every human has stuff they complain about that they struggle with. So optimization is another strategy that you can use to make sure that you're building things that grow revenue or reduce costs, but not at the expense of humans, actually as a amplification of the humans that work for you. And then what you'll notice, which is what Spotify and Salesforce have noticed, is that if you actually lay people off, you're cutting into the muscle of an organization that could actually help you grow. That if you decided as a leader to invest in those people, they would become value generators for you. As opposed to becoming a smaller organization, you now actually have limited that capability. Yeah, or even blockers. I think when you threaten people with AI, they will tell you all the reasons why it doesn't work. And then you get that inertia, whereas if we position that, oh, the optimize, it's, how can I do my job better, faster, takeaway, the drudgery, then they are more engaged. Exactly. And Garner has that like, right, this hype cycle. And it has this concept of the trough of disillusionment. And that's exactly where you're are talking about is the people shift into when you scare them about, oh wait, I'm going to, if I use this, I'll actually replace myself, which is what's happening. Many of these organizations, they're going to actually resist even using it. They won't even beta test it. And AI models don't get good unless you test them. So it's a vicious cycle. And we're going to see that play out more and more because we're only in year two of this game. So we're going to start to see this really get amplified. And one of the best mitigation strategies for avoiding this is to really get aligned with the core values of your organization, the ethics, the ethos of why you do what you do as a company. And that is the E part, is that it's not enough to just say, yeah, we're going to do the right things. And yeah, we have integrity in our corporate statements. But like, how are you going to embed your ethical foundation into the models that you build? And you can absolutely do that. Again, whether you build AI systems to check for those types of policies and processes, or you build a human team, like an AI red team to do that work for you. But that's going to need to be a core component of your success. And then finally, my favorite probably is trust in a world. I don't know if you noticed, but many organizations that generate content are now asking you to disclose in kind of as a predecessor to government policy that will force us to. But right now, a lot of YouTube, you know, meta, they're like, if you were using AI, let us know. And then they'll actually create a little icon on your content that says this was AI generated. Here's the fact of the matter is studies were done at Stanford, actually. Stanford did a study on trust and found that if you just if you are using AI and your users find out that you're using it or you tell them, everyone's going to take a 30% hit on trust, everyone, everyone, whether even in the funny thing is even if you use even if you don't use AI, people are going to think you're using AI. So it doesn't matter. Everyone's going to take this haircut. But the people that will be most hurt are those companies that decide not to disclose, the decide to not necessarily lie, but maybe lie by a mission and go, oh, I'm I can use this, I'm going to use a deep fake of my CEO to my employees, but they don't need to know. And when people find out that they've been lied to, that's a 70% hit on your trust. And Jeff Bezos would tell you, it's unrecoverable. When people lose 70% of their trust in you, they just don't buy from you. They don't work for you. They don't talk about you. They don't recommend you and your business starts to die. And so that I think that trust component is a human component, a relational component, and it is underpinning all the other philosophies that I have. For much of my 30 years in this industry, analytics has been a standalone application and data often a digital exhaust from business applications. In the Agentech AI era, you want insights at the point of impact embedded in workflows. When you embed Agentech analytics into operational applications, it becomes a competitive differentiator. Now Bivers' build has always been a decision point here. But now it's also about speed to market with AI. Adelauntwig is the co-founder and CTO of Navon, the all-in-one travel and expense management platform. Here's his take. The other decision point that you had to face at Navon when generative AI really became usable is do you buy or do you build? Take us through the thinking here. Yeah, that is again an excellent question, which I think today is very, very important. It's very relevant to all of the companies pretty much. Very, we kind of touched on it earlier when you asked me about how to educate yourself on the AI. I think that companies must make a decision and quickly. This or that, when it comes to Navon, so now I had this dilemma almost three years ago. In November, it would be three years. What I was trying to solve had nothing to do with AI. My task was to solve for a business problem that we had and it's the cost of support. We are a travel company, traveling expense, things happen when you travel a lot all the time and then you need someone to help you and to save you many times. So, we had a whole operation dedicated to solving this problem for travelers and they love it. But it's very expensive and the cost goes up very quickly. It's scheduled with your business and we said, "Okay, if you want to be a legit company, we must address components." It was my task and then obviously when I actually had a solution but when the chatty bit came out, I showed it to the garbage and said, "Okay, now I can see something else happening." I started just with chatty bit and very quickly realized that you cannot have a solution. So, chatty bit. Why one reason it hallucinates, left right and center, all models hallucinates, it's not just chatty bit. I'm saying chatty bit, but it's the equivalent of elements. So, just to make it very clear. Okay, so I very quickly looked for what exists because I'm like, "I need a solution." So my laser focus, I need a solution and I looked and there was no solution. It was too early, the world was not there at all. And then I started to build something in Python and what did I be? I'll give you an example. I'm saying that LLN's hallucinates and I wanted to prevent that and I thought, "How can I prevent it?" And I don't know if you personally got to experience when chatty bit hallucinates, it's kind of funny, it's amusing, it tells you something that is a complete nonsense and you kind of correct it politely and it says, "Oh, you're right, I'm sorry." And then gives you the right answer, that's hallucination. And, I basically said, "Okay, if I do it within the same conversation with the LLN, if I see a hallucination, I point it out and then it's corrected, I said what if a different LLN would point out the hallucination and then correct it." And this hypothesis actually worked, I tested it, it worked. So then I beat it in Python, I basically said there is the main travel agent to simplifying. And then there is the supervisor, the one that tries to catch undesired responses like hallucination and I orchestrated the dialogue between them and it worked. But I then realized that I would very quickly need another agent, so a supervisor is an agent, the travel agent is an agent. Today we all call it a general world or a framework, that's what it is, it's an LLN with a base prompt, that's what an agent is. So the first agent was a travel agent, you are a travel agent. The second one was the supervisor, you are the supervisor, and you need to look for hallucinations. And so we start building what's evolved into a full-blown, agentic framework that we call internally in a van cognition. And I recall in a van cognition because it takes LLNs that have the sparks of intelligence, but at the same time you can fool them and they may lie. And it turns it into a functional cognitive system. So Ava, our virtual travel agent that is built as a cognition application on top of cognition, does not have these hallucinations, does not, it has what we call also internally, it has zero critical hallucinations, I can rely on it. Critical hallucination is what we define as critical for us. For example, when Ava tells you that the cost of the upgrade would be 48 bucks, then no matter what, no excuses, 48 bucks is what the user will see in the statement of the bank or the credit card. Yeah. You know, we know compromise. You have to trust it. So you've built this trust layer through the conversational app. Because I think about travel admins who are looking at thousands and millions of transactions across their employee base, who did not book within the policy guidelines, who's always booking the day before travel, things like this. How did you approach the build versus buy here? Okay. So for Ava, the conclusion was to be prior to building Ava. Ava, the full blog. We had a live version, lightweight person, Ava. I also realized the potential of applying generative AI or conversational experience around data. And so I beat it. I literally beat it in Malibu. And I remember that Christmas Eve, it was ready. I remember it. 34th of December, 2022. And what did it allow? It was the trend on my computer. It was not production ready, but what? It allowed me to ask questions like, how many new customers did we have? Did we own board last week? How many bookings did customers, such and such, did break it down for a question, break it down by booking tag, hotels, flights, cars, rail, etc. And it worked. And so we decided to productize it. But I was in the beginning of my journey to understand this technology. So I kind of dumped it, that we proof of. of concept to one of the teams and they tried to figure it out. Long story short, it was not a success story. And I think that we still have some leftovers there, but it was not I moved on. They were struggling to productize it, et cetera. And ultimately, we actually end up using your product. Like two years later, we ended up using ThoughtSport. We used ThoughtSport, but we also applied generative AI, capabilities that you guys have been into your product. And now, I don't need to worry. Now, by the way, if something doesn't work, I can easily complain so that the benefits of the buy, but back in line, it is part of our product. In production, it works. And it generates value to our users, a value that we recognized more than two years ago. (upbeat music) Several of our guests have talked about the importance of trust, trust in data, trust in leaders, trust in AI. Trust is a core value at ThoughtSport, and just one of the reasons the company sponsors this podcast. So that you see us as your trusted partner in this generational shift to AI. Transparency is critical in fostering trust. And this is why ThoughtSport's CEO, Kate and Carcannus uses one spot and makes it accessible to the entire company. Here's Kate's advice to other leaders. Yeah, and so it sounds like for sure, as you work with CEOs and you advise them, you would tell them, no blessed power points, static, anything, real time, anything else that you would advise CEOs or CDAOs supporting them. What would you tell them? You know, sometimes all of us are really smart people and I'll tell you very simply, don't let perfection be the enemy of progress. That is where I see most people. I wouldn't use the word stumble, that's a harsh word to use, but slow down. And then slow down, sometimes results in stall. And stall sometimes results in, well, nothing's really happening. So perfect, what I mean by that? Look, when we started like, do I have all the metrics beautifully defined, KPIs beautifully defined, and every part of my data and business perfectly articulated in one spot on day one? No, I do not, but that's okay. Perfection cannot be the enemy of progress, especially with AI. But two is culture. I think so, these two and perfection enemy, it's part of culture. It's giving you a characteristic of the culture, if you may. But let's zoom out from that and talk about culture because I feel like, and we experience this, Cindy, you sit in so many of my meetings, we have experienced this, right? When you tell something like, when we were trying to use data for our meetings, you have to have a culture of, you know, you have to walk the talk. I have to show up to that meeting with one spot, and then my team does it. Or two, we have to have the culture of trust. Do you know, Cindy, that I have made one spot, everybody in Todd's spot has access to all data. All our book, like, we want everybody to have access to data. The culture of allowing what I call experimentation to flourish, because that is important. There's a lot more every organization. I would say that don't try to build a spaceship every day. I do indeed use one spot to see how my customers are doing and how this podcast is performing. I hope you have been inspired by all our guests throughout 2025, with practical insights, lessons learned, and personal journeys to becoming data chiefs. As always, thank you for tuning in and for the privilege of your time. Thank you for joining the data chief, as we explore data and AI leadership. To continue your journey, subscribe to our podcast on your preferred listening platform, and connect with me on LinkedIn or X at the iScoreCard. If you are loving this podcast, please rate or review it, so others can also discover the podcast. The data chief is brought to you by Thoughtsbot, the Agentech Analytics Platform company built for everyone to create personalized insights, drive decisions, and take action. Learn how companies like Lyft, Brambles, and Avon are leading the way at Thoughtsbot.com. [BLANK_AUDIO]