What Shared Intelligence Really Fixes in Enterprise AI - with John McMullan of Observe.ai

[Music] Welcome everyone to the Emerge AI in Business Podcast. Today's guest is John McMullen, head of Product Marketing and Strategy at Observe AI. Observe AI is a CX native AI agents platform that enables enterprises to deploy specialized agents that understand context, reason and take action across the customer experience life cycle. The platform includes orchestration, integrations and governance features that support consistent and reliable automation across CX operations. In today's episode, John unpacks why so many enterprise AI investments in customer experience fail to deliver measurable returns. He explains why roll out with our operational discipline leave frontline teams rechecking the tools work and how a shared layer of interaction data lets every AI agent and human pick up where the last one left off. Today's episode is sponsored by Observe AI. In this episode, we cover how AI investments in customer experience fail to deliver measurable returns. To go deeper on this topic and learn how to identify real AI trains by tracking where bench of funding is flowing and by listening to how leading CEO is described risk and competitive strategy, download our free PDF report three ways to discover AI trains in any sector. At Emerge.com/AIT1 that EMERJ.com/AIT1 to download your copy. Now the conversation with John. John, welcome to the Emerge AI and Business Podcast. Thank you. I'm excited to be here and look forward to our conversation. Yeah, I think this is a conversation that really needs to happen because we have gone past that part of what was it two, three months ago where we were still talking about oh, we need AI and why do we need AI? And I think everyone now has some sort of AI build into their platforms. And I like to think of CX as the first part where we really saw AI popping up even before what it is now. But something that we keep on hearing these days is that there's a lot of AI interest. There's a lot of budget, but the results still aren't there. So I want to get into what is that bottleneck? What's been preventing enterprises from getting real measurable value out of their AI in for the customer, in front line, and their back office workflows? The noise is feverish, right? In the fun part, I've been in conversational AI somehow some way, 15 plus years. A lot of these promises aren't new, but with obviously the ad end of LLM and these frontier models that are coming out, there's a lot of hype and a lot of FOMO driving a lot of these decisions. So companies have made investments, jumped in, feet first to say we're going to adopt AI. Some have even gone as far as saying we're going to be AI native, but a lot of that ROI has not come to fruition. And I think that really stems from interest without discipline. And what I mean is it's not that AI is not prevalent and available and there's certainly no shortage of problems to solve with AI. But this isn't the old world where you'd buy like a SaaS application and say, oh, we need a new billing system or a new CRM and grab it off the shelf and implement it or cut over from your old one and it just works and goes. AI is the continuous learning cycle, a continuous investment where the system is going to uncover new edge opportunities, need, constant tuning and tweaking. So there's been a lot of interest. I think companies have scrambled the bolt on AI or put it into workflows, but they don't fully understand what it means to adopt AI and what goes behind the scenes to make it work. And yeah, we can get into some of that more, but I think it's not an uncommon problem, right? Everybody's moving fast, there's a lot of noise, there's a lot of hype, but really it comes down to what are the tangible next steps for a business? And that comes with thoughtful experience design, which we can talk about here. Yeah, I want us to get into that. And officially, I think what you just mentioned is very relevant. That AI FOMO was so real and thinking about, you know, the platforms that we were used to back in the day, it had the functions that the box had and that's it. And that's what we expected of it. With AI, it's kind of like introducing new employees to your organization and they need constant learning and they need to professional development is part of their annual training and things like that. So we should be thinking about it. Is that the only friction points or the only reasons that we see our CX environment really flourishing and getting real ROI from AI? Complicated question, right? Everyone's unique. So there's definitely different challenges, different headwinds, you know, how it comes to manifest in everybody's individual organization, but a prevalent problem I see with the privilege I have of sitting down with, you know, dozens of customers every week and understanding what they're working on. Number one is write people around the table and this isn't necessarily like a council of AI or your AI steering committee. Those are super valuable, but they are not your implementers, right? Those are the people that step back and say, what are the ethical use cases? How do we want to think about that? But really that tiger team or red team that can jump in and it's not always the people you think it is to get into the mix. And that frankly is number one that we miss of like, okay, we want AI to deal with customer conversations or we need AI to analyze, you know, trends within our customer base. An AI council thinks about it globally, but your business analyst, your telephony engineers, your best service agents that know what customers think. Those are your tactical people you put on the red team that actually show up and have the insights of here is what the best case golden data set should look like. Here's how we build the system. Here's what it needs to know and doesn't need to know. That's usually a misstep is thinking almost too philosophically about the problem, rather than tactically what needs to go in it. So that's usually the primary issue I bump into every week is getting the right people around the table and really thinking about it. The second quick adenum there is fear of failure or maybe, you know, scared to start. It's another way to frame that. One phrase I always say is momentum brings momentum, right? I mentioned AI as a continuing iterative process. A lot of people show up and, you know, oh, we could do this or we could do this and they try to like over-engineer all the things a LLM could do and how this new agenteic future can take shape when really you need to build those battle scars. Like you need to pick something small and you need to commit to it with the understanding that it's going to more than likely fail, more than likely, you know, be challenging. But what you're going to do is figure out, again, who needs to be involved, figure out where it failed and why it failed and then build back stronger the next time. That momentum helps you move faster forward and it really is that hockey stick effect of companies that get going and with bumps and bruises along the way and all of a sudden they understand what it takes to implement AI and you see it take off and skyrocket across the organization and across the CX staff. Yeah, and that's exactly what we want. I think one thing and I'm going to just ask one more question on this before we move on to the fun part of the conversation where we can look at what solves for this. But one thing that I'm wondering about is one of the problem areas that we've been hearing about from our audience is that even though they have all of these different AI features built onto their system, it still feels like the humans are doing double the work. All right, so to do a full QA, they don't trust the systems. All the system is not capable of giving them something that they can trust. So they still have five to eight people evaluating transcripts, doing manual QA. What is it that causes our organizations or enterprises to feel that they still need five to eight humans to double check the algorithms work or to double check the AI's work? Yeah, I mean, trust is or not given. So anytime you introduce a new system, you know, your humans that are doing this day in and day out need to verify and that's just a natural cause of, you know, implementing a new process in a new system. That challenge again is not unique, but does rear its head in many different ways. And what it ultimately stems from is fragmented systems, fragmented processes and not thinking I mentioned earlier, but that end-to-end design. So I'll give you an example, you know, some of the solutions I work on and we have provides kind of after call wrap up work. So when a call concludes between a customer and a frontline rep, it fills out the disposition. You know, what are the key notes? What are the actions that need to be taken? It can trigger workflows. All the things a human do when you hang up the phone. Went on site and sat with some agents where this was rolled out and those agents were actually taking the call transcript copy and pasting it into a free instance of chat GPT and then measuring those notes back against their notes. So what they're taking sensitive company info out of the environment. And two, they're doing double the work that's already done for them, but they just didn't trust it because it wasn't necessarily rolled out and introduced the right way. And come to find out interviewing these reps, it was a non-mandatory training. They didn't understand why this new widget showed up on their desktop and it goes back to that process design of you have to work with your teams, you have to introduce them to why this is helpful, how it helps, and ultimately what's behind the scenes that's verifying this information. So you build that trust really through that thoughtful design process and pull together some of those fragmented workflows to streamline. And once that's built and shared understanding has had, then you can really take advantage of it. But again, getting to that point is only found out after you build some of those battle scars. Yeah, so I'm guessing that for the leaders thinking about, okay, we have AI, but it's not giving me what I want. It's not giving us the ROI that we expected. This could also be part of that. It was never deployed. correctly. And now they're too afraid to think about it from end to end because the little bits and pieces that they have already makes them feel nervous. But let's get into the fun part of this conversation and talk about the AI capabilities. Setting aside the failures for a second and thinking about what's actually working, what kinds of AI capabilities are proving most effective at improving reliability and cutting down human dependency across all the different workflows? There's obviously a lot of noise. The concept of a chatbot is nothing new. Concept of like smarter knowledge for the frontline teams, not new, but certainly more powerful these days. I actually kind of challenge that statement when I go out into the wild and think about this. In the old world or product-based world, you'd show up and be like, "Here's what my product solves for." In this new, agentic world where you can kind of build what you want and shape the experience, it's more about where are those friction points, right? What is the customer experience today? Having that real world understanding of, wait a minute, why am I waiting on hold for 18 minutes? Why am I pressing three for billing or stating in two short words the problem I have? Why is a frontline agent getting a coaching report every two to three weeks that's cobbled together? Starting with the problem and the friction points from the real world helps you understand where you can improve the process. Instead of trying to fit a product into a solution, start with the problem and work backwards. That ultimately, I think, lends to more successful adoption because you get a better view of what can be solved and really where the value will come from. Great. That's a good starting point and I'm thinking, every thinking end to end, we've both the trust, we started with the real friction point and you said earlier, failure is kind of inevitable. What happens when the AI fails us in that end to end process? If it happens early, if the fail happens early, does it get carried through to the end of the process? How do we then go back and gain that trust back? What do we do in those situations to also minimize it afterwards? What are we thinking here? No, I think it's an important note and I do want to circle back because when we say failure, we picture the spectacular failure that ends up on the front page of news articles or the evening storylines. I'm talking about much smaller failure, right? Like the internal test, think of it back to doing homework where you spent a lot of time and you got a B instead of an A. It maybe didn't change your trajectory in life, but at that moment in time, it felt like a failure. A lot of these failures are unseen on her. They're standard work that goes on in the development cycle and the building process. We want to make sure there's a lot of capabilities and recommendations I may have we would get to, but to be clear, these failures are not the throw it out into the wild and hope it holds together. Spaceship falling apart level failure. This is like, dang it, our test on Tuesday didn't work. Let's pick it up Wednesday. Let's pick it up Thursday. It's those little things that compound. When it gets to, how do you build and how do you prevent? There's a lot of tools and technology that are coming out right now. I think one of the hottest companies coming out of Y Combinator is AI agent evaluation and the premise of that is if everybody listening went to your favorite LLM and we all asked it the same question, we may all get different responses. How do you prevent that as a brand when you have a thousand customers come to ask you, you can't have a thousand different answers out there. You need to make sure it's adhering to policy. That's some of the capabilities built in for testing to make sure you can simulate all of those conversations, those edge cases, put them through and ensure that you're having consistency on every interaction or on every coaching scorecard or real-time recommendation to your front line team. Once you get through those and build that and test it and simulate those failures start to turn into wins. Again, these unseen small failures and ultimately it leads to a more durable product that you can trust in the hands of customers, front line or operations teams. I'm taking it back to thinking about not just deploying it in one specific part or one specific part of the customer journey into being the main goal here. How do we know it's ready? Is that based on our foundation, is the foundation built on the context from our data, our current data, is fragmentation going to play a role in there? How do we make sure that we minimize the failure from the start? How do you make sure it's ready? He's always a tough one. But ultimately, you are building this based on your business processes and the best existing data set. So starting with data readiness, right? If your data, bad data in is bad data out, making sure that your core systems are accurate and accessible for the AI to use. That's things like clean knowledge bases, making sure all that information is indexed appropriately and ready for AI consumption, making sure that your backend systems, your CRM, your ERP, your building systems, your scheduling systems, all are wired in. Should the, that be in scope for your AI agent and running those tests against it, making sure you're best humans. Again, I mentioned getting the right team together, but making sure your frontline reps, who already handle hundreds of these conversations every week, go and actually test it. They know what good looks like. They also know the edge cases and points of failure. So once you get through that, the final step is really simulation and testing, making sure you maximize the load on these systems to really uncover. And while it may be some synthetic testing, it is based on real conversations. So that is where ultimately, how you build some of that readiness to get out there. Yeah. And if we're thinking about data readiness, do you think unified data is going to make a unified platform more of a reality? Yeah. I mean, that's going to be the next, that is the next horizon frontier we are chasing. You already see it with some of these frontier models that are out there, you know, whether it's open AI or anthropic, it is the long term thinking and planning for multi-step procedural operations. Sort of like loan applications, right? Or going from start to finish on an insurance claim for car accident or a healthcare booking. That all stems from a unified shared data layer. And what is important for anybody to playing AI is to understand what that means and what that looks like. So going back to like bolting on AI and the initial ROI question, you know, oh, we threw out a chatbot. Are we getting ROI on this? It's like, well, you know, zoom in or zoom out, like cool. Some people talk to it. Maybe it saved you a few calls, but long term humans will still need to talk to humans. There are some conversations, some instances where a human is needed. And you need to make sure that that data is on a shared layer. So thinking from an AI first perspective, it's not just about building an AI widget or capability that's bolted onto an existing processor system. It's what that AI is pulling from and where it pushes to so that the next system can pick it up. And whether that next system is a human or a different AI agent workflow, that's ultimately where this is leading to. And I'll give you an example, right? I ordered a package. This is also probably real, but, you know, I think universally fits for everybody, you know, you order a package. It was supposed to be here Tuesday. It's Wednesday. It doesn't show up. I, in the middle of a work day, fire off a note to the online chatbot. And it's like, oh, looking into that, I don't like the answer. I chat again later in the day. I send an email. I finally pick up the phone by Thursday to be like, where the heck is this package? I ordered. And before I get to a human, I have to restate my name, my order number. Tell them I've already opened a claim. Tell them I already supported it. That's what I mean by unified experience. In this world, that web chat should enrich the data. My email should have created the claim number. And the human should pick up the phone right away saying, "John, I know that your package is supposed to be there Tuesday. I'm looking at it now. It is on the truck. It'll be there at four o'clock." But instead, I have to wait on hold. I have to resubmit my info. So that is where the vision has to come from is, can we build that shared intelligence layer that no matter which AI agent or workflow, human or not, has the same operating data as the preceding one? As a very frequent online shopper. I am voting for that. Where can I put in my ballot? That needs to happen. But also, so I read this article, and I'm not going to mention the source right now because it was a bit pessimistic. Where this writer said, it basically was about a concept very similar to this, but this writer said that connected AI systems sharing context is just a nicer way of describing more integrations. And from what I've seen based on my preparation for this conversation is that's not necessarily the case. So can you give us the difference between the two? What is the difference between what we're describing as the connected intelligence versus just more integrations? Yeah. I mean, and that, I will say, it varies because if you are viewing your solution or your approach as one-offs or bolt-ons, as I mentioned, of, "Oh, we're going to deploy this thing." And we're going to deploy this thing. It's also been said maybe by a different author, but like, IT is going to become the HR of AI. A lot of acronyms in there. But essentially, IT is going to have to manage all these different systems. And as you can imagine, it's going to be wildly complex with different policies and rules and chaos is there. The way to get to your optimistic view, right, is to build that interaction intelligence layer. And what that system is and does is take in every touch point that is currently happening and sits in between your core systems and your AI agents. That is the interaction intelligence layer. If you are trusting your CRM to go out there and have customer conversations, then your ticketing system or your building system is not gonna have visibility. And frankly, I don't know if these CRM applications or ERP applications that are building AI workflows have ever thought about integrating with a contact center application or a workforce management application. So it's almost thinking of it as a new layer to your stack. What is that interaction intelligence that can build the integration once into those core systems that are still very much needed but also can hold that context for a long time. Hold that memory and information so that when you do deploy AI agents in any phase of your business, it can go in and pull that information right away. Again, comes down to thoughtful design and the way you want to approach it, but given it's an iterative process, that's frequently where I see or the pessimistic side play out of, oh, we launched something and at some point it's gonna dead end or just be stuck in that silo until you step back and think about the interaction intelligence design. - Yeah, so it's not just about being optimistic or not it's about thinking about it intelligently as well. It's been great diving into this with you. We're gonna have, I'm gonna put in one last question before we wrap it up and that is our leadership question. We know that we have listeners listening to those thinking, yes, we had that AI FOMO. We bolted on everything that we could think of at that moment. Everything that our competitors were doing, we were trying to do better and now we don't see the actual ROI because nothing is connected. Nothing is speaking to each other. So for a leader listening and willing to admit the problem, let's look at where do they start at fixing it? What are the concrete first steps that our leaders should take right now to start closing that gap and getting real ROI? - Yeah, well, the good news if you have done that, you know, my best advice is start now. Get momentum, bring momentum. You have something to learn from. You've at least tried it and seen what works or maybe already have some positive progress going. So if you haven't started or if you have that FOMO that's driving you, hands on keyboard, right? Figure out what is possible, go play with these systems, even the open models, anthropic clawed, grog, whatever your favorite is, just go push the limits and see what's possible. That gives you so much familiar it. Other piece of advice is your existing processes and experts are in your organization. You are probably already have tens of thousands of great customer calls where you know it went right. You know the knowledge articles that were used. You can surface that information. Using that to help you understand the friction points and what to build, how to build it, where it's gonna make an impact exists. And it's not like a needle in a haystack search. Go talk to your front line reps, go talk to your supervisors, your managers. They can tell you their pain points and where they can see some automation improving things. They'll give you that set. So yeah, that's ultimately where I think, if you go look inside your organization, you can find a lot of those points. But as long as you keep building, keep learning and pushing forward, you're gonna uncover a lot. And then it comes down to, again, thinking through the process and continuing to build an iterative fashion. - Yeah, and I guess also updating the process as you go along and finding new and better ways to do things. And the one thing with AI is it highlights the faults that we've had in the process all this time, but no one really picked up on it. So being involved and constantly involved, I think that's also gonna make a big difference. So for our listeners, I think our big takeaways from what John has shared with us today is the first thing is, the reason why your AI is not paying off is not necessarily because of the technology. It's launching without having operational discipline. I think I really like the way that John framed that for us. And then the AI capability that will move the needle is having that intelligent layer, having the connection between all the different platforms if you wanna call it that. And starting with a single workflow, it worked in the beginning, but-- - One more thing to chime in. - Yes, John. - Obviously it's gonna come from my perspective inside as vendor, but having a trusted partner that goes through this, the battle scars, the learning, building things out. It's not that your team can't do it. Freak, we meet with a lot of customers and people every week that have teams focused on this. A lot of this is possible in this new future where the AI can almost build itself. You're bringing things out. Having a trusted partner is someone that helps you see around corners. They've already built the battle scars. They've already gone through it. And it's that ongoing management and maintenance you want to do that you mentioned, we deployed this system. And now it's, I moved 10 people from this process that we've automated, but now I got 12 people working on maintaining the AI. Having a trusted partner can help you streamline that process, uncover things, and is the owner of that program. You still have the technology, you get the outcomes, but somebody that has been there and done that is hugely immense, so you don't have to go through all the peaks and valleys of AI development. So that's one more thing I wanted to throw out there as a best practice is find your trusted partners who can help you see around corners that have been there and done that. - Yeah, absolutely. I think sometimes we forget that our internal teams, even though they could be brilliant, they're very boxed into a certain environment where if you partner up with the right partner, they have experience across different fields, across different organizations, and they've seen so many things come and go, and they've seen what works and what doesn't. And they might be able to borrow some creativity from a previous project and apply it to your project, which is something that your internal team might not be able to do. So very much agreeing on that point, and I think it's important to make sure that you pick the right partner as well. John, it's been lovely having you on the show. Great conversation. I'm very excited to see where this is going, and check in with you in a few months' time and see how this looks and how it's changed so many organizations, workflows, and the way that they see and experience their AI ROI. - Absolutely, greatly appreciate the time in the conversation. This is a lot of fun. (upbeat music) (upbeat music) - Rapping up, let's look at the three key takeaways from the conversation with John. First, trust comes from the role of not just the model. Teams need to understand why a tool is in their workflow and how its output is validated. Second, bold with the people closest to the customer, the analyst, engineers, and top-of-form groups, not only the leader's setting policy, and finally start small. Expect some failures in tasting, learn from them, and use those lessons to accelerate the next deployment. In this episode, we covered how AI investments in customer experience fail to deliver measurable returns. To go deeper on this topic and learn how to identify real AI trends by tracking where venture funding is flowing, and by listening to how leading CEOs describe risk and competitive strategy, download our free PDF report, three ways to discover AI trends in any sector, at emerge.com/ait1. That's emirj.com/ait1 to download your copy. On behalf of the team at Emerge, we'll see you in the next episode.