How Enterprise Leaders Should Measure the ROI of AI - with Darko Todorovic of HTEC

[Music] Welcome everyone to the image AI and business podcast. Today's guest is Dawko Todorović Chief Technology Officer at Edge Tech Group. Edge Tech is a global engineering firm focused on AI, centric software and hardware development. Working across financial services, Metic, Automotive, Telecom and Enterprise Software from more than 20 engineering centers. In this episode, Dawko examines why enterprise AI investments frequently produce successful pilots, but store-winded fluid-ed scale. A failure rooted in notting technology, but in organizational readiness, talent gaps and the absence of defined success metrics. He walks through how leaders can establish cost-con-unit baselines, build ROI measurement frameworks and treat AI agents as co-workers rather than tools. A mindset should be obvious is the critical precondition for scalable, measurable impact. Today's episode is sponsored by Edge Tech. In this episode, we cover how enterprise leaders can measure and prove AI ROI after deployment. To go deeper on this topic and then how to identify real AI trends by tracking where venture funding is flowing, and by listening to how leading CEOs describe risk and competitive strategy, download our free PDF report. Three ways to discover AI trends in any sector. At www.emerge.com/ait1. That's emeorj.com/ait1 to download your copy. Now the conversation with Dawko. Dawko, welcome to Emerge's AI and Business Podcast Studio. Thank you. Thank you very much for having me, Alan. That's great. I'm excited to talk to you today. I know that you've been inside engineering and delivery for quite some time, which means that you've been on the receiving end of what clients actually ask for, what they think they're measuring versus what they actually end up being able to show. And it always feels as if the ROI conversation question is very straightforward at the start of the project. We always have this direct idea at the beginning. We're going to do X and it's going to the level Y. But then somewhere in between the commitment and the close-up, our clarity kind of disappears. And we kind of lose track of what it is that we want from this. From your perspective, what is actually happening in that gap? Well, that gap is a multifaceted problem at this point of time. First of all, the technology is not mature enough. You don't have the basic components that you can choose from. So from the implementation side, it's always a question of how to go about it. We know that we need data, but nevertheless, we think that we have the right data, right sources, that everything is one data lake, etc. At the end of the data, it is maybe not usable in a proper way, but the AI agents. So on the technology level and technology layer, we need to make sure that we have the right setup for the problem that we are solving. Which brings us to the point is we shouldn't start by implementing technology for the sake of technology, and just putting a sticker on top of the process that it's AI process. We should completely rethink all of the processes that we want to augment with AI and really see how we implement it with making sure that AI agents are not the tools anymore, but they are our co-workers in the process. So when you start from there, there is a higher likelihood that the success of the implementation of the AI-enabled process is going to be successful, and that you can see the tangible ROI. Finding these processes is the first task that every organization is actually gets as a task. So you have a task force that is formed inside of the CIO office or a CTO office. It depends on the maturity of the company that we need to bring up our efficiency with the implementation of the AI tools, and you are asked to do that. So first, what they do, they find a process, they say, okay, this is going to be the process that we want to augment with AI. Then we are going to choose the tools that we are going to use in technologies, etc. And then we are going to implement the perfect process with the help of AI, and in demo this works perfectly, then we are going to deploy to a customer or a customer, depending on the maturity of the company and what the company actually does. And once that happens, usually things goes out. And this actually is a function of not the technology, the advancement of the technology that is implemented, not the function of the wrong process that was selected, is the function of the changes that need to happen inside of the organization. But more importantly, within the people that are using, that are on the receiving end of these AI agents to help them actually be more efficient in the processes they are delivering, or they are working on on day-to-day. So in a nutshell, I think that hearing the organizations realize that this is a huge change management problem is very important at the beginning of any AI and all. So would you say that problem comes in when organizations, if the technology works and if it's been implemented correctly and they did look at the problems, the business problem beforehand, they did look at different options and they still, they're still not able to prove what actually delivered from it. It worked, but they can't prove what it delivered. Is that a measurement problem? Are they measuring the wrong things? Is it a baseline problem? Or is it a conversation that was never had before they kicked off the project? Yeah, I think that most of our organizations are now implementing AI, not for the right reasons, basically, because of the most, in most of the occasions, because of the fear of the missing out, because they see that that organization implemented AI, they have efficiency of 200 percent or 400 percent, etc. And then you say, and this is normal, quite normal. What are we missing? What are we doing wrong if we don't have this kind of efficiency in the organization? More importantly, some of the organizations are asking themselves, okay, this organization based on the data created more revenue for their company. They have new revenue streams that they opened up, but ultimately, if you don't have the people capable in the organization to understand the technology limitations of the technology and how it should be implemented inside of the process where you have coexistence of AI agents and humans, they are going to fail. And this is the first thing that they need to realize, they need to have the right talent in the organization. They need to invest into education and understanding how these tools can help them. The problem is also that they are bombarded with the different agente platform, different approaches, promises of wonderland. And then it's very hard for them to say no. When somebody says, "Hey, I'm going to help you achieve these results in three months, I don't know, year and a half, but in three months you'll have this, you'll have that." And what happens is that they actually burn themselves in a couple of occasions. So when the technology is mature, when there is actually a real potential case with a real ROI, they are reluctant more to make the decision. And we have seen this happening in digital transformation as well. This is not the new for me or for our company. We've seen that process, but that process was 10 years long. Now everything is condensed in 18 months. And then pure cognitive load on the decision makers to make decisions very fast with little input, little to no input. And where nobody can for sure tell them that this is going to be a successful program, it's very hard. And we have to be very empathetic with these people. That's why everybody is opting in for POCs. But what they don't realize with the POCs is that the POC can work really good. Why? You choose the right process. You choose the most senior people in the organization, the best people that you have, that have already a decision making framework, good judgment. They know when the AI agent is flawed or the result or the outcome of the AI agent is flawed, they can correct it, etc. So they are an expert thing to room that can make decisions. But then if you go out of the POC and you go to the production, on the production level, if you have to release these agents to a thousand of people in your organization, it's completely uncontrollable. And usually the results are not compelling. And this is where the most of the organization that come to the POC and the POC is successful is actually failing on the scaling and implementation across the whole organization. And I think that this is the critical point in which you have to understand your organization, how it needs to change, what is the new role of the people in your organization. And usually the new role is not to use the AI agent as a tool, but use the AI agent as a coworker as your junior assistant. That is going to help you achieve something, but you have to make the judgment calls. You have to be the human and the decision factor in the loop. So until organizations realize this, it's going to be very hard to have scalable solutions really don't wonder in the organization. It's almost as if we're still using humans. We're just using AI to shift or change those normal humans into superhumans. That's the idea that you're giving me. Do you think the problem with being able to measure ROI and present it after deployment? From what you're saying at Sciences, if there's a gap, or if there's a reason why we struggle to prove ROI, it's because we did not have the correct people in the conversation at the start or the correct project managers from the start. So they just hopped on the bandwagon because they had to and because there was pressure applied to them. But the reason why they can't show the ROI is because they don't really know where to go and find it. Well, it's very hard. It also depends where you're implementing AI. If you implement to affect the bottom line, basically you're looking for deficiencies. And measuring ROI in very well-defined processes in the organizations, it's easy if you have the first of all very well-defined and then on top of that, you have the cost per unit defined in your organization. So if you have, for example, RFP documentation that you need to issue or you're issuing an RFP and then you get the offers, et cetera, the whole debt processing. So in that type of a scenario, you actually know how much time you are spending of each person and what is the cost of that particular RFP. And if you have thousands of those, then you know how much money you're spending per unit. If you can automate parts of the processes or the whole process of the RFP as an organization, you can easily see what are the benefits, what are the efficiencies that you're getting. The investment, you have to look it from two sides and in two components. One component is initial NRE investment, so basically CAPEX. And then you have to see how much is execution of these agents or air tools is going to cost you per each unit of measure. So if you have those two, you can easily say, hey, this is my investment. And this is what I'm getting out of that investment. This is the efficiency gain. On the other side, people are still reluctant to go into new value generation investment with the AI, except in the cases where organization is actually delivering AI products. So on that side is going to be even harder to see the ROI. Because if you're creating a new product line or you're creating the new service based on the data that you have on your customers or you want to upsell something, it's even in a manual world without AI, it's very hard to see what is the ROI and investment that you're making. So it's basically a gut feeling. It's institutional knowledge in the organization, judgment calls, decisions, investment committees, et cetera. Everybody is creating a model. So when you are creating the new value, you are creating a model based on the number of assumptions that you put into that bundle. We cannot expect from AI to do that for us. So in that case, we can only treat it as a tool. We cannot treat it as a solution to a problem, hey, I want a bigger ROI, a bigger revenue, or I want a new product that I want to sell to the market. So I think it's very important for people to understand and for the organizations to understand where these tools are going to be efficient. So if we are talking about the measurement of the ROI, of course, you have to have a metric. You have to have the KPI that you want to achieve. You have to show to yourself, actually, that the investment that you did is worthwhile. Where is also the problem? The problem with the pressure is that you are going to be under pressure to even fabricate the results of the AI investment, because you are responsible to do it. And then you are going to push for it to be successful. And sometimes-- and not sometimes, but a lot of times, people are going to look at it as their own child or their own responsibility. Or this is going to be looked as a failure. And out of that fear, people are going to do things that are basically not aligned with the company goals. And this is happening as well. So the result of this is why is this happening? There are no standardized metrics. There are no standardized approaches. There are no standardized stacks of technology. Everything is changing so fast that it's really hard to make these type of decisions. So people that are awfully risk-averse are not going to make them, which I don't really know is it a better or worse instead of making a bad decision. So I would experiment and do a bad decision if needed. But then the organization needs to be very supportive and to provide a safety net for the people that are actually going down that path. It makes a lot of sense. And I know our executive listeners, they understand what you're saying. But they still like to do-- I would call it tracking in real time. They would like to be able to track ROI, whatever ROI means for them for the specific product in real time. How do they do that? Is this going to be done on a dashboard? Is it a process that they need to monitor? Is it a governance structure? Or something else? How can we track in real time? Well, it's good that we have AI. So it's very, very fast to make any kind of a dashboard. You can pull out the data very easily. If you architect a solution in a way that can provide you valuable data in each step of the process, you can monitor the token consumptions per agent per person. You can actually see what is the bill at the end of the month, what kind of efficiencies you can make in the utilization of the AI tools as well. So if you have a sophisticated team, sophisticated team is going to give you the same solution for the last amount of spend, or less more amount of token spend, which is really important. If you don't have a sophisticated team, they are going to make a solution that is costing more than it is supposed to be. But this is going to level up over time. And serious organizations, they see these serious advantages of using AI. I'm not going to be worried about the cost in the first couple of years, because the cost is going to go down over time. The sophistication of the tools is going to go up. So if you want the latest model, if you want the latest thing on the market, you're going to pay premium. But if you look at what ChatGPT gave you as answer two years ago, compared to what you're getting now, it's like really, really different, comprehension, feeling like you are talking to a person, et cetera. So it's completely different. But that ChatGPT two years ago felt like wonder for everybody. So we are quickly forgetting how surprised we were when we first started using the tool. And now in two years from now, if you project yourself, in two years from now, you say, hey, this is the perfect tool that I have today. That tool is going to be very inexpensive to be used. But it's useful. It's to in today terms. So when you're calculating ROA, you have to calculate also how much you're going to spend in the future in that particular process. There is also a fear of exploding cost of tokens. And we need to calculate that in as a risk factor when we are building a model for our company. One thing which is really interesting also is that if you have successful scalable AI agent on the whole company level in one of the departments, in one of the functions, for example, what's going to happen is that the rest of the organization is going to look at it as a-- I want that. But that cost, that implementation cost, and then cost of running the agency is going to explode. And you have to calculate it in as well into your budgets and everything that you do. So you're going to think, how? I'm giving to some of the foundational model companies X amount of money. And now I have to give 10 X. 10 X is actually too much for our business. So what we do about it, what we do about inference cost, et cetera. These are already the questions that some organizations are asking themselves. But these are more mature organizations that see all of these efficiencies coming into play. But at the end of the day, the bottom line is the ROI is going to depend more on how you set up your organization and less and how you educate people and how much investment you make into the education of the people in your organization, how to use the tools in the proper way, how to use the AI in the proper way, then on the actual implementation and the technology behind. You touched on the sophistication of AI in this answer. And I wanted to just quickly dive in there as well for our executive listeners that need to sign off and budget and need to sign off on AI projects. How important is the sophistication of the tool for ROI? Because they obviously have budgets to take into consideration. Is it necessary for them to go ultra sophisticated? Or can they deliver acceptable ROI with middle-level AI? [LAUGHTER] I think it depends on the inspiration of the company and that individual. And if they were satisfied with mediocre results, with their own team and people working on the process, they're going to be satisfied with me, their results from AI. So that's not going to change. Like decision makers are not going to change their way of thinking overnight. So it depends on how critical that process is for the company. And if we go even deeper, you can see what separates successful companies from the less successful companies. But even less successful companies have their own place in the market. And they're satisfied with that. So at the end of the day, it depends on the aspirations and depends on how the organization is set up more than what they are going to be satisfied. - Okay, it makes sense. So if they have tighter budgets and I cannot necessarily afford the sophisticated AI at this point, they can still expect some ROI with a mid-level tool. Just what you put in is what you get out, is that the right assumption? - I think that the readiness of the organization to implement AI has to be looked through two dimensions. One dimension is technological readiness and the data readiness. And there are perfectly good metrics that can actually give you these measurements. On the organizational side, we already talked about it. So you have to have the organizational readiness to change in a way that's different from any change that they have up until now. So if you look from these two perspectives, you have to find the place where your organization is and be realistic about it. And if you're not realistic about it, then you are going to expect more and get less. So it's very hard to give you a definitive answer on this one because it depends on many different factors that you have in your organization when you're making these decisions. And when you say sophisticated AI versus less sophisticated AI, it only depends on the use that you are going to deploy this AI solution. So if you go with the latest model and you think that it's going to give you the best results, that's the wrong assumption out of the gate. You have to have the right model with the right data, with the right setup to give you the results that you need that are specific to your organization. The beauty of this AI revolution for the organization is that previously all of the organizations and all of the standard processes, like HR process, procurement processes, ERPs, et cetera, were built in a way that when you implement them, every organization is looking like the other organization have that tool. So all of the customizations were painful, really painful. And you know that you can be more efficient, but the software solution in order for you to change it is really expensive at the next time. You cannot do it. And it's very hard for you to actually build something that is custom for you. That's why you had all of these SaaS businesses exploding in the last decade. But the problem with that is that you are putting different organizations in the same bucket. With AI, the ultimate promise is that I can build my process for myself in very efficient and easy way and it's going to be easy to implement it. That's the promise. That's the promise that AI is giving to the organizations. But when they start implementing, this is where the problems are arising. Because you need to change your organization, you need to change the way you're thinking, the technology that you're implementing, and the thinking that the best models give you the best result is really flawed. And you need to choose the right weapon for the problem that you're solving. - That makes 100% sense. The right skill set with the wrong tool is not going to give you anything. Darka, I'm going to still, just a few moments of your time with one last question. We need to leave our listeners with an action plan. We need to give them something to start with. We've now spoken about implementation. We've spoken about what ROI could look like at the end and the conscious decisions that we make during the project. But where do we start? For a senior leader who is about the Greenlight and AI initiative and he wants to be able to show the work at the end of it, what needs to be in place before we even start with writing a single line of code? - It depends, yeah, on the maturity of the organization. So we need to assess the maturity of the organization that is, if I'm the decision maker, for example, I put myself in that position. I need to understand the maturity of the organization to implement and adopt AI tools. From the technology standpoint and from the organizational standpoint. On the technology standpoint, we need to look at the data. We need to look at the knowledge management layer on top of the data. We have knowledge management. It's really important if you want to have multi-agent systems that can share the same context so that they can be efficient across the whole company, because all of the processes that are interconnected are intertwined in each organization. And then from the organizational standpoint, we need to treat AI agents as employees that we are going to guard rail, that we are going to task to do something, that we are going to check if they did something right. I mean, have to look at them as younger associates at the beginning. And then over time, as the technology improves, as you realize that these agents can take more of an autonomy by themselves, then you're going to be able to leverage the true power of AI. So you should look at this as a starting point in evolution of your organization, where AI and humans are going to coexist, or AI agents in humans are going to coexist, and not as a project that starts and ends with the ROI. So if you start with that kind of mindset, you're going to do the right decisions from the start and not build on technical debt. And this is really, really important. How to do that practically is, again, depending if you have on the organization. If you're organization that has IT department that is used to buying software of the shelf, that's not going to work. You need to start from the department, from the people that deeply understand the processes in the organization, and then teach them where AI can help them. So don't start with the technology. Don't start with the CIO office. In most of the cases, start from the people that deeply understand the processes in the organization, and then teach them where AI can help them. Once you do that, you're going to realize what are the requirements, what you actually need. Then go on the market and find what you need. If you are IT savvy organization, or you have a development team in the organization, et cetera, this is going to be even easier for you because you can task some of your development teams to actually do the research, help you out, create POCs, build POCs, and do it internally. Try to do as much as possible internally. But if you don't have that critical talent in the organization, if you don't have results after three months, you know that you need to go out. And when you go out, it's also important to find somebody who actually delivered solutions on scale, on production, somebody who understands even the economics of the tokens, spend per process, per decision and the outcome that you get from the AI. Everything from there is going to be evolution. And be prepared that this is not just one off that you are going to do. AI is completely changing the paradigm of how we are doing operations in the businesses. And how the businesses of the future are going to look like. So if you are prepared for the evolution, then you're going to make the right decisions for the beginning. But invest in knowledge, knowledge. That's the only thing that you can do at the beginning of the journey. I feel like that's some very good tips that you've given our listeners. Dorka, I'm going to wrap this up by just repeating a few things that you say that will really stick with me. I think when we started, you said one of the biggest issues of not being able to show our eyes, because we actually never-- we were never sure what the project was about. We just jumped on the AI wagon because of fear of missing out. And we actually didn't have a business problem in mind or we didn't really specify what we're trying to solve with this. So that's going to be the first thing. It's make sure that when you start an AI project that you actually know what you want to achieve, because then the ROI will also be easier to just measure. And in terms of measurements, there's no standard metric. Figure out what ROI means for your organization or for that specific department that you're trying to implement and deploy it. So that's also something to sit out. And then lastly, when you say it, where we start is by assessing the maturity of our organization before touching a large AI project. And that is going to look different for every organization. So there's really no one size fits all. Is that a good summary of our conversation? Pretty much. Yeah. Very short one. Daga, it has been lovely having you in our studio. Thank you so much for wrapping up this series. I'm excited to hear more about this and to be able to implement these little tips that you've given us. I'm sure they're very practical. And our executive audience is going to be able to use them industry-wide. I hope so, so I thank you very much for hearing me under podcast. [Music] Rapping up today's episode I think there are three key takeaways from our conversation of Darko. First, most AI initiatives fail to scale because organizations launch them out of fear of missing out rather than a well-defined business problem. Establishing clear KPIs and cost baselines before implementation is the essential first step to proving ROI. Second, successful pilots typically rely on the most experienced people in the organization. Scaling AI across the entire work force requires equipping all employees to work alongside AI agents as co-workers who augment human judgment, not replace it. And finally, organizational maturity and readiness assist across both technology infrastructure and change management capacity is a stronger predictor of AI success in the sophistication of the model selected. In this episode we cover how enterprise leaders can measure and prove AI ROI after deployment. To go deeper on the topic and learn how to identify real AI trends by tracking where venture funding is flowing and by listening to how leading CEOs describe risk and competitive strategy, download our free BDiPriPort. At EMERG.COM/AIT1 that's EMERG.COM/AIT1. On behalf of the team at EMERG, we'll see you on the next episode.