Q+A: How to stop your AI budget blowing out

Welcome to Fear & Greed Q&A, where we are asking to answer questions about business investing, economics, politics and more, I'm Sean Aelma. Australian businesses are spending more on artificial intelligence, but many are struggling to work out what they're actually getting for their money. New research from Elastic suggests one in three businesses exceeded their AI budget last financial year while only 8% are tracking AI's contribution to revenue or cost savings. Despite that half plan to increase their AI spend over the next 12 months, Jeremy Pell is country manager A&Z at Elastic, great supporter of Fear & Greed, Jeremy welcome to Fear & Greed Q&A. Thank you, Sean. Great to be here. I reckon that this is the ultimate question right now. What are we spending on and are we getting a return for it? So let's break that down, one in three businesses went over their AI budget last financial year, yet half are increasing spending. What's going on? That doesn't sound rational. No, you're right. If you think about just how you would go and ask for money two years ago, you'd have to build quite a robust business case to say, "I'm going to get this outcome off the back of it." And for some reason, all of those basic fundamentals have gone out that all they are, it's spending more, it's not really linked to the business outcome yet. Yes, we're seeing some sort of benefits, but nowhere near amount as the budgets that are increasing. So people are going over those budgets. They're continually spending and people are actually even throwing more money at it. And I think the biggest concern there is that the budgets aren't growing the same pace that the AI is growing. So unfortunately, some businesses have had to make that decision of, how am I going to fund these AI initiatives? Where's it going to be coming from? So it's coming from different parts of the business to fund these AI strategies. I wonder sometimes with these new technologies, whether that's just part of the life cycle of it, people do throw money at it, and maybe you're not measuring it properly, maybe you're not quite sure where it's going, but you kind of have to do it, or is that not the case? If you aren't spending an AI, you're going to be left behind. So there is definitely an element of that. I also think that we're at the height of expectation of at the moment where people are now spending a whole bunch of money to try and keep up with it. And we'll start to see it plateau out as the cost of tokens and things come down. But also as it starts to measure the business outcomes, and people can start to really attribute it to how the business is either saving money or growing revenue. So attribution is a good one. The really striking number in the research is that only 8% are tracking AI's contribution to revenue or cost savings. Is that a function of people kind of not making the effort or is it just too hard to do? It's a bit of both, really. Being able to manage a cost in an AI world is quite difficult. It's a lot to do with the data that you're putting inside these models. So how you're actually storing your data, how you're putting your data in there is the thing that is creeping up those costs. So being able to bring the AI to your data versus always just putting your data into these models is crucial. But then being able to then measure it, I always think the token maxing was kind of so last week. And now we're getting to a point where it is all about token to revenue maxing or token to value maxing. So how could we actually link it to a business benefit being revenue up or value out? And once you get that equation right, that it's actually moving in the needle in either one of those buckets, then you can start to double down and invest further. Okay. So let's just explain token maxing the people. It might be last week, but many listeners won't quite know what you mean by it. Sure. And then what that next evolution of it is. Sure. So previously, people's productivity was measured by lines of code. And now in a new world where code is being less, less built by humans, token consumption inside organizations has been a measure of how much they're spending in AI. And when we're talking about tokens, it's a cost per million metric. And there's a consumption method of input, so what you're typing into these models, but also what the output is from that as well. And you can imagine the cost of an input is much cheaper than an output because you may be just asking a single question and you're getting paragraphs and paragraphs of data that you're paying for. What we're probably not being really clear about is how those tokens are being used. So putting the data in, how much data is going in, what's the cost of that data going in, and what's the output that you're getting out of the back of it? OK. So do people do organizations understand how their individual employees are using AI, are using agents? No, it's incredibly difficult to do it. It's important that you've got the ability to do it. So you can observe how token usage is across your organization. But at this point, we're still, as I said mentioned before, we're in that height of expectation where people are just consuming tokens for, it could be spelling mistakes in an email that people aren't even really looking at, which is just laziness. But then there's the other good parts of tokens, which is where people are using it to drive smarter ways and more efficient ways of working. But it's really important that you have visibility across all of your usage so that you can understand if it's being used to drive the business forward. Are there any particular methodologies where that works really well? Because what you just described to me, and I think in our own business, Michael Thompson and my co-host, I would have a clue what he uses AI for plenty, I suspect. Plenty. Yeah. And then I'm sure he doesn't know how I use AI. But I mean, in a sense, we should. Yeah. Absolutely. Smart businesses are putting a cap on it. But as I mentioned at the start of the show, those caps are being blinded through at the moment. So it's about having a cap so that everyone can use AI and use it to the ability. And if you need to go over, build a business case like you wouldn't any other business. Why do you need the extra usage? What's that going to drive across the business? Because it's not just the overarching cost of AI that is an issue. There's a compliance component as well. It's about putting the data in those models. So you need to think about a large organization and being able to police what data is going to these models. Where's that data going? Is it your business's IP that's been put into the public internet? You need to be able to put controls over how your data is being used as much as what there is with cost. In any sense, it's just another input into your business. And most inputs into your business, you have rules and regulations and processes around. Do we eventually get to that point and that's sort of what you're talking about when it comes to AI? We just don't have all that stuff yet, but a business needs to create it. Absolutely. A business needs to have understand what's right for their business. You know, at some point, someone is going to be left accountable if something bad happens. And you need to be across what's going on, how your customer's data is being used and how your employers are using that data to get efficiencies across the business. Okay. So let's dig into the cost and we talk about tokens. It's still pretty opaque, I would say. What exactly is a business paying for when it's using an AI model? And is it the outputs that you suggested before that are making the cost blow out? Yes. And it's how it comes to and how you use it. There's multiple different ways you can use it by holding back context or by starting a new chat every time you're doing it versus having that contents lag sitting in there. But where I think the businesses are going to be moving to is being able to use different models for different reasons. So the more we talk about those tokens, they consume more tokens at the more relevant newer frontier models, but you may be able to use some of the cheaper models for lower impact activities inside your organization. It's really interesting where we are at the moment too. We've had a period of a duopolies in regards to open AI and anthropic leading the frontier models. We've then started to see a whole bunch of different players come into the market in the last couple of weeks. The likes of Kimmy and being able to put those in an open source environment on your own self-managed data inside your organization. It's our meaning that you may be able to run more efficient open source models in your own building, which means you won't have to have the hefty cost of tokenisation at these frontier models. So it's getting really interesting. Additionally, GROC just announced their new model, new pricing model coming out as well. So what we thought was this duopoly just a couple of weeks ago, as all of a sudden come into this really competitive price war coming on with different models being your own machines, own data stacks inside your building, not just the cloud-based frontier models. So if I'm a CEO listening to this this morning, what should I be doing today? I mean, you have drawn this picture of a really competitive marketplace or increasingly competitive marketplace, but distinct models for distinct processes and parts of your business. What should a business person be doing listening to this thinking, okay, what do I need to do this morning to kind of start putting that in place? Yeah, starting with your data foundations is critical. Is your data fit for purpose? What data are you using? Where is your data sitting? What data are you going to be able to use to tell the right story? That's, understandably, the foundations for a great AI story. If you are putting bad data and paying for bad data to go into an LLM to produce the result, you're going to get a bad result. That's all about having the rate for data foundations to begin with. And then relying on your partners to be able to work through. How can they help you through this space? The amount of product releases coming out is incredibly hard to stand top of. Let alone the strategy that you've got inside your business. So being able to be on top of technology road maps as well as your own business is incredibly important. to belong your partners to be able to help you. you navigate through this space. Okay, where's the world going to be in two or three years? Jeremy. As I said, things are changing by the week at the moment, so two or three years, you've put me on the spot there. I think it's going to get incredibly, incredibly different to where we are at the moment. There's a lot of different theories out there and where we're going, but for me, if we're putting it back to what we're talking about today in customers here in Australia and CEO's, I think where they're going at the moment, we're going to be relying on not just data that it's going to sit in the cloud. You're going to have data that is within your building that you can have control over and you can have governance over. That's going to be incredibly important as we start to move forward. So you're going to have to have a hybrid approach of having data that sit to the inside that you have control over, but also you have the ability to tap into the the future of AI and the latest and greatest that's coming down the line from the Frontier models. Jeremy, thanks for talking to Fear and Greed. Thank you. Cheers. There's Jeremy Pell, country manager A&Z at Elastic, which is a great supporter of Fear and Greed. Search Elastic the Search AI Company to find out more about how Elastic enables AI. Also that again, Elastic the Search AI Company. I'm Sean Eyelma and this is Fear and Greed Q&A.