How to Forecast Your AI Token Costs with Jason Brisbane and Tanvi Lal

I think the biggest issue we have today is what is the ROI for AI. And frankly, that is the bookstops with the CFO, right? It hits a huge line item. On my expenses list, I need to be able to assess whether or not there's enough ROI to justify it. So I think from that angle, I really, really see the use case for it, the CFO owning it. I think the con on the flip side there is the CFO is not the voters actually in the weeds using the product and understanding exactly how things are being used across the board. So this is where I think if you and A and like the finance team really needs to understand how work is being done and why the costs are what they are in a more unique way than you have in the past because usage base means you have to understand how the work gets done. Welcome to another episode of S.P.N.A. Unlocked, where finance meets strategy. I'm your host Paul Barnhurst, AK, the F.P.N.A. guy. And each week we bring you conversations and practical advice from thought leaders, industry experts, and practitioners who are reshaping the role of F.P.N.A. in today's business world. Together we'll uncover the strategies and experiences that separate good F.P.N.A. professionals from great ones, helping you elevate your career and drive strategic impact. This week I'm joined by two guests. I'm thrilled to have with me Jason, Jason Brisbane. Welcome to the show. Thank you. And then I also have Kenvilla. Welcome. Thanks so much for having me. Hi. Yeah, excited to have you. So all right, so we're going to go ahead and jump into things. And before I get into the questions, I'm just going to take a minute and let each of our guests introduce themselves. So Jason, if you could just take a minute and we tell our audience a little bit about yourself. Yes, hi everyone. Jason Brisbane. My journey through F.P.N.A. I started my career at Adobe. I was fortunate enough to be one of their first financial rotation program individuals. I spent time in their corporate F.P.N.A. group, tax, treasury, internal audit. After that, I ultimately landed on their digital media F.P.N.A. team, which essentially became the creative cloud. And that journey brought me through product management and product development for the office of the CFO. And now I'm building Finhelm. Finhelm is the platform that I wish I had as an F.P.N.A. professional. We need a quality metric to grade our own work. And that's what Finhelm strives to do. Great. All right, over to you, Jeremy. Thanks so much for having me, Paul. And excited to be among good company, Jason. My name is Tanvi Law. I'm currently an investor at Intuit Ventures, which is Intuit's corporate venture fund. So we do growth stage investments across applied AI as well as Fintech. So I spent a lot of time in and around F.P.N.A. primarily from evaluating F.P.N.A. platforms and. understanding what our product teams at Intuit need as it comes to supporting our end customers, which are primarily small and medium businesses, as well as mid-market customers. So excited to dig in here. This is an area I'm really passionate about. I love to dig in. And Tanvi, I think it's a fascinating time for the F.P.N.A. platforms. It is for all finance software. I imagine the evaluation has changed a lot over the last few years. What you're looking for, what you see, how you think about things with AI. Yeah, absolutely. All right, so this is a question I like to start every episode with. And Tanvi, we'll start with you. If I asked you to define great F.P.N.A., how would you define it? Yeah. For my lens, and again, as a not F.P.N.A. professional, but just what I observed, I think great F.P.N.A. professionals truly, truly understand the way the business works. And I mean, they know how product gets shipped. They know which teams are involved with, and they bring that context into how they approach the financials into how they approach thinking forward, and also into how they build trust and rapport internally. So I think understanding of the product, how that works, and how you ship and deliver, as important as what the revenue stream is, and how you account for that and forecast it. Okay. I appreciate that answer on knowing the business. I think that you make some great points there, Jason. Any thoughts? I was saying all the same things Tanvi mentioned. I have to, a great F.P.N.A. team, you're beyond managing the reports. And as we talked about, you have a pulse on the business, know how to drive the business forward, and having ownership of the assumptions that matter. Got it. Thank you for that. So just our audience, those today, we're going to jump into kind of the economics forecasting, the cost behind generative AI tokens. So we're going to spend a lot of time talking about that. And where I want to start is, and we'll go with you, Jason, and then we'll reconvene. How did you kind of become interested in the economics of Gen AI? How did you kind of focus on that? For me, what drove me there, Paul, is when you're thinking about platform costs, typically an F.P.N.A. It's, for the most part, fixed. You know what that's going to be, and you can have ownership of it. You can grab your arms around it. I would say generative AI, we're just in a new world. Where the cost is variable as soon as you commit. So that creates an opportunity for us to have a forecasting layer in terms of being able to essentially own it. And so that's kind of drove my interest here. Make sense is, it's really interesting, because typically, yeah, your most platform costs are fixed. How about you, Tommy? How did you kind of get interested in the economics? I know you've done some reports and stuff on this. Yeah, similar approach to Jason. My vantage point is really from assessing and analyzing startup performance. So I started to realize, as I was digging into application layer AI companies that traditional SaaS metrics, which we relied on to assess how efficient and how well performing the businesses were financial in. It's just don't lend themselves very well to AI companies. What I really want to see and understand is how efficient is your AI spend? How effectively are you driving outcomes for your customers? And what is your customer specific margin? Your most engaged customer is actually released profitable with AI unit economics. So I started approaching it from that lens and really came to understand that there isn't infrastructure built for this or not a lot of startups are actually tracking these metrics. So that's where some of the work I've done to try to create what are the metrics I want to see and how do I want to think about this as an investor going forward to make the right bets. That's where I'm coming from with this. Yeah, I would imagine when it first came out, everybody was still looking at SaaS metrics. I'm like, okay, well, it's not that big. They still are. We still are, rather. Yeah, but I would imagine with some of these companies, it has to be a huge line on the PNL. I mean, it totally changes the economics. It's right, you want that super high-grows profit, you always want to see with low cogs and all sudden, with variable, it really changes the equation real quickly. Yeah, absolutely. And there's no standardization of where these costs are being held and how to really parse any AI or research spend that's on the statements. Yeah, like I was talking to somebody who works at a pretty much the biggest social media company in the world. Most people should be able to figure out who that is and they don't allocate any of their AI spend to cost a good soul right now. It's all in operating expenses. And I'm like, that's kind of tough to look at. Like, is that becomes bigger and bigger? It has to really kind of make it hard to really understand things I would think to a certain extent. But I imagine we'll get standards eventually kind of like SaaS. It was the Wild West. We never knew what bookings when somebody first reported it or what recurring revenue was and you dig into it and you're like, yeah, that, no, that doesn't work. That's not real revenue. Where I'd love to start, we'll start with some basics. I think, you know, most people have a basic idea of kind of what the LLM is. But I don't know that most people really know how it consumes information. We've all heard the term token. But how does a token kind of translate into a cost? How does the LLM absorb information? Tommy, could you share a little bit on that? I know you've done some work there. So a token to your point is the way LLM's consume information. And you can think of a token as more or less a word. Depending on the length of the word, maybe the word is absorbed in two tokens or three tokens. And you have, so LLM's are essentially anything you input into your chat GPT or into cloud, your input text, any files that you're sending, any web pages that you're sending, all of that will be consumed by the LLM and the form of input tokens. And these are typically priced per million. So you'll pay a set fee for, you know, one million tokens. The same concept applies when the LLM gives you an output. So the text, the document, whatever it's generating for you is also measured in tokens. And these are called output tokens. So these are also priced similarly, you know, X amount per million tokens. However, output tokens are priced much, much higher than input tokens. And the other important thing to realize is within one specific chat that you have, which is the context window for the LLM, every time you send in a new query, you're actually sending the entire chat back to the LLM. So the LLM re-reviews everything and then gives you, you know, the output. So as you keep using the same chat, your input tokens will exponentially increase because it compounds. And so that typically is what drives up input token costs. But again, those are quite, quite a bit smaller. I think output tokens. tokens are really where you see higher costs with the deliverable, or alternatively, if you have like hundreds and hundreds and hundreds of pages in your input, then you'll see a higher cost on the input. Yeah. I would imagine, especially people who are coding, right? Because your prompt is going to be much shorter. Even if you have instruction documents and everything, then the code that comes out. So I could see where the output gets really expensive. Yeah. Absolutely. And you need all of the contexts in the chat for the alumni sharing show. Have you tried what's what's worked, what hasn't, you know, LLMs give you the sense that you're talking to a person or somebody who's just like talking to you back and forth, but really every time it's just a net new call, and there's no real relationship there. So you need to resend the information every time. Yeah. Speaking of relationships, it's been interesting. I've heard something that like, you know, 40% or 50% of all people have like asked relationship advice from LLMs, you know, and so it's really interesting to see how people treat it like a human and conversations, but it's zeros and ones and probability. It's basically a very, very, you know, strongly enhanced prediction algorithm, and that's what you're talking to. Yeah, it's basically something that's been trained on pretty much the world's data. So it's really, really good at predicting what the next word should be, which the next output should be. Anything you want to add on kind of the whole, you know, kind of tokens and how they consume or any thoughts there, Jason, from your end? I would say just in regards to Tandy's work, which has given us a roadmap to kind of be able to better understand this in terms of project estimation. It's been eye opening to understand the impact of output tokens and how to account for tokens in regards to token ratio or orchestration ratio in terms of implementing a project leveraging AI. Yeah. So great, great segue. I know you've done quite a bit of work on that, Tandy, we want to talk a little bit about the project you did, because I know you do a work on modeling the cost, kind of what you did, what the takeaways from the exercise, maybe take us through that. Yeah, of course. So in my journey to better understand token economics and having seen headlines like Uber or blowing through their entire token budget and, you know, Microsoft, not renewing their cloud subscription, I really started thinking, I was like, listen, this tech just needs to be better managed, right, we need a framework to figure out what to use AI for our understand which types of asks are really driving up costs and really create some guidance and some structures so that organizations can incorporate AI in like a financially responsible way. So what I did was essentially bucket out what are the main types of tasks you do in a business. So maybe you're doing coding, maybe you're doing financial analysis, maybe you're doing content, you know, maybe you're doing something else. So I created six buckets there and I worked with Clawed to come up with dry runs. So I actually ran through various asks within those six buckets and I counted, Clawed counted, Clawed, Clawed it, most of it, but through my, through my guidance and so counted the number of input and output tokens that were coming up based on these tasks and you use the latest pricing to figure out what was the actual cost of AI usage to do this task and a low complexity, medium complexity and high complexity. So that's kind of the back end model that is powering this AI budgeting calculator I put together where you can describe whatever project you're thinking through in natural language and, you know, Clawed will assess, okay, what of the six categories is it, is it low medium high and it will spit out an approximation of how much this would cost from an LLM perspective. Some caveats when I did this work fable had not been released. So, you know, I'm sure actual costs are higher and really what I wanted to get a sense of is how much does it take to run through this one time where actually in reality you're iterating through the work a few times. So you're probably going to need to compound or, you know, multiply the cost that you have for once around and, you know, in addition to this, what I also did was try to estimate the labor spend that goes into it to see, hey, are you actually using that less labor spend as the overall cost more or less? And really what I found is that even though your token costs can be high, it's always always going to be the personnel and the labor cost that far outweigh the costs of two ends. So really at the end of the day, the concept isn't how do you like effectively like lower your tokens, that's how do you create a better process with what AI can do so that you are creating work faster and the aggregate cost is what we're with human intervention. So it sounds like, you know, you mentioned the token cost is really in the big scheme of things, for specific tasks, not when you're experimenting or, you know, big, huge runs multiple times, but just the basic tasks is usually really not very expensive. If you're, if you're using the right model, you've optimized the process. So it's really the labor and more about the optimization of the process with AI than the token cost. Is that kind of what I'm hearing? Yeah. Yeah. That's accurate. I think the type of task matters a lot. I mean, content creation tasks are basically free, they're so low cost, right? But coding, for example, just like continuously running your cloud or your codex in the background, like all day running coding prompts, like that is going to eat up a lot of tokens really deep or comprehensive financial analyses, actually, like generating a comprehensive Excel takes up a lot of output tokens. So I do think the tasks that you are doing matter as well. And I think this is where the ROI comes up. Okay. If I'm spending $10,000 on coding tokens, like what am I actually seeing and is it worth that or not? Yeah. And you make a great point. Get to you here on a question in a second, Jason, but I think when you mentioned Excel files, I hear a lot of people saying it did hardly anything in Excel and I blew my budget. Like, well, you're basically asking for me to reform file work and go anywhere in one formula unless you're using dynamic arrays can spill across a thousand places. That consumes a lot of tokens in a hurry. Yeah. Absolutely. I'm just laughing. Recently, he's using Clouds free preversion and he was trying to format a doc and he's like, well, before it even sped out the doc, it ran through my token using and I was like, yeah, well, it takes a lot of tokens to output a formatted document. Like, the formatting is actually really tough for the LLM. Like, he could spin out a really ugly version, but the formatting takes time and tokens and effort. You're kind of funny in that. When it first came out, Microsoft released, they did it. They built an LLM that was all around designed to analyze Excel files and the first one they did is about three years ago. I remember reading the article. It couldn't handle formatting or as long as the things it couldn't do. It had reduced consumption for reading it by like 97% by the way they were parsing it and everything. But compared to what they can do today, that I was like, okay, they're not getting there anytime soon on spreadsheets and the next thing, you know, they can do it, but it's costly. So Jason, would love to get your thoughts, we've talked a little bit, you know, the human cost is expensive. But from a finance standpoint, as companies are looking at Gen AI because FPNA, everybody's seeing the ubers of the world and all these others that blow their budgets or token maxing and all this stuff, what are some of the costs? How should we be thinking about this beyond tokens? I would say in terms of cost of labor, cost of resources. So I actually ran an exercise on Tammy's calculator. I took a typical FPNA project. I just asked it, hey, give me an estimate of building a three scenario cash runway model. When I ran the estimate, it was amazing. I got a range first off, and I got a confidence level at that range. But in terms of token costs, the orchestration, I would say the duration was 99 to 1 in regards to, hey, less than 1% of it was token versus orchestration was the other 99%. So it really points to when you're thinking about this from an FPNA perspective, but then also as an organization, you want to leverage that on exercises that are going to provide some sort of value. But hopefully when you do that, it's going to navigate your resources to really provide an of value at versus, hey, manual things that the LLM can, you can hand off to. And when you say orchestration, just so our audience is kind of clear, elaborate on that, you say 1% is token, what do you all, what do you consider all part of that orchestration? Guiding the LLM, doing checks, putting into inputs, some of those things that Tammy mentioned in regards to, hey, we're, we're, we're inputting, we're having a chat back and forth, we're iterating, hey, maybe a second person needs to come in and take a look at this, accounting for that in terms of the total project estimate scope. But I thought, hey, I could just do everything, you're telling me a quant, I have to actually steal work. I wish we're not there yet. You hear some doomsday rumors, but we're not, we're not there yet. We hear doomsday rumors all day, every day on LinkedIn. Yeah. But that's some other story. Yeah. Right. And I see you both smiling and laughing a little bit of that. That's when you know it's true. So, Tammy, if someone wants to see your calculator or get ahold of that, just real quick, where would they find that? Have you put that out there publicly? Is there a place they can go look at that? Yeah, yeah. It's on my LinkedIn and I'll share the link with you if it makes sense to add it in the session notes. But it's free. It's online. Oh, please use it. I would love your feedback and your thoughts and clearly I need to make some updates with the new model releases. And I also would like to run this back with codex and chat GPT2. you also see how it works across different files. - That would be great. How long did it take you to do it? - It took me about, I would say, 10 days, like a week to 10 days. And even with the cloud pro plan, I was kind of hitting the session limits. So I had to-- - Or that the $100 a month pro or the $200 a month. - I know, right? You're like, how much must I pay to actually assess these things? But yeah, I mean, even with the pro plans, I have limits and some of the tasks actually took quite a while to run. Like, they would take 20 minutes to run. So yeah, just got it out in and still much less time than if I had to do it all by myself with an Excel. - Yeah, isn't it funny how now we complain, I had to wait for my LLN for 20 minutes. And it's something we could have never done before. - Right, right. - Like when you would have attempted and working planning that we had to wait a whole 20 minutes to get something that was beyond our capability. - Well, I just like, hey, my laptops are reading. Like, come on, cloud, work faster. - It is so true, I was having edits at PowerPoint the other day and I was just like, come on, finish. Like, why is this taking, and it was doing stuff I would take me three, four hours to do. But yeah, it took 20 minutes or whatever and I was just sitting there complaining and I'm like, man, first world problems is what I thought. - But it's interesting, the LLN still like, it takes some time to figure out how humans do it. Like for example, I was trying to get cloud code to download a file from like a document repository. And it kept turning into issues. It was like, I'll try this, I'll try this. I'll try this in like 10 minutes later. I'm like, hi, do you need my help? Like, let me help you. And so I think it iterates in a very different way and we'll like find issues and like, you know, you know how to troubleshoot the issue. But the LLN still takes like 10 different tries to figure it out. - That's one of the most interesting things for me is you read. I feel like every day I see something failed. And the first time you see that you're thinking, all right, the LLN sucks. But, you know, it comes back with the right answer. It just keeps kind of iterating, I'll do this approach. I'll do that approach. It's almost like it has this checkbook. And you know, this checklist and it just keeps working through every possible solution to help find the one that works. - Yeah, and you're like, just ask for help. I'll tell you which one's going to work. - There are definitely times where I've stopped it. I was like, okay, stop searching. Here's how you, oh, yes, that will work. And you're like, thank you. I knew it would work because I've done it that way before. - Yeah, have you ever stopped mid-project while he's thinking and then check your usage? I've done that a lot to make sure I'm not running over. - I haven't done that one. I haven't stopped mid-project on my usage, but I'm pretty good on usage. I used to be running over a lot when I was on the $20 a month plan. Now that I've upgraded to 100, I've bumped into the limit a few times on some coding projects, but, which tells you I just need to use AI more. I'm a slacker, but we won't get into that. FP&A guy here, I want to take a minute and share something I've been working on. I've spent several months adding new affiliate programs to my website, probably wondering why affiliate programs? Well, I can't train you on everything, but I can show you and give you trusted resources with exclusive FP&A guy discounts in many cases on my website. I have several of the top FP&A AI people in the world. I have the top certificate programs. I also have other programs like Excel, Power BI. I'm bringing you some of the top people that train in the world. There's Nicholas Boucher with exclusive discounts, David 14, Madden Carl Sideman with exclusive discount, the top programs, FPAC, FMI, and more. So go ahead and check it out with my website, sign up. And if you have a question on what program is right for you, go ahead and message me. I'll get back to LinkedIn or you can email me. So go to the FP&A guy.com and check out my trusted list of resources for you to upskill yourself. So let's talk a little bit of new projects. Obviously, finance is gonna be asked whether atay we're working on this new software thing. We're doing something internally. We're creating these agents. There'll be a lot of them having to figure out estimating, which not what my ever did is an FP&A professional. Obviously now haven't been my old business for five years. And not something most people have experience in. So we'd love to get some of your thoughts. So where should they begin? And what are some of the key inputs? So 10V will start with you, your thoughts on that and then we'll go to you, Jason. - Yeah, I think the first thing to consider is what is the type of work the team that you're budgeting for has done? Like are they coding? Are they creating? Are they either parsing large amounts of input data? So let's say the data, like the data analytics team right there who are being across data lakes and like all these different BI tools, their inputs are gonna be very heavy. So either looking at teams which have very high volume of inputs or high complexity inputs or teams that have high volume or high complexity outputs. So I think teams which have either or both would require more tokens. And the other piece I would look at obviously is head count, right? And actually getting a clear sense of how many people in this team are actually gonna be using the tool versus are gonna be reviewing outputs of the tool. And then I think as always it's important to talk to the team understand how they are or how they plan to use. But with the obvious understanding that when folks have budgeting conversations, they will inflate their actual requirements. So that would be my thought is to focus on either complexity of inputs, volume of inputs or complexity and volume of outputs. Those teams at once would require the most. - Yeah, which makes a lot of sense. It's really kind of understanding the project and what's the key drivers to the cost, which are usually how much output or how much input do you expect? Jason, other thoughts on this, what else would you add here? - Yeah, I would just add on, I would say just overalls the organization, the hygiene and the behavior has to change where I would say you, as an organization at this point you have to move away from what I call kind of false precision. As we say, in Finhelm where we're just reporting from a single number perspective, I think now a range is critical and owning, hey, what's the floor, what's the expected, what's the blowout and manage to that band. And when the actuals come in, log it and just get the band tighter as you go. - No, you make a great point because the cost would be substantially different. If you assume that 80% of the project was going to be open and it turned out that 80% was Fable 51 for whatever reason, those are very different costs. And so little changes in assumptions 'cause there's so many different ones that can go into this can have a pretty big impact on what that cost is especially from a software product platform. You're kind of talking about your best users in some cases are your most expensive now. So do you really want all your best users if you define that by usage of the AI blowing out your cost? And so there's just some of those things you have to think about is is it the same? What does that mean in ways that you didn't have to before? - I think the other wrinkle here is as like AI monetization models evolved, this is gonna look different. Right, so startups and companies are starting to charge only for successful outcomes, which means that you're still gonna have a cost light item of unsuccessful items that then lead up to that. So obviously that's on the product team for building a product and the end team for building a product that actually meets the output metrics that you're seeing. But I think that's the other trend that I'm seeing coming down the line and actually to talk in more depth with startups that are approaching pricing like let's understand how they're thinking about it internally from a cost perspective. - Yeah, a couple of months ago I had on, I try to remember her name now. I'd have to go look it up, but I had on a pricing economist specialist in that and we talked about outcomes in pricing, right? It's an area that everybody talks to Holy Ghost, kind of that value or outcome based pricing, but that comes with some real risk. You gotta have a high level of confidence that you can deliver. Otherwise, it can be a disaster if you're not delivering. I think we all like the idea of paying on an outcome 'cause it's much easier to say, okay, how much is a successful outcome worth to me? And I'll pay this versus I pay this for unlimited usage. I pay this for token and there's more uncertainty. - Yeah, the other thing I'm seeing is folks are charging different costs for the type of word performs. So if you're doing like a simple query or things that are like testing for goal, I have like maybe you'll charge a certain amount less. And then if it's a production rating like customer facing agent and you're charging a little bit more. So lots of different lovers and approaches I think we're gonna see. - Yeah, which makes a lot of sense, customer facing. I mean, I think the higher the risk around the agent the more QC you have to be, at least in theory, the more you would expect to be in theory. - Yeah, you like the in theory part 'cause we all know how do I like to say it? There's theory in reality and rarely be with me. And anything else on this subject, Jason, before you kind of move forward, any other thoughts here? - I just think we're in an era now where the FPN-18 has to think in bands. Beyond a single point on this and communicating in bands, what's the expected, what's the floor, what's the blowout? I think that earns the confidence within the organization to what I would say kind of have arms around this line item. - Why I think? what's great is with LLMs and the fact they can do Python, they can do deterministic solutions, whether it's an Excel or other applications, it's much easier to do a monkey carto simulation. A lot of people would have struggled to build one or to do a bunch of sensitivity or scenario, but most people can understand the basics of it. You may not understand all the statistics, but you can understand it. So I think it's much easier now for people to get those ranges with some statistical confidence around them. Does that make them perfect? No, but you know, knowing it's 10 to $50 is 90% chance that's my range versus someone just saying $30 is really helpful. Our 30 million versus, yeah, 10 and 50 because sometimes those ranges can be high. We've all seen it with finance. They stick in the, how confident are you in that number? I would, I wouldn't stick my life on it. That's what you're asking. That's it. Yep. Maybe we've all put forward those numbers where you're like, I think you will land here. So you're learning how we really work when we do forecasts on these. You want to come over and grind us? I had enough of an experience when I'm doing my financial model. I was like, I don't know. It's like two years of historical data. And I'm like forecasting what a 15 year exit or like a 10 year exit. I feel you want that. Yeah. Yeah. I can imagine forecasting, especially if you're working with high growth companies, there's not a wide range of those outcomes. Is there? They're happy, but the good outcomes, no, there was a pretty specific range you want. Yeah. There's a pretty specific range of what you want. Unfortunately, reality tends to have a wider outcome. Exactly. Expectation versus reality again. So Jason, I want to ask a question. When we were chatting, if you remember when we were preparing for this episode, you mentioned a term or you said a phrase that I like, you said, treat variances of learning. It kind of reminds me the whole idea of failure leads to success or supposed to treat failures as a learning. So kind of talk a little bit about that. What does that look like in the terms of AI? How do you think people should look at that? Great question. Paul, I think what you want to do is, and we talked about it previously here, where when you move past false precision, when you can communicate the number and bands, you manage within the band. And from there, we get learnings from the band. And a miss one way or another is not failure. There's nothing bad. There's something to learn. And then we come back and we score it. And we come back and test ourselves against it. And then we're getting into a calibration type of exercise. And so I think if any organization that can install that, that's where you really create an environment to kind of what I said in the pre-call tree variances learning. So I'm going to ask a follow up here. I want to drill down, right? I agree with the concept, but we're often have to be held to a budget. There's a number that the board has given often external. So how do you manage between that range you need to do for projects and making sure you deliver in total? Because at the end of the day, we give Wall Street a pretty tight range, or hey, we expect to be, you know, 115, or we give our boarded directors. We expect it 150 million in revenue. Typically, we don't give them a wide range. It usually doesn't work there. So how do you kind of manage between, you know, these variances and make sure you're not blowing the, you know, kind of the top end off of what's expected when it all is said and done? Well, once you, you know, in kind of what we talk about operating in bands, what I would say owned and beginning to get your arms around what I would call the trip wire, owning the assumptions around that and being able to calibrate and adjust. I think when you're able to do that, you're able to kind of have those levers you can turn and pull what I would say on a dime. Because like, as you said previously, we have these tools now. So now we just need to bring them in house and manage to them and own things in terms of like, hey, what are the drivers with a trip wires on these drivers and be able to adjust accordingly? Makes a lot of sense. And I agree with you. I've like, so you said in the sense, you have to be able to know the dials because it's just like with any budget, there's going to be areas you blow, there's areas you're going to do way better, and you're going to have to adjust and move throughout all of it. It's just another level of complexity, so to speak. FPAA guy here, we'll get back to the show in a minute. This section is on scripted. When I started my business, I had two main goals to be trusted and for my content to come across as authentic and helpful. I've heard from many of you that I've accomplished those. Today, I have three goals to be trusted, authentic, and have the best beard and finance. Alright, enough with the jokes, enough with the fun. What I'm asking for is I'm asking for you to help me out. If you've enjoyed the show, if you've enjoyed an interview, if you found it helpful in your career, can you please leave a rating and review? That helps more people find the show, helps me grow the show, helps me grow my business. So I'd really appreciate it if you could take a moment and leave me a review. You can do it on Apple, Spotify, YouTube, wherever you listen, or you can go to my website, www.vfpnaguy.com. That's www.fpnaguy.com. Go ahead and go to Walla Love, go down to the bottom, and you can click to leave a testimonial. Thanks for doing that. Thanks for listening. So I'm going to ask you guys a question, something I didn't have in here, but I'd love to get your thoughts. I want to start getting into the role, finance, and FPNA should play in AI, because I think there's a huge role they can play. And I want to get your thoughts. So Glenn Hopper, I don't know if either of you know him, but I do a podcast called Future Finance with him. And he just released his, or he's releasing in October, he wrote his book, The AI Ready CFO. And he makes an argument because of the intersection of governance, which finance already has to do a lot of, and data, the AI adoption should be owned by the CFO, and finance. And I just love to get your guys' thoughts on this from a, from a perspective. So that's a big argument he makes in his book to CFO. So Tom Vee, we'll start with you. What are your thoughts on that? I guess I'm in two minds because I see the logical reason why the CFO should own it, because I think the biggest issue we have today is like what is the ROI for AI. And frankly, that is like the book stops with the CFO rate. So I think from that angle, I really, really see the use case for it, for the CFO or owning it. I think the con on the flip side there is like the CFO is not the voters actually in the weeds using the product and understanding exactly how things are being used across the board. So this is where I think if you day and like the finance team really needs to understand how work is being done and why the costs are what they are in a more unique way than you have in the past, because usage base means you have to understand how the work gets done. So I think to the extent the CFO is embracing that and like maybe we can talk about this later too, just you don't want to manage costs but stifle the innovation or stifle the way the work gets done, right? So I think you need a very specific type of CFO who's ready for that, which I'm sure he gets into in the book and all of that. I think from an ROI perspective that tends to sit with the CFO when it comes to the tool you're using. So that makes logical sense to me. Jason, any thoughts on that? Yes, I would agree. I think there's two pillars for the CFO. I think pillar one is at least in terms of your top LLM model that you're using. Having a grasp of token costs and within that for each model, but then also going back to what we said earlier when we were talking about 10 of these calculators in regards to the ratio of token costs versus orchestration costs for a project. The CFO is often tasked with directing what projects are done within your organization. So I think from that lens, the CFO has to manage that piece, but then also manage, hey, what is this costing from an overall LLM perspective? So yeah, I would agree, but I would just put it on to those two pillars. Thank you for that. Appreciate it. It was just kind of curious where you sat on that. So when when we chatted before, you know, where do you sound that Paul? I'm curious to get your tape. You know, I agree. Yeah, we can do it. I tend to agree that the CFO needs a lead adoption and not just from the ROI standpoint, but any company that's going to be successful beyond efficiency gains has to have a good good foundational data. And the the neutral person that should be helping define that data dictionary is really finance. If you let marketing define it, they're going to take one one angle. If you let another group define it. If you have a data group, yes, they're going to be involved. Yes, obviously, I do, but I think you've got the ROI side, but I also think you have a big data. There's a lot of governance in there that finance is really good at. So I think it's a natural spot. Do I think that means every situation should be that way? No. Does company size and complexity depend like you're a company that's huge in AI. It's a big part of everything you do in your products. Okay, maybe there's portions. I'm going to have the CFO own, but is he really owning all of it? Probably not. But I think in general, the philosophy makes sense just because the one who has to make sure that data dictionary is right for the external board is your finance. Even if IT owns some of that, they're the ones that the CFO is going to pay for. If he looks bad in that board meeting or something's reported wrong and so I think having that data, having that governance and then the ROI side, it is kind of a natural fit for the CFO to help lead out that adoption and make sure the data in governance along with that ROI or balanced in the process. But there are areas where they would have to get deep into the weeds and that may require the somebody else you're going to have the right team around it. So I don't think it's just a simple, hey, it's always the CFO, but I think the argument makes in one sense. Here's your perspective. It's good points. Yeah. No, you're welcome. All right. So when we chatted, both of you suggested that you think finance has an opportunity to kind of leave the organizations thinking about AI. So let me get a little bit more of your thoughts there. And we'll start with you, Jason. I mean, how does finance lead in this? We just talked about the CFO ordering a bit more broadly, FPNA, different things. How do they help lead? So one, we talked a bit and I do believe wholeheartedly the finance team in particular, FPNA should have direct ownership of understanding token costs. So the organization, obviously, there's going to be curiosity. You know, to hear these great things, hey, I want to use this. So I think I think the finance team should lead that and having an understanding. But then also, I think this creates when I think about FPNA, there was a cadence of when I could touch business partners. I think back to some of my times when we were doing head count outlooks, week sticks, nine and 12. I think this creates an opportunity to increase the cadence of working with our business partners. Like if I was in the seat right now, you know, if I'm owning token costs, I would want to be on top of my business partners when the new sexy model rolls out to say, hey, this model can do this and this. But then we're going back and we revisit some of these projects. So I think that's kind of creates the opportunity for the FPNA team to lead here. They should be the first ones that are talking about the new model, how they implement it. But then, hey, let's go back to these projects that we were looking at and again attack it from the band. Thank you. Tommy, your thoughts on this one. I think Jason hit the nail on the head there. The only thing I would add is I think the FPNA and the finance team is uniquely positioned to guide different views on how to think about use cases where AI is a good fit versus which model you shouldn't shouldn't be using. Like you don't need fable for a content task. This just and like even, you know, if you're doing, let's say you're creating a presentation, you can use fable for the outline. But when they're like, do you want me to create the deck? No, you use, you use a different model. You don't need fable to create the deck. It's a reasoning model. It's, it's there for the thinking and the partnering. It's not there for the execution. And you know, when you just get a $200, $500 budget, you're not necessarily thinking about these things because you're like, whatever it's my company, I can just go ask for more. But I think as your allocating budget, it's just explaining the thought process and the assumptions behind how you came up with this and like, what kind of model you such are assuming. I think is a really strong way to influence how work gets done and to Jason's point being on top of it when the new model comes out, say, hey, this is how you use this me model. This is what we're expecting in our budgets. And this is what we're being up. This is what we're applying consistently across the org and like, you know, yes, you're special, but no, you're not. I think that neutrality piece that you talked about follows really important here. And I like how you both kind of mentioned there's a real education opportunity there to help understand from a cost perspective, like you said, you know, fable, my basic content, basic tasks, all right, you're spending. If you compare to IQ, you might be spending 10 times as much. Really, like if I told you you could have a basic dinner, you know, you're drink for $5 or $50, though, the dinner is drink and they're roughly the same risk and take the five, but yet, we often just use fable 5.1, but it will be better. Not necessarily, you know, there's a lot of that kind of thing. I mean, I find myself doing it sometimes. I'm just going to run the eyes, but I was like, why did I really probably will need that. You know, but when you're especially when you're when you're on a subscription versus when you're paying for every single token call. You treat things differently and businesses are generally, you know, when you're by the API, you're paying for each token, which is different than just having a subscription. Yeah, and I think the issues that orgs have right now is there's no one centralized place of disseminating this information, like personalizing it for the business and how you be processes and like, you know, context, like, oh, when you're creating this type of analysis or when you're shipping this hydroproduct, where you're delivering this kind of service, right? Like, it's so different within each business. So actually, putting all of these basic concepts in practice for business. Like, there's no part of the org that does that right now. If I teams just figuring out on their own, every person's just figuring it out on their own. But yeah, influencing individual behavior like this is difficult. You need to educate your business partners, your business partners need to educate their team and like, that's how it successfully runs. And right now, we kind of have a cold start problem was like, nobody owes it and nobody's telling the business how to do these things. And I think, again, finance is very uniquely positioned to beat that person on that, that voice on the business. Yeah. And I like you said the cold start problem, great book, if anyone hasn't read it. So on networking, marketplaces, just don't do them as the advice. They're hard. Take everything with the greatest thought. Yes. All right. So Jason, I'll throw this over to you. How do we, you know, you talked about speaking in bands, but any other advice about communicating the uncertainty, kind of the senior leadership. Have really begin to educate your business partners in regards to, I would say the top 80% the assumptions that impact the top 80% of your forecast. I think now that's the cool thing of the tools that we have, we have the tools to be able to present that and own that and really get the business partners in thinking about that and having that in mind when we think about projects. I think in terms of when I mentioned thinking in bands, it's the assumptions that support that, but then also logging actualism in the quarter ends against that and pressure testing the assumptions that you win and will. I think you hit the nail on the head. In fact, you did on the pressure testing those assumptions. So often I've seen people you make assumptions, you do it and you just report and if you're close, nobody goes back and looks, but if you look at the assumptions, you might have been close, but you are completely off on every single assumption. And you know, that might change behavior. And so I think that pressure testing is critical because it's easy to just look at a top line number, our close good enough when what's going on underneath may be leading to problems or maybe very different than the top line number and I saw you smiling there a little bit, but you've never seen that right. Never, I'd never gotten totally incorrect in my assumptions, but the accurate at the top, you're like, yeah, good enough. Yeah, if any of us done forecast, we've all been there. I think you'll laugh at us. I had one time where I had a bunch of businesses. And for the month, it was probably, you know, like, we'll call it $5 million, $10 million, maybe even more like $10 million, and I came within a thousand on the businesses forecast. I was like, you're an expert at forecasting, like, no, no, no, I got lucky. If I can actually forecast that well, I'd be retired right now. I'd vote somewhere exactly. I said I'd be in Vegas or I would I wouldn't be here. So that's really about those assumptions and planning. Then yes, there are areas you have to be accurate. But if anyone's being accurate with their token costs, they either really don't have any. They got really lucky or hey, come talk to me. We can start a business. One of the three. So all right. So just a couple more questions. I don't want to move into our FPNA section. So how should a company think about kind of the AI related costs? I think there's a lot of people trying to figure out what do I put in cogs, what I put in operating expenses. So kind of from a PNL line item. Love your perspective. One is an investor. How you like to think it. Yeah. And they just in general, any advice you'd offer. You know, FPNA professionals as they're trying to figure this out. I'm sorry with you can be on this one. I'll go to you Jason. Yeah, I think this is actually quite a hard one because before you can really like you are going to have to sign it. Other AI costs should go into like overhead R&D or what have you and some of them should go into cogs rate. So customer usage of AI should go into cogs. If you're doing outcomes pricing like, you know, the outcomes that do and don't work all that should be in cogs, I thought that would be my dream of how to see financials. But the reality is we are in early innings of developing infrastructure do this because you can have your meet or a layer, but that you need a reasoning layer in between that's actually saying hey. This section of the meter goes here. This section goes here like actually this was a successful outcome. This was it, but both of these go into like cogs. And so I think we need that type of infrastructure and that type of regularity in what the LLM costs are going towards before we can do this, but ideally you are splitting across those items. Yeah, and I mean, I think all of us would probably agree in an ideal world. We want to see it nicely bucketed out. Realities like we said is never as clean as what I mean, a girl can dream with her financial statements, you know, again, I dream what my bank account will grow overnight. It rarely does. but I keep dreaming. (laughing) Jason, your thoughts on this? - Yeah, I would continue on the dream, but then from an FPNA kind of ongoing, we're measuring ourselves, we're trying to see if we're improving, that creates an opportunity to kind of put some metrics around our ability to improve. You know, if I'm thinking about it, you know, hey, after a quarter ends, we're looking back, hey, what's our forecast bias? What is the forecast accuracy level? What's the persistence? If we can break those items out, it gives us a better kind of frame of measurement and then we can isolate on particular items to improve on. Because again, it's interesting when you look at this stuff if we're looking at a project, we're looking at the ratio between token calls, being less stand or orchestration calls. So if we can kind of isolate that ideally perfectly in the dream, that's gonna give us some opportunities to kind of, I guess, measuring score ourselves after and make sure we're calibrating correctly. - Makes a lot of sense. So last question on AI, don't we're gonna move into the FPNA section? So last thoughts, is there a framework that you would recommend people use? Any kind of last minute advice, you know, kind of for the FPNA professional, as they try to figure out these token costs, kind of moving forward. So we'll start with you, Jason, any kind of last thoughts, frameworks, anything you wanna share? I wouldn't share a framework, but I would say this is a time for the FPNA team to really sharpen their, I guess, research tool, really grow to research muscle, really grow your communication abilities with your business partners, your business partners. You know, hey, if it's marketing, hey, if it's HR, they're gonna be using different HR, they're gonna be using different AI tools. So having an understanding, keeping the lines of communication open in terms of what they're using, what projects they're using it for, and then begin to understand the costs behind those. So I would say one, you know, sharpen the research muscle and really open up the lines of communication, business and organization that you're supporting. Don, be what you're thoughts. - Plus one, to everything, Jason said, the only other piece I add is to just be open and transparent about, hey, we're all figuring this out. And this is our first pass at it. Let's iterate on it together, because nobody really has this figured out, like from public companies, unless they're already usage-based company down to like, you know, fancy, injuries in Sequoia based startups, like even all the investors I talked about all featuring it out. So wait, I really think having that mindset and, you know, collaborating with your business partners to figure it out, it's the best move. - Yeah, collaboration is huge. We are all figuring this out. All right, so I'm gonna ask each of you a question here, we'll go in order, and there's just two questions on the FPNA section today. So just looking for kind of a quick short answer, you can add the, you know, maybe a couple sentences elaborating, but just real quick, if I was to ask you, Jason, today, what is the number one technical skill you believe FPNA professionals need to master? What is it? Research, research in terms of cost, but then also, I think, and this may be move, would move outside of research, really understand the businesses and the organizations you're supporting, understand what they're doing, and then I think that kind of drives you to AI technologies that they're using. And I think you kind of have to own those two things in this era. - Okay, what's your thoughts on that, Timmy? What would you say is that skill they need to master technical skill? - At the risk of the old generic, I really do think it's like building your own AI tooling, like you can do so much more to connect systems and spreadsheets, everything else under the sun to get the outcomes you need. So, how can you make your work faster, better, more accurate, smoother, using AI to create like a dashboard or a website or a little like a tool? What's so great about Cloud Code and Codex, et cetera, is it? Really build your old little mini products for yourself. So, being able to do that and being able to bring that to teams ago, I think it's the most important thing. - Alrighty, what about soft skill, Jason? What would you say for that? - Soft skill, getting comfortable and being comfortable, being uncomfortable. I think that's the skill you have to build in this era. As Tammy mentioned, this is new. We're all figuring it out together. So, I think you have to be comfortable having those uncomfortable conversations that we don't know, but we're gonna do our best to present a range and manage within that. And I think that creates a shift within the organization where, again, it goes back to what I said about the variances learning. We have to kind of train our brains to not look at variances as a bad thing. If we're off, it's not this bad thing that we're trying to explain and move it away. And so, I think FPNA in regards to a leading AI transformation, I think FPNA has to kind of lead the shift in terms of way organizations have to think now. - Yeah, okay, it makes a lot of sense. What about your thoughts on soft skill? - I think coaching and advisory is probably the biggest soft skill that I see FPNA needing potentially just becoming more and more machine critical to their role. - Perfect. - All right, so I'm gonna ask each one get to know you question and then we'll let you go. So we're gonna start here with you, Jason. If you could add dinner with one person in the world, get her alive, who would you take to dinner and why? - Yes, good question, Dolly Parton. I think there is so much to learn from Dolly in regards to staying power and branding. - Yeah, no, she was loved by everybody. And everybody knew who she was. I don't think I ever had anyone. They may not have liked your music per se, but I don't think there is anyone who didn't like her as a person. She had an amazing brand and personality. So that's a fun one. All right, so if I was to ask you, Tom, what is something maybe a hobby you have or something you do something that a for audience learned about you think they'd be surprised that they might not expect? - Well, I wish I had interesting hobbies inside of the normal things, but I think something is they can't have reading, walking, playing with dog. Yeah, but I, I think one thing that's the whole unique is I actually, I went to boarding school for about five years in high school. So eight through high school. That was a little bit of a different experience. It was a really strict school. So a very different high school than I think. Well, so that was the fact. - Yeah, I definitely have not had the boarding experience. So that would be, I don't know, I don't know how I would have handled that. And I'm glad I don't come out at this point. - Give it then I take it. I have an early riser now and yeah. - Yeah, no, I'm sure there's some great, like I said, they're, we like with anything. There's always things to learn, good things to come from it. - All right, well, thank you both for joining me. You know, if our audience wants to get in touch with you, I must say I can both reach out to you on LinkedIn. We'll put your information in the show notes. No, Jason, they can go to Finhelm, your website if they want to learn more about that. And I'll tell you, you have your budget tracker. And so thanks again for joining me for sharing some of your thoughts today. I think this is a topic, a subject we're all trying to figure out. There isn't a perfect answer out there. And rarely is there, but as time goes on, we'll kind of all figure it out together. So hopefully you learned a few things from today. I know I did and thank you both for joining me. I really appreciate it. - Thank you. - Thanks so much for having us. It's great to be on. - That's it for today's episode of FPNA Unlocked. If you enjoy FPNA Unlocked, please take a moment to leave a five star rating and review. It's the best way to support the FPNA guy and help more FPNA professionals discover the show. Remember, you can earn CPE credit for this episode by visiting earmarkcpe.com, downloading the app and completing the quiz. If you need continuing education credits for the FPAC certification, complete the quiz and reach out to me directly. I'm Paul Barnhurst, the FPNA guy, and I'll see you next time. (upbeat music)