J.R. & Sonali Niswander (MetLife) fireside chat about AI Value + Tokenomics

Speaker 1 So Sonali, all right. We, we talked about 9 weeks ago and at that time, a lot of the conversations were all the shifting left and fin OPS going beyond cloud and all these things. And we got on our prep call a couple weeks ago, you mentioned that we should change the questions, which, which we did at the last minute to be more about tokens in this area. So I'm looking forward to digging in that area. Before we get started, why don't you tell us a little bit about your purview at MetLife? Speaker 2 Yeah. So I'm a senior Vice President at Global technology at MetLife and my remit scan spans across cloud, AI, developer tooling, dev, SEC, cops, enterprise architecture and technology governance. This is where where fin OPS sits and and we operated MetLife operates in across 40 countries in six different regions. So it's truly a global remit. Speaker 1 Awesome. So as you work across those areas and you were sitting in the Exec Forum yesterday with a lot of your peers, what were some of the top takeaways that you heard in that room? Speaker 2 First thing I'll say is the conversations were very real and very practical, right? So, and, and, and what I realized is that most of us are dealing with the same problems, the same questions and the AI value conversation is now happening. And I think you should have said this is not just happening at the CIO and CFO level. The Cisos are getting in the act as as AI and security starts to, to intersect. The other thing I will, I, I kind of took away from those conversations was that a need, there is a specific need for a vendor neutral standards. Lot of the frontier AI model providers are, are coming up with different models. A lot of us are trying to deal with how do we create our own cost frameworks, how do we measure value. So I feel like there is a need for a vendor neutral standards and frameworks and this is where I think the Tokenomics Foundation can can play a big part to help all of us. Speaker 1 It does feel eerily similar to like 2017 Cloud where none of this was defined and everything was absolutely. It's interesting you mentioned about other types of leaders getting involved. One of the conversations I keep hearing from people in the exact forum rooms is fin OPS area and stuff used to be just, you know, the VPSVP. And now suddenly as we get into tokens and AI value and AI spend, our CEO is asking about this, our boardroom's asking about this. And for me and my journey toward this tokenomic world was realizing just how much of the Fortune 100 leaders I spoke to said this has become a board level concern. So for you at your organization, which is large and regulated, how would you define this emerging thing, this tokenomics concept? What does that look like? Speaker 2 I think for me, I look at it as a way to manage cost consumption and value of AI tokens and really like how what is the all of that spend all of that consumption, what is it, what value is it driving and what business outcomes is it produced? Speaker 1 OK. So I'm going to throw you an unplanned curveball question then. OK. Is it different than fin OPS? Speaker 2 I think it has different dimensions. OK. It adds more dimensions to fin OPS to standard fin OPS. Speaker 1 So as we talk about tokenomics in your in your organization, how is that playing out right now? What does it look like in effect? Speaker 2 It's definitely playing out and it's top of mind. But at MetLife, we're looking at it in three stages and a lot of the speakers that spoke here before kind of touched on these areas. First one is consumption visibility, right? We're seeing AI consumption happening from whether it's your own internal AI platforms, whether it's SAS platforms or vendor API's. There's so many different vectors to hot where consumption is coming from and the instrumentation that unified view of getting visibility does not come out-of-the-box, right? So we're focusing a lot, a lot on instrumentation of to get that visibility, right. Once you have the visibility, then the question is, are we using AI efficiently? Then it comes to optimization, right? So there's a couple things we're doing there is the pace at which new models are being released. You know, as you guys know is, is it's not weeks now it's days, but we're we're looking and that's where the model routing architecture comes into play in the sense where that the conversation thankfully is shifting from are we using the latest and greatest model to are we using the right model for the right task, right. And that requires instrumentation of a well, a more efficient model routing so that that that intelligence is built into our our platforms itself to make the decision for make the decision in terms of the cost context to what model to route to. The other things we're doing is in terms of, you know, building evals like this decision, the trade off between cost performance and accuracy is not an afterthought, right. We're shifting it left to a design time to allow our engineers and our business partners to make those decisions upfront and and make sure that that we're keeping that cost context in mind, not after implementation, but right at inception. So that's optimization. And the third thing is value, value realization, right? How do you tie token spend to business outcomes and business value, right? How do you answer at my level, how do we answer the questions like not only how much are we spending on AI, but what value how much is that spend generating in terms of efficiency and in terms of revenue? And this is not easy, right? This this requires a lot of correlation of data across different systems to make those tie outs between what is your consumption to to value realization. Speaker 1 Well, you said the question in there, which is how do you tie token spend to value. And one of the things that I've noticed a lot of the conversations coming back to which is fin OPS Deja vu all over again is unit economics. Because in a lot of ways we're seeing AIB used for customer facing uses and applications which should tie to some form of business outcome. You know that cost per blank. So you asked the question, how do you tie it? Like how are you starting to tie those unit economic connectors of business value back to those costs? Speaker 2 I, I will say that, you know, we, when we started looking at AI, see we, we kind of started in the efficiency framing first because that was the, the easier place to start. And, and then, and when it looks, when you look at efficiency, there is a, a baseline, right? You have a process that was running for X number of hours. Now AI can automate that process and bring that that cost down by 4050, whatever percentage. So there's a baseline and there's a credible business case and value that it can be associated with it. Now as we're starting to see these use cases move from efficiency to more growth or revenue oriented, it's becoming a little bit, little bit more tricky because the question now is AI is allowing us to do things that we weren't able to do before. So there isn't a baseline, right? In that case, it's the the how we're looking at it is cost is our value hypothesis. So we start with the value hypothesis. We're doing structured kind of time bound pilots and then through those pilots, we keep, we define what we right up front, we define what success looks like and we measure our leading indicators in terms of what that, you know, how that pilot is either, you know, confirming our hypothesis or not confirming our hypothesis. And I think that has worked well for us in terms of just kind of proving how from a value standpoint, whether a certain technology is is able to meet the the value hypothesis or not. And the other thing it has done is it's also like we were, we moved away from these pilots that run forever. And it's, it's gotten us to a point where the pilots are either result from the producing results or we move on to something else. So it's it's given us that flexibility as well. Speaker 1 So one of the things Ambit talked about in the last talk was splitting out the two types of AI. And we heard this a bit in the exec Summit, which was there is the developer productivity, developer assist side of things. And then there is of course the product AI. And we hear a lot about, you know, the, the day one thing about the, you know, people hitting 3X their spend unexpectedly coming from developer productivity. But we're also increasingly seeing companies are saying, you know, there's a set of end user behaviors in our products that are driving our product AI spend. And it reminds me a little bit about, and to your point of, of new use cases, new value we're pulling out. When we were talking about Clyde migrations 5710 years ago, a lot of it was rehosting. It was like, let's get out of this place and go to this place. And everybody talked about refactoring and modernization, but a lot of the times people are just lifting and shifting. And so I think one of the interesting challenges you're hearing about is, are we using AI in the right ways for the right things to get value out and not just using it for something that we could have done in a different way before. So this whole concept that was also in that last talk of, I think, you know, almost workload classification to make sure we're using the right model for the right thing. You mentioned shift left. How? How are you shifting left that conversation and classification to say we're using AI for something that actually drives net new value instead of just a more expensive way to do machine learning? Speaker 2 Right, right. I think, I think fin opsin in general, if you think about, you know, fin OPS right at the conversation about fin OPS right at the beginning yields value. If you think about fin OPS and the cost, once you've made those decisions, once you've signed those contracts or once you've implemented these platforms suboptimally, then it becomes an afterthought. And then you come around and you have to remediate with AI. It's the same thing, right? How do we, as I, as I mentioned, you know with these evals, we're trying to put these the cost context, the decision making, the tradeoffs between cost performance and accuracy right at design time. So our business partners and our engineering teams are able to make the right choices as they're building these solutions rather than after the implementation. And when, when the chargebacks come through, right, that's never a, never a good, good place to be. So that's one example of how we're kind of we're shifting left and having these decisions right at right up front. Speaker 1 So one of the first times I heard anyone say the word tokenomics to me, I was at a global CIO event and there were three CI OS and they were all from giant financial organizations. And one of them was saying that from their CEO, they're hearing speed, speed, speed when it comes to AI and, and, and this global CIO, she said, well, my job is to say speed, speed, speed and security speed, speed, speed and efficiency, speed, speed, speed and value. So what is the impact of this tokenomics work for you internally on speed and scale of AI, which is now a critical existential board level conversation? How do you how do you not slow the pace of innovation with tokenomics? Speaker 2 Yeah, and, and I can tell you that that pressure is real. And you know, the, the instinct to say slow down because we need to get this right. That is also real. So I'll tell you that it's, it's like top down. And and as Jr. mentioned, you know, the CE OS want to go faster. They want us to go want us to innovate more and, and scale AI. The CF OS are asking, how are you going to generate efficiencies to fund more investments in AI, right? See, So's are asking how, hey, are we, are we comfortable with the control environment that, that we have to as, as you're, as we're starting to scale this. And then there is bottoms up pressure in terms of democratization of AI tools, right? AI is not just for these tools are not just for engineers, they're for everyone in the company. So that, that pressure is, if you like colliding and, and it's really falling to the CI OS, the technic technology leaders like myself. And the key is, is for us to balance, not only balance the speed because speed is required, but balance speed with governance and how we building governance into our platforms that allow us then to go faster, but still within the guardrails, where the guardrails are still built into the system. So to answer your question, you know balancing speed and governance to create value is how I how I look at. Speaker 1 One of the things that does feel really different right now is historically we've managed growth in cloud or anything else through a set of engineering controls, right? You have their your hundreds or thousands or 10s of thousands of engineers spending up cloud resources. And as you mentioned, now it's a different set of people. I mean, at the Linux Foundation, after the engineering teams, I think some of the most aggressive consumers of AI and spenders is the legal team, right? And it's it's amazing to see that. But you know, you've got marketing teams and sales fees and customer success teams. So what does that look like in a world where technology leaders and CI OS are used to governing just engineers, where you've got to lay roll out policies to all these other non-technical users and again, recognize that they might be using the AI for really business changing, impactful transformational items. So like how do you, how do you reconcile that so broadly? Speaker 2 Right. I think the key thing is not to stop experiment. Experimentation, right? Not to constrain people from using the tools, but also give them build the guard rails in the system, build the spend limits in the spend caps within, within your systems. And it's a constant education in terms of where should you use AI versus not right? When does a, a more deterministic rules based engine makes more sense and is the right solution versus using an LLM, a probabilistic LLM to make make that decision. So I feel like it's a, it's a, a, a balance between the two and, and we're, we're kind of trading that, that balance as well within at, at MetLife as as our user base is growing and the interest to use more and more tools are, is growing as well. Speaker 1 Excellent. So you mentioned model routing, that's coming up a lot. I've heard people building model routers. I've heard a lot of vendors talking about, you know, selling them. You know, are are you looking from the top at saying we're going to define a loud set of models? Are are you are you picking vendors? Are are you doing any tearing of hey, you know, these principal engineers get to use five point O fable, but you know, the marketing folks get this level and sales folks get a different level. Are are you looking at it that way? Speaker 2 Yes, in fact all of the above. What we're also, what we're looking at is kind of persona based selection where you have engineers that are working on complex things. You know, we're, we're giving them the choice to, you know, have the most reasoning model, the most expensive reasoning model. But at the same time, you know, you have someone in, in marketing or, or legal that we know kind of based on their workload classification. We're making the decision being more prescriptive about the, the choices that that we're, we're offering there and you know, for, but at the same time, from an experimentation standpoint, we're not constraining anyone from using AI, but being more thoughtful in terms of building those, those guard rails and the guidelines in for what type of workload is best suited or what model is best suited for what types of types of workloads. Speaker 1 So I want to double click on something you said earlier about balancing speed and governance. I feel like there's a pain pendulum swing happening in the industry. And I mean we saw it in the some of the examples I gave yesterday in the day when keynote. There's a iron triangle of most of speed and governance and value, you know, good, fast, cheap, all these things like are you setting a target for where you need to be for the whole org? Is it changing, you know, as things evolve day over day? I mean, every day we turn around, there's a new model is a team by team. How do you balance those things? Yeah. Speaker 2 I think it's a. It's a constant balance that we're trying to trying to achieve. And it's, and it also depends on the on the problem at hand, right? Not everything needs speed. In some cases, you know, you need more, you need to be careful. Like some of the areas where we really differentiate underwriting, for example, we want to make sure it's accurate, right? So efficiency is more important. It's not about going fast, it's about getting it right, right. So that's the trade off there where in, in some cases speed is, is, is more important. So it's, so I, I would say it's based on the problem at hand and how do you strike that right balance between speed and efficiency. And, and but for for engineers like us, you know, how do you make sure that the underlying governance still supports both speed and efficiency? Speaker 1 OK. So I guess last question for, you know, folks who are out there in this type of role leading it, what's the thing people need to be doing like right now to prepare for the inevitable CEO question of how are you managing the tokenomics? How are you managing this AI spend this value? What? What's the thing that you think is most important? Speaker 2 I think the I think the first thing is get visibility into what AI, how you're consuming your, your AI tokens. As I mentioned, consumption is happening from different places. Get a handle on you can't govern where you can't see. So the first thing is to be able to gain that visibility. And then in terms of how you make your usage more efficient, you have to build those guard rails into your platforms, right? And, and give give the visibility and the, the, the trade off decisions to your business partners, to your engineers, so they can make the right choices. And then ultimately, you know, we're doing all this, we're using AI to generate value. So remember, it's all about generating value. So that balance between speed and governance, the ultimate goal is to meet business outcomes and to generate value for the company. Speaker 1 Amazing. Well, I feel like we're in a really interesting moment in time. Someone this morning, 5:30 coffee downstairs was like, you're really lucky that you had your conference in June because this is the right moment you have this conversation. So I'm really looking forward to seeing what we hear from you next year as the whole world I think will be very different. Absolutely. Thank you for your time. Thank you. Thank. Speaker 2 You.