ReThinking: Fixing social media with Pinterest CEO Bill Ready

agent decay also we have to think as an enterprise distributed systems so it is very critical now for us to make sure the agent decay is not just working for a demo application agent decay is going to scale for millions of conversation providing and also safeguarding the platform so one thing providing the remarkable user experience to the customer while protecting our platforms that is going to be the critical key to rolling out agent decay applications for an enterprise creator. I have a struggle to get management to care about performance testing what if you could walk in with the number and say we caught our AI token bill by half that'd be crazy right that's the kind of performance engineering we're talking about today and I think it's an awesome career proof skill to have in testing right now hey it's showing welcome to the guild I started my career as a performance engineer twenty six years ago back when I had bare metal a race floor and everything under my control and today's guess is going to help me how to learn a lot of that kind of Sami Salvaraj is a principal architect engineering leader in performance evangelist over twenty years in performance engineering and for the last three years he's been talking about agent decay applications from demo to production for the scale of millions of conversations he's also wrote a free book on it rethink in performance engineering for a genticae I will have a look for that down below and in this episode you'll discover how to performance test something that's non-deterministic why you need outcome based SLOs at the trace level and how to ship performance gates left it's every commit and shift right with sedentic monitoring how to cut token costs with prompt caching and model routing and the one question every team should be asking not how smart is my agent but what does my system let it do when the agent is wrong you want to miss it check it out hey could somebody welcome to the guild thank you Joe it's great to join me great to see you and thank you for having me on on this today's show absolutely no we may I forgot which IRL was was it a link or a Nashville Atlanta Atlanta I'm looking forward for you to meet up come again and host there was a great event and thank you for hosting that event in Atlanta absolutely and what I loved about the in-person events is I you know I get to meet people like you in person and you would tell me all about performance and how you know you really are going heavy into performance for AI and I was like oh you know I started my career as a performance engineer how's that different than than today so I thought we dive into those type of topics so I guess before we get into it you know you did write a book called re-ticket performance engineering for a gentle AI okay you know it takes a lot effort to write a book the book is free for people to know we'll have a link for it down below but why do you do write this book yes Joe so pretty much I mean since I'm into this performance engineering for last 30 years but last three years I started this performance in agent decay journey sometime the end of 2023 so over the last two years it was I mean all my production experience so how can I rethink basically I mean I have to unlearn some of these items what I what was the playbook which I was using before and how can I how can I approach the same blue playbook in a different angle treating I mean how the agent decay is going to operate at the scale not just system I mean basically difference between the systems working and the systems working at the scale right so that's a two different so basically this book is all about how can I help the community addressing those challenges whatever I mean basically part of my learning and taking your application from the demo to the production grade enterprise agent decay which is going to scale and handle millions of conversations per hour so that was always the issue I started my career as a performance engineer like 26 years ago now we had the bare metal we had a raised floor I had everything under my control and you know we used to test and staging and it was never the same as before as performance and production with AI I assume it's it's even more difficult because like how do you simulate like token usage and like one of the type of things I would need to think about now if I was a performance engineer and I have to rid of myself and maybe notions from 26 years ago. So just before this agent decay right it's always a fixed path pretty much the thing you cover most of the session now the biggest challenge is that non-deterministic and having and probabilistic replacing the deterministic year earlier when we design something we know the fixed path we it's a predictable workload that means if you know the response time if you know the request rate we can we can variable predict the workload but now right now with an agent decay the same request which takes 3 seconds in one execution in another execution it can take 8 seconds because it determines the number of tools calls it determines the number of reasoning on the fly not before we requisite to get whereas that is the core difference that means even when we as a capacity planning when we do the infrastructure planning we we mean if the way they we we planned our capacity for our deterministic systems now even we need to we need to plan the capacity planning for our infrastructure to handle the non-deterministic aspect all right so how do you handle something you don't know like the you know I mean like I knew this transaction shall return the response time in 3 seconds now it's like well good I don't know with how do you put godrails in a way that you can test it so it can be at least do you have to put godrails on it so you can make it deterministic for performance tasks or you just accept the non deterministic aspects of it correct that is where I like to bring that my book also is going to talk about a lot of ordnance topic you know like basically your prompt is what agent should do your ordnance hornest is what agents hello to do what system hello to do even when it is right when it is wrong that means harnesses the new I mean harnesses the new mechanism in the part of the design design right now it should tell in a worst case scenario how many tools calls it should stop the loop in like what is the number of I mean how many reasoning it can do and similarly what is the number of tokens it can exhaust per per the entire trace that means earlier we looked at the span level but right now instead of looking at the span level we have to look at the outcome based level that means did the agent answer that workflow did the agent completed the workflow completely within the within the given godrails for example before some change your agent is able to handle the answer with three steps basically completed the three tools calls three reasoning and I have provided the answer now with this with this godrails we have to but we have to particularly enforce I mean it cannot it cannot go more than three or four steps but in this case I mean if for some reason if someone changed a tool agreement if there is a change in the recent agent aka changes your step number of steps that means the efficiency the number of steps completed per trace increased from three to eight that's probably no go to production all right so you know it's always been difficult to get functional test part of a pipeline build before when people check in code before it's it's promoted to production and performance has always been a different thing so it's performance a different than environment that it goes to when you check it in and runs these godrails tests and then it and then this is okay performance is okay let's push it further it's a mix and match you for example I mean I did put together this shift lift performance engineering even the way back in 2016 and 2017 as well as this cloud so that means my goal always performance environment environment for some scalable environment similar to production whereas I mean but doing this performance at the later later cycle in the performance is always going to be too late before it goes to production so my goal is to shift lift performance even in the lower environments like a date fit environment but the biggest challenge week the those testing the lower environment it's not about scalability it is just purely an regression that means last time it completed three steps now is it going to do the same three steps or is it going to do the four steps so just putting the minimal load what system can take I mean that means not about stressing the infrastructure that means we understand the system capacity that capacity is not big enough and put the load that is what it is reasonable for the environment you have a baseline I mean saying like I mean last time for this conversation it took like a four second now with my recent changes that does the same conversation because my my input remains same whether my output remains same or not so taking the baseline and benchmark that is for my lower environments even before coming to performance when it comes to the performance that is the place where I am going to put 10 times of production load 10 times of production load and doing some kind of a scalability and and also running some and resiliency test also because the resiliency test means I need to do some kind of a fault injection and see basically I mean what happens if my database goes down what happens are basically how my agent decay reacts when one of my dependency tools of this goes down does it end up in retrace storm but because when it goes to the retrace storm basically it's going to I mean kind of it's going to cause a token extra saturation so that's the ideal difference between when I say shift lift shift lift on the lower environments it's purely about a standard set that means it it is doing the same thing every every commit before that code merge to master whereas the pipeline I mean whereas when it comes to the performance a dedicated environment something a comparable environments similar to production where it can run that test for extended period of time all right so we say shift left then is there a way we can shift to even more left with the developer are there like skills or are they like a performance agent they can run to make sure they're code is being performant, not necessarily stressed or just code itself is clean as performant as possible. Is that part of it as well? Yes. Yes. So basically, it can treat every commit. So even within the pipeline, even there, we have given a couple of options where developers can go to this some, we have a standard kind of like a, I mean, the developer platform where developers can go and prompt, okay, any performance gate. So basically, we call as a performance gate developers can prompt on on demand basis, even in the local desktop also, or else when it comes to the pipeline. So before all this, we have a two different gate. One is the functional regression. After the functional regression, there is a security and there is a performance. Once the security and performance gate is completed after the functional gate, then the code merge to master in the lower end numbers. In our CAC, pretty much, this is within our CAC, there are concourse or Jenkins, I mean, pretty much, I mean, basically, it's nothing but a job. So, but the job, the way we design, the job is a framework that is plugable to any environment. Maybe, I mean, the URLs, the endpoint may change, but the volume and the workload remains same except the performance where we keep to 10x of our load. And we do shift the right also. That means shift lift is a concept for a lower environment. Shift right is also another environment. That means for example, in production, we have a blue and green deployment. Whenever there is a lane change, we make sure, okay, after the lane change, run a quick sanity test, we call as a synthetic monitors. We run as synthetic monitors in the production and make sure everything is good to go. If there is a synthetic monitor, some failures, then we roll back. Love it. Yeah, a lot of people like, yeah, it's a good point. Focus on shift left, but shift right is also just as important, especially with these types of systems where you really don't necessarily know how they're going to act in performance in production, 100 percent, I would guess. Got it. Got it. All right. So, you know, I was looking at your book really quick. And I think in the first chapter, you have a chart that looked fine, the dashboard looked fine, but when you customers were reporting how it was timing out for them, what caused that? And is that a common issue? You see that a dashboard for certain performance issues with AI looked good, unless you actually are measuring the right things in the dashboard. Correct, Joe. So, typically the dashboard, like, let's come to the SRE, right? Typically, they are the one ensuring every application is healthy. They typically look at the response time, error rate, availability, any restarts, right? Whereas in this case, what is happening? I mean, the reasoning, in this case, we cannot, I mean, in addition to the response time, now it's a challenge right now. We have to measure the reasoning time, right? Because your reasoning time can differ based on the user query and the context, how the context, I mean, how the context is growing. At some point, what happened? Your system prompt might bloat. So, when the system prompt might bloat, at that time, it will end up bloating the conversation again. So, I mean, so in this case, for example, there, I mean, agent answering, even though it made some of the checkbox, like a response time looks good, your throughput looks good, no errors, but however, what if it takes five times of tokens? What if it's take five times of token compared to my baseline? So, that token ended up showing in the cost budget, not into the, I mean, not into the regular performance matrix. So, that is also like another key matrix, like how the cost is inflated. And similarly, and also another primary metric, we wanted to make sure one agent, I mean, basically this one agent is not going to impact the entire platform or infrastructure because we were one agent end up taking all the token limits on the one AA provider because at the end, pretty much 40 agents are using the same AA provider somewhere and this a platform is shared between this multiple agents. We wanted to make sure the one agent is not an anomaly, ending up taking all this token saturation, not putting the other 13 agents in the risk. Nice, nice. So, also in the book, I think you addressed this. This is something I used to know, I mean, is a lot of times people have tried to perform its test, something in the browser, like a file download for some reason. When I was like, well, that's not, that's not our application. That's the browser we would assume Microsoft had done that already. Is anything like that with performance LLMs that you see people trying to perform its test something that may be enthrapic or, you know, an LLM should have already covered that you shouldn't be worrying about, I don't know if that makes sense, but is there like certain functionality that people are trying to test that or he would have been tested as part of the model and that's not something that you should be focusing on? Yeah, yes, but our primary focus show, I mean, because pretty much hours is like, I mean, pretty much any of the application so far, I have worked is not about the user experience of this LLM, but user experience of her own application embedding that, that agentic models. That means, at the end, we are connecting to some this platform, A, I mean, A provider endpoints and we are, we are processing their inference and we are, I mean, we are providing the user experience in our own applications, right? So coming to that one, I mean, what it's very critical, even though my API matrix looks good, but at the end, for example, if it's a chat application, I mean, there is a, I mean, there is a one logic, even our, when it's streaming, that means, for example, one of the thing, when we are doing the streaming, the time to first token, the time to first token is always quick, but when it's not streaming, user don't even know whether basically, I mean, what is happening is, can simply waiting on the screen, looking for a response. So one of the first recommendations, we need to put together a streaming, that means, user's understand, there is some processing happening and user is a while, I mean, that way it doesn't, user does not perceive, even though it is slow, user does not perceive, it is kind of slow. And second thing coming to you were as, right, how, I mean, what exactly, we are testing, when LLM is already testing their application, for example, in this case, our ask is that user end-to-end experience. So even though not measuring from the infrared standpoint, from the time user asked the acoustic and how much time did it take to complete the entire workflow and give an answer back to an user, we use a standpoint, measuring this performance end-to-end from the user standpoint from the customer. So you would just start with a user experience with the video, is that like, like, I have an application that I do a scan, it could take 47 seconds, so I'd show quotes while it's thinking, so people know that it's still thinking. So even though it's, that necessary slopes, I expect it to be that amount of time, but for the user, I'm trying to give them an experience where they're not perceiving it as slow. Is that a common thing you think people should be doing more of? 100% 100% Joe, because before the Agente-KAA, there is not a much concept of streaming on, but however, in the performance world, always be measured, what is the time to, what is the first buffer for your page to download? That means you're just not downloaded fully, but at the end, what is the first buffer? That means servers started receiving, servers started sending some response back to the browser and browser is trying to load. Whereas when it's coming to the end user, maybe user does not, I mean, it does not matter to the user experience, but when it comes to the Agente-KAA, as you said, exactly, the streaming is very critical that way, because some of the reasoning, when it goes to the LLM, there are certain things even, which is not even controlled from our side, because LLM is going to take its own time. Something we cannot do a lot. So, what we can do? What data I can feed it to LLM? That data we can control, in terms of reduction, reducing the token, but in this case, streaming is what going to make user interactive, mentioning, okay, I'm doing the work, for example, if I'm uploading an application, I submitted a payment, it should say something like the payment is processing, giving you the confirmation, please, I mean, something like a meaningful conversation, even that way the user does not perceive, use it does not feel like he's waiting on something, because he understand, the system is doing some work at the back home. Right, right. And I can say a lot of people's wasting time trying to tune something that's not very tunable, where the user just wants to know works being done, and it's not fooling them, but I think there's a studies that have been shown that, if you kind of have even like a little hourglass of something, showing them that works being done that they don't perceive it as slow as much. Totally, Julia. I'm giving the context to what exactly the system is processing when he's waiting and making sense of there is a progress being made. Another thing in the book that popped out is I think you talked about stubbing, and the way you worked it for low tests is you were using latency distributions, sample from production traces. Correct. So, how do you make sure that that stub is being honest as the provider might drift normally? Correct. So basically, when it comes to the latency stubbing, for example, in this case, some places, first of all, in the traditional world, we have some kind of like, I'm just giving some examples. We have, let's say, some of the credit pool application, right. In credit pool, there are basically three vendors outside, right. I mean, like a TransUnion or Equipase or some, I mean, there are three credit pools. But when it comes to the scaling my own application, every time it's not challenged, it's not possible to run the end-to-end test with this credit pool application, because if they slowed on, they're going to slowed on our application also. But in this case, to make sure my application is scalable without having dependency, the stubbing is what is going to really help avoiding those dependency, keeping one variable same at one time. That way, I can focus on tuning the application like what is on our control, keeping that other variable same at the same time. Whereas if every time, if you don't stub, basically that end-to-end latency somewhere and some other dependent application, it's going to cause an impact on scalability and our application. So in order, the reason, the main reason why I why a depend on stubbing, I mean, basically to one thing, to get a consistent baseline because when, I mean, there are, we have a two different test. One is that stubbing test. That means keeping all the variables same in the other layer, keep, but, I mean, focusing the application on our own infrastructure, whatever, whatever the primary focus on our critical application. That way, the other very, I mean, that way, if something's closed on, it shows up immediately, that way, we say, it's probably no good to production. But also, we have another layer, where it covers end to end. That means that is what, typically, the user is going to experience in the production. But the challenge with end to end, it's always there is a slowdown on some other layer, which is going to probably slow down our internal application. So, for example, in this case, if I identify, if I need to find a breaking point, if I wanted to put a 10X load on my application, in this case, if it's a stubbing, I can achieve the 10X of workload. Whereas, when it's a end to end, if some other dependent application, not going to allow the 10X load, I may not be able to put that load of 10X even within my own application, if that makes sense. Totally. And also, this is completely automated. So, that stubbing also, your answer, your questioner, how can you make sure that stubbing is accurate? So, we have automated jobs that goes and measure the production metrics. For example, in this case, credit pull application, right, this occurred for the application. It goes and measure this metrics from the production on a weekly basis. It's a week, every time when my stub, my average time should look like, so that should be reflecting the production. That is very critical as well, to simulate the production type behavior. All right. So, do you need then to have SRI set up first to do these open telemetry type traces so you really understand performance and production to get good results, to be able to trace, you know, what good looks like in production, and without that, it's like you're kind of running blind. Yeah, definitely so, coming to this agent decay, that's another big challenge. I mean, see, earlier, with your APM application performance management tools, it's always a fixer path. That means, service A calls a service B, and we call it a database, it's a fixer path. Now, with an agent decay, it removes that, it's removed that fixer path and bringing the dynamic path, right, at the dynamic path, the most important advice to anyone starting getting started on this agent decay, observability is a key even before even your application goes to a production. That means start with something simple set up a open telemetry, and there is a lag, lag fuse, lag fuse is completely open source observability platform or a agent that shows you how many tools calls made. What is the token, whether in within the token did be incorporated some cash rate, cash input tokens are not, and how many reasoning it is happening? Does it end up making the so many reasoning loops, and whether any context bloat suddenly between the step, I mean step one and step two, we made a step one, we made a tool call, we got the complete responses, that's like a raw data, raw data of JSON. Now, between the step one and step two, if your context token grows from 5,000 to 50,000 token, that's something wrong, because your first alarm call may be like at three seconds, but when the tokens show grows from 5,000 to 50,000, your alarm calls takes up to like a 12 or 13 seconds, because in this case what we should have done, we should have taken that first tool call, we should have taken this summary data, instead of giving us a raw data to step two, if we could have given us a summary data, our token data could have could have grown from 5,000 to 7,000, but not from 5,000 to 40,000, because it comes with a challenge here, like basically speed cost quality, right? So for every second that we are trying to cut down, there is a tradeoff on the quality. So it's a speed cost quality is another decision, the product leader has to, and not this decision has to be made even at the product standpoint. So what kind of tradeoff that we can afford? You just talked about tokens, random thought, like is the power performance now also looking for token leakage maybe or make sure you have some sort of policy in place that if you get a request for X amount of tokens and X amount of time, not to do it, because you don't lose a lot of money, like are you now involved in any of that type of scenarios? Correct, Joe. I mean, definitely in the last one there, not before last, I mean, at least we started this in the mid of 2025. So right now our SLO is kind of outcome based SLO. That means what we measure within my 100% is my 95% of the traces completing the workflow with 5 steps and total of total 2,000 and we don't look at the step level, we look at the entire trace level. That means if it made 4 LLM call part of the same trace, we make sure at the trace level did it complete the entire trace under 5,000 tokens per trace, not at the step level. So that's something we started implementing this practice in the 2025. So the reason why it is important, because if you go at the span level, I mean between the spans, the token context grows and there is no compaction strategy, your token keeps on growing. But when we go with this journey, like when we go with outcome based SLO, as long as our 95% of the traces, 95% of the conversation, I mean, so the agent conversation completing this workflow with the 4 R5 steps, achieving that 2,000 tokens per trace or achieving that 4,000 tokens per trace, then we understand this is within our SLO. Once again, not part of the book, random thought in my head. So can you do performance testing to try to work on token optimization to get the token usage down? Like could someone be bad performance because something's consuming an amount of tokens that is ridiculous and that by optimizing somehow, knowing that like random sequels, not random sequels statements, but hard coded sequels, dynamic sequels statements, I would cause a lot of performance issues. I'm thinking, then seeing now token usage, almost like that where people aren't using the tokens correctly and therefore could be a performance issue that leads to a lot of money being lost. Correct. Yeah, definitely. Just to get started, Joe, like even when we go to the LLM provider, if you share cash, share it input tokens of every LLM provider, they have the cash input tokens as well. So we can configure how much cash we wanted to use also. If it's a regular tokens, we pay 100% cost. If it's a cash token, we pay only 10% of the cost. So prompt caching is the first layer. So we have categorized us like like a optimization layers within how to optimize the tokens. The first layer is the prompt caching. That means if your system prompt is effectively cached because let's say two conversations, it'd be the same system prompt, maybe it's the same data where you're in a traditional engineering world, like basically how we leveraged a caching, here is also like 100% we can translate the same logic into the prompt caching layer instead of caching, we call as a prompt caching. That means it's going to use 10% of the cost comparing to the 100% of the cost. That is one thing prompt caching and second thing, capping the conversation. For example, let's say user keep on asking 20 conversation or 70 conversation because user wants to type something and get, but are they we have to make sure in order to remember the context, how many conversation I can keep it part of memory because at the end of our context keeps growing, it's going to load the system conversation is going to load at some time, then it will put up with then we will then we will put more risk. That means the agent does not understand what user initial cost in, so it's very important what conversation we have to keep in the short term memory, what conversation we have to keep in the long term memory. In some cases what we do, we don't put everything in the agent context, we put this into something like AWS agent core memory, so agent knows what is my last state, after even if that way, it does not need to have this data every time in the system, but user has something it can go to the agent core memory, it can retrieve, this was the question user was asking me yesterday. So agent has a memory to that because that agent memory is something a new new new, it's a new new thing in performance engineering, we have to make sure giving that memory is going to make even agent even stronger. And the third thing coming to the token optimization show, so that way because with this prompt cashing, as I said, your tokens, that means even if it's $30,000 tokens, your cash tokens may be like a $10,000, maybe that means we are it's going to charge only $10,000 tokens at the full cost, another $10,000 token is probably the 10% cost. And third thing, we recently we came up with an optimization, like your system prompt, maybe some duplicate instruction, that is not even needed, we have to optimize that avoiding this duplicate instructions in the system prompt and primarily, like another thing, instead of feeding the raw data to LLM, it's always a best practice to summarize the data and feed the data to LLM, so that way our token count is going to the significantly coming down from $50,000 to $5,000, and when we are going with the summary of data, rather going with the raw data. And the reason why I'm holding in on this topic is I'm just thinking like a lot of times it's hard to get companies to care about testing and performance testing, it's like, oh, it's too difficult. But if you could say, look, we did performance testing, we actually have a number, we saved you X amount of dollars because we've found we've been token optimization. It is sure as your career basically, you're justifies your role, I don't know if that makes sense, but like I think being able to tell you a management that you have a real cost savings from the activity you're doing is a big win. Correct, Joe. And just coming to the cost saving topic, you know, that is something also we've been educating like pretty much every developer community user in this particular and in agent decay, there may be a different, we call utterances utterances, nothing but an user query user user query to the agent, right? So we can classify this query as a simple medium and complex. So we do sometimes I mean the mistake. that what I'm seeing in the market, people fix a model, no matter the respect of any user query, it is always going to some expensive models. Rather, we can classify as a simple questions, go to the simpler model. So that's going to cost like a, I mean, 50% cheaper than the other premium models. For example, if you have any complex query where it requires some reasoning, then go to the complex model. But if it's a simple query, that means you already have a PDF somewhere in the database somewhere store. Now user query show me show me show me my PDF. In this case, you just, we just don't need any reasoning. It can just go to the very simpler model. So they were cost is 30% comparing to the premium models. So that's another category that based on the skills. Basically, we have the skill document based on the query. It's all mapped to every skill document. If it's a simple skill, go to the simpler model. If it's a complex skill, go to the complex model, only if when it is complex. So this can cut down, I mean, a lot of cost and particularly when we are handling millions or because this may not be effective when we are doing like a hundreds of conversation, but this is going to make a meaningful difference when it's processing millions of conversation in a day. I mean, it's a big difference. If you're just using frontier models, I just said to teach myself, I was setting up a claw, claw instance open claw. And it was using a frontier model. But it sucked my tokens away like this. And I just switched it to Google Gemini Flash or something. And all of a sudden, it's like, so I mean, that's just my little piddly app. I'm just assuming like, I mean, developers are thinking of this before it goes to production. I'm guessing there's going to be a lot of performance bombs going on soon, where they're like, Oh, whoops, I didn't think of that. You know, that's a great point. How many people are thinking of that? What model are you using? And like, actually coding it to make sure that they're in the way correctly, that way. I mean, definitely, we are in. I mean, most of the architects, I mean, that's what we are trying to enforce this idea, but I mean, right now, if I look at the practically, what industry is doing, everyone is focusing on agent stability, agent functionality, giving agent giving accuracy. But as I said, the major thing, the systems working at the scale, because I mean, when we do the pilot, maybe we have a curated response, everything looks good. I mean, everyone is happy. But when we are processing millions of transactions, that is where this challenge has come. Pretty much, I think, right now, I think, I mean, this is something I think, I mean, but however, I feel like this should be the part of the design discussion, not being an afterthought, whenever there is an agent decay discussion. So this should be a part of any agent like, like some of the concept, like harness, which did not exist before, right, harness is like, okay, does it? I mean, basically, I mean, right now, everyone is thinking about how smart my agent is, but we also have to think about, okay, what system let agent to do when the agent is wrong? Because sometimes agent is wrong, but however, no system, no, God is being more important than whatever we had before. Absolutely. Now, this book, and we'll have a link for a down below once again, is it covers a lot of things, but you did leave out things like, I think you left out security, prompt injection, adversarial robustness as being out of scope. Why do you think that's a different topic by a different team is it just like you, there was out of scope for this book, because you want to keep it really tight on this one problem. Correct. So basically, this book, I mean, I wanted because there are so many things to cover even within the performance engineering. As I said, like, I mean, everything, I mean, it was my learning and practically deployed and there are what are my faith, I mean, what, what things work out well, what things did not work out well. So within the performance itself, there are too many concepts. So pretty much this book, I want to focus on the performance and observability, because because security is like another, like, I mean, it requires like a lot of different aspect. So that's what I'm holding on performing this book, particularly focused on performance and observability. But in my future, the future articles, then I'm going to cover some of this performance security also. But in the security, since it's not my primary core of expertise, too, I wanted to focus more on the performance and observability to be the major key area of this book. Gotcha. What's Ken? I wasn't able to get through the whole book. So I'm just curious, do you cover anything about this, maybe a dumb, dumb question? Once again, when I started performance testing years ago, I had different protocols and I had a bio license for this protocol, but I had to tool that handle all the protocols. So I was good. Does that, does that matter with LLM's like, do all, like, what chip performance tooling? And do you have to do anything special to have a handle testing LLM or AI? Correct. So in this case, Joe, pretty much we go to an HTTP layer or we have something kind of like a true, true client layer. In this case, user experience, if we go with the user experience, then we go by the kind of a chatbath application style. So where we go with the user interface layer, right? That means that user interface kind of a downward, say, entire JavaScript, pretty much doing whatever the browser rendering engine. So it is more focused on how the user experience from the UI standpoint. But when it comes to the scalability, because when we cannot, that's what I mean, I've been educating most of the team, we have to make sure we have to get started the AP layer first, because if the AP layer scales, then UI is going to scale. But when we don't even make that AP scalable, some people going jump directly at the UI layer without tuning at the AP layer, that means then it's going to like a reverse engineering. They have to come back again to the AP layer, I mean, my strategy being primarily starting this at an HTTP layer, right, HTTP, I mean, basically HTTP to layer and ensure this is and also along with the streaming platform, that is the biggest challenge, because now we did not have the protocol buffer and everything before. Now the streaming now, it's important to cover the protocol buffer. So coming to that, so HTTP to layer is primarily covering this protocol buffer layer. So start your agent decay journey in the AP layer first, because in this case, what we do, we don't necessarily load test the models, we primarily load test our applications, interact, how the application is interacting with these models. So our primary focus, how our our our our APIs, our application APS are scaling well with an interaction to these models. That's the first layer, then we go to the UI layer, Joe. Gotcha. Can you tell us what tooling you're using? I'm just curious to know or just about it. Yeah, correct, good. So we use primary J meter. For example, in this case, we use J meter for any pipeline level, because when we give it to the developer experience, I mean, developers are more comfortable with the J meter. So we have taken J meter as ever, anything on the pipeline, anything to kind of a stress test or scalability, we have a different set of tools. So one is called that so I start cloud test, I think right now, it's like I can back them. I that is the one tool, the cloud test and another one is probably the open text performance center tool. That is another tool that we use as well. Who is the name of the open open source tool you did for tracing was a wing? Langfews. Langfews. Langfews, RIs. I think RIs recently some company acquired RIs, RIs also also an open source, but some company recently acquired RIs. So pretty much we rely in terms of the observability, Joe. So we primarily use Langfews, but right now, RIs is an another standard platform, which is getting better in terms of an observability that basically that gives you the complete context of telling like how many tools calls happen? What is the modality? What is the model it invoked? Basically, people can, I mean, by looking at the observability, they can give the exact breakdown, what are the steps happened within their trace level? But I would recommend anyone, if I mean to get starting, Langfews would be a good start, Joe. Okay. Cool. Who does performance testing in your organization? Is it still a specific role as a performance tester? Is it an SRR or is it a tester? Or is it a developer that does everything? Like how does it work? So coming to the role in RIs, Joe, pretty much, we what we do our team performance, we say, I mean, right, there is no, we call us a performance engineering, the testing is, I mean, those kind of, I mean, we stopped doing like maybe a 10 years ago. So right now, I mean, a performance engineering, more about, I mean, more than two more about tuning and scalability and ensuring giving the, I mean, the production user experience. And also, testing is part of it as well, as you said, exactly. So coming to our role, right, so developer experience, giving the developer experience, we develop certain framework, we give the framework in the lower environment. So developer, the framework will take care of handling the performance in the lower environments, that is on the lower environments. The developer should get a handful of performance feedback, as soon as the code chickens into the branch, the master, the developer does not need to wait for a performance team to give a feedback, where developers get an instant feedback from the framework, from the pipeline level, whereas coming to the performance team, so that is where our team typically does the 10x of production workload. So coming to that, yes, we could go with the performance engineering team. So right now, the observability, we are part of that kind of a sorry team, like basically the performance and observability develops. So basically, we go with platform engineering team. So we are a part of platform engineering team, pretty much performance, observability, DevOps, we are all part of the same umbrella. Awesome. Alright, before we go, what piece of actual voice someone could take away right now and put into place to help them with their AI LLM testing? Yes, you. So just just to just to give you the real kind of a real simple layman layman terms, right? For example, if it's an empty road, the car can go at whatever the speed it wants, right? But when you when there is a company developing in 40 agents, and these 40 agents are trying to use the same a provider, we wanted to make sure this one agent does not impact that other 39 agents. And for example, like there are like a 43 drivers all trying to use the same three highways, meaning like a 3A providers. If one truck driver does not know how to merge on the highway, can significantly back up the truck drivers, even though they are doing everything correct. So that's one thing I would recommend. I mean, maybe I think it's a time for the team to ask these questions, right? Because can you, basically, I mean, can you bone the autonomy? Can you observe? Can you recover the agent? So these are the things. I mean, I want team to reconsider. For example, in this case, the observity is the key. It's going to be a critical pillar than ever before, understanding your agent take. In this case, one thing, as I said, Joe, like I'm not just asking how smart the agent it is and also asking what system let it do when even when the agent is wrong. That's something we need to have more confidence on the system than ever before. So this is the practical guide. So this is not a typical theoretical. This is the practical guide that team can take, can just read this book. This is not a long book. They, I mean, probably they can just complete the book and two sitting and they can apply this learning even within their first and two weeks of their agent decay application. And this is all practical to achieving the meaningful outcome. What are the significant, what is the outcome that you can drive your agent decay from like not just from a demo application to the production, enterprise, scale, agent decay application. And as I said, Joe, like I mean, just giving one thought, even this book saves you one incident from your week and incident and then this book has done its work. - Love it, love it. You can find links to all this awesomeness down below. How they recommend you check it out. - Thank you, Joe. I think yeah. - Thanks again for your automation awesomeness. The links have done a value we covered in this episode and I'm over to testgild.com/a603. And if the show has helped you in any way, why not rate it and review it in iTunes. Reviews really help in the rankings of the show when I read each and every one of them. So that's it for this episode of the Testgild Automation Podcast. Joe, my mission is to help you succeed for creating end to end full stack automation awesomeness. As always, test everything and keep the good. Cheers. - Hey, thank you for tuning in. It's incredible to connect with close to 400,000 followers across all our platforms and over 40,000 email subscribers who are at the forefront of automation testing in DevOps. If you haven't yet, join our vibrant community at testgild.com where you become part of our elite circle driving innovation and software testing and automation. And if you're a tube provider or have a service looking to empower our guild with solutions that elevate skills and tackle real world challenges, we're excited to collaborate. Visit testgild.info to explore how we can create transformative experiences together. Let's push the boundaries of what we can achieve. ♪ Over testgild automation testing podcast ♪ ♪ With loops and liars the bugs began their song ♪ ♪ Tooth of knowledge, a melody of code ♪ ♪ Through the air it spread like wildfire through the land ♪ ♪ Guiding testers showing the secrets to behold ♪