How Hex Builds AI Agents: Making Agents Reason Like Human Data Analysts | Izzy Miller, AI Engineer

- Over night, internally, we released this and everyone was just freaking out. I was like, "Oh my God, this is it." - Today, I'm talking to Izzy Miller, an AI engineer at Hex. They've been shipping data agents since before most teams were even thinking about them. - And it was increasingly feeling like we had this high horsepower beast driving 25 in a school zone. It's like the thing we are harnessing here needs more context, not just from the Selba, from the whole project. - Izzy describes one of the failure modes of building agents and how access to more context in user behavior unlocked big gains. - And if you inject in conflicting context, it will spend 30 minutes pondering, going, "Wait, let me see, hmm, actually, "and enter this crazy collapse mode." - Getting agents to reason as well as a human data analyst requires some cleverer approaches. I dig into how Hex started with separate agents and why that's changing. - People would be like, "Oh, why can threads do this, "but why can't threads write Python, "but the notebook agent can, "and so unifying these all together." Which has had interesting code level impact on the way that we organize our tools, prompts, and skills into bundles. - Right at the end, is Hex walks through the Eval test suite he's building, a 90 day simulation, where agents learn from context and improve over a longer arc of time. - If the agent is demonstrating the skills and behavior that we want it to by day 90, all of the questions and the tickets are carefully crafted such that it should get 100% of the questions right. Sonnet 4.6 gets 24% on day 90. - It's very expensive to run. I need credits, please. - Welcome to Max Agency, the podcast that goes deep into how the best agents are being built. Like builders like you. - Is he really excited to have you on? I think Hex has been one of the companies that pushed the boundaries of what's possible in the agents you guys have launched a bunch over the past year. We use a lot of them internally. I think the notebook agent is one that I hear rave reviews about. What is that and how did that come about? The core of Hex has always been this notebook interface in which you can, you know, intersperses, cells of SQL or Python or text or charts and construct very complex or simple analysis in a way that's very scalable, approachable, understandable, like literate programming. I think we were the first real product to ship a text to SQL feature to real paying users that they were using. - What was that product like at the time? Was it typed in you get SQL? - It was cell-contained, right? So we had this notebook that had all these cells in it and we scoped all of our AI features to operate within cells. And this is when way before anyone was thinking about agents, it was just like, you know, you open a SQL cell, you ask for a query, the query appears. This is running on like GPT 3.5 turbo or something and we were like, oh my God. Which all feels very quaint to think about now. And then for the next like year or so, we were very focused on this single-shot type workflow, which I think is kind of in many ways like uniquely cursed in data analytics because it is this like iterative domain in which you get an answer. And it's like, ooh, interesting. I want to follow up. And so I think that agents were not this thing that, I mean, maybe with the exception of you, I feel like people weren't like, this is coming. And so it took us a long time to like go from AI operating within the cell to AI operating across all of the cells and we tried and frankly, I think the models like weren't there the first couple of times we tried. And so we looked elsewhere and built all this other AI stuff and then there was kind of this moment where we were just like the models are here and we need to do this again and it's gonna work. And we put a sidebar in the notebook and we gave it all the same features or the same tools that like a user could access in the notebook and like overnight internally, we released this and like everyone but just freaking out. So like, oh my God. This is it. - Do you remember if there was anything that made you guys realize that it was time to come back to this idea? - I think it was two things. I think one tracking model capabilities and not necessarily where they were at the moment but like the obvious trajectory that they were increasing upon and it was increasingly feeling like we had this like high horsepower beast driving 25 in a school zone. It's like the thing we are harnessing here needs more context, not just from the cell but from the whole project yet. It needs to be able to take two steps, three steps, et cetera. So one side of it was the models and I think the other was that like, it just wasn't working candidly. Data I think is a uniquely difficult environment in which to try and do good single shot text to sequel or text to code or text to answer generation because it is iterative and it's about refining and sort of evaluating an answer and changing and pivoting and rabbit holeing. And so I think like we worked really hard and realized we were stalling out and like this more flexible approach was obviously what's necessary. - And what types of questions were people asking or wanting to ask whether it's in the single shot or in the agent that came after? Was it to build a notebook, was it to get an answer? Like what did you see people do? - Yeah, it's all the above. I mean this is a cool thing about working in data in general I think, but also about working with such a flexible product. Like hacks is that, you know, people are asking questions ranging from how many widgets did we sell last week and they just want to know the number to like build me a crazy predictive model of how our vehicles drive in snowy conditions when the temperature is like this and like we have such a wild variety of customers and they're all building crazy complex stuff in their domain and hacks like the, I don't know how early and user you were, but like long before all the AI stuff, the core promise of hex was like you do this notebook, you construct all of your logic and then everything there becomes a building block to build this app that is like much more flexible than a dashboard. And so it was kind of fun when we started working with the agent because hex has always been this tool that was come and ask your hardest, craziest, most flexible questions and go deep. And I think that because of the model and the harness limitations, the AI stuff was always like so tightly scoped compared to the like whole workflow that users wanted to answer. It's like you come in, you have this huge thing in your mind and it's like, okay, I'm gonna work with AI and I need to ask it like these teeny little bite sized bits. And I think that's the fun thing about the notebook is that like you have customers come in and they're like, okay, like we just launched this new feature, build me a report of how the users that have access to it are liking it. And like that's the remit and the notebook agent will work for like 20 minutes and come up with the answer to not just that, but to you know, many of the sub questions come up along the way. - What does it do and how does it do that? Does it write notebook cells that are then kind of like executed? - The notebook agent operates in the notebook almost exclusively with one exception is that we allow it to run scratch pad SQL queries that do not show up in the notebook. But all of the other work that it's doing is sort of contributing to this artifact that a, the user can follow along with as it's happening and b, can you know, ultimately use to turn it into some kind of a report that they might use or send to someone else or something like that. I'm forgetting the exact word, but Barry or I CEO went to one of those like VC summer camp things. And I think Sachin and Delo was there and said something inspirational about how many buttons Microsoft Word has and all of its toolbars. And it's like the agent needs to be able to use all of those. Like all the power of this very complex tool should be just like a prompt away. And I think that we came into hex or to building the notebook agent specifically with that same thing in mind where it's like this is a complex power tool for like technical users and it has a million little buttons and things to do. It's like can we just not change any of that? Not introduce any new capabilities. Can we just make all of that accessible through the simple natural language entry point? - I feel like the agents that most people use are coding agents and those are also kind of iterative right? Like how do you view the differences and similarities between data agents and coding agents? - I think there are two big things here. What I was referring to before is that a lot of the way that I code, especially now the models are much more capable, you lay out the plan up front and it executes. You write the spec, build it and it builds it and then you get it and you're like amazing. You're kind of either like yes or no. And I think that when you do good data science or good data analytics, there are many decision points along the journey of a project in which you're not just saying good job, bad job, but you're saying like, oh that's interesting, like should we look at it this way? Or like, oh, like I wonder if we break this out. Do we see a different trend? And I think that that is something that again really does not lend itself to just being like question, answer, done. And so I think agents really solve that. And I think that you can see with the notebook agent, the adoption of it and if you've ever tried it, it's a pretty remarkable experience for doing that kind of work. I think the thing that's still frontier difficulty problem is that all of those decision points are very hard to verify and validate the outcome of. And I think like coding increasingly is like a very verifiable task, right? And this is why the models are getting so much better at it because they can train on it with this verifiable reward and you say make a dashboard with these five buttons and it does it and you're like, there are five buttons and they work. And maybe the code is sloped behind the scenes, but like the task has been accomplished correctly. And I think with data, that's much less clear. And it's actually a very big challenge to how do you validate the accuracy of that question. And so how do you train the models to be very good at this and how do you use them in your harness? And so I think that that is still kind of like the interesting part of the puzzle, whereas I think agents versus single shot prompting kind of resolved the earlier part of like interesting AI data puzzle. I want to talk more about that evaluation step later on. But maybe one last high level question. What are the other agents besides notebook agents that you guys are building? And when we were talking earlier, you said that these are all maybe melding together in some way. We'd love to hear more about that. This is all very in flux in our code base right now. We started with the notebook agent, which we just talked about and was very much this kind of like, here's something that already exists in our user's love and get a ton of value out of. And can we just make it easier to work with basically? and automate it with this agent. The immediate next step was to build something. And so that's a product for technical people who are comfortable working in a notebook, a sequel and Python. And of course, once you make it more accessible, non-technical people start doing that as well. And so we have a lot of actually non-technical folks that work in the notebook now to build complex projects. But there's this other demographic in data, which is the true self-serve contingent of folks who want to ask a question again and answer. And so our second agent was what we call threads is really very, very, very similar to the notebook agent. But it's kind of a more abstracted, conversational interface. So just like a chat. Exactly. It looks much more like a chat, your BT conversation, but along the way, you're getting these data artifacts to pop up in our interactive and explorable in the same way. There would be in any other part of hacks, but all of the code has been kind of abstracted and hidden away. And it behaves slightly differently. It's not question, answer. It's like question a ton of stuff, maybe follow up branching rabbit hole. And then finally an answer. But it's designed for the kind of thing where it's like you go in with a question and you want an answer. Whereas a notebook agent is much more about perhaps building some complex unknown report or diving super deep into something. We also introduced some semantic authoring capabilities earlier this year, where you can write a semantic model directly in hex as well as import them from, you know, DBT or what have you. And we also introduced a model there. The interesting thing about that model is not necessarily the visible capabilities it has because it just, you know, writes Yemmel docs, which is pretty simple, but all of the context harvesting that it does in order to be able to accurately contribute to your semantic models. We talked about unification, but like we very early on gave the semantic modeling agent the ability to read a bunch of other artifacts from around the hex workspace that it could, you know, uncover interesting things people are doing and upstream them into the model. And that's now capability that's being rolled out to all of the agents in a first priority capacity. And when you say other artifacts, is that like other notebooks and stuff that people have created that might inform that you don't know? - Yeah, notebooks, other threads, conversations, all kinds of pieces of context inside of X. This is, I think, our big information architectural challenge is laying out this graph of all of the context that's like warehouse source semantic model, some user saying something in a thread, the admin saying something, how that all comes together and is synthesized. So we're actually behind the scenes somewhat, we have a context agent that helps synthesize all of this. We have the chat with your app agents. If you've built a data app on top of your notebook, you have a very sort of threads like experience with that as well. And then yeah, these are all basically trending towards being the same thing. - Same thing from what perspective from UI UX, from the underlying kind of like harness and architecture. - I think that the UI UX is a piece that's more up in there and interesting right now. I think it's like the capabilities that the agent has. If it turns out big surprise, if you're interacting with something that looks pretty similar in the same product in different places, you expected to have the same tools and capabilities and access. And our initial agents were all totally separate things. And so people would be like, "Oh, why can threads do this?" But like, why can't threads write Python, but the notebook agent can. And why can the semantic authoring agent, like, you know, read my other projects, but the notebook agent can't. And so just unifying these all together, which has had interesting code level impact on the way that we organize our tools and prompts and skills into bundles that we're calling capabilities now. - Great transition. Let's dive into that. Like, what do these agents maybe pick the notebook agent? Like, what does this look like under the hood? - There is kind of this high level workflow that runs of the notebook agent with a bunch of sub activities. We've built our own orchestrator to run all of this stuff. We can maybe later talk about how we eventually migrated from that very single shot oriented world towards this very long running, agentic world. - When I think of an agent, the simplest form of it, I think is like an LLM running in a loop calling tools. Is that basically what you guys have or do you have more opinionated workflows around that? - That's basically what we have. The opinionated stuff is mostly around context. - Great, how does context work? - I think we break our concept of context up into dynamic and static context. - And then the different agents you expose have different capability sets basically? - For now, although we're kind of exploring how it works when they have very similar looking capability sets, but yeah, the capabilities bundle together. Tools, static context, prompts, also some weird esoteric finicky stuff, like final turn behavior. How should the agent work when it's wrapping up its project? I think that like to what I said earlier, the actual high level agent loop, I mean, it's a fine piece of engineering, but it does kind of boil down, to just run the LLM in a loop. And the interesting stuff is really about the context harvesting pipeline and then like all of these finicky little things about how does it work on the last turn? How does it know when to wrap up? How long do we let it go for all of that interesting stuff? How does it work on the last turn? I think that this is like one of the things that we've always been trying to tune and tweak the behavior of to get it to wrap up appropriately. And I think actually like, it's kind of interesting with longer running tasks, more capable agents, compaction, becoming a thing, longer context windows, like I run my coding agents forever. Same, like I hope I wish there was no last turn. I wanted to just keep going. And today we still enforce, you know, like a hard ceiling on the number of iterations. And so if it's in the middle of something, when it hits that, we force it to wrap up in a certain way. I'm out of curiosity. You make these technical choices when the models are at a certain level of capability. And when you have five of those, it's easy to keep advancing them along as the ball rolls. But I think when you have 500 of them, I feel like every day we wake up and realize, oh, there's this thing holding the agent back. And like, why? And it's like, oh, well, you know, it used to help the agent. I think a remarkable amount of things that at one point we felt very proud of building. And we're in fact necessary are now hobbling the agent, actually, like just this week. We've been working on figuring out if we can remove this very complex system that we built for changing the way that our agents see the static IDs of artifacts and hex, whether they're cells, project, state of connections, et cetera. You know, we have a million of these things. They all have a static ID, long, unique numbers, and earlier iterations of models, the ones that we built the first agents on, how to kind of ceiling after which they started hallucinating these IDs are making them up or swapping them around. And so if you had a notebook with like more than 50 or 60 cells, you'd start to get crazy behavior. We built this crazy complicated system to basically map between short references and IDs. And this is like a core fundamental part of the agents. And just earlier this week, someone ran an EVA and was like, yeah, we don't need that anymore. They're fine. And we're always fixing bugs and dealing with stuff about this like, ref registry compensation system. And so there's probably like five more examples of that this week. And this is like when you get into the good and the bad of building your own thing. We've built our own very custom orchestration system. It allowed us a year ago to compensate for this like severe model deficiency and build at a very low intimate level, this mapping that allowed us to build and release the notebook agent and scale it and it to become this incredible thing and kind of change the trajectory of the hex product. But now it's like tech debt baggage unnecessary. And it's very hard to get rid of tech debt once it exists. When you do these changes to the harness, are they incremental or has there ever been a point where you're like, hey, we're just gonna completely rewrite this harness. It's gonna be agent V2, agent V3. And I guess like related at those points, did you consider adopting and off the shelf harness or not building yourself? And what was the considerations there and how do you think about that? We looked at a bunch of orchestrator tools and technologies and made this decision primarily on the basis of things moving really fast and us wanting to be able to continue to move fast with them on our own terms, even if it meant taking on a lot of this maintenance overhead on an ongoing basis in the background kind of unbeknownst to our users, we migrated everything from this very single shot, bulk you oriented flow to something based on temporal that kind of has proper long running workflow orchestration. It was a tremendous task. And in the worst part is you have to maintain two things at once and we're always going through two things at once as we introduce the new version of something. But late last year we kind of moved to this new, better world that has since enabled us to build all of these other agents and stuff. It wasn't a true blocker, but it was kind of a self and force blocker to building other agents was we want to actually be operating in the proper framework before we do that. You mentioned earlier that the notebook agent needs to be able to do everything in a notebook that human would be able to do. I imagine that's a lot of tools. And now you're talking about combining multiple agents in the game. How do you deal with so many tools? Oh, it's so many tools. I think it's too many tools. How many tools is it? Not quite 100,000, but something like that tokens worth of tools in the tool kit, which is too much to be clear. It's too many tools. I'm not proud of it. One thing here is to reduce the number of tools and simplify and consolidate. But another thing is to implement some kind of tool search for a tool retrieval, which becomes a big thing with the primary coding agent that I've to reach out to a bunch of MCPs. I think if you have a bunch of MCPs and sold cloud code can theoretically use thousands of tools. So we implemented tool search, which helped to take that pressure off. I don't necessarily purport to be an expert on this. We only have our empirical evidence, but we had a create chart cell tool, an update chart cell tool, delete all of these sort of very normalized tool sets, and it's always a little bit. interesting and unclear, unless you evaluate and test, do you need to repeat the same thing in every one of those tools? Can you say it in the system prompt is saying it in one tool is good as saying it in all and it's like, "kinda." But we basically see we haven't gotten around to it everywhere because it's just kind of work, but I did a test around an event and we can consolidate this all into just chart tool and it works fine. So there's that kind of just meaningless tool explosion, but there's also just a lot of temptation to introduce lots of little tools when you have a product that doesn't operate at the command line coding layer, right? Have you thought about making it operate at the command line coding layer? Because you guys also already run it in a in the notebook environment, right? Like we do, which isn't truly the command line, it was running in like an ipy kernel type, not exactly, but similar environment. But it has a coding environment with functions. It can run arbitrary Python code. And yet we built it a tool to check if a certain package is installed in the environment or not. And part of the reason for that is do you want the agent to run a quick little ephemeral tool or do you want it to write Python code that, you know, becomes a cell and is visible to the user like, oh, should we let it run a femoral Python code? And then you get into all kinds of interesting behavior around like once you let the agent, especially modern agents run secret ephemeral SQL queries and Python code, they really like these days to be pretty sure of themselves before they get back to you. And so like, especially GPT five series models, if you ask GPT five point four, a question, and it's the wrong question, and it turns out to be very complicated, depending on how it's feeling it might run like 50 ephemeral SQL queries just to be really sure before it actually starts doing any real work. And so these things have trade off. So yeah, we could give it, you know, very generic tools to just run code, but introducing very specific tools allows us to introduce these kinds of behavioral guidance around when to use this tool or when to take this action, I suppose, that you don't necessarily have if you're just like, here's a Python tool and use your best judgment. So I didn't realize this, but all the code it writes basically shows up as a cell in the notebook for the notebook, almost all the code it writes. But it does have this ephemeral SQL tool. I'd be curious to hear like why that and like, how do you see it using that? It comes back to what I said before about like just how important it is to be able to do a lot of work when you are answering a data question. If you ask, did we have more users of this product feature this week than last week? And if you have a well modeled beautiful semantic model, like maybe that's just boom, one shot, look at his split, but a lot of people don't. And I think a lot of people that think they do also don't. And you know, you might need to run some quick checks and be like, okay, does this table actually have the information I want? I have to join across these two tables. What format is the data in this column in? And there are all these things that you could discover iteratively by error driven debugging, like you could throw out a SQL query and get an error. It doesn't look right and refine, refine, refine. But we found that it's a more efficient workflow and better for the user. If instead of doing that, it kind of does these very small little atomic ephemeral investigative queries and then once it's gathered enough information to write the primary query correctly first go, then it will write that pop it in the notebook. And it's a much better experience for the user than having to refine and iterate on that query multiple times. But again, as with most things you give agents, like it does have kind of unintended consequences where we're constantly battling against the case where a user asks a really simple question. And it's just like tells you the answer, because it ran a little secret SQL query and these are like, where's the chart? Where's the proof? The agent's like, take my word for it. And so figuring out these little behavioral edge cases is always the thing when you introduce a new tool that isn't just write code. But also I use codex a lot. I like to use codex to code. And codex has a particular pension or if GPT 5.4 I guess, for running like crazy bash. And like I've noticed that writing like Pearl commands now it's like I kind of understand vaguely what's going on when it's writing bash. But not really. But it's right running Pearl commands like I have no idea what it's up to. And I think that again is one of these things where it's like, okay, maybe to do that for the like technical notebook agent audience. But we have these agents that are operating in different contexts for different users. And I think a lot of it does boil down to the UX of knowing exactly what the agent is doing because they take a while. They think and they ponder and they run a bunch of tools. And I think that engineers the hands are like coming off the keyboard. And it's like, yeah, run for 50 minutes. And I don't care what you do as long as you do it right. And I think that maybe the like business stakeholders will get there soon too. But I think more than you think they actually are interested in following what's going on. What do you show these business stakeholders? Do you show them every command that's running? Do you collapse them and show the little cloud code like, you know, noodle lean or whatever they have? I'm like, perhaps regarded internally as like a sort of class clown, just a type guy. And I hate the little new thing. It's like my biggest Grinch like Scrooge McDuck take. We like add some version of it. And I was like, no, no, this is like, I don't know. For some, it's there's no explanation. I'm just a Grinch about it. We say thinking. Very professional. While the agent is working, we show you what it's up to expanded. And then once it's done, we collapse it, which I think is for now kind of a nice paradigm. But I don't know if you've played with GPT five, three codex spark or whatever this, the open a is like newest, the really fast one, the cerebral test, basically, it's so fast. There's no beyond no need to show the user what's going on. There's no way to realistically show the user what's going on. It's like, and so I am operating under the assumption that this is going to be one of those things that we felt really stoked about today. But in a few months, they're going to be desperately figuring out how we rip out the user following along behavior because we want to unhobble our agents to do more and work more verbosely or do a million tool calls in parallel without overwhelming the user. And I can already see that this is going to be a like UX capabilities friction point soon. What about the end results that the agents create? Is it an answer? Is it a notebook cell or multiple notebook cells? Is it a chart? I'm assuming it's some combination of all the above. But what's the distribution that you see people using it for? Without being too much of a cool aid man, I think that this is one of the cool things about hacks is it's such a flexible product. And it's kind of like always been such a flexible product. I think that I sort of for lucker before hacks, which is a tool that lets you build dashboards and run explorers that are basically chart sale at the same. I love lucker actually. But one of the things that was most thrilling to me about hacks, which is the diversity of report and answer you could construct. And I think that we have carried that through to the agent world, like the threads agent, you know, notebook agent, you can build truly anything you dream of interactive apps that you click and they run some workflow and update crazy forms. You can build some really wild stuff. But even in threads, it really depends. And I think that flexibility of if you ask a question, it's very simple. I'm always trying to get the agent to give you the least cognitive overhead answer. And so if you're like a quick question, how many users does this company have? So you just want to know you don't want to report. Do you think it's important for the agent to like show its work? This is why the notebook is so great. And I think why we got so far so fast with the notebook agent because famously notebooks are literate programming. This is sort of their entire point is to be able to self document the work that's being done in a way that is remarkably well suited to allowing agents to write code. And I think also there's like a sense that when you are writing code with AI for a technical user that there is kind of this expectation that they can fall along and can verify and validate. But when you're doing this for a business stakeholder who doesn't know SQL, it doesn't know Python. It gets much more much more philosophically different. Right. This is codex running a pearl command that I cannot follow along with. I run codex with outputs off. So I don't even know what it's up to. I've moved past that. I think that the concept of showing your work is not the right concept for data work in terms of citation and verifiability. I think it needs to be some stronger form of verification or confidence or accuracy. This is something that we're working on right now. What does that mean? What are the ideas you have there? We don't have a massive amount of this figured out yet. There's one really obvious amazing way to do this in the data world, which is to use a semantic model. If you define an amazing semantic model and you let users use it within reason, you can probably be relatively sure that they're getting correct answers. People always figure out how to shoot themselves in the foot. I would always shoot myself in the foot with lookers. Perfect semantic models. But that is one way to know. Then you don't necessarily need this separate clever verification loop because you're basically always operating on trusted governed context. That's one half of it. We do this as semantic models. We do this with admin endorsement or verification of various assets, etc. But the models love to rate SQL and code. Sometimes you need to go beyond the semantic model because that's the whole point of the task. I think then you get into this interesting world where verification and verifiability becomes very difficult for data things. Often it's very vague what truth even is, depending on how two teams define a metric or a data pipeline update changing the number out from underneath you. I think it's just a really tough domain. I think the way that we're going to tackle it, the way that we're already tackling it is with a ton of feedback loops from the user. back up to the data team observability actually baked into the platform. We have this new context studio that basically gives admins on the data team a bird's-eye view of what kinds of questions people are asking, what answers they're getting. We flag out using a separate LLM as a judge, cases where we think something might have gone wrong or where the agent might have been confused or where there's a warning that conflicts with your semantic model. Today, all of this stuff runs as a sort of post-action and gives admins the opportunity to improve the context that allows the agent to do better. So it's kind of a human-in-the-loop system. For the model does something, the user perhaps gets a wrong answer. It's flagged up. The data team becomes aware of it. They can improve the guide or the semantic model. So it doesn't happen again. Maybe let the user know what happened, et cetera. Once you've built the human-in-the-loop system, it's like, okay, how do we begin to automate this and scale this and make it work better faster with AI and agents in the loop? Would that look like an agent updating the context itself? Yeah, we've been working on this context agent internally and prototyping what it might look like for the agent to do this. This is what you guys would consider memories for hex agents? A little bit. I think that the interesting thing with memory and how it plays into this is that memory, as users, think about it. I think it's a very user-level concept. Or chat, GPT memory remembers your conversations and that you told it you have a dog named Rover. So when you ask, "My dog is sick." It's like, "Oh, no, sorry to hear about Rover." I think that's less impactful and almost scary. Actually, I know, not almost very scary, to data teams and admins who are worried that the exact opposite of what we just talked about might happen within a user's little memory, where the user asks a question and gets a wrong answer or something, or the user tells the agent, "No, that's wrong." This metric is actually defined this way and maybe that user is wrong or it's outdated and that gets stored in memory and then you have these conflicting levels of context. Because the different levels would be user-level, team-level or how many? Yeah. Potentially even more? Potentially and the data team sets down these guides and semantic models, reconciling. I think one thing that agents and models do very bad with today is contradictory, dissonant situations. You actually ran an interesting evaluation on this and it wasn't actually necessarily looking at accuracy with regarding how can it resolve conflicting advice situation, but just the amount of time it spends thinking. Your agent, you know, clawed on it for six, what we were running this on and it actually runs fairly efficiently and quickly up to a point and then if you inject in a piece of conflicting context, it goes against some other information it has. It will spend 30 minutes pondering and going, "Wait, wait, but let me see. Hmm, actually, and enter this crazy sort of collapse mode." I think this happens a lot on a very smaller level and so I think, without having the secret sauce or the answer, I think this is something that we're just now starting to reckon with of like, admins need to be able to provide very strong governance and guidance. One way they can do this is a semantic model and then everything is just on rails. But if you do that, you miss out on this glorious world of more flexible work. There's kind of this like, "Oh, like thing on the horizon." It's like, "You know, what if you can just tell the LLM stuff and it's careful about it?" That's basically what people do with guides and rules files, etc. skills. So do your agents have a concept of skills in that way? They do. We call them guides, but they're modeled exactly like skills with sort of progressive disclosure. The agent, when it's running, can see all of the guides that are available from that workspace and then it can retrieve them and read them as it's operating. And is that how this memory gets passed into the agent or to some of it also get inserted into the system prompt and what determines what goes where? This has all been testing at the moment. For now, we're keeping memory as a separate thing from these other sources of context that are much more data team admin driven. I think that the core feedback loop of Hex improving today is driven by the data team. It's by users doing work, their work, or sort of the exhaust of their work being surfaced to the data team in an interface that allows them to notice mistakes, improve the guides, the models, the warehouse context, etc. And then that feedback loop kind of churns with the human. Memories are this confounding thing that can inject all kinds of clutter into that loop at the user level today. And so we're being very thoughtful about how we roll it out. I think we have more testing to do. You talked about observability and evals in the form of like, "Oh, I'm as a judge a few times already." How do you think about the observability that you guys have as developers of the agent versus what you expose to like admins or people? And I think it was agent control or a context studio. The context studio. Yeah, is that the same that you guys use internally? Is it different? What are the similarities? What are the differences? My hot take internally that people always argue about is I think they should be the same. So I'm guessing they're not the same. That's hot take. They're not the same. It's an internal observability and experimentation system. I sort of dream of a beautiful utopic future in which we use very much the same tools as our users do to understand what the agents are doing in the hex product and how we might make them better. I mean, this for observability and evaluation. We're just now starting to think about how we expose evaluation tools to users. We have a kind of very rudimentary setup for this now. If you edit your context, you can kind of run some tests. I'd like for these things to be convergent. Today, the observability tools that we built, I kind of built the first version of them in the beautiful before times before we launched this to any users. And it was only internal usage and we had full penopticon privileges and it was open season on all the data. And we still have that to some extent for our internal usage, which is really helpful to be able to see even just your own local developers to be able to really deeply introspect at all. I think that, A, obviously we can't do that for real user stuff. B, I don't know if admins or data teams want their job to become pouring through everyone's conversations and being sort of synthesizes of all of this. That to me feels like an agent's job. And so when I think about our internal observability tools and where they're going and also where the context studio feedback loop oriented tools that we exposed to our users are going, I think they ought to probably become more agent tech or at least higher level. The things that we can see about usage data from production are like the clusters of issues that are occurring, the kinds of failures that are occurring in the agent and what those clusters are and how they're shifting over time. So you guys are able to see that because you run some along as a judges, which clusters are tag things and then maybe you cluster those clusters or something and then that's the data that basically you guys have access to, which treats all the underlying traces. You don't have that broad data. You just have kind of like the clusters or the insights from the all-elums. That's right. There's a blog post by Anthropic. It's about this stuff. We actually built this into lengthsmith. Yeah, I think it's a very, yeah, it's a great idea. And privacy preserving something, something, something. Yes, exactly. I think that this again is another one of those situations where we built our own observability and evaluation stack here because of this. I don't know if I'll comment on whether it's truly correct or not, this kind of like perceived need to be able to move very, very, very rapidly and this uncertainty about what the agent or the product will look like or the model will look like three months in the future and a fear of limitation if we didn't build our own thing. And like the tax is tremendous. I have spent the bulk of last week actually doing very little besides updating and refactoring and working on our evaluation system to make it more user friendly. Our eVal system internally, we call this system the shoebox, which I tried. I tried not to come up with that name because I didn't want it to last. I wanted to be like the shoebox would just stuff all our shit, all our receipts into and put it under our bed. And then it caught on and is still around. The shoebox, which was some evaluation capabilities was always kind of a like high priesthood tool available to just the AI engineers that were like actively iterating on the agents and prompts. It was very difficult to use and how terribly you accent was kind of the kind of many foot guns such that you needed to know how to use it in order to actually make good use of it. But now the line between AI engineer and any other kind of engineer has very much blurred internally. And so we are overhauling the eVal system to make it really easy for anyone at the company to run eVal's on changes that they've made. The two questions you want to ask are, did my change have the desired effects and did my change have any undesired effects and being able to answer those two questions is the goal for any other company. What exactly does that process look like? Is there one big data set of a thousand examples and they all have a ground truth and you run it against that data set and then compare it to the ground truth with an element as a judge or is it different than that and does it branch off from that in some way? They all did. Okay. How many examples are in your eVal data set? Like this is something we hear a lot of people ask, how big do I have to get my eVal sets to be before and then maybe curious to hear if this is evolved over time as well. I have a lot of opinions here. I think as a general rule, most of them are in the same way. Eval sets are bad unless they are being actively worked on. I think almost are at least in the data space. Pretty much every time I've cracked open the hood on some data benchmark or evaluation, I've been very disappointed by what I saw. I've seen I don't want to slander anyone but just sir, but the rule applies to everyone that I've seen. Bad ground truth incorrect ground truths, problematic grading. Some folks try and do deterministic grading, which is a valiant quest, but your script has bugs or doesn't accept a percentage as a decimal point and like all of this stuff. Some questions are just bad or not representative. There's a very popular benchmark set out there that we actually do use a slightly altered flavor of and it's quite difficult unless you know the like one secret to the benchmark and many many many of the questions in this data-related benchmark revolve around whether or not the agent correctly treats an empty array as the same thing as a null or not. There's like this esoteric tiny little rule in a manual somewhere and that's make it or break it. And this is not like this isn't a data benchmark. There's like a needle in a haystack like context attending benchmark and I think that a lot of benchmarks out there can flate that it's talking specifically about analytical reasoning and data capability. I think that a lot of benchmarks can flate SQL syntax and retrieval and like this kind of needle in a haystack stuff with actual desirable analytical behavior and the kind of scientific reasoning that you would want to be able to do this that process I described at the very beginning where if you get an intermediate answer and I have a decision point of like do you accept it does this tell you what you want to do next do you reject it that I think is behavior is very hard to evaluate and that most of the eval sets available publicly do not evaluate at all. What does a good eval date of set look like? Well, I have some opinions. I think one is that a good eval set should be actually small enough that you as the interested party can sort of hold it in your mind. This may be very controversial. I don't know. I like to be able to know why all of my eval sets, like we have these aspirational eval sets that are agents to vary poorly on that all agents do vary poorly on like opus 4.6 max gets like 20% on super hard. I think that that eval set is much more useful if I know why g7 to eval fails or it's like there's these four reasons and so I've really handcrafted artisanally all of these failure modes traps. I've called them basically that I want the model to fall into and there's a lot of benchmarks out there like this one I was talking about with the empty array. It's like 390 470 ton of evals. Most of them are just variants on this one gotcha and I think that it's more interesting to have a couple of gotchas that you can maintain in your head and then just run a ton of repetitions on them rather than fan it out to a bunch of different cases and make things more complicated. So that's one opinion. Are evals sets are like 30 to 50 for these very very difficult cases. But again, we run multiple repetitions on them. I also think that a lot of data eval sets now are no longer representative of the kind of work that users are actually doing. They're much more of the thing I was talking about before actually this like single shot pub trivia type question where it's like how many users did we have on April 16th 2019. Can you syntactically rearrange this English into SQL. It's like Evalina coding model on tab complete whenever everyone's trying to ask you to write full on things. Yeah, exactly. And so a lot of our most interesting evals are notebook agent evaluations that begin in the middle of a very complicated notebook that has already been aggressively built out and the user says something like that's weird. That's not the number I expected to get. And the evals have been carefully crafted such that there's like a chain of three bugs in the data and the SQL queries that the agent needs to kind of unravel and explore and just looking at the state that the like notebook is currently in tells you about the first bug. But it actually obscures the second and third bug. And so it needs to be this actually like thorough process in order to actually resolve all of the bugs and get the answer. Do you take those from real trajectories or things that you see or are those completely synthetic hypothetical may ones? We have the luxury of having a ton of internal hex usage for like real stuff and I model a lot of it off of that because data's hard and we make mistakes internally also. My favorite eval that all current models that I've tested it on have failed. The newer ones take longer to fail on it is an internal dashboard real internal dashboard about our sales like AE quota attainment. And I took the dashboard and I intentionally introduced a fan out bug that makes it look like all the AE's are dominating like everyone is at at least 900% of quota like killing it best quarter ever. And then you ask the agent how are the top performers doing this quarter? And every agent is like, oh my god, like it's the best quarter ever like your business is popping when you compare this to last quarter. It's like step change. You know like Josie has 1200% of her quota like they're all stoked and I actually tested you can ratchet this up into the thousands of percent of quota before the models start to be like there may be a data accuracy pipeline error or something and almost not well literally none of them catch the bug itself. But if you then say that doesn't seem right, takes them 10 seconds to catch the bug. This is what I think is most interesting to evaluate for. I'm being very sort of grandiose. We also have just a totally normal set of like normal person evals that help us know if our product is like good day to day and we try and maintain a reasonable pass rate on them and it kind of helps us prevent regressions and lets us evaluate new features etc. It's like normal data questions that have a right answer and maybe there's a messy warehouse or whatever. But then I also think it's interesting to maintain this like very aspirational set that measures the things that the models are all very bad at right now. And there's a number of cases similar to that where it's just having the human in the loop for some of this stuff still makes a very big difference with regards to the agency ability to like catch mistakes or have that kind of like interesting like their ears don't perk up the same way a human analyst's ears perk up. But we have some really interesting evals for this. Speaking of the models as new models come out and to get better but also introduce kind of like new capabilities into the APIs. How do you guys think about upgrading these models but also upgrading the harness around the things in the API? Data analytics and data science is a task that requires general intelligence rather than like domain intelligence and so I think just smarter models is great for us and for our customers. So we always try and be on the smartest newest models. Which ones do you think those are right now open AI and throw up this 4.6 and GPT 5.4 are extremely capable in our domain. They have all these knobs and widgets to tune and sliders and so it's harder than I think people might assume to be like this one's the best. That one's the best like well GPT 5.4 like does a little better but like takes twice as long and it's like oh you know you can ratchet down the effort and then it's like takes actually half as long it's like oh but then it doesn't do quite as well and I think effort is this kind of new thing since opus 4.5 I think and GPT 5.3 both of those labs have built in this concept of effort. I think opening I calls it juice internally which is funny and this is something that I do not yet fully understand. We ran an internal test we had a like effort picker and the feedback we got was is it working? It does not seem to actually be having an effect because sometimes on low effort the model gets confused and goes into this spiral and sometimes on high it you know just answers a question like that. So to answer your original question we always try and be on the latest smartest model now we always try and run it at a high enough effort that we see evaluates well without making our users wait 10 minutes to get the answer to a simple question. What's increasingly interesting and challenging I think is that like maybe this is what you were referring to all these other little add-ons keep coming with them like tool search API or like server-side compaction and these are also all things that are very valuable to make use of and also kind of pain in the ass to maintain especially when we built all of our own stuff around it. So what do you guys do do you some we do well we use them where they help I think that part of the benefit of having our own thing and having evals is that we can actually say this helps does the 1 million context token window help or hurts like well it hurts at the margins because you can't fill up that window without incurring pretty severe intelligence or intelligence almost the wrong word is like the model starts doing weird stuff at the ends of that window so it's okay we want to like compact very early but you know maintaining access to the 1 million token model via some beta header is actually valuable so we can compact it you know 300,000 instead of 200,000 because we found that's the best so we do support all of these things whenever possible I do think though that like talk into the labs these days I think that they have a vested interest in you using their proprietary locked in stuff and the kind of technique the psychological warfare technique that I feel like they're using on us to try and get us to use this stuff is the words in distribution. And I don't yet totally know how I feel about this. We're there like, you know, if you use the Cloud Code agent SDK, like you'll be in distribution for what we've trained the model on or open it as like, you know, if you use it like stateful, server side execution or whatever, it'll be very in distribution for the new models. And I'm very open to that being true. I wonder how quickly that goes out the window once you like throw in your custom tool to build a chart or a SQL query. And it's like, are you suddenly out of distribution again? Or do you wind up in a weird codexd case where it says screw your tools? I'm going to use like, Pearl. So I don't know totally how I feel about that, but it does feel like that's where it's trending, right? It's like, there's all these little bits and bobs that are layered into the new models and some of them are very helpful and we try and adopt them. I think it's very difficult for us to think about moving from this like all a cart adoption to actually centralizing into the full on SDK or into like some, you know, stateful, programmatic tool calling like server side thing because you would give up a lot of the control that you like. I think because we would give up a lot of control and I think just at this point in time, the benefit quantitatively is a little unclear to me. I don't yet have a strong intuition for just how much it matters to be in distribution because these models are very good in context learning and I don't mean the model uses our harness pretty dang well. And it's unclear to me what being more in distribution might get us versus the trade offs of being slightly more locked in. Would you describe yourself as an agent engineer as an AI engineer? What is it that you do on a day to day and how did you? How did you get there? I think my official job title is AI engineer. I feel like I never really had a job title at hex anyway. I was an early-ish employee joined to do marketing to do community, dev role, technical developer marketing, all that jazz. And when you joined to do that at a very small company, we're the only marketer go to marketee person. You end up doing a whole lot of stuff. And I did that for like four years. And I think I always wanted to be an engineer. It's just the job didn't didn't exist. I was always technical. I was always a practitioner first, a data practitioner, not necessarily a full on engineer, but my kind of my dev role philosophy was always you shouldn't be a marketer. You should build stuff and then talk about it. And you shouldn't talk about anything. You can't build because that's what I call lying. I prefer a more honest form of marketing. And so it's always technical. Always very interested in AI stuff. I helped actually build our first version of this old single shot text to SQL workflows just out of interest. And the honest truth is like, I don't think I could have been an AI engineer until GPT 01. I don't know. I don't know when the models got capable enough to allow me to be a productive member of the engineering team and actually like contribute valuable, meaningful, clean code. But sometimes in the last year and a half, the models hit the point where my experience with hacks, my opinions, my like user empathy gathered over four years of doing dev role work for hacks paired with Claude or whatever. It's basically allowed me to become very rapidly an engineer, which is sweet. And I think there was like, I was just reflecting on this. There was like a very brief window where I was writing all of my code by hand. I was like, I'm an engineer. Like this is so cool. I can't believe I'm an engineer. And then I started like copying and pasting code from JGP team more and more. And now it's like, I'm kind of back in where I was before. I feel like I've forgotten how to actually code. And I'm right back to square one two years ago. Because I mean, I'm really good at reading code, but I don't write that much coding. More. It's interesting. What do you think the right profile for an AI engineer at hex is? I think there's a lot of different work to be done. We hire people from very diverse domains. We have mathematicians, all kinds of folks, marketers. I think I feel now, similarly about engineering as I do about dev realm marketing work, which is the best person is someone who really cares about the problem that you're trying to solve. That was always what I told people to hire for in dev realm. And it's like, you can teach them how to do marketing stuff. Like they're bad at writing. Uh-huh. You can probably teach them how to be a little better, but you can't teach them how to like really care about your domain enough to like stay up late and build that cool feature that's going to be the one that like goes viral on hacker news or whatever. And I think that was just something that didn't really exist for engineering because it's like, well, you have to be a good engineer. If you want to build incredible things, you can't just have opinions and really care. You got to also be able to do it. And I think it's still very important to have senior engineers. And I think I read and review and manually edit all of my code. But mostly I think my job is just like to care and have an opinion. And so I think if you can somehow find those people, I think that's really interesting. One of our most productive members of our engineering team is it was a user. And he gave us enough feedback finally that one day we grabbed him and threw a hood over his head and put him in a chair and we're like, you work for us now. And he's killing it. But he was a data scientist before not an AI engineer or any kind of engineer. And he's, you know, phenomenally productive primarily as a function of his opinions and his instincts and intuition for the things that are going to be useful for our users. And it's kind of funny like I almost model in my head. Maybe this is anthropomorphizing or worse. But the notebook agent and the threads agent is a data scientist data. And as we tell it, it's like, you are the hex threads agent like you're a data analyst. And so having that user empathy, I think actually makes people better at building the agent itself. Because you know how it should work. And you can tell when it's doing weird stuff that might be an indication, not that it's like in distress or whatever, but that something is holding it back or something is keeping it from this golden path that it should be allowed to run on. They can put themselves in the shoes of the agent and think well enough. Oh, if I were to solve this, this is how I would do it. They would care enough about that. I think so. But I think the domain expertise is irreplaceable. I think of it very specific place. I think it's irreplaceable is visualization, which is actually a place where I don't have that many opinions. I have lots of opinions about like the, you know, basics of how to not commit chart crimes. But not at an incredibly deep level. But we have two people on the team, Madeline and Nicholas, who I don't know what level is beyond expert. But they're like the like, Shukunin level of visualization experts. And increasingly, I think their job, I hope they don't mind me saying this, is just to have incredibly strong opinions about how this stuff should work. And half of the time they're directly encoding those into the agents. Half of the time they're telling other people that they did something wrong or that they should have built it this way. And helping other people build in a way that is more opinionated when you're building products in the age of AI. It's always a funny dance of your opinions. The user's opinions use a sometimes expect now to be able to just type something in and the product looks totally different. But I do think that with data and visualization and stuff, it's important to have these strong domain expertise driven opinions. You can codify into the products to make it delightful. And I think specific with regards like the business stuff, I think we get a lot of feedback. It's like, Dex is mind blowing. Like the hex threads agent is amazing. And sometimes when you like drill on it, it's almost hard for people to say exactly why they're like, I got the answer I wanted. It was way better than, you know, insert other tool here. And then you like really drill on why and why and why and a lot of the time, at least when I talk to you, is they're like, yeah, just was like better. And I think that that is like a very interesting june say quack because like accuracy, whatever. But if the answer is like a long report with five charts and these bullet points and this offer to look at this next, that's the kind of thing that I mean that is impossible to evaluate. I lose sleep at night trying to figure out how to evaluate that vagueness of what it means to be great at data outside of just getting the answer right in a very, very complex situation. Do you think that's the main value you provide is helping the agents be great at that and get those great answers, even when it is tough to evaluate quantitatively? I think so. I think that this is something we always need to be reevaluating is like how much of it is the model and how much of it is the harness. And I think Barry or CEO has gotten really into saying what sand and what is stone about the product, you know, if the models get twice as good as they are today, what turns out to have been sand that can be washed away and what is stone remains and is very meaningful. And I think as the agents get naturally more capable at data science and analytics and stuff, I do think that for a long time, at least I won't say always, but I think that for a long time, there will be value and the opinions and the domain expertise being encoded alongside the tools that will make hex better than a snowflake connector with a marketing document that you run in your command line. I will say though, I think that's like this much of the pie. And I think most of the other pie is the context exhaust that we talked about earlier in that like feedback loop of if you work in hex and you're producing artifacts and getting answers and building projects and those are all becoming sources of information potentially verified for the agent to, you know, validate or guide its future work. That is most of the value the product provides and I mean it's impossible to provide that value on day zero. So think like the day zero version of hex is better than your rando connector. I think mostly in just the form of being opinionated and user friendly and like I would actually maybe it's a hot take but I do think that on day zero clawed with the snowflake connector is probably like I don't know, probably just as good, but the value is that right this like overtime like it's not just a tool it's like a platform that everyone works in and the flywheel cause it to improve overtime. And so by day 90 you're operating a totally different ballgame. What are you most excited for next? There's one thing relating to eVal's. It's kind of interesting. The way that I constructed the warehouse that we use to power our internal eVal sets, it's done very aspirationally. I've left the door open for much more to be done in terms of eVal's here. So today, we have a snowflake warehouse with a realistic representative amount of business data in it about this made up company, Shoreline Commerce that does, you know, Chachky sales. And I very carefully hand injected all of these terrible data quality problems into the warehouse that cause the agent to have a difficult time and have to push through a bunch of noles and messed up columns and joins that don't quite work, all sorts of misleading, confusing things. That's like V0 of our benchmark is kind of all of these questions that are very realistic of what users ask of a very realistically messy like half modeled but half messy filled with quality issues realistic warehouse. In addition to this snapshot, the most interesting way to evaluate data agents and data work has to be long horizon. And I think this gets into some of the value that I think ex provides that we were talking about about like day zero performance versus like day 90 performance once the flywheel has churned. It's kind of unfair to make Claude answer a bunch of hard questions right on the first try and it doesn't ever have an opportunity to try again. And that's what most eVel sets are. It's like, who's the president of Malaysia? And you're like, good. And it's like wrong. And then like you never revisit that and no value came out of that. I think that hacks as a product is a kind of product that should get better over time every day. It should get better and better and better at your tasks. And so if we're not evaluating our product in a way that allows for the agent to you know actually demonstrate its ability to compound. We're only evaluating this like point in time part of the system. We're not evaluating the full flywheel. And so the full vision for this evaluation set is that day zero snapshot of the database basically is used to run a battery of benchmark questions which are that you've already have. Many of these are borderline impossible to answer with the information available to the agent on day zero. But I've built a 90 day simulation during which the clock ticks. And every time the day turns over a DBT models run and alter the warehouse to keep it sort of time shifted up to date. New rows are coming in. Things are breaking. New products are launching. Fraud is happening and tickets are coming in from stakeholders to the agent in the form of email tickets. It needs to answer and they're asking data questions and they're telling it information. And along the way it's kind of discovering things about the warehouse. And once it's replied to a ticket, the evaluation doesn't end or the conversation doesn't end. It gets told you have replied to the ticket. You now have access to an end turn tool. But if you'd like to do a little bit of proactive knowledge work before ending your turn, you're welcome to follow up on loose threads from this conversation document some of your findings clean up the wiki etc. And we let the agent sort of practically do its thing and then end it's turn whenever it wants and this happens every day for 90 days. And the conceit is that if the agent is demonstrating the skills and behavior that we want it to by day 90 all of the questions and the tickets are carefully crafted such that it should get 100% of the questions right. And so today it's very expensive to run. I need credits please. Again, I think that like for these very difficult questions, it's not a productive number to know day zero performance because it's not realist like it's not what the real world actually looks like when they're evaluating this stuff. This is sort of like mad science laboratory state right now. But in my spare time, I'm tinkering with making this something much more legitimate. And we actually can evaluate our and this is model based right now, this is running in sort of its own harness. It has very simple tools isn't evaluating the hex agent at all just evaluating the model's behavior part of it is that I want to observe I want to see. Thinking about like in distribution, how does the model like to organize its wiki what kinds of information nuggets does it like to store how does it retrieve them what is it find what is it miss like it's almost a fact finding like a research project so that we can make our harness better. Do you run the hex agent on the benchmark as well the before and after I figured out how to yet. It's very this is I guess a mad science laboratory at the moment, but I very much plan to make well that's the honest evaluation right I think any kind of one try you Val is uninteresting a lot of people report like pass at care whatever that just means in parallel like as far as I'm aware the only people that are doing really interesting long horizon evaluation work are and the end on labs of the vending bench, which is kind of what this is inspired by and that has sort of like this long horizon reward does the hex agent right now have a tool to like make notes in real life. Not at the moment so right now everything happens via this synthesis step after basically I think maybe it should I think it's an interesting thing to consider like I was arguing with a coworker about this actually recently about is there value in the context agent or the synthesis agent like being able to see 20 threads before it saves a piece of information versus the point in time saving of information it's like there are strong arguments both ways are pros and cons to both the thing that he had. And so the thing that he had implemented that I was kind of pushing on was the like synthesis step where it sees everything and so it can actually like validate something or just do it etcetera I think that scans but I also I wonder if there's some suggestion mechanism that needs to happen in real time during the threat as well like I sort of alluded to like these make things very difficult to evaluate if you are allowing the agent to dynamically change the system during the evaluation and then you have to run these more simulation style. Eval's judging becomes really difficult even just orchestrating this simulation becomes a total pain in the butt I don't yet have all the answers here but I do feel very strongly that the e valves we do today that are just one try. I think it's interesting very helpful they help us understand agent performance but I don't actually think they're honest evaluation of a system like hax it is platform that is designed to compound over time so I there's something missing there that I'm interested in and digging at what is the name of the benchmark vending bench. Vending bench is what and Thropic has built I've called mine metric city is vending bench open source vending benches closed source I think it's actually done by a small company called and on lab such as this weird Swedish company but if you read about like Claudius the vending machine yeah yeah this is sort of related to the vending bench and on runs the the vending machine to actually well it's not a source I want to run on it right I love to do but I'm just surprised that no one else has spent although I haven't released mine yet I hope to open source this metric city benchmark. Let's try that yeah I mean I think it's really tied into memory we think a lot about memory and our agent harnesses and there's yeah it's hard to evaluate that one so if they're I mean yeah we might make one as well but like I think something like this totally exist. Thanks for listening to my conversation with Izzy Miller this is the first episode of max agency a podcast where I talk to the builders behind the agents and get into the details the architecture the improvement loop and what's working or not if you liked this episode leave a review and subscribe send feedback questions to max agency at Langshane.dev we want to hear from you.