We've been struggling with this for a bit where we used to have our own hard-coded pipeline. Now we completely re-thought it. Today, I'm talking to Florian Jungerman, co-founder and CTO of Listen. They're known for their agents that can analyze hundreds of interviews, surveys and focus group feedback to pull out the signal from the noise. Right now in the main agent, it's not directly a fire structure. We think of it more as a table. So the table is every row is a response, and every column is kind of like a question or a feature that we extract. And then the agent can basically create new columns. Florian describes how their agent works to put structure to interview responses and gather implicit signal from media rich user conversations. We have been relying more on contextual prompt engineering. There's one feature we have which is emotional understanding based on the video and the audio and not just the text. We also go deep into their approach on quality control for the end report the agent rights and their eVal system to ensure accuracy. Basically, we have this sub-agent-richess-less reviewer agent that just knows what a good report looks like. So that's what we run it using the asynchronous runner. And then in the life runner, we use it as an evaluation system. Listen has figured out how to run agents at scale, solving some tricky problems on breaking up tasks so that they can be massively paralyzed. We have this hard coded workflow if you call this to a once. We spawn those 500 agents and then we aggregate it in a very specific way and then it returns it. Welcome to Max Agency, the podcast that goes deep into how the best agents are being built. By builders like you. And listen, you guys actually have a bunch of agents that make up your platform. Would you mind talking about the different agents you have and what they do and how the user interacts and sees them? So I think our platform is pretty broad. There's a lot of use cases and everything is now agent first. The first step is you actually create your product, your create your study. There we have this agent that is, you know, this interactive creation agent that, you know, we call composer. And it really works with the artifacts on the site. It can modify this but it's really like human and AI interacting in the same documents. There's some very interesting. What's the artifact on the side? To the artifact is your discussion guide, which will kind of be those are the questions we will ask in the interview. And this then actually goes as an input to the next agent, but it actually does the interview. So this is the the eye that has a conversation with our interview and goes back and forth. And you'll do this with like thousands of candidates. So yeah, so not necessarily candidates. It's more users or customers we'll talk to, talk to hundreds of thousands in parallel. And yeah, this agent, it's a little bit less interesting from like a, you know, agents building perspective because it doesn't have as many tools. It's more like a regular conversation, but it's also like multi-modal. It has like image input and you can do screen sharing and those kind of things. Is it voice as well or I think it's a voice based as well. Yeah. So that's the second agent and then the third big step is on the analysis side. And that's I think where we spend most of our time so far is building this what we call research agent. And that's really like imagine you have you've done, you know, 500 interviews now. And those exist like you have the transcripts of the videos, but now you want to explore the data you have questions about it can ask. And as research agent is really powerful, it can create everything from, you know, charts. It can, you know, obviously summarize things. It can even like cut video clips for you. It can even now create PowerPoint slide decks in your own company template. And that's I think we spend most of our time so far. Okay. So let's let's focus on that agent. So how do people interact with us? Do they chat with it? Is there some big background job that runs off after all these and how many interviews are there? You said 500. Is that a typical amount? Do you see hundreds of thousands? Yeah. So interaction. It's both. It is the, you know, we do run like one analysis run upfront and that can take like 30 minutes like we use the same agent architecture for that as for the live interaction chat. There's some different parameters. We want to optimize for latency versus, you know, just quality. What are those different parameters? Yeah. I mean, sometimes use different models. So for some like live calls, we use like faster smaller models or we tune thinking parameters. So, you know, in the live run, we have like minimal thinking versus in the long run, we have like medium or even more thinking. And then there's some, there's some things of like if you have more than 500, you have like 1000 of interviews. You can't always look at all of the interviews live because, you know, Rachel, Mets and and so on. Even if you spawn a lot of sub agent, there's some limits to that versus in the asynchronous workflow, you can actually look at most of them. So then we're doing some things like maybe we just sub sample some and we show you some results earlier. So there's some slight differences, but the same architecture, the same kind of agent tools overall for both of those tasks. And for the asynchronous run, like how do you decide what to kick it off with? Will the user set that ahead of time or is that a standard template that you guys have that you run? You know, we know kind of what works best. So there's a standard template to it, but it's also something you can customize on your kind of organization level. So if you have a specific format, if you have specific background information, you can enter that before and then it will take that into account. I mean, I think there's like different outputs. Sometimes you want to have like a long written document. Sometimes you want to have like multiple documents or multiple sections for maybe you have something that's multi enough like a study that's run in multi country. And you want to have like a comparison between those countries you can specify that up front and we can have a tailored report to you. But again, if you don't have what you need, you can also just chat with it with it after. And how do you specify that upfront? Is that in natural language or are there some boxes to check? It's all natural language. So you write basically a paragraph of, hey, I want a detailed research report. There should be one for each country. Yeah, exactly. And the eyes really good at understanding. I mean, if you used to have some some more rigid things, but I think we've all is all gone to natural language now. Okay. So what does this agent look like under the hood? What is it doing? And if there's differences between the lives and the more long running one, maybe let's focus on the long running one specifically, but it sounds like they're pretty similar. Yeah. No, overall they're pretty similar. I mean, how does the agent work underneath? It's actually built our own harness, which is something you can talk about. And it basically has access to these, you know, the main level is the transcripts of these 500 conversations. And then the goal is to, you know, really make that understandable. And it has a bunch of different tools. We're in this interesting domain where it's not infeasible to look at all of these individual interviews again, at least with like cheap LM. We're not in like the millions of conversations where it's infeasible. So we have some tools that can do like a, you know, more recursive summarization or even a classification of like, okay, if you want to know how many interviews mentioned this specific thing, we can actually look have a small model, you know, GPT mini or Haiku or something similar. Actually look at all the interviews, classified and they actually get robust quantitative data out of these like very open end conversations. So how are these files or how are these interviews presented to the LLM to the agent? Are they presented as files or they presented as variables in some programmatic environment? Yeah, so we've been iterating a little bit on that and we're, you know, constantly thinking about if you should change it to more file structure. Right now in the main agent, it's, it's not directly a file structure. It's more we think of it more as a table. So the table is every row is a response is a response. And every column is kind of like a question or a feature that we extract. And then the agent can basically create new columns so I can say something like, you know, what is the user sentiment towards this specific topic? And then please, it could be like an open ended like a summary of this, the use of sentiment or it could be a categorical value like, you know, positive negative neutral. And then it will basically add this column into this, you know, table and fill in the values for each one of them. And then you can use things like Python or things like, you know, like to chart the data basically based on that. Super interesting. And so, and when it fills in the values for each row, it kicks off basically a small sub agent or a small L or those small LLM small sub agent or those the same things. Yeah, you could call it a sub agent, but it's really like a very constrained agent. It doesn't have to have, you know, room to decide what to do. It just does like a call specification call. So that's one of the tools that have is basically like this. You can think of it more as like a map produce call and there's something that they've had coded as one of the tools. Yeah. So you can call it sub agent or you can call it just LM. So it's got this table. It's got a row for each of the transcripts, each of the interviews. It creates different columns. You mentioned like Python or plotting things. Like does it also have access to code? Can it write code? So by default, we give it some, let's say you call this, you know, this classification tool like with the sentiment, right? And by default, this tool actually returns a statistic on, you know, out of the 500 people interviews you did, 325 were positive, you know, 100 were, you know, it returns some default values that you can then use and give now, give the answer most of the time that's actually enough. Or can you use that, you know, we have some, you know, sophisticated way of creating charts based on columns. So you can say, please create a column chart based on this column or a chart based on this column segmented by this column. So we have some like logic to create charts and visualizations out of the box. But then obviously there's like a like a long tail of unlimited things people would want to do. And that's where the Python really comes into play. So you can also write custom Python code. And it's a little bit, you know, it takes more iteration. It will not look as nice because not in our UI, but I can do most sophisticated, you know, statistical analysis, great, very custom chart types that maybe we don't support in our platform. And then compute either compute values or even great specific charts that we then return and show to the user. So I can met both lip or something like that. What percent of queries do you find need this more open-ended long tail kind of like just raw Python? When we see it using a lot of Python, it's sometimes worth to give us the other specialized tool to do that task. So I think the number has been going up and down. The long-running default reports, it actually doesn't use Python that often, maybe in like 20% of the times. So you think it's maybe not as important, but actually it is super important, even if it's still 20% of the time, because we have the dynamic engine as well, which is I have a specific question and the given tools don't support that. So I fall back to Python and actually run the own and.
analysis and that's very powerful. So even if it's not in the majority of cases, it's very powerful to have it so you can drill down if you want to. So if you're writing this code, does this code run in a sandbox? How are you guys managing that? Yeah, so this code runs in a sandbox we're using E2B for execution, executing that code. There's some challenges on kind of making sure we get all the data because it's not tons of data, but it's still like a decent amount of data to spin up the sandbox load the data fast enough, especially in the life. So we do some like pre-warming and like setting up the sandbox to the head of time and those kind of things. So you have the sandbox separate from the agent. So the agent's kind of like running, doing its own stuff. It's got this table like thing, which is, yeah, actually, is that a real table under the hood? Like is it? It's purely a representation. So I think the agent, we tell the agent that's the table, but in our database, it's actually laid out differently, but you can kind of synthesize it as a table. And then if it actually runs the Python, then it gets it gets, it gets this pandas data frame, which is basically this table, but it's never stored as like a table. Like as it sees V5 or something, it's just stored in our postgres in a different format. Yeah, we've we've been experimenting with some stuff like virtual file systems and it seems like this is a virtual table. Exactly. And so you've got this agent running, it's got this table like thing and then it calls this tool and then that spins up a sandbox. So it's not like the agent's always running in the sandbox. It's got like a sandbox as a tool. And this approach, exactly. So most of the time, it can actually just run our back end and we don't need the sandbox because it's more hard coded regular tools and then only for the second backup we need to go to Python. We also do have some newer agents or sub agents that are using the sandbox more natively and definitely thinking about where the future going, definitely seems like the future is going to more agents that have it, I built based on code and run continuously. So it's definitely always something we're experimenting with. What are some of those newer sub agents? Yeah, so one thing and we actually just posted a blog post about this is the PowerPoint generation. So we've been working on the, you know, again, if you take this back, for our customers, oftentimes the end result is often like a PowerPoint slide that they want to present. And PowerPoint is like weird format where it's not, you could just create like an HTML like react page that looks like slides and then you can create a PDF out of that and that's great, but you can't really edit it. It's like our customers can't, they want to have something in PowerPoint. So we've been struggling with this for a bit where we used to have our own hard coded pipeline where we create templates and then we basically have like text replace in this PowerPoint slide X and image replace and you know, find the right templates and so on. So we used to build that pipeline. Now we completely rethought it as we've seen the agents being able to use tool calling and especially code, code generation to create PowerPoint. So basically what we have now is this and Cloud code agent SDK that runs and writes Python code and can then modify the PowerPoint file and kind of do a lot of iterations in that. And the way we implemented it so far is it's a sub agent to our main agent. So the main agent decides, okay, I want to create a slide deck. I should have this content, it gives all the data. And then the sub agent kind of iterates in the code on the code level with a specific skill of how to create a slide and then returns the PowerPoint in the end. And that's this agent that actually runs in another sandbox. - It sounds like definitely the sub agent, maybe the main agent they can create artifacts in addition to giving the final response. How does that work? - Yeah, I mean, out-effects in some ways is just a JSON output of a specific thing. So if it's the basic artifact we have is a chart, which is again, in our virtual table, we basically say create a chart. It's just a JSON tool call, create a chart with this column and this column and then we render the chart to the output. There's some other artifacts that can also create like memos or like, you know, like basically I can written output that combines charts, combines other things, can even create things like, you know, cut video clips from those interviews together, you know, based on the transcripts it selects. These are some interesting quotes. There's some like retrieval pipeline for that. And then it wraps that in a specific tool call and then we create this wheel and it can reference that in the output and then we render the real. - You mentioned tools and sub agents already. Do you also use skills? Is that a concept to that that's made its way into these agents yet? - As you can tell, like the really getting into like the longer tail of use cases where we've been struggling with, you know, the model's getting smarter and the problems are getting longer and that seems to kind of hold the balance. But if you want to go into the more detailed rare instances where we know this is how it's supposed to work, but the agent maybe doesn't want to, we don't want the agent to reinvent it every time then we're using skill. Think in this specific instance, we haven't used skill stat much yet. But we've been relying more on contextual prompt engineering I would call it where like if you want to use a specific tool or I guess the two things that we have the sub agents which we rely pretty heavily on and then the contextual things of like if it's a study that has the specific data structure then we, you know, include a specific, you know, how do you compare concepts in the prompt? Those kind of things. And then we rely on sub agents. So for example, we don't have the instructions how to create the PowerPoint in the main agent but we call the sub agent and that has basically the skill loaded, like preloaded by default with all the context. We have some other agents that we do skill so that more based on what we do. And I think the real opportunity is can you reduce the context, can you make more dynamic but also comes with some challenges? - For the contextual prompt engineering just to make sure I understand. So that's basically like you'll look at the study that was run and based on those properties you'll just insert different things into the prompt. And so the system prompt isn't really the same system prompt for every study or every agent. It kind of varies a little bit depending on the study. - Yeah, exactly. I mean, I think the main thing is just if you can cut out like a big chunk or like there's one feature we have which is emotional understanding based on the video and the audio and not just the text. But then sometimes some other studies don't have video and audio, right? And then we can remove all of these instructions from the prompt and then just have it less, be less confusing. Again, there's no magic trick because in the worst case, you have all of the cases in there and you don't save anything. So it doesn't really help you in that case. But oftentimes we see that, I can help a little bit. - How many different tools and how many different sub agents do you guys have? - For this research agent, it's probably, it's more than you think. It's probably like 15 tools or something like that. - And are most of these tools like running something over the table like a classification or no? - There's probably one or two that run everything across the table. There's maybe one or two different retrieval modes. There's then maybe one for creating PowerPoints. There's one for creating highlight reels. There's one for outputting specific chart and specific way. There's one for creating like a different artifact. So I think most of them are either output related or like processing related, like other compute's new data or create some artifact that you can then use in the artifact. And one other thing we also have is this, which is pretty interesting, is this feedback tool, which especially for the long running task, we can't really do it in the live one, but in the long running task, it can then self-request feedback. So it's basically we have this sub-agent, which is this reviewer agent that just has a clear context, doesn't have all the complicated instructions on, and then not necessarily sees all the history, but just knows what a good report looks like. And then has it lists this or criteria, like okay, it should not have any claims that are not backed up by certations or by data, and then it kind of goes through the reporting and gives feedback. And that's actually the loop that runs, you can run it quite often, and it will actually catch a lot of things and will make things much better. And then in the live runner, we use that as an evaluation system. So basically, I mean, we have to make some slight adjustments, but basically you want to know how many issues does this evaluation agent find in the report? And can you, you know, at least the first step is just at least know how bad it is or what the common issues are. And then if you make a change to the problem to the model, to the architecture, or maybe sometimes you think it's not even related, you'll see if it actually increases things either on a benchmark set or even in production of like if there's a spike in the specific problem. - When will you run this eval over the live things? Is it like offline evals that you run before production? Are you also running it over production data? And like if so, is that happening at the same time as it's chatting or like at the end of a day, you do a big crime job and score a bunch of things? - I mean, I think that's something where we're optimizing right now, we just kick that off after each run in like an asynchronous runner. So it doesn't block things, but it just gets eviled live. I mean, it's probably some cost optimization we can do on batch inference, but hasn't been the biggest priority right now. But it's just fast to kind of have internal metrics. And we might also move to sub sampling, but it's the cost is not the biggest concern for us right now. I think it's really can we make the product better and yeah, it's not the biggest concern right now. - What does make the product better mean to you? Is it just increasing accuracy on the existing things? Is it spreading out to more challenges and more development? - There's something we also thought about a bit recently because in some ways, if you have the signal of these are all the things that are broken, obviously you want to fix them and make progress on those. And that's kind of improving the reliability. That's very important, but sometimes you also want to take a step back and maybe just optimizing this local minimum and maybe sometimes you actually want to, you know, change the architecture overall or change the approach overall and then maybe the criteria has even changed. The sales climbing is very important because otherwise you ship a product that is aspirationally great but then in practice doesn't work. Again, you can only test it in so many cases yourself and the customers, the first customer tries it will use it in a way that you didn't anticipate. So I think it is very important to challenge the e-votes and production. - Are there any fun stories of how people were using it in ways that you didn't anticipate when you first launched it or that you're discovering now? - There's a bunch of things people definitely very soon try to like
not break it in like a prompt injection way, but like really test the limits of like, you know, can you do like a cluster analysis? Can you do like, can you run like a, you know, classification model in the Python and so on? And then, you know, usually the, like, the Python leverage image we have is limited. So you can't do everything. And then like, it tries to like create code things up from scratch. Like, can you show them? You write like a classification algorithm, like myself in my own pipeline, why not? Because it doesn't have access to those tools. And then we're like, okay, maybe we should give it some some, some of our tools. So it doesn't have to do those things manually. It's always interesting. And it happens much sooner than you, than you expect. When you're like iterating on these insights from production usage, are the most common things you're iterating on like changing the prompt or giving it more tools or updating like the environment that is running code in like, where have you seen the distribution of things go? I think obviously the updating and prompt is the lowest lift. The hard part is, you know, if you change the prompt, it hopefully improves the thing that you want to improve it on. Does it then, especially if you're like, change it to all caps and repeat it three times, does it then mean something else regresses? Because there's less attention to some other intersections. And I think that's the hard part. How do you gain confidence in that right now? Yeah, it's, I don't think we have a good solution. I think obviously as we deployed and sometimes we roll it out slowly and see, you know, other number of issues, but I mean, we have like an example, like test set of like maybe 20 different product. We can always run it on. We can evaluate. We can see at least in that small sample size, it's anything. And that's what we actually manually look through the outputs and see, okay, this does seem good, but it's, it's really hard to compare one like five page output with another five page output. They're different. People would probably argue this one is better. Some people would argue this one's better. Like, I don't know, but at least you will see if something clearly breaks. How many times, if at all, have you guys like completely rearchitected this agent? It sounds like maybe you're thinking of that now or starting to think of that? I've done it a bunch of times. I mean, over the last two years, I would say, it was like been the main impetus is the models just getting better or I don't know, you hear a great talk from someone at Lengshane maybe or something like that. But no, like what's been the impetus for the marketing thing? I mean, I think the first iteration was just like before, kind of agents really worked like the first version was just like a simple rack bot, which I guess probably most companies start out doing and that's still a primitive that we have, but I think both like smaller models getting good enough that you can actually do, like look at all the data, classify things live that's going to change and then the big models getting smart enough that can really orchestrate those things. It's been the biggest change. And I don't think it was like a single thing. I think we've been talking about this a lot and it's been a big investment from on our side to re-architect this. And we definitely did a lot of explorations before trying it out and yeah, now I mean, obviously most people are moving throughout like a file system based agent, something we have explored a couple months ago and there felt like especially in our specific use case, it wasn't that kind of still want to have these specific tools. Is that largely because of like the table that you guys have and the ability to quickly run over the surface? Exactly, the table is pretty powerful. But there are arguments against that, especially if like the model company starts to post train on the specific harness or the specific tool calls, it feels like you're kind of fighting against that movement if you have your own harness. I think right now all these like coding agent harnesses are pretty bad at like calling sub-agents programmatically. Like they can call them one at a time, but what you guys really want, it sounds like it's basically called it 500 times and have that always happen. And this is similar to some of the stuff in like the recursive language model paper, but I don't think any of we're thinking a lot about this. How can we take those ideas and put them into like these coding agent harnesses and there's some stuff we're thinking about, but I don't think we or anyone has really nailed that. Yeah, exactly. I mean, you know, obviously the way we do it is we don't actually, the model doesn't actually call it 500 times and it just calls it once and then we have this hard coded workflow. If you call this to at once, we spawn those 500 agents and then we aggregate it in a very specific way and then it returns it. Can it like call it over a subset of rows? Could it pass in some like filter criteria to filter? Exactly. So that's one of the use cases and maybe also coming back to a question about whatever we changed. It's from changes, but then something like, oh, oftentimes it would have to filter. And what it used to do is run it on everything and then write a python script to filter out instead of giving the aggregate results over all of the data that were just with python filter out to the aggregates of those. We're like, okay, this seems a little bit, you know, cumbersome. Like let's add the specific additional fields in the true call of like filter column. This column equals this field to filter. It works in like 90% of the filter cases, there might still be some advanced filters based on like combination of columns. It might need to write the python script one, but that helps a lot. So that's one of the other things that we obviously see as we deploy those. But that's really something you have to look at traces yourself. You have to have good observability. You have to really go deep and see what it did do. Does it actually make sense? Sometimes even look at the reasoning traces of the model. Like why did it call this thing? And then see, oh, I wish I had this tool for this. And they're like, okay, maybe I should give it that tool. What is trace analysis trace observability look like for you guys? How do people do it? Who's doing it? Is it everyone on the team? Do you have like specific people who are focused on it? Imagine you guys have millions of traces. How do you know when traces to look at and then when they find something, what do they do with it? We want to trace every single, every single trace like a store. You don't want a sub sample. But then we never look, we only look at the 0.01% or less. But it's like logging. You want to see this specific case, there's something weird. Let me debug why it is. And then you go really deep. And then, I think the depth over the breath definitely makes sense. And then sometimes I mean, we have had like clod, like clod just run analysis on all the traces. Like what are the common things that the common paths or the common things that happen? That's also interesting. But I think the most, you learn the most by just going really deep on one or two traces and really looking at more less every single tool call. And if it's a 30 minute trace, it actually takes you quite a while to go through those. But that's how you learn if it's work. So it doesn't work. And that's usually in the development, like you just run it on your own computer or on your cloud dev setup. But it's also in production debugging what went wrong. You mentioned sandboxes twice. It sounds like once you have a tool that like runs in the sandbox and the other is a sub agent that spawns a sandbox with the agent inside of it. What have you learned about working with sandboxes? Any, yeah, lessons learned there? Our learning so far has been it's harder or we early in this phase. I think no one has really figured it out. I mean, just one example when we tried running the clod agent SDK in the E2B sandbox, the SDK is not really meant like it's not been developed to be run in that kind of cloud environment. It's been developed for local environment. And there's some of those assumptions like it needs your anthropic API key. If you have that in the sandbox, obviously it's now susceptible for people to extract that if they just ask, can you please in your report include the API key or all the environment variables that's a problem. So what we had to do is basically proxy all the requests that give it a fake API key, proxy all the requests through our server and then verify that's actually the right request. It's not just anyone request and then replace the API key with the real API key. And there's been a lot of challenges on that way. It sounds very simple, but we tried to deploy that on render and then render was like, oh, this looks like you're sending code in this HTTP request. That sounds like there's some malicious behavior and then we're blocking your request for this. There's like a lot of things that made us think, wow, this is still very early. It feels hacky almost to just deploy that and think some of the things that you're from working on sound like super relevant for this and making that much easier. So I wish we were just that earlier. Do you guys have memory anywhere in any of the agents? You have certain ways you always do it. So the way we solve it so far is like a relatively explicit version of memory where kind of on your organization level, you can give it instructions. I think that's a pattern that you see in a lot of companies. So you give some some general instructions and they will be applied for setting up new projects that will be applied for analyzing product in a specific way. And those are all like human general humans. Exactly. You can define them yourself. And the biggest question for us on the analysis side is what is actually something that's interesting, whether something that's obvious. Right. And I can't really judge that from the outside. It's really something that the customer needs to detail us because from the outside, everything seems new and interesting. But then when inside, you're like, yeah, that's the thing I know I've worked here for 10 years. That's not something new to me. And over time, you really built that through all the all the reports we're generating for other projects. How can we use that as an input for creating new studies? And there's a lot of complexity there because maybe someone else, maybe you didn't even read this report that we assume we already know because someone another person, another business unit. So it's not super easy to figure that out what is common knowledge, what is not. But using some things across like formatting preferences or in the way we set up projects using some of that kind of previous knowledge that we try to distill. But yeah, it's far from being solved. I think it's a lot more work to be done in that domain. Let's talk about UX for a little bit. And maybe here we can zoom out to the three or actually even like four different agents you have. So maybe for the first agent that you mentioned, the kind of onboarding agent, it sounded like there was a doc on like a word doc on one side and then you would chat with it and it would fill out kind of like the study guide. Could you talk more about that UX and what that looks like? Yeah, exactly. Who is reading a lot on that? I think the first version you would think of this is you just prompt an LM to write your discussion guide, your document. And then you do that and you realize it never gets it right 100%. Not to play in the model. I think the model is great. But just like you give it one sentence, you expect it to write like a whole page out of it. Like that's probably not going to work. So there needs to be some interaction. And then basically the second version we build is can we
just have the AI, can you just chat with that, right? And you can kind of make modifications to that, right? And then you have the two basic principles that you have to decide between this, like either the LM re-rides your entire document and that works for shorter documents or it works if you're actually making sub like changes that can you change the tone or something like that. Or do you say like we have some kind of edit functionality that is either string replaces or you have some IDs that you replace only specific IDs, which usually works better, but sometimes can also be confusing if you're making a lot of changes. You stack them is like ID number two is now ID number three and then you insert a new element here and then all like it's also not perfect, but that's kind of the approach that that we picked can we just make because our documents are pretty long can we just make target changes and so on. That's pretty cool. And then you realize like sometimes chatting with it is not the fastest way to to modify things. Sometimes you just want to delete that and then telling the I please delete question number three feels a bit cumbersome or sometimes you just want to reformulate it yourself. So you do you want to have a way of also manually modifying it and you know open a an atrope like they have some version of that. I don't think do you access like supernatural and they've also iterated a lot on those. I've seen like they rolled back some of the changes that it and so on. I think we have a pretty good solution now where you're both working on the same edit history. So you can actually undo and do changes and you have to kind of have this change dog. You can compare things and you can manually make changes and then you can also make changes with the chat and it kind of knows about the changes you made. So it doesn't undo the changes immediately and how does it know about those changes are those inserted how yeah, how does it know about that. Basically the way it works is every change we make is formatted as an edit operation. So you see a lock of all the edit operations. So then it this knows. Okay, you just modify this question because the main problem is you can't always have like this is the old approach like this is the old document is the new document or somehow you modify the diff. So we kind of formalize that in like an edit operations way and then the model knows. Okay, if you just touch that or if in the history, do you see that you're having touched that and you probably don't want to rewrite that because those are passed. So like if I I'm chatting with it, it gets a document. I go into the document, edit something and then I chat with it again. Like that edit is passed in like prior to my message in some way. It's the same like the LM rise up edit edit operations and the human edits also edit operations that kind of fit into the same kind of message history. Yeah. So that's the approach that worked pretty well for us. Cool. So that's the onboarding agent. Then there's the interviewer agent and that sounds like you've got voice there. It's multimodal. It's more just like a chatbot style thing. Anything interesting there that you guys have been playing with the interface like the voice interface is still still not quite solved. And we've seen even like, oh, may I like the TFT app has been going back and forth right there used to have this like blue bubble level speaking. And now that their voice mode is actually you see the text and it's right it's out text as well. It's kind of the approach that we have been taking for a while as well because you can actually read text faster than you can listen to it and sometimes can actually be annoying. Like, you know, I wanted to expedite or something. But then sometimes to speed us too fast and actually want to go back. So I don't think it's fully solved yet. And same time for our use case, we really don't want to interrupt people in almost all cases. We just want to we listen company right here to listen to the customers. We don't want to interrupt them. Even if they may be saying something that's you know, rambling or maybe going slightly off tangent, oftentimes there's a reason for that. And in only very rare case we actually want to interrupt. So even if I take a break for a second to think about something, I actually don't want the eye to jump in. And are you guys using kind of like the speech to text to speech sandwich or using the real time APIs? But now we're mainly using the, you know, speech to text to speech. But I'm just because it's so important to have the smartest models. And oftentimes these real time services and last time we evaluated them at least, I think the models are not getting pretty good. But at least when we evaluated them, they were like one or two tiers, you know, faster and dumber than they, you know, top tier opus and so on models that it is so important to ask that it sounds very simple to just have a conversation, then asking the right questions. I mean, guess that's what you're doing today. It is actually pretty hard to ask. And that's why we don't want to compromise on that and rather compromise a little bit on the real time aspect of it. Yeah, you've got the fast and dumb interviewer here today. So I feel like most people, myself included, have largely stayed and kind of just like the text domain of agents. When you think about adding on voice, like how much extra work is that? Is it easy? Is it hard? It depends. I think the hard part is not necessarily the AI. I think it's more like, you know, we're collecting fastest. It's the interview runs on like millions of people's devices, right? And the more modalities you have, the harder it becomes for micro compatibility perspective, and kind of see all kinds of issues on like somehow the microphone stopped working or you know how like everyone used to have like trouble with like zoom, like microphone, not getting recognized and those kind of things. So we see all of that across the globe, right? And globally. So I think those are more the challenges with the multi-modality fast than like on the AI side. And the AI models are getting pretty good. I think there's still some challenges on, you know, transcription. You'd think that it's a solve problem. And you know, models are getting pretty good, but there's still something like you release a new product tomorrow. The transcription model doesn't know about, you know, that name. And it probably uses some other name. And you know, the only way we could solve that for now is actually having an LM that has the context on the interview and maybe even know some of the terms it might come up correct. The transcription model basically on the fly just to give this additional like smartness that the transcription model self-count can't use. So this is some of the challenges and there's obviously like traditional things of like now we're storing like, you know, thousands of hours of video data on you to, you know, do that, but those are more like traditional infrastructure problems. And then going on to UX of the final research agent in the two different modes it has. So when it's running kind of like long in the background, how long does that take? And do you kick one off? How do you kick one off? Does it happen like automatically after all 500 interviews are done? Yeah, that's a big challenge. So again, it runs for like 30 minutes and so on. So the cost is like, you know, significant. It's nothing crazy. And you know, we're usually like a higher priced offering. So it's not the biggest concern, but you don't want to re-kick it like after every time there's a new response or someone updates. And you know, we have 500 responses. At the same time, we do want to show you results early, right? You get the first 10 responses that that's a magical moment and that might happen like 30 minutes after you launch the study or they get the first 10 people to respond, which is very magical. So we do want to give you something. The way we do it right now is we run it at a certain thresholds of interviews. We do like a full analysis and kind of rethink all the hypotheses. And, you know, if the data has changed from 10 to 20 or from 20 to 100 interviews, we actually want to completely rerun this. And that's one part. But then now you have we run it after 100 interviews and now then 100 and first comes in 100 seconds come in like you don't want to rerun everything. But at the same time, you have we have like numbers in our report where we have percentages, we have charts and all these kind of things. And those things we constructed in a way that we can actually replace those things. So they never outputs those numbers. It only outputs place orders. And then we can run all the classifications, all the Python code again. But keep the core thing the same. So this way we always, our numbers are always very viable and updated. So you can always click on them. You can see the data is backed up and there's a new response coming in or you remove from response because, you know, maybe you don't like them, or a second quality. Those things were updated immediately. Obviously, there's like a limit to that. If you say like, if in the text it says like, you know, this is definitely the best idea because 80% of people liked it. And then more and more response come in and suddenly it's no longer 80% only 20%. Then actually the, you know, qualitative takeaways change so that you do need to run it occasionally. But yeah, that's one of the one of the things we work with them. You know, at the end, you usually, in our use case, usually said, okay, now I'm done or you haven't seen any new responses for two days and then we run like a full new analysis. But I think the real-time component I'd really believe in this, you know, delayed gratification, like the faster it gets your results, like the more, like better the user experiences. So you really want to do that. Interesting. And then for the end then for the real time, kind of like asking if you ask questions, how does that work? Because I imagine it's doing a bunch of tool calling on the hood. Do you surface those to the user? Do you hide those? Like how, how transparent are you about the agent's work that it's doing? Right now we do show like an abstracted version. We don't, like our customers don't really care about what happens under the hood too much, but they do want to see something's happening and maybe things are like, okay, I'm actually now looking at all the responses again. And this way it might take a while and we have a loading state. That's kind of things. The more tricky thing there is what happens if it now starts writing Python code. Because in some way, if you ask it a complicated question and says, you know, the answer is 42. You're like, okay, I guess usually all of our findings are very traceable. As I said, you can click on the number as you can see the breakdown. You can even go down to the individual level. This is everyone reclassified and why we did it and can really cannot explore the data. But the Python that's no longer the case. And at the same time, you know, no one wants to read Python. And in our case, your customers will probably also not understand the Python. So can we make it, can we build the confidence that it's the right answer? And obviously there's always been, there's always assumption assumptions going into that. The instructions are never one, like completely clear. So, you know, basically we're currently summarizing exactly his assumptions we're taking when we're executing Python in like a little box that you can expand. And if you want to look at that, but we don't show you the raw Python script, which is something we're thinking about. It's kind of in the middle of gone. It's not perfect, but it gives you some confidence that what it does is actually write. And how are you summarizing? Is that another LLM call that's like looking at the trace so far? Yeah, exactly. It's also
the Python code is run, it will, first we showed us some messages like running Python. I don't think we actually saying that anymore, we're saying like running some computations or something. And then off the script and written in the script and why it's executing the script, we actually summarize it to have that text. And it's more, it's less for the status, what it's doing right now and it's more for, you know, where it has come from. Let me go deeper and look at what it's come from. When you guys are running like these sub agents or small m's over the 500 documents, what if the documents are like really big or really massive? Do you do any chunking there and like further kind of like sub-setting the text and chunking into three things and then running a small lm overall three things or do you always just treat it as one big thing? Yeah, so there's multiple layers. Our interviews are what we call semi-structured. So we have a rough idea of what people are talking about. So there's different sections that, you know, in this section we'll talk about, you know, this concept and this section we talk about, this concept or we talk about different ideas. And so we basically the interviews are annotated that way and we can filter it to a specific relevant section. So that helps. And that filter would be part of the filter that you pass into like the main agent would decide, you know, for this second question, we don't need to look at the entire interview, we just need to look at their background information and something like that and then we just cut it to that section. It's not perfect, right? Sometimes people might say like, oh, by the way, I forgot like the very different setting of like, oh, I actually forgot, I change my mind, whatever. Again, not perfect, but I think that's a pretty reasonable assumption. We do use some trunking and some retrieval for if you ask a question like, oh, did anyone mention something like this or can you find clips where people talk about this specific topic? Then, of course, in theory, we could run this, you know, my previous function call over all of them, but in fact, it's usually faster, especially if you're going to like the thousands and tens of thousands interviews to use a retrieval like a semantic search on trunking. We do some hierarchical summarizations as well for this extraction step because sometimes if you imagine doing like a thousand, you know, summaries on individual interviews that's still a thousand times maybe 200 tokens, it's still a lot of text. So then we actually using like another layer to summarize that. As your use of retrieval change over time, so it has to change a lot. We still using retrieval. Again, the first version of the research agent we built like two years ago, it was just a rack, I would just say semantic search, and then the second thing we added was can we add some robust filters? Can we just if you ask, can we just filter for man like we extract metadata, we filter based on that? That was the first version. The second version we're like, okay, we saw all the problems and we actually completely moved away from retrieval. We were only doing these like small elams looking at everything. We didn't have any retrieval five-pointed there at all. But then we realized for some cases, again, especially if you're scaling to larger samples, it's still useful and it's sometimes faster, in some cases it's better. But it's definitely less critical than it used to be. But even if it's not that critical, it still means we embed everything. And the cost is not not prohibitive. So we just do that. Especially if you're working towards a platform where you can search through all of your findings, not scope to a specific product, but kind of over time. And then of course the data becomes much like a much larger corpus and I think retrieval will continue to be important. I know the coding in the coding world, people now just use keyboard search or a Grap. Have you implemented that over your transcripts? We haven't really implemented that. I think the main thing is, in code, I guess you have symbols that are like types, like strongly types and you can actually search for exactly that string. In a conatrial conversation, it's much harder if you want to think, if you want to retrieve all cases where people are frustrated, that's maybe a semantic search, but it's really hard to show you can search for list of adjectives that people might have used. It's really hard. So in our use case, we were just sticking with the regular retrieval. On a completely different note, what does the team that does all this agent engineering look like for you guys? It's engineers right now. We're relatively small team. But one thing I look for in hiring engineers is kind of this product sense because I think what this maybe has two types of engineers that do well in this world. I think that's one that is really good at building large scale systems. I have seen that have a good taste and maybe there's something at the labs at least right now can't do super well. The other side is these product engineers that really you understand the customer iterate fast and try out things. I think it's super hard to whiteboard. This is how the agent is going to work. These are going to be the problems and this is how it's work. You need some part of that. You need to have some some idea of where you want to go but then you need to try out things. How reliable is it? You need to adapt. For that, I think it's very important that the engineer itself is the one that that is evaluating that it's talking to the customer and adhering how it's going looking at the logs and the traces themselves. You mentioned previously someone else is looking at the logs. No, it's actually the person that that built a system that's looking at the logs and trying to understand this model, do what I wanted to do. I don't really believe in people that just prompt, just write the prompt because again, it's a very nuanced thing of if you change the prompt then you also need to change the tools and then all the tools need to understand some of the products and infrastructure and so on. I think this entire ownership is how we've been building the product and I think that's where I think it's going to stay the same even as we grow the team. Do you have any non-engineers contributing to the agent, whether that's a product person or a design or some subject matter expert? I don't know. Not directly. I think the problem I see is it's relatively easy to change something like the prompt that I go, "Oh, yeah, I just add the sentence to the prompt." People definitely want to do that. But then from my perspective, I'm like, "Okay, sure. It's easy to change it, but what about the validation? Who's actually going to take the blame?" If it breaks, who's actually going to fix it? Right, right now, the number of PRs we have in our organization is exploded. It's so easy to write code and I go, "Oh, can I go and get into the code and write code?" But I don't want the engineers to just be the ones that review code and improve changes and then later on, the ones that fixing it. That's a little bit of the problem that I see. We obviously working closely with customer facing people and getting feedback, but then it's typically the engineer that will consolidate with all the other requirements and try to improve that. Then get feedback from them again. Do you care if people joining the agent engineering team or the team that works on it? Do they have to have previous AI or agent experience? Or is that something that, "Hey, if you're a good software engineer, I think you can pick this up on the fly." I changed my mind a little bit on that. Maybe a year ago, I was like, "We just sound very smart people and if they haven't worked with AI, I think that's fine, they can learn that." Now, I think we're at the time where it's a bit strange if you've never worked with AI. It's been like three and a half weeks, exactly. Or even if you haven't worked on your job because for whatever reason, you're a company doing it, you should at least be intellectually interested and curious about how does it work behind the scenes and build something on the side or at least know what it's called doing if I'm asking it to build my PowerPoint. How does it actually build that? So I think if you don't at least have that level of experience, I think it's no longer fit. Yeah, I think I'm kind of the same way, I also updated some beliefs. Awesome. Thank you for that. Thanks for listening to Max Agency. If you liked this episode, leave a review and subscribe. Send feedback or questions to
[email protected]. We want to hear from you.
PodcastTranscript AI