How Cogent builds AI agents that have to be right every single time | Geng Sng (Co-founder & CTO - Cogent)

This is basically Cess 2.0, attackers only need to be right once and defend us from right all the time, and today the mean time to exploit is in minutes. Today I'm talking to Gangseng, co-founder and CTO of Cogen. They're building agents for autonomous cyber defense and closing vulnerabilities at machine speed. We don't use a graph database, but we have a very high right throughput system, like billions of events flowing through the system every single day. He walks me through how Cogen builds trusts with customers before letting agents act autonomously. Even if the agent is solutionating and wants to do some kind of right action, it's not allowed to do that. And how they make sure their LOM judges are actually qualified to assess the agent's work. At the most complex level, it's LOMs that have environmental parity with the actual agent that we're e-velling. Cogen also has an interesting way of thinking about context. We're differentiating between hot and cold contexts. It's not that this context has existed for a short amount of time. It's that this context is active being used. Right at the end, Gang talks about why defensive security won't be replaced by AI anytime soon. The nature of a defensive security task is I have a castle here on my walls. How do I patch every single little hole in the wall? And every single castle is built differently. There really is no data. Welcome to Max Agency, the podcast that goes deep into how the best agents are being built by builders like you. Cogen is building a platform for autonomous cyber defense. What does that mean and how do you think about building that platform? Yeah, it's actually really super interesting because autonomous cyber defense was not quite a thing. I think up until very recently. What's really motivating this is the meantime to exploit for vulnerabilities has really compressed over the last couple of years. So two and a half years ago, if you were a security team of vulnerability comes up, you have about two and a half years to basically backlog it, triage it before it first gets exploited in the wild. And today the mean time to exploit is in minutes. Why? It's a combination of one, a lot of digital sprawl. So the attacks are physicists like increasing at a pace at which it's not possible for defenders to keep up. So software and vulnerabilities are two sides of the coin. More and more software is being invented, created, and more and more vulnerabilities are being created, but not being patched. And on the other hand, the frontier models have really accelerated the exploit capabilities of any attacker. So I think cloud recently was talking about how they found all zero day exploits in Mozilla. And this is one of the most battle tested environments that have gone through decades of pen testing that now with a frontier model like Opus 4.7, you could probably, you know, with 500 turns, find your way through, find the exploit and find a way to basically find a zero day. It seems to panic totally that there's been a ton of more security incidents recently. So how do you guys think about, I'm assuming the code you can put in the platform helps remediate that in some way or detect those in some ways? How exactly do you guys think about the role that you play in that space? Yeah, exactly. So the challenging aspect is not finding the vulnerabilities. It's really fixing and patching in time. And in a world where you need to move at machine speed, it's not enough to have humans in the loop. And the reason why we kind of think of ourselves as a autonomous cyber defense platform is because you need to have these agents that can react in a timely fashion that have contacts on how your organization wants these things patched. And ultimately, it has to write judgment to make the right decisions, to either apply a patch, contain a vulnerability or do anything on behalf of a security team that would have consequences in a high stakes environment. Walk me through what that looks like in practice for a customer. CodeJun's customers basically plug into our like platform. And the first thing that we do actually is building contacts. And a big part of that is plugging into their vulnerability scanners, threat intelligence. We scrape a lot of things on the internet where we are basically trying to understand if there is a newspaper article that's trending, you know, somebody like a researcher says something on X, something Reddit, factoring that into what we consider as a risk signal, as well as other proprietary threat and tell feeds. We also work with a lot of asset inventory information, so CMDBs, as well as like any kind of like IT asset inventory. And the most interesting aspect of CodeJun actually is business context. So there's a lot of information in your ticketing systems that show your password radiation history in the comments, live in a ticket that tells you why something was impatched. There's also a lot of context in things like your share points, your Google drives, policy information, information about product launch. So being able to take all that context, build that into this big context graph and not just understand the system context, but also how decisions are being made. How are agents which are operating at the mix, like I recall it like a intelligence layer if you will, of how to take actions when things change in the organization. That's kind of like the prerec as we kind of see it in terms of ticking action in autonomous fashion. Okay, so you build up all this context. How do you do this? Is this an agent that's building that context? We call it agent lake, basically it's an agentic data lake and we kind of have two parts of our system. So the first part of it is basically a context form. So agents forms that have live access to data. We have this context engine that essentially has a lot of memory harnesses around how we're extracting information. These agents are capable of going on long running tasks and they basically answer the really hard questions about system ownership, about a certain remediation playbook and this information may not actually live as a codified artifact in the organization. It might live in little pieces of data that needs to be corroborated and ultimately contextualized by the customer. That's the first piece. The second piece actually is like a high scale data platform. And so we're kind of like combining agents with high scale data processing and in parts of how we construct our graph, the topology is just the first part. So agents are actually coloring the graph over time to create more and more interesting lower high confidence edges. And that's really what allows us to do really high quality risk classification for a lot of our customers. And that high fidelity of the risk classification is what allows in the cut through the noise because a lot of customer environments in the enterprise, which are most of our customers, there's like hundreds or millions of vulnerabilities and you need to have the ability to say, hey, these are the top 10, top 100, top 1000, not only that, but given the risk, it needs to have an action taken place right now, mapping it to your custom SLA mapping. There's so much complexity that goes into any of these decisions. And so you mentioned graphs a few times now. And that was going to be my next question because sometimes in RAAG or retrieval or context, there's just a whole bunch of different data structures that you can use from vector stores to graphs to just files. And I feel like graphs have a reputation as being the most complex, both good and bad. And so it sounds like you guys are using knowledge graphs as the database itself. Is that correct? So we don't use a graph database. Okay. And the reason why is like, I think graph databases are proven to not scale very well. We have a very high right throughput system. So like tons and tons, like billions of events flowing through the system every single day. And what are these events? Changes in life cycle entities. So let's say a vulnerability. A scan can happen on a cadence or it can happen in real time. So any single entity like a vulnerability, like an asset, like a person, any identity could have some kind of life cycle change, right? Like it's a van coming in. We want to know about it and we want to model in the graph. So instead of using graph database, we actually use a data lake. So we actually store things in S3. This really cheap is really fast, very like right throughput optimized. And it gives us the capability to basically materialize a graph that we need that is like kind of custom for all use cases. Is the entology fixed up front or does the agent actually decide the entology as it is? It builds the graph. The ontology needs to be fixed at the start. But parts of the ontology does not need to be fixed. So maybe the way to think about this is there are structural properties of the vocabulary of how you might think about an entity. So for example, a vulnerability probably has some kind of identifier, right? Probably has certain attributes that are well known. So those need to be fixed because it helps us identify and resolve the entity in the stream of events, right? So without these almost like signatures, we don't actually know what this thing could be. But once you're past that initial level of hey, here are the properties, the agents can really come up with any kind of derived attributes on the fly. An example of that is Internet exposure. Like I don't really know if this thing is Internet exposed, but here are the five data sources, including your wafelogs, including some security tool that you're using that make me think that based on some reasoning, this is an Internet exposed asset. And it sounds like there's an up front step to create this, but there's it's also like always ongoing. Is that correct? Yes, that's right. So even the up front step is actually agent. So the discovery process of what these properties should be and how they map to all customers environments across different vendor integrations is super complex, right? So you know, you might have an asset, but every customer will have different combinations of data sources and being able to do that level of any de resolution across like, you know, basically very high cardinality of data is super difficult. And so the agents actually do kind of the first pass EDA and actually one of our like more, I would say like harder e-veils that we have for our agents is long running data science tasks. And it's really these agents are like highly capable, you know, have access to Jupyter notebooks, they run Spark and they're able to do high scale data processing. And that's what allows us to look at, you know, a million rows or like a hundred million rows in like a couple of seconds. And that's all just to fill in the graph during kind of like the end of the class. How long does that take like the initial setup of the graph? The good thing about like the graph is that once we have these kind of like default, I guess like on top of that. apology vocabulary. It's actually a lot easier for us to just apply like a bunch of templates, but there is kind of like two phases to the way we think about context enrichment. So the first pass is always going to be lossy, but it's going to help us paint the picture of the topology. And once you have the topology, you can start answering questions around, oh, how are these things connected? Right? Like, are there interesting stories that could be told? An example of that might be, does this asset have sensitive data on it? Right? So like being able to do a second pass, a third pass within agent is super helpful. So you do an initial pass, kind of like course high level, just set up the entology. And then you do second and third pass on kind of like more narrow focused questions to kind of like fill out the info at a high level. Yes. I think what's interesting also is like there's two types of tasks that the agents are doing. So one is answering open-ended questions on things that are unverifiable. When you say they're answering these questions, is this like when the users now using code like your end users are using it when after they set up the graph? So not quite. So you plug your data in and usually that's through APIs. And there is like a class of unverifiable context where even the users themselves have no idea. The perfect example, and this is generally true, not just in security, but also in engineering, is system ownership. Right? Like some of these organizations we work with people leave the company all the time, like engineering teams reorg. It's really hard to know if this AWS tag is the right to also truth, or maybe I need to consult an org chart, right? I need to go figure out all these things. Being able to answer those types of questions is the role of this context form that I was talking about. So having agents basically ask very pointed questions on data that is unsure, building confidence through like these like many, many like reasoning loops is like a core use case. And then once you have that, we kind of call them context artifacts that creates the provenance for the downstream enrichment, right? So almost like it's like the pre-step of your cleaning your data. And once that data is cleaned, we're able to say, oh, actually, like, you know, we do think that this is the right system owner because we've done all this work. Super interesting. Okay, so this is all to set up the context graph in the first place. Yes. What happens after that? So that's where things get really fun. You know, Kojian, one of our big value-probstoral customers is auto-fixing. And this is historically something that's super challenging, right? Like I'm sure everybody who's software engineer has gone through dependency hell before. You upgrade one software, you like, break something else. There's no the invariant. You end up going on this like well goose chase. And at the end of the day, you've spent like a couple of hours and maybe you haven't achieved anything because you realize, oh, actually, this thing can't be like upgraded because now I need to do all these things and I can't deploy into production. That same story is true vulnerabilities, which basically tend to like require patching. And that's like a software version upgrade. What makes it even more challenging is this concept of vulnerability chaining, where the upgrade not just is maybe causing some dependency issues, it may lead to more vulnerabilities. And so where our agents really have, I think, a lot of value and so a lot of room for improvement is helping customers go on this journey where they're doing a lot of the automation of tailoring the remediation guidance to validating the actual steps that it takes to deploy something into production, whether that's through staging first, like basically running through all the steps, codifying it. And then finally, maybe it's through automation or auto remediation, getting the patch into where it needs to be without any kind of human oversight. And users trust these agents to merge things into their production code base. So it's definitely a trust building exercise, right? All of our customers start in read only mode. This is the purely assistive version of cogent where the AI is making recommendations and users have the easier proof button that starts from ticketing. Right? And the next trust letter, if you will, wrong is auto routing of the tickets. Right? So over time, you build confidence that for certain classes of remediation actions, cogent recommendations are really good. And then you essentially start smartly auto routing these tickets. And the next kind of like level of automation is auto validation, where you let the agents basically start testing in your debt environment, staging environments, they're spinning up things in dev sandboxes. They're running the patch, right? They're showing you the proof and evidence of like whether or not this will break something. And actually, like the the really challenging piece here is not applying the patch or like knowing if the patch will work. It's knowing if the patch, if it's applied, has production impact. Right? This is usually the key blocker for true automation. And you know, like we work with a lot of big enterprises, some of them like our banks and like something that a person will say is like, look, never, ever will you like autonomously deploy onto my like payment processor. Right? Yes, I trust the AI. Yes, everything's good, but that's just never happening. And so you can use cogent in any way that you want. And really it's more so customization that customers prefer. But our thesis is there will be a lot of low blast radius things that you can fully automate. And when you use cogent, it's not just, hey, let's go auto fix everything. It's what percentage of your attack surface can be truly automated, right? Like within, let's say, five minutes off an attack happening, it can be auto fixed without you needing the care about that effectively shrinking your attack surface. And for everything else, you have the AI can be autonomous. It can be semi-autonomous. It can be completely manual. Kind of opted to customer. What is a reasonable percentage of things that can be fully remedied? I think it depends on the customer profile. So a lot of, I would say, like startups and tech companies tend to have very high risk appetite because they value productivity. Right? You're like, it's fine. Like, let it break, right? Like, what would the oil of it? And then when you work as some of our customers who are like, like, fortune 1000 type customers and have like huge scale, right? Like, presence in like, you know, hundreds of countries and like a lot of manufacturing, the stakes are a lot higher. And I think in those examples, it's more so the modality of it. So cloud is very difficult, but endpoints are a lot easier. So people are like more okay with laptops being maybe in like a weird state versus a cloud server having some issue. And the reality of it also is like a modern enterprise has maybe I want to say like 80% of their tech surface in dev and staging. Right? So in some ways, like if you can really isolate the types of things that you're automating, there's huge attack surface that you can fully automate. But it is a journey and we want to make sure that customers feel comfortable and usually where we start is like in the smallest slice. It sounds like there's two main agents then one that builds a graph and then the other that actions on these vulnerabilities that come in are they the same agent architecture under the hood or are they different? Yeah. So they're actually quite different. At Cogeant, we have three types of agents. So you have these interactive agents that work with customers and they have different latency requirements compared to the second type which are background agents. So background agents work in the cloud. Those two examples are just two examples of agents that we have actually like there's a lot of automation across the platform. Even when you upload something into our like portal, there's a follow analysis agent, right? It's going to go understand what you uploaded, map that to the rest of the system. It's going to maybe write some code to plug it into an existing framework that we have. And so there's tons and tons of background agents. And then the last type of agent is actually coding agents. So we have a I would say like pretty AI native stack in the way where we've invested a lot in our software, the LUM lifecycle. So native tooling around how we build software, like agent harnesses that make it easier for people to think through what invariance look like write tests. And I think that type of agent is also very powerful because it allows us to ship quickly but durably. And yeah, I definitely think it's like one of the areas that like a lot of folks on the team are super excited about. And that's like an agent that's internal to the company. But then the interactive in the background agents are the one setter and the product is that correct? Exactly. Maybe diving into those. So understand kind of like the latency requirement differences. How does that end up manifesting in the architectures themselves? Do they have drastically different system prompts, different tools, different harnesses completely? Maybe to answer that question, you have to understand the problem constraints that we have. So security data is very high scale. Right. So easily like terabytes of data pro customer for each data source. And so what that means is when agents doing data analysis, it needs to do some like heavy duty lifting to figure out like how to analyze things. And a lot of our systems are built for performance because of this requirement. The second piece of this is safety, right? So we are a security company and we are dealing with highly sensitive information, right? Like vulnerability information and like all types of contacts that needs to be very carefully handled. And so the way we think about the architecture of anything that touches customer data requires a lot of defense and depth. And concretely like, you know, we do run agent sandboxes, but the level of isolation that we apply to these sandboxes is very, very deep. So these sandboxes basically have many layers of permissioning where most of them basically operate at the box level with read only permissions. And we have this kind of really interesting policy engine that allows us to like elevate access. And so you never ever allowed to like even if the agent is solutionating it once they like do some kind of right action. It's one not allowed to do that. And then two, when it needs to do things like that, it actually needs to do it outside of the box in a way that is like fully secure and deterministic. And so we never want to be in a place where the agents are doing things that we're unaware of or it's capable of doing something that the system is this allowing are both the background agents and the interactive agents to both of them have access to these sandboxes exactly. Yes. So the only difference between the interactive agent and the background agent from an architecture standpoint is that interactive agents need to have way faster latency. And so there's a human behind it and we tend to kind of short-circuit a lot of the prompts that we're using as well as the trajectories. And a lot of the MCPs that these agents are using tend to have very fast latency requirements. Background agents go on long-running tasks, they can do whatever they want to do, they can retry, there's a lot of value that you can get. And it's almost like when you don't have this speed and latency constraint, you don't really have to build in a certain way. And that gives the agent a lot of freedom actually to get the most like maximal potential of his reasoning capabilities. What does it mean to short-circuit an agent trajectory? Is that like, hey, after I guess step four, we're going to force you to give an answer or something like that? Not quite. So when customers are kind of like looking to an agent to answer a question, so like to make things one create, a lot of our customers will use Cogen as chat GPT for your security data. Because we built this graph, it's super helpful, and in the enterprise, it used to take four hours or in some cases like multiple weeks to answer a really hard data question. And today the agent can do that in three or four minutes. But this three or four minutes has a lot of architectural decisions behind that, right? It's a lot of textosequal, a lot of like schema, I would say modeling, to make sure that the agents are able to understand exactly like kind of what the universe looks like and can come to an answer in a very fast fashion. When I say short-circuit, it's really more so around being right the first time and not being able to have the luxury of doing multiple retries, as well as ensuring that there's enough context before the agent is taking any action. So another way to think about this is our agents that are interactive have a like deeper planning phase. And if it doesn't pass any of the tests that we have in the planning phase, it will short-circuit its like activity. So we kind of like want to make sure that there's enough context before the agent moves on to the NICS execution stage. Doesn't a deeper planning phase mean that it will take longer to run? No, it does take longer to run. But I think how we've kind of like figured this out is the planning phase actually makes like one-shotting a lot easier and like one-shotting not just with like textosequal, but also with like writing code. And so it's almost like if you plan correctly, you're able to do it in one or two loops. And that basically means that the agent can return with like pretty high accuracy within like a couple minutes. It's the user affected by the planning phase like they'll wait for the minute if it plans and the 30 seconds for it to one shot or something like that. Yeah, this is a good question on UX. And you know, I think the users have like instant feedback as the agent is planning. The way we think about like product in general is explainability is a key aspect of how users kind of consume the outputs of our agents. And so everything is transparent to the user from the planning, the reasoning, as well as the tools as using. And what's helpful about that actually is one, they can catch us when we make mistakes and give us good feedback. And two, it's a lot more interactive and users can kind of steer the agents as they're kind of doing their work. So it's not like a, you know, it's almost like the concierge that goes away and like comes back with this like pristine thing. I feel like our flavor of interactive agents is a lot more interactive and iterative. You kind of get to see what's happening. And if you think there's like some problem like, yeah, feel free to interrupt the agent and tell you what to do. So how do you surface that? Is it in like a typical chat interface? Like I would expect with chat GPT or is it different? Yeah, so it is in a typical chat interface. And the primary reason why is because a lot of our customers are like kind of enterprise. And we wanted to basically have a UX that's very familiar to them. And I think it's been very helpful actually like for folks who've like starting to onboard into much more agentic user experience. And I think what's also cool about the like, I guess UI that we have is the agents actually capable of creating agentic finance. So creating charts, right? Like kind of like building little like building blocks that help the user with like different tasks. And those are deliters. I feel like for a lot of users who have never seen this before. But ultimately also, I feel like humans are very visual. And text may not always be the best way. And so like having the capability to show the user exactly what they're looking for is very helpful. Interesting. How many, how often does it do that? How often will it generate something that's not text? Pretty often actually. So reporting is one of the, I would say like top use cases for any security analyst. In any given day, like at least, you know, a couple minutes through like hours is spent during reporting. And what you're doing is you're like pulling data, pulling metrics, you have somebody that has a slide that you know needs to see. And yeah, like I think the agent has basically effectively removed a need to like know how to do these things anymore. You said earlier that showing kind of like the trajectory of what the agent's doing is a good way to give visibility and show users what's going on. Do you think that as people get more comfortable with AI and trust things more that desire will go away and people will just want to see like a final response when it's ready or do you think that will always be a part? You know, as like engineers, we've all done the like, oh, like after the first time of testing like a cloud code or like a codex kind of like harness, you're just doing whatever it tells you to do. And at the same time, we are like our customers are security people who are really, really skeptical of everything, right? And like definitely a lot of AI kind of like skeptics in the security world. I think it's like very human to start trusting something once you build a workflow around it. And I actually think it's more so on the tool providers to have the responsibility of ensuring that AI is safe. And the reason for that is I feel like the behaviors that we're seeing in a lot of our customers is that people trust the agent almost too much. And that happens after the agent performs really well for a lot of tasks. But they sometimes forget that agents have tasks that they've never seen before. And for tasks that they've never seen before, they were likely not going to do well. And there's almost like a feedback loop that needs to be part of your mindset as an agent manager. And I think that's just like really important. And it will take some time for us to get there where this now becomes the new way of how people think about using these tools. Talking about kind of like being an agent manager and you mentioned some kind of like background tasks that happen to platform. If stuff is triggered in the background and runs in the background, how is it surfaced to the user? How does the user know what's happening outside of the chats that it's like kicked off? Sometimes the way I describe gogen is like if we're doing our job, like you don't need to come into the platform. And we effectively let the security team be on autopilot, right? The agents are figuring out what your risk is. They're prioritizing your tickets. They're doing your remediation for you. And you're really logging into like manager agent and like report on what happened. However, there's always going to be drifts, right? So in our production system, there's a lot of monitoring around drift both at the e-veh level for like models and prompts, but also for behaviors, right? So we talked about the data science agent, right? So when a new type of context enters the environment that we've never seen before, it's a high chance that there's no context on how to deal with this type of data. And a perfect example might be big company goes through M&A, right? All of a sudden there's a new set of assets that are now sort of like showing up in the environment. So what happens? We have to go tell the user, hey, Mr. Manager, I noticed that there's this new thing and you have not told me what I'm supposed to do given this like here's my recommendation on what needs to happen. Like what do you think? And so there's actually this like feedback loop that is really important in any kind of context layer, which is the entry point, right? I actually think that entry points are more important in context layers. What do you mean by entry point? Exactly. Entry point is where the user is able to provide context, right? And that can be through the product, it can be through Slack or Teams, that can be through a ticket where you're responding to like a bot, it can be through anything. But the more entry points you have in your context layer, the more live and real the data is, and the more real time you're able to kind of surface the right things to your end users. Completely agree with that. I mean, we build Langsmith, which is like observability for things in a big part of what we emphasize, is just getting more feedback in there. So you can slice and dice all your traces and look at things in particular ways. And we're not doing anything yet intelligent around it with like a context graph or anything like that. But I can totally see how that feedback helps kind of like, yeah, build this knowledge that you use. How do you guys try to do that to the max? Like what like do you send a bunch of Slack messages? Like what advice would you have for others as they think about, hey, I've got this data. I've got these agents interacting with the data. But yeah, the more feedback, the better. How do you get more feedback? Yeah, that's like, I think a super challenging chat like problem. And I think the way to think about feedback is you need to have a product that creates value to the user. And your product needs to have enough engagement for the user to see value to give feedback. I think that's like pre-req, right? Like independent of like any technology choices. And with that as kind of the main, I would say like pre-req here, I think it's really important to have an agent that lives where the work is. And that's like a product principle that we have. So a lot of SaaS tools, and I've built a lot of SaaS tools before in the past, kind of living this world where you come to our tool. We lock you in. You have to log in to this thing. They're like workflows that you need to learn in order to do your job. I think in order for you to maximize for a context capture, you need to one, like be where the work is happening, whether that's like our teams or in some ticketing system, or in some other cases, it could even be like offline, right? Like there are ways to capture that delivery. I think two, you need to build what we're calling supply side agents. So this is basically the context forms that I was talking about, right? So you don't really need to have access to the user. What you need access to is the systems of context as well as decisions are being made throughout the organization. And with enough agents that can do enough long running tasks, you can basically figure out, oh, like here are the five. take us, it will close in the last day. Here are the people that comment on it. Here are the decisions that were made as a result of closing this ticket. And capturing that in the way that is almost like a nervous system of like how this organization works is highly valuable. So those are probably the two things that I would think about. You mentioned that both of the agents have access to a sandbox and it sounds like for some they're writing at least some Python code or some code to manipulate data in some way. How similar are these agents under the hood to coding agents? So I think the coding capabilities are probably similar in the regard that we use models that are coding models for coding tasks. But the harness of what needs to be true, the tool calls are you're making the title data that you're dealing with is very different. So for coding tasks, what's actually really important is understanding almost like next token prediction for the next line of code. And that type of task is very different from an analysis task where you're looking through tons of data and you need to understand, oh, like what is correctness in the kind of like universe of schemas in an organization? It's also very different from actually executing a production workflow to validate a patch. And that often requires you to understand the deployment path. It requires you to understand the kind of like dependencies between different systems. And so I would say it's actually quite different from like coding agents. How does that manifest in the harness itself? Do you guys use skills that are like popular in coding agents? Do you have, I think a lot of coding agents have access to a file system and bash? Do you do you give your agents access to file system and bash? Do they have other other tools that are even more critical than those for doing work? Yeah, so I think from the structural standpoint, our agents are very similar and they use file systems. They have bash. They have skills. And I think the file system and text as kind of like a way to pass context around has emerged as like a very efficient way to do what's called these days agentic rag, right? Like being able to retrieve all in a file system. And I think we do benefit from a lot of that. But there are aspects that are really challenging to do within that context window or file system. And this pertains to things that have really huge amounts of data, right? So you can't store like one terabyte of data in your file system and have it figure out, okay, like how do I build a in-memory graph and figure out what's going on? So I think for things like that, you still need some kind of structure. And that's where like having like databases, indexes, things like that are super helpful. And then you just give the agent tools to read, right? Make queries to it. Exactly. One interesting thing about what we do, which I think is kind of like novel is we actually have a lot of really interesting tasks that require like sub agents. And the sub agent problem is super interesting because you have an agent. It's easy to orchestrate when you have multiple agents that can create agents. This like orchestration loop kind of fails, right? Like how do you pass the context? Like how do I know like if it's correctly instantiating the right sub agent? And the reason why you need like a sub agent actually is like primarily for two reasons. One is for parallelization. And then two, the other one actually is for some kind of like sub task that is not in the critical path of your primary task. And this is actually like a really interesting way to think about how like agent activities can formulate, which is just like a human, right? Like you're doing your main task, but as you're doing your main task, you actually have a lot of like side tasks, kind of like side tasks that help with doing your main tasks. And oftentimes you don't know in that trajectory yet, whether or not the side task will help your main task. And I think this kind of capability to spin off different thoughts or like almost like side quests is super helpful for agents because it's one, not coupled in the same runtime event loop. And two, it actually creates more context. And the agent is able to basically figure out if any sort of that context away. How does that work exactly? Like if you've got the main agent running it spins up a sub agent to some work, will that work in like the same sandbox with the same context as the main agent or does it get a copy and then like does the main agent wait for it to finish or is it running kind of like, are they running in parallel? And if so, how does the sub agent communicate back to the main agent? Yeah, so we kind of leave it up to like the use case. So there's not like one way in which this is supposed to work. And I think what we found anecdotally and empirically is you only really need the wait for the sub agent to come back if you're doing a sequential task. And if you're really doing a sequential task, I'm not sure if you should be using a sub agent. But you know, there could be like interesting elements where let's talk about like a long running task within agents form. Right. So there are many different ways you could do this. You could have like multiple agents spin up with one single orchestrator. You might have a route loop where you're just doing a bunch of things. You might also have within each agent that's running in that form spinning off its own things. And I think it's more so about like decomposing the tasks. So, tasks that can be decomposed are well suited for parallelization. And that's where like I would recommend actually using sub agents. But things that require like agent agent communication, where it's not just like singular process, right? I think that's where you probably don't want to use a sub agent. You probably need more orchestration at like the outer level. What models do you guys find work best or do you use a whole family or or mixture of models? So we definitely use a mixture of models. And the way we kind of think about this is we're agnostic to the model provider. We just run benchmarks and like figure out kind of what models are better now compared to like, you know, yesterday because things move very quickly in the space. What I will say like for a long time, we are big fans of like Anthropic. They've had really good coding models. But we've also I think seeing a lot of really like good success recently with a lot of these like models that been more trained on the cyber side. So with like open AI is trust access for cyber and like there's like some of these things where there are like really interesting ways in which they've pre-trained the model for like cyber tasks. And it's been really interesting to see how like the eVALs pair out and like how they're more capable, less capable and like where the calibration should be for some of these newer models. What is your eVAL system look like? So our eVAL system is kind of like a two tiered system. You have the offline harness where you know it's like multi-dimensional, I guess like eVAL metrics. And you have an online kind of harness where it's evelling our product as it's being used. And we feed those scores back into like this like I guess like global score and it's kind of like helping us figure out if there's live drift in the models and the prompts. I think what's interesting about like some of these dimensions is there's obviously like some of the obvious ones right like correctness usefulness things like that. But we also have some like kind of subtle eVAL metrics. So an example of this is like we have this metric called feel safe recover. And basically what it is is did the model basically feel at doing a simple task and they did try to recover and was it successful at recovering. And high frequency of this metric basically correlates with us messing up the prompt right basically it's helping you differentiate between a is this a really hard problem versus they reintroduce something that's like causing the agent to trip up for the offline tests that you're talking about how many of them are like unit tests so so to speak of like a single alarm call versus like you guys have your own version of terminal bench or something like end to end long long horizon. Yeah so definitely a spectrum of the simple like tests all the way through to like more like long horizon almost like process oriented testing. And I think the eVALs are there for different reasons. So on the simpler side you have golden labels, golden data sets and we're doing a lot of like outcome eVALs. This is especially important for the interactive agents I was talking about especially if there's a Q&A involved you need to make sure that the agents are returning things that are like well defined. And the longer term or like more long running side of things is where we're really looking at the traces right like understanding tool sequence understanding trajectories understanding if there is a faster way for the agent to converge and so like all of those are really important. And when you look at those trajectories are you looking at those manually or are you do or you now feeding them into coding agents and asking them to do that or do you have eVAL metrics themselves that look at those traces and score them on you know this is five out of 10 in terms of like effectiveness or something like that. Yeah so we have I would say pretty big fleet of LLMs as judges and they are a big part of how we think about eVAL harness and it's basically like different capabilities of LLMs as judge. So at the most simple level it's just at ever serial LLM and then at the most complex level it's LLMs that have full environmental parity with the actual agent that we're eVALing and so it's able to fully basically replicate or like reproduce some of the outcomes that it saw in the trajectory and ever serially assess if it was correct or wrong. Interesting. So like agent as a judge almost exactly. For the online eVALs that you guys do do run those over 100% of online traffic do you do sample down to 5 10% yeah so we definitely sample down I think for online traffic it's not super helpful in general to like run it on everything and I think this comes from like some past life where we're doing a lot of more classical machine learning. For online I feel like the key is like making sure that you have represented distribution and making sure that these eVALs are actually reflected back into some kind of feedback loop and like the whole point of like online eVAL is understanding if if you need to call an incident, right? Like in some ways, it's like a symptom for like an actual problem that's like degrading in your like online system. And so like that is where we wanna spend most of our time. And a lot of the efforts of, you know, making sure that the system works really well is kind of like offloaded to offline e-vails. I completely agree with what you said around kind of like using online e-vails to kind of like help spend this kind of like feedback loop. What does that look like in practice for you guys? - Yeah, so what that could look like is we have different products that are agentic and the agents are producing outputs as they're like one, either doing background task or two, interacting with a user. And those traces basically get fed through like some e-vail system. There's this kind of like a harness that's like running on the different, I guess like metrics that we care about. And when we see some kind of regression or some kind of like a, I guess threshold that drops, we basically want the one be notified. And then two have an agent look at it and say, "Oh, like what's happening here?" Right, and a good example of like how you might do this is through some kind of clustering to figure out if there's like a more deep-seater problem. And sometimes it's, we need to go talk to the user, right? Like there could be like instances where the agent is not misbehaving. It's just operating under a situation where it's never seen the context before. And so naturally the e-vails are not kind of up to speed there. For a lot of the long-running tasks that you talked about earlier, it seems like a key part of it is kind of like learning and adapting over time. I feel like one of the things that has been really hot recently is memory or continual learning in various forms. What do you think this looks like in the limit for a really good, long horizon learning a genetic system? Yeah, I think in the limit is like the key which here. I think we're very early days in terms of memory research as well as like what people are learning kind of in the field. What we are doing, which I think is also kind of interesting, is we're differentiating between hot and cold context. And what that actually looks like for us is how context is kind of like your session context, right? You're doing combation, you're generating change logs. There's like all this metadata. Cold context is basically almost like archived context. You're building a book with like a contents page of all the things that you've seen before in the past. And I think what's kind of interesting is that we are also plugging that into our knowledge graph. I was going to ask, yeah, so hot and cold, is that the same as like short term and long term memory kind of like hot is short term within the thread and then cold is long term between threads across threads? So I think it's like similar but maybe not quite the same. So I think when people talk about like short and long term memory, it's more so is this context that I will use right now and like throw away long term memory, something that you just like store and you have access to. And kind of the reason why we use hot and cold versus short and long is there are a lot of hot contexts that stay in the context for a long time. And for good reason, right? So an example of this is in our context engine, we actually have a much more higher level context graph that's not our knowledge graph. So the knowledge graph that we have is like a high scale property graph that's like really good for like running queries across like huge amounts of data. You can do multi-hop traversals and things like that. But in the context engine where the agents are like operating kind of in their like memory, they're building these like little graphs that can stay hot for a very long time. And that can be across a long running task, that can be across like handoffs to like other agents. And I think the distinction there is not that this context has like existed for a short amount of time. So it's almost like active versus inactive. I mean, maybe short terms kind of like the wrong, but it's like session level where session is like an ongoing thread. And that could be like, yeah, it could be it could be going on for a day. It could be going on for a week like it, but it's a continuous thread. And when you start talking about like sub agents and sharing contexts across says that's becomes maybe a little bit overlapping because you could have some piece of context that's shared between two threads, but like it still could be hot as opposed to cold. What is the difference between the cold storage and the context graph that you guys have? Yeah, so I think that the key there is like we want to differentiate between organizational context and agentic task context. So organizational context is context about the system, context about decisions. And that goes through its own agentic ETL. That's very high scale, right? There's like all these events propagating. And the key of that system is to maintain a source of truth and to have agents that can basically color the graph with high confidence inference that does enrichment of almost like the state of the world. And then you have these agents that operate on top of this context layer, right? And these agents themselves have kind of like a context harness or like a memory harness. And that's really like the world of like making sure that agents have access to the things based on the tasks that they need. So maybe the way you think about this is the hot and cold context that agents have are task dependent versus the context graph that we're building is kind of like a much more rich version of what the organization could look like, as well as kind of like a brain almost of all the things that are happening in the organization. What does the format of the cold context end up looking like? You mentioned kind of like a table of contents for Booker Something. Is it, and I think memories like super early on in general, but I've seen some version of this where it is basically just a bunch of files on a file system. And that could include skills actually. And an agents dot MD files, which are kind of like instructions, but but skills for a bunch of different things that the agents learn over time. And yeah, it is it is literally just like a table of contents for Booker. You've got all these different skills. Is that similar to like what is the data structure you guys use for this cold term? Yeah, so we kind of have like two different, I guess, types of data structures here. One is something that makes it easier to like do in memory graph traversal. So it's like an actual data structure that lives in memory. And then two, it's basically text files and text files that you can also plug into things like SQLite or like some kind of like, you know, indexable database. And the reason why is like the cold context as it accumulates can get pretty big. And there you may not want to like the value of archiving cold context may not be very high. If it's very well constrained to the type of problem they were working on. So you always almost want to have like context on processions and like a lot of this like metadata that could or cannot be useful. How do you guys think about accumulating the cold context? Does it happen? Like does the agent write to it directly as it's running? Or is there some background process that accumulates? It's a background process. And so like the the consideration here is not is there more value in like generating more context like as the agents doing its work? I think the consideration here is one when agents are doing work we want to make sure that they're like very focused. And two, there's actually a lot of like almost additional context that needs to go into how code context is being curated. That has nothing to do with the actual task that the agents are doing. We experiment with memory in a few different ways. And I think the first way we added it was basically just treating memory as like files on a file system and letting the agent read and write those as it kind of goes along. But to your point, reading is actually kind of easy for it to grow. But writing and especially like, okay, like where should this fact go? And like do I need to reconcile it with other things? That quickly starts to yeah, add other instructions into the prompt or become just too long for it to do as it's running. And so I do think background you see kind of like anthropic release dreaming as part of cloud managed agents. Letta has sleep time compute. I think this idea of like running processes in the background to store context will become more and more popular. I do think it shares a lot of similar lyridys to what you're doing with the context graph that you guys already have. Like you have these processes that seem to be acting in the background and using LLMs to take raw data and distill that into the context graph. And so I don't know if you think about the processes for that versus the cold as like in the limit. Are they just the same processes and they'll kind of converge in the techniques and the agents or are there things that are uniquely different about them? So I think the pattern of basically distilling data over time in a background process is probably the same. But the properties of the task at hand is probably quite different. So the scale of data actually matters a lot. And how much time you have to run these things also matters. But you know besides that I do feel like given enough compute and like you know enough data you can probably go pretty far. And who knows maybe there's a world where agents can do this really fast. Talking about kind of like compute. I mean it sounds like you guys are running a ton of agents. How do you guys think about pricing? Do you charge? Do you directly kind of like do you let people bring their own API keys? Do you charge directly for tokens that you guys use? Do you have a higher level kind of like unit that you charge for? Yeah, this is like a pretty contentious topic. So we charge based on number of assets. And this is like industry standard for cybersecurity for volume management. And the reason why is that number of assets that you own is kind of analogous to the attack surface. You kind of pay for what you're being protected on. And the reason why we charging this way is because customers understand it. And it's really familiar. There's like it doesn't open up questions around like, oh, like why are you charging in this way? We I think have had a lot of conversations and thoughts around like, you know, token based pricing, right outcome based pricing. There's like definitely a lot of thoughts in that space. And I think where we are today as a company and also as a product is we care more about customer usage. I think the nice thing about like us is also that way in a space where customers are willing to pay for the product independent of like usage kind of like you're paying for these autonomous agents to protect your organization. And so we don't really have that problem where we need to figure out how to aggressively price our customers yet. You recently launched Cogent Research. What's the thesis behind that? Yeah, so I'm been really excited about Cogent Research. I think it's one of the highlights in the last couple of weeks. The interesting thing about Cogent Research is, I think we're at this bottleneck for security problems where it's no longer a product problem. It's really a research problem. The reason why I say this is a lot of what we do is verification. In order for me to deploy something into a customer's production environment, I need to provably show that this will not cause any issues in their environment. That really leads us into this frontier of open problems in the intersection of AI and security. The charter of Cogent Research is really about taking some of these techniques that we know of, formal verification, building these provable systems, and marrying them with a lot of AI techniques, like RL, open models that we're going to open for us at some point, building these security benchmarks that help the community think through what it actually means to do defensive security at scale. I think ultimately that research will allow us to lead the way people are thinking about the problem. One really exciting thing is our research labs actually headed by two X Google DeepMind researchers. I think one of them is the X8 primary author of Google's CodeMender, which is their version of Cloud Code Security. The reason why it's really important to do this in an applied AI lab versus just a big tech company or an AI lab is precisely because we're well positioned with actual enterprise data. I think the deep amount of customer access that allows us to do the interesting research that helps answer questions around, "Oh, how would I autonomously do I am? How would I autonomously have privacy preserving policy enforcement?" There are some interesting questions that you can only answer outside of a research setting. Almost I think the vision of Cogent Research is basically applying a lot of these learnings from what we're seeing in the field towards a lot of our research problems and thinking through what the newer version of autonomous security should look like. How big is the team already and how big do you want to grow it? Yeah, so the team right now in total is about 30 plus people and we're actively looking to grow. We have offices in SF and New York. I think we're at a very exciting time where the timing is really good for both building an AI but also thinking about autonomy. I personally think that being an applied AI lab is one of the best places where you can one, grow a ton and two work on really interesting things. I think we have really ambitious plans to grow the team this year and we're looking to hire. What is the right profile for you guys or for an applied AI lab in general? What is the right profile of builder? Yeah, it's such an interesting question because I think the matter question is like in an age of AI, what is the right profile of a person who will be successful? I think Cogent is one of those companies where there's so much breadth of things that you could do where it doesn't really matter. We have huge data and infrared platforms to run the high-scale data analysis that we're doing and processing. We have, we've got our own in-house agent development platform for our agents. We want to do post-training, there's all this stuff, we have security research, there's all these things going on and of course this product engineering. I think the number one kind of property that I see a lot of people tend to succeed in in today's world is actually having high agency. I think what that means is being willing to figure out what's going on and use the tools even if they're in a nascent state. I think creativity and taste are also really important but you do need to have the right fundamentals as it pertains to the type of tasks you're working on. An example of that is, Echogent, we like to think of ourselves as having a high bar and what that means is we really try to stress test the way people think about problems and that transcends just knowing how to write code. It's kind of like are you able to reason about problem from first principles? You have the right frameworks to think through what that means and then with that as the foundation, is that like a 10X or is that a 100X improvement with AI coding? Do you have any tips on how to interview for agency? Yeah, it's really challenging, I guess, I think the assess. I think one thing that I look for is actually the way people have behaved in the past. Agency to me is actually a trade and people leave marks throughout their careers and having someone tell a story about this is kind of the best ways to kind of dive deep. I think the way to maybe like interview for it is to go like maybe five or six levels deep. Really question the motivations, the reasoning, what was the circumstance, why was it hard, what decision did they make and why was it actually something that was really difficult in that moment and what did they learn from that? I think the feedback loop of both learning from an experience as well as like delivering on the experience are both equally important. You mentioned open source models and post training or you mentioned post training in RL a few times. Are you guys actively using open source models? Have you experimented with them? How do you think about their place in your platform? Yeah, so I think open source models are great and I think the way I wish to open source models are improving is actually been really, really impressive. The way to think about open source models is it's really just a better way or like a cheaper way for us to do some of the experimentation that we want to do. I think over time as the open source models become a lot better, probably they will be a staple in every kind of like applied AI setting. And for us, actually a lot of our tasks don't necessarily require a lot of reasoning. So there's like almost no reason why we would not be using an open source model. I feel like there's two reasons that people use open source models sometimes. One is they argue basically I can RL it, I can post training and it's going to be better for this task. The other one is it's just going to be way cheaper. I don't need the reasoning for you guys. I'm assuming it's a blend of both the summer right, but is it like 90% cost and 10% performance or is it 90% performance or 10% cost? How do you think about it? Yeah, so I know we're less conservative costs and for that reason, I think the fun to your models have been like really good for us just like in general, thinking through like kind of you know, you go to sleep and then you get model performance improvements, that's like pretty good and pretty sweet. I do think that there are some like really interesting points of leverage. So for us, it's not so much about what the models were pre trained on. It's what interesting context we can post train on and working in the enterprise context. It's really these highly complex environments that need to be tailored, right? So the giving example, some of our customers have 10 ticketing systems, right? Like it's just like impossible. Like how do I know what changed control process to use? Who should I route it to and like where that context should live? And these things trip up the agents all the time. I think from like a post training standpoint, there's actually a lot of alpha for companies that work in this space where the nature of the problem is just like highly complex that the models have never seen this data before, right? And so like there's almost like a lot of calibration that we have to do in the day to day that hurts the e-velle, right? So even for a lot of long running tasks, the like best in class frontier models are not necessarily performing at percentages that we need them to be. And I think the way we kind of like meet that golf is through post training. If you had to estimate, when do you think a key driver of the Cotient Platform will be post trained open source model? I think within six months. Yeah. And I think the reasoning is this, right? Like the reasoning is already there and it's more so a matter of having data. And so the question is flipped. It's not about model capabilities. It's about context. As a company, do you have enough context to do the post training that you need to solve your problem? And I feel like we're there already having been around for like, I guess year and a half. Beyonds in the AI era. Yes, exactly. And I think that's a luxury for us, but I'm super excited to see like kind of what ends up happening after we go on this path. I feel like there's been a lot of buzz around quad mythos and other frontier models that are getting really good at security related things. How much of this is hype? How much is this is true? Where do you think it feels like there is? I mean, even in the open source space, we get a ton of security issues filed against Langtrain thinks some valid, some not valid. There's this explosion of security related code generation code work being done. How do you see the space right now and how do you just see it evolving over the next few months? Yeah, it's a really good question. And I think what mythos is really done for the industry is basically highlighting the extent of model capabilities around offensive security. But the reality is pre-methos like Opus 4.6 was pretty good. And in fact, like Opus 4.6, I think was capable of a lot of the exploits they had reported essentially in the mythos report. And we had known about this for a long time, right? I think part of the thesis of Cogen is that attackers only need to be right once and defend us right all the time. And the kind of compression of the almost like window of exposure will mean that you need to have these autonomous cyber defense systems. And so answering your question of like what this means for the industry and kind of like how much of it is hype is definitely very real. And one interesting thing that I've starting to observe is all the coding companies, including the AI labs, cursor, Versel, anybody in the dev space, they're all coming up with essentially security tools. They all are, it feels like they're all, the first day launched a coding agent, then they launched a review agent. Now it's now in the last week, it's all security agents. Exactly. And to me, this is basically SAS 2.0. So like five, six years ago, we had this thing called SAS. It's basically like fuzzy matching for like code scanning. And this was like the cool end thing, like to basically figure out if you had first party code vulnerabilities. And in some ways, I think discovery is like really easy. How to fix this, like, basically it's still the bottleneck and like super challenging. I think this will be not good for the industry because there's a lot of vulnerabilities you can't find using existing techniques that the LLMs will help to catch. And the real value actually is, I kind of think about it in like two ways. There's two simulans of agentic capabilities. There is the dev tool chain. And then there's the operational tool chain. Most applied AI companies like us operate an operational tool chain where we're automating everything outside of code. The AI labs, these like IDEs, like all these companies that kind of like work around the dev tool chain will have coding agents. They'll do all the things. And I think where it's going to benefit a lot of folks is in the shift left movement. So if you have a skill and you're writing some code, the agent will be able to help you figure out how to write better code. I think that's great, right? That will help you have better code that's like secured by default, secured by design. That's going to be amazing. But who's going to go curate the contacts that feeds into that agent is still like an open question? And what happens when you have vulnerabilities that need to be fixed? So I think there's a lot of operational things that still need to be figured out. And the way I think the industry is going to move is the AI labs and probably a lot of these kind of companies are in like the dev space will try to commoditize a lot of skills or any kind of like coding agent based skill, right? Like it's not just security. It's going to be Texas, right? The other day, I had a friend who was basically filing his Texas using ClawD and it worked really well. So I think like a lot of that's going to be true. But you will still have use cases where things are very operational and challenging. And you need to have correctness guarantees that you can't get with like a coding agent. It seems like security is top of mind for everyone these days. But you guys started a year and a half ago. How did you guys get the idea for Cogin? So I was part of the founding team of this company called Abnormal Security and eventually became like leading their product engineering team. And we were basically building one of the world's best machine learning platforms for FOT detection. And today Abnormal basically protects more than half of the Fortune 500. And the problem that we were solving, I thought was actually very foundational the way I think about just AI systems in general, but also how to build agents. And that really planted the seed for Cogin. So just for some quick context here. So Abnormal is like an email detection platform. And the really interesting thing about finding malicious emails in an enterprise is that the likelihood of finding and malicious email is actually really, really small. It's like less than one in 100 million. So you need really high precision. And on top of that, you need really, really high recall as well because what happens when I take an email out of your inbox and it turns out it's from your boss. I've disrupted your business. And even worse, I could leave it in there. Like you click on it, spearfishing email, and your organization gets breached. So we were basically building the system processing trillions of emails every single day. With near zero, I would say false positives and false negatives. And the key to solving that entire problem was basically context. It was not really the models. And I think that insight really kind of feeds into how we think about Cogin or building here. And when you're building behavioral classical machine learning models, what you're doing is a normally detection. But what you're trying to understand is the organizational behavior and the relationships between the people, not the actual email signature. So what we ended up doing actually was spending a lot of our time investing in and building understanding of how people communicated. And that became, I think, the way we thought about how the way you understand if your CEO is getting spoofed, even though the email looks exactly identical compared to what your actual CEO would send to you, is precisely because you understand some of these patterns. And in some ways, what we're doing at Cogin is just solving that exact same problem, but with a much larger scale on the much more important problem. And so I had a normal idea. I imagine a lot of those manual feature engineering to get these signals that would go into the traditional machine learning models. How much of what you guys do now do you think is manual context engineering versus letting the agent build its own context? Yeah, I would say it's half and half. And so we have a four-deploy engineering team. And the reason why we have four-deploy engineering is because AI deployment is not just about data and context. It's also about people and organizations. And there's a lot of battles that you have to fight to help people understand. Oh, we need this thing. Here's how it's going to be valuable for you. And a lot of the tuning oftentimes that makes our agents so powerful and used to happen through the deeper understanding of the customer's environment. And that's where having our FDA is actually a superpower, because they can dive into a customer's environment, understand what's going on, feed that back into the agent, without the customer's needing to do a single thing. And on the flip side, we use agents in the day to day of everything that we do. And feeding that back into our product into the systems that we use is a core part of our DNA. And we're getting a lot of value on that just by itself. But I think there are still limitations to how the agent feedback loop is being structured. And the core learning, having only built this company for a less than a year and a half, and the landscape, having like kind of rock pull this like every six months, is that your architecture is always going to change. And your AI hardness is always going to change. And the models are always going to keep improving. So I think we kind of graduated from a world where there's a lot of spectraven development, and you kind of like teaching the AI through skills, into a world where you need to do a lot of invariant testing, where people are talking about like autonomous software development. We do autonomous workflow development for our customers. And I think in that world, you really need to invest more in re-architecture, as well as making sure that you understand what to test versus implementation speed. And so I think in that world, it's actually more important for us to go back to the fundamentals of like, what is this system trying to do? Like are we actually trying to solve the correct problem here? And that's definitely I think something that we're really excited about, both from a research standpoint, but also from like a product standpoint. In the short medium and long term, how do you guys think about positioning cogent against the model labs themselves? The way we kind of think about this at cogent is in the short and medium term, probably there is a lot of work to be done. And so in the short term, I think we're completely complimentary. And what that means is like a lot of these AI labs are launching security tools. They are just other sources in the cogent. And the problem that we're solving actually is a context problem for a lot of our customers that I think the AI labs will not work on. And there's a lot of like nitty-gritty challenges to building integrations and like making sure that things work correctly in like a highly complex enterprise. But on top of that, I also think that there are like highly customized and like domain-specific agents and like post-training that you have to do to get some of these problems truly correct. And let me give you a perfect example, right? So a lot of people in the coding agent space not just want to augment how coding agents are able to increase the productivity of engineers. They want to reach to 100x kind of outcome of running the mythical cobalt-er-rust kind of migration for the big bank. And yeah, that probably will be like a eight to nine figure kind of like contract if you ever get it done. And I think in that space, it's really interesting because like if you think about it, what needs to be true for a coding agent to be able to do something like that? Why didn't model capabilities need to be there to long running tasks need to be commoditized? Three, you need to have a lot of context on what you can and cannot do. And then four, you actually need to have very strong synthesis capabilities to reach basically parity of the existing system and build the right guardrails almost like integration tests, right? For the invariants that need to be true. And so out of all of those things, what is the thing that is customer-specific? It's really like context, right? And I think that will need to be true for any kind of, I guess, like coding agent that wants to do that. And that's also generalizable for any domain. So for us in security, it's really doing true auto-mediation in production environments. And trusting the agents to basically be your last line of defense. And yeah, like when the next zero day drops, right? Like when like LLM or one of these other like very common tools has the next zero day, you want to be able to trust your agent to do something that you won't have time to do, lock something down, contain maybe deploy a Wath, like a firewall. Or in some cases, like just run the patch, right? Like I don't care. I need this thing to be safe and locked down. I need to decommission my asset for me. And so what it would take to actually do some of those things, I feel like is kind of in the, like I guess, like a hot seat of building the right context for a lot of companies. So in the short term, I think we're completely complimentary. We focus on fixing and like a lot of these AI labs are basically kind of data sources into what you would plug into cogent. In the medium term, I think everybody that's an applied AI company is building a world model for their customers. The fact that we're doing these things is just making up for the gap that these customers themselves do not yet have a context layer. So every applied AI company ends up building two things, the context layer and the intelligence layer. And it's just required because that's just how we are today. But tomorrow will like fast. for three to five years. Yeah, I'm pretty sure most companies will have something. And there's a lot of value in being the first provider for that context layer. I think it gives you a lot of moat in terms of first-movers advantage of use cases. And when you're talking about this context layer, just to be clear, this is the same as the context graph that we were talking about earlier that you have agents going build. Exactly. Another way to think about this is who is going to help companies understand how decisions get made in their organizations. And this needs to be a real time system. It needs to be able to change as events are happening. And this is the stuff that the agents feed off of. Like the high quality context that tells you how judgment should happen. So the reasoning of the agents can be met with almost like historical data. And it's really interesting because there's two ways to solve the problem. One is historical data through this context layer stuff. The other way to solve the problem is through prediction. And the way you do prediction, probably is some amount of like post-training, where you can build these hyper realistic, like, our environments, you're like training with one model, right? You're like simulating all the possible paths. I think both are very valid. And both probably will be in the hot path of any like applied AI lab. And I think where this gets really interesting is like even in the world where the context layer is commoditized and you're just plugging in to a context layer, the context that you generate as a result of your intelligence layer is highly valuable. And that in and of itself actually would be, like, I think the key to a lot of the workflows that organizations need to automate. Do you ever get asks from your customers to make this context graph, this context layer that you guys are creating more open or use it outside of your intelligence layer? Yeah, certainly. I think once customers see value, like the first question is like, can I use it? And this happens a lot. A lot of customers are asking us for like API access to like MCPs and things like that. I think it's also like different maturities of customers. So the ones that tend to be more tech-forward will tend to have these asks versus the ones that tend not to be want to have UI. They want to like live in like familiar interfaces. What do you tell those customers? Do you let them have access to that? So I think the pencil in the use case, we tend to want to basically go on design partnerships of customers that are a bit more forward thinking. But we set expectations around how the product works and like kind of what this could lead to for both companies. And I think I did on the day the way we kind of frame this is like, that's just another way for us to get context. Do you think the models themselves will commoditize the security landscape? Yeah, this is like really good question because in some ways, GPD55 cyber as well as cloud meet those have kind of proven that the models have commoditized offensive security, right? Like today, any script kitty, any person on the street, you and I, we could have access to cloud code or codex and we can compromise our neighbors down the street or like down the office block. And I think what's interesting about that is that on the outside, it looks like, yeah, security is like already commoditized. I think if you understand the type of data offensive security is trained on, it actually helps you understand that like, I don't think security will actually be commoditized. So here we are, pen test things are very well defined industry, right? People get sort of five or these things, it's like well defined data sets and like it's a very well defined task. And so it's actually like generally straightforward for you to go pre-trained a model and get very high quality labeled data on how you should exploit something. And it's also a field that I think has seen a lot of resurgence in like recent years where people are kind of like curating these data sets. On the flip side, defensive security data is actually really, really hard to get and it's also very complex. So the nature of an exploit is here's like bunch of like data, let me figure out how to like get in, right? The nature of a defensive security task is I have a castle here are all my walls, how do I patch every single little hole in the wall? And every single castle is built differently, has you know, some are made of sand, some are made of rocks. They have different guards who have different, almost like schedules that you need to like maintain. There's so much complexity and customization for the defensive stack of every enterprise that a defensive security task is actually like really, really hard to one like model and then to create good label data for. And this is also the reason why there has not really been a definitive security benchmark for defensive tasks. So from that perspective, actually, I think security is one of probably the last bastions that would truly be commoditized because there really is no data. All right, thank you. That was fun. Awesome. Thank you so much. Thanks for listening to Max Agency. If you liked this episode, leave a review and subscribe. Send feedback or questions to Max Agency at Langchain.dev. We want to hear from you.