The AI Control Loop: When AI Goes Rogue - with Craig Thomas of Wallarm

Hello listeners. Today we are dropping another episode in our series, The AI Control Loop. Our enterprises govern the AI they've already deployed, sponsored by our friends at Wallarm. Wallarm is the AI control platform for enterprise AI, protecting every AI workload, API, and application in production, giving CSOs the governance they need and CIOs the speed they demand. Friends choose Wallarm for a complete inventory of APIs, AI agents, and AI apps using patented AIML-based threat detection and blocking that operates at production speeds. In this episode, Craig Thomas, Senior Solutions Engineer at Wallarm, examines what rogue AI actually means in practice, where the risk materializes, and what it takes to move from detection to control. Craig, thank you for being on the show today. Thanks for being on Code Story. Yeah, great to be here. Appreciate it, Noah. Absolutely. Really excited to dive into our topic today around when AI goes rogue. Excited to dive into that. Excited to hear your perspective and bringing in all of your experience, including your experience with Wallarm. Before we jump into that topic, tell me a little bit about yourself. Tell me in my audience a little bit about you. Actually, my background, I spent over 10 years in Department of Energy in multiple roles from a network engineer all the way up to CIO, the nuclear weapons plants. Bring the experience from the customer side in a highly regulated environment. Then spent several years running a consulting company where working with those same highly regulated industries. When I joined Wallarm, they had worked with APIs, had worked with automation for quite a while, and really interested to how do we bring that security to the customers, right? Both from an API and an AI perspective is that that continues to evolve. And then outside of work, pretty hands on, like a tar, spend time in the yard, like to mess with my long quite a bit. A lot of my weekends revolve around kids' activities, horse shows and baseball and football. So, it keeps me pretty busy and always somewhere to be. I appreciate that overview. Let's dive into the meat of it. Again, our topic is when AI goes rogue. When we say rogue AI, what do we actually mean there? What do you guys actually mean there? Is it only malicious AI or can legitimate systems become risky too? Level setting where we're coming from too, right? AI is really interesting. The fact that it really does permeate all parts of the business from IT, cyber, compliance. Obviously, the business itself. It's adding value and accelerating work. But how do we better discover, observe, enforce and govern that and do it at the speed of AI? Right? And so, some of those are starting with rogue. And to your question, rogue doesn't necessarily mean malicious. And in a lot of times, it isn't malicious. Most common rogue AI scenarios we're seeing are similar to what we saw in the API world, right? They're not attacks. They're legitimate systems. But they're behaving outside. They're intended boundaries. So, their intent was fine, but the outcomes weren't from that perspective. But we kind of look at that and I'll look at that from three various categories. Unsanctioned AI, that AI that nobody approved it. It's running in your environment and it gets into shadow a bit. Unmonitored AI. So, we have AI that no one's watching at all and then unpredictable AI. So, that gets in a bit of the point to the rogue. It's approved and watched, but it's doing things no one anticipated whether those be from the business side or the security side. So, that you've got agents emerging, kind of, is that biggest emerging category because they're doing things on behalf of users and also rather than just a specific question to a chatbot and a specific answer. Now it's going and taking actions on that behalf and that's introducing even more rogue actions at scale. Absolutely. Now, that makes it clear when we think about rogue AI, doesn't necessarily have to be something malicious. It's going to be tough on its own and what are the most common ways that AI systems drift outside of intended boundaries? Once an organization understands what rogue AI looks like, where does that loss of control typically begin and who is responsible for preventing it? The operating definition I like to use is it's working as a design, but not as intended. And so, this goes back to computer science 101, right? It's, we're capturing a developer or a designer designs it one way, but then it's being used for other ways, whether maliciously by bad actors or just from the software responding, different than you expect. But one of the most common ways is prompt injection. It's an underrated drift mechanism, but malicious content in a data source is support ticket, scraped web page, user message, it can redirect an agent's behavior. No one touches the model, right? With drift events on an AI particularly, you've got various components and several of those are out of your control. Model updates may change the behavior without you changing a single thing or single code deploy. If you're calling a hosted LLL and the provider updates the model, the agents behavior can shift, no pull requests, no change management, no one reviewing the Delta. So back to the conversation of what broke and why you've got to go a layer deeper. And that's a big way that AI systems can drift because of from the user interaction or the agent interaction to an API, to a tool to an LLL. There's various components in there that they can all be changing and outside your general control. Let's move into the shadow world, right? And shadow AI is the unsanctioned or unmonitored use of AI tools, apps or embedded features by employees within an organization, basically usage of unapproved AI, right? In looking at shadow LLMs, how do these things create risk even when no attacker is involved? Right? Let's define that. And if AI drift often starts with normal business activity, where do shadow AI systems fit into that picture? I'd define shadow LLMs or that they are the new shadow IT, right? A developer spins up an LLM endpoint in AWS or Azure or somewhere else because the internal approval process takes weeks, if not months, for various things. And so typical behavior with shadow AI and shadow LLMs, right? It's because it takes too long. They have a legitimate use case, a business case, but it's just taking too long to get what they want done. And so there's no security review, no data handling policy, no logging. Or you bring in a new third party tool with its own AI built in, calling LLMs that you never approve. You need to be able to see that, surface that, and stop it, or at least provide feedback to your vendors, right? So that you understand what's going on there and you understand the full risk picture. But it's maybe a little bit outside of the scope of this conversation, but S-bombs, software bill materials and AI bill materials that are crucial to understand truly what makes up your software and understand all your risks. Those are pointing to them, maybe from your vendor, but the fact that you need to be able to understand and build those so you can understand the risk and stop that as well. And that asks, or kind of add one other thing, unsanctioned agents. They inherit the developer's credentials, right? So the developer goes in, just puts his credentials in, and now they get a broader role than they probably should have. So the agent, same person as me, or you, right, maybe in an administrative role, which is much more than it needed to actually do the task at hand. And so those are a couple of things from Shadow LM. You're really looking at one big stat that kind of comes out, 72% of corporate AI tools and active use are rated high or critical risk. Not future state problem, that's current state today. Still, we have to get that under control for sure. Yeah, and no doubt. And it's clear already from our conversation that this is important and it's a hard problem to solve. In an AI action, look legitimate in isolation, still create serious business security or compliance risk when viewed as part of a larger sequence of actions or a workflow. As these shadow systems become more embedded in everyday workflows, why is it so difficult to recognize this risk in real time? One, because our tools aren't keeping up right. Tools that are using signatures are specific. This is allowed, this isn't allowed. Just can't keep track when you're doing, changing actions together. In that case, individual actions pass every check, but the sequence could fail the audit. Maybe reading a customer record is fine. You keep summarizing it's fine, passing the summary to a third party API, all good, right? But doing all three automatically at scale without a human review step, it could be a conversation around GDPR or other things that you just don't want to have. It goes back to even different parts of data and isolation are fine, but once you chain those together, you can have a classified or other security events in the national security space and similar things are happening here. Then by next step, know as a chained agents, right? They multiply the blast radius because rather than just a single agent doing a single thing, agent A calls agent B, which calls agency. Each step looks good, authorized API calls, but the end result, there's a capability. No one really explicitly designed or approved. They just emerged from the chain and these agents learning what they can do and how to get what they want done. And so then a lot of times you're losing context. Indian session tracing really is the tech capability that makes this sequence level analysis possible. We can't believe you have to be able to trace every AI session from the initial prompt through every model call tool, identification, API response, because it's a single record because otherwise you're losing that context and you're not able to reconstruct that and under understand the true risk of what's going on to be able to eliminate or minimize risk and take action in a larger environment. That makes sense. And I think your answer probably opens up or leads into an answer to my, for my next question, how do APIs and integrations and connected systems amplify the impact of these seemingly legitimate actions? I think I could probably, as an engineer, cherry pick some of the things you're probably going to say from the last answer, but what changes once those actions begin flowing across these workflows, across APIs, applications and interconnected systems? Yeah, it goes back to, I know conversations you've had in previous weeks, right? APIs really are the central nervous system of AI, right? Every meaningful action or AI agent takes eventually, it becomes an API call. Well, that's to a database, to a SAS tool, the payment system, identity provider, MCB servers. You're not watching that API layer, or you're not watching the agent. And then that integration surface is enormous. It's unambatory. How do you protect what you don't know? Modern enterprise environments have thousands, if not tens of thousands of API endpoints. Many of those are undocumented. They might be third party, YAS, it might be your own APIs. And so it might be endpoints that should have been deprecated a long time ago. The issue now becomes even more because an AI agent with broad permissions, they'll find those, they'll call them, they'll be able to do what they need to do to get things done. You're no longer dealing with just users or even just scripts that are dealing with specific APIs, you're dealing with agents that multiply this and it's being able to integrate, AI security, API security, because of this amplification effect, becomes crucial. A lot of folks treat these as separate problems, API gateway for API traffic, AI gateway for AI traffic, separate tool for AI behavior, but AI agents interact with all these external systems to APIs. That's where data exfiltration risks, typically materialized, separating the two, creates a blind spot exactly where the risk is highest. And so you've got to be able to see that full picture. Right on. Okay, let's switch over to leadership of CIOs and CSOs. What kinds of unexpected outcomes worry these folks the most today? When AI systems are operating across these interconnected environments, and adds that connectivity expands, what are security and business leaders most concerned about? Great question. A couple things. One that jumps out to me right away is PI or sensitive data showing up where it shouldn't. Right, and these have real world consequences and real world fines. So an AI workflow process of support requests routes it through a summarization step, stores the output in the log and that log contains the customer's social security number. No one really designed for that data path, but it happened because the model was helpful. It got what it needed, but it now logged that information where it shouldn't have. And then I think another big thing is, yeah, I can't explain what it did problem. And so that can be from the CSOs hat or from be from the CIO or businesses hat. A CSO can generally reconstruct a human attacker's path, reconstructing the AI agents decision tree across a multi-step workflow with external data and puts. It's generally, it's just a hard thing to do, right? Regulators, board members, incident responders, they all want the story, but today a lot of organizations can't tell it. And any other kind of interesting story is unexpected answers or behaviors. We actually have an ongoing conversation with a potential customer. They built their own chatbot and going to change some details here to not reveal anything, but you go ask this chatbot a question, for instance. So what do I feed my new puppy, right? 99 times out of 100 or 9 or 9 times out of 1000 responds with a dispute for puppy food, a link you can buy it straight from their website. However, one time out of that, it actually responds completely randomly and not only randomly, but in a really bad way to potentially feed your puppy something. That's poisonous. So that can't happen. We have to understand what those responses are and make sure they're accurate 100% of the time because otherwise the company can be liable there. So those are a couple of the two big things know about unexpected outcomes, both from a cyber side as well as the business and liability side that jump out to me. And then kind of the third is just that reputational risk from outputs, not just access on AI system that publishes incorrect or biased content at scale or gives inappropriate recommendations of customers like I talked about. It's a brand and legal risk doesn't fit neatly into traditional security frameworks. I appreciate you walking through that. And given these concerns, what does meaningful oversight actually look like when AI systems can act at machine speed and I can we talk about interconnected systems and impact there. That can really amplify the effects of some of these actions. So what does that meaningful oversight look like from a security practitioner, your policy alone is not oversight. We've always said that right paperwork doesn't make you more secure policy document does not stop an agent mid call right. So real oversight fires actual runtime, enforce the ability to inspect block or rate limit that AI generated API traffic in real time. That closed loop model really to know see stop improve. You need to know what the AI systems are running and what they've connected to need to see what they're doing in real time. You need to be able to stop the behavior before the consequences compound. But increasingly you need to be able to prove to auditors that you did this right most organizations have no at least partially and they've got some logs but see stop improved definitely are the gaps. And then humans in loop do matter right but they need to be at the right point. You can't put a human in the loop for every API call you put humans and decision gates for high risk actions boundary crossing transactions. But then the system has to surface those moments automatically right so as you're looking at tooling and other things those key aspects need to bubble up so that so that humans can take action quickly. Understand the context around that and then the tool has to be able to enforce these policies like you said at machines feed that makes sense and enforcing the policies as quick as the actions are being done that seems clear that would be a meaningful that would be meaningful oversight. Okay how should organizations distinguish between the experimentation they want to encourage and the unmanaged AI behavior that they need to control one challenges balancing governance with innovation. And as an engineer as a startup founder myself that that holds true day to day how do organizations avoid slowing down AI adoption while still maintaining control security organization can't put the know in an innovation right we've heard that many times right. So the answer's not slow everything down organizations that over restrict AI experimentation one you'll lose your best engineers you lose your best people to competitors that don't as well as the business is asking all of these quarterly earnings calls and other things are asking specifically what the AI strategy is in plan so we have to be able to support that but do it in a secure way. So I think the first thing is hey define the sandbox clearly make it easy to use know if you can go out you can innovate you can test but do it truly with a dev environment isolated credentials scope API access no production data logging enabled so you can understand that you can experiment quickly and that sandbox needs to be a first class experience not just an afterthought and it needs to be usable right we've all dealt with dev environments that didn't mirror production at all and caused also. The issues went to prod being able to do that be able to support continuous a is bomb going forward here as well so that in those environments you can understand what all is in your AI what builds that out as well and then create a fast path to production right shadow AI grows when that official path is slow so your point is you can't slow it down people are going to get around it one way or the other right either they're going to do their own thing outside your boundaries or they're going to leave. So a lightweight AI system intake process the security room is data classification runtime monitoring the completes and days not months or years in some cases removes the incentive to go around it and then I think the last thing is you're tooling being able to easily enforce new new sandboxes or new environments that are stood up is important to right so rather than having to go from your security team or your developer is to the infrastructure people and all it be able to just click a button from some of these AI governance and observability tools to do the enforcement monitoring and enforcement in these new environments with a click of a button in seconds rather than like as weeks or months also is important so you get that visibility enforcement as you see new things spin up. Okay I'm tracking with that okay Craig there's one more question I've got and we know that many organizations can detect risky behavior that's a fact but if they can't stop it real time what critical gap still remains even with these governance programs in place many or is still operating reactively kind of close out our conversation what's the key difference between detecting a iris can actually control. As a security professional or is the sea so you don't want just another dashboard to tell you how bad you're doing right and detection without prevention is just documentation of damage right knowing and agent X filtered data after the fact doesn't on X filter it it doesn't keep you from finds another regulations. Yeah systems operating at machine speed the gap between that action taken and the action detected can mean thousands of API calls and gigabytes of data before. you're ever alerted to that. And also at machine speed looking at logs and some of your traditional seams just aren't able to keep up with AI speed actions. These metrics, the traditional metrics, they just look even catastrophic. So one minute response time is just way too slow of an agent made 50,000 API calls in that minute. And regulators even are moving toward real-time accountability, right? So you've got the EUAI Act, which is coming fast upon us. And this AI, RMAF, emerging SEC regulation, it's all signal that we detected eventually just won't be sufficient. So the expectation is shifting towards a demonstrated ability to prevent, not just discover. And so that's what definitely we are looking at. How do we close this gap? Most organizations have that seam, they have APM, they have API gateways, but not at least tools see inside a session and stitch all those pieces together. They just see headers, they see latency, they see logs, but they don't see the prompt, the payload, the data return. We're looking to be able to see all of that at the kernel level and act on it in line at machine speed. That just makes so much sense and thinking about reactive versus control and processing that in real time just really brings it to light. How important and how fast this moves. So Craig, I really appreciate you being on the show today. It's clear that rogue AI doesn't necessarily mean malicious AI actions can appear as legitimate business actions and are often difficult to determine. And the chaining of these events through interconnected systems, obviously and clearly through your answers amplifies the impact quickly. You have to bring the speed of governments to be as fast as the speed of action. A business must keep that balance between control and innovation, but to control AI risk businesses have to move to real time action to these issues in order to prevent the damage and to get out of that reactive mode. So I really appreciate you explaining all this, sharing your perspective and being on a show. Craig, thanks now I really enjoyed the conversation and I hope you have a great day. As you can see from Craig's answers, rogue AI is something to take seriously. Even if it's not malicious, AI is powerful and businesses need to take stock of how people are using this technology with a balance of ensuring open posture towards innovation. If you'd like to learn more about Wallarm, you can visit wallarm.com. That's w-a-l-l-a-r-m.com. And thanks again for listening.