#38 - The Basics of Workload Identity

How often have you seen your on call support lead looks like they haven't slipped days or included in an unavoidable outage? You get notified of the outage, you join the call and they say payments are down again. Now, you could assume that payments are important, so there has to be a big reason for it. Maybe your cloud provider is having a global outage or maybe your data center burned down or because some over-enthusiastic intern robbed the entire production database. But more often than not, this is not the case. Firstly, it's because of a string, a 64-character API key that was rotated or revoked in the middle of the night and took down half your microservices with it. It's written in the post-mortem, someone asked the question that should be offensive to every modern identity architect. Why are we still running a multi-million or in some cases multi-billion dollar business on copy-pasted secrets in MLFI? And today, we are going to answer that question. This is the identity navigator and I am Rohit. And then this episode, we are going to talk about workload identity. What it is, why it matters, how it actually works under the hood, and why it's the only sane way to secure a word full of microservices, Kubernetes bots, CICD pipeline, and increasingly autonomous AI agents. By the end of this episode, our goal should be simple. If someone drops SPIFI, Spire, Workload Identity Federation, or SLSA in the meeting, you will not only understand it, you will the one explaining it. So when I was growing up an identity, there were these two peoples who showed up in all the examples. I'm not sure if you have heard of them, but they were called LIS and Bob. Honestly, at one point, when I was thinking about launching this podcast, I was about to name this Chronicles of Alice and Bob. And I thought, you know, I will try to explain identity fundamentals using Alice and Bob and it's as an example. And I seriously considered that. But then I realized podcasts, you know, they don't show up in feeds or things like this. So that would have been too clever for my own good. Like you don't search an identity podcast and put in Alice and Bob in the search field. So that is how the identity navigate came about and not the Chronicles of Alice and Bob. So back to Alice and Bob. In the examples that I was taught on, Alice always forgot her password and Bob always had the right access and some of the entire field of identity security and management was built around their extremely dramatic office life. Alice was the user and Bob was the human and also sometimes service. He was also the server, he was the machine and Bob sometimes was the thing in the basement. Nobody talked to unless it needed to print something. The word this worked and that worked when identity mostly meant humans logging into apps. But all of us have all of us know the word has changed. Now the companies aren't just staffed by people, it's staffed by software. So software, calling software, pipeline, calling clouds, pods, calling databases, agent, calling tools. So and a thousand little Bob's carrying the business on their backs. So let's talk about the real Bob here. Bob is a loyal company employee. Bob shows up early, drinks, back off, he answers tickets and never complains. He reads the run box, he gets phased at 2 a.m. And he somehow still says, I'll take one for the team. Bob used to get by with a client secret or an API key or a service account credential or a token stored in vault but Bob is tired and tired Bob is carrying a secret around in a work where secrets leak. So let's start from the basics. In the land of identity, we have human identity, that's Alice, that's Bob, the person, that's Charlie, that's Rohit. The login with user name, password, MFA, maybe a phyto 2 key, they get sessions, tokens and rules but modern systems are dominated by humans, they are dominated by software talking to software. That might be a microservices calling another microservices, a Q1 at the spot calling a database or a GitHub actions pipeline deploying infrastructure into Azure or an AI agent calling an internal recommendation API. All of those are workloads, non human software prospects that need to authenticate and be authorized to access something. So here is the first core definition for today, a workload identity is the identity of a software workload like an app, service, container, script or pipeline used to authenticate it to other systems and APIs. So instead of who is this user, the question becomes who is this workload really, right now on this machine in this cluster and that really right now is where things get interesting. So before we go deeper, let's clean up a bunch of terms that people often mix together, non human identity management or NHIM. Now this is a broad governance umbrella. It must ask what non human identities do we have, who owns them? What are they allowed to access, when do they expire, are they still in use? That's the inventory and governance layer. So if identity was a city, NHIM would be the records office. Now you have all heard me or read my thoughts around non human identity management. This is a good concept for talking to the executives. This is a good broad umbrella term that can be used for governance. But when you start to design and architect your platforms for non human identity management, the things get tricky because every non human identity is drastically different. And this is what we called out in our previous episode of identity categorization done repeat. There are so many different type of identities, but they should be differentiated or categorized based on their behavior. Otherwise we just create taxonomic debt, and I think NHIM is actively creating taxonomic debt in many environments today. So if you haven't listened to that episode, I would really advise you to go back and listen to it. I'm sure you would like it, but we are here. And now we talk about API keys. The old school witness Bob proves who he is, you give Bob a secret string, Bob presents a string. The system says, yep, that matches. It's simple. It's familiar. You know, honestly, this is basically from 2000 years. This is the way to authenticate each other. But now we realize that it is also a terrible long term answer. Because if Bob knows the secret, anyone with the secret can be Bob. At least just to Walt, Walt is the clever butler. It's two secrets. It brokers secrets. It's hands out credentials and it does a lot of heavy lifting. Walt is useful, but Walt is not same as identity. Walt manages secrets. Workload identity minimizes the needs for secrets in the first place. And then there are cloud native workload identities, AWS, IRSA, Azure workload identity, GCP workload identity. These are platform specific ways of doing the same underlying thing. The platform says, I know which workload this is. I know where it is running. I can issue a short-lived credential. You don't need to carry a long-lived secret around anymore. And then. comes the workload identity. This is the modern model underneath all of it. So instead of asking Bob to carry a secret, it asks, who is Bob really? Where is Bob running? What code is Bob executing? What runtime context is Bob in? And can the platform attest that Bob is the right Bob and that is the shift. So you should ask this right now if you aren't already. Why should you care? If you have built anything in the last decade, you probably already implement workload identity and some of you must have done it in the worst possible way, I know I have, because it starts inocently. You put an API key into a configuration file. Then into a CICD variable, then into a secure password manager that everyone screenshots into slags or teams. At small scale, it feels fine. And then your architecture grows 5,000 pieces, each with at least a client ID and a client secret. Let's go into the 5 figures of credentials to protect, rotate and audit. So now you have token theft risk because you must have heard the podcast episode about the bearer tokens and the bearer tokens can be replayed until they expire. You also have a supply chain risk because if CICD is compromised, attackers can steal secrets and mint their own tokens. And you have obviously operational pain because teams spend real time and money rotating secrets and fixing outages caused by the expiry. And cloud providers basically admits this in their own docs. Storing long lived credential secrets is brittle, risky and operationally heavy. So the why should we care is simple. At scale, manual secrets for workloads do not work. There are too many, too long lived and too easy to leak. As we move into a word of ephemeral containers, serverless jobs and AI agents that can spawn and disappear in seconds trying to pre-assume static credentials to every possible workload becomes literally impossible. We need something better. And that something better is workload identity. You would also hear concepts about authorization layers and some of the new startups creating that as a session broker. And we will deep dive into them in one of the future episodes. But in this one, we really want to talk about the workload identity and did not wanted to dilute the concepts here. So you know, in addition to Alice and Bob, what else did you always see as an example in your identity books? For me, it was a fictional company called Acme Corp. So because we are living in the nostalgia today, let me tell you a story about this fictional company called Acme Corp. Acme starts with a monolith, a big Java app, talking to a database. To call the third party payment API, they store an API key in a properties file, ship it and done. This is the first era, era of hard-coded secrets. You authenticate by knowing a magic ring and if anyone copies it, they are used. Then Acme modernizes. They adopt microservices and Kubernetes. Good for you, Acme. And now they have 200 services, each needing secrets for databases, queues and external APIs. And so they modernize again. Move secrets into the world, inject them as environment variables or mounted files and rotate them more often. I don't get me wrong, that's better. But still, long-lived credentials, rotation burden and any process that can read the secret can impersonate the workload. But then Acme gets cloud-native, it starts using OAuth client credentials. So now each service has a client ID and a client secret. They request short-lived access tokens from an authorization server. And they use those to call APIs. Security is better. But the route secret is still as static strings somewhere, you're client secret or certificate. If attackers steal it from CI, vault or the code repo, they can continue minting valid tokens until you rotate and detect. You know, in many, one of the earlier episodes, I remember we talked about the secret zero problem and this is exactly what I'm talking about here. That's a total on a total on a total. But then Acme still has more. Acme goes all in on Kubernetes and serverless. And now they have started deploying AI agents that can spawn other agents on the fly. So now the workloads are ephemeral, auto scaled and created dynamically by other workloads. And for AI agents potentially changing behavior at runtime. So you can't pre-register every possible agent in a directory and you can't safely ship secrets to every pod that would exist. And that is a crisis that forces evolution. Static credentials don't scale to dynamic and agentic workloads. I can use a different model. And instead of proving identity by what secrets you know, you prove it by where and how you're actually running right now. And that is a jump to a station based workload identity. So we will talk about what good looks like. But before we get there, before we get into spiffy inspire, let's just talk about what it is all about. If I say this Kubernetes pod should be allowed to call the payments API, I want to be confident. It's running the image I built and not something an attacker so I've been right. That is a reasonable task and it's running in the right environment. Again, it's got its credentials automatically from the platform based on those facts and its credentials are short lived and automatically rotated. So if an identity executive or security executive asking you to do all of these things, these are pretty rational, right? And that is the promise of a more workload identity. Strong cryptography, platform issued credentials, short lived tokens and binding to runtime properties like namespace, image digest and service accounts. And this basic pattern in their workload identity and federation docs. This is this is also covered in micros of docs and some of the other bad other vendor. This is the basic pattern. That is, prove the workload through trusted runtime context, then exchange that proof for access booker without a static secret. So fundamentally, workload identity is equal to automatically issued, short lived, strongly bound credentials for software based on runtime attestation. Now this is a mouthful. That is why I wanted to walk you through of, I was just asking for it's running the image I built, not something an attacker swapped in, right? So stronger cryptography and it's running in the right environment and it's got credentials automatically from the platform. So platform issued credentials and the credentials are short lived. So short lived tokens, right? All of those things made sense. But this line itself might sound heavy. Let me, let me say it one second workload identity is equal to automatically issued, short lived, strongly bound credentials for software based on runtime attestation. Sounded heavy, right? And I know many of you are experts and you would say now, right? But for, for my purpose, it does, it does sound heavy and that is why I wanted to talk about it. It's a rational ask. So let's talk about spiffy and spire. And these are the names people love to drop in architecture meeting. So from now on, you would be in that meeting. So spiffy stands for secure production identity framework for everyone and it's the standard. And spire is the open source implementation of that standard. It acts like the identity provider for workloads. So think of spiffy as the passports standard and spiral. the passport office. And at the heart of Spiffy is the Spiffy ID which looks like a URL. It has the trust domain and the path identifies the workload. So how Spire works? Remember Spire is the open source implementation of the Spiffy standard. So a Spire server acts as a coordinator and signing authority. They're the Spire agent on each node. And then Spire also uses workload registration that maps selectors to identify identities. So workload registration maps selectors to identities. And Spire uses the workload API to issue SVIDs or SVIDs. So let's talk about the flow because it can get messy real quickly. So Bob starts up, and Bob is a service. Bob starts up. Bob asks the local Spire agent who am I? I wish I could ask somebody who am I, but we will stick to the topic today. So Bob starts up and Bob asks the local Spire agent who am I? And the agent checks runtime selectors. For name, namespace, service counts, image digest and process context. And the server checks the registration entry. And if the workload map is its guess and SVID which may be at 509 or jot. Then Bob can use this identity to do MTLS, talk to other services or exchange it for cloud credentials. So this is a two step beyond. Here is a secret. Please don't leak it. So how do you translate into the cloud? Because that is where I believe most of you are working in today. So Microsoft, Entra workload identity federation. Let's an external workload like GitHub actions, Kubernetes on another trusted IDP, exchange a token for Azure access without needing a client secret. And that is huge because instead of managing a static credential, you can now configure trust relationship. You remember the federation concept for the human domains. So now, so this is you configuring trust relationship. You have an issue or you have a subject and you have an audience. Now, AWS, the same basic thing with EKS and IRSA, a Kubernetes service accounts can be mapped to AWS permission without a long lived secret. Sounds nice, right? And then GCP has similar patterns with its own workload identity model. So the common idea is this is respective of which cloud platform you are on. The platform issues or trusts are token. Token is short lived. The workload proves its runtime identity and the cloud exchange is that proof for access. And that is workload identity in action. But wait, still have more things to discuss in this episode. So when people say cloud workload identity, they are usually talking about platform native federation. When people say workload identity, they often mean the broader runtime trust model. So don't get confused on how these terms are being used in your meetings normally. Now, a lot of these things, you would say, I don't hear them in my identity and architecture meetings. I wrote with I hear about what an API keys and cloud services. So let's get a little explicit here because this is if this is what you hear in real life, this is what we should talk about. So we spoke about nonhuman identity management that you hear. We spoke about API keys. We spoke about fault, right? But I don't think we spoke about AI agents, right? And how can we be in a podcast at about 25, 26 minutes and do not talk about AI agents? Right? So obviously, I will not deep dive into AI agents today, but you know, like an ordinary service is predictable and AI agent is not. An AI agent can typically decide what tool to call. It can change its plan, spawns up agents, use new context and in some cases adapt as it runs. And this changes the identity problem completely, right? So that is why AI agents is a part of workload identity, but there is more to it when it comes to like you'll have to design more controls. So if you don't assign identity to an agent and you get chaos, and that is why I really like agent 365, the new solution from Microsoft that went into public preview. I think they start of this month may 2026 because you need to assign identity to an agent so that you can do attestations. And you can assign accountability because without an identity, you can't tell which agent made which call, you can't scope access properly, you can't revoke one agent without breaking others and you can't audit behavior, you can't govern delegation. So yes, hurry to believe assigning identity to agents is crucial because an agent with identity is just a floating trust problem with a language model attached and nobody wants that. So based upon my, I would say still not so much, but I'll just be playing around with Microsoft agent 365. So the direction simply makes sense to me. I know the product has so much more scope to develop. I think it still is like the first version or V1 of it also for $15 per user per month. It is, it is not enough thing for small to medium businesses and mostly for an enterprise. But the direction makes sense to me agent needs identity, life cycle governance and policy, just like human users and workloads do. And I'm pretty sure this is the direction the industry is heading, right? And that is why the identity uncertainty principle is such a powerful idea. And the point that it makes is simple, but I would say it's brilliant. As the point that identity uncertainty principle which the foundation law of agent AI authorization makes is brilliant because, you know, it simply says as an AI agent becomes more autonomous, static identity confidence decreases. And I really like this because it gets it from a problem and not a solutioning point of view. Like what is the first thing that that is being told to us? Like define the problem in in terms of what the business should care about. You cannot lead in with a technical concept. And that is where I think identity uncertainty principle does a lot of justice to how identity practiceness can frame this problem, right? Because now we are saying your identity or certainty of the identity itself is diminishing exponentially as your agents are becoming more autonomous. And that would lead into the conversation of, okay, I don't want that. So what should I do? And then you can have these additional controls and CIEP and shared signal framework and all the dynamic controls over that authorization layer come into. Because the more the system learns adopts and spawns new behavior, the less a one-time startup token tells you about what it is doing right now. And it also proves that the point in time authentication is not enough, right? So you can start with that identity and identity principle and define the controls and can get business by in based upon how assured or what is the level of confidence that they would like their identity. So if point in time authentication is not enough, you would also need any statistician and continuous monitoring and dynamic relocation or kill switch and request by policy evaluation. And that is why workload identity is necessary. But not sufficient for the most advanced AI system. Because for systems like these need identity as behavior, identity plus observation and identity plus policy. So let me give you a clean maturity model here. So level one was hard copied secrets. Level two was centralized secret management, right? And I'm intensely keeping it at a high level because you should be able to tell at what level your company operates just by listening to it and not after a six weeks engagement by a consulting firm. So Level. level 3 is oath client credentials and token based authentication and I know a lot of your stuck here. Level 4 is attestation based workload identity. So, your spiffy inspire your AWS IRSA, your Azure workload identity, your federation and your short-lived credentials. And level 5 is your continuous behavior attestation. So, your signed bits, SLSA providers, runtime monitoring with EBVF or or tetragon or salco, policy engines like OPA and just in time relocation. So, for level 4s and level 5 that is why let me add just one thing to it. Level 4 answers who is this workload? And level 5 answers is this workload still behaving as intended and that is a major jump. So, let's briefly unpack the level 5 pieces. Six store and co-sign, right? These are used to sign artifacts at build time. So, you can prove the image came from the right pipeline. And as LSA formalizes the trustworthiness of the supply chain. So, your build is recorded, it is isolated, your source is controlled, your provenance is captured, your reproducibility is improved, right? And then there are also in level 5 you have EBVF or tetragon, falco and key line to monitor your runtime behavior. And they watch this calls and file access and network traffic and suspicious deviations, right? And all of us use P back in some way form of another or opa, right? So, policy engine like opa enforces authorization at request time. So, now your architecture should all come together and it will have a trusted origin, a trusted runtime, a trusted behavior and a trusted decision. And this my friends is the future of secure AI and secure workloads. So, let's get back to Bob. Rob started as a tired employee with an API key, then Bob gets client secret, then Bob gets vault, then he gets cloud federation, then he got attestation, then he gets short lift credentials, then he gets monitored and then he becomes a trusted workload and that is the heerless journey. Bob did not become the hero because he memorized the secret. Bob became the hero because the plate form finally stopped making him carry one. And that is the point of workload identity. So, if you're trying to push this in your own company, here is how I would do it. Start with the pain, show the secrets, Paul, how many secrets, how many pipelines, how many manual operations, how many incidents. And then start with Laurie, great wins, move CI-3 to frustration, move Kubernetes workload to cloud native workload identity. And these are easy wins because the vendors already support them. And then introduce Piffy and Spire and start in Wattluster, don't go big bang, demonstrate registration and attestation and SVD students and issuance and MTLS and workload to workload identity trust. And then prepare for agents. Define identity policy, lifecycle, delegation, monitoring and revocation. And don't wait until the agents are already in production and asking other agents for access to things they should have never seen. So, let's rewind the whole episode. We started with Alison Bob, Alice was the human and Bob was the birth load. And then Bob grew up, Bob became a service, Bob became a pod, became a cloud workload and an AI agent. And suddenly the old word of static secrets stopped making sense. Okay, so we need a new model, non-human identity management for governance. Don't use it to architect your platforms. Watt for secret brokering. Cloud native federation for plate form trust. Piffy and Spire for workload identity. SLSA, cosine EBVF, opa for continuous trust, and agent identity for the AI future or AI present. If you remember one thing from this episode, remember this, workload identity is the modern way for software to prove who it is without carrying a reusable secret like a bad habit from the old world. That is the shift and that is the future. I'm Rohit and this is the identity navigator. And next time somebody says, can we just put the secret involved? You will know exactly what to say. Please let me know. You've felt about this episode. You can always email me or find me on LinkedIn. Both of these contact details I will put in show notes. Thank you so much for listening. This is Rohit, your identity navigator.