What is an AI Harness? | Mid-Market AI | Episode 107

What is an AI harness? I know, it sounds kind of naughty, but I assure you it isn't. What it actually is and why it determines whether your AI project succeeds or dies in production is probably the most important thing I will explain on this show so far. You're listening to Mid-Market AI. I'm Ariel Jalali, CEO of Paragon. We're a managed intelligence provider. We put Chief AI Officer-led Data and AI Engineering capabilities inside PE-backed and Mid-Market companies. A few times a week, I get on here and I cut through the noise for the operators and the investors who need field experience rather than conference slides. Quick note on today's episode. I'm going to speak to a few different people at once. If you've got a portfolio company or your CEO and technology makes your eyes glaze over, stay with me. I got you. If you're a PE operating partner or a fund level AI person, there's something in here for you to and if you're an MSP or channel partner, try to figure out how AI fits your practice. Don't go anywhere because the harness is actually your golden opportunity. I've been doing this long enough to know how these episodes can sometimes land a bit arrogantly. So I'm going to say this upfront. I'm not here to tell you that Silicon Valley has yet again figured something out and you need to catch up. I'm here to tell you the opposite. The manufacturers, the home health operators, the distributors, service companies, the companies and private equity mid market portfolio, they're already living these buzzword. They built exception routing that runs without a human touching it. They have inventory alerts that fire purchase orders automatically. They have care scheduling logic and staffing scheduling that matches staff to customer and patient needs across dozens of variables. They didn't call it AI infrastructure. They didn't call it a harness. They just called it Tuesday. Silicon Valley is in a theoretical tower right now wondering what enterprise AI looks like in the real world. You are the real world. They need you more than you need them. So when I use terms like harness governance layer, optimization API, I'm not bringing you new ideas. I'm just giving you the vocabulary for what you already built or are in the process of building. So you can do it more intentionally at scale and with governance. And so you can tell that story to an acquirer who will pay a premium for it. You are the men and women in the arena. This episode is your briefing. Use a pattern around these AI buzzwords and once you see it, it becomes hard to unsee. Every single one landed on your desk as a board agenda item before anyone explained what it actually meant. The pressure to have an answer always came before the tools came to form one. It started with generative AI or Gen AI. The moment the general public got their hands on chat GPT and they realized that an AI could create amazing things, not just sort things into buckets. Someone came prompt engineering, the craft of how you talk to a model, then context engineering, not just what you ask of the AI, but what you load into the model's memory before you ask it. Then agent AI and AI agents models that can actually take actions, run tasks, call other systems, do things, not just say things. And now we're at the harness. Every one of those was real. Everyone was overhyped for about 18 months before just became how things work. The harness is going through that cycle right now and the difference is the harness is the actually the one that determines whether any of the others actually work. It's the most practical concept in AI right now and it has the worst name. The term came from a guy named Mitchell Hashimoto who built Hashikorp and created Terraform. He published a post describing a habit he had developed while working with AI agents. Every time an agent made a mistake, he engineered a permanent fix into the agent's environment. We call that engineering the harness. Within weeks open AI and anthropic both published pieces, expanding on the idea, one engineer's blog post led to an industry vocabulary and just under a month. One quote reframes this entire conversation and it comes from Andre Carpati, the former head of AI at Tesla, who co-founded open AI. I feel he's one of the most credible voices in this field. He said this in March. The customer is not the human anymore. It's the agents acting on behalf of humans and this refactoring will probably be substantial. Let's sit with that for a second. We will come back to exactly why this sentence makes the harness non-negotiable. The cleanest formula in AI right now, AI equals model plus harness. The model is the AI, the brain that raisins and generates. The harness is everything else. I'll be honest with you. I've been trying to give people a clean stable definition of the harness and honest answers this. The definition keeps moving. Six months ago it was the orchestration layer around the model. Then it absorbed the runtime. Now it's absorbing the cloud infrastructure itself. Every vendor is rebranding their piece of it. So instead of what it is, here's what it does because that part doesn't change. The harness is whatever sits between your business and the raw model, making sure five things happen. The data gets in. The write outputs get out. You can change any piece of it without rebuilding everything out. Five functions. One, governance. Two, routing. Three, memory. Four, integration. And five, control. If something in your AI stack is doing those five things, well, that's your harness. Whatever anyone's calling it this week. If nothing is, that's why the demo worked in the production environment didn't. That's why the board is asking questions that you can't perhaps answer yet. Think about a new employee, brilliant, capable day one. They don't know where everything is. You wouldn't hand them the keys to your production database, giving them no training, no guidelines, no manager or mentorship and say go figure it out. But that's exactly what most companies are doing with AI. They wire the model directly into their data, point it to a task, and then they wonder why it doesn't work reliably. The harness is the onboarding, the structure, the context. This is the manager who sets the rules, routing is the team who leads and assigns the right task to the right person. Memory is the institutional knowledge they build over time. Integration is making sure that the work actually lands somewhere useful and controls the ability to redirect them, correct them, or swap them out if something better comes along. Five functions. One job, which is to make the AI reliable inside of your real business. Now back to Carpathy, because I think it really changes the stakes on everything. Automated traffic on the internet is growing now at eight times faster rate than human traffic. Eight times. The Cloud for CEO put it this way. If a human task visits five data sources, the AI agent is doing that same task visit times 5,000. By next year, bought an agent traffic is projected to exceed all human traffic online. So what does that mean for your company? Your systems are increasingly being used, not by your employees or stakeholders who are human clicking around, but by AI agents acting on behalf of humans or teams. Running at machine speed at machine volume, mostly silently, and nearly half of enterprises right now, 48.9% are completely blind to that. Machine to machine traffic, they can't monitor what their agents are doing inside their own systems. They can't tell a legitimate agent from a malicious bot or a swarm. That's not a model problem that's a missing governance and control function to out of the five big ones. When a primary user of your system is an AI agent, instead of a human, the harness doesn't just become more important. It becomes the only thing standing between your operations and complete invisible care. A human makes a mistake, you catch it. An agent on agent swarm makes a mistake at machine speed at machine volume with no one watching it. And nearly half of companies have no idea that it's even happening. For the port co-CEO, the harness is why your AI experiments haven't turned into fruitful results, not because the AI is bad, more because nobody built the structure around it. For the PE operating partner, the harness is the difference between the demo that works, and a system that runs reliably at 2 in the morning when nobody's watching in a measurable way. And for the MSP channel partner, governance and control, two of the five big functions are exactly where you already live. The harness is your natural expansion. Let's talk about single player mode versus multiplayer mode. This frame is going to matter enormously for mid-market operators over the next 18 to 24 months and onwards. Today when most people use AI at work, they're in single player mode. You have your agent, it helps you do your job. You ask it to draft something, summarize something, research something, one person, one model, one conversation, or a number of chat threads that you're having one-on-one with your AI agent. Carpoth is not in single player mode anymore. He said that in March, he hasn't typed a single line of code since December. He runs 20 AI agents in parallel. He's not using AI, he's orchestrating it. That's what multiplayer looks like at the frontier. In multiplayer mode, your department has an agent. Not you personally, your entire team does. That agent coordinates work across the whole group, knows what everyone is working on. It routes tasks, it escalates exceptions, and any person on the team can step up and orchestrate. It's not one person's tool anymore, it's really the team's operating system. If we extend that one level further, your sales agent coordinates with your finance agent. Your operations agent talks to your supply chain agent. Agent's orchestrating agents, that's the agent to enterprise. Single player mode is forgiving. If your AI agent gives you a bad answer, you can catch it. You're in the loop. The last radius of the mistake is relatively small. Multiplayer mode is not forgiving. When agents are coordinating work across a team, routing decisions. triggering actions in your ERP, sending outputs to your customers, the blast radius of a bad output is enormous. Gartner is already saying that over 40% of agentech AI projects are at risk of cancellation by next year, specifically because governance and observability aren't in place. That's the multiplayer cliff. Rush to multiplayer without a harness and you're the cautionary tale. The harness is the governance layer that makes multiplayer safe and without it multiplayer mode is chaos with the budget. With it, it's your competitive advantage. For the PE operating partner, the companies in your portfolio that get to multiplayer mode safely with a real harness underneath are the ones that show operational leverage that AI promises. The number that should reframe every AI conversation you have with your board, global investment in AI has crossed over 200 billion. That's probably going to change by the next episode. Things are moving fast and only 6% of organizations report meaningful bottom line impact. Nobody has published a study that says the harness is specifically why. But every time we get a call that says that an AI pilot is going sideways and they need our help, we see the same thing. The AI going into the model was wrong. The outputs went nowhere. There was no exception handling. Nobody owned the environment. Nobody defined what failure looked like. Nobody built the controls. That's not the model failing. That's nothing around the model. All five functions or some portion of them missing at once. The pattern is always the same. The POC works in a controlled environment with a single player, mode developer with clean sample data, someone watching carefully, then the model performs beautifully. Leadership got excited. The board slide looked great and then they tried to roll it into production and everything falls apart. What they didn't account for, the POC typically costs just 15 to 25% of what production actually cost. The demo proves the idea works. The demo proves it works reliably at scale with real users on data that's messier than the sample set. Most companies scope for the demo and skip the 70% of the hard problems. The data cleanup, exception handling, integration, governance. The data problem is real and it's expensive. 68% have failed AI projects under-invest in data foundations. They discover quality issues and average of 5.2 months into development. The budget is committed and the board is excited and the vendors are paid. When they finally address a data remediation cost, 2.8 times the original project budget. Nearly three times on top of what they already spent. For an establishment market company, Azure Shop, multiple ERPs, data accumulating for 15 plus years across systems that don't talk to each other is not a hypothetical. This is just another day. For the manufacturers and distributors in PE port codes right now, there's a specific version of this that is particularly brutal. Terriffs broke demand planning, not the people, the cycle time. Supplier costs shift overnight, freight lanes reprice weekly, the forecast your planning team built on Monday is wrong by Tuesday. One large retailer reported costs up 20% year over year from tariffs alone, many more. Traditional demand planning assumes stable inputs for months at a time. That world is gone. The company's surviving margin compression are the ones whose data is watching signals in real time, modeling scenarios, routing decisions before the window to act closes. That's the data latency problem. We call that the typical thermostat problem in terms of system archetype challenges. It's really a harness problem when you get down to it. The second failure mode is one nobody talks about because it feels embarrassing at hindsight. Nobody gets fired for choosing GPT or CLAWD. That's the safe call, the brand name. The thing you can defend in a board meeting. It's the same instinct that drove people to say nobody gets fired for buying IBM for 30 years. But at production volume, you pointed a front to your model, the biggest, most capable, most expensive model available at something narrow. Like classifying invoices, routing support tickets, extracting fields from a contract. A task that's genuinely simple once the data is clean, and you're paying front to your model prices to do it. At scale, that can be $50,000 or $100,000 to your monthly cloud bill. On domain-specific tasks, a fine-tuned small model, or what we call an SLM, achieves 94% accuracy on contracts versus a frontier model is 87%. The smaller model is not just cheaper, it's more accurate on the narrow job. And it's 10 to 30 times cheaper to run. Small language models now dominate six out of eight major enterprise use cases. So you didn't pick the wrong vendor, you just picked the wrong model size. And nobody built the harness routing function that would have told you that or giving you the ability to swap when you figured it out. Third, and this one connects specifically to every Microsoft shop listening right now. Most of you are an Azure M365, probably have co-pilot running or in conversations about it. And that's the right foundation. Azure is actually ideal for a harness. Your data stays in your tenant, your security posture is already there, your IT team knows the environment. The question is not whether to use Azure. The question is whether you've built the harness layer inside it. A co-pilot license is not a harness. Co-pilot is a model with a thin wrapper. It doesn't govern your data across systems, and it doesn't wrap between models. It gives you the full auto trail, and you don't own the architecture. It doesn't govern your data across systems. It doesn't route between models. And it doesn't give you the full auto trail, and you don't own the architecture fully. The harness is what protects your Microsoft investment. It lets you take advantage of wherever Microsoft goes next or the underlying tools or models. Without rebuilding from scratch, every time they ship something new. There's also a data sovereignty point that PE operating partners need to hear. When you run a model API wired directly into your data, outputs going wherever, you're functionally letting a third party touch your operational data. Every query, every document, every context load, that's your business intelligence going through someone else's infrastructure. Acquire is just starting to ask about AI architecture and technical due diligence. We do a lot of it. And a company with a clean harness has a defensible answer. A company with a naked model API wired to their production database, not so much. Two examples from actual portfolios, especially food and beverage manufacturer, every batch run, every QC flag, every shift change historically meant somebody reading a report, deciding what to escalate, rounding the exception manually, a full time job, just managing the queue. Then they built the system that monitors production line data, detects the anomaly, cross references specific spec tolerances and routes, the exception to the right person with context. Before it becomes a line stoppage or recall, they didn't call it a harness. They call it their exception management system. They built it because their margins demanded it. A home health operator, two of the most painful workflows in the sector, both already running without humans in the loop, claims management where agents monitor payer portals catching the nile the moment it hits, pulling the patient record, identifying the fix, resubmitting within hours instead of days or even minutes. That's cash flow. That's days sales outstanding. That's a metric the P.E. sponsor tracks weekly care staffing. A caregiver calls out patients needs change. Once a certification lapses, the system reroutes automatically, surfaces the exception to the human that need human intervention, keeps the schedule intact. Neither company called what they built a harness. They just called it survival. Now we have a word for it. Something is happening right now that almost nobody in mid market is paying attention to. Cloud code and open AI code X started as coding tools. Your developers probably think of them that way still. A really powerful assistant that writes code, but that's not what they are anymore. They are full agentic platforms, model, harness, and now getting into cloud infrastructure where your agents can actually run all bundled together, all under someone else's roof. Thropic builds the model and the topic builds the harness around it. Now anthropic is providing the cloud environment with the agent executes. Same story at open AI with code X. They started the model layer moved up into the harness layer and now they're moving down into the infrastructure layer going vertical fast. Everything is much of the stack as possible. You don't blame them given that their startups in hyper scale mode. Microsoft is running the same play through Azure, although they have a more diversified data platform play, not just the AI model harness. Sales force just announced edless 360. It's the same play from the enterprise sass side. Jensen Wong at Nvidia has been saying for over two years that the application layer is the one and the five layer cake that he says captures the most value. Everyone just heard him. Everyone is now acting on it. The mid market risk your port code developer got approved to use cloud code,thropics infrastructure. Not your Azure tenant, not your governed environment. There's with their audit trail or no audit trail at all. That's not a criticism of cloud code or code X. They're excellent tools. Your people should be using them. The question is where they run and who owns the layer around them. When an anthropic changes their pricing and they have and they will, when open AI changes their terms and they have and they will, when everyone of them gets acquired or better tool emerges or your P e sponsor asks in due diligence who owns your AI. infrastructure, what is your answer? If your agents are running in someone else's cloud, inside of someone else's harness, they do not you. This is where the two-door principle becomes a business decision out of technical one. The winners in AI over the next three years won't be the ones who picked the right AI model. They'll be the ones who can change their mind in 2026 and 2027 next year every year as models keep improving, vendors keep shifting and prices keep changing. Two doors mean your data, your environment, and your architecture belongs to you. The models that run inside are interchangeable. For example, Microsoft updates co-pilot you slide it in. A better model comes along you slot that into. You have a workflow that needs several models involved for certain sub components of that workflow. You can do that as well. If a smaller specialized model does your invoice or order classification better and cheaper, you can swap it in or swap it out without rebuilding anything. One door is always left open. That's not by accident. That's good design philosophy. What we had paragon have been building and calling it the reference architecture. Technically we call it the MIPSRA which stands for Managed Intelligence Provider Reference Architecture. I know. We need better names in AI. But this is what the AI industry is now calling the harness. It's just a buzzword that caught up to us. We didn't build it because of a trend. We build it because clients kept needing it. Five functions. Governance, routing, memory, integration, control. Four layers in an actual architecture. Layer one is your data store. That says your memory. Your AI is filing cabinet except that it's yours. It's organized and only you have the key. Not the vendor's cloud, your Azure environment or AWS bedrock. Governed inversion. That's your institutional memory. The thing that lets your AI get smarter over time because it remembers what happened last quarter or last month. Not just what's in this conversation or chat. Azure blob, Azure SQL, Azure fabric. It's already there. The harness is what governs it and how AI talks to it. What's important about the operational data store is more the uses of the data rather than sources of the data. Unlike a data warehouse where the goal was to get every pipe flowing in and build these perfect representations of data, the operational data store is actually built one automation at a time or what we call minimally viable operational data. Layer two is the analysis engine and this is the part that handles governance. It's a gatekeeper that cleans and prepares your data before it even touches AI. The most skipped layer in almost every mid market deployment that I've seen. Everyone wants to jump into the fun AI model part, but 68% fail projects never even assess their data before committing dirty data in wrong answers out. No matter how smart the model is the analysis engine is also your security layer. It sanitizes inputs. Nothing untrusted ever gets near the model for MSP partners. This is where you plug in you already governed the data access. This is a natural extension. Layer three, what we call the model engine or the model jukebox handles routing the ability to pick the right AI for the right job and switch when needed. Not every task needs a frontier model. Some task need Claude, some need codex, some need an open source model, some need an SLM that you've tuned. Some need a smaller specialized model running inside your own Azure environment, no external calls, no token costs. The jukebox is model agnostic by design. This is where two door principles live inside the architecture. You turn your dial, you don't rebuild a system. Claude code slots in codex slot in. They run inside your governed Azure environment, your tenant, your security posture, your auto trail, not anthropics, not open a eyes, full power of those tools within your own architecture underneath. There's also an availability concern. In the last two months, we've all seen these models go down randomly. That's unacceptable for production environment. You also want to have some failover models just for redundancy and availability purposes. Layer four, the automation API or what we call the optimization API, it gives you integration and control. The one control door that outputs leave through, we call it shipping the goodies. Your AI makes a decision or produces a result and it goes exactly where you say it goes. You can go into your ERP, your SAS, your legacy system, your BI dashboard, your email system, your workflow, not everywhere at once. One gate, one audit trail. This is also what keeps your humans in the loop. High stakes outputs route to human for approval before anything happens, especially in compliance, driven environments. Remember that nearly 50% of the companies blind to their own agent traffic is the answer to that problem. Every output is tracked, every action is logged, you own the record. That's what we call MIPRA deployed inside your Azure or AWS environment. When we're done, you own it, swap the models in the jukebox without rebuilding your integrations, change vendors, bring in a different team, the architecture is yours. One more thing, model drift. As agents run longer, more complex tasks, hundreds of steps, maybe even millions, multi-day workflows, models can start to go sideways, following instructions less precisely and taking shortcuts. Without a harness, you don't even know what's happening. Without a harness, you detect it and correct it before it even touches production. Building a harness requires skills that are generally hard to hire and retain, and most mid-market companies don't have them. A harness needs a data engineer running your pipelines, a harness engineer, a role that barely existed 18 months ago, sitting at the intersection of platform engineering, ML engineering and security architecture. A security specialist who understands this specific context and an integration engineer who connects outputs to your actual business systems. Four specialized roles for a company that just finished writing a job wreck for an AI engineer singular. That's a sobering list. BCG's 2026 research described it well. Most companies are stuck in deploy mode, handing out AI to licensing without changing the workflows. This is like giving a Formula One card to someone driving on city streets. The power is there. The infrastructure just can't support it. That's the gap. Not the model. Everyone has a model now. The gap is the harness, and the harness requires a team and the skill profile that most mid-market companies can't realistically build internally fast enough on any reasonable timeline. For channel partners and MSP partners, this is your opening. The harness is to manage AI infrastructure service your clients need and cannot build themselves. At Paragon, this is what MIP delivers. It's our managed intelligence provider solution. The team and the architecture without the hiring cycle. It's a CTO-led harness team deployed inside your Azure or AWS environment. You own the architecture. You've built the institutional knowledge. If you want to have your internal team take it from there, you totally can. Here are three questions that tell you immediately when you have a harness problem. One, where is your AI's data come from and who owns that environment? If the answer is the vendor's cloud or the AI platform cloud or I'm not sure, that's a flag. Two, if your AI vendor change their pricing tomorrow or shut down or went offline, how long would it take you to rebuild or even notice? If the answer is a long time or we'd have to start over, that's a flag. Three, is there a human who reviews outputs before they trigger an action in a production system? The answer is not really. That's a flag. Three flags mean you're running on a model with no harness. Find for single player experimentation or hacking. Not find for anything you're counting on to run the business and definitely not find when you're moving towards multiplayer mode in agents, not humans, where they are the primary users of your system. At Paragon, we help companies go from three flags to zero using a crawl walk and run approach. Get the data foundation right, then the routing layer, then the agents. Mip is how we deliver all of it. Links are in the show notes if you want to talk through what this looks like for your come. The buzzword cycle, generative AI, prompt engineering, context engineering, agentic AI, harness engineering, each one real, each one eventually table stakes. The harness is the one that determines what the other ones actually work inside a real company at scale or not. $200 billion invested so far and growing exponentially, but only 6% meaningful impact. The money seems to be going in. The harness is why the results aren't coming out the other side. The food manufacturer whose quality exceptions route themselves didn't need a keynote to tell them it was a good idea. The healthcare operator whose claims management process runs without a human didn't need to read about it in an industry or VC newsletter. They built it because their operations demanded it. That's you. You've been building the harness without knowing what it's called. Now you have the vocabulary and the architecture and now also the team available to scale what you've already started. Get the harness right and the model almost doesn't matter. Get it wrong and the best AI in the world won't save your POC from going kaput in production or your agents from running wild in the dark. Next episode we're going headless. Sales force just rebuilt their entire platforms or agents can run it without a human ever touching the screen. Open AI dropped the model native harness. Carpati said it himself. Apps shouldn't even exist. It should be just APIs and agents talking to them directly. The market caught up to that idea this week. That's episode two of the series and we're calling it headless. Send us a one person, a portco CEO and operating partner who's trying to figure out why their company's keep hitting a wall or an MSP channel partner who knows that their clients need AI help but isn't really sure where to start. I hope you're outside. I don't know, walk right now, go enjoy it. And I guarantee you, the AI space will have changed by the time you get back. Cheers. (upbeat music)